ONC RPC version 2 over TCP/IP

This post discusses message structure of the Open Network Computing (ONC) remote procedure call (RPC) version 2. The protocol is specified in IETF RFC 5531. RFC 4506 specifies the C-like data representation syntax used in RFC 5531. RFC 1833 specifies an RPC service (portmapper) used to discover RPC services provided by a host.

ONC RPC in Wireshark
Wireshark RPC dissector

The ONC RPC message structure is defined in the specification as follows

struct rpc_msg {
    unsigned int xid;               /* transaction id */
    union switch (msg_type mtype) { /* message type */
    case CALL:
        call_body cbody;
    case REPLY:
        reply_body rbody;
  } body;

An unsigned int, according to the XDR specification, is a 4-byte unsigned integer value in big-endian byte order. Transaction id is therefore a 4-byte value. Message type is also an unsigned int value. A value of 0 indicates a call, 1 indicates a reply.

Message fragmentation is used over a stream oriented protocol such as TCP. Transaction id is therefore preceded by a unsigned int value that indicates the size of the fragment in bytes. The most significant bit (MSB) of the unsigned int is a boolean value that, when set, indicates the last fragment of a sequence of fragments.

The call body in turn is defined as follows

struct call_body {
    unsigned int rpcvers;  /* must be equal to two (2) */
    unsigned int prog;     /* program identifier */
    unsigned int vers;     /* program version number */
    unsigned int proc;     /* remote procedure number */
    opaque_auth cred;      /* authentication credentials */
    opaque_auth verf;      /* authentication verifier */
    /* procedure-specific parameters start here */

enum auth_flavor {
    AUTH_NONE = 0
    /* and more */

struct opaque_auth {
    auth_flavor flavor;    /* authentication flavor */
    opaque body<400>;

If authentication flavor in use is AUTH_NONE, authentication credentials is an unsigned int value of 0, followed by another unsigned int value indicating an authentication credential body size of 0. Authentication verifier is encoded in the same manner.

A reply is defined as follows

union reply_body switch (reply_stat stat) { /* Reply status */
    accepted_reply areply;
    rejected_reply rreply;
} reply;

struct accepted_reply {
    opaque_auth verf;
    union switch (accept_stat stat) { /* accepted status */
    case SUCCESS:
        opaque results[0];
        * procedure-specific results start here
        struct {
            unsigned int low;
            unsigned int high;
        } mismatch_info;
        * Void.  Cases include PROG_UNAVAIL, PROC_UNAVAIL,
    } reply_data;

union rejected_reply switch (reject_stat stat) {
    struct {
        unsigned int low;
        unsigned int high;
    } mismatch_info;
    auth_stat stat;

Reply status is an unsigned int value, followed by the authentication verifier encoded as explained earlier. A reply status value of 0 indicates an accepted message, which is followed by an unsigned int indicating accepted status (0 is success). A reply status of 1 indicates a rejected message, which is followed by an unsigned int indicating rejection status.

Blocks – string or opaque data, are padded with 0 to 3 residual bytes so that their length is a multiple of 4.


SSL/TLS decryption in Wireshark

Wireshark’s dissector for SSL is able to decrypt SSL/TLS, given the private key in PFX/P12 or PEM format. If you want to figure out whether you’re using the right private key, you can derive the public key from it, and compare its modulus with the first certificate in the chain of certificates sent in the SERVER HELLO.

$ openssl rsa -text -in key.pem -pubout
Private-Key: (2048 bit)


Enable IP multicast routing in Linux kernel

In this post I discuss how to enable multicast routing in a Linux system. It is a continuation to the post Wireless Router with Buildroot and Raspberry Pi, where I discussed how to build a basic Wi-Fi router with a Raspberry Pi. You’ll want to read that first.

Linux kernel configuration

Besides the Kernel modules mentioned in the post(s) linked above, you’ll need a few additional modules.

IP multicast routing and tunneling

IPv6 protocol

Under Networking support, Networking options, enable

  • IP: multicasting
  • IP: tunneling – this is required if you want to use tunneling with mrouted
  • IP: multicast routing and its sub-options
  • The IPv6 protocol

In the absence of IPv6 smcroute fails with an error such as

Starting static multicast router daemon: INIT: ICMPv6 socket open; Errno(97): Address family not supported by protocol
INIT: MRT6_INIT failed; Errno(97): Address family not supported by protocol

IPv6 Multicast Routing

Under Networking support, Networking options, The IPv6 protocol, enable IPv6: multicast routing and its sub-options.

Packet Mangling

Enable packet mangling with TTL target support if you require support for changing TTL values with iptables.

Buildroot package configuration

The following Buildroot packages provide daemons for performing multicast routing. Enable mrouted and smcroute under Target packages, Networking applications. mrouted requires a glibc based toolchain, you will have to enable it instead of uClibc if you want to use mrouted.



Perform build and prepare the SD card.

Setup multicast routing

The following procedure is performed from a root console. I usually use the serial console through the expansion header.

Use mrouted when proper IGMP signaling exists


The default configuration file /etc/mrouted.conf should be enough, unless you want to perform tunneling.

If you don’t have proper IGMP signaling happening, you can still perform static multicast routing using

smcroute -d

smcroute requires a configuration file, which in my case is /etc/smcroute.conf and looks something like

mgroup from wlan0 group
mroute from wlan0 group to usb0

If you don’t have an application and want to use ping to test mutlicast, you can enable ICMP echo responses thus

echo "0" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

You can use ping requests and receive responses from destination hosts

ping -t 10

Note the use of the time to live (TTL) parameter -t. Linux and Mac OS X will set TTL to 1 before forwarding message to the default gateway. You can dump ping messages with TTL parameter using

tcpdump -v host or

Note change in TTL from 10 to 1 in a packet routed through Mac OS X in the following dump

14:49:19.642140 IP (tos 0x0, ttl 10, id 0, offset 0, flags [DF], proto ICMP (1), length 84) > ICMP echo request, id 4113, seq 92, length 64
14:49:19.642190 IP (tos 0x0, ttl 1, id 32573, offset 0, flags [none], proto ICMP (1), length 84) > ICMP echo request, id 4113, seq 92, length 64

If all is well with the routing daemon, IP variable /proc/sys/net/ipv4/conf/{all,interface}/mc_forwarding will be set to 1.

Some other files offer useful hints related to multicast routing. The following lists interfaces where multicast routing is active

cat /proc/net/ip_mr_vif

This lists multicast routing cache entries

cat /proc/net/ip_mr_cache

When using static multicast routing with smcroute, routing will work only when TTL is greater than 1. If the downstream hosts are transmitting packets with TTL at 1, you can use iptables to set TTL thus

iptables -t mangle -A PREROUTING -i wlan0 -j TTL --ttl-set 64

I’ve also had to wait a while after executing smcroute for NAT to kick in, so that source IP address is translated to address of interface on the destination network. Note the change in source IP address in a message sequence captured using

tcpdump -v -i usb0 host or
03:03:25.393056 IP (tos 0x0, ttl 63, id 16103, offset 0, flags [none], proto UDP (17), length 41) > UDP, length 13
03:04:22.277348 IP (tos 0x0, ttl 63, id 8095, offset 0, flags [none], proto UDP (17), length 41) > UDP, length 13

IP multicasting

IP multicasting is used to target a group of hosts by sending a single datagram. IP addresses in the range through are reserved for multicasting.

To find out which hosts on your subnet support multicasting, try


Here’s a Node.js code snippet that sends UDP datagrams to multicast group at port 8001

var dgram = require('dgram');
var s = dgram.createSocket('udp4');
var b = new Buffer("Hello");
s.send(b, 0, b.length, 8001, "", function(err, bytes) {
  console.log("Sent " + bytes + " bytes");

A quirk observed on Linux (nay Unix, including OS X) is the need to add a route to forward multicast IP packets

route add -net netmask gw


route add -net gw

Check that the route has been added with

netstat -nr



A host that desires to receive a datagram sent to a multicast group, must first request membership to that group. Here’s a Node.js code snippet that receives datagram sent by the code above

var dgram = require('dgram');
var s = dgram.createSocket('udp4');
s.bind(8001, function() {
s.on("message", function (msg, rinfo) {
  console.log("server got: " + msg + " from " +
    rinfo.address + ":" + rinfo.port);

Receiving multicasts on Linux does not work when you bind the socket to a specific interface, for instance s.bind(8001, does not work. It looks like a Linux (nay Unix, including OS X) quirk because it is not required on Windows with either Mono .NET runtime or Node.js.

.NET code that does something similar can be found in the UDP Tool at GitHub.

Using ping to determine MTU size

ping is a ubiquitous and versatile utility available from the command line of most operating systems. Here’s how it can be used to determine maximum transmission unit (MTU) size i.e. the maximum amount of data the network will forward in a single data packet.

On Ubuntu GNU/Linux

ping -s 50000 -M do localhost

Here, 50000 is the size of the payload in the ICMP echo request. The -M do option prohibits fragmentation. The ICMP message as a whole has 8 more bytes, as that is the size of the header. The command shows that the loopback adapter’s MTU size is 16436, and the ping fails.

On a Mac

ping -s 50000 -D localhost

Does not print the MTU size, so you’ll have to try different payload sizes until you hit the limit. -D prohibits fragmentation by setting the Don’t Fragment (DF) bit in the IP header.

On Windows 8

ping -l 50000 -f localhost

Prints the default MTU size as 65500, so the ping above works.

IP fowarding

Normally, commercial operating systems do not act as routers. They discard any IP packet with a destination IP address not assigned to an active network interface. It is fairly easy to enable IP forwarding, there are instructions elsewhere to do so on Windows and Linux.

You’ll also need to add appropriate route settings to the routing table, so that the IP packets are routed through the correct network interface. This can be done using the route add command on Windows and Linux.

I have had problems routing multicast packets (IP addresses through, formerly referred to as Class D addresses). Windows does not route multicast traffic by default. Using EnableMulticastForwarding does not automatically enable it either.

Applications that communicate

You are building an application that needs to communicate over a network, maybe you have decided to build your own communication protocol. I hope you’re doing it because TCP over IP does not meet your needs. I cover some points to keep in mind when developing an application or protocol that communicates over a network.

Use an existing transport protocol

You’ll find it easier to layer your protocol on top of an existing transport protocol such as UDP or TCP over IP. It will require more work otherwise.

Protocol header

A protocol usually requires a header to transmit relevant information about the message. It can contain information such as version, sender address, receiver address, payload size, sequence number, and so on. One important consideration is the size of the header itself, make it as small as possible so that it does not become a significant overhead.

Message oriented vs stream oriented

It may be desirable to have message boundaries preserved. For instance, if the protocol has been asked to deliver a particular set of bytes, it should ideally provide the receiver those same set of bytes as a cohesive whole.

TCP is an example of a stream oriented protocol in the sense that there are no clear message boundaries. UDP is message oriented, each message or datagram can be up to approximately 65,000 bytes long.

Fragmentation and reassembly

Depending on the the size of the data, it will need to be broken into smaller fragments, these are reassembled when received. To reassemble data, data fragments need to be put in the order they are sent. The order can be indicated by adding a sequence number to each fragment. The application may also segment data as required, a protocol does not care for the contents of the data itself, it is blissfully unaware that data is segmented.

IP, and therefore TCP and UDP, transparently perform fragmentation and reassembly of data. TCP also further segments data sent by the application. The segments are reassembled at the receiver and provided to the application as a stream. The segment size needs to be such that the total length of the network packet does not exceed the maximum transmission unit (MTU) of the network.


If your communication link is unreliable, such as a noisy wireless link, you’ll need to retransmit data that does not arrive at the receiver. Retransmission may also be required if data arrives but is corrupted.

One way to implement retransmission is by requiring the receiver to send an acknowledgement when data is correctly received. The sender can use a timer to resend data when an acknowledgement is not received. If multiple simultaneous retries fail, the data transfer attempt may be abandoned, and an error reported to the software layer that uses the protocol. Each data fragment needs a unique identifier that should be used during acknowledgement.

Another way to implement retransmission is for the receiver to request it when a fragment with a particular sequence number is not received, after a more recent fragment has been received. This eliminates the need for acknowledgement.

IP is best effort, it neither retransmits nor prevents duplicate messages from arriving. UDP retains these drawbacks, large datagrams may be dropped if the network is unreliable, they may also arrive out of order. TCP handles retransmission, making it reliable and robust at the cost of throughput.

Error checking and correction

Error checking codes such as CRC codes can be added to data fragments so that errors during transmission can be detected. Redundancy in the data can ensure that data can be corrected even when there are errors. This is useful in scenarios where retransmission is expensive or not possible at all.

UDP and TCP are capable of checking header and data integrity based on a checksum value. They do not have data correction capability. Since TCP does retransmission, it can recover from errors by asking the sender to retransmit.

Multiple networks paths

The communication protocol stack may have to deal with multiple network paths to the destination, for instance a Bluetooth PAN and a WiFi link. The decision to choose one over the others may be based on the knowledge of which is more reliable, is currently active, has better throughput and so on. IP prioritizes one interface over the other using routing metric.

Connection (re)establishment

The state of the connection can be detected using keepalive or heartbeat messages. If the receiver responds to heartbeat messages, the connection is alive. Otherwise, it is considered broken and an error reported to the application. Heartbeat messages compete with regular data, so they may be used when no data activity is present. Connection reestablishment may require user intervention in case of persistent problems with the network.

Protocols such as TCP initiate and maintain a session with the receiver. A termination in this connection is negotiated. TCP supports keepalive, it can be enabled on a per connection basis. UDP on the other hand does not maintain a connection, it is entirely stateless. Termination of a connection due to persistent problems in the network is not handled gracefully by UDP.

Compression and encryption

Compression can reduce the bandwidth required to transport data. Domain specific compression algorithms are usually more efficient than generic compression algorithms like Deflate, for instance JPEG is better at compressing image data, and MP3 is better at compressing music. Encryption ensures that data cannot be read by parties other than the sender and the receiver. Encryption is quite an elaborate and complex topic involving key exchange, and several kinds of crypto algorithms.

Rate/flow control

Rate control, also called throttling, prevents the network and network nodes from being overwhelmed, averting effects such as packet loss. It can also be used to divide the available bandwidth between users, when it is scarce. Rate control can also be applied when available bandwidth changes, commonly referred to as adaptive rate control.

Store and forward

Some considerations need to be made as to what happens to messages when delivery fails, when there is a power outage for instance. The protocol can store messages in a persistent queue and forward them at a later time. This is also sometimes referred to as fire and forget, since the application fires a message and is assured that the other end will receive it, even after a significant delay.

Software design patterns and data structures

Certain data structures and patterns that can be very useful are queues, priority queues, observer, and chain of responsibility.