• Table of Contents
• Previous Chapter
• Next Chapter

Chapter 3: Networking

Open Systems describes the networking protocol using the seven layers of the Open Systems Interconnection basic reference model, or the OSI model. Each layer has its specific properties:

1. The first level is the physical layer, such as a twisted pair cable, an RS232 port, or a 802.11g wireless receiver.

2. The next level is the data link, such as Ethernet or PPP (Point to Point Protocol).

3. The next layer is the network layer, e.g. ICMP (the 'ping' protocol) and IP (the Internet protocol).

4. The layer above that is the transport layer, such as TCP (Transmission Control Protocol) or UDP (User Datagram Protocol). The difference between the two is that TCP:
   • is guaranteed, continuous flow. Individual TCP packets will be retransmitted until they arrive, or until an error is detected and the tranmission aborts;
   • is in order; TCP receivers will reshuffle packets until a whole series is matched.

   In the remainder of this document we will only focus on TCP. This transport protocol is most widely used.

   UDP is non-guaranteed, and might be delivered out of order (first sent packets may arrive last); but it has less overhead than TCP and is therefore faster. UDP has its uses, of which we name three:

   • Remote system log messages (syslog) are often delivered by UDP. The idea is that it is preferable to have some logging getting lost due to network malfunction, than having a process wait until logging finishes.

   • Domain name resolution (DNS) often runs via UDP. Here the idea is similar; a program can decide to re-query DNS, or to ignore name resolution alltogether. (Incidentally, DNS over TCP is also quite widely used.)

   • P2P connections for file sharing and streaming often use UDP. If an individual datagram is lost, a peer can always request it from another source.

   (If you want a look at a UDP client and server, you may want to look at the package logfwd on http://public.e-tunity.com.)

5. The fifth layer is the session layer, which denotes an active network connection. In our discussion this is the layer where 'sockets' occur. A program opens a network socket and reads from it, or writes to it, like from a file. Throughout this time the socket is open and represents one network session.

6. The layer above that is the presentation layer, where standards are defined to represent data. Examples are MIME (mail encapsulation), XDR (external data representation, which e.g. ensures that a double value is sent correctly, despite of Endianness), and SSL (which has phases like key negotiation and exchange, encrypted data transmission, end-of-session warnings).

7. The final seventh layer is the application layer, where application-specific formats are defined. Examples are SMTP (mail transfer protocol) and HTTP (hypertext transfer protocol).

The advantage of separate OSI layers is that one can substitute alternative mechanisms for one or more layers, without affecting standards of other layers. E.g., UMTS (wireless phone transfer) replaces the first three layers. IP doesn't exist under UMTS. However, above that, all protocols still work, up until HTTP which brings your favorite site to your mobile phone.

Another example is SSL in layer 6. Whether enabled or not, the above application protocol remains the same. If you have a mail reader that doesn't know how to talk to secure POP3 mailboxes (spop3), then you can simply use a program like stunnel to set up an SSL link to your spop3 provider. Next you have your mail agent talk to the non-SSL part of the tunnel and you're up and running.

In the following sections we will talk about layer 5 (the session layer). Other layers will go largely unnoticed. Regarding sockets, we will handle server side and client side separately -- preparations and actions on these are quite different depending on your point of view.

In the sections on client-side sockets we'll discuss all that is necessary to behave as a client that connects to some server. In the sections on server-side sockets we'll discuss all that is necessary to behhave as a server that presents a networked service.

3.1: Client-side Socket Handling

• The first step is to create a socket itself, in a given domain, and having a given type. For our purposes the domain is the ARPA Internet (PF_INET), and the type is a sequenced two-way stream (SOCK_STREAM):

  int sock = socket (PF_INET, SOCK_STREAM, 0);
  

  The final 0 argument is a protocol specifier, which is unused at this time.

• The second step is to look up a host that we're going to connect to. This is done using function gethostbyname(), which returns a struct hostent *. The returned pointer contains information on the IP address of the host, its aliases, and so on. This step may fail, in which case gethostbyname() returns 0, and the hostname cannot be resolved.

• The third step is to prepare the binding between the socket and the looked up host. Also, the port that we want to connect to, is now stated. This step consists of filling a 'socket address structure' struct sockaddr_in with relevant fields.

• The final step is to connect the socket to the server's resolved IP address, and the port. This step may fail when the IP address offers no service at the given port. Or this step may take indefinitely if the connection cannot be established (this often happens when firewalls silently drop network packets for unsupported ports).

3.1.1: How to handle connection timeouts

In order to neatly prevent endless connection waits, one should set up an acceptable timeout period that is an acceptable wait time. When exceeded, the program would stop.

Waiting for a timeout of the connect() call in most cases consists of the following steps:

1. Before a potentially blocking action, we issue an alarm call in say 5 seconds.
2. We define a signal handler for the SIGALRM event. This signal handler 'does nothing' except raise a flag, called e.g. timed_out.
3. Next, we attempt the connect() system call.
4. If connect() returns a non-zero value, then something's afoot. The call may have been interrupted due to a signal; which might actually be our alarm call. We check the flag variable timed_out to see whether a timeout condition has occurred.

3.1.2: Putting it all together

So, having sorted out the timeout problem, here's a demo client connector. You can run it as ./client smtp.gmail.com 25. It will connect to smtp.gmail.com, port 25, and show the first transmission (up to 255 characters) what the server is sending. In this case this would be the greeting line of a mail server. You can also try out ./client smtp.gmail.com 26 to see the timeout code in action. It should be pointed out that the timeout applies only to the connecting of a socket; not to the reading of data. If you direct the client at e.g. www.google.com port 80, then the client will sit there indefinitely, waiting for the server's message. A webserver however doesn't start the chat; it waits for the client to request something, which is not what this client does.

The relevant part of creating a client socket and connecting it, is coded in a function client_socket(). This function uses a hardcoded timeout value of 5 seconds, and exits the program upon failure. It may however be a starting point for your own code.

3.1.3: Handling connect() timeouts in multithreaded programs

The approach shown above in section 3.1.1 is applicable only to non-threaded programs; ie., in the shown sample client program, or in forked daemons. The approach is not thread-safe:

• A global, context-free variable timed_out is used;
• An alarm for the whole program is set. This clears all other pending alarms.

The approach for multi-threaded daemons is somewhat different. It will not be fully described here, but the outline is shown below. A full example can be found in section 1.

1. Just as is the case with the 'standard' approach, a socket is created, the remote host is looked up, and a binding structure sockaddr_in is filled.

2. Next, the socket is put in non-blocking mode. This can be done with fcntl().

3. Now the connect() is attempted. Since the socket is non-blocking, the connect() won't block either, but will almost immediately return a non-zero value. The variable errno will be EINPROGRESS.

   As a global variable errno is used, this piece of code must be protected by mutex locks.

4. Following the connect(), the socket can be placed back in blocking mode.

5. After this, a select() call is executed, with the socket in the writable set. The required timeout is stated. The select() call will try to wait until a next write() to the socket is possible; which might time out.

6. However, the select() call will apparently succeed in situations where a connection was refused. In such cases, the next read() or write() will yield errno ECONNREFUSED.

3.2: Socket IO and Timeouts

• Sockets aren't character- or line based. E.g, you can't expect to read a line from a socket; instead, you read a 'chunk' of data and you have to see for yourself if it's a line. (You could emulate character-based IO by handling chunks of 1 byte, but that would cause a lot of overhead.)

• Sockets are unbuffered. Each read() or write() is a system call in itself that the kernel has to handle. In contrast, fprintf() or cout are buffered; you use call the function or use the object, and the underlying library flushes from time to time. If you want buffering with sockets, then you have to implement it yourself.

• Sockets don't return 'ready to use' information, as a 0-terminated string. If you want to read a string and you want to display it, you have to append a 0 character yourself.

• Reading from a socket, or writing to a socket, may require several read()'s or write()'s. Each atomic read() or write() may handle only a part of its assignment. You have to check the return value to see how many bytes were actually handled, and optionally you may need to re-issue the system call for next parts of the buffer you're handling.

3.2.1: Looped socket IO

The following sample code shows how to write() to a socket. The sample program just uses file descriptor 0 (stdout), but the same methodology applies to network sockets. The trick is in function fd_write(): it loops until the total number of written bytes equals the total number of bytes we want to write. Inside the loop atomic write()'s send the information. The return value of each write() is checked to verify that at least some information could be sent.

The same mechanism applies to the reading from a socket. If you expect to read e.g. 1000 bytes, then you may have to issue several read()'s before all information is collected. Each read() provides a signal for the next course:

• When read() returns 0, then there is no more information to be read. Usually this means that the other network party has hung up.
• When read() returns -1, then there has been an error. Usually this means that the network connection failed.

3.2.2: Handling read- and write timeouts

We've seen in section 3.1 that connecting to a network server may result in a timeout, and how to handle it. However, reading from a network socket or writing to one, may cause timeouts too.

The basic principle here is that the system calls read() and write() are blocking calls: they will stop execution of the program until they can do something. In normal situations that is just what you want. For example, tail -f /var/log/messages will monitor /var/log/messages and show text as it is appended to that file. In this case, you'd simply use a blocking read(), and once it signals that information has arrived, display that information.

In contrast, you'll most often want to use some timeout value when handling network sockets -- especially if you're writing a daemon program. The trick here is to see whether a socket can be read or written before the actual read() or write(). The system call select() allows you to do exactly that: peek at a file descriptor to see if a read() or write() would block.

The select() call works as follows:

1. First you need a timeout specifer, struct timeval. This specifier is set to the timeout length (in seconds and microseconds).

2. Next, you need two fd_set variables. Each fd_set is cleared, and then the file descriptor you want to watch is added. One of the fd_set's is used as either the read or write watcher; the other fd_set is used as a watcher for exeptions (errors).

3. After this, the select() call is issued, with the timeout specifier and the file descriptor sets. select() will return the number of available descriptors, or 0 when a timeout occurred.

4. If there are file descriptors that are ready for reading or writing, then the read() or write() can take place.

The above setup is illustrated in the below listing. The program is identical to the client socket initializer of section 3.1.2, except that main()'s read is replaced by a function socketread(). This function is implements the above mechanism. Note also the checking of exceptions on the socket.

In order to test this yourself, try reading from e.g. an HTTP server. The server won't initiate chatting and will patiently wait for the client. This client will time out, while attempting to read from the server.

3.3: Server-side Socket Handling

• The first step is the creation of a socket, which is similar to the client-side process. This is done via the system call socket().

• The second step isn't necessary but very much recommended. It consists of setting a specific socket option, which enables address reuse. This is done with setsockopt(), using option SO_REUSEADDR. If this step is skipped, then restarting the server process would require waiting for the old network socket to free up. This usually takes a few seconds.

• Similar to client-side socket handling, the following step is to set up the binding between the socket and the host. The host is always localhost, but the IP address can be specified separately. This feature is e.g. used in the SSH daemon sshd which allows you to bind the daemon to a specific IP address; e.g., only to the address of one network card.

• After this, the socket is bound to the established address and a given port, using bind().

• Next the kernel is informed of the fact that this daemon is listening to activity, using the listen() call. Contrary to what the name of this system call suggests, listen() is non-blocking.

• Finally, the program forks so that it daemonizes (see section 2.1). After this the standard streams are gone, so that all messaging is done to either a separate log file, or via syslog() to the system log.

Having established a server network socket, the program waits for client activity. The server enters an endless loop to serve clients. The loop usually consists of the following:

• Using the select() call, the server waits for activity on the socket.

• Once some activity is seen, a connection is accepted, using accept(). In this step the kernel can give us information on the client, such as their IP address.

• After this, the connection is served. The actual serving of the connection should run in a separate process fork, or in a separate thread. That way the socket immediately becomes available for next clients. When using forked processes, a signal handler for SIGCHLD must be set up, so that stopped children are seen and their status is collected.

• When the server should stop, the socket is shut down.

Without further ado, let's dive into some code. The following program is started as ./server {port}, where port is a numeric TCP port. The server listens to all network interfaces; it doesn't bind to a specific server address. Once the server establishes its service, it says hello to the client, also stating the client's IP address, and then hangs up.

The following is a rather lengthy listing, which is however self-explanatory in its comments. You can try out the program with the following steps:

1. In one terminal window, run tail -f /var/log/messages (or whatever your syslogd output file is);

2. In a next terminal window, start the server using ./server 10000. Unix will allow a user-space process to start a networked process on an unprivileged port (greater than 1024). If you want to run the server on say port 80, then you have to start it as root.

3. Test the server using the client program from section 3.1.2, as in ./client 10000.

4. When you're done testing, stop the server using killall server.

• Table of Contents
• Previous Chapter
• Next Chapter