Internet Programming with Pascal
Compilers and Operating Systems
Many people (such as me) came to Pascal programming through Borland's great Turbo Pascal series for DOS. A lot of them still consider it the best development environment around and want to stay with it as long as possible. Unfortunately, there are a few points that let it seem advisable to say good-bye to Turbo Pascal and set out for new shores.
The 16 bit platform is obsolete, and has been so for quite some time. It is unlikely that you vex a lot of users if you require "386 or better" for your program to run. On the other hand, having modern processors run 16 bit code is waste of ressources at best.
Then there are the operating systems. With Turbo/Borland Pascal, the best you can get is Windows 3.x; what I said about 16 bit platforms applies. Nowadays, most users have 32 bit preemptive multitasking systems like Windows NT or Linux, and we should make use of these systems. With focus on Internet programming, DOS is an especially problematic system, because TCP/IP stacks for DOS are incompatible, poorly documented, and not very wide-spread.
To make a long story short, this guide will use Free Pascal. It is a free (GPL'd) 32 bit compiler that is available for a number of platforms such as Windows32 and Linux (for a full list see their web site below; more platforms are expected to be added in the future). It has a well-documented runtime library, a number of useful additional units, and a large base of third-party packages can be found on the Internet.
More information, sources and binary packages are available at the Free Pascal website.
Most information in this guide doesn't rely on operating system specific features; if it does, I have pointed that out. The example programs have been written under Linux and additionally tested under Windows NT.
All files referred in this document are available from the download link.
Getting Started
So, we want to do "Internet programming". In other words, we want to write programs that communicate with other programs (that may or may not be written by us) through the Internet. As you may already know, two transport protocols are normally used on the Internet, namely UDP and TCP. The former is a so-called connection-less protocol that lets you send and receive small packets of data (datagrams) to and from remote computers, while the latter establishes an enduring connection between your computer and a remote host. In this document, we will only use TCP.
I assume that you are familiar with the basic terms of TCP/IP networking, such as IP addresses and ports, since I conjecture that this kind of knowledge is common among "expercienced computer users" such as programmers. If you think this assumption is wrong, please let me know.
Byte Order
This is a tiresome issue which keeps arising now and then, and which will be a permanent companion in all kinds of network programming. The problem is as simple as this:
When you store words of 2 or even 4 bytes in memory, such as Free Pascal's Integer and LongInt types, there are several possibilities. You could store the least significant byte at the lowest address and the most significant byte at the highest, as it is done on Intel and VAX machines, to name the most important examples. Sounds resonable, doesn't it? However, many other platforms such as the Motorola 68000 and the PowerPC, do it just vice versa (for 32 bit words, there are more possibilities, but fortunately, no-one has yet used these!). Just why do they do that? Well, look at the written representation of a 16 bit word in hexadecimal:
$4587
On an Intel box, the first byte would be 87 and the second 45. In the hex-editor, this looks like 87 45. That is why.
Communication on the Internet would hardly be possible if people hadn't agreed upon a network byte order. If your machine uses this, you don't have to change anything, but if it doesn't, you have to swap the bytes. The version that was chosen for this purpose is Most Significant Byte First. For PC programmers this means that they have to swap.
I have written a unit called inetaux containing routines for
byte order conversion on 16 and 32 bit integers. The functions are called:
-
hton– Host to network byte order for LongInts -
ntoh– Network to host byte order for LongInts -
htons– Likehton, only for Integers -
ntohs– Likentoh, only for Integers
(The nomenclature is inspired by a certain set of C macros.)
There are two other interesting routines in inetaux which I will be using in the examples below. IP addresses are basically 32 bit numbers, yet what you normally see are notations such as '192.168.2.1'. For better readability, the four bytes have been split up and written as decimals. The two functions can translate a LongInt to the so-called dotted decimal notation and vice versa. They are:
-
StrToAddr- Dotted decimal to LongInt -
AddrToStr- LongInt to dotted decimal
Note that the LongInt arguments are in network byte order.
Some interesting questions arise when performing these translations. First, what to do with strings like '34.324.45.3' - should the 324 be taken as "more than a byte" and the surplus bits be added to the 34? Or should they be truncated, or, maybe, should a roll-over be performed, such that 324 becomes 324 - 256 = 68? Well, StrToAddr would treat the above string as invalid and would return 0 (the invalid address).
The other case would be strings such as '3232236033' (the decimal version of 192.168.2.1). Apparently, many programs accept these - a fact which spammers tend to make use of for confusing their victims. StrToAddr will treat them as invalid because of missing dots. I don't know if this behaviour is approved of by the "net.gods", but as long as I don't have further information, it seems only reasonable to me.
Error Messages
If you want to write stable, reliable programs, you have to do error checking. You should write an error message to stderr (preferrably one that is helpful for the user), and then either continue operation or, in the case of an unrecoverable error, you should abort. When dealing with the network interface, we are going to use a lot of system functions that will, on error, set a special variable. The system provides us with a "translation service" to turn this number into a human-readable string. I wrote a bunch of procedures that I will use in the examples to deal with such errors:
-
Say– Write a message to stderr. -
SockError– Write a message to stderr, then then a description of the last Sockets error, then abort. -
SockSay– Like SockError, but don't abort. -
GenError– Write a message to stderr and abort.
Note Windows programmers: Since strerror is not available under Windows, converting the numbers to human-readable messages is currently impossible. This has probably to be done using Winsock functions. Until I have found a way, there is a special Windows version of the myerror unit that only prints numbers (see the link below).
Introductory Example
From the application's point of view, the network is accessed through sockets. Any program that wants to send or receive data through the network has to request such a socket from the operating system, which can then be used to open a TCP connection or to handle UDP datagrams. From the network's point of view, a socket is associated with a port.
When sockets were invented for Unix, the authors had the idea to use the normal Unix file descriptors (handles) for this purpose. That way, you can read from and write to the network as if it were a normal text file. However, opening a socket is still a bit more work than just opening a file...
Let's now have a look at a simple example, and describe on a step-by-step basis, what the program does. The task is to write a simple server that waits for incoming connections (contrarily, a client would actively be opening a connection). When a connection is opened, the server reads a line and writes the length of the line back to the client. This (rather tedious occupation) is repeated until the client disconnects.
As all servers, we have to select the port we can be contacted at. Ports below 1024 are reserved for system servers such as the FTP daemon etc. Some systems won't let user processes bind to these ports. Also, many of them are already assigned as well-known port numbers for standardized services such as FTP, HTTP and so on by the Internet Assigned Numbers Authority – certainly nothing we want to mess with. We will therefore use a big number such as $AFFE, which is used throughout the examples.
Ok, let's start off with the initial declarations, and our first action: To request a Socket from the operating system. As you can see, a socket is nothing but a LongInt:
program simserv; uses sockets, inetaux, myerror; const ListenPort : Word = $AFFE; MaxConn = 1; var lSock, uSock : LongInt; sAddr : TInetSockAddr; Len : LongInt; Line : String; sin, sout : Text; begin lSock := Socket(af_inet, sock_stream, 0); if lSock = -1 then SockError('Socket: ');
The socket is created using the Socket system call. Let us have a look at the
parameters. First of all, you have to know that sockets are not only used
for TCP/IP networking, but also for IPX/SPX, AppleTalk and what have you.
Since we want to use the Internet, we specify the address family
af_inet. The next parameter, sock_stream, indicates that we
are going to use a TCP connection. The last parameter specifies a certain
protocol. We set it to 0, as there is only once choice, and that is TCP.
A return value of -1 returns an error; we call SockError to abort.
Now that we have a socket, we want to bind it to a certain address/port. For
this, we use the variable sAddr of type TInetSockAddr. It
has three (relevant) fields; the address family, the address and the port
number:
with sAddr do begin Family := af_inet; Port := htons(ListenPort); Addr := 0; end; if not Bind(lSock, sAddr, sizeof(sAddr)) then SockError('Bind: ');
The family is again af_inet. The port is set to the constant defined
earlier, and please note that it is converted to network byte order. The
address is set to 0 (invalid address), which makes the operating system
automatically fill in our own address.
With the address record ready, we call the Bind function (the third parameter is the size of the address record), which returns False on error. Now we are ready to tell the system to open this socket for incoming connections:
if not Listen(lSock, MaxConn) then SockError('Listen: ');
MaxConn (also defined in the const part) is a number, also known as
backlog, that specifies how many incoming connections should be kept in
a queue until we get around to answering them. Anyhow, now that the socket is
listening, how do we check for connections and answer them? The
following code shows:
repeat Say('Waiting for connections...'); Len := sizeof(sAddr); uSock := Accept(lSock, sAddr, Len); if uSock = -1 then SockError('Accept: '); Say('Accepted connection from ' + AddrToStr(sAddr.Addr));
The Accept call blocks the program until a connection arrives (unless we
use it with a non-blocking socket, which I'm not going to cover here). It
fills sAddr with the address of the connection's other end. The
return value is a new socket for the new connection (the old socket keeps
listening), unless an error occurs, in which case it returns -1. The last line
produces an output that reports the new connection.
Having successfully opened a connection, we use the procedure Sock2Text to retrieve two text files - one for input, one for output: Then we use it as if it was a normal file to communicate with the remote host:
Sock2Text(uSock, sin, sout); Reset(sin); Rewrite(sout); Writeln(sout, 'Welcome, stranger!'); while not eof(sin) do begin Readln(sin, Line); if Line = 'close' then break; Writeln(sout, Length(Line)); end; Close(sin); Close(sout); Shutdown(uSock, 2); Say('Connection closed.'); until False; end.
The Shutdown procedure is used to permanently close the connection, with the parameter 2 meaning that no further sending or receiving is allowed. Other choices would be 0: No further receives and 1: No further sends.
To summarize, the course of events is as follows:
Create a new internet socket of type stream.
Bind this socket to our own address, with our custom port.
Call Listen to let the system know we want to accept connections.
Wait for connections with Accept.
After a connection has been accepted, convert the socket to Text files, then open the files.
Exchange data with the client.
Close the files and call Shutdown to close the connection.
To test our server, we can use a standard Telnet client (port $AFFE = 45054). It is a good idea to open two xterms and start the server in the first one and the Telnet client in the other one (in Windows, open two Telnet windows). The telnet output should look like this:
basti@clever:~ > telnet localhost 45054 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Welcome, stranger! short line 10 this is a very long line 24 close Connection closed by foreign host.
The Other End
Ok, now we know in principle what the server has to do. The client is even
simpler: Instead of binding to an address on the local host and listening for
connection, it just opens one. The function Connect is used for this. It takes
as first argument the socket, and as second an address of the type
TInetSockAddr that we already know (a third argument is the size of
the address record).
The following program implements a client for the server we just wrote:
program simclient; { Simple client program } uses sockets, inetaux, myerror; const RemoteAddress = '127.0.0.1'; RemotePort : Word = $AFFE; var Sock : LongInt; sAddr : TInetSockAddr; sin, sout : Text; Line : String; begin Sock := Socket(af_inet, sock_stream, 0); if Sock = -1 then SockError('Socket: '); with sAddr do begin Family := af_inet; Port := htons(RemotePort); Addr := StrToAddr(RemoteAddress); end; if not Connect(Sock, sAddr, sizeof(sAddr)) then SockError('Connect: '); Sock2Text(Sock, sin, sout); Reset(sin); Rewrite(sout); Writeln('Connected.'); Readln(sin, Line); Writeln(Line); repeat Write('> '); Readln(Line); Writeln(sout, Line); if Line <> 'close' then begin Readln(sin, Line); Writeln(Line); end; until Line = 'close'; Close(sin); Close(sout); Shutdown(Sock, 2); end.
A few remarks: As you can see, the loopback address 127.0.0.1 is hard-coded
into the program as a string. We use the function StrToAddr from
the inetaux unit to convert it to a LongInt. Another possibility
would be to define it as $7F000001 (host byte order!), but this technique
quickly becomes impractical. Also, if we use strings, we can easily change
the program to accept IP addresses as user input, as we shall see in the
next subsection.
The rest of the program is nothing new. Go ahead and try it out, but make sure that the server is already running somewhere.
Something Useful
It's time that we write a program that actually has a purpose. I mean, determining the length of a string is not really one of the services you want to offer other people over the 'net (and also other people don't offer it). One of the lesser known services on the Internet is "daytime". Some hosts offer a clock service at port 13. The protocol is simple:
You connect to the server.
The server sends a line with the current date and time in some human-readable format.
The server disconnects.
Of course, this is not very accurate, since the message needs some time to travel through the network. For precise time transmission, there is the network time protocol (NTP), as defined in RFC 1305. Let's just say it's very mathy, if that is a word. However, if you awake in a dark, locked room after a long, dreamless sleep, not knowing whether it is day or night, you could simply connect to a daytime server to gain this essential information.
The daytime protocol is defined in RFC 867. I suggest that you read it, it is an easy read compared to your average RFC. RFC documents can be retrieved from the RFC-Editor.
Without further ado, this is the source:
program daytime; { Simple client program } uses sockets, inetaux, myerror; const RemotePort : Word = 13; var Sock : LongInt; sAddr : TInetSockAddr; sin, sout : Text; Line : String; begin if ParamCount = 0 then GenError('Supply IP address as parameter.'); with sAddr do begin Family := af_inet; Port := htons(RemotePort); Addr := StrToAddr(ParamStr(1)); if Addr = 0 then GenError('Not a valid IP address.'); end; Sock := Socket(af_inet, sock_stream, 0); if Sock = -1 then SockError('Socket: '); if not Connect(Sock, sAddr, sizeof(sAddr)) then SockError('Connect: '); Sock2Text(Sock, sin, sout); Reset(sin); Rewrite(sout); while not eof(sin) do begin Readln(sin, Line); Writeln(Line); end; Close(sin); Close(sout); Shutdown(Sock, 2); end.
The syntax is "daytime <ip-address>". Many server
administrators disable the daytime service. Just try some server at a
university that you know. For example, public.uni-hamburg.de works
fine:
basti@clever:~/pas > nslookup public.uni-hamburg.de Server: rzaix240.rrz.uni-hamburg.de Address: 134.100.33.240 Name: public.uni-hamburg.de Address: 134.100.32.55 basti@clever:~/pas > daytime 134.100.32.55 Sun Apr 23 16:54:50 2000 basti@clever:~/pas >
Suggestions
If you want to try your newly gained knowledge, here are some things you could do:
Write a server that reads file names from the client and, for each name, sends back the content of that file.
The server described above is the worst security hole imaginable. Modify the program so that it will interpret the file names relative to a directory specified either in the code or by the user. Make sure that your program cannot be fooled by file names like
'../../../etc/passwd'. Once you start writing network programs, you have to start thinking about security issues!Read RFC 1288 - The Finger User Information Protocol. Write a finger client. Find out why this service is called Finger and tell me about your findings, since I'd like to know.
Why not write a daytime server?
Using the Domain Name System
As every schoolkid knows, hosts on the Internet are uniquely identified by an IP address, which is – as we have already seen – nothing but a 32 bit number normally written as 4 consecutive bytes. Now if there is something we can learn from the telephone system, it is that numbers are hard to rememeber. Therefore, the domain name system was created. This is not the only use of the DNS – email exchange hosts can also be determined, among other things.
The DNS is a distributed database of a hierarchical name space. Distributed means that there is not one central name server that holds all the information, but instead the administrator of a zone such as "microsoft.com" has to provide name services for all hosts in that zone (zones can be partitioned into sub-zones, as is probably done in the case of Microsoft). There are, however, root servers that know what the name servers for each zone are. This information should partly be available at every provider, but if it is not, there is a hierarchical system of name servers that your provider's server can ask, ending up with the root servers.
Interface
Fortunately, we don't have to deal with all this. We can use a simple set of operating system calls to resolve a name to an IP address. For this, you have to tell your operating system the IP address of your provider's name servers, which is something you probably have done when you set up your computer.
Michaël van Canneyt of the Free Pascal Development Team has written
a unit called inet to provide some basic services including name
resolution. Unfortunately, it is only available for Linux (included in the
distrubution since version 0.99.14). Windows programmers will therefore have
to use the WinSock unit manually.
The inet unit provides an
object type THost that encapsulates the needed calls to system
functions such as GetHostByName etc. The reason is that more than
one IP address can be returned, and also additional names. (Multiple names can
point to the same host, and multiple hosts can be specified for a name.)
The returned data has a clumsy structure that only C programmers might have
fun with. Luckily, the THost object will examine it for us. This is its
type declaration from inet.pp:
THost = Object FHostEntry : PHostEnt; FAlias,FAddr,FError : Longint; Constructor NameLookup (HostName : String); Constructor AddressLookup (Const Address : THostAddr); Destructor Done; Function Name : String; Function GetAddress (Select : TSelectType) : String; Function GetAlias (Select : TSelectType) : String; Function IPAddress : THostAddr; Function IPString : String; Function LastError : Longint; end;
We are going to call the constructor NameLookup, and then use the fields
Name (the official name) and, most importantly, IPAddress, which can
be cast to a LongInt without problems and is already in network byte order
(for more details please see inet.pp).
However, IP address strings such as '127.0.0.1' will not be resolved to the address. How do we detect these?
The clean way: Check if the string could be a correct IP address, that is, four numbers in the range 0 - 255, separated by dots.
Our way: Try to convert it, and if it fails (0 = invalid address), try to resolve it.
This is an improved version of the daytime client with DNS support:
program dtname; { Simple client program with DNS support } uses sockets, inet, inetaux, myerror; const RemotePort : Word = 13; var Sock : LongInt; sAddr : TInetSockAddr; sin, sout : Text; Line : String; Host : THost; begin if ParamCount = 0 then GenError('Supply hostname as parameter'); with sAddr do begin Family := af_inet; Port := htons(RemotePort); Addr := StrToAddr(ParamStr(1)); { Maybe it's an IP address } if Addr = 0 then begin with Host do begin NameLookup(ParamStr(1)); if LastError <> 0 then GenError('Name lookup failure'); Writeln('Official name: ', Name); Addr := LongInt(IPAddress); Writeln('IP address: ', AddrToStr(Addr)); end; end; end; Sock := Socket(af_inet, sock_stream, 0); if Sock = -1 then SockError('Socket: '); if not Connect(Sock, sAddr, sizeof(sAddr)) then SockError('Connect: '); Sock2Text(Sock, sin, sout); Reset(sin); Rewrite(sout); while not eof(sin) do begin Readln(sin, Line); Writeln(Line); end; Close(sin); Close(sout); Shutdown(Sock, 2); end.
As an example of its usage, let's see if the folks in the capital will tell us what time it is:
basti@clever:~/pas/collection/inet > dtname news.fu-berlin.de Official name: news.fu-berlin.de IP address: 130.133.1.4 Sun Apr 23 19:44:25 2000 basti@clever:~/pas/collection/inet >
Suggestions
DNS is a powerful system. If you want to find out more about the information you can retrieve through your name servers, you might want to get a hand on the package "dig" which is available for most operating systems. DNS is described in RFCs 1034 and 1035.
The THost object can do more than just retrieve one address and one hostname. It can find secondary addresses (for example if the host is multi-homed), and it can find alias names. Many organizations use systematic names for their computers, and in addition to that, name one of them 'www', one 'ftp' (might be the same), and so on. Use GetAddress and GetAlias to determine these.
Handling Multiple Clients
The servers we have written so far can only serve one client at a time. This may be acceptable for very simple services such as daytime, but otherwise it is unacceptable, especially when connections can last longer (NNTP, FTP or Telnet come to mind). There are two basic techniques to write multi-client servers, which we will explore in the following section.
Unfortunately, at this point we are leaving the zone of operating system independence. We are going to use mechanisms of parallelism that are not part of the sockets network interface, and thus there will be differences between operating systems. So where are we going? As you might have guessed from the text so far (especially the example outputs with a bash prompt, ahem...), I am doing this mainly under Linux, so I am a bit more familar with that. This is why this section will be Linux specific. On the other hand, I had the opportunity to use the WinSock DLL some time ago, so if there is a strong interest in Windows, let me know and I'll see if I can dig out some old sources. (Besides, Delphi offers great sockets components.)
Multitasking
This is the first approach to the task. If multiple clients can be handled independently, it is possible to start a new process for each client. That means that we proceed like in the simple server presented earlier, but after accepting, issue a Fork command. Fork creates a new process that is an exact copy of the old one, all variable values, file descriptors etc. included. With one small, but subtle difference: The return value of the Fork function is 0 in the child process, and another number in the parent process (more precisely, the process ID of the child). So the basic structure looks like this:
if Fork = 0 then { we are the child } begin handle the new client; Halt; end else proceed with the accepting loop;
However, after executing the Halt, the child process is not really dead. On the other hand, it surely doesn't live anymore, so it is fair to say that it is in a mysterious state between life and death. It is called a zombie. (I am not making jokes, please read the manual page for ps!) The process will stay in this state until it is delivered by its parent through a call of WaitPid.
Before proceeding with the loop, the parent version has therefore to call WaitPid (with no particular process ID to wait for) until all childs that are currently in zombie state are deleted from the process list. The appropriate call is
WaitPid(-1, nil, wnohang);
where -1 means that we are not waiting for a particular process, nil means that we don't want to know more about the child's state and the flag wnohang indicates that if there is no zombie, the function should return immediately. The return value is -1 on failure (no zombie) and the process number of the child otherwise.
This is the complete program:
program mtserv; { A multi-threaded server program. } uses sockets, inetaux, myerror, linux; const ListenPort : Word = $AFFE; MaxConn = 1; var lSock, uSock : LongInt; sAddr : TInetSockAddr; Len : LongInt; Line : String; sin, sout : Text; begin lSock := Socket(af_inet, sock_stream, 0); if lSock = -1 then SockError('Socket: '); with sAddr do begin Family := af_inet; Port := htons(ListenPort); Addr := 0; end; if not Bind(lSock, sAddr, sizeof(sAddr)) then SockError('Bind: '); if not Listen(lSock, MaxConn) then SockError('Listen: '); repeat Say('Waiting for connections...'); Len := sizeof(sAddr); uSock := Accept(lSock, sAddr, Len); if uSock = -1 then SockError('Accept: '); Say('Accepted connection from ' + AddrToStr(sAddr.Addr)); if Fork = 0 then { we are the child } begin Sock2Text(uSock, sin, sout); Reset(sin); Rewrite(sout); Writeln(sout, 'Welcome, stranger!'); while not eof(sin) do begin Readln(sin, Line); if Line = 'close' then break; Writeln(sout, Length(Line)); end; Close(sin); Close(sout); Shutdown(uSock, 2); Say('Connection closed.'); Halt; end else { we are the parent } repeat until WaitPid(-1, nil, wnohang) < 1; until False; end.
After starting mtserv, we get the following result from the ps output (try ps | grep mt):
3735 ? S 0:00 mtserv
Then we connect to the server (e. g. with Telnet, or with our own sophisticated client), and voilá, we have two processes running:
3735 ? S 0:00 mtserv 3753 ? S 0:00 mtserv
A second Telnet session, and we have already three of them running. Now let's close both connections and look at the results in the process list:
3735 ? S 0:00 mtserv 3753 ? Z 0:00 (mtserv <zombie>) 3760 ? Z 0:00 (mtserv <zombie>)
As expected, two zombies are wandering around. We open a new connection, and the parent process will kill off both of them, but on the other hand, a new one will be created:
3735 ? S 0:00 mtserv 3800 ? S 0:00 mtserv
Now this guide is not an introduction to Unix process management, so we stop here and just note that with this type of management, there might always be some processes in zombie state, but at least we free all zombies each time we set a new child into the world.
All in all, this technique is very useful for servers where there is no need for the clients to interact, which includes a lot of well-known protocols such as FTP, HTTP and so on. On the other hand, there are a lot of situations where clients are supposed to interact with other clients, such as chats and MUDs (multi-user dungeons). With multiple processes, we would have to employ inter-process communication, which is not exactly what I want to do here. Fortunately, there is a simpler technique which also has the advantage of an enormously impressive name.
Synchronous I/O Multiplexing
This facility is a feature of most Unices, and is not specific to sockets - it works for other types of file descriptors as well. Its heart is the Select system call, which lets the operating system wait for events to happen on one or more files. Of course, regular files are not the most interesting examples, since watching them tends to be rather boring. But as Select works for sockets as well, it will be a great help.
You can tell Select to watch three sets of file descriptors, for read, write and exception events. For example, if you put stdin in the read set, you are informed when new data is available to be read from stdin. The sets are modified upon return of Select to indicate which file descriptors are concerned. In our examples, we only use the read category, which covers connection establishment and termination as well as incoming data.
A number of routines are available to manipulate file descriptor sets:
-
fd_zero– Empty a set -
fd_set– Add a file descriptor to a set -
fd_clr– Remove a file descriptor from a set -
fd_isset– Determine if a file descriptor is in a set
(For more information, see the documentation about the Linux unit).
Select can either block until something happens or you can set a timeout parameter. If nothing happens within the specified period of time, Select will return anyway. The time can be 0. In our case, using a timeout is not useful (except maybe for printing "Still waiting..." messages), so we will not set this parameter. The basic structure looks like the following code snippet:
repeat put all sockets currently connected in the read set; put the listening socket in the read set; Select(read set, no other sets, no timeout); respond to all sockets that are left in the read set; until false;
As an example, let's assume we want to write a simple chat server. All data that is received from one client is sent to all other clients that are online. To maintain the clients, we are going to use a record ClientRec that keeps all data we have about one client. More precisely, that means the socket descriptor and the two text files. On a more advanced level, you might want to store additional information such as a nickname, and so on. For simplicity, we use an array to hold the client records.
After one or more events have occured, we have to follow the procedure outlined below:
If an event occured on the listening socket, accept the connection. If there is still room for the new client, store it in the clients list, otherwise send an error message and close the connection.
Check all active clients for events. If one occured, test if the connection has been closed (with eof). If so, remove the client from the clients list. If not, read the input and process it.
I believe that a piece of code says more than a thousand words of prose, so here is it:
program mulserv; { A server that can handle multiple client connections. } uses sockets, inetaux, myerror, linux; const ListenPort : Word = $AFFE; MaxConn = 5; MaxClients = 5; type ClientRec = record cSock : LongInt; adstr : String; sin, sout : Text; end; var lSock, uSock : LongInt; sAddr : TInetSockAddr; Len, i, j : LongInt; Line : String; Clients : array[1..MaxClients] of ClientRec; NumClients : LongInt; MaxFD : LongInt; ReadSet : FDSet; sin, sout : Text; begin lSock := Socket(af_inet, sock_stream, 0); if lSock = -1 then SockError('Socket: '); with sAddr do begin Family := af_inet; Port := htons(ListenPort); Addr := 0; end; if not Bind(lSock, sAddr, sizeof(sAddr)) then SockError('Bind: '); if not Listen(lSock, MaxConn) then SockError('Listen: '); Say('Waiting...'); fd_zero(ReadSet); NumClients := 0; repeat MaxFD := 0; for i := 1 to NumClients do with Clients[i] do begin fd_set(cSock, ReadSet); if cSock > MaxFD then MaxFD := cSock; end; fd_set(lSock, ReadSet); if lSock > MaxFD then MaxFD := lSock; Inc(MaxFD); Select(MaxFD, @ReadSet, nil, nil, nil); { No timeout! } { New connections? } if fd_isset(lSock, ReadSet) then begin Say('Incoming connection.'); Len := sizeof(sAddr); uSock := Accept(lSock, sAddr, Len); if uSock = -1 then SockSay('Accept: ') else begin if NumClients < MaxClients then begin Inc(NumClients); with Clients[NumClients] do begin cSock := uSock; Sock2Text(cSock, sin, sout); Reset(sin); Rewrite(sout); adstr := AddrToStr(sAddr.Addr); Say('Accepted connection from ' + adstr); end; end else { client limit reached } begin Sock2Text(uSock, sin, sout); Rewrite(sout); Writeln(sout, 'Sorry, we are fully booked.'); Close(sout); Shutdown(uSock, 2); end; end; end; { And/or an event on an existing connection? } for i := 1 to NumClients do if fd_isset(Clients[i].cSock, ReadSet) then begin if eof(Clients[i].sin) then { Connection has been closed? } begin with Clients[i] do begin Close(sin); Close(sout); Shutdown(cSock, 2); Say('Disconnected ' + adstr); end; for j := i to NumClients - 1 do Clients[j] := Clients[j + 1]; Dec(NumClients); end else { No -> Data can be read } begin Say('Received message from ' + Clients[i].adstr + '...'); Readln(Clients[i].sin, Line); Say(Line); for j := 1 to NumClients do if i <> j then Writeln(Clients[j].sout, Line); end; end; until False; end.
(Note: I once expercienced a mysterious CPU hogging by this server after a client had disconnected. I don't know why, and I couldn't reproduce it reliably. If you find anything suspicious, please let me know.)
Suggestions
The concepts of serving multiple clients should be clear so far. Things to try to get used to client interaction:
Improve the chat server. If you are familiar with dynamical data structures, you might want to add dynamical client management. It also could use support for nicknames, multiple channels, private conversations, operator mode, etc etc... It might be a good exercise to document your protocol.
Write a multi-threaded version of the file server from section 2.
Binary Data
In the example programs so far, we always assumed that communication through the network is like text file I/O - lines of reasonable length are exchanged with Readln and Writeln (if you are concerned about the line length, consider compiling your programs with AnsiString support). Many protocols work this way, but it also has its drawbacks. If we want to exchange raw, binary data, CR/LF sequences should not be treated in a special way, and buffers should be used to hold the data to be sent or received. For this purpose, we will use the functions Send and Recv. Both have four parameters:
- the socket
- the buffer
- size of the buffer
- flags
To illustrate the point, let's assume that we want to conduct bandwidth measurements in a network. We need a server we can tell, "send me so many bytes of random data", and it will begin pumping the bytes into the network. (Actually, a research group at Hamburg University once developed a web server that would accept URLs like http://hostname/50MB for similar purposes.) We also need a client to initiate the transmission and receive the data.
The protocol is as follows: After starting the connection, the client sends a four byte LongInt indicating how many bytes it would like to receive (with a maximum of 2 GB). The server then starts sending the data. The clients writes the data to disk, so that we can examine it (if you think this affects your delicate bandwidth measurements, you can change OutFileName to '/dev/null', but see the Fhlushstone benchmark).
Anyway, this is the server code:
program binserv; { A binary data server program. } uses sockets, inetaux, myerror; const ListenPort : Word = $AFFE; MaxConn = 1; BufSize = 1024; var Buffy : array[0..BufSize - 1] of Char; procedure FillBuffer; var i : LongInt; begin for i := 0 to BufSize - 1 do Buffy[i] := Chr(Random(256)); end; var lSock, uSock : LongInt; sAddr : TInetSockAddr; Len : LongInt; Amount : LongInt; begin Randomize; lSock := Socket(af_inet, sock_stream, 0); if lSock = -1 then SockError('Socket: '); with sAddr do begin Family := af_inet; Port := htons(ListenPort); Addr := 0; end; if not Bind(lSock, sAddr, sizeof(sAddr)) then SockError('Bind: '); if not Listen(lSock, MaxConn) then SockError('Listen: '); repeat Say('Waiting for connections...'); Len := sizeof(sAddr); uSock := Accept(lSock, sAddr, Len); if uSock = -1 then SockError('Accept: '); Say('Accepted connection from ' + AddrToStr(sAddr.Addr)); Len := Recv(uSock, Amount, sizeof(Amount), 0); if SocketError <> 0 then SockError('Recv: '); if Len < sizeof(Amount) then GenError('Couldn''t receive a LongInt!'); while Amount > 0 do begin FillBuffer; if Amount < BufSize then Len := Amount else Len := BufSize; Len := Send(uSock, Buffy, Len, 0); if Len = -1 then SockError('Send: '); Dec(Amount, Len); end; Shutdown(uSock, 2); Say('Connection closed.'); until False; end.
Please note that it is not guaranteed that Send will send all the data we want to have sent, although it is very likely. If not, we don't resend (as the bytes are random anyway), and just take the actual amount of bytes sent (as returned by Send) into consideration.
The client works similar:
program binclient; { Simple client for binary data } uses sockets, inetaux, myerror; const RemoteAddress = '127.0.0.1'; RemotePort : Word = $AFFE; OutFileName = 'received.bin'; const_Amount = 10005; BufSize = 2048; var Sock : LongInt; sAddr : TInetSockAddr; Buffy : array[0..BufSize - 1] of Char; fout : File of Char; Amount, Len : LongInt; Closed : Boolean; i : LongInt; begin Sock := Socket(af_inet, sock_stream, 0); if Sock = -1 then SockError('Socket: '); with sAddr do begin Family := af_inet; Port := htons(RemotePort); Addr := StrToAddr(RemoteAddress); end; if not Connect(Sock, sAddr, sizeof(sAddr)) then SockError('Connect: '); Writeln('Connected.'); Amount := const_Amount; Len := Send(Sock, Amount, sizeof(Amount), 0); if Len < sizeof(Amount) then GenError('Couldn''t send a LongInt.'); Assign(fout, OutFileName); {$I-} Rewrite(fout); if IoResult <> 0 then GenError('Couldn''t open file ' + OutFileName); {$I+} Closed := False; while not Closed do begin Len := Recv(Sock, Buffy, sizeof(Buffy), 0); if SocketError <> 0 then begin Close(fout); SockError('Recv: '); end; if Len = 0 then Closed := True else for i := 0 to Len - 1 do Write(fout, Buffy[i]); end; Close(fout); Shutdown(Sock, 2); end.
Of course, Send/Recv and the Text file functions can be combined. This might be useful in the case of HTTP, which on the one hand works with text-oriented headers, on the other hand is able to send binary files (e.g. pictures). I have not checked the HTTP specification, so I don't know how this is handled. Another thing worth mentioning is the procedure Sock2File which turns the socket descriptor into two untyped files (instead of Text files). This allows for record-oriented file I/O, since you can call Reset and Rewrite with the record size parameter.
Suggestions
Unstructured data flow has a number of advantages over line-oriented communication. It seems wise to become familiar with it, although there are many situations where it only adds management overhead.
In the earlier sections, we developed a file server that could be used to transfer files from one host to the other. However, the transmission technique that we used - line oriented - was suboptimal, as it would produce a tremendous overhead for files with many empty lines, and would scale very badly for files with very long lines.
It is obvious that it would be a major improvement for the program to change the transmission technique to byte-oriented. On the other hand, aren't there some advantages to line-oriented transmission (hint: think about heterogenous networks with communication between Linux and Windows hosts)?
The SMTP protocol is completely line-oriented, regarding the client-to-server commands as well as the transmitted emails themselves. QMTP however, devised by qmail author Dan Bernstein as an SMTP replacement, uses so-called "netstrings", which are not unlike classic Pascal strings, to simplify the protocol and improve throughput. Write a simple message exchange service using netstrings. Using AnsiStrings will be a great help. And don't forget the no. 1 optimization rule for AnsiStrings: If you know the final length beforehand, set it with SetLength.
Obtaining the Software
Throughout this document, I have quoted quite a lot of program sources, and I have also mentioned several units. All these files are available in a ZIP archive accessible via the link below:
Michaël Van Canneyt's inet package is available from the following address:
Final Remarks
As an overview over the subject of sockets programming, this tutorial is far from being complete, and it is still evolving. Topics that might be added in the future include: Connectionless transmission, FPC's overloaded Accept and Connect functions, non-blocking sockets, various fine-tunings (e.g. with the Send and Recv flags) etc. Several problems exist with Windows, and I would like to address them more specifically - but before I can do that, I have to have a closer look at them.
Second, while all programs on this page have been tested, that does not mean they are bug-free. The same holds true for all other informations: I try to get everything right, but there might still be errors and inaccuracies. Therefore, any feedback is greatly appreciated; if you find anything to criticize - factual errors, bugs, whatever - feel free to mail me at the address below.
For now, have fun!