We base our parallel computer on Communicating Sequential Processes [8] (CSP). We use OS processes, i.e., Linux processes or threads, for the CSP processes. The CSP channels are virtual links between these processes. The Application Programming Interface (API), see Table 1, provides full CSP communication functionality, including facilities for active channel selection.
Table 1: The communication API.
We add non-blocking communication to the blocking CSP interface, since non-blocking communication allows concurrent computation and communication without task switching. This improves the performance of the DSNIC, since task switching is a time consuming operation in general purpose OSs. The definition of blocking and non-blocking corresponds to the definition in the Message Passing Interface[12] (MPI) standard. In non-blocking communication, the send or receive call may return before the operation is completed, and before the user is allowed to re-use resources, such as buffers, specified in the call. In blocking communication, returning from a send or receive call indicates that the user is allowed to re-use resources specified in the call.
Furthermore, we add asynchronous communication to the synchronous CSP interface. In synchronous communication, the sender and receiver processes not only exchange a message, but they also synchronise during this communication: the send and receive operations only terminate after they both have been executed. In asynchronous communication, the send and receive operations do not necessarily synchronise: the send operation may terminate even before the receive operation has been executed, by exploiting the use of buffers. By using buffers on the receiving host, asynchronous communication may result in latency improvements, since the message data may already have arrived at the destination host even before it is requested by the receiving process. The API allows the user to specify the amount of buffer space on the receiving host for each virtual link.
To avoid memory-to-memory copying on the host CPU, we make the DSNIC collect and deposit the message data directly in user space. Direct user space access puts constraints on the kind of memory that can be used: the memory must be locked, word aligned, and accessible via Direct Memory Access (DMA). Therefore, the API is extended with special allocation and release functions for message memory.