The requirement to study different topologies, plus the need to do this for hundreds of nodes, imposes a system design and implementation which is highly modular and flexible, for quick and reliable system re-configuration, as well as having a very low cost per node. The system is built up from three elements, each one housed in VME 6U mechanics:
A traffic node can simultaneously send and receive data at the full link speed of 100 Mbits/s. A series of packet descriptors define the traffic pattern. The packet destination address, the packet length and the time delay to wait before dispatching the next packet is programmable. Each traffic node has memory for up to 8k such packet descriptors. The nodes are all synchronised with the same clock.
The dispatch algorithm is implemented in an FPGA
which can be reconfigured under host control. A control processor
is used to supervise the operation of a group of 4 traffic nodes and
all these processors are connected via a control network.
To reduce cabling, sixteen traffic nodes are hard-wired to an on board STC104 packet switch. The remaining 16 ports of the switch are brought out to the front panel for inter module connection. Boards can be interconnected either directly, or via packet switch units which contain one switch with all 32 ports brought out to the front panel.
To measure latency, the timing nodes transmit and analyse time stamped packets which cross the network between chosen points. The same modules, in spy mode, can be inserted into any cabled connection to provide a snapshot of the traffic passing through that point. This provides debugging information and additional data on congestion "hot spots".
A VME crate contains 128 traffic nodes and the entire 1024 node system
can be housed within eight crates. All crates have an Ethernet port
which drives an OS link daisy chain connection to the control processors. The STC104
packet switches have their own separate DS control network which is
independent from the main data path.
Figure 2 shows how a two dimensional grid network topology can be constructed. Each packet switch has 16 on board connections to traffic nodes and four external cabled connections to each of its four nearest neighbours.
So far 656 nodes have been built and tested. They have been assembled as a range of 2D grid, and multistage Clos networks [7]. An example of a 256 node Clos is shown in Figure 3. Results are presented for these configurations. Further details on the design of the testbed are presented in [8].
Figure 2: Architecture of the Macramé testbed
Figure 3: 256 node Clos network