Network performance has been measured under the traffic patterns expected within the second level trigger of the ATLAS experiment [11]. The traffic shows a fan-in pattern, i.e. several sources (level two buffers) are sending to the same destination (Feature Extractor processors or FEX). Several FEX processors are active per event. The results presented use parameters based on the Silicon Tracker (SCT) subdetector.
The requirements of the SCT have been taken from ATLAS internal documents [12, 13]. The method used to generate event description files is described in an accompanying paper [14]. The total number of buffers in the SCT has been estimated to be 256. The number of FEX processors has been chosen to be 80.
Two different distributions of FEX processors across a 512 node Clos network have been investigated: grouped and distributed. In the grouped case all 80 FEX processors are connected to the last 5 terminal stage switches, i.e. 16 FEX processors per switch. In the distributed case the FEX processors are connected to the last 16 terminal stage switches, i.e. 5 FEX processors per switch. In all measurements the 256 buffers are connected to the first 16 terminal stage switches.
Figure 7 shows the total network throughput versus attempted event rate for a 512 node Clos. The maximum sustainable event rate is about 120 kHz which is greater than the 100 kHz expected rate within the second level trigger. The improvement from grouped to distributed FEX processors is due to reducing the contention at each terminal stage switch and therefore reducing the effect of head-of-line blocking. The average network latency is shown in figure 8.
Figure 7: : Network throughput versus attempted event rate for a 512 node Clos
Figure 8: : Average network latency versus attempted event rate
The average receive rates for the individual FEX processors were 5.3 and 7.2 Mbytes/s for the grouped and distributed cases respectively. These amounts correspond to 53 and 72% of the theoretical maximum for an individual link (9.97 Mbytes/s [7]). Under random traffic the individual link data rate for a 512 node Clos is 4.2 Mbytes/s.