S-LINK News 1999

TileCal plans to use G-LINK based S-LINK at front-end link

15 December 1999. Jim Pilcher and his collegues of the University of Chicago who are responsible for the front-end links of the TileCal, have made radiation tests on the Simplex G-LINK card that is based on the 'old' G-LINK HDMP-1022 and Actel components. Their article "Neutron Irradiation Tests of an S-LINK-over-G-link System" has the following conclusions:
The survivability of the S-LINK-over-G-link system under neutron irradiation has been satisfactorily demonstrated. The system continued to operate at 10 times the required neutron fluence.
The observed rate of single-word errors would correspond to 6.7 per month from the entire ATLAS TileCal system for the nominal neutron fluence. Since these errors are detectable from parity information, and since the data itself has considerable redundancy, this error rate should not pose a problem; nevertheless, the ROD modules which receive the data must test for these errors and take appropriate action. The actual error rate in ATLAS should be lower because the link will only be used with a duty factor of ~50% of that used here.
The observed rate of block errors would correspond to 0.53 per month from the entire TileCal system. Such errors would result in the loss of all data from one electronics drawer (1/256th of the system) for a number of consecutive events. If these errors correspond to loss of link synchronization the recovery time is estimated to be ~ 0.5 ms. At the maximum LVL1 trigger rate this would correspond to a block of ~ 40 events. The fraction of LVL1 triggers affected by such an error would be ~10-10. Such a loss appears entirely acceptable but an ATLAS-wide policy would be helpful for reaching a conclusion. These block errors correspond to a bit error rate on a single link in the nominal TileCal neutron environment of 4x10-13.
During this test the LSC was exposed to 9.9 Krad of ionizing radiation. Separate studies have indicated that the system remains fully functional to over 20 Krad.

The S-LINK team encourages this type of radiation tests as they cannot guarantee that the links they build are tolerant to radiation. This remains the responsability of the physicists who are building the final system. The Simplex G-LINK card was a best-effort to design a radiation tolerant link, and even that one has shown that there were problems with certain components and bit error rates. Note that the Simplex G-link cards can be made commercially available if there is sufficient demand. For the testbeams where there is much less radiation, the TileCal is currently using the FCS-LINK, which is commercially available.
 

TileCal testbeam Read-Out Crate based on commercial components

15 December 1999. The Read-Out Crate of the ATLAS Tile Calorimeter that is being designed for the testbeams in the year 2000, will be completely based on commercially available components. 

One RIO2 processor from CES will be able to handle three front-end links that come from the detector and one read-out link that goes to the Read-Out Buffer. 

Both the front-end links and the read-out links are S-LINK based. The front-end links used in the testbeams are the commercial available FCS-LINK, which have been used already since the 1999 testbeam. Those links will not withstand the radiation levels that exist in the final ATLAS system.

ODIN: a new optical link running at 128 or 160 MByte/sec

13 December 1999. Based on the new low-power G-LINK chip from Hewlett Packard a new S-LINK has been built. As the link is fully compatible to the S-LINK specification, it can be used as a drop-in replacement of any of the other links such as the ones based on Fibre Channel or SCSI/LVDS components. The newly developed link is able to sustain a data rate of 128 MByte/sec, or, when two fibres are used in the forward channel, it can even sustain a rate of slightly less than 160 MByte/sec.

All of the components on the new card are powered by 3.3 Volt. As currently most motherboards power the link cards with 5 Volt, a drop-down regulator can be put on the board. When ordering, one should mention the required supply voltage. The supply current is typically 1.0 Amp for the single G-LINK version. This is only 3.3 Watt when the card is supplied with 3.3 V, or 5 Watt when supplied by 5 Volt. The numbers are about 20% higher for the double G-LINK version. Other features are that the link uses the new Small-Form-Factor (SFF) optical transceivers from Infineon, which use the new VF-45 fibre-optic connector, also called Volition. The advantage of this connector is the low-price for both the optical cable and the transceiver as it works with connectors without a ferrule. This helped considerably in getting the component price down by 25% compared to that of the FCS-LINK2 card in 1998 (while even improving the performance by 25%).

Erik Brandin, a Swedish student from KTH, nicknamed the link ODIN, which stands for Optical Dual g-lINk. The other developers were Zoltan Meggyesi, who also designed part of the Fibre Channel S-LINKs, Erik van der Bij who designed the PCB and Robert McLaren who guided the project so it could be finished in time before both Erik Brandin and Zoltan Meggyesi left CERN.

The ODIN will be commercialised by the Hungarian company CERNTECH. Zoltan Meggyesi is also active in CERNTECH, which means that CERNTECH can update the PROM of the link when necessary. The Fibre Channel link is also still available from CERNTECH and is the fibre-optic link of choice until the first production run of ODIN cards is available in May 2000. Until then, a few of the prototypes built at CERN will be available to selected experiments.

All features of the link can be found in the ODIN Datasheet (.pdf).

1 nA/bps

30 September 1999. A typical S-LINK such as the Fibre Channel link, draws only 1 Amp from the 5 Volt power supply when it transfers data at one gigabit per second. This is only one nanoamp per bit per second. This 5 Watt is less than the maximum of 7.5 Watt that the S-LINK specification allows. Newer links likely will consume even less as mainly 3.3 Volt components will be used.

Main features revisited

29 September 1999. After four years of existence, the S-LINK specification is well known. But as the specification has been stable for over two and a half years, for many designers it has been quite a long time that they read the complete specification. Therefore it may happen that some of the features are forgotten or not completely understood when an S-LINK motherboard is being designed. Four of those features will be revisited here: control words, error detection, flow control and reset.

All S-LINKs have a way of sending control words, as opposed to sending normal data words. Those words can be used for encapsulating the data to transmit in between "start of fragment" and "end of fragment" control words, like used in the ATLAS experiment. In this case the "end of fragment" control word will also contain the transmission error information of the fragment's data and the control word itself. Like normal data words, control words can contain any data, except that only 28 bits can be used (see the table below). Two of the lower four bits are used for error detection information and another two are reserved for future use. It is very advisable to use control words as if you don't, your data will be transmitted as a single stream without you knowing where the begin and end of each data block is. Also it would be impossible to get error detection information, as that is only sent out together with control words (except for links that have an word-by-word error detection mechanism like parity).

All S-LINK links have a mechanism (using either CRC or parity) that can detect errors that are introduced during the transmission of data. If an error is detected, the S-LINK destination card (LDC) will set a bit in the control word that ends the block of data sent (see the table below). So the S-LINK inserts the error detection information inside the stream of data. Therefore it is advisable to also store the control words in the memory.

Data line meanings when LCTRL# is low
LD[31..4] LD[3] LD[2] LD[1] LD[0]
Same as UD[31..4] Reserved for future use Reserved for future use When 1: Transmission error in Control Word. 

When 0: Control Word OK

 When 1: Transmission error in previous data block 

When 0: Previous Data Block OK
All duplex S-LINKs have a simple mechanism for flow control, i.e. with the UXOFF# signal the read-out motherboard can tell that it is "almost full". The LDC will transmit this to the LSC where the Link Full Flag LFF# will tell the front-end motherboard that it should stop sending data immediately. In fact it may still send two words after LFF# became active. The LFF# signal not only gets asserted when the receiving side is almost full, but also when the front-end motherboard sends data faster than the physical link can handle. E.g. on a Fibre Channel S-LINK card, the words are transmitted at a speed of 27 MHz, while the S-LINK LCLK can be up to 40 MHz. To throttle the speed, the LSC will make LFF# active for on average one third of the time (1 - 27/40). This still gets you up to a considerable speed of 103 MByte/sec.
Also simplex links have the LFF# signal as also there the S-LINK clock which may be up to 40 MHz, may be faster than the physical link can handle. Therefore all front-end motherboards should be able to handle the LFF# signal as otherwise data will get lost.

An S-LINK needs a reset to start working. On the links that have been built up to now, both the LSC and LDC need an independent reset. Duplex links that will be built in the future, such as the ODIN, will allow you to reset only one side, after which that side will also reset the remote side. Note that there is an handshake protocol between URESET# and the LDOWN# signal which should be respected. Having URESET# in a writeable register and LDOWN# in a readable register and some software will do the job.

NA48 has taken this year 170 TeraByte of data over S-LINK interfaces

6 September 1999. Last year the NA48 experiment has made an major upgrade of the data acquisition system in which every bit of the physics data goes through one of eleven S-LINK to PCI interfaces, which are controlled by a Linux driver. Last year the interfaces had moved 75 TeraByte of data, while during this year's run they moved another 170 TeraByte. None of the eleven interfaces has shown any failure during those two years.

S-LINK in CERN's Technology Database

27 July 1999. S-LINK is one of the technologies that comes from CERN that has applications outside the High-energy Physics community. It has already been used for moving display data (Olivetti and Oracle research labs) and in telescopes (Megacam), while people have shown interest in using it in areas such as in the offshore industry (underwater towed vehicle) and for moving data out of a digital camera.

Of course there are many other developments at CERN that may be of interest to others. The CERN Technology Database contains information on technology developed and in use at CERN. It is compiled and maintained by the CERN Industry and Technology Liaison Office (ITLO) in collaboration with the relevant CERN Divisions and Groups. It aims to make available information on such technology to industry in CERN's member states, "as is" or via a licence agreement, and to disseminate such information within CERN and its collaborating institutes.
The database also serves as a point of contact for the flow of technology information to and from CERN. It is used to gather information on CERN technology directly from sources within CERN as well as to channel enquiries on CERN's technology from Member States, collaborating institutes and from CERN itself to the appropriate sources of information. These enquiries will include requests for details of technologies as well as information on consultation services.
To have an idea on how the information is presented, have a look at the S-LINK entry of the CERN Technology Database.
 

COMPASS received 20 Fibre Channel S-LINK cards from Hungarian company

9 July 1999. On 22 June Cerntech, the Hungarian company who produces the FCS-LINK, shipped 20 FCS-LINK cards (10 LSC, 10 LDC) to COMPASS. Together with the cards COMPASS has already, they will make up for the 12 links that COMPASS needs for testbeams to be held in the second half of 1999. The data will be received by 12 spill buffer cards (S-LINK to PCI cards with a large FIFO-like organised memory on the S-LINK side) for this purpose as well. The design of the final read-out driver, called CATCH, is also well under way.

Gazing in space with SHARCs, S-LINK and VxWorks

9 July 1999. Jean de Kat and his collegues from CEA-Saclay, Dapnia/Sei have written a VxWorks driver for the S-LINK to PMC card. The software has been tested on a Motorola 2604-333MHz-64MB VME processor board. With the SLIDAS as a full-speed data generator, a transfer rate from S-LINK into the Motorola's memory of 82 MByte/sec has been measured. This is largely enough for his MEGACAM telescope application in which the CCD camera data is coming from a SHARC link (max 40 MB/s) before it is transferred over an FCS-LINK to the Motorola processors.

April-June: too much progress

9 July 1999. In between April and June many things have happened, which made there was little time left for updating the pages.

S-LINK pages split

19 March 1999. Like with stock values, when the value gets too high, they do a stock-split. This time the S-LINK web pages became too long to be easy to use and therefore the S-LINK Products and Projects pages got each their own page. Also the area "people using S-LINK" got removed as the list was becoming too long to maintain.

G-LINK Link Source Card ready, installation in test beam

19 March 1999. The G-LINK Link Source Card that uses the Hewlett Packard HDMP-1022 serialiser is working. The board, which uses an Actel A54SX16 FPGA as protocol chip worked first time. This was a great accomplishement (and a fortunate one) as the Actel is a one time programmable device. Zoltan Meggyesi implemented a data generator in the Actel. In fact three different generators are implemented, each one designed in a different way so the influence of the design method (e.g. triple redundancy) on the radiation tolerance can be tested. At the end of March the card will be installed next to the SPS in building 889 to do the radiation tests. The corresponding G-LINK Link Destination Card was working already before and has been used on 11 March at the Paul Scherrer Institute to test the sensitivity to upset of the serialiser chip that Peter Denes designed.

COMPASS CATCHes on

9 February 1999. For the COMPASS experiment a readout-driver and buffer-module called CATCH (COMPASS Accumulate, Transfer & Cache Hardware) is in development. The FPGA based VME module serves as an interface between the front-end of the detector systems and an on optical S-LINK, which transmits the data to the Readout Buffer (ROB). It also acts as an fan-out for the COMPASS trigger distribution and time synchronisation system (TCS). The readout-driver monitors the trigger and data flow to and from the front-ends. In addition a specific data buffer structure and sophisticated data flow control is used to pursue local pre-event building.

LHCb will use S-LINK in the prototype Readout Unit

2 February 1999. The LHCb experiment will use S-LINK in the prototype of the Readout Unit. The Readout Units (RU) receive event fragments from several front­end links and assemble them into larger sub­events. Once a subevent is assembled, the RU transfers it to the next stage for further event building. In the final system 100 Readout Units are needed. Each unit will receive between one and four S-LINK inputs (LDCs) and may have one S-LINK output (LSC). The design of the RU has been started and it expected to have a first working prototype ready in Summer 1999.
With this decision, LHCb is the fourth high energy physics experiment (after NA48, ATLAS and COMPASS) where all of the physics data taken may be sent over S-LINK.

S-LINK Studies in the TileCal ROD Environment

2 February 1999. Two S-LINK cards have been tested at Valencia ROD Lab using different S-LINK hardware devices and hosts. In the source side a PC PentiumII 266Mhz running Linux kernel 2.0.35 allocate a PCI to Slink board. At Receiver side a Motorola 2604 VME card under LynxOS 3.0 allocate a S-LINK to PMC card. The main goal of the test was to try to parametrize in terms of Bandwidth the current commercial S-LINK hardware devices. One of the conclusions is that CTRL DMA Transfers (DMA buffering with CTRL words) the transfer rate can reach up to 85 Mbytes/sec for 8Kwords (the maximun Packet size provided by the SLIDAS data generator). In terms of Bandwidth, the penalty coming from the software CTRL words switch, which is nicely seen in the curves, it's about 10% for 1Kword. Without control words, a speed of 89 MByte/sec can be measured.

Q: Can S-LINK logic easily be integrated onto a motherboard?
A: Yes!

1 February 1999. Currently all S-LINK links are implemented as daughter cards of the size of a PMC card. To reduce cost, it is foreseen that in final applications the link logic may be integrated on the front-end motherboard or on the read-out motherboard. As currently most designs are prototypes and as link technologies are still rapidly changing and are becoming cheaper, it is still too early to integrate the link logic on the motherboards.

However, we have to learn already now if any difficulties will arise if S-LINK logic is integrated with other logic. One may think of problems of the co-existence of gigabit signals on boards with other frequencies, EMC, noise on power supplies etc. To get experience with this, Wieslaw Iwanski of the Institute of Nuclear Physics in Krakow has integrated the Fibre Channel Link Destination Card logic with S-LINK to PMC card logic. In the last week of January he has tested a prototype in both a VME environment and inside a PC (with a PCI to PMC adapter). The board has been working without any problems during more than a day while it received full speed data with different patterns.
FCS-LINK LDC and S-LINK to PMC card integrated

As the board has the gigabit signals differently routed than the original FCS-LINK2 it shows that when integrating it is possible to redo the layout. Of course one has to keep in mind the high-speed design techniques such as putting decoupling capacitors as close as possible to the pads of the chips and optical transceiver, having a controlled impedance for the differential lines carrying the gigabit signals etc. Therefore it is recommended to have someone from the S-LINK team review your design before doing the final layout.

Wieslaw will send three of those boards to CERN, where they will be tested over a long period. After that, they may be used in testbeams or in the DAQ Prototype-1 project. In those projects the new board will be more practical as it is only one slot high, while with the solution used until now, two VME slots were needed. Until now it is not foreseen to have this board commercially available. Soon a web page will be made describing the integrated FCS-LINK2 and the S-LINK to PMC card.

Simplex G-LINK LDC robust

25 January 1999. In the ATLAS experiment there are some detectors that would like to use the S-LINK protocol on the front-end links. That means that they like to use the signal description and the protocol. In the area where those links are going to be used there is a certain level of radio-active radiation, which needs special radiation tolerant IC's that can withstand this. This area is full of unknowns, so there are several projects working on aspects of the radiation tolerant IC's and logic.

Zoltan Meggyesi is researching if normal, commercial, programmable chips are radiation tolerant enough, and is also looking at the different design techniques that can be used so that the logic can withstand single bit errors.


Another problem area is the serialiser IC's that need to be radiation tolerant as well. Currently there is no chip available that is known to withstand the radiation levels that ATLAS needs it for. Preliminary tests have been made with the G-LINK serialiser from Hewlett Packard, but during radiation the receiving G-LINK lost its synchronisation too often and also the error output was too often active. Note that in those tests the data itself has not been tested for errors. But it might still be that the G-LINK is suitable as the radiation levels used during the tests were much higher than those during the running of the LHC. Therefore extra tests will be performed. Also, in the CMS experiment, a serialiser chip is being designed which sends data in a format so that it can be received by a G-LINK receiver.

To help those different projects in testing chips for radiation hardness, the G-LINK Link Destination Card has been built. Prototypes have been tested and are very robust. Because of special power decoupling and EMC techniques the board even works with a supply down to 4 Volt and even at high temperatures (without a cooling tower), the board receives data without failure. As an engineer from a company that uses G-LINK chips in commercial designs stated: "That is absolutely amazing! I definitely think I'll try your localized ground plane on the next design we do with G-LINKs. I would love to be able to leave those cooling towers off of the G-LINKs".

We have designed a G-LINK Link Source card as well, which will be used in Zoltan's radiation testing project. Prototypes are expected by the middle of February.

NIKHEF makes CRUSH

8 January 1999. The Compact Robin Using the SHarc (CRUSH) is a design of an ATLAS Read-out buffer. The board receives it's data over an S-LINK connector, stores in a 1 MByte Zero Bus Turnaround RAM and can process it with the SHARC processor. All communication with the board is done over the six SHARC links. Although the board is in the PCI form factor, the connector is only used to draw power from it.

Old S-LINK News

CERN - High Speed Interconnect - S-LINK
Erik van der Bij - 2 May 2000