S-LINK

S-LINK Implementation Guide

Erik van der Bij

CERN - European Laboratory for Particle Physics

ECP division

8 June 1998

Table of Contents

1. Scope
2. Frequently asked questions
3. Choices to be made
4. Design tips
5. Shopping list
Updates to this document

1. Scope

This S-LINK implementation guide gives help for S-LINK designers. These are the people that are making the Link Source Card (LSC) and Link Destination Card (LDC), as opposed to the users, who are building the Front-end Motherboard (FEMB) and Read-out Motherboard (ROMB).

This guide is not part of the S-LINK specification. The S-LINK spec should already have told you everything that you really need to know to build a link. This guide will try to ease your task as it gives suggestions on how to implement certain features.

It will also give you the reasoning behind certain decisions we made while writing the S-LINK specification. This will make you better understand the reason for certain features. However, the main aim of this write-up is to give you hints and tips; just to make the job of designing the boards easier and to show you where the pitfalls are. It also gives suggestions on places where the spec allows various implementations.

Of course, S-LINK users may read this text as well. Especially in the chapter "Frequently asked questions" they might find answers to questions about the use of the S­LINK or the concepts of it.

If after reading this implementation guide you still have questions about the design or usage of S-LINKs, please don't hesitate to contact Robert McLaren, Owen Boyle or me, Erik van der Bij.

Robert McLaren
Tel: +41 22 767 4907
Fax: +41 22 767 9495
E-mail: Robert.McLaren@cern.ch

Owen Boyle
Tel: +41 22 767 8737
E-mail: Owen.Boyle@cern.ch

Erik van der Bij
Tel: +41 22 767 8645
Fax: +41 22 767 9495
E-mail: Erik.van_der_Bij@cern.ch 

2. Frequently asked questions

2.1 How much will it cost?

The S-LINK only specifies the protocol that runs over a connector. It does not at all specify the type of components or even the physical medium that has to be used. It might be serial, parallel, optical or electrical. This freedom makes that a designer can choose the cheapest components that will fullfil the task. The S-LINK protocol itself is kept as simple as possible, so it does not add any costs compared to other protocols.

2.2 Is it Synchronous or Asynchronous?

What type of link is the S-LINK, is it synchronous, asynchronous and what type of data can be sent over it? For the user, the data link from LSC to LDC is a synchronous link in the sense that the data is given together with a clock. So the data is synchronous to that clock. However, there is no synchronization between the source and the destination as the time it takes to send the data from the source to the destination (the transmission latency) is undetermined. In practice there will only be a small variation in latency.

The return link is a separate case. That one is really asynchronous in any sense. First of all, there is no clock associated with it, and second, the four return lines are even asynchronous with each other, which makes that a user cannot use the lines to send a codeword over it. See other sections in this chapter for a more detailed explanation of this and an explanation on how the return link may be used.

2.3 How high is the latency of a link?

The latency of an S-LINK is really dependent on the implementation of the source and destination cards. In principle the latency should not be higher than say 10 to 20 clock cycles plus the transmission time in the physical link. But if a Link Source Card uses a large FIFO on its input, the latency may be much higher, so it is recommended to link designers to only implement a small FIFO (maximum 4 words), if one is needed at all. Again, depending on the implementation of the link cards, the latency can vary each word, but the S­LINK specification has nothing in it that prevents making implementations with a fixed latency.

2.4 Is it too big?

S-LINK cards are relatively large you may think. Indeed, a CMC sized card uses quite a surface. Also the connector and the component height mean that a height of more than 15 mm will be used. If that makes it too big to use on a detector, you may integrate the design of the link on the Front-end Motherboard (FEMB) or on the Read-out Motherboard (ROMB). In that case only the components needed will take the space. I expect that most designs will consist of one large programmable chip that does the control and multiplexing of the data and one block of logic that drives the link. So that would take not much more space than say 40 square centimeter. On places where the space is not a problem and for prototyping, I recommend you to follow the mechanics and the connector from the S-LINK specification as it has many advantages such as interchangeability of cards and for testing.

2.5 What's the lifetime?

Is the S-LINK specification flexible enough to let it survive until well into the LHC age? It does not at all prescribe the components that have to be used to implement the link. So the specification leaves you the freedom to take the cheapest or best components for your application. Furthermore, the specification leaves enough freedom to use lower voltage components, fast or slow components etc. What about the functionality then. Is that enough? We have built in all the main features that were needed until now in data acquisition links. For example, flow control and lines back are all included. So I believe there is enough flexibility so that it can survive for quite some time.

2.6 Is it only a point-to-point link?

The S-LINK is really designed for point-to-point usage. That should be fine for most detector links, but indeed, it is not possible to make a bus out of it. Nor is it foreseen to have a switch in the link. However, you can imagine that other protocols could be included later to make those things possible. How? By using the control codes. Currently, all 32 bits of the control data are free for anything you like. But what if we gave them special meanings? Such as "switch off the laser", or "tri-state drivers". This opens up the possibility for a bus. Or we could make one bit mean: "bits 10 to 0 give the destination address", or "next word is logical address". Then we can use switches in the path. Never thought about that, eh? Nor did we in too much detail. But it shows that there is some growth path for the link. I believe we could even make S-LINK converters to be able to use switches from other technologies such as HIPPI. In due time, before LHC, we will have learnt enough about the usage of the S-LINK to expand its functionality. The good thing is that the connector and usage of the signals will not change. We would only start inventing protocols on top of the S-LINK spec.

2.7 Is 32 bits enough?

The datawidth of 32 bits could limit the throughput, but with a clock frequency of 66 MHz we can still go up to 264 MByte/sec. That is something that currently cannot be implemented with cheap components. But you may argue that it could be in a few years from now. In that case, it will be easy to expand the specification by adding another 32 datalines. It would need an extra connector though.

2.8 How can I use the error detection?

There are two types of S-LINKs as far as error detection concerns. One type only signals that there was an error in the datablock ended by a control word. S-LINKs that can detect errors on a word-by-word basis, mark in addition to this exactly which word was in error. The type of error detection that a user needs, is dependent on his application. If a user sends control words over the link, errors are not only shown by the seperate signal LDERR#, but also inside the control word itself. This has the advantage that a user may use either method to handle the errors. E.g. for hardware devices a seperate pin showing the error may be easier to handle, while if the data and control words are stored on disk, the error detection bits are also automatically stored without the need for storing LDERR# seperately.

2.9 What about flow control?

On the duplex version of the S-LINK, flow control is implemented. The mechanism is simple, just XON/XOFF. There is no sophisticated system like look-ahead flow control. This is done to simplify the design, which is possible as the Read-out Motherboards will have a large enough FIFO to handle the data that it will receive after UXOFF# is set low. The S-LINK spec gives as an example that on a 50 MHz link about 100 words can be received after UXOFF# is set low.

2.10 Why the Write Enable?

You may ask yourself why the Write Enable mechanism is used on the LDC. Why would you not just use a simple edge of a clock that says that the data must be written? The reason is that with a Write Enable there is no need for any controlled signal that runs at the frequency of the clock. The maximum frequency of the LWEN# and UWEN# signals is only half the clock frequency, just like all data lines. As the WEN lines are controlled (I mean, dependent on LFF# for example), the WEN mechanism means that you can use logic that runs only half the speed of what you would need if you had only a clock signal. This makes the design easier and cheaper. Implementing a Write Enable is very easy in programmable logic anyway (see section "Write Enable")

2.11 What's this asynchronism with the return link?

The four lines of the return link on the duplex S-LINKs are specified to be asynchronous. Even worse, they are not only asynchronous to any clock, but even to each other. At least, that is how it is specified to the user. Why is that? And, could we not do it a bit better?

The problem is that the user of the LDC receives a clock (LCLK), which is basically generated by the LDC. Of course, we could let the user make the return lines synchronous to that clock. But in general, the LDC will generate it's own clock for sending things over the link back. This clock is asynchronous to LCLK. Sure, we could have invented some system with a FIFO here to implement some decoupling, but then the return link would have needed also some clock or enable line. More importantly, it adds complexity and extra hardware which is not needed for most users. It is even worse if you also consider that the LSC also would need a FIFO and extra logic. In the LSC this would mean more power consumption, which should absolutely be avoided.

But why are the lines even defined as asynchronous to each other? Most link designers would use the clock of the link back to latch the return lines. As this clock is not given to the user, the return lines come in really asynchronous. So even if they all change at the same time, it could be that sometimes some lines fall within the setup and hold time of the latch and others don't. Only the change on the lines that do, will be transmitted on that clock edge. The others will come one clock cycle later. As a result, changes on the return lines are not all sent at the same time and are thus asynchronous to each other.

2.12 Are four return lines enough?

The return lines: four seems a small number. The fact that the lines have to be considered as completely independent makes it even worse. Basically this limition is caused by the number of pins on the connector. This should not be a problem as in most cases there is no need for fast data transfer and the amount of data will be relatively small. And that opens a new possibility: a link user may implement with those four lines a simple serial protocol to send over some data. You can imagine using one line as a clock, a second one as a dataline and a third as a start signal. Or he can just send over RS-232, which uses only one line, but that is more complex to implement. Or he can use an I2C serial protocol for example. So even with simple solutions like this, you can easily send hundreds of bits over the few return lines that the S-LINK offers.

2.13 What's the data rate on the return link?

What to expect of the speed if you use a serial protocol on the return link? Probably you need 64 clock cycles to send one bit, as the spec says the minimum pulse width shall be 32 clock cycles and you might want to send a clock signal in parallel. On a 33 MHz link this would mean more than 0.5 Mbit/sec, which is probably more than enough to send some slow data. If you would use two bits to send data and one as a clock line, a speed of over 1 Mbit/sec can be reached and you would still have one line free to send some other signal.

2.14 Is there error detection on the return link?

Yes, there is an error detection mechanism for the return link, but it is invisible for the user. A link will send a parity bit together with the return lines. When an error is detected, the Link Source Card will not update the value of the return lines, which prevents glitches on the return line outputs. The next time a sample of the return lines is sent without an error to the Link Source Card, the lines will be updated with the new value. This effectively means that the update took longer than normal and in case a slow clock was sent over the return lines, the duty cycle would have changed. This is not a problem as in the specification is written that on the input the minimum pulse duration shall be 32 clock cycles and on the output it will be minimum 16 clock cycles. If only once an error occurs, in practical implementations the pulse lengthening will not be more than say 4 or 8 clock cycles. If more errors occur in a row, the link will pull LDOWN# low, showing there is a serious problem on the link.

2.15 Can you use a cable instead of an S-LINK?

If you are a motherboard designer, and don't have a set of S-LINK cards, could you still do some testing? Indeed you can; by using a simple cable between the S-LINK connectors of the FEMB and ROMB. This would even implement a duplex link with XOFF protocol and return lines. Never thought about that, eh? There are only a few limitations: You should make the following connections in the cable to make the cheapest possible S-LINK:

Table 1: Cable connections for testing S-LINK motherboards 

UD31-UD0  - LD31-LD0
LFF#      - UXOFF#
UCTRL#    - LCTRL#
UWEN#     - LWEN#
UCLK      - LCLK
Vcc       - LDERR#
Vcc       - LDOWN# (LSC and LDC)
LR3-LR0   - URL3-URL0
Gnd (LSC) - Gnd (LDC)
Make sure you have enough ground wires in the cable to reduce the crosstalk. And be careful to not connect the Vcc lines of the LSC and LDC together as the S-LINK is not supposed to send smoke signals. It is really dependent on the motherboard designs if the drivers are suited to drive a cable. So to be honest, it's not really certain it will always work. But it is an interesting idea to build this simple S-LINK. If you use a register to retime LFF#, it can be even fully compliant to the S-LINK spec!

To run other simple tests, you might want to make a dummy LSC. You just need to pull LFF# high, and have possibly a dipswitch for the Link Return Lines. That's all you need to get rid of the data!

A dummy LDC is a real design: you would need a real crystal oscillator to drive LCLK, pull up LDERR# and do whatever would suit you with the link data lines and the LCTRL# line.

In fact we are building testing devices that emulate an LSC or LDC. Those contain logic state analyzer adapters and some extra logic so that a motherboard designer can easily check the functioning of his devices.

2.16 What about the mechanics?

The idea of the physical description of the S-LINK is to use it for prototypes of detectors and for final applications where you only need a few links. But for applications where thousands of links are needed, a design will probably be integrated on the FEMB. This means that it will use less space and will radiate less as the connections to the FEMB or ROMB logic will be shorter. Also the separate PCB and the connectors are not needed then, which reduces the cost. Still it can be handy to keep the connector in the design, or at least keep the pads on the motherboard to ease debugging of the full cards.This way you can connect a logic state analyzer adapter or other test equipment. It is not recommended to mount the connector in the final system, as the stub-length of the connector pins introduce ringing on the signals and can cause EMC problems.

2.17 Can't I do it on my own?

"Why don't I invent my own link and protocol?" is a question you may ask. You surely know really well what is needed and you could make a customized protocol for your own application. But if you have read the S-LINK specification and continued reading this document until here, I'm pretty sure that you will build a link that follows the S-LINK spec. If not, let me remind you that all features included are the result of many discussions with potential users of the link and with people that have used similar types of links. So all features that are in the S-LINK spec are really needed. You can also be pretty sure that it is also the minimum needed, so it can be designed with minimum cost and complexity. When you follow the specification, it will be very easy for you to explain to a user how to use your link and what features it has. And that will save you lots of work. Even the user documentation is very easy for you to make: you only have to fill in the S-LINK Data Sheet template with the functional and speed parameters! 

3. Choices to be made

The S-LINK specification basically only tells you what signals exist, the signal levels, the protocol and it suggests a physical description of a link card itself. This leaves a lot of freedom to you as a designer. The S­LINK Data Sheet shows you all parameters that you have to define. These are:

Other choices, that are not specified on the S-LINK Data Sheet, that you have to make are: So when you design a link, you have to make many design decisions. However, to make your card useful, you should first check with the end users what they need. If you are designing the link for specific users you may ask them to fill in the S-LINK Data Sheet. Or you may ask them to write a User's Requirements Document (URD) to help you in making your choices.

One thing that helps to solve the non-standard things related to the physical link, is that you normally are designing both ends of the link, i.e. both the Link Source Card and the Link Destination Card. In fact, I recommend you to indeed keep the design of both sides in the hands of one group, as otherwise there will be quite a risk that the LSC and LDC will not work together.

However, most points of the freedom that you and the user have, are pretty obvious. For example, simplex/duplex, data width and the transfer rate required are really things that a user cares about and knows what he needs.

The supply voltage is less obvious. We think that the S-LINK idea will live long enough to enter the age of 3.3 Volt Vcc, so we have foreseen this. At the moment of writing this document, 5 Volt is still the most common supply voltage for use in the detector arena, so therefore I recommend you to design 5 Volt boards when you design in 1996. In the prototyping stage of detectors and DAQ systems power consumption is not yet a major concern. But for the final LHC system, 3.3 Volt boards will be needed.

The timing of the signals is partially left open in the S-LINK spec. You have three options for the maximum clock frequency that your boards must accept. This again allows you to choose components that are the most suitable for your application, I mean why would you use high-speed components if in the application your specific S-LINK will only be used in a rather slow fashion? Note that as the LSC may throttle the FEMB with the Link Full Flag (LFF#), the maximum clock frequency does not specify the maximum data transfer rate.

For the error detection mechanism, you have the choice to implement one that works on a word-by-word basis, or on a block basis, i.e. a datablock between two control words. Some users may really need error detection on a word-by-word basis, so you should check this with the potential user of your link. Within each choice, you have the freedom to select the detailed error detection algorithm, e.g. how many parity bits, over which bits are they used, do you use CRC and with what polynomial etc. As this is left open to the designer, he may use a system that is the best for the physical link that is used.

So you see that you have quite some freedom to design an S-LINK. But even so, it is a specification having well thought out capabilities and options, based on user requirements. There is the possibility that you, as a designer, would like to add other functionality. I can imagine you want to give more error reporting on the link. Or you might want to add error correction codes, or want to add framing information to the data as you can do it so easily with the technology you want to use. Or you like to make some control and status registers on your card. These things are allowed, but only when it conforms to the S-LINK specification. And that will be difficult in most cases. Before you implement anything special, consider very carefully if it is really needed, if it does not increase the cost, and moreover, will there be anyone that will use it anyway? In case of doubt, just give me a call. 

4. Design tips

4.1 A standard S-LINK

Before starting the design, you have to think about what you want to make. Should it be simplex or duplex, what should be the speed, etc. The previous chapter already gave you a few hints on how to decide this, so I won't repeat them.

For the interchangeablity of cards, the many choices definitely will pose problems. Therefore I have defined a Standard S-LINK. This is a link with certain minimum requirements which assures a minimum level of interoperability. Such an S-LINK would fulfill already many applications and help in reducing the number of different links.

The Standard S-LINK should be:

If possible for your application, try to design an S-LINK that would conform to this Standard S-LINK description, or one that is better in any of the features.

The further discussion in this document is not limited to Standard S-LINK designs.

4.2 Source/destination

Of course, you must make both the source and destination cards. To start designing, you must think about how you are going to send the data over the link itself. That is, how do you send the databits, the control bit, test commands etc. Here you need to consider quite a few things for which you'll get some more ideas when you continue reading. Once you have defined the protocol, and that is completely up to you, you can start designing the source and destination. They can then even be made by separate design teams.

About the physical layout: take care to include the hole for the pin that marks if a board is a source or a destination. The Link Source Card should have a hole in it. This prevents you from putting a Destination card on a board that expects a Source card. This way you can't blow up all drivers on both the motherboard and the link card.

4.3 Power supply

As written before, I recommend that you design 5 Volt boards when you design in 1996. When you are designing link cards that work at 5 Volt, make sure to have a hole in your link card at the location as specified in the S-LINK specification. Otherwise your board will not fit. Boards that run at 3.3 Volt should not have the hole. This prevents blowing up 3.3 Volt boards. A 5 Volt link card would still fit in a 3.3 Volt motherboard. The link will probably not work then, but at least it will not blow up.

4.4 Data transfer

4.4.1 Data width control

So now we have to move the data from one side to the other. You remember that we have the option of different datawidths. If your link supports only one or two datawidths, e.g. 8 and 16 bits, what should you do when you detect with the UDW lines that the motherboard wants a different bandwidth than you can support? I suggest that in that case you disable the link and set LDOWN# low. This way the user will immediately notice that he has done something wrong as no data will be sent, nor received.

In most cases a link will be able to handle several datawidths. This is because anyway some multiplexer is used to multiplex 16-bit or 32-bit data onto a physical link that is, for example, 8-bit wide. You have the choice to optimize the transmission speed or not. If you don't, you can for example just ignore datalines 31 to 16 when your 32-bit link card is used in 16-bit mode. With that you throw away half the bandwidth. This is not necessarily a problem if the link still goes fast enough for the user. You might optimize the physical link usage by sending the 16-bit datawords from the user twice as fast over the link. Or, if the link is put in 8-bit mode, even four times as fast. It is up to you if you want to optimize like this, just don't forget to specify in the datasheet what throughput you support with the different datawidth modes.

Practically, it will often be impossible to use the full bandwidth in 8-bit mode. Imagine that your physical link is 100 MByte/sec. That would mean that the clock frequency that the user should give is 100 MHz to keep using the full link bandwidth. This is not feasible with cheap components and anyway the maximum permissable clock frequency for UCLK is 66 MHz.

4.4.2 Memory

How much memory should there be on the link? You will probably include some FIFO, or FIFO-like structure at the input of the LSC. This FIFO has two functions. First of all it is there to decouple the clock given by the FEMB (UCLK) from the clock you use on the link, as in many cases those will have different frequencies. The second function is to have a buffer for the disassembling of the full dataword in chunks of the size your physical link can handle. You should minimize the number of words that you buffer on the LSC to two or four. This will reduce power consumption, component area and will mean that you can implement this buffer inside the same chip that handles all the control and multiplexing. You should not use a large FIFO as the user does not expect the link to buffer data. That is not a function of the link. You may just stop the user from sending data by asserting the Link Full Flag (LFF#) if you need to.

At the LDC you should not buffer any data either. You can directly output the data that is received, as there is no handshake to the Read-out Motherboard. Remember the UXOFF# only signals back to the LSC, but has no active function on the LDC. Probably the only buffer you need is the one that assembles a 16-bit or 32-bit dataword and makes a short pipeline to get the LDERR# line in parallel with the dataword.

When you select the chips to implement your design, you must check that it is able to implement a FIFO. Simple PALs, for example, only have a single clock signal for the registers and it will be very difficult to design a FIFO out of this.

4.4.3 Write Enable

How to make a Write Enable? Some programmable chips have special inputs on their logic blocks that implement a Write Enable. If your chip does not have this, be careful to implement it the correct way: it is not possible to generate a write clock out of an AND between the clock and the inverted UWEN#. This does not work since when UWEN# goes low while the clock line is already high, the data would be written, while it should only be written on the edge of the clock. The correct way is rather simple: the registers for the datalines and the UCTRL# line should all use an equation like the following (in ABEL language): 
DATAOUT.C   = UCLK;
DATAOUT0.D := (DATAOUT0 & UWEN_N) # (UD0 & !UWEN_N);
This makes that the output of the register keeps its old value when UWEN# is high, and that it gets the value of UD0 when UWEN# is low at the next clock cycle. The setup and hold time of UWEN# are the same as the data inputs (like is specified in the S-LINK spec).

4.4.4 Byte synchronization

Your link will probably be based on some physical link that works in a byte-wide fashion. Even many serial links work this way. If you have to send 32-bit or 16-bit words and some extra control bits over this link, you will chop up the word into byte-wide pieces that you transmit one after the other. When you do this, you must make sure that the LDC can still see which byte belongs where in the total transmitted word. You may call this byte synchronization. So you need a mechanism to mark either the first byte of a word transmission, or to mark each transmitted byte with its place in the total word. Of course you can also have something in your link protocol that guarantees in some way that you know where your bytes are. For example, at startup you can use some special protocol to make the byte synchronization. In that case, you still should have some system with which you can verify, on a regular basis, that the byte synchronization is still correct.

4.4.5 Throttling the link

You read in the spec that the Link Source Card has a signal called LFF#, Link Full Flag. Maybe you have not noticed, but this signal is available not only on the duplex version, but also on the simplex version of the S­LINK. On the duplex version, it's use is clear: when the LDC has given an XOFF, the LSC will stop sending data which means that the buffer on the LSC will fill and that LFF# will go low. But on both the simplex and duplex links, LFF# can also go low when a user transfers data faster to the LSC buffer than the LSC can move the data out onto the physical link. Also in that case the buffer will fill and LFF# should go down to prevent overrun. This means that Front-end Motherboard designs should always look at LFF# if no data may be lost at the LSC side.

4.4.6 Clocks in and out of the links

Just in case you did not realise yet: the clock that comes out of the LDC has in most cases a frequency that is not the same as the clock that goes into the LSC. The clock going into the LSC is coming from the Front-end Motherboard and can be decoupled from the physical link by the buffer on the LSC. The LDC clock (LCLK) has a frequency related to the clock that is used on the physical link.

4.4.7 Clocks on parallel links

When you implement an S-LINK with a technology that has a parallel physical link, you probably will use one line as a clock line and the others as datalines. You have to take care with a few things here. The first thing is that you should pick a line that has the least skew to all other lines as the clock line. This will mean that you have more setup and hold time for all other lines. You may test a few samples to see which line is the best. It could be one of the lines in the middle, e.g. bit 4 or 5 on a 10-bit link.

In fact it is not so simple just to use one line as a simple positive edge-triggered clock. The problem is that the clock frequency will be twice as high as the frequency of the datalines. This might become a problem if you want to get the highest speed out of your physical link. I can think quickly of two solutions for this problem:

The first solution uses, of course, one line less. But it could be that it is very difficult to implement. The problem is that most programmable chips can only latch data on the positive clock edge. That means that on the receiving side you will have to generate a double frequency clock, so you need some phase locked loop to do that. PLLs are normally a pain as they generate high-frequency noise and insert some jitter. You may also invert the clock and have one part of your destination chip run on one clock and another part on the inverted one. But in that case be very careful with the extra delay you introduce and the skew between the clocks and the data. On the source side you also must be careful with the extra skew you introduce when generating the half frequency clock.

The second solution seems more easy to implement, but indeed it consumes an extra line on your physical link. It also can become complicated if you try to send an odd number of bytes for each word transfer.

4.4.8 Timing

You have three options for the timing. You make link cards that can accept UCLK clock frequencies and may give out LCLK frequencies of up to either 16.5 MHz, 33 MHz or 66 MHz. With each choice corresponds a set of timing parameters such as Data Set-up time that the link cards must be able to accept or to give out. The choice of maximum clock frequency does mean that the link cards must be able to work with frequencies lower than the maximum, so you may not count on that the LSC will indeed get the highest frequency. For links that use a serial physical link this will mean that in most cases a FIFO is needed on the LSC to decouple the UCLK frequency from the clock used for the physical link.

For the LDC it means that the LCLK frequency that it may give out may not be higher than what it is specified for. It may give out a lower frequency. If you see a use for it, you may use this to your advantage. E.g. if you need four clock cycles on the physical link to receive a word, it may ease your design to only give out an LCLK with a quarter of the frequency of the clock used on the physical link.

Note that the maximum clock speed does not specify the maximum transfer rate an S-LINK can handle. The LSC may throttle the link by asserting the LFF# signal. E.g. a 33 MHz, 32-bit S-LINK that has a physical medium of 33 MByte/sec will only allow one word to be transferred per 4 clock cycles (if LCLK is 33 MHz). With the overhead of the error detection and control bits, the transfer rate may even be lower.

4.5 Control words

4.5.1 How to implement

What is this control word transmission really? It is a concept to send control information over the link. The idea is that the user does not have to introduce a protocol to get control information over the line. He just sets UCTRL# low and can then send data as control data. Note that the user cannot use the lowest 4 bits, as they are used by the link itself. The lowest 4 bits are chosen because that makes that the position will be the same for links with any datawidth.

You as a link designer, have different ways of implementing the sending of control words. If you have enough bits available on your physical link, you might send UCTRL# as a seperate bit, just like a normal databit. On other physical links you might want to send a special command to put the link in "control mode" or use a special type of frame to mark that data as being control data.

The link itself should not care about how and even if control words are used. It is expected that the users will use the control words only to delimit normal data in functional blocks. E.g. each event could be marked with control words showing the beginning and the end. This means that normally many more normal datawords than control words are sent. This means that if you need to, you may take more time to send a control word than a normal data word.

4.5.2 How can it be used

So you see that a user is completely free to define his own way of using the codewords. But since many users will have needs to show the beginning and end of datablocks, table 2 suggests a few codes.

Table 2: Suggested control words 

 Databit (UCTRL#=0)
 31-7  6  5  4   3-0  Meaning
 --------------------------------------------------------------- 
  x    0  0  0    x   Start
  x    0  1  0    x   Stop
  x    1  0  0    x   Reserved for future use 
  x    1  1  0    x   Reserved for future use
  x    x  x  1    x   All bits except bit 4-0 available to user
The Start and Stop code could be used to mark the beginning and the end of events. You could use bits 31-7 to show the type of event (e.g. physics data or calibration data). If there is a need to insert control words between Start and Stop codes, you should use a control word with bit 4 set to 1.

It is not obliged to use those control words, but if your application can map its needs on those, please do it. You will find that software will become available that supports it.

4.6 Data error detection

4.6.1 Error detection options

So you have seen that your link should have some error detection capability for the data sent. That is really what the users want: just error detection and no error correction.

You have the option of error detection on a word-by-word basis or on a block basis. Error detection on a word-by-word basis looks to the user as a superset of the block basis error detection; the behaviour of the bits in the control word and LDERR# is the same. The only difference is that the LDERR# line is not set directly during an error on a data word.

The user has two ways to see an error has occurred. It is shown by LDERR# low, and, if control words are used, the errors are shown in the databits of the control word itself as well.

4.6.2 Error detection on a word-by-word basis

If you implement error detection on a word-by-word basis, you must be able assert the LDERR# line together with the data that is in error. So your choice is rather limited. I suggest to just implement a parity over the data and send this in parallel with the data. You may implement a parity bit for each byte plus a parity over the parity bits. This latter parity bit may also include other bits such as the control bit. This means that for a 32-bit datalink, you would need to send 5 parity bits with each 32-bit dataword. If you run out of bits and can easily implement a parity over 16 bits, you may do with 3 parity bits in total.

If your physical link has its own way of detecting errors, you may use that as well to drive the LDERR# line. But make sure that you really mark the word that is in error and not a word that comes out later.

If the LDC detects an error in a dataword, it should set some internal flip-flop. The value of this flip-flop will be sent as bit LD[0] in the first following control word, showing there was a transmission error in the previous datablock. In that case, you also should pull LDERR# low. After transferring this control word, you have to reset the flip-flop to arm it for the next datablock. If you detect an error in the control word itself, bit LD[1] has to be set in the control word, and LDERR# has to be pulled low as well.

4.6.3 Error detection on a block basis

If you implement error detection on a block basis, you may only pull LDERR# low during the transfer of a control word. The behaviour of bits LD[1] and LD[0] in the control word and the LDERR# state for the control word are exactly the same as for the error detecion on a word-by-word basis.

For block basis error detection, you probably will use some CRC-type of error detection. As a CRC is only reliable when calculated over a certain maximum number of words, you must make sure that you send a CRC regularly. You don't know how many datawords a user sends between two control words, so you may not postpone sending the CRC until a control word is sent. How many words you send before inserting a CRC is dependent on the type and size of the CRC. This means that in case you detect a CRC error, you have to set an internal flip-flop and output the value of this flip-flop as LD[0] in the first control word that comes. The behaviour is exactly the same as you do for the error detection on a word-by-word basis.

Make sure that the control word itself has its own error check, and is not included in the calculation of the CRC of the data. This is because they have independent status bits (LD[1] and LD[0]). It means that when a control word must be sent, first a CRC over the data transferred until then has to be calulated and sent before the control word with its own CRC or other error detection will be sent.

4.6.4 Error LED

The S-LINK specification defines a Data Error LED. During normal operation, this LED should turn on when a data error has occurred, that is, when LDERR# gets asserted. It should only go off when the button on the front panel is pressed, or when the LDC is reset. The LED may not go on if you detect a problem on your link such as when you lose synchronization on a serial link. For that you have the Not LDOWN# LED, which you should turn off during that time. Note that when your link is "self recovering", the user will not care that there has been an intermittent problem while he was not sending any data.

4.7 Flow control

4.7.1 How to implement

If you are building a duplex link, you should implement the flow control as described in the S-LINK spec. The mechanism is pretty simple. When you see, on the LDC, that the UXOFF# line goes low, you have to send an XOFF token to the Source, which then must stop sending data. When the UXOFF# line goes high again, you may send an XON token and the transfer can resume. In fact, you should send an XOFF or XON token regularly or even continuously dependent on the level of the signal UXOFF#. This way, even if a token gets lost, the link will continue to work. How you implement the token is up to you: it may be a bit or some special code you send back.

4.7.2 Source filling

When the LSC receives the XOFF token, the LSC must stop sending data to the LDC. When the FEMB transfers data to the buffer in the LSC, the buffer on the LSC will fill which will ultimately set LFF# low. There is no need for the LSC to set LFF# low immediately after it receives an XOFF token.

4.8 Return Link

4.8.1 For slow pulses

The physical link back is not only used to transfer the return lines, but also the XON and XOFF codes. If your implementation of the link cannot send XON/XOFF codes and return line data at the same time, it could be that even more time will be introduced between the changes on the line. As the minimum pulse width on the return lines is defined as 32 clock cycles, it gives you an idea on how much delay you may introduce. Note that indeed the S-LINK may change the pulse width on the return lines to minumum 16 clock cycles. Note as well that the S-LINK does not define the transmission delay. It may change from line change to line change, so it cannot be used to send high-resolution timing information. With some physical implementations of the S-LINK, e.g. on parallel links where the return lines are just sent over as separate lines, it may be better though.

Note that with clock cycles the clock cycles of the clock coming out of the LDC (LCLK) is meant, not the clock that goes into the LSC.

4.8.2 No buffering or FIFO

As the return lines really behave as separate lines that go from the LDC to the LSC, there is no need or even possibility to use a FIFO for the return lines. Only some registers will be needed to latch the data. Note that on the LDC the signals come in asynchronously, so metastability problems can occur here. This should not be a worry as the signals don't carry any timing information.

4.8.3 Errors on the Return Lines

There is no way that the link can feed back to the user that an error occurred on the return path. But there are a few things that you must do as a link designer.

First of all you should implement some error detection system such as a parity. This error detection may include the XON/XOFF bit as well. In case you detect an error, you should just keep the last value on the URL lines and hope that the next time the return line data is sent it will not contain an error. This way you hide the problem and the system will recover automatically. The only thing that will happen is that the pulse width may change, but the system was not set up to send short pulses, anyway. For you as link designer, it means that the LDC should send regularly the status of the URL lines, and not only when changes occur.

The second thing that you should do, is to pull the LDOWN# line low when you receive so many errors in a row that the specification of a minimum pulse output of 16 LCLK cycles can't be met. As the minimum pulse width on the input is 32 LCLK cycles, it means that if you can't update the return lines for 16 LCLK cycles because of the errors, you should pull LDOWN# low.

4.9 Testing

4.9.1 Aims

The aim of the test mode as specified in the S-LINK spec is not to do a full functional test of the boards. It's aim is to test the physical link (fibres, connectors) and much (not all) of the link card electronics. In most cases this is enough since in practice boards work completely or don't work at all. This means that you as link designer should not overcomplicate it.

4.9.2 Test pattern

The test pattern to send over the physical link is a "walking 1" over all databits and the control bit. This is something that is relatively easy to implement with programmable logic. Dependent on the datawidth selected by the user, you should send 9, 17 or 33 datawords before starting to send the same data again. You should always send a multiple of this number of datawords, as this allows the user of the LDC to check if extra words are generated or if words are lost. Before sending the test pattern, or maybe on each word you send, the LSC must send a signal over to the LDC to say that the data coming is test data.

It could happen that the LDC is not ready to receive the data, e.g. it is just coming out of reset. I recommend that the LDC, when it comes out of reset and sees that the data is test data, should only start checking the data, or outputting the test data to the user, when it receives the first word of a complete test cycle, the one with LCTRL# low. This will make it easier for the software on the ROMB to verify the data, as the first dataword is always the same, whether the LDC just came out of reset or not.

Remember that you should only output the test data (i.e. have LWEN# low) when the UTDO# input is low.

4.9.3 How to implement

The LFF# signal has to stay functional during the test mode. So on the LSC you should just stop emptying the buffer that you have at the input, such the LFF# ultimitely will go low. This is exactly the same behaviour as when the LSC received an XOFF command, so you may reuse some logic here. Possibly you can use the last stage of the buffer to implement the shift register for the test data. Note that you need a global reset for those registers to get all 0's in.

4.9.4 No gaps between test words

If you implement some kind of command to put the LDC in test mode, you should make sure that the LSC will start sending the test data immediately and without gaps. The LDC should indeed expect that it will receive immediately the test data after the command. Otherwise, if the LSC does not start sending the data after the "go to test" command, a user has no way to detect this. He would just see the Test Mode LED go on on the LDC. The Data Error LED would not go on as the LDC would not have received any data. Or, you can make the test LED go on only after the first dataword is received. After that, the LDC must expect all test words in one block, since if the transfer was blocked after the first dataword, again no error would show up. When I mean in one block, or without gaps, it does not mean each clock cycle. It may be that only one dataword will come out every four clock cycles. But then the LDC should expect the data indeed every four cycles, and not more.

4.9.5 Function of LEDs

As a link designer you must make sure that the LEDs "Test Mode" and "Data Error" only light up during the test mode. You should turn both of them off immediately after going out of test mode. This is just to make sure that the person who runs the test will verify that the link indeed went into test mode. Otherwise, he could start the test, stop it and only after that look at the "Data Error" LED to see if there had been any problem. But if the link had not gone into test mode at all, he would not see this. Implementing it so that the LEDs always go off after going out of test mode, will oblige the user to take a look at the LEDs while the test runs.

4.9.6 Testing duplex links

Testing duplex links is more complicated than testing the simplex links. In this case you should also test the physical return link. You should not test the return lines as they have to stay fully functional during the test mode.

As there is no Error LED on the LSC, errors on the return link should be reported back to the LDC. Also the user must be able to see if both directions of the data transfer are correctly put in test mode. Therefore some protocol is needed. Table 3 shows a protocol where commands are sent and actions are taken that handle the duplex link test.

Table 3: Duplex Link test protocol 

A.  LSC -> LDC  "Request Test Mode" (Test Mode LED on on LSC)
B.  LSC <- LDC  "Request Test Mode Received"
C.  LSC -> LDC  "Start Test"
D.              (Test Mode LED on on LDC)

E1. LSC -> LDC  Test data on data bits and control bit
E2. LSC <- LDC  Test physical return link
F1.             if error detected by LDC, turn on Data Error LED
F2.             if error detected by LSC, LSC should send error command
Steps E1,E2 ,F1,F2 are running concurrently. If the LSC detects an error, it may signal this to the LDC by sending out a testword with an error in it instead of sending an "error command". This would save hardware.

The test of the physical return link to the LSC should not include the return lines. These shall stay functional as they may control critical functions in the system. However, you may include the XON/XOFF bit, if in your implementation that is a separate bit sent. Or you can use some spare bit to see if the link is working. You also should make sure that data in both directions is received within a certain time after the "Request Test Mode Received" and "Start Test" commands are received. On the other hand, don't make the test mode too complicated, as the aim of the test mode is not to do fully functional tests.

4.9.7 Priority of reset, test and normal data

So we have, kind of, three modes of the link: reset, test and normal operation. The reset has the highest priority. It is really a hard reset and should make all activity stop. The user may try to transfer data, but it may be lost. When the link is put into test mode (UTEST# low, URESET# high), this behavior is similar. Only when both UTEST# and URESET# are high, is the link in normal operation. Note that even when UTEST# goes high, the link still should finish off the transfer of the current test pattern. You as link designer should make sure that you will accept data transfers when UTEST# is pulled high again. You might fill up the FIFO on the LSC and put LFF# low to signal that you cannot accept any more data, or even keep LFF# low until you finished the test cycle.

4.9.8 What to do if a test fails?

It is specified that if during the test mode an error is detected, the LDERR# line should be pulled low. That is indeed the only thing it should do. You should not pull LDOWN# low, or disable the link. So when the link gets out of the test mode, it should try to work as normal, even when an error is detected. In case one bit is failing, many users will still prefer to receive 31 bits of data instead of nothing at all.

4.9.9 Production tests

As said before, the test mode is not meant as a production test. For this you should write test programs that send complex patterns over the link and put the link into several states, such as XOFF and so. This type of test is very dependent on your implementation of the S-LINK.

I will not go into detail on how to do extended manufacturing tests of your boards. But during the debugging of your cards, you may run the board at a low Vcc voltage (e.g. 4.5 Volt) and at a high Vcc voltage (e.g. 5.5 Volt). This would make the logic behave as if the cards were operating in a high or low temperature environment respectively. I'm sure that it will give you some extra information on the robustness of your boards.

It could be that the chips that you use on your boards support JTAG. The S-LINK connector does not have any pins left to put the JTAG signals on, so you may want to put a little connector on the board that carries those signals. When your design is integrated onto the motherboard, you can make it that the JTAG loop can always be accessed under program control. This will give you extra possibilities for testing when the system in installed.

4.10 Reset

The reset can be the most complex part of the design of your link. Especially on serial physical links, you will need to do something to get both LSC and LDC synchronized and in a known state. It is not specified in the S­LINK specification as the way you implement this is very dependent on the technology you choose.

For a duplex link, you may not assume any order in which the LSC and LDC are put into reset and when they go out of reset. By having a kind of "link reset protocol" you should make your reset so robust that it will not block. On a simplex link, you may assume that the LDC will be reset first.

To ease your job, you may throw away data that is sent to the LSC when it is in reset; the user is not supposed to send any data during that time. The LDC may not send any data to the ROMB when it is in reset. It should keep LWEN# high.

For each side of the link, unless LDOWN# was already low, the LDOWN# signal should go low within 4 clock cycles whenever it is put in reset. When the link card has finished the reset, or has finished setting up the link to the other side, it should get LDOWN# up again.You don't need to do a complete test of the link before getting LDOWN# up.

The user will not attempt to transfer data to the LSC within 4 clock cycles after it removed URESET#. You will need some of those 4 cycles to synchronise the URESET# line to the internal clock.

You should prevent the Data Error LED on the LDC lighting up because of problems during reset. The Data Error LED should only go on when an error on the data has occurred. So you must make your link in such a way that you don't have to press the reset button on the LDC after every reset!

I'm sorry that I can't help you anymore in describing the reset procedure. I realize it is a trickiest part of the design and hope you manage to make something robust here. Just take your time and do some simulations to test out a few ideas!

4.11 Hardware

4.11.1 Component selection

There are a few important things that you should look at when you choose components for the link cards:

4.11.2 Design tips

When you design the boards, there are of course all the normal precautions to take that you always do for high-speed designs. I can't make an exhaustive list here, but there are a few points I would like to draw your attention to.

As there are only five Vcc pins on the LDC connector you must make a good decoupling of the Vcc. So as well as the normal decoupling of each chip on your board you should put a capacitor or two of, say, 6.8 µF near the connector.

As the bus is 32-bit wide, it is possible that you will get some crosstalk. Crosstalk between the datalines should not be a problem because it will only occur when the lines are changing anyway. But you must be careful with any crosstalk that can influence critical lines, such as WEN# and the error lines. As they are not related to any clock, the most critical lines will be the return lines, so you should keep those a bit away from the datalines (like they are already at the connector). 

5. Shopping list

5.1 PMC connectors

The connector that is used on the S-LINK is exactly the same one as used on the PMC boards. You may buy them from AMP or from Molex. Two different sets of connectors exist, one with locating pegs, and one without. Both have a stacking height of 10 mm. The impedance is around 72 Ohm.

The AMP part numbers are:
- With locating pegs: 120527-1 (plug), 120521-1 (receptable)
- Without locating pegs: 120534-1 (plug), 120528-1 (receptable)

The ones with locating pegs are now CERN Stores items with the following SCEM numbers:
09.55.40.064.7 (plug) and 09.55.41.064.3 (receptable)

You may also buy AMP connectors from
Mr. Georges Viñals
AMP (Schweiz) AG
Tel: 032-342 52 49
Cell: 077-96 57 82
Fax: 032-342 52 50

The Molex part numbers are:
- With locating pegs: 53483-0649 (plug), 52763-0649 (receptable)
- Without locating pegs: 53508-0648 (plug), 52795-0648 (receptable)

In Switzerland, you may buy the Molex connectors from
E M E  AG
Interconnection & Motion
Lohwisstrasse 50
CH-8123 Ebmatingen
Tel:   ++41 1 982 11 11
Fax: ++41 1 982 11 22
http://www.eme.ch
 

5.2 Hardware

The front panel recommended for the S-LINK cards is the same used for PMC boards. They are made by the company Schroff and have an EMC protection joint. Also the keying pins and the standoff pillars are made by Schroff. The part numbers and approximate pricings are: 
20835-598  Mezzanine front panel with EMC protection joint  SF 18.00
60827-026  Keying pin                                       SF  0.60
60897-278  Standoff pillar                                  SF  0.60
You may buy Schroff components from
ROTRONIC AG
Tel: +41 32 341 80 05
Fax: +41 32 341 80 06

5.3 CMC spec

The S-LINK follows pretty closely the mechanical specification of the IEEE standard P1386, called Draft Standard for a Common Mezzanine Card Family: CMC. If you need more detailed mechanical descriptions of the card size, you may want to get a copy of this CMC spec. In September 1995 it was Draft 2.0, dated April 4, 1995. For your purposes, the version would not make a big difference.

If you can't get a copy from one of your friends, in Switzerland you should be able to get one from

Technisches Fachbuch-vertrieb
Tel: 032-322 61 41
Fax: 032-322 61 30

The price is around Sfr 100 and it can be sent to you within two weeks. 

Let me use the last line to wish you to have fun designing or using an S-LINK! And, if you have any problems or requests about the S-LINK, just send me a mail. Erik 

Updates to this document