RD-12
Approved April 91
Status Continuation

Readout System Test Benches

CERN, LPNHE Ecole Polytechnique Palaiseau INFN Padova, INFN Pavia, KFKI Budapest, LAPP Annecy, MIT Boston, SEFT Helsinki, Tampere University of Technology, University of California San Diego.

Spokesman: S. Cittolin

1 Main goals

The RD12 project [1,2,3,4] encompasses a range of studies on topics related to the front-end electronics and data acquisition systems of future LHC experiments. They include:

2 Main results and current developments

2.1 Timing, trigger and control distribution

All the subdetectors of the proposed ATLAS, CMS and ALICE experiments require quite extensive distribution systems for the transmission of timing, trigger and control signals to large numbers of front- end electronics controllers from single locations in the vicinity of their central trigger processors. The systems must provide for the synchronization of the front-end electronics, compensating for different time-of-flight, electronics and propagation delays, and deliver properly phased bunch numbers to the controllers with each trigger decision as well as broadcast and individually-addressed commands and data.

In conjunction with RD27, RD12 is developing a multi-function optoelectronic distribution system [5,6,7] which has the potential to meet the requirements of all the proposed subdetectors of the LHC experiments. The implementation of a common solution to this problem would result in economies of scale and result in a rationalization of the development, operational and support efforts required.

The RD12/RD27 system is based on the use of a few relatively high power laser sources which distribute the signals via entirely passive all-glass networks composed of a hierarchy of optical tree couplers. The couplers have small size, low mass and unlimited bandwidth, require no power and are potentially highly reliable, while the single-source architecture allows the laser to be well protected and easily upgraded as technology progresses. The laser operates at 1310 nm, at which the chromatic dispersion of the fibre is negligible.

It has been shown that it is possible to time-division multiplex and encode the first-level trigger decisions, broadcast commands and individually addressed controls and data for optical transmission to several thousand front-end electronics destinations per transmitter. (Successful operation has been demonstrated through three levels of 1:32 couplers, corresponding to an optical fanout of 1:32768).

The ‘prompt’ TDM A-channel which is dedicated to the broadcasting of the first-level trigger-accept signal has been designed for low latency. The total delay from the trigger-accept input from the central trigger processor to the output from the timing receiver is less than 2 bunch-crossing intervals plus one bunch-crossing interval per 5m of optical fibre in the distribution path. The bunch-crossing timing reference can be recovered from the received encoded data stream with a time jitter of 50 - 200 ps rms (depending on the optical fanout), comparable with the spread in event origin time due to the LHC bunch collision length and expected longitudinal phase modulation of the circulating beams.

These facilities can be implemented with the relatively low transmission rate of 160.32 MBaud, permitting the use of InGaAs photodetectors manufactured in increasingly large volume for the expanding Sonet OC-3 telecommunications market. Extensive deployment plans for fibre in Europe and the USA suggest that these components will become available at affordable cost during the time frame of LHC preparation, and industrial collaboration is in progress for the development of sub-miniature optical fibre connectors and device packaging.

The development of a single-crate transmitter subsystem has been completed and a small series production is in progress for the first users. The subsystem comprises the clock generator and receiver, 160.32 MHz VCXO/PLL, TDM biphase mark encoder, laser transmitter, thermoelectric cooler and laser diode controllers, protection system, RF modulator, 1:32 root coupler and a test module. External VMEbus B- channel serializer and master phase modules are currently being developed. A lower cost mini-system has also been produced which generates compatible signals for front-end electronics development and integration work.

The architecture of an associated timing receiver ASIC has been defined and its design is being undertaken by CERN ECP Microelectronics Group in conjunction with Padova University/INFN. This chip, which will be required in large quantities, accepts a single input from the photodetector and generates a full range of decoded and deskewed signals for front-end electronics controllers. The ASIC comprises an analog part (including the postamplifier, AGC circuits and clock recovery/fine deskew PLL) and a digital part (including the decoding, demultiplexing, coarse deskew and command processing sections).

Technology evaluation chips including alternative implementations of basic elements of the analog part are currently being fabricated, while the digital part has been simulated and an FPGA implementation will be tested before proceeding to custom silicon. It is anticipated that hybrid timing receivers will be available early in 1995 while a fully integrated timing receiver ASIC should be completed about 9 months later.

2.2 Dual port memories and data link interfaces

High speed data buffering is required at different levels in the readout chain of LHC experiments. The basic data acquisition unit for all sub-detector readout systems is expected to be a programmable message- driven multi-port memory (DPM) with high throughput (100 MB/s). Moreover, during the present phase of design and evaluation of readout components, dual port memories are the basic test tools to generate and acquire data to/from a high speed digital system such as a front-end readout or event builder switch. RD12 is developing a series of DPM modules for applications ranging from testing in realistic LHC conditions to final data acquisition systems.

A VME-based Fast Dual Port Memory (FDPM) [8] has been designed and 15 modules have been produced in versions with ALTERA and MAG control logic [9]. They are used in several R& D projects and by external laboratories. The modules, which allow up to 400MB/s input-output from a 512 MB static memory, are mainly used as test tools to generate and acquire data in front-end digital systems such as ADCs, digital filters and fast data links.

The continuation of this study is the design of a more advanced unit able to support the full data acquisition functions required in an LHC experiment, in order to implement the front-end readout and to generate and collect data in an event builder switch-based system. The new design consists of a 100 MB/s two-port memory of 2MB, expandable up to 100MB, with microprogrammable sequencers to read out front-end electronics and to drive a fast output data link.

The unit behaves as a disk controller operating at an event rate of up to 100 kHz, with the capability of opening/writing/closing files (event buffer) and opening/reading/deleting files under the control of external command signals. The memory control may be implemented by hardwired logic or by embedded processors (both ways are being studied). The exploitation of standard buses, such as PCI, for peripheral interconnections is also being pursued. A Fibre Channel (FC) interface to connect the memory output to a switch port is under construction. The protocol generation is entirely hardwired in order to communicate with a commercial FC switch at the maximum speed.

2.3 Digital signal processing algorithms and tools

The work on digital signal processing algorithms has concentrated on the optimal filtering problem and efficient base line estimation. In optimal filtering the amplitude and the time position of acquired pulses is extracted with the highest possible accuracy. The pulses are corrupted by several artefacts: Gaussian-like electronics noise, Laplacian-like pile-up noise and sample timing jitter. As the sample timing jitter results in a multiplicative error in the sample values it is the dominant source of error for high energy pulses. The optimal filter is dependent on the noise distribution: in a Gaussian-noise environment it is the matched FIR filter, while in Laplacian-noise conditions it is the matched median filter. To obtain the maximum performance in a mixed-noise environment we combine linear and non-linear operators into a hybrid filter. The FIR - order statistic hybrid (FIR-OS) filter is a two-level structure in which the outputs of the parallel FIR filters are processed by the OS operator to obtain the final filter output [16,17,18].

To perform the filter optimization a design tool called FIROSMIN has been created (the software is available on the CERN caeftp server). The applied method is ‘design by training’, where the filter parameters are found by minimizing the mean squared error (MSE) between the filter output and the desired output. The procedure is very similar to the learning process of neural networks, but the operator derived is a digital filter with a predefined structure. Adaptive filtering has a common optimization strategy but the adaptation is performed during operation whereas the training is performed prior to operation. Thus adaptive filters can adapt to changing conditions in real-time but they have the drawback of being computationally intensive. The training method is especially suitable for cases where the specifications are given in the time domain and not in the frequency domain, and for adaptive non-linear hybrid filters which are difficult to design using the analytical approach.

Stochastic algorithms offer an alternative to gradient-based optimization. To investigate their utilization the SAFIR tool, incorporating simulated annealing as the optimization algorithm, was developed. The tool also includes a VHDL output generator to test the designed filters in a Xilinx environment.

For signal base line estimation a set of sparse nonlinear filters has been developed. As the signal pulses can be considered as outliers, linear operators like moving average give a strongly biased estimate [11]. Non-linear filters perform better, producing a totally unbiased base line estimate when a sufficient window size is used. In order to overcome the problem of sorting a large number of samples, sparse structures are used: the input is obtained by downsampling the original set. The sparse filters evaluated include the OS filter, the L-filter and the Ll-filter [13,14]. The operators were optimized using the training approach.

Filter structures have been implemented in hardware as part of the FERMI readout microsystem (RD16). Some algorithms have been tested using the DataWave processor simulator developed by ITT Intermetall. DataWave is a single-chip multiprocessor system, originally designed for image processing tasks in digital television applications. Each processor cell is an autonomous RISC with local program and data stores. Due to the high level of parallelism and the optimized instruction set the DataWave processor achieves a maximum performance of 4 GOPS [12,19].

2.4 High speed video processor applications and VLSI developments

The application of industrially developed digital circuits for High Definition Television has been studied by LAPP. An 8-bit digital chain including a linear filter and a pipeline was implemented in 1991-92 using the MULAC8 (8-tap FIR) and LRV16K (digital pipeline) circuits designed by the Centre National des Télécomunications (CNET) in Grenoble. The filter module was extended in 1993 to run a 16-bit filter and it was integrated with a logarithmic amplifier, a 10-bit flash ADC and a look-up table. A digital channel of this type has been used to acquire and process the calorimeter data in a test beam (RD3) [23]. A silicon compiler for general linear filter design has been written in collaboration with CNET. This software has been used to synthesize the design of an 8-tap 17-bit linear filter FIR chip running at 100 MHz [24]. The prototype circuit realised in 0.7 µm CMOS technology is under test.

Another application of video industry developments to LHC trigger and front-end digital systems is being studied in a collaboration between LPNHE-X and Philips. The design of a circuit composed of 16 parallel 16-bit integer DSPs running at 40 MHz in pipeline mode has been completed. LPNHE-X is responsible for the data communication part, the DSP software development and the prototype tests which were started in July 1994. The study will continue with the completion of the VHDL description of the chip behaviour and the conception and realisation of VME boards with multiple L-Neuro chips for applications to calorimeter signal and level-1 trigger processing.

In Padova the Memory Address Generator (MAG) circuit has been produced using CMOS 1.2 µm technology and integrated in the FDPM modules. It allows programmable address generation at a rate of 100 MHz. A switching capacitor CMOS discriminator [35] suitable for LHC instrumentation has been designed and 1.2 µm CMOS prototypes tested. It runs at 66 MHz with 10 mV threshold and 20 mW power dissipation. The design is the building block for front-end ADCs.

2.5 Modelling and simulation

Modelling and simulation of all the readout elements, at the single component and global system levels, complements the prototyping test activity for the specification and design of data acquisition architectures. The system environment of the modelling and simulation tools must include the facilities to perform the component specifications in a framework common to the hardware and software design. The industry- standard description language VHDL has been chosen as a means to specify component structures and to simulate the behaviour of real time systems. The simulation of front-end and readout components has been initiated in a systematic way, starting with the general logical structure of the front-end readout, the timing distribution system and the event builder elements. Detector specific models will be implemented in collaboration with CMS. The component graphical editor SPeeDCHART (SpeedSA) and the VHDL compiler Voyager (Ikos) are the software tools adopted so far.

2.6 Test bench software tools

LabVIEW is an icon-based programming system for building software modules called virtual instruments (VIs) which has been developed by National Instruments since 1986. It is a general-purpose programming tool for data acquisition, data analysis and instrument control based on data flow graphical, object-oriented programming technology. RD12 adopted LabVIEW as the main software system environment for the test benches running on Macintosh and Sun workstations.

The set of virtual instruments developed by RD12 supports CAMAC and VMEbus systems, accessed by MacVEE or CES and NI-MXI interfaces, as well as histogram, data acquisition and filter design libraries [28,29,30]. The LabVIEW packages and documentation are available by FTP: Machine caeftp, Login anonymous, Password e-mail address, Access /home/ftp/labview/CERN.LIB/.

References and main publications

  1. RD12 proposal. CERN/DRDC 90-63.
  2. Status report on the RD12 project. CERN/DRDC 92-10.
  3. Status report on the RD12 project. CERN/DRDC 93-22.
  4. Status report on the RD12 project. CERN/DRDC 94-16.
  5. B. G. Taylor, Multichannel Optical Fibre Distribution System for LHC Detector Timing and Control Signals, Conf. Record IEEE Nuclear Science Symposium, Orlando, Florida, 25-31 October 1992.
  6. B. G. Taylor, Optical Timing, Trigger and Control Distribution for LHC Detectors, Conf. Record IEEE Nuclear Science Symposium, San Francisco, California, 31 October - 6 November 1993. To be published in IEEE Transactions on Nuclear Science, Vol. 41, August 1994.
  7. B. G. Taylor, Timing, Trigger and Control Distribution for LHC Detectors, working review document, CERN/ECP-RA.
  8. A. Fucci, G. Heguesi, J. Kolbinger, M. Lomo, H. Masuch, C. Pirotte, Fast Dual Port Memory Module, CERN/ECP/93-4.
  9. S. Centro, R. Martinelli, D. Pascoli, F. Dal Corso and A.J. da Ponte Sancho, MAG Memory Address Generator. DFPD 93/EI/30.
  10. L. Pollet. A Racz, Fiber Channel data link. RD12-TN October 1993.
  11. S. J. Inkinen, in Proc. Workshop on Image Processing for Future HEP Detectors, Erice, Italy, 1991.
  12. S. J. Inkinen, in New Computing Techniques in Physics Research, World Sci. Publ. Co., 1992.
  13. S. J. Inkinen and Y. Neuvo, in Proc. IEEE Winter Workshop on Nonlinear Digital Signal Processing, Tampere, Finland, 1993.
  14. S. J. Inkinen and Y. Neuvo, submitted to IEEE Trans. on Instrumentation and Measurement, 1994.
  15. S. J. Inkinen et al., in Proc. European Conference on Circuit Theory and Design, Davos, CH, 1993.
  16. S. J. Inkinen and J. Niittylahti, submitted to IEEE Trans. on Circuits and Systems II, 1993.
  17. S. J. Inkinen and Y. Neuvo, in Proc. European Signal Processing Conference, Edinburgh, Scotland, 1994.
  18. S. J. Inkinen and Y. Neuvo, in Proc. IEEE Nuclear Science Symposium, Norfolk, Virginia, 1994.
  19. S. J. Inkinen, IEEE Trans. on Instrumentation and Measurement, 1994.
  20. J. Niittylahti et al., in Proc. IEEE Winter Workshop on Nonlinear Digital Signal Processing, Tampere, Finland, 1993.
  21. J. Niittylahti et al., submitted to IEEE Trans. on Neural Networks, 1993.
  22. J. Niittylahti et al., submitted to IEEE Trans. on Neural Networks, 1993.
  23. Boniface et al., Chaine d'acquisition avec filtre numerique. Groupe de Recherche et developpement LHC-TVHD. LAPP internal report 11. 10. 93.
  24. M. Aqachmar et al. A compiled 100MHz programmable FIR filter chip for data acquisition. Paper in preparation (LAPP-CNET) 1994.
  25. S. Centro et al., Switching Capacitor CMS Discriminator. Submitted to IEEE J. of Solid State Circuits.
  26. R. Bonino, JP. Gros, L. Pollet, Analog memory test bench. RD12-TN February 1992.
  27. JP. Gros, L. Pollet, MEC2-VME system test board. RD12-TN February 1992.
  28. M. Demoulin, LabVIEW Fast Dual Port Memory library. RD12-TN February 1992.
  29. M. Hansen, O. Ledorz, ADC Test and Evaluation System. RD12-TN March 1992.
  30. S. Cittolin, LabRSTB LabVIEW libraries for VME, CAMAC and histograms. RD12-TN February 1993.
    THESES
  31. M. Kuusisto, Fast Dual-Port Memory for the Large Hadron Collider's Readout System Test Benches. Master's Thesis Tampere University of Technology, 13. 11. 91.
  32. S. Inkinen, Digital Signal Processing Methods for Event Triggering in Particle Detectors. Master's Thesis Tampere University of Technology, 16. 10. 91.
  33. S. J. Inkinen, Advanced Digital Signal Processing for Particle Detectors, PhD Thesis, to be published.
  34. M. Aqachmar, Developpement d'un filtre digital RIF a grande dynamique. Thesis, Univ. Grenoble 1994.
  35. J. M. Tomŕs Fernández, Introduccion al sistema de adcquisición de datos para LHC. Thesis, Universidad Autónoma de Barcelona 1994.