Long-Wire Data Acquisition Specification

Contents

Introduction

The name LWDAQ stands for Long-Wire Data Acquisition, and refers to the length of the cables that connect LWDAQ devices, LWDAQ multiplexers, LWDAQ Repeaters, and LWDAQ drivers. These cables can be up to 130 m long. They supply power and control signals from the drivers to the devices, and carry analog and digital signals from the devices to the drivers. A LWDAQ makes only one measurement at a time. Large LWDAQ systems perform thousands of measurements in sequence, and take thousands of seconds to complete their entire data acquisition cycle.

All LWDAQ cables are interchangeable. Each cable is a network cable, very similar or identical to a CAT-5 cable, that contains eight wires. All eight wires take part in the connection. Four carry analog and digital power. The reminaing four make up two twisted pairs. One pair carries digital comminucation from the driver. The other pair carries digital or analog communication from the device. Both the digital and analog signals are LVDS (low-voltage differential signals).

The LWDAQ transmits sixteen-bit addresses to its multiplexers and sixteen-bit commands to its devices. Each sixteen-bit transmission takes 4 μs. Some devices turn on lasers or LEDs when they receive the correct commands. Some return analog voltages that the driver digitizes and records. Image-capture devices allow the driver to read out pixel voltage levels directly, and digitize them with correlated double-sampling. Devices that produce digital data instead of analog data transmit their data to the driver at the driver's command one byte at a time. Data transfer in this way takes place at roughly 1 MByte/s.

The LWDAQ performs correlated double-sampling at up to 2 MSPS over a 130-m cable by transmitting a 2 MHz pixel clock from the driver to the device. The device returns a new pedestal voltage on the HI portion of the clock and a new sample voltage on the LO portion. The driver digitizes the difference between the sample and the offset and stores the eight-bit result in its own memory. Each sample takes 500 ns, but the time taken for the clock signal to travel to the device along a 100-m cable, plus the time taken for the analog signal to return over the same cable to the driver, is 1000 ns. The driver synchronizes its digitization with the returned analog signal by measuring the loop time of the cable before it begins clocking the analog signal.

All contemporary LWDAQ systems use some form of LWDAQ server. The server makes the LWDAQ available over TCPIP using the LWDAQ message protocol. We provide software that communicates over TCPIP with LWDAQ servers, and we invite you to download this software from our website. Other users have used Visual C++ and LabView to communicate with LWDAQ servers.

This specification may not be exactly the document you need to answer your questions about the LWDAQ. The following table lists several other likely documents.

If you are a user of the LWDAQ, your best place to start is the LWDAQ User Manual.

Architecture

The LWDAQ protocol supports the architecture shown below. LWDAQ drivers connect to repeaters, multiplexers, devices and sensors. Sensors connect to devices. We use the term "sensor" to include not only sensors, but also light sources, actuators, and any other outputs a device might have. Devices are slaves of the driver. Communication between the driver and its devices can pass through repeaters and mulitplexers. Repeaters restore outgoing control signals mid-way along lengthy cables. Repeaters also allow the driver to switch off power on the other side of the repeater. Multiplexers are simple switches that direct communication to one of several devices. The following figure shows the various ways in which LWDAQ components can be connected together.

Figure: LWDAQ architecture. Connection to the outside world is via TCPIP.

As you can see from the diagram, single device can control multiple sensors, light sources, actuators, and electrical outputs. We refer to all device inputs and outputs as sensors.

Example: The Resistive Sensor Head (A2053) is a LWDAQ device that provides eleven resistive sensor connections. The A2053A allows us to connect eleven 1000-Ω RTDs. The LWDAQ reads the sensors out sequentially with precision 0.02 mK. The twisted-pair wires to the sensors can be several meters long.

Example: The Polar BCAM Head (A2051) is a LWDAQ device that controls four lasers and reads out two TC255P image sensors. The lasers and image sensors are arranged on two auxilliary boards connected to the A2052 by flex cables.

The LWDAQ connects to the outside world over a TCPIP network, such as the Internet or a Local Area Network. We use our LWDAQ Software to control our LWDAQ systems, but some users write their own control software. One user has written a LabView interface.

All LWDAQ cables are inter-changeable. Two wires carry LVDS serial transmissions from the driver, two carry acquired data transmissions from the device, and four carry power. The driver can transmit sixteen-bit address words, or sixteen-bit command words. Multiplexers respond to address words, and devices respond to command words. A single TCPIP socket might give access to only a few devices, or it might give access to thousands.

Example: The LWDAQ Driver with Ethernet Interface (A2037E) has one TCPIP connection and provides eight driver sockets for root cables. Image data download speed from the driver is 160 kBytes/s, or two TC255P eight-bit gray-scale images per second.

Example: The TCPIP-VME Interface (A2064F) sits in a VME crate and provides a single TCPIP connection. The A2064 gives access to as many LWDAQ Driver with VME Interface (A2037A) as you can fit in the crate. In ATLAS, our crates each contain 20 A2037As, each with eight driver sockets, so that each TCPIP socket controls 160 driver sockets. Each driver socket is connects to a LWDAQ Multiplexer (A2046) at the end of a cable over 100 m long, and each multiplexer connects to up to 10 devices. That makes 1600 devices under the control of a single TCPIP socket. The average device contains a few sensors and light sources, so we have around 10,000 sensors-actuators under the control of the TCPIP socket. Image data download speed from the TCPIP-VME Interface is 340 kBytes/s, or four TCP255P eight-bit gray-scal images per second.

All the cables and sockets in LWDAQ systems have names.

Figure: LWDAQ cable and socket names. Cable-mounting plugs are named after the sockets with which they mate.

Any number of multiplexers and devices may be connected to a driver, but only fifteen devices may be connected to a single multiplexer, and we cannot connect a multiplexer to a multiplexer. The figure above defines the names of LWDAQ cables and sockets. We name plugs after the sockets with which they mate. When we insert a repeater in a root cable, the cables on either side of the repeater are both called root cables. The one between the repeater and the driver is the upstream root cable, and the other is the downstream root cable.

Cables

All LWDAQ cables are category-five (CAT-5) cables, all plugs are 8-way modular plugs, and all sockets are 8-way modular jacks (RJ-45). The LWDAQ guarantees against ground loops in the data acquisition cables, enclosures, and multiplexers. In order to make this guarantee, some sockets must be shielded, and others must be unshielded.

Rule: All LWDAQ driver and branch sockets must be unshielded, and all root and device sockets must be shielded.

The cables and plugs may be shielded or unshielded, as the system designer sees fit.

Rule: A LWDAQ receiver circuit must operate without error across 100 m of solid-core CAT-5 cable.

Rule: The receiver circuit on a LWDAQ multiplexer or LWDAQ device must operate without error across 10 m of stranded-core CAT-5 cable.

The reason we specify solid-core CAT-5 in the above rule is because the dispersion and resistance specifications for CAT-5 cable are stricter for solid-core cables than they are for stranded-core cables. Likewise, we have a relaxed specification for stranded-core cables. We find that the LWDAQ functions perfectly with a fully-loaded ten-slot multiplexer at the end of a 130-m shielded, solid-wire cable. The devices begin to fail at around 150-m, and this is because the Control Signals transmitted by the driver are not received properly by the multiplexers and devices. Stranded cables, on the other hand, must be far shorter. With a single device on the end of a stranded cable, we can operate reliably up to a length of 13 m. We begin to see occasional failures at 15 m, and frequent failures at 20 m.

Repeaters allow you to extend the range of the LWDAQ by restoring the outgoing logic signal from the driver at some point along the root cable before the signal has degraded significantly. If we want the length of cable from the driver to the multiplexer to be 200 m, the best place to put the repeater is 100 m from the driver.

Devices, repeaters, and multiplexers may be mounted in metal enclosures, with shielded connectors making direct contact with the enclosures through their shields. To avoid ground loops, there can be no low-impedance low-frequency connection between a cable shield and the zero-volt supply on either a multiplexer or a device.

Rule: A device, multiplexer, or repeater enclosure may connect to the local circuit's zero-volt potential through a resistance ≥ 1 kΩ and a capacitor ≤ 1 μF.

Recommendation: Connect the shield of a device or repeater socket to the device zero-volt potential with a 10-nF capacitor. Connect the shield of a multiplexer socket to the multiplexer zero-volt potential with a 1 μF capacitor.

Rule: The plug and socket pin-outs, and wire color-codes, for all LWDAQ connectors and cables, must conform to the Connector Pin-Out and Color Codes Table.

Rule: LWDAQ cables will carry only low-voltage differential signals (LVDS). The differential voltage must not exceed ±0.7 V, and the common-mode voltage must lie between 1 V and 2 V.

For more information about cables, see the Cable-Making Manual. There you will find a graph showing data acquisition quality with total cable length. When a driver clocks analog data out of a device, it synchronizes its digitization of the signal returned from the device by first measuring the round-trip propagation delay to the device. We call this propagation delay the loop time, and it increases with cable length by 5.0 ns/m.

Devices

The LWDAQ specification allows up to sixteen devices per multiplexer. When these devices draw current, their power supply voltages drop because of the resistance of the wires that connect the multiplexer to the driver. Although one device may tolerate a large drop in the supply voltages, another may not. The LWDAQ specifies maximum ranges for power supplies at the devices. We do not want inactive devices to waste the current-delivering capacity of the multiplexer.

Rule: All LWDAQ devices must provide a sleep state, in which consumption from each of the three supply voltages (+5V, +15V, −15V) is less than 5 mA.

Rule: All LWDAQ devices must power-up asleep.

Rule: All LWDAQ devices must be such that it is impossible to damage them by any sequence of transmissions from the driver.

We reset a LWDAQ by turning off the power and turning it on again. When the devices turn on, they will be in the sleep state, and they will not over-load the power supplies.

Multiplexers

To select a device, a LWDAQ driver disables all its driver sockets except the one connected to the device. We call this the active driver socket. The driver transmits an address word through the active socket. Assuming there is a multiplexer connected to this socket, the multiplexer will disable all its branch sockets except the one specified by the address word, which becomes the active branch socket. By means of the address transmission, the driver selects a unique device as the target of its next command word transmission. We call this unique device the target device.

Rule: On a LWDAQ multiplexer, address bit DA1 enables branch socket 1, DA2 enables socket 2, and so on up to bit DA15 enabling socket 15.

Example: The LWDAQ Multiplexer (A2046) provides ten branch sockets.

Repeaters

A repeater performs two functions. It receives and re-transmits all logic signals from the driver. This allows us to place multiplexers and devices farther from the driver. The repeater also allows us to turn off power to its multiplexer or device. When we transmit address word $0001, which is the same as selecting branch socket zero, the repeater recognises the zero socket selection and switches off its downstream power.

Example: The LWDAQ Repeater (A2058) is a single repeater. The Patch Panel (A2059) is a six-way repeater circuit.

Drivers

A large LWDAQ may consist of many drivers, each controlling many multiplexers, repeaters, and devices.

Example: The LWDAQ Driver with VME Interface (A2037A) is a one-slot wide, double-height VME card that uses the +5V and ±12V supplies on the VME backplane to power its multiplexers and devices.

Other drivers combine the functions of a driver with those of a server.

Example: The LWDAQ Driver with Ethernet Interface (A2037E) is a stand-alone black box that connects to the local ethernet and allows its LWDAQ to be controlled by LWDAQ Software over the internet.

Servers

A server makes the LWDAQ available over TCPIP. Clients connect to the server and control it by means of LWDAQ messages. We define the LWDAQ Message protocol in the TCPIP Communication section of the A2037E Manual.

When we have many drivers that do not include the server function, we use a dedicated server to provide TCPIP access to all the drivers.

Example: The VME-TCPIP Interface (A2064) is a one-slot wide, double-height VME card that allows all the A2037As in a VME crate to be controlled over the internet by LWDAQ Software.

Software

Our LWDAQ software is an accessory to the LWDAQ. You are welcome to use it, but some LWDAQ users prefer to write their own data acquisition software. Our software uese the LWDAQ messaging protocol to acquire data from a LWDAQ server over TCPIP. You can download our software here. We describe the installation and operation of the software in our LWDAQ User Manual.

Transmit Signals

The transmit signal is the signal carried from the driver by the T+ and T− wires. Unlike the receive signal, the transmit signal is always a logic signal, never an analog signal. The driver uses the transmit signal to switch on repeaters, select sockets on multiplexers, and control devices.

The transmit signal carries information with sequences of transmit bits. Each transmit bit begins with a 50-ns LO followed by a 50-ns HI. The following figure shows the three valid transmit bit patterns. Each pattern occupies 425 ns.

Rule: All rising edges on the transmit signal must be part of a valid transmit bit pattern, as defined in the Transmit Bit Patterns figure. The timing of the transmit bit pattern must be correct to within ±5 ns at every edge.

The driver combine transmit bits to perform the following control functions.

Address and command words are both control words. A control word transmission must follow either a stop bit transmission or a power-up reset. The control word begins with a type bit. If the type bit is a one bit, the control word is a command word. If the type bit is a zero bit, the control word is an address word. Sixteen data bits follow the type bit. Each data bit can be a one bit or a zero bit.

Repeaters and multiplexers respond only to address words. Devices respond only to command words. The circuit on a device or multiplexer that receives command or address words is a control receiver. It may be a command receiver, an address receiver, or both at the same time.

Every time a control receiver sees a LO-to-HI transition on T, it must determine which of the three bit patterns follows the transition. A stop bit always restores the receiver to its rest state. If the receiver is already in its rest state, the stop bit does nothing. When a receiver powers up, it must enter rest state.

When an command receiver sees the first bit of an address word, it enters its a passive state in which it until a stop bit arrives. An address receiver responds to a command word in the same way.

When a command receiver sees the first bit of a command word, it enters its active state. In its active state, the receiver records the subsequent command bits. We number the command bits from sixteen down to one. The driver transmits bit sixteen first and bit one last. The last command bit is followed by a stop bit. The receiver does not apply the command bits to the device until it receives the stop bit that marks the end of the command word. When it sees the stop bit, the command receiver applies all the command bits at the same time.

An address receiver responds to an address word in the same way. But we number the address bits from fifteen down to one. The driver transmits bit fifteen first and bit zero last.

Note: A receiver need not count data bits as they come in. The driver is responsible for making sure there are exactly sixteen bits.

Note: A receiver need not save all the data bits. It can discard the bits that it does not use.

We designed the transmit bit patterns so that they could be received without a clock oscillator. The receiver needs only two delayed versions of the transmit signal's rising edge to identify incoming bits. One delayed edge occurs after 125±25 ns and the other after 250±50 ns. We can generate these delayed edges with a monostable flip-flop, such as the 74VHC123. The 74VHC123's quiescent current is only 20 μA.

Example: The Proximity Mask Head (A2045) uses standard logic chips to implement a command receiver. The receiver consists of two 74VHC74s, 1 74VHC123, and 74LV594. These are U8-U11 in the schematic. The A2045 uses the LV594 to latch the bottom eight bits of the incoming command. It throws away the top eight bits. The VHC123 provides the two 125-ns delays necessary to sample T at the required moments after the rising edge that marks the commencement of bit activity. The two VHC74s distinguish between address and command transmissions, and act upon the stop bit that terminates a command reception. The logic that performs this termination is a little hard to understand. Two flip-flops reset one another and latch the shift register at the same time. The quiescent current consumption of the A2045 is less than 1 mA. The command receiver itself consumes less than 100 μA.

Example: The Polar BCAM Head (A2051) adds another 74LV594 shift register to the command receiver used in the A2045, and so retains all sixteen command bits.

Example: The Inclinometer Head (A2065) uses programmable logic with a ring oscillator to receive commands. When the logic sees a rising edge on T, the ring oscillator starts up and provides timing for reception. The command receiver consumes only 11 mA, but this still exceeds the LWDAQ limit of 5 mA when asleep. You will find the logic program here.

A command or address receiver in its rest state ignores stop bits. Consequently, the driver is free to transmit stop bits as often as it likes do a receiver in its rest state. Some devices allow the driver to their synchronize data return with stop bits. We call this use of stop bits data synchronization. For more on data synchronization, see Receive Signals.

Recommendation: Use the SN65LVDM180 transceiver, manufactured by Texas Instruments, to send and receive low-voltage differential logic signals on T+/T− and R+/R−. These transceiver operate from a 3.3-V supply. With the driver portion disabled, their typical current consumption is only 1.7 mA. The driver and receiver have separate enable lines. When the driver is disabled, it enters a high-impedance state that allows analog circuits in a device to drive the R+/R− lines.

Recommendation: Do not use the SN65LVDS180 transceiver in devices that use R+/R− for analog data return. Because of an error on the part of Texas Instruments, the SN65LVDS180 does not enter its high-impedance state properly.

Note: The SN65LVDS180 transceiver works fine in devices that do not use R+/R− for analog data return, and has the advantage of lower current consumption.

Example: The Proximity Mask Head (A2045) uses the SN65LVDS180 transceiver. Even thought this chip does not enter its high-impedance state properly, we do not mind because we transmit only LVDS logic back from the A2045. We could instead use the LVDM chip, but its quiescent current is slightly higher.

Rule: The circuits driving T+ and T− must assert a defined logic level on the two lines at all times.

The above rule guarantees that the circuit receiving T+ and T− does not receive spurious commands. If the circuit driving T+ and T− allows the T+ and T− lines to float, spurious commands can be received at the other end of a long cable, even if the receiving circuit has its own pull-up resistors. The SN65LVDM180D uses 300 kΩ pull-up resistors to define its state when its input floats, but these are not adequate at the end of an 80-m cable.

Recommendation: Connect the source of T+ to +3.3V with a 10 kΩ resistor. Connect the source of T− to 0V with a 10 kΩ resistor.

Pull-up and pull-down resistors at the source of the transmit signal guarantee that the transmit logic level will remain HI during and after the LVDS driver enters its high-impedance state.

Receive Signals

The receive signal is the signal carried from a device by the R+ and R− wires. Unlike the transmit signal, the receive signal can be either analog or digital. Either way, it is always a low-voltage differential signal.

Rule: The driver's differential analog input range for DC-coupled signals must include -0.5 V to +0.5 V.

The driver may clamp the returned data stream, and measure the returned differential voltage with respect to the clamped value.

Rule: The LWDAQ driver differential analog input range for synchronously-clamped signals must include the range 0 V to 1 V.

The driver puts a device into its loop-back state by transmitting a command word with bit seven set to 1. When in the loop-back state a device drives the transmit signal onto the receive signal, so that whatever logic level arrives from the driver will be returned to the driver.

Rule: Every LWDAQ device must implement the loop-back state.

The return of T+ and T− allows the driver to check the functionality of the device, and also to measure the propagation time of a signal traveling to the device and back again. We call this round-trip propagation time the loop time. The driver uses its knowledge of the loop time to synchronize its reception of a stream of data arriving at one sample per 500 ns, even when the clock signal it transmits takes 500 ns to reach the device along a 100-m cable, and the data takes another 500 ns to come back again.

A device can switch from digital to analog receive signals by disabling its LVDS driver.

Example: The Polar BCAM Head (A2051) uses the SN65LVDM180D to drive R+/R− with a logic value (U1 in the schematic). But R+/R− are also connected through 100-Ω resistors to two op-amps (U19 in the schematic). With U1's driver enabled, the op-amps cannot alter the logic level on R+/R−. But once U1 is disabled, the op-amps drive an analog voltage onto the R+/R−. Thus the act of disabling U1's driver switches the circuit between logic and analog return. No analog switch is required.

By transmitting stop bits, a driver can synchronize the return of analog data from a device.

Example: A driver transmits 425-ns stop bits continuously. The stop bits form a 2.35-MHz clock that a device can use for 2.35-MHz data synchronization. For another device, the driver inserts a 1075-ns LO period between the stop bits, and so generates a 1-MHz clock. Each clock period consists of a 375-ns HI followed by a 1125-ns LO.

Example: The Inplane Sensor Head (A2036) clocks pixels out of its TC255P image sensor using a stop bits with 75 ns of LO inserted between them. The result is a 2-MHz pixel clock whose period consists of a 125-ns LO followed by a 375-ns HI. The 125-ns LO clocks out the pixel intensity, and the 375-ns HI clocks out the background intensity.

Devices can transmit data bytes to the driver using the receive signal. The driver synchronizes this transfer with stop bits. First the driver sets the device into a state where it is ready to transmit bytes. In this state, the device drive enables its logic-level LVDS driver and asserts a logic HI on the receive signal. After that, each stop bit the device receives from the driver will provoke a byte transfer on the receive signal.

The byte transfer consists of ten 50-ns bits. The first bit is a zero, and is called the start bit. The next eight bits are the data bits. The most significant data bit is first, and the least significant is last. After the last data bit is a one, which is called the stop bit.

The entire byte transfer takes 500 ns, suggesting a maximum transfer rate of 2 MBytes/s. In practice, however, the device needs 425 ns to receive the stop bit before it begins its byte transfer. Loop time also adds to the byte transfer period.

Example: Block byte transfers from the ADC Tester (A2100 to the LWDAQ Driver (A2037E) take place at 1.1 MByte/s.

Rule: The timing of every feature of a byte transfer must be accurate to ±10 ns.

Because the ten-bit transfer takes 500 ns, and the final bit must be accurate to 10 ns, we see that the clock used on the device to generate the byte transfer timing must be accurate to 2%. This precludes the use of self-calibrated ring oscillators such as those used in the Inclinometer Head (A2065). A low-power device that performs byte transfer must turn on a precision oscillator when the driver wakes it up, and turn off this oscillator when the driver sends it to sleep.

Power Supplies

The voltage drop in the resistance of a CAT-5 cable limits the number of devices that may share the same root cable. The following table gives the maximum current consumption of a device when asleep and awake, and the range of power supply voltages that must be acceptable to a LWDAQ receiver.

With 20-mA current consumption, the drop across a low-power, low-drop-out regulator (such as the MAX6329 from Maxim) is less than fifty millivolts. But regulators that rely upon internal band-gap references are vulnerable to ionizing radiation, and the feedback loop that controls their output can latch up when hit by a high-energy proton. For radiation resistance, therefore, the LWDAQ specification allows us to use an emitter-follower logic supply. The incoming +5 V supply connects to the collector of an NPN transistor, and the base receives current through a resistor from the +15 V supply (for an example schematic, see the A2045). The saturation voltage of an NPN transistor (such as the XN4501 from Panasonic) is 100 mV. The SN65LVDS-series chips will operate with 3-V supplies, as will low-voltage logic chips. Thus the minimum logic supply voltage is +3.1 V.

Rule: Every LWDAQ device must obey the limits given in the Device Current Consumption Limits and Supply Voltage Ranges Table.

Rule: Every LWDAQ multiplexer and repeater must obey the limits given in the Multiplexer and Repeater Power Supply Constraints Table.

The CAT-5 specification limits the resistance of solid CAT-5 conductors to 10 Ω per 100 m. This resistance, combined with the maximum device current consumption and power voltage ranges, limits the number of devices that may share a root cable. With the limits given in Tables 2 and 3, sixteen sleeping devices and two waking devices can share the same ninety-meter root cable and remain operational.

Rule: It must be impossible to damage a LWDAQ component by exceeding the current consumption limits at another LWDAQ component.

Rule: All devices, repeaters and multiplexers must tolerate the power supply voltages given in the Absolute Maximum Ratings table.

We have not yet specified the anti-static discharge tolerance of the device and multiplexer inputs. We make the following recommendation.

Recommendation: Clamp the T+, T−, R+, and R− signals to +3.3V and 0V with silicon diodes on every root, branch, and device socket.

Components suitable for the clamping are diode arrays such as the MA125 from Panasonic. Clamping makes the circuits less vulnerable to static electricity and to hot-plugging power surges.

Subtle problems arise in large LWDAQ systems when we make a device's receiver logic power supply dependent upon anything other than the incoming +5V power from the device socket.

Example: The Proximity Mask Head (A2045) uses +15V to supply base current to a radiation-tolerant 3.3V regulator, as shown here. The 3.3V logic supply depends upon a 300-μA flow of current into the regulator from +15V. Suppose the mask is faulty, so that turning it on shorts the +15V supply. Now the logic supply fails also. As soon as the mask turns off again, the +15V supply rises and powers the logic once more. Unless there is a perfect power-up reset, it's possible that the mask will turn on again, beginning an endless cycle. It so happens that the A2045 doe snot have a perfect power-up reset. Its power-up reset is produced by a capacitor and resistor. The problem with power-up reset circuits is that they tend to be vulnerable to radiation, just like low drop-out regulators. For more on large-system problems see Large Systems.

Recommendation: Make the power supply used by command and address receivers dependent only upon the incoming +5V supply.

Device Types

Whenever we execute a device-dependent job in a driver, we must let the driver know the type of device the job is operating upon. We give the driver a device type number. Associated with each device type is command bit allocation.

Rule: All LWDAQ drivers must respect the device types given in the Reserved Device Types.

Command Bits

A device's command bit allocation is the use it makes of each bit in its most recent command word. We name the device command bits DC1 to DC16. A manual describing a LWDAQ device will tell you its command bit allocation, and a multiplexer or repeater manual will tell you its address bit allocation.

Rule: Command bit DC8 is the wake bit (WAKE or !SLEEP on schematics). The device wakes up when the wake bit is one, and goes to sleep when the wake bit is zero.

Rule: Command bit DC7 is the loop-back bit (LB on the schematic). When the loop-back bit is one, the device drives onto R+/R− the logic level it receives on lines T+/T−.

The LB and WAKE bits must be respected by all devices. If WAKE is 1, the device must enter its lowest-power state. A device can ignore WAKE if it enters its lowest-power state when it receives a command of all zeros. If LB is 1, the device must drive the R+/R− lines with the same logic value it receives on T+/T−. But a device can ignore LB if it always drives T+/T− onto R+/R−

Rule: All command and address bits on all devices and multiplexers must reset to zero on power-up.

Some command bits have reserved functions depending upon the device type that receives them. The following table lists reserved command bits for various device types.

Rule: All LWDAQ drivers must respect the command bit allocations given in the Reserved Command Bits Table.

Address Bits

A multiplexer or repeater's address bit allocation is the use it makes of its most recent address word. We name the address bits DA0 to DA15.

Address words instruct multiplexers in the same way that command words instruct devices. Each of the sixteen address bits selects one of sixteen hypothetical branch sockets on a multiplexer. If multiple bits are set, then multiple sockets will be enabled. In this way, it is possible to send command words to any subset of devices attached to a multiplexer simultaneously, or to only one device. Repeaters make use of the DA0 bit to turn off downstream power.

Rule: Multiplexers use bits DA1 to DA15 to select branch sockets 1 to 15. Repeaters use bit DA0 to turn off power to multiplexers and devices.

Element Numbers

When a target device has multiple sensors and transmitters, we select individual sensors and transmitters with a device element number. Just as a driver must interpret a device-dependent jobs with the help of a device type number, it must interpret the element number with the help of a device type number also.

Rule: All LWDAQ drivers must respect the element numbers given in the Element Numbers Table.

Example: A flash job with device type 2 and element number 1 causes the driver to set bit 10 in the command word to turn on source number 1. A flash job with device type 1 and element number 1 causes the driver to set bit 1 to turn on source number 1.

Driver Jobs

We control LWDAQ drivers by instructing them to perform LWDAQ jobs.

Example: The LWDAQ Driver (A2037) with firmware version 3+ recognizes sixteen LWDAQ jobs. The Device Driver (A2031) with firmware version 3+ recognizes nine LWDAQ jobs.

The following table is a list of job numbers and their associated names. We use these names in our LWDAQ driver firmware and software.

Rule: All LWDAQ drivers must respect the job numbers and functions given in the Driver Jobs Table.

A device-dependent job is any job whose implementation depends upon the target device. Examples of device-independent jobs are sleep, wake, and loop. Examples of device-dependent jobs are flash and read. When a driver executes a flash job, it must turn on and then turn off a transmitter in the target device. The command bits the driver must set to turn on and off the first transmitter on a device vary with the device type.

Example: The first transmitter on a BCAM Head (A2051) turns on with command bit ten. The first transmitter on an Inplane Mask Head (A2052) turns on with command bit one.

TCPIP Messages

All LWDAQ drivers provide a TCPIP interface at some point in the communication between the devices and the data acquisition computer. The LWDAQ driver acts as a TCPIP server, listening upon a particular port for connections from TCPIP clients. We call it the LWDAQ server. The LWDAQ server looks for messages in the following format from TCPIP clients that we call LWDAQ clients.

The LWDAQ Message Protocol runs on top of TCPIP, and defines the way in which the LWDAQ client will communicate over TCPIP with the LWDAQ server. The first byte of LWDAQ message is the start byte, which has value $A5 (that's hexadecimal A5). The final byte of the message is the end byte, which has value $5A. Following the start byte is the message identifier, a 32-bit integer with the most significant byte first. Following the message identifier is the content length, another 32-bit integer that gives the number of bytes in the content of the message. If we send a message that begins with any byte other than the $A5 start byte, the driver will close the TCPIP socket.

Example: The A2037E is a LWDAQ server. Its TCPIP stack and Ethernet interface run on an RCM2200 embedded processor. We implement the LWDAQ server functions with a C program. Today's version of the program (July 2008) is P2037E12.c.

Example: The LWDAQ Software runs on a Windows, Linux, or MacOS computer, and implements a LWDAQ client with graphical user interface and automatic data acquisition and recording to disk. The software uses Tcl to open TCPIP sockets to a LWDAQ server. It uses Tcl to send and receive messages. The script called Driver.tcl contains the Tcl code that implements the LWDAQ client. This script is one of the files included in our software, which you can download here.

The message identifier tells the LWDAQ client or server how to interpret the remainder of the message, and what action to take in response to the message. Here are the message identifiers currently defined by the message protocol.

We describe each of the message identifiers in the sections below. Some messages request information from or exercise control over the TCPIP portion of the LWDAQ server. We call this portion of the server the LWDAQ relay. The main function of the LWDAQ relay is to convey instructions to the real-time state-machine-driven logic circuits that control LWDAQ devices and multiplexers. We call these logic circuits LWDAQ controllers.

Example: The A2037E is a LWDAQ server containing one LWDAQ relay and one LWDAQ controller. The A2064 is a LWDAQ server that contains only a LWDAQ relay. The relay conveys messages to one or more VME-resident LWDAQ controllers, such as the A2037A.

Most messages instruct the LWDAQ relay to write to or read from locations in control address space. This address space uses 32-bit addresses and provides byte-wise access to locations. All interactions with the LWDAQ controller take the form or reading from or writing to locations in the control address space.

version_read

When a server receives a version_read message, it transmits a data_return message. The content of the data_return message is four bytes long. These four bytes contain a 32-bit integer giving the server software version.

byte_write

When it receives a byte_write message, the relay writes to a controller location. The first four bytes of the message content contain the controller address. The fifth byte gives the value to write. The server transmits no message in response to a byte_write.

byte_read

When it receives a byte_read message, the relay reads a single byte from a controller location. The first four bytes of the message content contains the contoller address. The server returns the byte it reads in a data_return message. The content of the data_return message is one byte long. This byte is the byte the relay read from the controller.

stream_read

When it receives a stream_read message, the relay reads repeatedly from a single controller location and returns all the bytes it reads from in a data_return message. The first four bytes of the stream_read message content contains the controller address. The next four bytes give the number of times the relay should read from the controller location.

Example: We use the stream_read message to read images out of A2037E controller RAM. Address $3F (decimal 63) is the RAM portal address. The value we read from the RAM portal is the value pointed to by the controller's internal data address. Each read from the RAM portal increments the data address, so the stream_read from the RAM portal ends up being a block move out of the controller memory. The stream_read is more efficient because it requires no change in the address presented by the relay to the controller. To read a block of RAM, we set the data address to point to the first byte of the block, and send the stream_read message with the block length.

byte_poll

When it receives a byte_poll message, the relay goes into a loop waiting for a controller location to assume a particular value. The first four bytes of the message content contain the controller address. The fifth byte contains the value the relay should wait for. The relay will drop out of this loop when the LWDAQ client closes its socket to the LWDAQ server.

With the byte_poll job, you can send a list of instructions to the server in one Ethernet packet, and have them executed in the most efficient way by the relay. The last instruction can be a stream_read, which causes the A2037E to send data back to the data acquisition computer.

Example: To obtain an image from a camera, the client sends the sequence of instructions required to obtain and return the image and waits for the image data to arrive. Instead of polling the controller's BUSY bit over TCPIP, we use byte_poll to instruct the relay to poll the BUSY bit for us.

login

Depending upon how a server is configured, it may require a login message with the correcct password before it will respond to any other message. One such login is required for each TCPIP connection. If the server security level is 1, the client must send a login message with a valid password in order to execute a config_write. If the security level is 2, the client must send a login message to execute any command. The contents of a login message is the ascii-encoded password.

config_read

When the server receives a config_read, it sends back a RAM-resident copy of the its EEPROM-based configuration file. The RAM-resident copy is made after any hardware reset, but does not get modified after it is made, even if the EEPROM copy of the configuration file gets modified by a config_write instruction. The contents of a config_red job are empty. The message returned contains the ascii-encoded characters of the configuration file.

config_write

When the server receives a config_write, it writes the contents of the message to its EEPROM-based configuration file. The contents of the file must be compatible with the server's TCPIP Interface software, or else the server will ignore its contents. The file does not take effect until after a hardware reset. You cannot re-configure a server remotely with config_write. You must be able to press its hardware reset button or turn off its power supply.

stream_delete

When the server receives a stream_delete, it writes a single byte value repeatedly to a single-byte control location. The first four bytes of the message content give the location to which the relay will write the byte value.

Example: In the A2037E, this address will invariably be $3F (decimal 63), the RAM Portal, because consecutive writes to this location set consecutive locations in the controller's RAM space.

The next four bytes of the stream_read message content give the number of times the relay will write the value to the control location. The final byte of the nine-byte message content gives the value the relay will write.

Design

We designed the LWDAQ to meet the demands of a particular data acquisition problem, that of the end-cap alignment system in the ATLAS muon detector, as we describe in The Optical Alignment System of the ATLAS Muon Spectrometer Endcaps. The ATLAS detector is a cylinder forty meters long and twenty meters wide. The alignment system's cameras and light sources are LWDAQ devices.

One of the first problems we faced in the ATLAS detector was how to provide electrical power to our devices. We could not use batteries, because the power consumption of the light sources cannot be reduced below certain limits, and no battery small enough to fit in the light source enclosures could supply the required power for the ten-year operating life of the experiment. If we were to deliver high-voltage power and convert it to low-voltage power, we would have to do so in the strong magnetic field of the detector. This magnetic field would saturate ferrite inductor cores. High-frequency converters with air-core inductors are vulnerable to ionizing radiation. We expect some devices to receive as much as 7 krad of ionizing radiation during the ten-year operating life of ATLAS, and we would like them to be able to endure 20 krad to give us a margin of safety.

We considered delivering power to our devices through a separate, low-resistance cable, while communicating through a network cable. But if we deliver power with one cable, and signals with another, we invite ground loops. We could try coupling the signals into the devices optically, but LEDs (light-emitting diodes) in opto-couplers are vulnerable to neutron radiation. We expect some of our alignment devices to receive as much as 10¹² 1-MeV equivalent n/cm² (1 Tn) during the ten-year operating life of ATLAS, and we would like them to be able to endure 10 Tn to give us a margin of safety. The most neutron-resistant LED we have found can lose up to 90% of its transmitting power after 10 Tn. Aside from the ground-loop problems introduced by separate cables, there is the problem of placing power supplies in the experiment hall to deliver power over these separate cables. Such power supplies would have to operate in a magnetic field, or else the power cables would be tens of meters long.

We decided to deliver power and signals to each alignment device through a single cable. There are no LWDAQ power supplies in the ATLAS detector hall.

But we do not have space in ATLAS to bring a cable into the detector for every one of our thousands of cameras and light sources. We must provide some kind of multiplexer. With multiplexing, our power supply problems become more severe. Some cables from the service hall to the devices in the detector are over 100 m long. Despite their length, such cables must be able to provide power to all devices connected to a multiplexer.

The ATLAS end-cap alignment system, is one in which only one or two devices out of the thousands in contains need to operate at one time. We reduce the power consumption of the multiplexers by putting to sleep any devices that are not active. The LWDAQ allows us to put a device to sleep either with a single command from its driver, or by cutting off power to the device and then turning the power on again. With the sleeping power consumption reduced to tens of milliwatts, we can supply power to two active devices on the same multiplexer through a single 130-m solid-wire CAT-5 network cable. We can connect multiplexers to their devices with CAT-5 cables as well. We tend to use stranded-core cables for the shorter connections between multiplexers and devices. Stranded wires are more flexible. Solide wires are stiff, but they are faster. The CAT-5 specification for solid-wire cables is much more strict than for stranded-wire cables. It is more difficult to control the dielectric properties experienced by a signal traveling down a stranded wire. In ATLAS, all our root and branch cables are shielded and halogen-free.

In each cable, the same allocation of conductors applies: one twisted pair transmits commands from the driver, another twisted pair returns data from the devices, and the remaining four wires, which may be twisted or not, carry ±15V, +5V, and 0V power.

Another problem we faced in ATLAS, as we have mentioned already, is the pervasive ionizing and neutron radiation to which our circuits will be subjected for the ten-year operating lifetime of the detector. The highest ionizing dose is approximately 7 krad, and the highest neutron dose is roughly 1 Tn (10¹² 1-MeV equivalent n/cm²). Our most-severely irradiated devices will be inaccessible for years at a time. We would like them to be resistant to radiation, rugged, and long-lived, which suggests that they should be simple. On the other hand, we would like them to be versatile, so that a single device, with a single cable, can perform all the alignment functions needed in its immediate neighborhood.

Neutron radiation damages the image sensors and the infra-red LEDs we use in our alignment devices. Our TC255P image sensor is a CCD (charge-coupled device) from Texas Instruments. It suffers an increase in dark current in neutron radiation. After absorbing 10 Tn, its pixels fill up with dark current in 50 ms. If we are to capture images with these sensors after a does of 10 Tn, we must capture and read them out in less than 50 ms. There are eighty thousand pixels per image, and we must allow at least 10 ms for exposure to light, so we must retrieve the pixels at a rate no slower than two million per second.

Our HSDL4400 LED is an infra-red emitter from Hewlett-Packard. The HSDL4400 is more resistant to neutron radiation than any other we tested. Nevertheless, it can lose up to 90% of its optical output power after 10 Tn. We can use these diodes up to a dose of 10 Tn only if the time for which an undamaged diode must be flashed to obtain an adequate image is no more than 1 ms. In order to provide 1-ms flashes, our data acquisition system must be able to turn on and off light sources, and switch between one device and another, in a fraction of a millisecond.

We decided upon a number of policies designed to keep alignment devices simple, but at the same time versatile and fast. All timing signals required by a device are provided by its driver, with the exception of the short pulses required to decode the serial transmissions from the driver. Devices do not digitize analog signals, but transmit them directly to the driver. To preserve the integrity of these analog voltages, they propagate as low-voltage differential signals (LVDS) and all ATLAS-resident LWDAQ cables are shielded. Likewise, the driver transmits its commands as LVDS logic levels.

And so we arrived at the Long-Wire Data Acquisition System, with its generic drivers, multiplexers, repeaters, and cables. The devices can contain as many sources and sensors as we need, provided their waking power consumption remains below the LWDAQ specified limits (see below).

Example: The BCAM Head (A2051), of which there will be several hundred in ATLAS, provides four laser diode light sources and two image sensors. We connect the BCAM Head to the LWDAQ with a single CAT-5 cable. The Bar Head (A2044), of which there will be two hundred in ATLAS, provides four platinum resistance-temperature devices (RTD), two image sensors, and two infra-red light-emitting diode arrays. Assuming a perfect RTD, the Bar Head provides temperature measurement accuracy of 40 mK, and resolution of 20 mK. No digitization takes place in the Bar Head. Instead, the device returns analog voltages, which the driver digitizes for temperature measurements with its sixteen-bit ADC. Both the BCAM Head and the Bar Head have a typical sleeping current consumption of 1.8 mA at +5 V, 400 μA at +15 V, and 100 μA at −15 V. Their sleeping power consumption is therefore 17 mW.

The ATLAS LWDAQ Driver with VME Interface (A2037) resides in a VME crate, and provides eight CAT-5 sockets. We can connect a device or a multiplexer to each one of these sockets, although in ATLAS, only multiplexers will be connected directly to the drivers. The ATLAS detector requires nearly eight hundred LWDAQ Ten-Way Multiplexers (A2046) with eight hundred CAT-5 cables running out of the detector and into the service hall, where one hundred VME-resident LWDAQ drivers will receive them.

We might have included a second layer of multiplexing in the ATLAS detector, to reduce the number of cables that run into the service hall. A second-layer multiplexer might provide ten sockets for ten first-layer multiplexers, and thus allow one hundred devices to be connected to a driver with only one cable. But this cable would have to be larger than our existing CAT-5 cables, and would require a larger connector. We would have to design the second-layer multiplexer itself, and test it. In the end, we found that the cost of designing, implementing, building, and testing a second layer of multiplexing was greater than the cost of building, testing, and installing eight hundred cables.

Figure: Block Diagram of ATLAS DAQ.

We are wary of connecting a large number of devices to a single detector-resident circuit, or of making a large number of devices dependent upon any single cable. With a second layer of multiplexing, the failure of a single device, or of a single device cable, could, by shorting the power supplies, cripple a second-layer multiplexer and disable ninety-nine other devices at the same time. A fault on a single device could damage, by its affect on the power supplies, every other device connected to the second-layer multiplexer. To avoid such disasters, a second-layer multiplexer would have to be sophisticated in its distribution and monitoring of the LWDAQ power supplies. Such a circuit would be difficult to design and complicated to test, and its failure at any time during the running of the ATLAS detector would cut off a hundred fully-functional alignment devices from their LWDAQ driver. The LWDAQ, therefore, provides only one layer of multiplexing.

When the LWDAQ was a few years old, we began to suspect that cables in the ATLAS detector hall would be longer than the 100 m for which we initially designed the system. We had to increase the maximum cable length to over 130 m. To this end, we designed the LWDAQ Repeater (A2058). The repeater restores outgoing logic signals. It allows us to extend our operating range to 200 m. The repeater also allows us to shut off power to the downstream circuits. Turning off power to individual multiplexers allows us to increase the proportion of time for which each device is without power. Devices are more resistant to ionizing radiation and single-event upsets when they are without power. The repeaters also allos us to isolate in software any faulty cables, multiplexers, and devices that would otherwise bring down the driver power supplies.

We discuss the problems we encountered when installing the ten-thousand device LWDAQ of the ATLAS end-cap muon spectrometer in the following section.

Large Systems

Two weaknesses in our LWDAQ devices conspired together to cause problems in our large LWDAQ systems. The first weakness was the dependence on +15V of the command receivers in radiation-tolerant devices. When the +15V power turns off, the NPN transistor in our radiation-tolerant 3.3V regulators are deprived of base current and turn off. The second weakness was our use of a capacitor-resistor network to provide our power-up reset signal.