Timing and Control Module Specification

Contents

Description

The Timing and Control Module (TCM) is a proposed component of the Large Synoptic Survey Telescope (LSST) camera. Brandeis University's High Energy Physics lab has committed to designing and building the TCM for the LSST camera, a few spares TCMs to keep the camera going for ten years, plus a dozen or more prototypes for use in laboratories during camera development.

Figure: The Large Synoptic Survey Telescope.

The camera is the black cylinder just to the left of the gymbol axis in the figure above.

Figure: The LSST Camera.

The Cryostat Outer Cylinder (green) encloses the camera's image sensor plane and the sensor readout electronics. These readout electronics are divided into two parts, called the Front-End Electronics (FEE) and the Back-End Electronics (BEE). The FEEs mount directly beneath the image sensors. They driver the image sensor clock lines, read out the pixel intensities, and digitize these intensities. Because the FEEs are directly beneath the image sensors, they reside in the cold region of the cryostat, and operate at −100°C. The BEEs, on the other hand, lie beyond an insulating plate in the warm region of the cryostat, and operate at −40°C. They deliver timing signals to the FEEs, and in return receive digitized pixels. The BEEs plug into Raft Controller Modules (RCMs). The RCMs transmit digitized pixels over an optical fiber, out of the camera to the Raft Network Adaptor (RNA).

The proposed TCM resides in the camera outside the cryostat. It connects to all twenty-five RCMs within the cryostat with wires that penetrate the cryostat wall. The TCM is master of bi-directional communication with the RCMs. The TCM has the following operational functions, which support the operation of the camera.

1. Generate and distribute a synchronised clock signal to the BEEs so that pixel readout from all image sensors can be synchronous.
2. Read out and set configuration registers on the BEEs.
3. Provide a TCPIP interface with the Camera Control System (CCS).

The TCM also has the following diagnostic functions, which support development in the laboratory and camera diagnostics.

1. Slow read-out of image pixels from the RCMs.

The Brandeis University High Energy Physics lab is responsible for the design and construction the TCM for the LSST camera. We bring to LSST our experience with TCP/IP data acquisition in ATLAS. The TCM is similar to the LWDAQ Driver with Ethernet Interface (A2037E) we developed for our work on the ATLAS End-Cap Muon Spectrometer. The TCM will use newer parts, and its TCP/IP interface will be modified to conform with a standard laid down by the Camera Control System.

[05-SEP-08] Each TCM allows the Camera Control System to communicate with a number of RCMs. An eight-slot TCM allows communication with up to 8 RCMs. This year we are building ten 8-socket TCM prototypes. We have finished the circuit design and submitted the printed circuit board for fabrication. We will build and test the first prototype in September. We will work on the TCP/IP messaging software in October and November. In December we will complete the set of ten prototypes and distribute them to various LSST collaborators in Chile, Boston, Arizona, and California. These ten prototypes will support LSST camera development in 2009. In 2010 we will design and build ten second-generation 8-socket prototypes and distribute these to LSST collaborators. In 2011 we will design and build the prototype 25-socket TCM. We will build auxilliary circuits that will allow us to test the 25-socket TCM. From 2012 to 2013 we will build five 25-slot TCMs for use in LSST. We will ship three to Chile, one to SLAC, and keep one at Brandeis. Of the three in Chile, one will be in active use and two will be spares.

Specification

Here is a summary of specifications for the TCM. We intend to modify these numbers in response to comments from collaborators.

• Size: 9" x 2" x 6 "
• Enclosure: Anodized Aluminum
• Heat Dissipation: 10 W
• TCPIP Connection: RJ-45, PC104 Linux Module with Digital IO ports
• TCPIP Data Rate: >500 kBytes/s
• RCM Connection: RJ-45, CAT-5 Unshielded.
• RCM Communication: custom bi-directional serial interface.
• Pixel Clock: 50 MHz to each RCM, ±5 ns total jitter and offset.
• Timestamp: counts cycles of a 32.768 kHz clock.

Functions

The Camera Control System (CCS) will instruct the TCM to perform the following functions.

1. Synchronize TCM real-time clock with CCS real-time clock.
2. Write to a single configuration byte on a single RCM.
3. Write simultaneously to a shared configuration byte on all RCMs.
4. Read a single status byte on a single RCM.
5. Program firmware on RCMs.
6. Upload image sensor control programs to RCMs.
7. Assemble RCM status reports with timestamps and return to CSS.
8. Reboot RCMs.

We hope our readers will feel free to extend and correct the above list.

Architecture

Figure: TCM Block Diagram. Marked in red are choices for the prototype, TCM1, for use in laboratories in summer 2008.

TCPIP Interface

One way to control the nodes in the Camera Control System (CCS) is with a TCPIP interface, and one possible implementation of this TCPIP interface is a PC104-format, PC-compatible computer running Linux. The Linux computer would publish its data and functions on the local network and subsribe to the sources of instructions and data that it needs. It could use DDS or some other publish-subscribe protocol.

We bought and tested just such a Linux machine, as we describe in our report Embedded Linux System. Our conclusions were not favorable, as we described at the Camer Control Workshop in Tuscon in January, 2008, in our talk Camera Control Nodes.

Figure: A Camera Control Node.

Instead of placing a Linux machine next to the control input-output boards, we could instead put the Linux machine outside the camera, and connect it to the control input-output boards with a long cable. Each control input-output boards communicates serially with its Linux computer. The Linux computers, being together outside the camera, can be combined into one Linux computer. The serial communication can join at a multiplexer and proceed out of the camera on one shared serial communication cable.

Figure: TCPIP Between Linux Machine and IO Boards

Now suppose we use TCPIP to transfer simple messages to and from the control input-output boards. Each board needs a translator that runs a TCPIP stack and passes incoming TCPIP messages to the control input-output board as register reads and writes. We use just such a TCPIP receiver in our LWDAQ Driver (A2037E) and our TCPIP-VME Interface (A2064. The TCPIP translator is the RCM2200 and the RCM3200 respectively, both manufactured by Rabbit Semiconductor, and both costing below $100. Both these translators are eight-bit embedded microprocessor boards with a 10-Base-T Ethernet socket and a bunch of byte-wide IO ports.

The TCM will use an RCM4200 to provide its TCPIP interface on the TCM, and to implement the SIAP messaging protocol, as described in the following section.

CCS Communication

The TCM (Timing and Control Module) acts as an intermediary between the Camera Control System (CCS) Master and the camera's twenty-five Raft Controller Modules (RCMs). Here we consider communication between the CCS Master and the TCM. This communication will follow SCL (SOAR Communication Library) protocol developed by German Schumacher for the SOAR Telescope.

The SCL protocol is a client-server protocol that provides security based upon the IP address of the client, and parsing of the two-way data flow with four-byte length fields passed at the beginning of any SCL message. The SCL implemented on the TCM will run on top of TCPIP.

Note: German is planning another version of SCL that runs on top of DDS (data distribution service). We have no plans to implement DDS on the TCM, but there is no reason why the TCM cannot implement DDS if there is some advantage in doing so.

Here are the steps required by a two-way SCL communication. At any point during this communication, either the client or the server can terminate the communication asynchronously by aborting the socket.

1. Client opens a TCPIP socket to the server using the server's IP (internet protocol) address and the SCL port number.
2. Server accepts connection.
3. Server checks IP address of client against its list of permitted clients. If the client's IP address is on the list, the server sends back an SCL message containing the ASCII string "DONE". The message contains eight bytes in total: four bytes to give the length of the content and four for the content itself. If the client's IP address is not on the list, the server sends back an error message and aborts the socket.
4. Client checks that socket opens within a timeout. Otherwise, aborts the socket with an error.
5. Once socket is open, client tries to read the "DONE" message within a timeout. Otherwise, aborts the socket with an error.
6. Client sends an SCL message with an instruction for the server as its content.
7. Server receives the SCL message from the client. Server interprets and executes the instruction. The interpretation and execution of the instruction is nothing to do with SCL.
8. In this example, the instruction requires an answer. The server sends the client an SCL message with the answer as its content.
9. Client receives SCL message containing the answer from the server.
10. If the client needs to send more instructions, go back to Step 6. Otherwise, close the socket.
11. Server sees the client has closed the socket. Server ends the SCL session.

The length field of an SCL message is four bytes long. The most significant byte comes first. The length field gives the number of bytes in the SCL message that follow the length field. Thus the length field for content "DONE" is 4, because "DONE" has 4 bytes, even though the entire message contains 8 bytes: 4 for the length field and 4 for the content field.

We propose the following messaging protocol to run within SCL and carry instructions and answers between the CCS and the TCM. We call our protocol the Simple Instruction-Answer Protocol (SIAP). Each SIAP message begins with the SCL length field of four bytes, and is itself an SCL message. Following the length field is a four-byte message identifier field. The message identifier tells us the function of the message and the format of its contents. All SIAP fields are big-endian. The most significant byte comes first.

Note: We base the SIAP message identifiers upon the ones we use in our LWDAQ. This set of identifiers has grown over the years in response to our needs. We describe this protocol in the TCPIP Communication Protocol section of the A2037 User Manual. We increase the address fields of the read and write instructions from four bytes to eight bytes, in order to support direct access to pixels in astronomy image sensors with more than 4 GPixels.

The TCM, and any other SIAP server, presents the client with a 64-bit address space. The layout of this address space is entirely up to the individual server. No memory locations are reserved by the SIAP protocol. The client can read from and write to this address space using SIAP messages. Reading and writing is byte-by-byte to avoid any confusion resulting from byte ordering, and to give the greatest flexibility and efficiency when dealing with a variety of server platforms.

The SIAP messages provide three different ways to read bytes from the server address space. The byte_read reads a single byte at address A and returns this byte in a data_return message. The block_read reads a block of bytes and returns them in a data_return message. The number of bytes, N is L-12, where L is the SCL content length. The first byte comes from address A and the last from address A+N-1. The stream_read asks the server to read repeatedly from the same byte address and send back all the bytes it read as a single data block. The server reads address A a total of N = L-12 times and returns the N bytes in a data_return message.

The TCM divides its 64-bit SIAP address space between the RCMs. Each RCM will have a private address space, through which registers may be accessed independently of other RCMs. The RCMs will also share a public address space, through which registers may be accessed simultaneously. The public address space will have a mask that allows it to be shared by any subset of the RCMs. The TCM will have its own segment of address space in which it can present its own status and configuration.

Example: To set the TCM timestamp, we write to a set of registers on the TCM using a block_write with the timestamp register address. To read the TCM timestamp, we read the same registers with a block_read.

Example: To start simultaneous image transfer from all rafts into all RCMs, we write to a register in the public address space. To start simultaneous image transfer in seven out of twenty-five rafts, we set the public address mask to exclude all but these seven rafts from the public address space.

Example: To read an image window from a raft, we write instructions to the raft RCM through its private address space. We read the image data out of the RCM with a block read message.

Because the SIAP protocol reserves no locations in the server's 64-bit address space, there are SIAP messages that permit any SIAP client to interact with any SIAP server and query its status and configuration. The version_read message requests the server software version. The server should implement the version_read processing as efficiently as possible, so that SIAP clients can use the version_read as a way of punctuating message transmission. If the client sends too many messages to the server, none of which require a response from the server, the server message buffer will over-flow. The client uses version_read to introduce a pause in such sequences. When the client receives a data_return message with the server version number, it proceeds.

Example: The client wants to direct the TCM to accumulate in its address space twenty-five 1-KByte blocks of ADC samples from the RCM temperature sensors. To improve the efficiency of this exchange, the client sends the TCM a long sequence of commands that causes the samples to be transferred into the TCM memory and assembled into a 25-KByte block. During the sequence of commands, the client uses version_read to avoid over-flowing the TCM message buffer. When the data block is ready, the client reads it out with a single block_read.

The TCM must abort whatever it is doing whenever its client closes its TCPIP socket. The TCM must check the status of the socket frequently. The byte_poll instruction puts the SIAP server into a loop, waiting for a particular byte to take on a particular value. The client must be able to abort this loop at any time by closing the socket. No harm must come to the TCM, nor to any RCMs, as a result of sudden abandonment of any data acquisition activity when the client closes the socket.

Example: If, during the readout of an image from a raft, we want to abort the data transfer, we abort the TCPIP socket to the TCM. Closure of the socket causes an immediate halt to the TCM's participation in the data transfer from the RCM. We open a new socket and send a Stop instruction to the RCM to stop it transmitting any further data. When the TCM aborts communication with the RCM, the RCM returns to its rest state and is immediately ready for further interaction with the TCM.

No command register on the TCM or on the RCMs will contain more than one parameter. We have plenty of address space, so each independent parameter will have its own address. Thus we disallow the use of separate bits in the same register byte for independent parameter.

Note: Our one register, one parameter restriction frees us from the need to read a register before we write to it. When two independent parameters share a byte, we cannot set one of them without re-writing the other. By assumption, the other parameter is independent, so we have to obtain its current value from somewhere before we set it again. Either we read the register itself, in which case the register must support reading, or we keep a copy of the register in memory (the shadow byte).

All SIAP servers have a configuration file that lists the permitted client IP addresses, specifies its own IP address, its timeout behavior, security level, its name, its serial number, and various other such parameters, as well as any server-specific configuration information. This configuration will reside in EEPROM so that it is available upon start-up. We can re-write the configuration file dynamically over TCPIP with a configuration_write message. Because the new configuration file can specify a new IP address for the server, it is possible to cripple a SIAP server by giving it an IP address that makes any further communication with the server impossible.

Example: The TCM IP address is 120.1.1.2 in a local area network in the camera utility crate. The utility crate has its own local area network, and communicates with the outside world through a router. The router accepts TCPIP communication with any IP address in the range 120.1.0.0 to 120.1.255.255. We re-write the TCM configuration file and give it IP address 10.0.0.2. We send a reset message. The TCM re-starts with this new IP address. Now the router makes it impossible for us to contact with the TCM from outside the camera. We have to unplug the TCM and re-configure it with a separate cable.

A SIAP server can protect itself against such a disaster by requiring that the client log in with a password for supervisory access. Such a requirement makes it very unlikely that normal SIAP client procedures will re-write the configuration file by accident.

A SIAP server need not implement all SIAP messages, but it must at the very least implement the version_read and error_message. If the server receives a message that it does not implement, it must return an error_message containing an ASCII text string describing its failure to implement the client's command.

The SOAR control system used its own messaging protocol, which they called the SOAR Messaging Library. This messaging protocol tended to use ASCII command lines in the place of numerical message identifiers. The SIAP string_command message allows SIAP servers to receive ASCII command lines. A SIAP server that implements string_command will pass the string to a command interpreter and act accordingly. The string_echo message instructs a SIAP server to return the its string field in a data_return message.

RCM Communication

Communication with the RCMs will conform to the protocol defined by Document 4512 in the LSST Arcvhive. You need a password to get into the archive, so we provide a PDF copy of the document here for the benefit of those who have forgotten their password or who don't have one at all.

The RCM-RCM communication requires four LVDS logic signals between the TCM and each RCM. We plan to provide thirty sockets for RCM connections, and each socket will carry four LVDS signals.

The first LVDS signal is the RCM system clock, SCLK. The system clock is a 50-MHz square wave. Generation and distribution of the system clock is one of the most important duties of the TCM.

Note: Although the system clock will be in phase with itself when it leaves the TCM, the cables connecting the TCM to the Rafts may not be all of one length. An LVDS signal propagates at close to 20 cm/s, so a 1-m difference in cable length will lead to a 5-ns difference in system clock phase. We do not expect there to be any ill-effects resulting from such phase differences. Indeed, we expect there to be some beneficial effects resulting from a distribution of the clock edges in time. But if there are ill-effects from these phase differences, the TCM will be able cooperate with the RCMs to measure the round-trip propagation delay of each cable, and will set the outgoing system clock phases (and the phases of the SDO and SYNC that travel with SCLK to the RCM) so that the phase of the clocks arriving at the RCMs matches a requirement delivered by the CCS Master.

Example: We observe synchronous noise in the images we obtain from one raft. The noise disappears when we turn off an adjacent raft. We leave both rafts turned on and adjust the relative phase of the system clocks arriving at both rafts so that they are at 90° to one another (in a 50-MHz clock, that's 5 ns). If this action affects the synchronous noise, we learn something by observing the effect. If this action does not affect the synchronous noise, we learn something also.

The second LVDS signal is synchronise, SYNC. The protocol uses SYNC to mark the start and finish of serial transmissions. As master of communication with the RCMs, the TCM drivers SYNC.

The third LVDS signal is serial data out, SDO, which carries serial bits from the TCM (the Master) to the RCM (the Slave). By default, SDO is 0. It takes the value 1 to present a start bit to the RCM, and in subsequent data bits it may be 1. It takes the value 0 to present a stop bit at the end of a data transmission, and remains 0 until the next start bit.

The fourth LVDS signal is serial data in, SDI, which carries serial bits from the RCM (the Slave) to the TCM (the Master). By default, SDI is 0. It takes the value 1 to present a start bit to the TCM, and in subsequent data bits it may be 1. It takes the value 0 to present a stop bit at the end of a data transmission, and remains 0 until the next start bit.

Note: The start and stop bits are sufficient to organise communication between the TCM and the RCM. The RCM is invited to use SCLK to clock the incoming serial data, and SYNC to mark the start and end of transmissions. The intention of SYNC is the make the communication more robust. The intention of SCLK is to make the 50-MHz clocks on all rafts synchronous.

The TCM transmits 32-bit commands to the RCMs. When an RCM sees SYNC asserted, it begins clocking thirty-three bits on the rising edges of SCLK. The first bit is a start bit. The next thirty-two are data bits.

Note: The protocol does not define whether the RCM must keep clocking bits until SYNC is un-asserted, or whether it can take the first thirty-two bits as the data regardless of the behavior of SYNC.

Following a 32-bit command, the TCM can transmit data words to the RCM. We expect these data words to be 16 bits, but they may of another length, depending upon the command they follow. The RCM will expect these data words as a result of the content of the 32-bit command: the command was a WRITE. Each word begins with a start bit and ends with a stop bit. If multiple words are coming, SYNC will remain asserted throughout the transmission.

Alert: According to the current specification, the TCM puts the RCM in a receiving state when it sends a WRITE command. The RCM expects a certain number of data words to follow. It is not clear how the TCM can abort its WRITE operation without leaving the RCM locked in a waiting state. If the TCM un-asserts SYNC before the RCM has received all its WORDS, the RCM can return to its rest state. But if the TCM does not want to send even the first word, the RCM will interpret the next 32-bit transmission as a partial transmission of two 16-bit data words. Perhaps the RCM should always reset when it sees SYNC asserted without a start bit.

Suggestion: The RCM communication state machine gets reset by a rising edge on SYNC accompanied by a 0 on SDO.

The RCM responds to READ instructions by transmitting data words to the TCM. Each data word begins with a start bit. The start bit allows the TCM to synchronise its reception of the data bits regardless of the length of the cable between the TCM and RCM.

Note: We found in the LWDAQ Driver (A2037E) implementation of the LWDAQ byte transfer, the optimal clock speed for the simplest state-machine-based, start-bit-synchronised serial receiver is four times the bit rate. In this case, the bit rate is 50 MBPS, so we need a clock frequency of 200 MHz. The TCM will first synchronise the incoming SDI signal with its 200-MHz clock. Its receiver will sit in a loop waiting for the start bit on SDI. When the start bit arrives, the receiver waits 3 clock periods and clocks the first data bit, and so on. If the start bit does not arrive within a timeout period (which will be only a few microseconds), the receiver will abort and pass an error flag to the TCM status register, and an all-zeros data word to the TCM memory.

Packaging

We have not yet determined the package outline or volume limitation.