Embedded Linux System

Kevan Hashemi, alignment.hep.brandeis.edu
ELS-Hosted Website

Contents

Introduction
ELS Computer
Flash Disk Mounting
Flash Disk Copy
Flash Disk Modify
Bootable Disk Creation
Boot Desktop with ELS
Available Utilities
Compiling IO Programs
Performance
Conclusion

Introduction

The Brandeis University High Energy Physics Deparement (BNDHEP) is committed to designing and building the Timing and Control Module for the camera in the Large Synoptic Survey Telescope (LSST). At the time or writing (07-NOV-07), the LSST Camera Control System group plans to build all independent pieces of the Camera Control System around industry-standard PC104 embedded computers running Linux.

Here we keep a record of our efforts to prepare a PC104 module running an Embedded Linux System (ELS) for use in the the Camera Control System. Our entries are casual. They proceed from earliest work first, so you must scroll down for the latest work.

ELS Computer

We purchased a PC104 Embedded Linux System (ELS) and development hardware from WinSystems. As shown on our purchase requisition, the computer itself is part number PCM-SC520-0M. We also purchased a digital input-output module, part number PCM-UIO48.


Figure: PC104 ELS Computer, PCM-SC520 from WinSystems. To one side are a monitor and console for a terminal interface with the computer. The black box in the background is the power supply.

The computer you see in the above photograph consists of a stack of four boards and a bunch of adaptor cables. The bottom board is the PC104 mother board. The mother board holds the processor, an AMD i486, and a flash card that serves as a hard drive. The next board in the stack is the VGA graphics board, which allows us to connect a monitor and console. On top of that is our digital I/O board. The topmost of the four boards is a diagnostic board that provides numeric error codes if the computer fails during its boot sequence.

The computer comes with a flash card socket on the bottom side and a 128 MB flash card containing Winsystem's compact ELS operating system. The computer boots off the flash card and loads Linux. According to WinSystems, the flash card is a custom design that acts as a bootable hard drive when connected through its socket to the computer's IDE drive. We ordered three more flash cards from WinSystems, two of 512 MB and one of 128 MB. We connect the flash cards to our desktop Linux box with a flash to USB adaptor, as shown below. Our desktop is running Scientific Linux 2.4.21-40.


Figure: Flash Memory Cards for PC104 Computer. The flash card holds the boot program and a compact version of Linux. The silver enclosureis a flash card reader, for connecting the flash card to a USB socket on another computer.

We followed Winsystem's Quick-Start Tutorial, and soon had the computer connected to the internet at address 129.64.37.40. We started up the web server, telnet server, and other utilities. You can ping the computer from anywhere on the web, or click here to connect to its home page.

To edit configuration files on the ELS computer, we had to use the vi editor. We soon became passably competent with the venerable editing program, but it's clear that we will be severely restricted in our efforst to edit, compile, and run code on the ELS computer if our only means of editing files is through an editor like vi.

Flash Disk Mounting

[26-OCT-07] We remove the flash card from our ELS computer, plug the flash card into the card reader that came with our development kit, and plug the card reader into our Linux desktop. The green light on the reader turns on. Nothing happens on the Linux desktop. We plug the card reader in again. The green light does not turn on. We re-started the computer and the green light turned on when we plugged in the card reader. We later find that unplugging the card reader, waiting for five minutes, and plugging the card reader in again, causes the light to turn on.

Despite the shining green light, nothing appears on the Linux desktop. We tried the mount utility. Sometimes we could mount the disk as the root user, but we could not write to it.

You can see the flash card reader in the photograph above. Our Linux PC runs some version of Red Hat Linux. We can't figure out yet how to determine the operating system version.

We run Linux's fdisk utility. We can run fdisk as any user, but it won't do anything unless you run it as root user. Our fdisk is in /sbin, which is not in our default path, so we run it by giving it's full name. Here's the output from fdisk when we pass it the -l option on one of those occasions when the green light on the reader is illuminated.


Disk /dev/sda: 131 MB, 131334144 bytes
8 heads, 32 sectors/track, 1002 cylinders
Units = cylinders of 256 * 512 = 131072 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sda1             1       980    125424   83  Linux

Disk /dev/hda: 163.9 GB, 163928604672 bytes
255 heads, 63 sectors/track, 19929 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1             1      1275  10241406    5  Extended
/dev/hda2          1276     19929 149838255   83  Linux
/dev/hda5   *         1       653   5245159+  83  Linux
/dev/hda6           654       784   1052226   82  Linux swap
/dev/hda7           785       915   1052226   83  Linux
/dev/hda8           916      1046   1052226   83  Linux

The first disk fdisk reports upon is our flash card. Linux has given the flash card the name /dev/sda. The /dev/sda device has one partition, which Linux has called /dev/sda1. The main hard drive, meanwhile, is /dev/hda with eight partitions /dev/hda1 to /dev/hda8.

We try to mount /dev/sda1 partition on the flash card with the mount command. As root user, this works, but as another user, it does not. To get the mount to work properly, we edit the /etc/fstab file. This file is a text file that tells Linux how to respond to requests from users to mount new devices. Here is our enhanced fstab file. We add the last four lines to allow users to mount USB devices like the card reader.


LABEL=/     /          ext3    defaults        1 1
none        /dev/pts   devpts  gid=5,mode=620  0 0
none        /proc      proc    defaults        0 0
none        /dev/shm   tmpfs   defaults        0 0
LABEL=/tmp  /tmp       ext3    defaults        1 2
LABEL=/var  /var       ext3    defaults        1 2
LABEL=/home /home      ext3    defaults,rw     1 2
/dev/hda6   swap       swap    defaults        0 0
/dev/cdrom  /mnt/cdrom udf,iso9660 noauto,owner,kudzu,ro 0 0
/dev/sda1   /mnt/sda1  auto  auto,rw,user,exec,async 0 0
/dev/sda2   /mnt/sda2  auto  auto,rw,user,exec,async 0 0
/dev/sdb1   /mnt/sdb1  auto  auto,rw,user,exec,async 0 0
/dev/sdb2   /mnt/sdb2  auto  auto,rw,user,exec,async 0 0

The fourth line from the bottom tells Linux how to handle requests to mount a device /dev/sda1. We instruct Linux to make the contents of /dev/sda1 appear in the /mnt/sda1 directory, which we create ourselves. The /mnt/sda1 directory is th mount point for the sda1 device. We created this directory ourselves. The word "auto" in the third column tells Linux to detect automatically the file system that exists on the device. The final sequence of words, separated by commas, are the mount options. These are "auto" to say the device should be mounted automatically after a re-boot, "rw" to mount it as read-write, "user" to allow any user to mount and unmount the device, "exec" to allow us to execute binary code on the device, "async" allows the operating system to write changes to the device whenever it is convenient. Because we mount with the "async" option, we must unmount the disk before we unplug it. The zero in the fifth column tells Linux's back utility, which is called dump, not to back up this device. The zero in the sixth column tells Linux's disk-checking utility, which is called fsck, not to check this device.

We can now mount sda1 with the following command when logged in as any user.


mount /dev/sda1

The device appears as an icon named "sda1" on our Linux desktop. We can unmount it from a terminal with the following command, provided that no terminal or window is using the device.


umount /dev/sda1

We have not figured out yet how to get the flash drive to mount automatically when we plug it in. Nor do we understand why sometimes the computer refuses to notice the flash drive for five minutes. When the green light is off, fdisk lists no information about the drive.

We unplug our card reader and plug another USB memory stick in its place. Linux names the stick /dev/sdb. We unplug it an try another memory stick. Now fdisk reports the following about the new memory stick.


Disk /dev/sdc: 128 MB, 128450048 bytes
8 heads, 32 sectors/track, 979 cylinders
Units = cylinders of 256 * 512 = 131072 bytes

Device Boot    Start   End    Blocks   Id  System
/dev/sdc1      1       979    125263+   6  FAT16

We can't mount this drive with our existing fstab file entries. We need an entry for /dev/sdc1. We unplug the memory stick without mounting it, and go back to our original ELS flash device. The green light does not turn on. We wait five minutes and plug the device in again. The green light comes on and fdisk tells us that device /dev/sda1 is available to mount.

As an ordinary user on our Linux desktop, we find that we cannot write to the ELS flash drive. The drive always comes up as owned by "root". Our memory sticks come up as being owned by whoever mounted them ("hashemi" or "root"). Meanwhile, when we plug the ELS flash drive into our ELS computer, we can read and write to it as the "admin" user. We have no explanation for this behavior.

We consulted two Linux experts in the ATLAS collaboration. They tell us we have done what we can to make it easy to mount flash devices. One of them says, "Linux's philosophy is: if you do not know how to access the USB stick, you are not worthy of accessing it ;-)"

Meanwhile, we can plug any of our USB sticks into any Windows or MacOS computer and see them pop up on the desktop or in the file browser, regardless of how many times we have plugged them in and taken them out again.

Flash Disk Copy

[09-NOV-07] The flash card in our ELS computer acts as its hard drive. We call it the flash disk. We have only one copy of the ELS operating system, and this copy exists upon the 128 MB flash disk that came plugged with the computer. We bought extra flash cards compatible with our PC104 from WinSystems: 1×128 MB and 2×512 MB. Today we set out to make a copy of our existing ELS flash disk on our new, blank 128 MB flash card.

The method we arrive upon after a couple of hours of work is to use the Linux dd utility. First we plug the flash card we want to copy in our card reader and connect it to our Linux desktop. We do not mount the flash disk before we use dd.

The flash card appears to fdisk as device /dev/sda. We are drop the number after "sda". The entire flash card, including its partition table and all its partitions, appear in fdisk as /dev/sda. The first partition (and the only partition in this case) appears as /dev/sda1.

To make duplicate our flash card, we copy its entire contents including the partition table. The dd utility (we think dd stands for disk duplicate) reads bytes from a device or file and writes them to another device or file. We log in as root. We cannot read from or write to the /dev/sda device unless we are the root user. We copy the contents of our flash card we use the following command.


[root]# dd if=/dev/sda of=sda.image

The contents of the flash card are copied into the sda.image file. We unplug the flash card. We don't have to unmount the flash card because we never mounted it in the first place. We don't have to mount the disk in the file system for dd to be able to read bytes out of the disk.

We plug in another 128 MB flash card. It can be blank or filled with other stuff. Whatever is on there will be wiped out by the following command, and replaced by the contents of our original flash card. Once again, we must be logged in as root to execute our dd command on /dev/sda, and it's important that you have not mounted the /dev/sda1 partition in the file system, or else Linux will get confused and may fail to perform the copy properly.


[root]# dd of=/dev/sda if=sda.image

We unplug the newly-written flash card, plug it into our ELS computer, and turn on the power. The ELS computer boots up. It behaves exactly as it did with the other flash card. The webserver works, telnet works, ping works, and the account passwords are the same.

Now that we have a copy of the operating system, we can set our original aside as a backup.

Flash Disk Modify

[09-NOV-07] One way to work make changes to the ELS hard disk is to mount the flash disk on another computer and copy files to and from the flash card via the card reader. We mount our ELS hard disk on our Linux desktop. We find we cannot write to the disk as an ordinary user. The disk is owned by root, and protected against changes by other users. We take care of this problem by logging in as root and entering:


[root]# chmod -R 777 /dev/sda1/*

Now everything on the disk is readable and writeable by anyone. We copy this web page to the www user home directory and see what happens.

The transfer almost works. We have some conflicts with protection settings so that the pages don't load properly. After we change the protection on all the files with the following line, the website works fine, including images.


cd ../www
chmod -R 666 *.*

Another way to move files to and from the flash disk is by ftp. We connected to the ftp server on the ELS computer and copied the same files into the webserver directory. Before we copied the files, we set their protections so that everyone has read and write privelidge, but nobody has execute privelidge. The directories, on the other hand, we set so that everybody has execute and read privelidge.

With the privelidges set properly on the originating computer, the website on the ELS computer works correctly immediately after ftp transfer.

Bootable Disk Creation

[17-JAN-08] We previously created a backup of our ELS flash disk with the dd utility. Now we create a new bootable flash disk by formatting a new flash card with fdisk, installing a file system with mke2fs and tune2fs, copying files to the flash card from our Linux desktop, and making the card bootable with the grub utility. In doing so, we are following the instructions provided by WinSystems in their Creating WinSystems ELS-2.0 instructions, and we are grateful to George Hilliard of WinSystems for his support and patience with our many questions.

To begin with, we mount out 256-MByte ELS flash disk (the copy we made with dd of the original) on our Linux desktop and copy all files to a folder called ELS. We use the -a copy option (equivalent to -dpR) copy folders recursively, and to preserve links, mode, ownership, and timestamps.


[root]# mount /dev/sda1
[root]# cp -a /mnt/sda1/* ELS

These are the files we will copy to our new flash disk when it's ready.

We plug a new 512-MByte flash card into our card reader and plug the card reader into a USB socket on our Linux desktop. We run fdisk to create a primary partition on the disk as follows. A primary partition is one that leaves space at the beginning for a boot sector that will contain the master boot record. We create the master boot record later.


$ su
Password:
[root]# /sbin/fdisk /dev/sda

Command (m for help): h
h: unknown command
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)
 
Command (m for help): p
 
Disk /dev/sda: 526 MB, 526417920 bytes
16 heads, 63 sectors/track, 1020 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
 
   Device Boot    Start       End    Blocks   Id  System
/dev/sda1   *         1      1019    513544+   6  FAT16
 
Command (m for help): d
Selected partition 1
 
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1020, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-1020, default 1020): 1020
 
Command (m for help): p
 
Disk /dev/sda: 526 MB, 526417920 bytes
16 heads, 63 sectors/track, 1020 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
 
   Device Boot    Start       End    Blocks   Id  System
/dev/sda1             1      1020    514048+  83  Linux
 
Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.

Now we install a file system with mke2fs, which is, "a tool that allows you to create a Linux second extended (ext2) file system on any block device." The command takes a twenty seconds to complete.


[root]# /sbin/mke2fs -m0 /dev/sda1
mke2fs 1.32 (09-Nov-2002)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
128520 inodes, 514048 blocks
0 blocks (0.00%) reserved for the super user
First data block=1
63 block groups
8192 blocks per group, 8192 fragments per group
2040 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
 
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

As the utility recommends itself, we use tune2fs to disable mount counting and periodic file system checks.


[root]# /sbin/tune2fs -c0 -i0 /dev/sda1
tune2fs 1.32 (09-Nov-2002)
Setting maximal mount count to -1
Setting interval between check 0 seconds

We mount the flash drive as root, and copy all our ELS files into it.


[root]# mount /dev/sda1
[root]# cp -a ELS/* /mnt/sda1

Now it's time to make the drive bootable with the grub utility. We are still logged in as root, and our flash disk is still mounted. When we install grub on a drive, grub writes a short piece of code in to the flash drive's master boot record. The master boot record is small, so grub creates a directory called /boot/grub and puts the rest of the code and configuration parameters it needs in there. Among the files it puts in this directory is menu.lst, which contains instructions to grub telling it how to set up the boot option menu, and how to execute each option.

When our ELS computer boots up, it runs the master boot record code, which loads the full grub utility out of the /boot/grub directory. Now that grub is running it reads menu.lst and creates the boot menu. The user selects a boot method, say ELS-2.0. Associated with this option are a sequence of grub commands that select the boot device and the kernel file. The kernel file contains the code that starts up the operating system. We will show and explain these commands later.

Right now, we want to install grub into the master boot record of our flash card and make sure that all the necessary grub code and configuration files are on the flash card. We tried this with the /sbin/grub on our Linux desktop, but the resulting installation did not recognise the menu.lst file. It turned out that the grub on our desktop was version 0.93, while the one in ELS-2.0 is 0.95. So we run ELS/sbin/grub instead. Note that we are still logged in as root.


[root]# ELS/sbin/grub
grub> find /bin/cp
 (hd0,4)
 (hd1,0)

When we install grub on our flash card, we have to name the flash card and its partition. The grub device-naming convention differs from that of Linux. In grub, our flash card appears as a hard drive named hd1. The hard drives being numbered from zero and up. Thus hd0 is the desktop's hard drive. The way we determine our flash card name and its partition number is by telling grub to find a file we know exists on all the bootable Linux drives: the cp utility. Now we can tell grub to direct its attention to our flash card and install itself.


grub> root (hd1,0)
 Filesystem type is ext2fs, partition type 0x83
grub> setup (hd1)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd1)"...  16 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd1) (hd1)1+16 p (hd1,0)/boot/grub/stage2 
   /boot/grub/menu.lst"... succeeded
Done.                                                                                                      
grub> quit

We unmount the flash card, plug it into our ELS computer and turn on the power. We get the grub boot menu, as defined by menu.lst. Here are the commands that boot ELS-2.0.


title  WinSystems-ELS-2.0
root (hd0,0)
kernel /boot/bzImage-2.6.10-els-2.0 root=/dev/hda1 console=tty0 console=ttyS1,38400n8

Our ELS system the kernel file is bzImage-2.6.10-els-2.0. The grub utility understands the file system on the flash card. It knows how to find a file whose name is given as a path in its kernel command. It loads the kernel into memory and executes it. When grub starts kernel, it passes parameters to the kernel as a text string. The text string is the string at the end of the kernel command.

We select the ELS-2.0 boot option and the ELS machine boots off our newly-built flash card. Everything works exactly as it did with the original 128-MByte flash card we received from WinSystems.

The entire ELS file system occupies only 13.4 MBytes on our 512-MByte flash card.

Boot Desktop with ELS

To satisfy our curiosity, we take our flash card and plug it into our ELS machine. We get a grub command line. For some reason, grub is not finding the /boot/grub/menu.lst file on the flash card.

We now try to boot off our flash disk on the desktop machine with the following grub commands, which we put in the menu.lst grub configuration file.


title WinSystems-ELS-2.0
root (hd1,0)
kernel /boot/bzImage-2.6.10-els-2.0 root=/dev/sda1 console=tty0

We re-start the desktop. A grub menu comes up. We select the ELS-2.0 kernel. The computer loads the image and starts to boot. But it gets stuck when the kernel tries to open the root device, which we say is /dev/sda1. We belive Linux cannot open the root device because it has not yet mounted the flash drive, but we are not sure that's the reason.

Available Utilities

[13-JAN-08] We are sitting at home. Our ELS computer is running telnet and ftp servers. We cannot connect to either from home because our ELS computer is on campus, and our campus network blocks both telnet and ftp. Our ELS computer is not running ssh or sftp (the encrypted versions of telnet and ftp).

We connect by ssh to our webserver on campus, and connect by telnet from the webserver to our ELS computer. We now explore the commands and utilities available on the ELS computer.

UtilityAvailability
telnet/usr/bin/telnet, /usr/bin/, /usr/sbin/vsftpd
ftp/usr/bin/ftpget, /usr/bin/ftpput, /usr/sbin/telnetd
sftpnone
sshnone
gccnone
tclshnone
bashnone
X Windowsnone
Table: Utilities Available on ELS Computer.

We will try to add a new utility to the ELS computer. It may be that C-compilers and X-windows are too large for our flash card, so let's start with the bash shell, which is not available as a separate utility in ELS.

We mount an ELS flash card on our Linux desktop. We copy the /bin/bash to /mnt/sda1/bin/bash. We fully expect the bash code to call some shared libraries, so we use the nm utility to find out what it needs.


$ nm /bin/bash
nm: /bin/bash: no symbols

We put the flash card back in our ELS computer and boot up. When we try to run bash, it says it can't run because it can't find a file called libtermcap.so.2.

We copy our own LWDAQ Software onto the flash drive. We try to run the tclsh, the TCL shell we bundle with our distribution. The shell fails, saying it can't find libtcl8.4.so. When we do the same thing on our desktop machine, the shell runs fine. It finds the libtcl8.4.so sitting in the ../lib folder. When the ELS computer says it can't find libtcl8.4.so, we suspect it really means to say that it can't find another shared library called by libtcl8.4.so, which tclsh assumes is in the kernel.

Compiling IO Programs

[18-JAN-08] The ELS computer contains no C-compiler. Nor does it have a copy of the kernel files necessary for compiling new drivers. We cannot compile our IO (Input-Output) programs on the ELS computer.

One solution to this problem is to install a C-compiler on the ELS machine. We would like to install GCC. We spoke to George Halliard of WinSystems. He said he has never installed a C-compiler on an ELS computer. He says the GCC compiler is too large. He suggested we build our own embedded Linux from scratch using the recently-improved Buildroot system. By this means, we could include the MicroC compiler on our embedded system. The MicroC compiler is small. We enjoyed reading this explanation of Buildroot and the problem of compiling for and on embedded Linux systems. He also recommended we look at Linux from Scratch, which gives instructions on how to build your own Linux from source code.

Not only must the C-compiler we install upon our ELS computer be small, it must also be compatible with the C libraries and compiled kernel image that make up the heart of the ELS. The Linux on our PC104 computer is based upon Simply MEPIS. Its C libraries are stripped down, as is everything else. That's why the ELS takes up only 13.4 MBytes on disk, and can run with plenty of room in the ELS computer's 256-MByte RAM.

Instead of installing a compiler on the ELS computer, WinSystems recommends we compile code for the ELS on our Linux desktop, and copy the compiled executable files to the ELS computer by ftp or by flash card. Our immediate objective is to use the PCM-UIO48 to flash some lights, and measure the maximum switching speed of its logic outputs. To compile an IO program, we must do the following.

  1. Choose an IO device (PCM-UIO48A)
  2. Compile a driver for the IO device (uio48.ko).
  3. Install the driver in the kernel (with insmod)
  4. Compile a library of routines that talk to the driver (uio48.o)
  5. Create device nodes that present the device to Linux (uio48a, uio48b, etc.)
  6. Compile our program source code and link to the library (flash.c)
  7. Run our programs (flash and flash_fast)

In order to compile the driver, you have to have a copy of the expanded Linux kernel directory. Our WinSystems ELS is based upon Linux 2.6.10. We download linux-2.6.10.tar.bz2 from here and expand it. We extract the UIO48A support files from uio48io_kernel_2.6.zip supplied by WinSystems. We run make on the example programs. It compiles the driver uio48.ko. We take uio48.ko to our ELS computer. As the root user we try to load it into the kernel.


[root]# insmod uio48.ko io=0x240 irq=5
insmod: error inserting 'poll': -1 Invalid module format

After consultation with WinSystems, we conclude that we cannot build a driver for our ELS Simply-MEPIS computer using the C-compiler on our Scientific Linux desktop.

WinSystems showed us where we could download a copy of the driver that they compiled in Simply-MEPIS. We are able to load this driver in our ELS computer.


[root]# insmod uio48.ko io=0x240 irq=5
$ Id: uio48.c,v 1.1 2005/04/22 20:10:58 steve Exp $
UIO48 - IRQ 5 registered ok Chip 1

Our UIO48A is jumpered to occuby 16 bytes of address space starting at address 0x320 and to use interrupt request line 5 when changes take place on interrupt-enabled inputs. As you can see, the kernel accepts the driver.

The remaining steps go smoothly. We don't need the linux kernel to compile uio48.o from uio48.c. Nor do we need it to create the device nodes. We compile our two programs, flash and flash_fast. These both refer to uio48.h. The flash program is one supplied by WinSystems. The flash_fast program we created ourselves. It sets and clears a bit as fast as it can.

We copy our uio48a folder onto our flash drive. To copy the device nodes, we need do use the -a copy option, and we need to be root.


[root]# mkdir /mnt/sda1/usr/lib/uio48a
[root]# cp -a uio48a/* /mnt/sda1/usr/lib/uio48a

We plug the flash card into our ELS computer and boot. We cd to /usr/lib/uio48a. We re-install the uio48.ko driver. We run flash. We plug ten LEDs with 1-kΩ resistors into the first ten IO ports of the UIO48.


Figure: The UIO48A with Flashing LEDs.

The programs run as they should.

It seems likely that we can compile code for our ELS computer on any desktop Linux computer, provided we link to the correct drivers. We may run into problems, however, when we try to compile TCPIP socket code on Scientific Linux and expect it to run with the greatly-reduced TCPIP stack and Ethernet driver of our Simply-MEPIS Linux. In the end, we may be better off building a copy of our embedded Linux on our desktop. Such a copy is called a toolchain. With the help of the chroot utility, we could switch our desktop from compiling within its own root file system to compiling within the toolchain's root file system.

Performance

[18-JAN-08] The ELS computer provides a 133 MHz 486 microprocessor, 256 MBytes of RAM, and we are booting it off a 512 MByte flash disk. We can't read and write indefinitely from this flash disk. If we want a scratch disk for current operations, we should make one in RAM. We transferred a large file through an Ethernet switch to our ELS computer. The fastest data rate we obtain through the switch is 4 MBytes/s.

Our flash_fast program is compiled from C. It sets and clears bits on our UIO48A board as fast as it can. The figure below shows what this switching looks like on an oscilloscope.


Figure: The UIO48A with Flashing LEDs. Note the 5-μ delay between transisions and the jitter caused by other system activity interrupting the switching.

The switching frequency is only 90 kHz. The UIO48A Operations Manual does not tell us how fast we can expect the outputs to switch. Our experience has been, with eight-bit embedded microprocessors, that code compiled from C is ten times slower than code written in assembler. In this case, an IO write with no counter increment will take no more than twenty clock periods, or 150 ns. Ten times that would be 1.5 μs. Here we have 5 μs. It looks like the driver and the code that calls the driver are slowing down the computer by a factor of 30. We doubt very much that the UIO48A itself is responsible for the delay.

Nor can we speed up the performance of the UIO48A by reading and writing bits in parallel. The driver provides only a serial bit interface. Let us suppose, however, that we obtained a different IO board, one with a driver that could read and write 32 bits at a time, all in 5 μs. This would represent a data rate of only 800 kBytes/s, which is 5 times slower than the Ethernet transfer rate on the same computer, and 3 times slower than the data rate we obtain with the 22-MHz 8-bit embedded processor we use on our existing TCPIP data acquisition driver, the A2037E.

The only way we are going to obtain reliable, high-speed communication between the ELS computer and whatever circuit it controls is by writing drivers and low-level IO code in assembler. We are not sure how hard it is to write Linux drivers from scratch in assembler.

Conclusion

[18-JAN-08] Our ELS computer is fast over Ethernet but slow over its digital IO board. The Ethernet transfer rate is 4 MBytes/s. The maximum digital IO switching speed is 90 kHz. Each call to the IO device driver takes 5 μs. If we were to use one IO bit for serial data transfer, our maximum data rate would be around 20 kBytes/s. Furthermore, our data rate would be intermittent because Linux's pre-emptive multi-tasking interrupts our IO activity frequently.

Suppose our IO board accepted 32-bit data instead of 1-bit data, and suppose further that our external circuits are willing and able to make use of 320bit data transfers. The IO board would need to provide a data strobe. The maximum frequency of this strobe would be 90 kHz. Our maximum data rate would be under 400 kBytes/s, which would still be ten times slower than the Ethernet data rate.

If we want our ELS computer to communicate with external circuits at more than 1 MByte/s, we must re-write its digital IO device drivers in i486 assembler and make block data transfer available to our IO programs. Instead of taking ≈150 instructions to perform one IO cycle, a driver written in assembler would take ≈20 instructions. Our callse to these functions would provide the driver with a pointer to a block of memory. The driver would perform the block transfers with a loop written in assembler.

One of the advantages forseen by using ELS computers was that we would obtain a high-performance, low-power embedded computer with fast IO for very little effort. We spend five working days figuring out how to duplicate the ELS file system and compile IO programs on our Linux desktop that would run on the ELS computer. Now that we have put in this effort, however, we believe others trying to duplicate our efforts will make more rapid progress, even if they choose a PC104 computer form a different manufacturer. But writing drivers in asssembler ties our solution to particular hardware and will take weeks of engineering effort, if not months. Indeed, we cannot even guarantee that such an effort will be successful in the long run. Subsequent versions of Linux may be incompatible with our custom-made drivers. We may be unable to take advantage of Linux bug fixes.

Another advantage of PC104 ELS computers is that the operating system is robust and multi-tasking. We can connect over TCPIP at any time, stop IO existing processes, load new IO programs, and re-start our IO processes. We can have several IO processes running at the same time without waiting for one another. We can force a re-boot over the network without turning off the power or touching a switch.

Nevertheless, if we are to proceed with PC104 ELS computers as our control nodes in the LSST camera, we must first demonstrate that we can switch an ELS computer's digital outputs at sufficient speeds to satisfy the requirements of such boards as the Timing and Control Module. If we cannot demonstrate sufficient speed, there is no point in using ELS computers. There are other embedded options, such as the RabbitCorre modules from Rabbit Semiconductor, which we have used for years, with great satisfaction. The latest modules provide 500-kByte/s over Ethernet, and 5 MBytes/s over an eight-bit parallel interface. The modules are one tenth the size of a PC104, cost one tenth as much money ($100 instead of $1000), and consume less than half the power.

[21-JAN-08] We summarise our work, and present the problems with the ELS as a choice of control node for the LSST Camera Control System is a talk we prepared for the Camera Control Workshop in Tuscon, 21-23 January, 2008, Camera Control Nodes.