Porting the Linux e1000e Driver to RTnet

My client just switched his hardware to a SOM which comes with an on-board E1000 PCIe chip. The RTnet (or Linux) e1000_new driver did not recognize the card so I had to hack the e1000e driver.

Here are the steps (mostly in netdev.c):
- borrow Makefile from RTnet e1000_new;
- junk Ethtool code, disable IPv6;
- short out all of the TCP offloading code -- this is a real time networking stack;
- rip out the Linux stack calls and replace with RTDM if applicable;
- ditto for skb->rtskb;
- kmalloc / kcalloc / kzalloc wrapped for RTDM;
- hash-define rtskb_copy_to_linear_data, rtskb_copy_to_linear_data_offset to some memcpy;
- pre-allocate a pool of 256 rtskbs for RX and initialise it;
- changed IRQ allocation to legacy (not MSI);
- connect to STACK_manager in e1000_open;
- added I/O RT task which is the RTnet bottom half -- waits on a semaphore, calls e1000_clean() and if packets received signals RTnet to process the packets;
- changed the ISR (legacy) to signal the semaphore above is IRQ belongs to the driver;
- removed the QoS stuff;
- removed the VLan stuff;
- disabled multicast code -- RTnet's support for it is flakey;
- disabled set MAC code -- RTnet does not support it;
- disabled skb frags code -- RTnet does not support it;
- disabled change MTU code -- RTnet does not support it;
- disabled power management code -- RTnet does not support it;
- modify RTnet's configure to have "--enable-e1000e".

I spent most of the time trying to compile the hacked code. After I got an IRQ kicking I spent a lot of time making sure the driver is stable and that inserting/removing its module does not thrash the box.

The ported driver (base kernel version is 2.6.29) is here.

-ulianov

A Very Nice and Very UNIXy Bug

I encountered a very puzzling bug in a user-land app I helped write. I wrote the socket comms library and the database backend (using SQLite3).

Once in a blue moon one of the sqlite3 databases would become corrupted and would be written off by the integrity_check pragma. The first thing to blame was sqlite3 with multithreading (tho I had it compiled for that). It ended up not being this.

What happened was soo much better: the socket library was being used by a thread pool so when a new request came it was handed to a worker thread which had to talk to some innards of the app and formulate a result. In the meanwhile the accepted socket was being kept open.

The comms library was used on the client side by ephemeral CGIs that were called by a YUI2 webapp. The web server was thttpd -- it has a CGI timeout cut-off after which it kills the child CGIs.

The innards of the app could sometimes take longer to respond that the CGI cut-off time so the CGI vanished and the client socket was closed. But on the app side (=server) the TCP socket was being used. When finally the innards finished doing whatever they were doing the data would be written to the dangling socket using write(2).

But in the meanwhile in another galax^H^H^H^Hthread a sqlite3 db would be opened, used and closed. UNIX has a policy of reusing the lowest numerical socket that becomes free.

See the conflict? The worker thread would write some late-arriving data to what it thinks it's a socket but now it's a file descriptor!

I fixed this by changing all calls for read/write in the comms library to recv/send -- the latter pair only works on sockets. Also for added paranoia I sprinkled the comms code with getpeername(2) and would log a critical error if a socket descriptor did not look as a socket.

Only took me two days to get to the bottom of this.

-ulianov

How to Measure the Duration of a RTAI Semaphore Operation

Today I was asked the question "How long does it take to signal a semaphore?".

This is important as I had to do it in an ISR [which serviced a quasi-timer IRQ] to signal an RT task to start doing a job (this is akin to Linux tasklets).

Here is how I measured it (interrupts are disable to keep the measurement accurate):

#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/kernel.h>
#include <linux/types.h>

#include <rtai_sem.h>
#include <rtai_sched.h>

SEM pollSem;

static int __init testsem_init_module(void)
{
  rt_typed_sem_init(&pollSem, 0, BIN_SEM | FIFO_Q);

  rt_global_cli();

  volatile unsigned long long ticks_start;
  __asm__("rdtsc\n\t"
          "mov %%edx, %%ecx\n\t"
          :"=A" (ticks_start));

  rt_sem_signal(&pollSem);

  volatile unsigned long long ticks_end;
  __asm__("rdtsc\n\t"
          "mov %%edx, %%ecx\n\t"
          :"=A" (ticks_end));

  rt_global_sti();

  long long dT = ticks_end - ticks_start;

  rt_sem_delete(&pollSem);

  printk(KERN_DEBUG "rt_sem_signal took %lld ticks\n", dT);

  return -EBUSY;
}

module_init(testsem_init_module);
MODULE_LICENSE("GPL");

The answer was (on a PIII/1.2GHz) about 300 cpu ticks which is ~0.25 uS.

-ulianov

Porting the Linux e100 Driver to RTnet

My client has been using the E100 Intel card that came with his embedded mobo. When using the RTnet driver (eepro100, an antique version of the Becker driver which did not do firmware download to the card) they were experience strange TX lockups which could only be cured by a power cycle.

It was either trying to fix the eepro100 driver (and maybe download firmware borrowed from the newer E100 driver) or port the stock Linux e100 driver which had no lockup issued, quod fecit.

Here are the steps (in e100.c):
- alter Makefile, Makefile.in from RTnet;
- junk Ethtool code, disable IPv6;
- junk eeprom write code;
- junk loopback test code;
- rip out the Linux stack calls and replace with RTDM if applicable;
- ditto for skb->rtskb;
- replace schedule_work with a semaphore and rtdm_nrtsig_pend()
- kmalloc / kcalloc / kzalloc wrapped for RTDM;
- pre-allocate a pool of 256 rtskbs for RX and initialise it;
- connect to STACK_manager in e100_open;
- added I/O RT task which is the RTnet bottom half -- waits on a semaphore, calls e100_poll() and if packets received signals RTnet to process the packets;
- changed the ISR (legacy) to signal the semaphore above is IRQ belongs to the driver;
- disabled set MAC code -- RTnet does not support it;
- disabled change MTU code -- RTnet does not support it;
- modify RTnet's configure to have "--enable-e100".

I spent most of the time trying to compile the hacked code. After I got an IRQ kicking I spent a lot of time making sure the driver is stable.

The ported driver (base kernel version is 2.6.29) is here.

-ulianov

To Have a CLI or Not?

Any embedded product for the telco market will eventually have a CLI. Most have Web GUIs but they are not good enough as scripting a JavaScript-heavy GUI is hell.

So customers usually demand a CLI so they can script their operations and perhaps make the configuration job easy -- any network admin worth his salt dislikes clicking a 1000 times to bring a system up to scratch.

Cīsco reigns king here. Everybody in the industry is familiar with their style of CLI and expects it. (The alternative is to give access to the native OS commands, i.e Linux, but this can be dangerous and does not make the configuration job easier.)

In a previous life I have seen this implemented ad-hoc in C (using ncurses) and in a different shop a horrendous mess made with Python (the latter was quasi-unmaintainable). Both had problems and did not conform to the Cīsco style.

Luckily a smart cookie published a LGPL libcli (written in C) which actually works well.

The only modification I had to make [and submit back] for adapting it to a serial port was to cut out some funny characters it was sending at the beginning of the session.

The downside was that I had to convince the powers that be in my company that this won't blow up their IP. Also I had to link in statically libsqlite3 so I can manipulate the password database which pushed the size of my custom CLI to 0.5 megabyte.

-ulianov

M$ Vestigials: nmake and CMD.EXE

I was trying to integrate my Linux PPC build with TFS build and the only way I found this possible was via a "nmake" VS project. This is just contorted: a VS 2010 project wrapping a nmake Makefile is promoted to "build" status.

All this fuss just to map a Samba share, SSH into the Linux build machine and perform the build (the denx cross-compiler we use is only hosted on Linux).

The nmake is a pale and withered imitation of the one true make, the GNU make.

And I am not saying it for lack of trying: nmake does not pick up environment variables correctly and it certainly does not allow me to say

TIMESTAMP = $(shell unixdate '+%%Y-%%m-%%d_%%H.%%M.%%S')

Also it does not follow the respected idiom CC ?= gcc (i.e. Set the CC variable to "gcc" if it wasn't set alteady in this Makefile or the environment.)

This may seem obscure but if one wants to get TFS to get a snapshot onto uniquely named directory onto a Linux samba share then one is out of luck.

Which brings me to the need to have a CMD batch file to handle all this. The backticks implementation in CMD is heinous:

for /F "usebackq" %%a IN \
    (`"unixdate +%%Y-%%m-%%d_%%H.%%M.%%S"`) \
    do @set TIMESTAMP=%%a

Yes, these are the wrong way to do stuff on a Win32 machine but you do what you have to do to get the job done.

-ulianov

When make(1) Starts Spinning in Circles

I have this Linux project where the "depend" Makefile target reads:

${DEPDIR}/%.d:  %.cpp
 @echo "Determining dependencies for $<"
 @${CXX} ${CPPFLAGS} -E -MM $< | \
   sed -e 's~\($*\)\.o[ :]*~\1.o $@ : ~g' > $@

In one instance this snippet would start to run in an infinite loop.

On a closer look it turned out that the build machine I was trying to make this the system clock was waaay behind the `real' time and there was nope in h*ll to get it sync'ed as the corporate firewall blocks the S/NTP.

The files were checked out out of TFS via Samba which somehow preserves the time stamp of the client Win32 machine (whose clock was right).

The clock being behind it means that all builds artefacts were indeed older than the files that came out of revision control, thus the infinite loop.

The solution is rather simple (as this is build machine, not a dev machine):

# The time on the Linux build machine may drift
# and be behind the time of the TFS agent. Yet when
# files are deposited on the Samba share their
# time stamps come from the agent and that can
# be "newer" than any file we produce locally
# on the Linux machine thus triggering an IFINITE loop.
#
# We time stamp all files with the (same) local time
# stamp to avoid this.
CO_TIME=$(date '+%Y%m%d%H%M'); export CO_TIME
find -type f -exec touch -t $CO_TIME {} \;
unset CO_TIME

What's frustrating is that I had this problem 4 years ago when I did not have this blog to remind myself of build oddities.

Also during the same exercise I learned that the Win32 DEL command does not cope well with symlinks on the Samba share so it needs a small assisst at the end of the build:

find -type l -delete

-ulianov

Speeding up XFree86 Startup on a x86 Target

I have an embedded motherboard with the ?i945G? graphics chipset. XF86 4.6.0 would take 11 seconds of blank screen to start up which is unacceptable for an embedded applicance [the user might think that the system went belly-up].

While playing with vesafb I learned that if I start Linux in graphics mode vga=0x314 (which also enabled me to claim half of the boot screen with the company logo) the start up time of the XF86 (same server) is cut to only 1.5 sec.

Prior to this I tried Xvfb which starts instantaneously but it's horrendously slow.

-ulianov

P.S. Bolting the logo into the kernel using a 224 colour ppm ASCII bitmap is gross.

Circumventing a Corporate Firewall

Having work in the NA corporate world for a while now I can list a few ways of circumventing firewalls and freely communicating with the outside world.

Keep in mind that corporations have edge firewalls and restrictive firewalls on the users' computers which in most cases run Windows XP.

Here are a few ways I got it working for edge firewalls:
1. running OpenVPN over UDP/33400 [apparently this port and and a few after are used by tracert];
2. running OpenVPN over TCP/21: this is used for FTP and some corporations allow direct FTP connections;
3. running OpenVPN over TCP/1194, which comes as no surprise at this is the IANA OpenVPN port;
4. tunnel over HTTPS with CONNECT but this is short-lived.

Sometimes when one sells a device which needs to be controlled over Ethernet then it is next-to-impossible drive it for people have those pesky local firewalls.

Some esoteric ways to confound a local firewall would be:
1. use a Windows named pipe to talk to a Samba program that is the server for the named pipe;
2. use payloads for icmp-request and icmp-response in fact implementing a UDP/ICMP; some firewalls block incoming ICMP;
3. make the device respond with icmp-destination-unreachable (with payload) which are never blocked else all the IP stack is thrashed.
4. use UPnP to exchange data.

-ulianov