Thursday, February 23, 2012

IEC GOOSE Application -- Network Latency On a RTAI/RTnet Box

I have an embedded box running Linux 2.6.29/RTAI/RTnet. I hacked the e1000e driver for use with RTnet and also I eviscerated its ISR as (due to PCI limitations) its IRQ sits on the same line as an FGPA IRQ my client is using.

The FPGA RTAI ISR is calling the old e1000e ISR by
a) signalling a semaphore which
b) wakes an RTAI tasks which
c) calls the e1000e ISR.

On top of this there is an LXRT hard RT user-mode application (proprietary IEC GOOSE stack adapted by me to RTnet) which scoops all GOOSE packets from the wire.

Alas rteth0 must be put in promiscuous mode as Ethernet cards generally only allow 4 or 8 MACs to be programmed in their multicast list and GOOSE frames are emitted with the destination MAC to be the same as the source MAC plus the multicast bit turned on. So we can easily shoot past 4/8.

The LXRT application also knows how to send GOOSE packets.

The application is hooked to a Linux/RTAI driver which executes every 100 uS and does stuff. The comms between the app and driver are
a) RX -- RTAI shared memory for the incoming data and a volatile change counter
b) TX -- an RTAI semaphore signalled by the driver and RTAI shared memory for the outgoing data.

So the app stuffs data (GOOSE points) into the driver and sends driver points packaged as GOOSE frames.

One of the things the driver can be programmed to do is to respond to incoming point changes by reflecting them into outgoing points.

So the data flow is
e1000e->RTnet->LXRT app->driver->LXRT app->RTnet->e1000e
We timed using wireshark running on an external box the response time to a change in a GOOSE point and the answer came to ~1ms ±10% which is not bad at all considering the PCIe latency, scheduling latencies and FPGA ISR -- the latter can account to up to 200 uS.