Monday, December 14, 2009

How to Learn What Code is Actually Disabling IRQs?

When one applies the PREEMPT_RT patch the spinlocks & mutexes get horribly overloaded. cli/sti use is frowned upon (x86) so one ends up with macros of macros of macros for locking so it is a head ache to see what functions are actually disbling interrupts.

Yet we are on x86 and the locking sequence generated by GCC is simple:
cli
...
sti
or could by a bit more complicated:
pushf
pop %reg
cli
...
push %reg
popf
Luckily I used objdump and a bit of massaging come to the rescue in the form of a neat script that finds all the object files that make up the compiled kernel and then searches the disassembler output for these sequences.

Bonus: the function names that do the locking are also printed.

Here comes findcli_x86.sh [run it at the root of the kernel tree after you compile the kernel]:
#!/bin/sh

# This file is licensed under GPLv2

# Note this works only with x86 code

obj_list='obj';
trap "rm -f $obj_list" 0 1 2 15

function extractcli()
{
file=$1;

[ ! -r "$file" ] && return 1;

[ -z "$OBJDUMP" ] && OBJDUMP=${CROSS_COMPILE}objdump

$OBJDUMP -dSCl $file | awk 'BEGIN {
idx="nosuchfunc"
}
/(^[a-zA-Z_]|cli[^a-z]|popf|sti[^a-z])/ && !/(file format|Disassembly)/ {
if($0 ~ /^[a-zA-Z_]/) {
idx=$1
} else {
F[idx]=(F[idx] "\n" "\t" $1 "\t" $NF);
}
}
END {
for(idx in F) {
print (" " idx F[idx])
}
}'
return 0
}

# main

find -type f -name '*.o' > $obj_list
for obj in $(cat $obj_list)
do
o=$(echo $obj | sed 's/^\.\///g')
[ "$o" = 'vmlinux.o' ] && continue; # this is the whole kernel
echo $o | grep -q 'built-in\.o' && continue; # these are aggregations

echo -en " \r$o" 1>&2
cnt=$(objdump -dS $o | grep -cw cli)
[ "$cnt" = '0' ] && continue;

echo -en " \r" 1>&2

src='???';
c=$(echo $o | sed 's/\.o$/\.c/g')
S=$(echo $o | sed 's/\.o$/\.S/g')
[ -f "$c" ] && src="$c"
[ -f "$S" ] && src="$S"

echo "$o: $cnt, src: $src"
extractcli $o
done
As usual awk comes to the rescue.

-ulianov

P.S. I looked for a decent decompiler for Linux other than objdump but I found only ancient ones and all broken. This is annoying as sometimes I need to see what external functions are called by an object file (.o).

Monday, December 7, 2009

DTS Headaches on a PPC440EPx Board

I had problems trying to get an ST RTC chip to be recognized by Linux. The chip was attached to the I2C bus 0 at address 0x68, the chip is supported by Linux but the two did not talk.

This is a custom board made specially for my US employer. The vendor brought up U-Boot, tested the peripherals with it and that was about it. I tried the denx.de kernel in a Sequoia configuration and no luck, I haggled the board vendor and they gave me a Linux kernel that booted on the board but did not detect much of the hardware.

By playing with the kernel config I got it to sniff the NOR flash and to partition it using a command-line scheme but the RTC chip was a mistery.

I dug a bit in the kernel source and docs and learned that the kernel expects from the bootloader a flattned device tree yet U-Boot does not provide that. The vendor provided a DTS file that has a textual representation of the devices but it did not get the RTC right.

By poking around I learned that the PPC kernel is matching text labels ("compatible") provided by the device drivers against the labels in the device tree. I hacked those a bit and the RTC works fine now.

-ulianov

Friday, December 4, 2009

Linux 2.6.31/PREEMPT_RT and Hard IRQs

I was trying to improve the latency of an ISR which serves an IRQ which fires every 100 uS. As a Softirq even when running at the highest priority the latency would some times slip to 1500 uS which is unacceptable (dd if=/dev/hda of=/dev/null comes to mind as a reason, hda is a slooow Compact Flash).

It does not help that the ISR is crunching numbers and doing float calculation in interrupt context. This ISR munches about 25 uS and should execute as close as possible to the beginning of the 100 uS interval.

So I tried to use a hard IRQ (IRQF_NODELAY) which was not good as the kernel would BUG_ON in rtmutex.c:807 when a process was trying to read from the character device which belonged to the driver which used the said IRQ.

I learned that spin_lock/spin_unlock are being overloaded by PREEMPT_RT by rt_mutex_lock/rt_mutex_unlock which do not seem to be compatible with hard IRQs: the mutex gets double-locked by the "current" process altho the two contenders are the hard-IRQ and code executed on behalf of a user process. I think this is bonkers.

I managed to get it working by using "atomic_" spinlock primitives which basically boil down [in disasm output] to cli/sti.

-ulianov

Thursday, December 3, 2009

A Nice Kernel Trick for x86

Today I learnt a nice way to obtain the number of microseconds since the power-up of the CPU using a P5 performance counter via (rdtsc). This instruction returns the number of ticks since power-up but it can be scaled to microseconds by dividing the number by the CPU frequency (in MHz).

The kernel code reads:
#include <linux/kernel.h>
#include <linux/cpufreq.h>

unsigned long long uSecSinceBoot()
{
volatile unsigned long long int cpu_ticks;

__asm__("rdtsc\n\t"
"mov %%edx, %%ecx\n\t"
:"=A" (cpu_ticks));

unsigned long long div = cpu_ticks * 1000;
do_div(div, cpu_khz); // TRICK! div is modified, do_div() returns reminder

return div;
}
I am using do_div() as on x86/32 bits long-long division is supported by GCC via an external function (_udivdi3) which usually lives in libgcc but it is not provided by the kernel.

-ulianov

P.S. I did extract _udivdi3 from libgcc and then disassembled/reassembled it but it's a pain. do_div is the right thing to use in the kernel.

Friday, September 18, 2009

The Strange Case of Multiplying Zombies

While working on a Linux/PPC embedded board I changed /etc/rc.sh (the system startup script as known by Busybox) to start our application in foreground.

This was good as we could see its output on the serial console and be able to interact with it. The application is a standalone binary which is interrogated by CGI and thru a Web interface. The web server is thttpd.

So I had everything running and I looked at the process table and noticed it being filled by zombie (Z) entries of the logger CGI (which gets invoked thrice a minute).

I tried to trace (strace) thttpd, I even put a waitpid(-1) at the top of its main loop [it's a single-threaded web server] and still could not get the damn zombies reaped!

This was baad as the system could stay up only for half a day before filling up the process table.

I did some hard thinking and remembered some APUE and Bach bits and concluded that Busybox [which alas contains init] must be still waiting for the termination of /etc/rc.sh before it starts the prescribed init behaviour!! I.e. Reaping orphaned processes and zombies.

So I put our application in background via nohup and voila! everything was good again.

-ulianov

Saturday, August 1, 2009

Mutex Problems on PPC, Again

One would think that doing
pthread_mutex_t mutex;
pthread_mutex_init(&mutex, NULL);
would yield a usable mutex.

Yet this is not always the case: on denx/PPC 440 strange things can happen (see the previous post When Classes Instantiated as auto Vars on Stack are Evil).

It turns out that the "correct" code sequence is
pthread_mutex_t mutex;
memset(&mutex, 0, sizeof(mutex)); // Voodoo for PPC
pthread_mutex_init(&mutex, NULL);
otherwise under some circumstances (e.g. mutexes ending up on the stack in the belly of an Object) one would have pthread_mutex_lock block on this mutex forever at the first invocation.

-ulianov

Tuesday, April 7, 2009

When Classes Instantiated as auto Vars on Stack are Evil

I have a threaded C++ app that's using message queue to pass data among threads like so (UML sequence diagram follows):
  Thread A (with response-Q) enqueues request in Thread B's input-Q
Thread A blocks on response-Q empty
Thread B wakes up [input-Q non-empty], dequeues request
Thread B munches on the request from A
Thread B enqueues result in A's response-Q
Thread B blocks on input-Q empty
Thread A wakes up [input-Q non-empty], dequeues response
Thread A goes its merry way.

Nice and easy, eh? And it has worked OK for a while...

You know how it is when one keeps adding code the bugs tend to be shifted on the shelf and rear their nasty heads? It happens in my case that Thread A was the main thread and its response-Q was declared as
Queue respQ("A's response Q");
Now this an auto var that lives on Thread A's stack.

Thread B was doing its job but when it wanted to enqueue the response in Thread A's respQ [it got a pointer to &respQ via a param] it would block in
pthread_mutex_lock()
in libc.

Bummer! I spent three hours writing LOCK/UNLOCK macros in class Queue that would confess who called them and in what thread and I was matching the results (yes, gdb was borked on that ppc target and was useless with inf thr et al.) and I really saw that the Queue instance of Thread A was indeed blocking in
pthread_mutex_lock()
but nobody had locked that mutex before!!

The funny part is that I had that mutex properly initialised in Queue::Queue(); I even changed its attribute to error-checking but it just did not help! That mutex would behave as if uninitialised and containing garbage!

After a while you get bored of this kind of debugging so I changed Thread A's code to read
Queue* respQ = new Queue("A's response Q");
and everything went smooth afterwards.

This yields the following article of faith:
Objects declared as class auto on stack are evil.
-ulianov

Thursday, April 2, 2009

When GCC is Not Smart Enough to Help

Have you noticed in the later versions of gcc that you have added lots of bogus warning-errors that suck the joy out of programming? Well, even with
-Wall -Werror
sometimes it won't help:
class X {
public:
typedef enum {
ERR1,
ERR2
} IOErrorCode;
const char* IOstrerror(IOErrorCode c); // string describing the error
};

struct S {
// ...
X::IOErrorCode status;
};
On another day and in another file I coded:
struct S* res = malloc(sizeof(struct S));
if(! DoSomeIO(res))
printf("IO/Error: %s (%d)\n", \
X::IOErrorCode(res->status), res->status);//(*)
and when I ran the program I would get a segfault at line (*) and GDB was indicating that the stack was partially smashed! Niice!

After scratching my head for half an hour it occured to me that I made a mistake: I coded
IOErrorCode(res->status) // BAAD
instead of
IOstrerror(res->status) // OK
The former is [in C++] a typecast to type IOErrorCode and will cause a crash inside printf().

The latter is a function call.

Ha! Not paying attention to my own code! And I had this sequence in five places handling I/O errors!

This is the most dangerous kind of error as one hits this code path infrequently (sometimes it only happens at a client's site thus driving the client mad).

-ulianov

Tuesday, March 31, 2009

When One Needs to Hide Things in Plain Sight on the WWW

Let's say that a web page needs to show content according to various local sensibilities and that it's hosted by a 3rd party that does not provide server-side scripting. What to do?

It's simple! Javascript+DOM+CSS come to the rescue: simply have a <div> with style="display:none". Store your content there in a plain/text scrambled form.

Then load a JavaScript script from a server that you control as the output of a server-side script.

Decide in your server-side script whether you wish to authorize the user based on the $REMOTE_ADDR to view the content and spit out either the decrypting JavaScript code or some dummy code.

Simple, eh?

The negative side-effect is that the search engines won't index your content. The positive side-effect is that the search engines won't index you content (maybe you don't like other people to use your content for their own purposes, e.g. keywordspy·com) and spammers won't be able to grab e-mail addresses from your pages ;)

-ulianov

Monday, March 30, 2009

When Networking != IPC

To some people it stands firmly to reason that
Networking != IPC
This boggles the mind as they see the only way for two (or more!) application [living on the same machine] to communicate is via SHM and semaphores (aka. "mailbox & lock"). They say it's faster.

It is -- Stevens states in UNP that such an approach is 30% faster than AF_UNIX sockets.

However there are some minor drawbacks in this communication pattern:
a. it cannot be extended across hosts (for sockets it's transparent, endianess non-withstanding)
b. it is only half-duplex (no better than the venerable pipes/FIFOs) so you need two such message queues;
c. there is no notification of a peer's death (TCP/IP sockets can send keep-alives and these can be tuned);
d. the notification of a pending message is wasteful and at best awkward: one needs to dedicate a thread on the receiving side to block on the semaphore; this can be mitigated with pthread_cond_timedwait but it cannot be mixed with select-ing on sockets and you'll end up with a thread babysitting the mailboxes;
e. if there is more than one receiver process then the receivers must hunt down the messages addressed to them in the mailboxes; worse if one of the receivers gets bored and ends execution its messages are cleaned up by no-one (can I smell a DoS?);
f. the data that can be transported via mailboxes is limited by the size of the mailboxes and one may have to resort to fragmentation -- things can get very hairy at this point;
g. this pattern is not observable, i.e. one cannot use tcpdump to look at the packets going to and fro; one must build custom tools to observe the data flow;
h. depending on the type of SHM used (e.g. SysV SHM) the mailboxes and their contents may persist after the death of all the parties involved; this can be good if one wanted that of very bad if the server process must clean up at start-up.

To me such a socket-phobia is unexplainable (that is thinking with a UN*X programmer's mind). I do recall tho the contortions and the horrendous API and sad performance of Winsock. Yet have I mentioned that this whole mailbox/lock brouhaha has to happen on Linux?

-ulianov

Saturday, March 28, 2009

A Nightmarish Fantasy

Please stand by...


Friday, October 10, 2008

Unintended Spam Consequences to Webapps

Some time ago I wrote a webapp that would convert long URLs into fixed-length ("tiny") ones. Yes there is tinyurl.com out there but I keep forgetting the short URLs.

Thus it was easier for me to write it from scratch and be able to look at the database backend whenever I wish. The application has a verification code embedded in an scrambled image so I am sure that only humans can generate the short URLs.

I made the mistake to post its location on my home page (which is being prowled by Google and other bots).

It just happens that some scammers/spammers have used it today to spam people: they made a redirection to an image containing an ad for counterfeit watches. Now I get lots of hits from people who read their e-mails (and I see scores of webmail providers being hit by this spam).

Naturally I changed the stored links so people now get a logo of a US law enforcement agency and when they click the hook & bait link in their e-mails they end up at the website of the said agency.

It pisses me off that I get one hit every other second and my web server is getting a high load average as I installed the Perl webapp as CGI scripts instead of using mod_perl.

This will eventually force me to upgrade my ancient system (both as hardware and OS version) and put mod_perl in place.

-ulianov

Saturday, August 30, 2008

Log messages to dmesg/kernel buffer from user-space

From time to time a buddy of mine who's not a professional software developer asks questions that are interesting albeit they go against the grain of UN*X programming.

This time he asked whether printk is accessible to user-mode applications. As per entry.S it does not but the guy is stubborn so I got bored explaining the impossibility I got to work and wrote a Linux kernel module that allows him to fulfill his fantasy.

The technical article is here: http://spamchk.no-ip.org/uprintk.html. The source code is here.

-ulianov

Friday, August 15, 2008

Fooling Around with jiffies

Some time ago a QA buddy asked whether the Linux uptime can be changed on the fly in order to study the behaviour of a closed-source application that would only exhibit a certain bug after two weeks of running.

I looked at sys_gettimeofday (Linux kernel 2.4) and learned that it depends on jiffies. So I wrote a small module that adjusts jiffies at will.

The technical article is here: http://spamchk.no-ip.org/kernuptime.html. The source code is here.

-ulianov

Thursday, July 3, 2008

Go easy on "rm -fr"

Here I am trying to improve and extend an old script that read:
#!/bin/bash

[ ! -d ~/stage ] && exit 1

(cd ~/stage; rm -vfr *)

...
The new version read:
#!/bin/bash

STAGE=$HOME/stage

[ ! -d "$STAGE" ] && exit 1

(cd $STAGE; rm -vfr *)

...
Guess what? I managed to wipe out a good chunk of my $HOME!!

The explanation is that (cd $STAGE; rm -vfr *) executes in a new (child) copy of the shell [I wrote the code like this so I don't have to cd back after the removal].

The STAGE environment variable is set in the parent shell but is not inherited by the child shell because I did not export it so what got executed was in fact (cd; rm -vfr *) [just saying "cd" in bash gets you back to your $HOME].

The proper way to do this was:
#!/bin/bash

STAGE=$HOME/stage; export STAGE;

[ ! -d "$STAGE" ] && exit 1

(cd $STAGE; rm -vf *)

...
Note that I added the export STAGE statement and dropped the "-r" from the remove command as I really had no sub-directories in ~/stage

-ulianov

Tuesday, June 24, 2008

Cargo-Cult Programming

Here is an example I encountered of a chronic Cargo-Cult Programming affliction in a Junior engineer -- all code he wrote read:
if(CONSTANT == var) {
// something
} else {
// something else
}
The reason behind this is a fake "defensive programming" strategy as he protected himself against
if(var = CONSTANT) { ... }
which is does not do a comparison, it does an assignment which sometimes fails (when CONSTANT=0).

This is just bad coding. First, it's an eye-sore, second this programmer is not aware that new GCC iterations warn about this when using -Wall and even halt compilation when used in conjunction with -Werror

The fact that he was very stubborn did not help either.

-ulianov

Monday, June 23, 2008

The Forgotten Junk in libc

libc is a patchwork of stuff: syscalls and helper functions that got set in stone aeons ago, warts and all. During the bad times unthinking people stuck in it junk such as:
  • atoi() -- does not return errors;
  • gets() -- has no way of knowing the length of the buffer it populates;
  • strcpy() -- the all time favourite way of causing a crash: if the src pointer is junk and has no '\0' then the buffer pointed by dst will be filled beyond its boundaries with garbage;
  • ditto strcat();
  • sprintf() -- what if the stuff you wish to print into the buffer exceeds the buffer's length?
  • heck, even strlen(NULL) will crash;
  • strtok() -- this one is just evil.
DO NOT USE THESE CALLS! I cannot count how many time I had to look for crashes and refactor code using this garbage.

-ulianov

Friday, June 20, 2008

Why atoi() is NOT Your Friend

atoi() is a hold-over of the bad times when people put junk such as strcpy() and gets() into libc.

It is unsuitable as it has no way of returning and error, e.g.
atoi("adhgsjgdas") = 0
Alas many people still use it.

Please, please use instead:
if(sscanf(str, "%d", &int_var) != 1) {
// handle the error!!!
}
-ulianov

Thursday, June 19, 2008

Passing Params thru Registers Woes on x86

One year ago I was asked to look into a problem some colleagues had with a networking framework that lived as a set of Linux kernel modules.

The problem they saw was that when certain framework functions were called a parameter [which was a pointer] contained garbage.

They had the hairy idea to start using "-mregparm=3" to compile the kernel altho until them we lived happily with stack-based calls.

I looked at the code and the makefile and here is how gcc was invoked:
gcc -ggdb -O2 -pipe \
-mregparm=3 \
-Wall -Wstrict-prototypes -Wno-trigraphs \
-fno-strict-aliasing -fno-common \
-fomit-frame-pointer \
-mpreferred-stack-boundary=2 \
-march=pentium code.c
and here is how the offending code looked like [not a verbatim copy]:
#include <stdio.h>
#include <stdlib.h>
#ifdef FORCE_STDARG
#include <stdarg.h>
#endif

#define ENOMEM -2
#define EBADSLT -3
#define ENODEV 0

typedef int (*func_t)(void*, ...);

struct _S2; // forward declaration
struct _S1 {
char filler1[3];
func_t f1;
int (*f2)(long, long, struct _S2*, long);
};
struct _S2 {
char filler1[13];
long filler2;
struct _S1* s;
char filler3[7];
};

struct _S1 g_S1;
struct _S2 g_S2;

#ifdef FORCE_STDARG
int f1(struct _S1* s, ...)
#else
int f1(struct _S1* s)
#endif
{
#ifdef FORCE_STDARG
va_list ap;
va_start(ap, s);
#endif
if(s != &g_S1)
return -1;
#ifdef FORCE_STDARG
va_end(ap);
#endif
return 1;
}
int f2(long i1, long i2, struct _S2* s, long i3)
{
if(s != &g_S2)
return -1;
return 1;
}
int main()
{
g_S1.filler1[0] = 'A';
g_S1.f1 = (func_t)f1;
g_S1.f2 = f2;

g_S2.filler2 = 13;
g_S2.s = &g_S1;

if(g_S1.f1(&g_S1) < 0)
return -ENOMEM;
if(g_S2.s->f2(1, 2, &g_S2, 3) < 0)
return -EBADSLT;

return -ENODEV;
}
I noticed comparing disassembled code (objdump -dSl code.o) that the call setup for
g_S1.f1(&g_S1);
was the same regardless of "-mregparm" -- this is because the compiler applies the template
typedef int (*func_t)(void*, ...);
which forces it to put the arguments on the stack.

However the declaration
int f1(struct _S1* s);
in connection with "-mregparm=3" has f1() looking for its first argument in %eax which contains some garbage!!

Hint: compile the code with -DFORCE_STDARG and without and see the difference in execution.

The moral is two-fold:
1. if you use "..." in a function prototype then also use it in the function implementation!!;
2. in the kernel passing function arguments thru registers will yield little or no gain in execution speed (as only the leaf functions will fully benefit from the stack-free operation).

-ulianov

Tuesday, June 17, 2008

Bad Idiom: Using Strings on Stack

Here's what I've seen recently:
#include <stdexcept>

Class::Class(int arg)
{
...
if(/* error */) {
// original code
//throw std::runtime_error("Class::Class: Wrong argument!");

// later addition
char str[100];
sprintf(str, "Class::Class: argument %d is too big", arg);
throw std::runtime_error(str);
}
...
}
int main()
{
try { c = new Class(100000); } // say it's allocating memory
catch(std::runtime_error& e) {
printf("Got an error: %s\n", e.what());
}
}
The code sins in many ways:
1. the catch() is wrong as the code does not throw a reference but an object!
2. the throw() is poisoned: it throws a string from the stack which will go out of scope when the constructor is exited; the application won't crash immediately [the stack pages are not unmapped];
3. text of the exception will be junk (and may not have a \0 terminator) and attempting to print it may crash the app in funny and random ways.

The person who butchered the code and used the class hierarchy suffers from the cargo cult programming syndrome because:
1. did not pay attention to original code;
2. did not bother to read the Doxygen documentation of the original classes;
3. does not understand the life cycle of an auto var allocated on stack;
4. does not analyse all the implications of using and changing other people's code.

-ulianov

Monday, June 16, 2008

Compressed Output from Perl CGI Scripts

You know how you say in PHP
<?php ob_start("ob_gzhandler"); ?>
I wanted the same for a Perl CGI script so I dug a bit on the Net and here's the code I came up with:
use strict;
use Compress::Zlib qw(gzopen);

print <<EOF;
Content-Type: text/html
Content-Encoding: gzip

EOF

my $print;
binmode STDOUT;
{ # closure
my $gz = gzopen(\*STDOUT, "wb");
$print = sub { $gz->gzwrite(@_) };
}

$print->("Some content....");
Mind you I am not using CGI.pm as it's a memory hog and in this particular script I did not have to parse form variables.

On another note importing just what you need from a Perl module [i.e. qw(gzopen)] is a great way to cut down on memory consumption.

-ulianov

Saturday, June 14, 2008

Smashed Stack in a Multithreaded Application

One of the most annoying thing in debugging an app is to have it crash and GDB show garbage for the topmost 3-4 frames.

There is not much to do other than to do a diff between your current code (that is crap!) and the last known working version that you have in revision control (I hope you do have that!).

-ulianov

Friday, June 13, 2008

The Fine Distinction Between a Pointer and an Array

I keep finding crashes cause by people not comprehending what C pointers and arrays really are.

Definitions:
1. array = some auto variable that was declared as a collection of objects:
char x[100];
be it as a global or even worse on stack;
2. pointer = something that has been malloc'ed or something that holds the address of another object:
char* x1 = (char*)malloc(100);
or
char c;
char* x2 = &c;
A pointer may live as a global or on stack, regardless of this its memory footprint is 4 bytes (on a 32-bit architecture).

To many people pointers are arrays and arrays are pointers. True, but not all the time.

To simplify the discussion let us assume that a segfault occurs when we go beyond the boundaries of a buffer regardless of access type (read/write), its size and alignment.

Let's assume you populated x and x1 with a 99-byte message and you say:
write(STDOUT_FILENO, &x, 99); // mind the \0!
This is correct.

If you say:
write(STDOUT_FILENO, &x[0], 99); // mind the \0!
This is also correct.

If you say:
write(STDOUT_FILENO, x1, 99); // mind the \0!
This is correct.

If you say:
write(STDOUT_FILENO, &x1, 99); // mind the \0!
This is a SEGFAULT because you will try to access (99-4) bytes on the stack that follow x1 and there is nothing mapped there and you program will crash and burn!

The explanation is that
x == &x == &x[0]
but
x1 != &x1
as x1 means the allocated memory buffer to which x1 points to whereas &x1 means the address in memory where the pointer x1 lives!!!

In real life it gets even worse if x1 lives on the stack and you write to &x1 -- here you put garbage on stack and you may go past the red page. If there are some other auto vars on stack after it and they happen to be pointers then you have a recipe for a clusterfuck.

If x1 is a global then you will corrupt other globals that live after x1. This is even more fun!

-ulianov

Unlock of Mutex Failed

This was the one of the most puzzling problems I faced. The Teja application we had was failing once in a blue moon printing the message "unlock of mutex failed".

Looking at he code I saw that pthread_unlock_mutex returned an error which should never happen.

I spent a month writing a wrapper for the pthread/mutex library (a mutex is in this particular case a 64-byte memory area, futex was not called).

I put signatures before and after the mutex, checked signatures & various mutex fields before entering the true mutex call to no avail.

It was a random memory corruption that I solved elsewhere so this error ceased to happen.

-ulianov

Thursday, June 12, 2008

Re-Inventing the Wheel is Evil

Once I had to debug a program that ran in the Teja application framework for IXP 1200. We experienced random crashes at sites that were using a Walled Garden wireless authentication method.

By another name the Captive Portal consist in a TCP stream hijacker. The intercepted stream is HTTP. The component that handles this functionality simply pretends to be any HTTP destination, analyses the request and sends a crafted HTTP/307 redirection message in which the original URL is encoded as a GET parameter.

The Teja module that implemented the redirection got as an input an Ethernet frame (TCP reassembly was not performed but that was not a practical problem as most people simply go to http://yahoo.com, i.e. use short URLs that fit within one frame).

The data structure that enveloped the Ethernet frame was called Packet and it looked like:

struct Packet {
char buffer[1500];
short start;
short end;
};
To an astute reader this resembles an skbuff.

The code was parsing the input request and sending back a 307 that looked like "Location: http://auth.host/login.php?dest=OriginalURL"

The response was kept in a 6000-byte buffer which was large enough but here's what they were doing:

memcpy(packet.buffer+packet.end, response, response_len);
packet.end += response_len;
See anything wrong? If response_len>1500 then the memcpy thrashes packet.end, then one adds something to it resulting in a bigger junk. How about the junk in packet.start?

The Packets were malloc'ed [thus living in a pool of objects and having neighbours] so this had the potential (if len>1508) to corrupt the "secret stuff" the GNU allocator puts before each malloc'ed pointer. This will cause a happy crash when you want to free that pointer!

Later on the packet was copied into a hardware buffer that lived in IXP DRAM mapped in the Arm CPU address space like so:

memcpy(pHwBuf->buffer, \
packet.buffer+packet.start, \
(packet.end-packet.start));
Here you could copy garbage from a garbage address (packet.buffer+packet.start) [no crash if the memory was mapped] that had a garbage length (packet.end-packet.start) thus thrashing memory left and right in unexpected places.

Murphy's Laws dictate that the memory belonged to other threads of execution and that these threads will crash at another time in a such a way to turn you green.

In order to track that I had to change the Packet like so:

struct Packet {
unsigned sig1 = 0xdeadbeef; // initialised when
// allocating a Packet
char buffer[1500];
unsigned sig2 = 0xfeedabba;
short start;
short end;
unsigned sig3 = 0xdeafabba;
};
and check the signatures after each Packet operation. Took me only a month and one very pissed customer to fix this (not my code).

-ulianov

P.S. The Linux skbuff infrastructure comes with a lot of accessory functions that prevent this kind of crap from happening by Oops'ing.

Tuesday, June 3, 2008

Shooting your Own Foot and Jumping on it Afterwards

While I was developing my anti-recruiter Sendmail milter I was relying on nohup(1) to collect stderr output. Each time I would change code I would re-start the milter and wipe nohup.out like so "> nohup.out".

Then I changed code to redirect stderr to "milter.err" and at every restart I would do "> milter.err". One night at midnight I did not pay attention to the BASH-completion and wiped the code itself!

Now if you delete a file on an ext2 filesystem you can recover it using some utilities. But wiping the content of the file severs the relation between the i-node and its data blocks. debugfs ain't no help.

Moral: I had to "dd if=/dev/hda1 of=/storage/hda1.saved" and find the source code block by block by searching code patterns I remembered in hda1.saved. Trust me, file blocks are not contiguous on-disk!

I managed to piece together everything but the top of the file where the Perl use statements live. No biggie but I also skipped a small function and the code only complained at run time.

Took me three hours in the middle of the night and a huge head ache.

-ulianov

Thrashing the Kernel (Linux 2.4.x)

Today a co-worker told me that a library I've been writing is causing a Segfault but he could not give me an IP. Hmm. The library calls a custom driver and tries to pull data from the driver via ioctl's.

Then I used the lib in a utility and presto! I hosed the PPC 450. Funny is that the utility executes picture-perfect, does not crash, the kernel does not panic(), but everything executed after that segfaults.

Now that's artful.

Jun 4 update: I traced the code by #if 0'ing code and the problem is in the kernel driver.

Jun 10 update: The root cause was that they changed the way they were DMA-ing data into my user buffer that got mapped into the kernel space. As I was malloc-ing the buffer and not touching it the associated hardware pages were not actually mapped and that was confusing the PPC DMA engine.

memset-ing the buffer to zero in user space to fault the pages in or forcing the mapping of pages in kernel mode fixed the problem.

-ulianov