When Classes Instantiated as auto Vars on Stack are Evil

I have a threaded C++ app that's using message queue to pass data among threads like so (UML sequence diagram follows):

  Thread A (with response-Q) enqueues request in Thread B's input-Q
  Thread A blocks on response-Q empty
    Thread B wakes up [input-Q non-empty], dequeues request
    Thread B munches on the request from A
    Thread B enqueues result in A's response-Q
    Thread B blocks on input-Q empty
  Thread A wakes up [input-Q non-empty], dequeues response
  Thread A goes its merry way.

Nice and easy, eh? And it has worked OK for a while...

You know how it is when one keeps adding code the bugs tend to be shifted on the shelf and rear their nasty heads? It happens in my case that Thread A was the main thread and its response-Q was declared as

Queue respQ("A's response Q");

Now this an auto var that lives on Thread A's stack.

Thread B was doing its job but when it wanted to enqueue the response in Thread A's respQ [it got a pointer to &respQ via a param] it would block in

pthread_mutex_lock()

in libc.

Bummer! I spent three hours writing LOCK/UNLOCK macros in class Queue that would confess who called them and in what thread and I was matching the results (yes, gdb was borked on that ppc target and was useless with inf thr et al.) and I really saw that the Queue instance of Thread A was indeed blocking in

pthread_mutex_lock()

but nobody had locked that mutex before!!

The funny part is that I had that mutex properly initialised in Queue::Queue(); I even changed its attribute to error-checking but it just did not help! That mutex would behave as if uninitialised and containing garbage!

After a while you get bored of this kind of debugging so I changed Thread A's code to read

Queue* respQ = new Queue("A's response Q");

and everything went smooth afterwards.

This yields the following article of faith:

Objects declared as class auto on stack are evil.

-ulianov

When GCC is Not Smart Enough to Help

Have you noticed in the later versions of gcc that you have added lots of bogus warning-errors that suck the joy out of programming? Well, even with

-Wall -Werror

sometimes it won't help:

class X {
public:
  typedef enum {
    ERR1,
    ERR2
  } IOErrorCode;
  const char* IOstrerror(IOErrorCode c); // string describing the error
};

struct S {
 // ...
 X::IOErrorCode status;
};

On another day and in another file I coded:

struct S* res = malloc(sizeof(struct S));
if(! DoSomeIO(res))
  printf("IO/Error: %s (%d)\n", \
         X::IOErrorCode(res->status), res->status);//(*)

and when I ran the program I would get a segfault at line (*) and GDB was indicating that the stack was partially smashed! Niice!

After scratching my head for half an hour it occured to me that I made a mistake: I coded

IOErrorCode(res->status) // BAAD

instead of

IOstrerror(res->status) // OK

The former is [in C++] a typecast to type IOErrorCode and will cause a crash inside printf().

The latter is a function call.

Ha! Not paying attention to my own code! And I had this sequence in five places handling I/O errors!

This is the most dangerous kind of error as one hits this code path infrequently (sometimes it only happens at a client's site thus driving the client mad).

-ulianov

When One Needs to Hide Things in Plain Sight on the WWW

Let's say that a web page needs to show content according to various local sensibilities and that it's hosted by a 3rd party that does not provide server-side scripting. What to do?

It's simple! Javascript+DOM+CSS come to the rescue: simply have a <div> with style="display:none". Store your content there in a plain/text scrambled form.

Then load a JavaScript script from a server that you control as the output of a server-side script.

Decide in your server-side script whether you wish to authorize the user based on the $REMOTE_ADDR to view the content and spit out either the decrypting JavaScript code or some dummy code.

Simple, eh?

The negative side-effect is that the search engines won't index your content. The positive side-effect is that the search engines won't index you content (maybe you don't like other people to use your content for their own purposes, e.g. keywordspy·com) and spammers won't be able to grab e-mail addresses from your pages ;)

-ulianov

When Networking != IPC

To some people it stands firmly to reason that

Networking != IPC

This boggles the mind as they see the only way for two (or more!) application [living on the same machine] to communicate is via SHM and semaphores (aka. "mailbox & lock"). They say it's faster.

It is -- Stevens states in UNP that such an approach is 30% faster than AF_UNIX sockets.

However there are some minor drawbacks in this communication pattern:
a. it cannot be extended across hosts (for sockets it's transparent, endianess non-withstanding)
b. it is only half-duplex (no better than the venerable pipes/FIFOs) so you need two such message queues;
c. there is no notification of a peer's death (TCP/IP sockets can send keep-alives and these can be tuned);
d. the notification of a pending message is wasteful and at best awkward: one needs to dedicate a thread on the receiving side to block on the semaphore; this can be mitigated with pthread_cond_timedwait but it cannot be mixed with select-ing on sockets and you'll end up with a thread babysitting the mailboxes;
e. if there is more than one receiver process then the receivers must hunt down the messages addressed to them in the mailboxes; worse if one of the receivers gets bored and ends execution its messages are cleaned up by no-one (can I smell a DoS?);
f. the data that can be transported via mailboxes is limited by the size of the mailboxes and one may have to resort to fragmentation -- things can get very hairy at this point;
g. this pattern is not observable, i.e. one cannot use tcpdump to look at the packets going to and fro; one must build custom tools to observe the data flow;
h. depending on the type of SHM used (e.g. SysV SHM) the mailboxes and their contents may persist after the death of all the parties involved; this can be good if one wanted that of very bad if the server process must clean up at start-up.

To me such a socket-phobia is unexplainable (that is thinking with a UN*X programmer's mind). I do recall tho the contortions and the horrendous API and sad performance of Winsock. Yet have I mentioned that this whole mailbox/lock brouhaha has to happen on Linux?

-ulianov

A Nightmarish Fantasy

Please stand by...

V unq na LZ pung jvgu n ohqql bs zvar sebz gur Havirefvgl lrnef.

Guvf thl vf n uboolvfg cebtenzzre (ur'f ohvyqvat pbbyvat gbjref nf n qnl wbo). Ng svefg ur jnf n Yvahk naq FhaBF raguhfvnfg ohg guvatf punatr: ur vf abj nggenpgrq zber naq zber gb gur Qnex Fvqr bs cebtenzzvat, ur'f cebsrffvat naq haqlvat nqzvengvba naq raguhfvnfz sbe P# naq .ARG.

(Zlfrys V jnag gb unir abguvat gb qb jvgu gur Qnex Fvqr bs gur vaqhfgel naq ubyq gur ivrj gung Wnin vf tbbq naq gung gur PYE naq P# ner vasrevbe evc-bssf bs n dhnyvgl abg hayvxr gung bs Puvarfr snxr jngpurf.)

V gevrq gb fgrre uvz gbjneq gur yvtug (Wnin-tbbq, Wnin+FJG orggre nf lbh pna ohvyq qrprag pebff-cyngsbez THVf) ohg ur jbhyq abg urrq zl nqivpr. V guvax gung ur unf na hafgnoyr fgernx tvira uvf erpbeq:
n) unpxf TGX jvqtrgf naq vagreanyf [TGX vf lrg nabgure fho-cne evc-bss ba Zbgvs naq Dg];
o) vf n tebhcvr sbe P# naq Zbab (juvpu vf gur Gebwna Ubefr bs gur Qnex Fvqr guehfg vagb gur BFF pbzzhavgl ol gurve urapuzna -- lbh xabj uvz, vg'f gur fnzr zvavba bs Qnexarff jub hayrnfurq hcba hf TGX);
p) guvaxf gung NYY THVf zhfg or pbirerq va tenqvragf, fcvaavat guvatf naq rlr pnaql [ohg ur erwrpgf gur BF K jnlf];
q) unf wbvarq gur Frpg bs Zvpebxrearyf naq ur'f n sreirag oryvrire guvaxvat gur guevpr-snvyher bs UHEQ gb or npghnyyl n cbfvgvir qrirybczrag;
r) unf n qrenatrq ivfvba (naq unys-onxrq vqrn) bs n gur hygvzngr fbsgjner fgnpx juvpu vf UHEQ xreary->Zbab->P# THV jvgu ybgf naq ybgf naq ybgf bs oyvat-oyvat.

Uvf yngrfg syvtug bs snapl vf gb cnl zr gb cbeg nyy Yvahk qeviref gb UHEQ guhf znxvat vg (UHEQ gung vf) npprcgnoyr sbe gur znffrf. V ubcr ur trgf gb zrrg EZF fbzr qnl.

Nyy guvf vf shaal ohg ur'f nyfb gur xvaq bs crefba jub jbhyq bhgfuvar Fulybpx fb V erpxba vg jbhyq or naq hajvfr guvat gb qb. Ur'f n oryvrire va gur erag-n-pbqre fpurzr. V hetrq uvz gb gel vg.

Zne 31 hcqngr: urer vf uvf erfcbafr nobhg gur TGX nssnve:Lbhe gjvfgrq negvfgvp frafr qbrf abg qvfprea gur htyvarff bs n ornfg yvxr Zbgvs sebz gur ryrtnag ornhgvrf fhpu nf TGX.
Gur cebtenzzvat zbqry bs TGX vf nf jbaqreshy nf vg vf ubeevoyr.
Gung vf jul crbcyr ner fb nggenpgrq gb vg. Vg vf irel rnfl gb perngr ynathntr ovaqvatf.
Vgf havdhr qrfvta bs cynpvat jvqtrgf va obkrf nyybjf n irel ryrtnag erfvmr bs jvaqbjf jvgubhg nqqvgvbany unffyr sbe cebtenzzref.
Naq vgf orfg fryyvat cbvag: vgf znva ynathntr cebtenzzvat vf P!! naq lrg fbzr zber nobhg UHEQ: Guvf jbhyq or n fznyy pbagevohgvba gb BFF pbzzhavgl. Vg pbhyq zrna znff-nqbcgvba gb UHEQ , vs vg jbhyq unir fhccbeg sbe Yvahk qeviref.-hyvnabi

Unintended Spam Consequences to Webapps

Some time ago I wrote a webapp that would convert long URLs into fixed-length ("tiny") ones. Yes there is tinyurl.com out there but I keep forgetting the short URLs.

Thus it was easier for me to write it from scratch and be able to look at the database backend whenever I wish. The application has a verification code embedded in an scrambled image so I am sure that only humans can generate the short URLs.

I made the mistake to post its location on my home page (which is being prowled by Google and other bots).

It just happens that some scammers/spammers have used it today to spam people: they made a redirection to an image containing an ad for counterfeit watches. Now I get lots of hits from people who read their e-mails (and I see scores of webmail providers being hit by this spam).

Naturally I changed the stored links so people now get a logo of a US law enforcement agency and when they click the hook & bait link in their e-mails they end up at the website of the said agency.

It pisses me off that I get one hit every other second and my web server is getting a high load average as I installed the Perl webapp as CGI scripts instead of using mod_perl.

This will eventually force me to upgrade my ancient system (both as hardware and OS version) and put mod_perl in place.

-ulianov

Log messages to dmesg/kernel buffer from user-space

From time to time a buddy of mine who's not a professional software developer asks questions that are interesting albeit they go against the grain of UN*X programming.

This time he asked whether printk is accessible to user-mode applications. As per entry.S it does not but the guy is stubborn so I got bored explaining the impossibility I got to work and wrote a Linux kernel module that allows him to fulfill his fantasy.

The technical article is here: http://spamchk.no-ip.org/uprintk.html. The source code is here.

-ulianov

Fooling Around with jiffies

Some time ago a QA buddy asked whether the Linux uptime can be changed on the fly in order to study the behaviour of a closed-source application that would only exhibit a certain bug after two weeks of running.

I looked at sys_gettimeofday (Linux kernel 2.4) and learned that it depends on jiffies. So I wrote a small module that adjusts jiffies at will.

The technical article is here: http://spamchk.no-ip.org/kernuptime.html. The source code is here.

-ulianov

Go easy on "rm -fr"

Here I am trying to improve and extend an old script that read:

#!/bin/bash

[ ! -d ~/stage ] && exit 1

(cd ~/stage; rm -vfr *)

...

The new version read:

#!/bin/bash

STAGE=$HOME/stage

[ ! -d "$STAGE" ] && exit 1

(cd $STAGE; rm -vfr *)

...

Guess what? I managed to wipe out a good chunk of my $HOME!!

The explanation is that (cd $STAGE; rm -vfr *) executes in a new (child) copy of the shell [I wrote the code like this so I don't have to cd back after the removal].

The STAGE environment variable is set in the parent shell but is not inherited by the child shell because I did not export it so what got executed was in fact (cd; rm -vfr *) [just saying "cd" in bash gets you back to your $HOME].

The proper way to do this was:

#!/bin/bash

STAGE=$HOME/stage; export STAGE;

[ ! -d "$STAGE" ] && exit 1

(cd $STAGE; rm -vf *)

...

Note that I added the export STAGE statement and dropped the "-r" from the remove command as I really had no sub-directories in ~/stage

-ulianov

Cargo-Cult Programming

Here is an example I encountered of a chronic Cargo-Cult Programming affliction in a Junior engineer -- all code he wrote read:

if(CONSTANT == var) {
  // something
} else {
  // something else
}

The reason behind this is a fake "defensive programming" strategy as he protected himself against

if(var = CONSTANT) { ... }

which is does not do a comparison, it does an assignment which sometimes fails (when CONSTANT=0).

This is just bad coding. First, it's an eye-sore, second this programmer is not aware that new GCC iterations warn about this when using -Wall and even halt compilation when used in conjunction with -Werror

The fact that he was very stubborn did not help either.

-ulianov

The Forgotten Junk in libc

libc is a patchwork of stuff: syscalls and helper functions that got set in stone aeons ago, warts and all. During the bad times unthinking people stuck in it junk such as:

• atoi() -- does not return errors;
  
• gets() -- has no way of knowing the length of the buffer it populates;
  
• strcpy() -- the all time favourite way of causing a crash: if the src pointer is junk and has no '\0' then the buffer pointed by dst will be filled beyond its boundaries with garbage;
  
• ditto strcat();
  
• sprintf() -- what if the stuff you wish to print into the buffer exceeds the buffer's length?
  
• heck, even strlen(NULL) will crash;
  
• strtok() -- this one is just evil.
  

DO NOT USE THESE CALLS! I cannot count how many time I had to look for crashes and refactor code using this garbage.

-ulianov

Why atoi() is NOT Your Friend

atoi() is a hold-over of the bad times when people put junk such as strcpy() and gets() into libc.

It is unsuitable as it has no way of returning and error, e.g.

atoi("adhgsjgdas") = 0

Alas many people still use it.

Please, please use instead:

if(sscanf(str, "%d", &int_var) != 1) {
   // handle the error!!!
}

-ulianov

Passing Params thru Registers Woes on x86

One year ago I was asked to look into a problem some colleagues had with a networking framework that lived as a set of Linux kernel modules.

The problem they saw was that when certain framework functions were called a parameter [which was a pointer] contained garbage.

They had the hairy idea to start using "-mregparm=3" to compile the kernel altho until them we lived happily with stack-based calls.

I looked at the code and the makefile and here is how gcc was invoked:

gcc -ggdb -O2 -pipe \
        -mregparm=3 \
        -Wall -Wstrict-prototypes -Wno-trigraphs \
        -fno-strict-aliasing -fno-common \
        -fomit-frame-pointer \
        -mpreferred-stack-boundary=2 \
        -march=pentium code.c

and here is how the offending code looked like [not a verbatim copy]:

#include <stdio.h>
#include <stdlib.h>
#ifdef FORCE_STDARG
  #include <stdarg.h>
#endif

#define ENOMEM  -2
#define EBADSLT -3
#define ENODEV  0

typedef int (*func_t)(void*, ...);

struct _S2; // forward declaration
struct _S1 {
     char filler1[3];
     func_t f1;
     int (*f2)(long, long, struct _S2*, long);
};
struct _S2 {
     char filler1[13];
     long filler2;
     struct _S1* s;
     char filler3[7];
};

struct _S1 g_S1;
struct _S2 g_S2;

#ifdef FORCE_STDARG
int f1(struct _S1* s, ...)
#else
int f1(struct _S1* s)
#endif
{
#ifdef FORCE_STDARG
    va_list ap;
    va_start(ap, s);
#endif
    if(s != &g_S1)
     return -1;
#ifdef FORCE_STDARG
    va_end(ap);
#endif
    return 1;
}
int f2(long i1, long i2, struct _S2* s, long i3)
{
    if(s != &g_S2)
     return -1;
    return 1;
}
int main()
{
    g_S1.filler1[0] = 'A';
    g_S1.f1 = (func_t)f1;
    g_S1.f2 = f2;

    g_S2.filler2 = 13;
    g_S2.s = &g_S1;

    if(g_S1.f1(&g_S1) < 0)
         return -ENOMEM;
    if(g_S2.s->f2(1, 2, &g_S2, 3) < 0)
         return -EBADSLT;

    return -ENODEV;
}

I noticed comparing disassembled code (objdump -dSl code.o) that the call setup for

g_S1.f1(&g_S1);

was the same regardless of "-mregparm" -- this is because the compiler applies the template

typedef int (*func_t)(void*, ...);

which forces it to put the arguments on the stack.

However the declaration

int f1(struct _S1* s);

in connection with "-mregparm=3" has f1() looking for its first argument in %eax which contains some garbage!!

Hint: compile the code with -DFORCE_STDARG and without and see the difference in execution.

The moral is two-fold:
1. if you use "..." in a function prototype then also use it in the function implementation!!;
2. in the kernel passing function arguments thru registers will yield little or no gain in execution speed (as only the leaf functions will fully benefit from the stack-free operation).

-ulianov

Bad Idiom: Using Strings on Stack

Here's what I've seen recently:

#include <stdexcept>

Class::Class(int arg)
{
  ...
  if(/* error */) {
    // original code
    //throw std::runtime_error("Class::Class: Wrong argument!");

    // later addition
    char str[100];
    sprintf(str, "Class::Class: argument %d is too big", arg);
    throw std::runtime_error(str);
  }
  ...
}
int main()
{
  try { c = new Class(100000); } // say it's allocating memory
  catch(std::runtime_error& e) {
    printf("Got an error: %s\n", e.what());
  }
}

The code sins in many ways:
1. the catch() is wrong as the code does not throw a reference but an object!
2. the throw() is poisoned: it throws a string from the stack which will go out of scope when the constructor is exited; the application won't crash immediately [the stack pages are not unmapped];
3. text of the exception will be junk (and may not have a \0 terminator) and attempting to print it may crash the app in funny and random ways.

The person who butchered the code and used the class hierarchy suffers from the cargo cult programming syndrome because:
1. did not pay attention to original code;
2. did not bother to read the Doxygen documentation of the original classes;
3. does not understand the life cycle of an auto var allocated on stack;
4. does not analyse all the implications of using and changing other people's code.

-ulianov

Compressed Output from Perl CGI Scripts

You know how you say in PHP

<?php ob_start("ob_gzhandler"); ?>

I wanted the same for a Perl CGI script so I dug a bit on the Net and here's the code I came up with:

use strict;
use Compress::Zlib qw(gzopen);

print <<EOF;
Content-Type: text/html
Content-Encoding: gzip

EOF

my $print;
binmode STDOUT;
{ # closure
  my $gz = gzopen(\*STDOUT, "wb");
  $print = sub { $gz->gzwrite(@_) };
}

$print->("Some content....");

Mind you I am not using CGI.pm as it's a memory hog and in this particular script I did not have to parse form variables.

On another note importing just what you need from a Perl module [i.e. qw(gzopen)] is a great way to cut down on memory consumption.

-ulianov

Smashed Stack in a Multithreaded Application

One of the most annoying thing in debugging an app is to have it crash and GDB show garbage for the topmost 3-4 frames.

There is not much to do other than to do a diff between your current code (that is crap!) and the last known working version that you have in revision control (I hope you do have that!).

-ulianov

The Fine Distinction Between a Pointer and an Array

I keep finding crashes cause by people not comprehending what C pointers and arrays really are.

Definitions:
1. array = some auto variable that was declared as a collection of objects:

char x[100];

be it as a global or even worse on stack;
2. pointer = something that has been malloc'ed or something that holds the address of another object:

char* x1 = (char*)malloc(100);
or
char c;
char* x2 = &c;

A pointer may live as a global or on stack, regardless of this its memory footprint is 4 bytes (on a 32-bit architecture).

To many people pointers are arrays and arrays are pointers. True, but not all the time.

To simplify the discussion let us assume that a segfault occurs when we go beyond the boundaries of a buffer regardless of access type (read/write), its size and alignment.

Let's assume you populated x and x1 with a 99-byte message and you say:

write(STDOUT_FILENO, &x, 99); // mind the \0!

This is correct.

If you say:

write(STDOUT_FILENO, &x[0], 99); // mind the \0!

This is also correct.

If you say:

write(STDOUT_FILENO, x1, 99); // mind the \0!

This is correct.

If you say:

write(STDOUT_FILENO, &x1, 99); // mind the \0!

This is a SEGFAULT because you will try to access (99-4) bytes on the stack that follow x1 and there is nothing mapped there and you program will crash and burn!

The explanation is that

x == &x == &x[0]
but
x1 != &x1

as x1 means the allocated memory buffer to which x1 points to whereas &x1 means the address in memory where the pointer x1 lives!!!

In real life it gets even worse if x1 lives on the stack and you write to &x1 -- here you put garbage on stack and you may go past the red page. If there are some other auto vars on stack after it and they happen to be pointers then you have a recipe for a clusterfuck.

If x1 is a global then you will corrupt other globals that live after x1. This is even more fun!

-ulianov

Unlock of Mutex Failed

This was the one of the most puzzling problems I faced. The Teja application we had was failing once in a blue moon printing the message "unlock of mutex failed".

Looking at he code I saw that pthread_unlock_mutex returned an error which should never happen.

I spent a month writing a wrapper for the pthread/mutex library (a mutex is in this particular case a 64-byte memory area, futex was not called).

I put signatures before and after the mutex, checked signatures & various mutex fields before entering the true mutex call to no avail.

It was a random memory corruption that I solved elsewhere so this error ceased to happen.

-ulianov

Re-Inventing the Wheel is Evil

Once I had to debug a program that ran in the Teja application framework for IXP 1200. We experienced random crashes at sites that were using a Walled Garden wireless authentication method.

By another name the Captive Portal consist in a TCP stream hijacker. The intercepted stream is HTTP. The component that handles this functionality simply pretends to be any HTTP destination, analyses the request and sends a crafted HTTP/307 redirection message in which the original URL is encoded as a GET parameter.

The Teja module that implemented the redirection got as an input an Ethernet frame (TCP reassembly was not performed but that was not a practical problem as most people simply go to http://yahoo.com, i.e. use short URLs that fit within one frame).

The data structure that enveloped the Ethernet frame was called Packet and it looked like:

struct Packet {
    char  buffer[1500];
    short start;
    short end;
};

To an astute reader this resembles an skbuff.

The code was parsing the input request and sending back a 307 that looked like "Location: http://auth.host/login.php?dest=OriginalURL"

The response was kept in a 6000-byte buffer which was large enough but here's what they were doing:

memcpy(packet.buffer+packet.end, response, response_len);
packet.end += response_len;

See anything wrong? If response_len>1500 then the memcpy thrashes packet.end, then one adds something to it resulting in a bigger junk. How about the junk in packet.start?

The Packets were malloc'ed [thus living in a pool of objects and having neighbours] so this had the potential (if len>1508) to corrupt the "secret stuff" the GNU allocator puts before each malloc'ed pointer. This will cause a happy crash when you want to free that pointer!

Later on the packet was copied into a hardware buffer that lived in IXP DRAM mapped in the Arm CPU address space like so:

memcpy(pHwBuf->buffer, \
       packet.buffer+packet.start, \
       (packet.end-packet.start));

Here you could copy garbage from a garbage address (packet.buffer+packet.start) [no crash if the memory was mapped] that had a garbage length (packet.end-packet.start) thus thrashing memory left and right in unexpected places.

Murphy's Laws dictate that the memory belonged to other threads of execution and that these threads will crash at another time in a such a way to turn you green.

In order to track that I had to change the Packet like so:

struct Packet {
    unsigned sig1 = 0xdeadbeef; // initialised when
                                // allocating a Packet
    char buffer[1500];
    unsigned sig2 = 0xfeedabba;
    short start;
    short end;
    unsigned sig3 = 0xdeafabba;
};

and check the signatures after each Packet operation. Took me only a month and one very pissed customer to fix this (not my code).

-ulianov

P.S. The Linux skbuff infrastructure comes with a lot of accessory functions that prevent this kind of crap from happening by Oops'ing.

Shooting your Own Foot and Jumping on it Afterwards

While I was developing my anti-recruiter Sendmail milter I was relying on nohup(1) to collect stderr output. Each time I would change code I would re-start the milter and wipe nohup.out like so "> nohup.out".

Then I changed code to redirect stderr to "milter.err" and at every restart I would do "> milter.err". One night at midnight I did not pay attention to the BASH-completion and wiped the code itself!

Now if you delete a file on an ext2 filesystem you can recover it using some utilities. But wiping the content of the file severs the relation between the i-node and its data blocks. debugfs ain't no help.

Moral: I had to "dd if=/dev/hda1 of=/storage/hda1.saved" and find the source code block by block by searching code patterns I remembered in hda1.saved. Trust me, file blocks are not contiguous on-disk!

I managed to piece together everything but the top of the file where the Perl use statements live. No biggie but I also skipped a small function and the code only complained at run time.

Took me three hours in the middle of the night and a huge head ache.

-ulianov

Thrashing the Kernel (Linux 2.4.x)

Today a co-worker told me that a library I've been writing is causing a Segfault but he could not give me an IP. Hmm. The library calls a custom driver and tries to pull data from the driver via ioctl's.

Then I used the lib in a utility and presto! I hosed the PPC 450. Funny is that the utility executes picture-perfect, does not crash, the kernel does not panic(), but everything executed after that segfaults.

Now that's artful.

Jun 4 update: I traced the code by #if 0'ing code and the problem is in the kernel driver.

Jun 10 update: The root cause was that they changed the way they were DMA-ing data into my user buffer that got mapped into the kernel space. As I was malloc-ing the buffer and not touching it the associated hardware pages were not actually mapped and that was confusing the PPC DMA engine.

memset-ing the buffer to zero in user space to fault the pages in or forcing the mapping of pages in kernel mode fixed the problem.

-ulianov