While I was developing my anti-recruiter Sendmail milter I was relying on nohup(1) to collect stderr output. Each time I would change code I would re-start the milter and wipe nohup.out like so "> nohup.out".
Then I changed code to redirect stderr to "milter.err" and at every restart I would do "> milter.err". One night at midnight I did not pay attention to the BASH-completion and wiped the code itself!
Now if you delete a file on an ext2 filesystem you can recover it using some utilities. But wiping the content of the file severs the relation between the i-node and its data blocks. debugfs ain't no help.
Moral: I had to "dd if=/dev/hda1 of=/storage/hda1.saved" and find the source code block by block by searching code patterns I remembered in hda1.saved. Trust me, file blocks are not contiguous on-disk!
I managed to piece together everything but the top of the file where the Perl use statements live. No biggie but I also skipped a small function and the code only complained at run time.
Took me three hours in the middle of the night and a huge head ache.
-ulianov
Tuesday, June 3, 2008
Thrashing the Kernel (Linux 2.4.x)
Today a co-worker told me that a library I've been writing is causing a Segfault but he could not give me an IP. Hmm. The library calls a custom driver and tries to pull data from the driver via ioctl's.
Then I used the lib in a utility and presto! I hosed the PPC 450. Funny is that the utility executes picture-perfect, does not crash, the kernel does not panic(), but everything executed after that segfaults.
Now that's artful.
Jun 4 update: I traced the code by #if 0'ing code and the problem is in the kernel driver.
Jun 10 update: The root cause was that they changed the way they were DMA-ing data into my user buffer that got mapped into the kernel space. As I was malloc-ing the buffer and not touching it the associated hardware pages were not actually mapped and that was confusing the PPC DMA engine.
memset-ing the buffer to zero in user space to fault the pages in or forcing the mapping of pages in kernel mode fixed the problem.
-ulianov
Then I used the lib in a utility and presto! I hosed the PPC 450. Funny is that the utility executes picture-perfect, does not crash, the kernel does not panic(), but everything executed after that segfaults.
Now that's artful.
Jun 4 update: I traced the code by #if 0'ing code and the problem is in the kernel driver.
Jun 10 update: The root cause was that they changed the way they were DMA-ing data into my user buffer that got mapped into the kernel space. As I was malloc-ing the buffer and not touching it the associated hardware pages were not actually mapped and that was confusing the PPC DMA engine.
memset-ing the buffer to zero in user space to fault the pages in or forcing the mapping of pages in kernel mode fixed the problem.
-ulianov
Subscribe to:
Posts (Atom)