Friday, September 18, 2009

The Strange Case of Multiplying Zombies

While working on a Linux/PPC embedded board I changed /etc/ (the system startup script as known by Busybox) to start our application in foreground.

This was good as we could see its output on the serial console and be able to interact with it. The application is a standalone binary which is interrogated by CGI and thru a Web interface. The web server is thttpd.

So I had everything running and I looked at the process table and noticed it being filled by zombie (Z) entries of the logger CGI (which gets invoked thrice a minute).

I tried to trace (strace) thttpd, I even put a waitpid(-1) at the top of its main loop [it's a single-threaded web server] and still could not get the damn zombies reaped!

This was baad as the system could stay up only for half a day before filling up the process table.

I did some hard thinking and remembered some APUE and Bach bits and concluded that Busybox [which alas contains init] must be still waiting for the termination of /etc/ before it starts the prescribed init behaviour!! I.e. Reaping orphaned processes and zombies.

So I put our application in background via nohup and voila! everything was good again.