Showing posts with label kernel. Show all posts
Showing posts with label kernel. Show all posts

Friday, October 25, 2013

NULL Pointer in __up() in Custom Driver

My client calls and says he has a panic; I request stack traces and one comes to me as cute as lemon pie (see below).

This is Linux 2.6.29 with RTAI and has been running well for three years in production systems.

So the bug cannot be in __up(). I disass the crash point and it looked like
*(%ecx + 4) := %eax
This looks like "NULL->offset4". The kernel code looked like "waiter_list->prev". Hmm.

This must be something sprinkling memory. I reviewed custommodule and indeed some debug code looked like
array[index++] = debug_info;
No checking on array bounds, see?

Customer reworked the code and lived happy until the next crash ;)

-ulianov

PS. There were other crashes caused by the same problem from other power cycles but none as beautiful and explicit as this.
BUG: unable to handle kernel NULL pointer dereference at 00000004
IP: [<c0286e8a>] __up+0xb/0x2e
*pde = 365c1067 *pte = 00000000 
Oops: 0002 [#1] 
Modules linked in: custommodule(P) module3x20(P) moduleDSPcode(P)\
                   rdtsc customdebug coretemp fakertnet(P) e1000e \
                   irqregistrar(P) \
                   rtai_smi rtai_mbx rtai_sched \
                   rtai_math rtai_hal uhci_hcd

Pid: 1873, comm: customproc.bin Tainted: P (2.6.29.6-kernel8-ipipe #54)  
EIP: 0060:[<c0286e8a>] EFLAGS: 00010007 CPU: 0
EIP is at __up+0xb/0x2e
EAX: 73694c67 EBX: 00000200 ECX: 00000000 EDX: 00000000
ESI: f65cdbe0 EDI: f9feed50 EBP: f65cdb14 ESP: f65cdb14
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process mts5000.bin (pid: 1873, ti=f65cc000 task=f70eacc0 task.ti=f65cc000)
I-pipe domain Linux
Stack:
 f65cdb20 c012627a f9f286d4 f65cdbec f86a32c4 00004e1f 00004e20 f9e72441
 00000013 00000002 000c0f19 00000001 f86b819b 20203130 65532f3c 6e697474
 73694c67 00003e74 00000000 00000000 00000000 00000000 00000000 00000000
Call Trace:
 [<c012627a>] ? up+0x2e/0x44
 [<f86a32c4>] ? dequeueCommsRequest+0x217/0x225 [custommodule]
 [<f86a0382>] ? customdriver_read+0x15e2/0x1cdb [custommodule]
 [<c01331b8>] ? __ipipe_restore_root+0x16/0x18
 [<c01331b8>] ? __ipipe_restore_root+0x16/0x18
 [<c0131e6e>] ? cpu_quiet+0x71/0xcb
 [<c0118ff1>] ? __do_softirq+0xc5/0xcd
 [<c0119110>] ? irq_exit+0x28/0x2a
 [<c0104285>] ? do_IRQ+0x55/0x68
 [<f86b0024>] ? pfc_runInInterrupt+0xe0/0x6cf [custommodule]
 [<f86ae1a4>] ? sampleInterruptHandler+0x2944/0x2958 [custommodule]
 [<f86ae1a4>] ? sampleInterruptHandler+0x2944/0x2958 [custommodule]
 [<f86b0024>] ? pfc_runInInterrupt+0xe0/0x6cf [custommodule]
 [<f86ae1a4>] ? sampleInterruptHandler+0x2944/0x2958 [custommodule]
 [<c011222a>] ? enqueue_task_fair+0x12b/0x133
 [<c0110df5>] ? check_preempt_wakeup+0x82/0xa5
 [<c0112922>] ? try_to_wake_up+0xa2/0xad
 [<c0112944>] ? wake_up_state+0xa/0xc
 [<c011d59f>] ? signal_wake_up+0x51/0x55
 [<c011d717>] ? complete_signal+0x174/0x18c
 [<c011d8b1>] ? send_signal+0x182/0x197
 [<c01331b8>] ? __ipipe_restore_root+0x16/0x18
 [<c011df89>] ? group_send_sig_info+0x54/0x5d
 [<c011dfbd>] ? kill_pid_info+0x2b/0x35
 [<c011e129>] ? sys_kill+0x6f/0x114
 [<f869eda0>] ? customdriver_read+0x0/0x1cdb [custommodule]
 [<c014ffbe>] ? vfs_read+0x87/0x101
 [<c01500d1>] ? sys_read+0x3b/0x60
 [<c0102c07>] ? syscall_call+0x7/0xb
EIP: [<c0286e8a>] __up+0xb/0x2e SS:ESP 0068:f65cdb14
---[ end trace 2aa77bbc7c743932 ]---

Thursday, March 1, 2012

On Doing Ugly Linux Kernel Hacks for Need & Profit

A client has two targets ("new" and "legacy") and they insist they run the same system image on both. The two have subtle differences, the most annoying being that the legacy one does not support (U)DMA for CompactFlash cards.

Linux can be told to disable (U)DMA support by passing "ide-core.nodma=0.0" in the command line but there's no way to re-enable later on, performance is dead-slow and these folks have tons of code that has to be loaded from the CF and they like fast boot times.

I cut some of their load time by upx'ing their XFree86 binary and proprietary app and that helped somewhat on both platforms. And it shrank their installer by 1M which is always a bonus.

I tried to muck with LILO but second.S is written in assembler (mine is quite rusty) and getting to the actual command line sector is yucky.

The fall back was to butcher the kernel. I have simple target detection routine which uses cpuid, let's call that chkLegacyCPU().

I changed start_kernel() in init/main.c to append " ide-core.nodma=0.0" to the command line (which is now a global in main.c) if the legacy platform is detected.

Also the legacy target does not support booting in graphics mode (vga=8) so I modified main() in arch/x86/boot/main.c to force the VGA mode to NORMAL_VGA if legacy platform.

Interestingly the two modifications are in two different parts of the kernel -- the one in x86/boot executes in x86 real mode and has a totally different software stack so I had to duplicate the code of chkLegacyCPU() for it.

-ulianov

Tuesday, May 25, 2010

How to Measure the Duration of a RTAI Semaphore Operation

Today I was asked the question "How long does it take to signal a semaphore?".

This is important as I had to do it in an ISR [which serviced a quasi-timer IRQ] to signal an RT task to start doing a job (this is akin to Linux tasklets).

Here is how I measured it (interrupts are disable to keep the measurement accurate):

#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/kernel.h>
#include <linux/types.h>

#include <rtai_sem.h>
#include <rtai_sched.h>

SEM pollSem;

static int __init testsem_init_module(void)
{
rt_typed_sem_init(&pollSem, 0, BIN_SEM | FIFO_Q);

rt_global_cli();

volatile unsigned long long ticks_start;
__asm__("rdtsc\n\t"
"mov %%edx, %%ecx\n\t"
:"=A" (ticks_start));

rt_sem_signal(&pollSem);

volatile unsigned long long ticks_end;
__asm__("rdtsc\n\t"
"mov %%edx, %%ecx\n\t"
:"=A" (ticks_end));

rt_global_sti();

long long dT = ticks_end - ticks_start;

rt_sem_delete(&pollSem);

printk(KERN_DEBUG "rt_sem_signal took %lld ticks\n", dT);

return -EBUSY;
}

module_init(testsem_init_module);
MODULE_LICENSE("GPL");
The answer was (on a PIII/1.2GHz) about 300 cpu ticks which is ~0.25 uS.

-ulianov

Monday, December 14, 2009

How to Learn What Code is Actually Disabling IRQs?

When one applies the PREEMPT_RT patch the spinlocks & mutexes get horribly overloaded. cli/sti use is frowned upon (x86) so one ends up with macros of macros of macros for locking so it is a head ache to see what functions are actually disbling interrupts.

Yet we are on x86 and the locking sequence generated by GCC is simple:
cli
...
sti
or could by a bit more complicated:
pushf
pop %reg
cli
...
push %reg
popf
Luckily I used objdump and a bit of massaging come to the rescue in the form of a neat script that finds all the object files that make up the compiled kernel and then searches the disassembler output for these sequences.

Bonus: the function names that do the locking are also printed.

Here comes findcli_x86.sh [run it at the root of the kernel tree after you compile the kernel]:
#!/bin/sh

# This file is licensed under GPLv2

# Note this works only with x86 code

obj_list='obj';
trap "rm -f $obj_list" 0 1 2 15

function extractcli()
{
file=$1;

[ ! -r "$file" ] && return 1;

[ -z "$OBJDUMP" ] && OBJDUMP=${CROSS_COMPILE}objdump

$OBJDUMP -dSCl $file | awk 'BEGIN {
idx="nosuchfunc"
}
/(^[a-zA-Z_]|cli[^a-z]|popf|sti[^a-z])/ && !/(file format|Disassembly)/ {
if($0 ~ /^[a-zA-Z_]/) {
idx=$1
} else {
F[idx]=(F[idx] "\n" "\t" $1 "\t" $NF);
}
}
END {
for(idx in F) {
print (" " idx F[idx])
}
}'
return 0
}

# main

find -type f -name '*.o' > $obj_list
for obj in $(cat $obj_list)
do
o=$(echo $obj | sed 's/^\.\///g')
[ "$o" = 'vmlinux.o' ] && continue; # this is the whole kernel
echo $o | grep -q 'built-in\.o' && continue; # these are aggregations

echo -en " \r$o" 1>&2
cnt=$(objdump -dS $o | grep -cw cli)
[ "$cnt" = '0' ] && continue;

echo -en " \r" 1>&2

src='???';
c=$(echo $o | sed 's/\.o$/\.c/g')
S=$(echo $o | sed 's/\.o$/\.S/g')
[ -f "$c" ] && src="$c"
[ -f "$S" ] && src="$S"

echo "$o: $cnt, src: $src"
extractcli $o
done
As usual awk comes to the rescue.

-ulianov

P.S. I looked for a decent decompiler for Linux other than objdump but I found only ancient ones and all broken. This is annoying as sometimes I need to see what external functions are called by an object file (.o).

Friday, December 4, 2009

Linux 2.6.31/PREEMPT_RT and Hard IRQs

I was trying to improve the latency of an ISR which serves an IRQ which fires every 100 uS. As a Softirq even when running at the highest priority the latency would some times slip to 1500 uS which is unacceptable (dd if=/dev/hda of=/dev/null comes to mind as a reason, hda is a slooow Compact Flash).

It does not help that the ISR is crunching numbers and doing float calculation in interrupt context. This ISR munches about 25 uS and should execute as close as possible to the beginning of the 100 uS interval.

So I tried to use a hard IRQ (IRQF_NODELAY) which was not good as the kernel would BUG_ON in rtmutex.c:807 when a process was trying to read from the character device which belonged to the driver which used the said IRQ.

I learned that spin_lock/spin_unlock are being overloaded by PREEMPT_RT by rt_mutex_lock/rt_mutex_unlock which do not seem to be compatible with hard IRQs: the mutex gets double-locked by the "current" process altho the two contenders are the hard-IRQ and code executed on behalf of a user process. I think this is bonkers.

I managed to get it working by using "atomic_" spinlock primitives which basically boil down [in disasm output] to cli/sti.

-ulianov

Thursday, December 3, 2009

A Nice Kernel Trick for x86

Today I learnt a nice way to obtain the number of microseconds since the power-up of the CPU using a P5 performance counter via (rdtsc). This instruction returns the number of ticks since power-up but it can be scaled to microseconds by dividing the number by the CPU frequency (in MHz).

The kernel code reads:
#include <linux/kernel.h>
#include <linux/cpufreq.h>

unsigned long long uSecSinceBoot()
{
volatile unsigned long long int cpu_ticks;

__asm__("rdtsc\n\t"
"mov %%edx, %%ecx\n\t"
:"=A" (cpu_ticks));

unsigned long long div = cpu_ticks * 1000;
do_div(div, cpu_khz); // TRICK! div is modified, do_div() returns reminder

return div;
}
I am using do_div() as on x86/32 bits long-long division is supported by GCC via an external function (_udivdi3) which usually lives in libgcc but it is not provided by the kernel.

-ulianov

P.S. I did extract _udivdi3 from libgcc and then disassembled/reassembled it but it's a pain. do_div is the right thing to use in the kernel.

Saturday, August 30, 2008

Log messages to dmesg/kernel buffer from user-space

From time to time a buddy of mine who's not a professional software developer asks questions that are interesting albeit they go against the grain of UN*X programming.

This time he asked whether printk is accessible to user-mode applications. As per entry.S it does not but the guy is stubborn so I got bored explaining the impossibility I got to work and wrote a Linux kernel module that allows him to fulfill his fantasy.

The technical article is here: http://spamchk.no-ip.org/uprintk.html. The source code is here.

-ulianov

Friday, August 15, 2008

Fooling Around with jiffies

Some time ago a QA buddy asked whether the Linux uptime can be changed on the fly in order to study the behaviour of a closed-source application that would only exhibit a certain bug after two weeks of running.

I looked at sys_gettimeofday (Linux kernel 2.4) and learned that it depends on jiffies. So I wrote a small module that adjusts jiffies at will.

The technical article is here: http://spamchk.no-ip.org/kernuptime.html. The source code is here.

-ulianov