Subject: Re: bisect results of MSI-X related panic (help!)

From: Tejun Heo <[email-masked]>
Date: Thu, 15 Oct 2009 16:30:25 +0900

Hello,

Brandeburg, Jesse wrote:
> On Mon, 12 Oct 2009, Tejun Heo wrote:
>> Can you please apply the following patch and try to retrigger the
>> panic?
>>
>> diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
>> index c166019..f5a1482 100644
>> --- a/kernel/irq/chip.c
>> +++ b/kernel/irq/chip.c
>> @@ -63,6 +63,9 @@ void dynamic_irq_cleanup(unsigned int irq)
>>  	struct irq_desc *desc = irq_to_desc(irq);
>>  	unsigned long flags;
>>
>> +	printk("XXX dynamic_irq_cleanup() called on %u\n", irq);
>> +	dump_stack();
>> +
>>  	if (!desc) {
>>  		WARN(1, KERN_ERR "Trying to cleanup invalid IRQ%d\n", irq);
>>  		return;
> 
> I'm working on it, but now that I've added a bunch of debug including the 
> above printk, my system panics (with a stack protector canary overwrite) 
> when loading the first network adapter with 30+ MSI-X vectors.  I can boot 
> single user mode and bring up netconsole, but then as soon as I brought up 
> the first port with lots of MSI-X vectors, the system hard locks, no panic 
> message.
>  
> I have a bit of a theory that the node = -1 (numa_node) stuff might be 
> playing some havoc with the code in numa_migrate.c.  I'm not sure if that 
> is contributing, but the code in there doesn't seem written to handle node 
> = - 1 very well.  As in I never see it do an smp_processor_id at the 
> bottom before accessing the node value.
> 
> Not sure if that is relevant, but I wanted to mention it before I went 
> home.
> 
> What next?  I made it worse so I guess that is something.

I don't know.  At this point, I can't think of anything other than
sprinkling printks and dump_stacks around.  :-(

Thanks.

-- 
tejun