Subject: Re: what's causing "ip_rt_bug"?

From: Tomasz Chmielewski <[email-masked]>
Date: Fri, 17 Jun 2011 23:37:46 +0200

On 17.06.2011 22:36, Eric Dumazet wrote:
> Le vendredi 17 juin 2011 à 22:00 +0200, Tomasz Chmielewski a écrit :
>> I have a system pushing around 800 Mbit/s, ~130 kpps.
>>
>> It uses 2.6.35.12 kernel.
>>
>>
>> Several times a minute, I can see entries like (a.b.c.d - IP of this system):
>>
>> Jun 18 02:39:19 KOR-SV22 kernel: [37187.665951] ip_rt_bug: 110.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:39:19 KOR-SV22 kernel: [37187.685419] ip_rt_bug: 110.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:40:31 KOR-SV22 kernel: [37259.199315] ip_rt_bug: 124.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:40:36 KOR-SV22 kernel: [37263.828000] ip_rt_bug: 124.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:44:16 KOR-SV22 kernel: [37484.120689] ip_rt_bug: 110.x.x.x ->  a.b.c.d, ?
>> Jun 18 02:44:19 KOR-SV22 kernel: [37487.114357] ip_rt_bug: 110.x.x.x ->  a.b.c.d, ?
>>
> 
> Hi
> 
> What your routing table looks like ? (ip ro)

It's just a proxy, no special routing set:

# ip ro
58.185.117.18 via 119.46.110.193 dev eth0 
119.46.240.13 via 119.46.110.193 dev eth0 
58.185.117.29 via 119.46.110.193 dev eth0 
119.46.241.13 via 119.46.110.193 dev eth0 
58.185.117.28 via 119.46.110.193 dev eth0 
119.46.110.192/26 dev eth0  proto kernel  scope link  src 119.46.110.197 
169.254.0.0/16 dev eth0  scope link 
default via 119.46.110.195 dev eth0 

The box is also crashing every few days; and I really had no clue why (just connected a serial console to catch any new oops/panic).

The last time it crashed, I have this entry in syslog:

Jun 17 16:16:17 TRUE-SC02 kernel: [172488.602629] ip_rt_bug: 124.121.155.197 -> 119.46.110.197, ?
Jun 17 16:17:00 TRUE-SC02 kernel: [172531.239041] BUG: unable to handle kernel NULL pointer dereference at (null)
Jun 17 16:17:00 TRUE-SC02 kernel: [172531.239409] IP: [<ffffffff81361cae>] dev_queue_xmit+0x1e3/0x441
Jun 17 16:17:00 TRUE-SC02 kernel: [172531.239760] PGD 43c30b067 PUD 439e63067 PMD 0
Jun 17 16:17:00 TRUE-SC02 kernel: [172531.240103] Oops: 0000 [#1] SMP
Jun 17 16:19:58 TRUE-SC02 syslogd 1.4.1: restart.

Right now, it uses the newest igb driver, and I started seeing "Out of socket memory" quite a bit (didn't have it with the original igb driver from 2.6.35.12).

So I doubled this value to be:

 net.ipv4.tcp_max_orphans = 256000

and "Out of socket memory" stopped showing up.

Instead, "ip_rt_bug" shows up.

-- 
Tomasz Chmielewski
http://wpkg.org