Subject: Re: Multicast packet loss

From: Kenny Chang <[email-masked]>
Date: Mon, 02 Feb 2009 11:13:19 -0500

Neil Horman wrote:
> On Fri, Jan 30, 2009 at 11:41:23PM +0100, Eric Dumazet wrote:
>   
>> Kenny Chang a écrit :
>>     
>>> Ah, sorry, here's the test program attached.
>>>
>>> We've tried 2.6.28.1, but no, we haven't tried the 2.6.28.2 or the
>>> 2.6.29.-rcX.
>>>
>>> Right now, we are trying to step through the kernel versions until we
>>> see where the performance drops significantly.  We'll try 2.6.29-rc soon
>>> and post the result.
>>>       
>> 2.6.29-rc contains UDP receive improvements (lockless)
>>
>> Problem is multicast handling was not yet updated, but could be :)
>>
>>
>> I was asking you "cat /proc/interrupts" because I believe you might
>> have a problem NIC interrupts being handled by one CPU only (when having problems)
>>
>>     
> That would be expected (if irqbalance is running), and desireable, since
> spreading high volume interrupts like NICS accross multiple cores (or more
> specifically multiple L2 caches), is going increase your cache line miss rate
> significantly and decrease rx throughput.
>
> Although you do have a point here, if the system isn't running irqbalance, and
> the NICS irq affinity is spread accross multiple L2 caches, that would be a
> point of improvement performance-wise.  
>
> Kenny, if you could provide the /proc/interrupts info along with /proc/cpuinfo
> and your stats that I asked about earlier, that would be a big help.
>
> Regards
> Neil
>
>   
Hi Neil,

Here's the information you requested.

Kenny

kchang@beast8:~$ uname -a
Linux beast8 2.6.24-19-server #1 SMP Wed Aug 20 18:43:06 UTC 2008 x86_64 
GNU/Linux
kchang@beast8:~$ cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 0
siblings    : 4
core id        : 0
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5322.91
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 0
siblings    : 4
core id        : 1
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.03
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 2
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 0
siblings    : 4
core id        : 2
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.06
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 3
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 0
siblings    : 4
core id        : 3
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.06
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 4
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 1
siblings    : 4
core id        : 0
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.07
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 5
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 1
siblings    : 4
core id        : 1
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.07
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 6
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 1
siblings    : 4
core id        : 2
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.08
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

processor    : 7
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Xeon(R) CPU           L5430  @ 2.66GHz
stepping    : 10
cpu MHz        : 2659.999
cache size    : 6144 KB
physical id    : 1
siblings    : 4
core id        : 3
cpu cores    : 4
fpu        : yes
fpu_exception    : yes
cpuid level    : 13
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx 
est tm2 ssse3 cx16 xtpr dca sse4_1 lahf_lm
bogomips    : 5320.08
clflush size    : 64
cache_alignment    : 64
address sizes    : 38 bits physical, 48 bits virtual
power management:

kchang@beast8:~$ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       CPU4       
CPU5       CPU6       CPU7      
  0:         67          0          1          0          0          
0          0          0   IO-APIC-edge      timer
  1:          0          0          0          0          0          
0          0          0   IO-APIC-edge      i8042
  8:          0          0          0          1          0          
0          0          0   IO-APIC-edge      rtc
  9:          0          0          0          0          0          
0          0          0   IO-APIC-fasteoi   acpi
 14:         12         13         13         13         12         
10         13         13   IO-APIC-edge      libata
 15:          0          0          0          0          0          
0          0          0   IO-APIC-edge      libata
 17:        294        295        293        294        294        
296        293        288   IO-APIC-fasteoi   aacraid
 22:          6          5          5          5          6          
6          6          6   IO-APIC-fasteoi   uhci_hcd:usb3
 23:          7          8          8          7          7          
8          7          8   IO-APIC-fasteoi   ehci_hcd:usb1, 
uhci_hcd:usb2, uhci_hcd:usb4
2294:         48         46         48         48         49         
47         48         51   PCI-MSI-edge      eth0
NMI:          0          0          0          0          0          
0          0          0   Non-maskable interrupts
LOC:       5088       3394       3129       2835       2561       
2938       2576       2798   Local timer interrupts
RES:         59        119         58         36         34         
71         50         17   Rescheduling interrupts
CAL:        132        128        149        141        138        
140        152        140   function call interrupts
TLB:        285        178        278        183        297        
191        295        157   TLB shootdowns
TRM:          0          0          0          0          0          
0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          
0          0          0   Threshold APIC interrupts
SPU:          0          0          0          0          0          
0          0          0   Spurious interrupts
ERR:          0