Subject: Re: weird problem

From: Paweł Staszewski <[email-masked]>
Date: Thu, 25 Jun 2009 21:45:16 +0200

Ok

After this day of observation im near 100% sure that this cpu load is 
made by route cahce flushes
When route cache increase to its "net.ipv4.route.gc_thresh" size or is 
near that size
system is starting to drop some routes from cache then cpu load is 
increase from 2% to near 80%
after cleaning / flush cache when cache is filling cpu load is again 
normal 2%

Someone know how to resolve this ?
on kernels < 2.6.29 i don't see this, all start after upgrade from 
2.6.28 to 2.6.29 - then i try 2.6.29.1 , 2.6.29.3 and 2.6.30 and on all 
this kernels >= 2.6.29 problem with cpu load is the same.

I can minimize this cpu fluctuations by changing of route cache /proc 
parameters but the best result for my router was

15 sec of 2% cpu
and after
15sec of 80% cpu

Regards
Pawel Staszewski

Paweł Staszewski pisze:
> Another info
>
> i look at some stats from rtstats and mpstat and try to join them 
> together and here is output:
> 19:06:50     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
> %steal  %guest   %idle        rt_cache entries
> 19:07:01     all    0.00    0.00    0.12    0.00    0.59   12.00    
> 0.00    0.00   87.29        77726
> 19:07:02     all    0.00    0.00    0.12    0.00    0.49    6.11    
> 0.00    0.00   93.28        80127
> 19:07:03     all    0.00    0.00    0.00    0.00    0.11    1.06    
> 0.00    0.00   98.84        82596
> 19:07:04     all    0.00    0.00    0.16    0.00    0.00    2.07    
> 0.00    0.00   97.77        85147
> 19:07:05     all    0.00    0.00    0.00    0.00    0.12    1.81    
> 0.00    0.00   98.07        87641
> 19:07:06     all    0.00    0.00    0.00    0.00    0.27    1.50    
> 0.00    0.00   98.22        90085
> 19:07:07     all    0.00    0.00    0.00    0.00    0.23    1.72    
> 0.00    0.00   98.05        93586
> 19:07:08     all    0.00    0.00    0.00    0.00    0.00    1.56    
> 0.00    0.00   98.44        96865
> 19:07:09     all    0.00    0.00    0.00    0.00    0.00    1.45    
> 0.00    0.00   98.55        99803
> 19:07:10     all    0.00    0.00    0.00    0.00    0.27    1.65    
> 0.00    0.00   98.08        102123
> 19:07:11     all    0.00    0.00    0.10    0.00    0.30    4.07    
> 0.00    0.00   95.54        104618
> 19:07:12     all    0.00    0.00    0.00    0.00    1.30   18.96    
> 0.00    0.00   79.74        105524
> 19:07:13     all    0.00    0.00    0.00    0.00    0.84   13.58    
> 0.00    0.00   85.58        101085
> 19:07:14     all    0.00    0.00    0.00    0.00    1.36   20.45    
> 0.00    0.00   78.18        97518
> 19:07:15     all    0.00    0.00    0.00    0.00    0.98   15.89    
> 0.00    0.00   83.13        94308
> 19:07:16     all    0.00    0.00    0.00    0.00    1.32   17.00    
> 0.00    0.00   81.69        91811
> 19:07:17     all    0.00    0.00    0.00    0.00    0.67   14.57    
> 0.00    0.00   84.75        92278
> 19:07:18     all    0.00    0.00    0.00    0.00    1.51   17.40    
> 0.00    0.00   81.10        90936
> 19:07:19     all    0.00    0.00    0.00    0.00    1.24   13.46    
> 0.00    0.00   85.29        87951
> 19:07:20     all    0.00    0.00    0.14    0.00    0.99   18.75    
> 0.00    0.00   80.11        85071
> 19:07:21     all    0.00    0.00    0.00    0.00    0.84   15.20    
> 0.00    0.00   83.96        82477
> 19:07:22     all    0.40    0.00    0.00    0.00    1.71   17.65    
> 0.00    0.00   80.24        80301
> 19:07:23     all    0.00    0.00    0.11    0.00    1.08   13.33    
> 0.00    0.00   85.48        78480
> 19:07:24     all    0.00    0.00    0.00    0.00    0.87   19.25    
> 0.00    0.00   79.88        76880
>
> When cpu load is rising rt cache entries are decreasing
> and when cpu load is from 1 do 3% rt cache entries are increasing
>
> some sysctl-s that i have and are route cache specific:
> net.ipv4.route.gc_thresh = 190536
> net.ipv4.route.gc_min_interval = 0
> net.ipv4.route.gc_min_interval_ms = 0
> net.ipv4.route.gc_timeout = 15
> net.ipv4.route.gc_interval = 1
> net.ipv4.route.gc_elasticity = 4
>
>
>
>
> Paweł Staszewski pisze:
>> oprofile output per cpu
>> CPU: Core 2, speed 3000.08 MHz (estimated)
>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a 
>> unit mask of 0x00 (Unhalted core cycles) count 100000
>> Samples on CPU 0
>> Samples on CPU 1
>> Samples on CPU 2
>> Samples on CPU 3
>> Samples on CPU 4
>> Samples on CPU 5
>> Samples on CPU 6
>> Samples on CPU 7
>> Samples on CPU all
>> samples  %        samples  %        samples  %        samples  
>> %        samples  %        samples  %        samples  %        
>> samples  %        samples  %
>>    image name               app name                 symbol name
>> 29520    21.7362  0              0  0              0  0              
>> 0  0              0  0              0  9571     34.0254  
>> 0              0  4701      0.22
>> 55  oprofiled                oprofiled                (no symbols)
>> 18441    13.5785  0              0  0              0  0              
>> 0  0              0  0              0  0              0  
>> 0              0  2666      0.12
>> 79  vmlinux                  vmlinux                  rt_worker_func
>> 18428    13.5690  132089    2.1756  135138    1.4238  21735    
>> 42.7703  775       1.9819  893       5.3180  847       3.0111  
>> 1056      3.3130  37949     1.82
>> 01  vmlinux                  vmlinux                  mwait_idle
>> 15797    11.6317  212       0.0035  2879      0.0303  17        
>> 0.0335  2         0.0051  4         0.0238  14        0.0498  
>> 185       0.5804  2895      0.13
>> 89  vmlinux                  vmlinux                  free_block
>> 9034      6.6519  1218      0.0201  2727      0.0287  0              
>> 0  0              0  0              0  0              0  
>> 0              0  2004      0.09
>> 61  vmlinux                  vmlinux                  dst_destroy
>> 7113      5.2375  632       0.0104  483       0.0051  6615     
>> 13.0170  543       1.3886  461       2.7454  1951      6.9359  
>> 1274      3.9970  4073      0.19
>> 54  vmlinux                  vmlinux                  
>> mutex_spin_on_owner
>> 4382      3.2266  652       0.0107  1880      0.0198  2         
>> 0.0039  0              0  0              0  3         0.0107  
>> 7         0.0220  1050      0.05
>> 04  vmlinux                  vmlinux                  __call_rcu
>> 4337      3.1934  418       0.0069  830       0.0087  58        
>> 0.1141  18        0.0460  8         0.0476  20        0.0711  
>> 39        0.1224  917       0.04
>> 40  vmlinux                  vmlinux                  
>> __rcu_process_callbacks
>> 3991      2.9387  51       8.4e-04  325       0.0034  136       
>> 0.2676  2         0.0051  1         0.0060  1034      3.6759  
>> 205       0.6432  1110      0.05
>> 32  vmlinux                  vmlinux                  
>> copy_user_generic_string
>> 3292      2.4240  275838    4.5432  391639    4.1263  869       
>> 1.7100  431       1.1022  350       2.0843  673       2.3925  
>> 1069      3.3538  89939     4.31
>> 37  vmlinux                  vmlinux                  _raw_spin_lock
>> 1909      1.4056  79        0.0013  376       0.0040  0              
>> 0  0              0  0              0  0              0  
>> 0              0  348       0.01
>> 67  vmlinux                  vmlinux                  dst_rcu_free
>> 1461      1.0758  0              0  1163      0.0123  0              
>> 0  0              0  0              0  0              0  
>> 0              0  0
>> 0  vim                      vim                      (no symbols)
>> 1122      0.8262  11514     0.1896  6590      0.0694  1857      
>> 3.6542  7638     19.5325  2138     12.7323  1213      4.3123  
>> 4025     12.6278  9676      0.46
>> 41  libc-2.8.so              libc-2.8.so              (no symbols)
>> 1035      0.7621  9027      0.1487  22216     0.2341  4         
>> 0.0079  0              0  0              0  5         0.0178  
>> 15        0.0471  4310      0.20
>> 67  vmlinux                  vmlinux                  kmem_cache_free
>> 1016      0.7481  218       0.0036  465       0.0049  0              
>> 0  0              0  0              0  0              0  
>> 0              0  255       0.01
>> 22  vmlinux                  vmlinux                  ipv4_dst_destroy
>> 951       0.7002  4290      0.0707  5514      0.0581  3283      
>> 6.4603  867       2.2172  673       4.0079  1096      3.8963  
>> 1316      4.1288  2593      0.12
>> 44  vmlinux                  vmlinux                  tg_shares_up
>> 708       0.5213  104       0.0017  279       0.0029  0              
>> 0  0              0  0              0  0              0  
>> 0              0  159       0.00
>> 76  vmlinux                  vmlinux                  call_rcu_bh
>> 581       0.4278  21812     0.3593  38364     0.4042  68        
>> 0.1338  23        0.0588  23        0.1370  65        0.2311  
>> 96        0.3012  8000      0.38
>> 37  vmlinux                  vmlinux                  _raw_spin_unlock
>> 563       0.4145  1027      0.0169  2528      0.0266  0              
>> 0  0              0  0              0  0              0  3         
>> 0.0094  545       0.02
>> 61  vmlinux                  vmlinux                  local_bh_enable_ip
>> 509       0.3748  0              0  3        3.2e-05  0              
>> 0  0              0  0              0  4         0.0142  3         
>> 0.0094  76        0.00
>> 36  vmlinux                  vmlinux                  can_dirty
>> 430       0.3166  74        0.0012  148       0.0016  25        
>> 0.0492  13        0.0332  12        0.0715  9         0.0320  
>> 12        0.0376  96        0.00
>> 46  vmlinux                  vmlinux                  
>> rcu_process_callbacks
>> 372       0.2739  0              0  0              0  0              
>> 0  0              0  0              0  3         0.0107  2         
>> 0.0063  48        0.00
>> 23  vmlinux                  vmlinux                  do_journal_end
>> 347       0.2555  9991      0.1646  16066     0.1693  143       
>> 0.2814  66        0.1688  25        0.1489  46        0.1635  
>> 71        0.2228  3514      0.16
>> 85  vmlinux                  vmlinux                  rb_get_reader_page
>> 346       0.2548  952       0.0157  566       0.0060  871       
>> 1.7140  321       0.8209  174       1.0362  324       1.1518  
>> 269       0.8439  655       0.03
>> 14  vmlinux                  vmlinux                  find_busiest_group
>> 275       0.2025  1194      0.0197  1228      0.0129  1019      
>> 2.0052  203       0.5191  151       0.8992  290       1.0310  
>> 653       2.0487  799       0.03
>> 83  vmlinux                  vmlinux                  find_next_bit
>> 238       0.1752  3053      0.0503  4711      0.0496  116       
>> 0.2283  82        0.2097  54        0.3216  101       0.3591  
>> 101       0.3169  1174      0.05
>> 63  oprofile                 oprofile                 (no symbols)
>> 221       0.1627  0              0  0              0  0              
>> 0  0              0  0              0  0              0  
>> 0              0  18       8.6e-
>> 04  vmlinux                  vmlinux                  ahci_interrupt
>> 212       0.1561  38335     0.6314  60370     0.6361  329       
>> 0.6474  165       0.4220  136       0.8099  133       0.4728  
>> 205       0.6432  12821     0.61
>> 49  vmlinux                  vmlinux                  
>> apic_timer_interrupt
>> 210       0.1546  117       0.0019  1001      0.0105  2         
>> 0.0039  0              0  0              0  55        0.1955  
>> 16        0.0502  270       0.01
>> 30  vmlinux                  vmlinux                  memmove
>> 207       0.1524  0              0  7        7.4e-05  34        
>> 0.0669  0              0  0              0  93        0.3306  
>> 62        0.1945  214       0.01
>> 03  vmlinux                  vmlinux                  search_by_key
>> 204       0.1502  26603     0.4382  25086     0.2643  45        
>> 0.0886  18        0.0460  19        0.1131  19        0.0675  
>> 25        0.0784  6580      0.31
>> 56  vmlinux                  vmlinux                  __do_softirq
>> 170       0.1252  2618      0.0431  3493      0.0368  69        
>> 0.1358  42        0.1074  51        0.3037  64        0.2275  
>> 87        0.2729  856       0.04
>> 11  vmlinux                  vmlinux                  __schedule
>> 151       0.1112  7590      0.1250  8909      0.0939  273       
>> 0.5372  86        0.2199  73        0.4347  99        0.3519  
>> 142       0.4455  2114      0.10
>> 14  vmlinux                  vmlinux                  sched_clock_cpu
>> 142       0.1046  200       0.0033  165       0.0017  65        
>> 0.1279  23        0.0588  20        0.1191  15        0.0533  
>> 36        0.1129  94        0.00
>> 45  vmlinux                  vmlinux                  raise_softirq
>> 141       0.1038  8048      0.1326  11386     0.1200  2         
>> 0.0039  0              0  0              0  2         0.0071  
>> 4         0.0125  2589      0.12
>> 42  vmlinux                  vmlinux                  __phys_addr
>> 141       0.1038  23726     0.3908  24837     0.2617  89        
>> 0.1751  58        0.1483  55        0.3275  66        0.2346  
>> 62        0.1945  5398      0.25
>> 89  vmlinux                  vmlinux                  
>> tick_nohz_stop_sched_tick
>> 139       0.1023  43230     0.7120  68500     0.7217  105       
>> 0.2066  66        0.1688  86        0.5121  100       0.3555  
>> 100       0.3137  13725     0.65
>> 83  vmlinux                  vmlinux                  getnstimeofday
>> 136       0.1001  0              0  0              0  0              
>> 0  0              0  0              0  0              0  
>> 0              0  36        0.00
>> 17  vmlinux                  vmlinux                  
>> remove_journal_hash
>> 133       0.0979  8624      0.1420  9518      0.1003  217       
>> 0.4270  68        0.1739  64        0.3811  79        0.2808  
>> 106       0.3326  2348      0.11
>> 26  vmlinux                  vmlinux                  native_sched_clock
>> 129       0.0950  270       0.0044  233       0.0025  355       
>> 0.6986  93        0.2378  60        0.3573  102       0.3626  
>> 105       0.3294  214       0.01
>> 03  vmlinux                  vmlinux                  rebalance_domains
>> 117       0.0861  0              0  36       3.8e-04  57        
>> 0.1122  0              0  10        0.0596  66        0.2346  
>> 60        0.1882  220       0.01
>> 06  vmlinux                  vmlinux                  find_get_page
>> 112       0.0825  287       0.0047  397       0.0042  121       
>> 0.2381  89        0.2276  63        0.3752  85        0.3022  
>> 114       0.3577  220       0.01
>> 06  vmlinux                  vmlinux                  scheduler_tick
>> 107       0.0788  13702     0.2257  16191     0.1706  102       
>> 0.2007  28        0.0716  44        0.2620  41        0.1458  
>> 66        0.2071  3890      0.18
>> 66  vmlinux                  vmlinux                  
>> tick_nohz_stop_idle
>> 105       0.0773  4277      0.0704  7002      0.0738  53        
>> 0.1043  26        0.0665  24        0.1429  24        0.0853  
>> 38        0.1192  1561      0.07
>> 49  vmlinux                  vmlinux                  
>> ring_buffer_consume
>> 103       0.0758  0              0  2        2.1e-05  5         
>> 0.0098  0              0  0              0  20        0.0711  
>> 13        0.0408  61        0.00
>> 29  vmlinux                  vmlinux                  __find_get_block
>> 103       0.0758  49251     0.8112  85999     0.9061  166       
>> 0.3267  105       0.2685  97        0.5777  122       0.4337  
>> 138       0.4330  17108     0.82
>> 06  vmlinux                  vmlinux                  read_tsc
>> 98        0.0722  17406     0.2867  40897     0.4309  15        
>> 0.0295  2         0.0051  2         0.0119  62        0.2204  
>> 51        0.1600  7913      0.37
>> 95  vmlinux                  vmlinux                  kmem_cache_alloc
>> 98        0.0722  14817     0.2440  21291     0.2243  41        
>> 0.0807  49        0.1253  48        0.2859  62        0.2204  
>> 64        0.2008  4582      0.21
>> 98  vmlinux                  vmlinux                  ktime_get_ts
>> 94        0.0692  0              0  1        1.1e-05  2         
>> 0.0039  0              0  0              0  7         0.0249  
>> 6         0.0188  16       7.7e-
>> 04  vmlinux                  vmlinux                  
>> __find_get_block_slow
>> 92        0.0677  1161      0.0191  1411      0.0149  41        
>> 0.0807  39        0.0997  40        0.2382  53        0.1884  
>> 49        0.1537  408       0.01
>> 96  vmlinux                  vmlinux                  
>> select_nohz_load_balancer
>> 88        0.0648  11160     0.1838  54990     0.5794  69        
>> 0.1358  43        0.1100  35        0.2084  661       2.3499  
>> 55        0.1726  4513      0.21
>> 65  vmlinux                  vmlinux                  
>> get_next_timer_interrupt
>> 75        0.0552  0              0  0              0  0              
>> 0  0              0  0              0  0              0  
>> 0              0  6        2.9e-
>> 04  vmlinux                  vmlinux                  make_request
>> 69        0.0508  803       0.0132  1048      0.0110  35        
>> 0.0689  26        0.0665  15        0.0893  24        0.0853  
>> 58        0.1820  334       0.01
>> --More--
>>
>> Paweł Staszewski pisze:
>>> I dont know where to put this message so i start here because server 
>>> that i see this funny thing is make only networking
>>> Routing (bgpd) / traffic management (iproute2)
>>>
>>> This starts on kernels >= 2.6.29
>>>
>>> On server i have almost constant traffic that is about 300Mbit/s RX 
>>> and 300Mbit/s TX
>>> Server have two interfaces:
>>> 04:00.0 Ethernet controller: Intel Corporation 82573E Gigabit 
>>> Ethernet Controller (Copper) (rev 03)
>>> 05:00.0 Ethernet controller: Intel Corporation 82573L Gigabit 
>>> Ethernet Controller
>>>
>>> iBGP peer < - > [ eth0 - forward + shaping - eth1 ] < - >  eBGP
>>>
>>>
>>> And what is weird is CPU Load
>>> it fluctuate from 10% to almost 80% - but traffic is  constant :
>>> bwm-ng v0.6 (probing every 10.000s), press 'h' for help
>>>  input: /proc/net/dev type: rate
>>>  |         iface                   Rx                   
>>> Tx                Total
>>>  
>>> ============================================================================== 
>>>
>>>               lo:           0.00  b/s            0.00  
>>> b/s            0.00  b/s
>>>             eth0:         200.22 Mb/s          266.74 Mb/s          
>>> 466.96 Mb/s
>>>             eth1:         264.00 Mb/s          192.59 Mb/s          
>>> 456.59 Mb/s
>>>  
>>> ------------------------------------------------------------------------------ 
>>>
>>>            total:         464.22 Mb/s          459.34 Mb/s          
>>> 923.56 Mb/s
>>>
>>>
>>> bwm-ng v0.6 (probing every 10.000s), press 'h' for help
>>>  input: /proc/net/dev type: rate
>>>  /         iface                   Rx                   
>>> Tx                Total
>>>  
>>> ============================================================================== 
>>>
>>>               lo:            0.00 P/s             0.00 
>>> P/s             0.00 P/s
>>>             eth0:        51314.95 P/s         49880.20 P/s        
>>> 101195.14 P/s
>>>             eth1:        43309.85 P/s         41885.79 P/s         
>>> 85195.64 P/s
>>>  
>>> ------------------------------------------------------------------------------ 
>>>
>>>            total:        94624.80 P/s         91765.99 P/s        
>>> 186390.79 P/s
>>>
>>>
>>> mpstat -P ALL 1
>>> 18:00:05     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:06     all    0.00    0.00    0.00    0.00    1.33   13.65    
>>> 0.00    0.00   85.02
>>> 18:00:06       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:06       1    0.00    0.00    0.00    0.00    3.77   39.62    
>>> 0.00    0.00   56.60
>>> 18:00:06       2    0.00    0.00    0.00    0.00    7.92   70.30    
>>> 0.00    0.00   21.78
>>> 18:00:06       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:06       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:06       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:06       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:06       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:06     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:07     all    0.12    0.00    0.12    0.00    1.19   13.35    
>>> 0.00    0.00   85.22
>>> 18:00:07       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:07       1    0.00    0.00    0.00    0.00    2.75   42.20    
>>> 0.00    0.00   55.05
>>> 18:00:07       2    0.00    0.00    0.99    0.00    5.94   65.35    
>>> 0.00    0.00   27.72
>>> 18:00:07       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:07       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:07       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:07       6    0.00    0.00    0.99    0.00    0.00    0.00    
>>> 0.00    0.00   99.01
>>> 18:00:07       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:07     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:08     all    0.00    0.00    0.12    0.00    0.85   13.42    
>>> 0.00    0.00   85.61
>>> 18:00:08       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:08       1    0.00    0.00    0.00    0.00    1.85   38.89    
>>> 0.00    0.00   59.26
>>> 18:00:08       2    0.00    0.00    0.00    0.00    5.94   68.32    
>>> 0.00    0.00   25.74
>>> 18:00:08       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:08       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:08       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:08       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:08       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:08     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:09     all    0.00    0.00    0.00    0.00    1.33   14.18    
>>> 0.00    0.00   84.48
>>> 18:00:09       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:09       1    0.00    0.00    0.99    0.00    2.97   50.50    
>>> 0.00    0.00   45.54
>>> 18:00:09       2    0.00    0.00    0.00    0.00    6.93   65.35    
>>> 0.00    0.00   27.72
>>> 18:00:09       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:09       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:09       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:09       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:09       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:09     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:10     all    0.00    0.00    0.24    0.00    1.20   11.81    
>>> 0.00    0.00   86.75
>>> 18:00:10       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:10       1    0.00    0.00    0.95    0.00    4.76   35.24    
>>> 0.00    0.00   59.05
>>> 18:00:10       2    0.00    0.00    0.00    0.00    4.95   59.41    
>>> 0.00    0.00   35.64
>>> 18:00:10       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:10       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:10       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:10       6    0.00    0.00    0.95    0.00    0.00    0.00    
>>> 0.00    0.00   99.05
>>> 18:00:10       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:10     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:11     all    0.00    0.00    0.24    0.00    0.36    5.97    
>>> 0.00    0.00   93.43
>>> 18:00:11       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:11       1    0.00    0.00    0.97    0.00    0.97   17.48    
>>> 0.00    0.00   80.58
>>> 18:00:11       2    0.00    0.00    0.00    0.00    2.88   31.73    
>>> 0.00    0.00   65.38
>>> 18:00:11       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:11       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:11       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:11       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:11       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:11     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:12     all    0.00    0.00    0.25    0.00    0.37    5.76    
>>> 0.00    0.00   93.63
>>> 18:00:12       0    0.00    0.00    0.00    1.06    0.00    0.00    
>>> 0.00    0.00   98.94
>>> 18:00:12       1    0.00    0.00    1.92    0.00    0.00   18.27    
>>> 0.00    0.00   79.81
>>> 18:00:12       2    0.00    0.00    0.96    0.00    1.92   26.92    
>>> 0.00    0.00   70.19
>>> 18:00:12       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:12       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:12       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:12       6    0.96    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00   99.04
>>> 18:00:12       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:12     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:13     all    0.12    0.00    0.25    0.00    0.12    2.34    
>>> 0.00    0.00   97.17
>>> 18:00:13       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:13       1    0.00    0.00    0.00    0.00    0.97    7.77    
>>> 0.00    0.00   91.26
>>> 18:00:13       2    0.00    0.00    0.98    0.00    0.98   10.78    
>>> 0.00    0.00   87.25
>>> 18:00:13       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:13       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:13       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:13       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:13       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:13     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:14     all    0.00    0.00    0.37    0.00    0.37    2.33    
>>> 0.00    0.00   96.94
>>> 18:00:14       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:14       1    0.00    0.00    1.92    0.00    0.96    7.69    
>>> 0.00    0.00   89.42
>>> 18:00:14       2    0.00    0.00    0.97    0.00    0.97   10.68    
>>> 0.00    0.00   87.38
>>> 18:00:14       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:14       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:14       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:14       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:14       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:14     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:15     all    0.00    0.00    0.61    0.00    0.25    2.45    
>>> 0.00    0.00   96.69
>>> 18:00:15       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:15       1    0.00    0.00    3.85    0.00    0.96    7.69    
>>> 0.00    0.00   87.50
>>> 18:00:15       2    0.00    0.00    0.96    0.00    1.92   12.50    
>>> 0.00    0.00   84.62
>>> 18:00:15       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:15       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:15       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:15       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:15       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:15     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:16     all    0.00    0.00    0.37    0.00    0.37    1.83    
>>> 0.00    0.00   97.43
>>> 18:00:16       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:16       1    0.00    0.00    1.96    0.00    0.98    4.90    
>>> 0.00    0.00   92.16
>>> 18:00:16       2    0.00    0.00    1.00    0.00    1.00   10.00    
>>> 0.00    0.00   88.00
>>> 18:00:16       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:16       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:16       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:16       6    0.98    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00   99.02
>>> 18:00:16       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:17     all    0.00    0.00    0.25    0.00    0.25    2.73    
>>> 0.00    0.00   96.78
>>> 18:00:17       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:17       1    0.00    0.00    1.96    0.00    0.00    8.82    
>>> 0.00    0.00   89.22
>>> 18:00:17       2    0.00    0.00    0.99    0.00    1.98   11.88    
>>> 0.00    0.00   85.15
>>> 18:00:17       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:17       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:17       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:17       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:17       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:17     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:18     all    0.12    0.00    0.12    0.00    0.95   11.51    
>>> 0.00    0.00   87.31
>>> 18:00:18       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:18       1    0.00    0.00    0.00    0.00    3.03   33.33    
>>> 0.00    0.00   63.64
>>> 18:00:18       2    0.00    0.00    0.00    0.00    5.00   64.00    
>>> 0.00    0.00   31.00
>>> 18:00:18       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:18       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:18       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:18       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:18       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:18     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:19     all    0.36    0.00    0.12    0.00    1.46   14.48    
>>> 0.00    0.00   83.58
>>> 18:00:19       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:19       1    0.00    0.00    0.00    0.00    4.00   46.00    
>>> 0.00    0.00   50.00
>>> 18:00:19       2    0.00    0.00    0.99    0.00    7.92   72.28    
>>> 0.00    0.00   18.81
>>> 18:00:19       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:19       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:19       5    2.86    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00   97.14
>>> 18:00:19       6    0.95    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00   99.05
>>> 18:00:19       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:19     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:20     all    0.00    0.00    0.00    0.00    1.58   14.86    
>>> 0.00    0.00   83.56
>>> 18:00:20       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:20       1    0.00    0.00    0.00    0.00    3.96   49.50    
>>> 0.00    0.00   46.53
>>> 18:00:20       2    0.00    0.00    0.00    0.00    9.00   73.00    
>>> 0.00    0.00   18.00
>>> 18:00:20       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:20       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:20       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:20       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:20       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:20     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:21     all    0.00    0.00    0.00    0.00    1.71   14.86    
>>> 0.00    0.00   83.43
>>> 18:00:21       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:21       1    0.00    0.00    0.00    0.00    5.05   51.52    
>>> 0.00    0.00   43.43
>>> 18:00:21       2    0.00    0.00    0.00    0.00   10.00   70.00    
>>> 0.00    0.00   20.00
>>> 18:00:21       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:21       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:21       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:21       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:21       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:21     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:22     all    0.00    0.00    0.24    0.00    1.70   13.97    
>>> 0.00    0.00   84.08
>>> 18:00:22       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:22       1    0.00    0.00    0.00    0.00    4.95   42.57    
>>> 0.00    0.00   52.48
>>> 18:00:22       2    0.00    0.00    1.01    0.00    8.08   72.73    
>>> 0.00    0.00   18.18
>>> 18:00:22       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:22       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:22       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:22       6    0.00    0.00    0.95    0.00    0.00    0.00    
>>> 0.00    0.00   99.05
>>> 18:00:22       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:22     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:23     all    0.12    0.00    0.00    0.00    1.58   14.13    
>>> 0.00    0.00   84.17
>>> 18:00:23       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:23       1    0.00    0.00    0.00    0.00    5.00   45.00    
>>> 0.00    0.00   50.00
>>> 18:00:23       2    0.99    0.00    0.00    0.00    8.91   70.30    
>>> 0.00    0.00   19.80
>>> 18:00:23       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:23       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:23       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:23       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:23       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:23     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:24     all    0.00    0.00    0.00    0.00    1.34   14.13    
>>> 0.00    0.00   84.53
>>> 18:00:24       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:24       1    0.00    0.00    0.00    0.00    4.04   42.42    
>>> 0.00    0.00   53.54
>>> 18:00:24       2    0.00    0.00    0.00    0.00    6.93   73.27    
>>> 0.00    0.00   19.80
>>> 18:00:24       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:24       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:24       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:24       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:24       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:24     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:25     all    0.12    0.00    0.12    0.00    1.45   14.67    
>>> 0.00    0.00   83.64
>>> 18:00:25       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:25       1    0.00    0.00    0.99    0.00    3.96   45.54    
>>> 0.00    0.00   49.50
>>> 18:00:25       2    0.00    0.00    0.00    0.00    8.00   75.00    
>>> 0.00    0.00   17.00
>>> 18:00:25       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:25       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:25       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:25       6    0.95    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00   99.05
>>> 18:00:25       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:25     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:26     all    0.00    0.00    0.00    0.00    1.22   14.60    
>>> 0.00    0.00   84.18
>>> 18:00:26       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:26       1    0.99    0.00    0.00    0.00    2.97   50.50    
>>> 0.00    0.00   45.54
>>> 18:00:26       2    0.00    0.00    0.00    0.00    7.07   68.69    
>>> 0.00    0.00   24.24
>>> 18:00:26       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:26       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:26       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:26       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:26       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:26     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:27     all    0.00    0.00    0.00    0.00    1.59   15.02    
>>> 0.00    0.00   83.39
>>> 18:00:27       0    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:27       1    0.00    0.00    0.00    0.00    6.00   52.00    
>>> 0.00    0.00   42.00
>>> 18:00:27       2    0.00    0.00    0.00    0.00    6.93   71.29    
>>> 0.00    0.00   21.78
>>> 18:00:27       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:27       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:27       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:27       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:27       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> 18:00:27     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>>> %steal  %guest   %idle
>>> 18:00:28     all    0.00    0.00    0.00    0.00    1.33   14.56    
>>> 0.00    0.00   84.10
>>> 18:00:28       0    0.00    0.00    0.00    0.00    0.00    0.99    
>>> 0.00    0.00   99.01
>>> 18:00:28       1    0.00    0.00    0.00    0.00    4.00   50.00    
>>> 0.00    0.00   46.00
>>> 18:00:28       2    0.00    0.00    0.00    0.00    6.06   69.70    
>>> 0.00    0.00   24.24
>>> 18:00:28       3    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:28       4    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:28       5    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:28       6    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>> 18:00:28       7    0.00    0.00    0.00    0.00    0.00    0.00    
>>> 0.00    0.00  100.00
>>>
>>> at the same time top shows:
>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>   28 root      15  -5     0    0    0 S    2  0.0   0:16.60 events/1
>>>   29 root      15  -5     0    0    0 S    2  0.0   0:15.62 events/2
>>>   10 root      15  -5     0    0    0 S    2  0.0   0:41.01 ksoftirqd/2
>>>    7 root      15  -5     0    0    0 S    1  0.0   0:31.17 ksoftirqd/1
>>>   30 root      15  -5     0    0    0 S    1  0.0   0:03.00 events/3
>>> 6545 root      20   0 11112 3244 1780 S    0  0.0   0:08.69 oprofiled
>>> 6845 root      20   0 19084 1384 1008 R    0  0.0   0:00.07 top
>>>    1 root      20   0  3812  612  520 S    0  0.0   0:01.00 init
>>>    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd
>>>    3 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/0
>>>    4 root      15  -5     0    0    0 S    0  0.0   0:01.04 ksoftirqd/0
>>>    5 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/0
>>>    6 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/1
>>>    8 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/1
>>>    9 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/2
>>>   11 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/2
>>>   12 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/3
>>>   13 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3
>>>   14 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/3
>>>   15 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/4
>>>   16 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/4
>>>   17 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/4
>>>   18 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/5
>>>   19 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/5
>>>   20 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/5
>>>   21 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/6
>>>   22 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/6
>>>   23 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/6
>>>   24 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/7
>>>   25 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/7
>>>   26 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/7
>>>   27 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/0
>>>   31 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/4
>>>   32 root      15  -5     0    0    0 S    0  0.0   0:00.01 events/5
>>>   33 root      15  -5     0    0    0 S    0  0.0   0:00.02 events/6
>>>   34 root      15  -5     0    0    0 S    0  0.0   0:00.01 events/7
>>>   35 root      15  -5     0    0    0 S    0  0.0   0:00.01 khelper
>>>   41 root      15  -5     0    0    0 S    0  0.0   0:00.00 async/mgr
>>>  159 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/0
>>>  160 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/1
>>>  161 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/2
>>>  162 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/3
>>>  163 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/4
>>>  164 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/5
>>>  165 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/6
>>>  166 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/7
>>>  167 root      15  -5     0    0    0 S    0  0.0   0:00.00 kacpid
>>>  168 root      15  -5     0    0    0 S    0  0.0   0:00.00 
>>> kacpi_notify
>>>  246 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/0
>>>  247 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/1
>>>
>>> and oprofile output:
>>> CPU: Core 2, speed 3000.08 MHz (estimated)
>>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with 
>>> a unit mask of 0x00 (Unhalted core cycles) count 100000
>>> samples  %        image name               app name                 
>>> symbol name
>>> 11138304 12.3598  vmlinux                  vmlinux                  
>>> rb_next
>>> 7714260   8.5602  vmlinux                  vmlinux                  
>>> e1000_clean
>>> 5380466   5.9705  vmlinux                  vmlinux                  
>>> e1000_intr_msi
>>> 4881445   5.4168  vmlinux                  vmlinux                  
>>> u32_classify
>>> 3977072   4.4132  vmlinux                  vmlinux                  
>>> _raw_spin_lock
>>> 3471611   3.8523  vmlinux                  vmlinux                  
>>> hfsc_dequeue
>>> 3103493   3.4438  vmlinux                  vmlinux                  
>>> hfsc_enqueue
>>> 2636824   2.9260  vmlinux                  vmlinux                  
>>> delay_tsc
>>> 2464293   2.7345  vmlinux                  vmlinux                  
>>> e1000_xmit_frame
>>> 2035779   2.2590  vmlinux                  vmlinux                  
>>> e1000_clean_tx_irq
>>> 1947231   2.1608  vmlinux                  vmlinux                  
>>> mwait_idle
>>> 1919540   2.1300  vmlinux                  vmlinux                  
>>> ipt_do_table
>>> 1908544   2.1178  vmlinux                  vmlinux                  
>>> e1000_clean_rx_irq
>>> 1865996   2.0706  vmlinux                  vmlinux                  
>>> update_vf
>>> 1829293   2.0299  vmlinux                  vmlinux                  
>>> rb_erase
>>> 1777847   1.9728  vmlinux                  vmlinux                  
>>> ip_route_input
>>> 1430773   1.5877  vmlinux                  vmlinux                  
>>> rb_insert_color
>>> 1329674   1.4755  vmlinux                  vmlinux                  
>>> __alloc_skb
>>> 1091678   1.2114  vmlinux                  vmlinux                  
>>> sfq_enqueue
>>> 985105    1.0931  vmlinux                  vmlinux                  
>>> rb_first
>>> 944928    1.0486  vmlinux                  vmlinux                  
>>> sfq_dequeue
>>> 756452    0.8394  vmlinux                  vmlinux                  
>>> read_tsc
>>> 728703    0.8086  vmlinux                  vmlinux                  
>>> nf_iterate
>>> 665551    0.7385  vmlinux                  vmlinux                  
>>> ip_forward
>>> 662616    0.7353  vmlinux                  vmlinux                  
>>> dev_queue_xmit
>>> 648919    0.7201  vmlinux                  vmlinux                  
>>> __qdisc_run
>>> 615592    0.6831  vmlinux                  vmlinux                  
>>> ip_rcv
>>> 589743    0.6544  vmlinux                  vmlinux                  
>>> apic_timer_interrupt
>>> 550581    0.6110  vmlinux                  vmlinux                  
>>> irq_entries_start
>>> 536172    0.5950  vmlinux                  vmlinux                  
>>> getnstimeofday
>>> 527467    0.5853  vmlinux                  vmlinux                  
>>> rb_last
>>> 508305    0.5640  vmlinux                  vmlinux                  
>>> skb_release_data
>>> 506763    0.5623  vmlinux                  vmlinux                  
>>> kfree
>>> 506269    0.5618  vmlinux                  vmlinux                  
>>> dst_release
>>> 464412    0.5153  vmlinux                  vmlinux                  
>>> __hrtimer_start_range_ns
>>> 456354    0.5064  vmlinux                  vmlinux                  
>>> e1000_alloc_rx_buffers
>>> 435339    0.4831  vmlinux                  vmlinux                  
>>> dev_hard_start_xmit
>>> 371439    0.4122  vmlinux                  vmlinux                  
>>> local_bh_enable
>>> 369266    0.4098  vmlinux                  vmlinux                  
>>> __rb_rotate_left
>>> 361627    0.4013  vmlinux                  vmlinux                  
>>> net_tx_action
>>> 357797    0.3970  vmlinux                  vmlinux                  
>>> rtsc_y2x
>>> 350595    0.3890  vmlinux                  vmlinux                  
>>> _raw_spin_unlock
>>> 349359    0.3877  vmlinux                  vmlinux                  
>>> kmem_cache_alloc
>>> 336255    0.3731  vmlinux                  vmlinux                  
>>> skb_dma_map
>>> 328809    0.3649  vmlinux                  vmlinux                  
>>> netif_receive_skb
>>> 315826    0.3505  vmlinux                  vmlinux                  
>>> fn_trie_lookup
>>> 313065    0.3474  vmlinux                  vmlinux                  
>>> __kmalloc
>>> 307684    0.3414  vmlinux                  vmlinux                  
>>> __do_softirq
>>> 298676    0.3314  vmlinux                  vmlinux                  
>>> skb_pull
>>> 292352    0.3244  vmlinux                  vmlinux                  
>>> __napi_complete
>>> 288878    0.3206  vmlinux                  vmlinux                  
>>> get_next_timer_interrupt
>>> 280449    0.3112  vmlinux                  vmlinux                  
>>> ip_finish_output
>>> 264482    0.2935  vmlinux                  vmlinux                  
>>> consume_skb
>>> 251622    0.2792  vmlinux                  vmlinux                  
>>> __remove_hrtimer
>>> 227178    0.2521  vmlinux                  vmlinux                  
>>> tc_classify_compat
>>> 226224    0.2510  vmlinux                  vmlinux                  
>>> handle_edge_irq
>>>
>>>
>>>
>>> proces that this machine is running is only quagga (zebra + bgpd + 
>>> oprofiled)
>>>
>>>
>>> Regards
>>> Paweł Staszewski
>>>
>>>
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>> the body of a message to [email-masked]
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to [email-masked]
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to [email-masked]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>