Subject: Re: weird problem

From: Eric Dumazet <[email-masked]>
Date: Thu, 25 Jun 2009 22:18:33 +0200

Paweł Staszewski a écrit :
> Ok
> 
> After this day of observation im near 100% sure that this cpu load is
> made by route cahce flushes
> When route cache increase to its "net.ipv4.route.gc_thresh" size or is
> near that size
> system is starting to drop some routes from cache then cpu load is
> increase from 2% to near 80%
> after cleaning / flush cache when cache is filling cpu load is again
> normal 2%
> 
> Someone know how to resolve this ?
> on kernels < 2.6.29 i don't see this, all start after upgrade from
> 2.6.28 to 2.6.29 - then i try 2.6.29.1 , 2.6.29.3 and 2.6.30 and on all
> this kernels >= 2.6.29 problem with cpu load is the same.
> 
> I can minimize this cpu fluctuations by changing of route cache /proc
> parameters but the best result for my router was
> 
> 15 sec of 2% cpu
> and after
> 15sec of 80% cpu
> 
> 
> Regards
> Pawel Staszewski

I believe this is known 2.6.29 regressions

Following two commits should correct the problem you have

Your best bet would be to try 2.6.31-rc1, and tell us if this recent kernel
is ok on your machine ?

commit 1ddbcb005c395518c2cd0df504cff3d4b5c85853
Author: Eric Dumazet <[email-masked]>
Date:   Tue May 19 20:14:28 2009 +0000

    net: fix rtable leak in net/ipv4/route.c

    Alexander V. Lukyanov found a regression in 2.6.29 and made a complete
    analysis found in http://bugzilla.kernel.org/show_bug.cgi?id=13339
    Quoted here because its a perfect one :

    begin_of_quotation
     2.6.29 patch has introduced flexible route cache rebuilding. Unfortunately the
     patch has at least one critical flaw, and another problem.

     rt_intern_hash calculates rthi pointer, which is later used for new entry
     insertion. The same loop calculates cand pointer which is used to clean the
     list. If the pointers are the same, rtable leak occurs, as first the cand is
     removed then the new entry is appended to it.

     This leak leads to unregister_netdevice problem (usage count > 0).

     Another problem of the patch is that it tries to insert the entries in certain
     order, to facilitate counting of entries distinct by all but QoS parameters.
     Unfortunately, referencing an existing rtable entry moves it to list beginning,
     to speed up further lookups, so the carefully built order is destroyed.

     For the first problem the simplest patch it to set rthi=0 when rthi==cand, but
     it will also destroy the ordering.
    end_of_quotation

    Problematic commit is 1080d709fb9d8cd4392f93476ee46a9d6ea05a5b
    (net: implement emergency route cache rebulds when gc_elasticity is exceeded)

    Trying to keep dst_entries ordered is too complex and breaks the fact that
    order should depend on the frequency of use for garbage collection.

    A possible fix is to make rt_intern_hash() simpler, and only makes
    rt_check_expire() a litle bit smarter, being able to cope with an arbitrary
    entries order. The added loop is running on cache hot data, while cpu
    is prefetching next object, so should be unnoticied.

    Reported-and-analyzed-by: Alexander V. Lukyanov <[email-masked]>

commit cf8da764fc6959b7efb482f375dfef9830e98205
Author: Eric Dumazet <[email-masked]>
Date:   Tue May 19 18:54:22 2009 +0000

    net: fix length computation in rt_check_expire()

    rt_check_expire() computes average and standard deviation of chain lengths,
    but not correclty reset length to 0 at beginning of each chain.
    This probably gives overflows for sum2 (and sum) on loaded machines instead
    of meaningful results.

    Signed-off-by: Eric Dumazet <[email-masked]>
    Acked-by: Neil Horman <[email-masked]>
    Signed-off-by: David S. Miller <[email-masked]>

> 
> 
> 
> Paweł Staszewski pisze:
>> Another info
>>
>> i look at some stats from rtstats and mpstat and try to join them
>> together and here is output:
>> 19:06:50     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>> %steal  %guest   %idle        rt_cache entries
>> 19:07:01     all    0.00    0.00    0.12    0.00    0.59   12.00   
>> 0.00    0.00   87.29        77726
>> 19:07:02     all    0.00    0.00    0.12    0.00    0.49    6.11   
>> 0.00    0.00   93.28        80127
>> 19:07:03     all    0.00    0.00    0.00    0.00    0.11    1.06   
>> 0.00    0.00   98.84        82596
>> 19:07:04     all    0.00    0.00    0.16    0.00    0.00    2.07   
>> 0.00    0.00   97.77        85147
>> 19:07:05     all    0.00    0.00    0.00    0.00    0.12    1.81   
>> 0.00    0.00   98.07        87641
>> 19:07:06     all    0.00    0.00    0.00    0.00    0.27    1.50   
>> 0.00    0.00   98.22        90085
>> 19:07:07     all    0.00    0.00    0.00    0.00    0.23    1.72   
>> 0.00    0.00   98.05        93586
>> 19:07:08     all    0.00    0.00    0.00    0.00    0.00    1.56   
>> 0.00    0.00   98.44        96865
>> 19:07:09     all    0.00    0.00    0.00    0.00    0.00    1.45   
>> 0.00    0.00   98.55        99803
>> 19:07:10     all    0.00    0.00    0.00    0.00    0.27    1.65   
>> 0.00    0.00   98.08        102123
>> 19:07:11     all    0.00    0.00    0.10    0.00    0.30    4.07   
>> 0.00    0.00   95.54        104618
>> 19:07:12     all    0.00    0.00    0.00    0.00    1.30   18.96   
>> 0.00    0.00   79.74        105524
>> 19:07:13     all    0.00    0.00    0.00    0.00    0.84   13.58   
>> 0.00    0.00   85.58        101085
>> 19:07:14     all    0.00    0.00    0.00    0.00    1.36   20.45   
>> 0.00    0.00   78.18        97518
>> 19:07:15     all    0.00    0.00    0.00    0.00    0.98   15.89   
>> 0.00    0.00   83.13        94308
>> 19:07:16     all    0.00    0.00    0.00    0.00    1.32   17.00   
>> 0.00    0.00   81.69        91811
>> 19:07:17     all    0.00    0.00    0.00    0.00    0.67   14.57   
>> 0.00    0.00   84.75        92278
>> 19:07:18     all    0.00    0.00    0.00    0.00    1.51   17.40   
>> 0.00    0.00   81.10        90936
>> 19:07:19     all    0.00    0.00    0.00    0.00    1.24   13.46   
>> 0.00    0.00   85.29        87951
>> 19:07:20     all    0.00    0.00    0.14    0.00    0.99   18.75   
>> 0.00    0.00   80.11        85071
>> 19:07:21     all    0.00    0.00    0.00    0.00    0.84   15.20   
>> 0.00    0.00   83.96        82477
>> 19:07:22     all    0.40    0.00    0.00    0.00    1.71   17.65   
>> 0.00    0.00   80.24        80301
>> 19:07:23     all    0.00    0.00    0.11    0.00    1.08   13.33   
>> 0.00    0.00   85.48        78480
>> 19:07:24     all    0.00    0.00    0.00    0.00    0.87   19.25   
>> 0.00    0.00   79.88        76880
>>
>> When cpu load is rising rt cache entries are decreasing
>> and when cpu load is from 1 do 3% rt cache entries are increasing
>>
>> some sysctl-s that i have and are route cache specific:
>> net.ipv4.route.gc_thresh = 190536
>> net.ipv4.route.gc_min_interval = 0
>> net.ipv4.route.gc_min_interval_ms = 0
>> net.ipv4.route.gc_timeout = 15
>> net.ipv4.route.gc_interval = 1
>> net.ipv4.route.gc_elasticity = 4
>>
>>
>>
>>
>> Paweł Staszewski pisze:
>>> oprofile output per cpu
>>> CPU: Core 2, speed 3000.08 MHz (estimated)
>>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
>>> unit mask of 0x00 (Unhalted core cycles) count 100000
>>> Samples on CPU 0
>>> Samples on CPU 1
>>> Samples on CPU 2
>>> Samples on CPU 3
>>> Samples on CPU 4
>>> Samples on CPU 5
>>> Samples on CPU 6
>>> Samples on CPU 7
>>> Samples on CPU all
>>> samples  %        samples  %        samples  %        samples 
>>> %        samples  %        samples  %        samples  %       
>>> samples  %        samples  %
>>>    image name               app name                 symbol name
>>> 29520    21.7362  0              0  0              0  0             
>>> 0  0              0  0              0  9571     34.0254 
>>> 0              0  4701      0.22
>>> 55  oprofiled                oprofiled                (no symbols)
>>> 18441    13.5785  0              0  0              0  0             
>>> 0  0              0  0              0  0              0 
>>> 0              0  2666      0.12
>>> 79  vmlinux                  vmlinux                  rt_worker_func
>>> 18428    13.5690  132089    2.1756  135138    1.4238  21735   
>>> 42.7703  775       1.9819  893       5.3180  847       3.0111 
>>> 1056      3.3130  37949     1.82
>>> 01  vmlinux                  vmlinux                  mwait_idle
>>> 15797    11.6317  212       0.0035  2879      0.0303  17       
>>> 0.0335  2         0.0051  4         0.0238  14        0.0498 
>>> 185       0.5804  2895      0.13
>>> 89  vmlinux                  vmlinux                  free_block
>>> 9034      6.6519  1218      0.0201  2727      0.0287  0             
>>> 0  0              0  0              0  0              0 
>>> 0              0  2004      0.09
>>> 61  vmlinux                  vmlinux                  dst_destroy
>>> 7113      5.2375  632       0.0104  483       0.0051  6615    
>>> 13.0170  543       1.3886  461       2.7454  1951      6.9359 
>>> 1274      3.9970  4073      0.19
>>> 54  vmlinux                  vmlinux                 
>>> mutex_spin_on_owner
>>> 4382      3.2266  652       0.0107  1880      0.0198  2        
>>> 0.0039  0              0  0              0  3         0.0107 
>>> 7         0.0220  1050      0.05
>>> 04  vmlinux                  vmlinux                  __call_rcu
>>> 4337      3.1934  418       0.0069  830       0.0087  58       
>>> 0.1141  18        0.0460  8         0.0476  20        0.0711 
>>> 39        0.1224  917       0.04
>>> 40  vmlinux                  vmlinux                 
>>> __rcu_process_callbacks
>>> 3991      2.9387  51       8.4e-04  325       0.0034  136      
>>> 0.2676  2         0.0051  1         0.0060  1034      3.6759 
>>> 205       0.6432  1110      0.05
>>> 32  vmlinux                  vmlinux                 
>>> copy_user_generic_string
>>> 3292      2.4240  275838    4.5432  391639    4.1263  869      
>>> 1.7100  431       1.1022  350       2.0843  673       2.3925 
>>> 1069      3.3538  89939     4.31
>>> 37  vmlinux                  vmlinux                  _raw_spin_lock
>>> 1909      1.4056  79        0.0013  376       0.0040  0             
>>> 0  0              0  0              0  0              0 
>>> 0              0  348       0.01
>>> 67  vmlinux                  vmlinux                  dst_rcu_free
>>> 1461      1.0758  0              0  1163      0.0123  0             
>>> 0  0              0  0              0  0              0 
>>> 0              0  0
>>> 0  vim                      vim                      (no symbols)
>>> 1122      0.8262  11514     0.1896  6590      0.0694  1857     
>>> 3.6542  7638     19.5325  2138     12.7323  1213      4.3123 
>>> 4025     12.6278  9676      0.46
>>> 41  libc-2.8.so              libc-2.8.so              (no symbols)
>>> 1035      0.7621  9027      0.1487  22216     0.2341  4        
>>> 0.0079  0              0  0              0  5         0.0178 
>>> 15        0.0471  4310      0.20
>>> 67  vmlinux                  vmlinux                  kmem_cache_free
>>> 1016      0.7481  218       0.0036  465       0.0049  0             
>>> 0  0              0  0              0  0              0 
>>> 0              0  255       0.01
>>> 22  vmlinux                  vmlinux                  ipv4_dst_destroy
>>> 951       0.7002  4290      0.0707  5514      0.0581  3283     
>>> 6.4603  867       2.2172  673       4.0079  1096      3.8963 
>>> 1316      4.1288  2593      0.12
>>> 44  vmlinux                  vmlinux                  tg_shares_up
>>> 708       0.5213  104       0.0017  279       0.0029  0             
>>> 0  0              0  0              0  0              0 
>>> 0              0  159       0.00
>>> 76  vmlinux                  vmlinux                  call_rcu_bh
>>> 581       0.4278  21812     0.3593  38364     0.4042  68       
>>> 0.1338  23        0.0588  23        0.1370  65        0.2311 
>>> 96        0.3012  8000      0.38
>>> 37  vmlinux                  vmlinux                  _raw_spin_unlock
>>> 563       0.4145  1027      0.0169  2528      0.0266  0             
>>> 0  0              0  0              0  0              0  3        
>>> 0.0094  545       0.02
>>> 61  vmlinux                  vmlinux                  local_bh_enable_ip
>>> 509       0.3748  0              0  3        3.2e-05  0             
>>> 0  0              0  0              0  4         0.0142  3        
>>> 0.0094  76        0.00
>>> 36  vmlinux                  vmlinux                  can_dirty
>>> 430       0.3166  74        0.0012  148       0.0016  25       
>>> 0.0492  13        0.0332  12        0.0715  9         0.0320 
>>> 12        0.0376  96        0.00
>>> 46  vmlinux                  vmlinux                 
>>> rcu_process_callbacks
>>> 372       0.2739  0              0  0              0  0             
>>> 0  0              0  0              0  3         0.0107  2        
>>> 0.0063  48        0.00
>>> 23  vmlinux                  vmlinux                  do_journal_end
>>> 347       0.2555  9991      0.1646  16066     0.1693  143      
>>> 0.2814  66        0.1688  25        0.1489  46        0.1635 
>>> 71        0.2228  3514      0.16
>>> 85  vmlinux                  vmlinux                  rb_get_reader_page
>>> 346       0.2548  952       0.0157  566       0.0060  871      
>>> 1.7140  321       0.8209  174       1.0362  324       1.1518 
>>> 269       0.8439  655       0.03
>>> 14  vmlinux                  vmlinux                  find_busiest_group
>>> 275       0.2025  1194      0.0197  1228      0.0129  1019     
>>> 2.0052  203       0.5191  151       0.8992  290       1.0310 
>>> 653       2.0487  799       0.03
>>> 83  vmlinux                  vmlinux                  find_next_bit
>>> 238       0.1752  3053      0.0503  4711      0.0496  116      
>>> 0.2283  82        0.2097  54        0.3216  101       0.3591 
>>> 101       0.3169  1174      0.05
>>> 63  oprofile                 oprofile                 (no symbols)
>>> 221       0.1627  0              0  0              0  0             
>>> 0  0              0  0              0  0              0 
>>> 0              0  18       8.6e-
>>> 04  vmlinux                  vmlinux                  ahci_interrupt
>>> 212       0.1561  38335     0.6314  60370     0.6361  329      
>>> 0.6474  165       0.4220  136       0.8099  133       0.4728 
>>> 205       0.6432  12821     0.61
>>> 49  vmlinux                  vmlinux                 
>>> apic_timer_interrupt
>>> 210       0.1546  117       0.0019  1001      0.0105  2        
>>> 0.0039  0              0  0              0  55        0.1955 
>>> 16        0.0502  270       0.01
>>> 30  vmlinux                  vmlinux                  memmove
>>> 207       0.1524  0              0  7        7.4e-05  34       
>>> 0.0669  0              0  0              0  93        0.3306 
>>> 62        0.1945  214       0.01
>>> 03  vmlinux                  vmlinux                  search_by_key
>>> 204       0.1502  26603     0.4382  25086     0.2643  45       
>>> 0.0886  18        0.0460  19        0.1131  19        0.0675 
>>> 25        0.0784  6580      0.31
>>> 56  vmlinux                  vmlinux                  __do_softirq
>>> 170       0.1252  2618      0.0431  3493      0.0368  69       
>>> 0.1358  42        0.1074  51        0.3037  64        0.2275 
>>> 87        0.2729  856       0.04
>>> 11  vmlinux                  vmlinux                  __schedule
>>> 151       0.1112  7590      0.1250  8909      0.0939  273      
>>> 0.5372  86        0.2199  73        0.4347  99        0.3519 
>>> 142       0.4455  2114      0.10
>>> 14  vmlinux                  vmlinux                  sched_clock_cpu
>>> 142       0.1046  200       0.0033  165       0.0017  65       
>>> 0.1279  23        0.0588  20        0.1191  15        0.0533 
>>> 36        0.1129  94        0.00
>>> 45  vmlinux                  vmlinux                  raise_softirq
>>> 141       0.1038  8048      0.1326  11386     0.1200  2        
>>> 0.0039  0              0  0              0  2         0.0071 
>>> 4         0.0125  2589      0.12
>>> 42  vmlinux                  vmlinux                  __phys_addr
>>> 141       0.1038  23726     0.3908  24837     0.2617  89       
>>> 0.1751  58        0.1483  55        0.3275  66        0.2346 
>>> 62        0.1945  5398      0.25
>>> 89  vmlinux                  vmlinux                 
>>> tick_nohz_stop_sched_tick
>>> 139       0.1023  43230     0.7120  68500     0.7217  105      
>>> 0.2066  66        0.1688  86        0.5121  100       0.3555 
>>> 100       0.3137  13725     0.65
>>> 83  vmlinux                  vmlinux                  getnstimeofday
>>> 136       0.1001  0              0  0              0  0             
>>> 0  0              0  0              0  0              0 
>>> 0              0  36        0.00
>>> 17  vmlinux                  vmlinux                 
>>> remove_journal_hash
>>> 133       0.0979  8624      0.1420  9518      0.1003  217      
>>> 0.4270  68        0.1739  64        0.3811  79        0.2808 
>>> 106       0.3326  2348      0.11
>>> 26  vmlinux                  vmlinux                  native_sched_clock
>>> 129       0.0950  270       0.0044  233       0.0025  355      
>>> 0.6986  93        0.2378  60        0.3573  102       0.3626 
>>> 105       0.3294  214       0.01
>>> 03  vmlinux                  vmlinux                  rebalance_domains
>>> 117       0.0861  0              0  36       3.8e-04  57       
>>> 0.1122  0              0  10        0.0596  66        0.2346 
>>> 60        0.1882  220       0.01
>>> 06  vmlinux                  vmlinux                  find_get_page
>>> 112       0.0825  287       0.0047  397       0.0042  121      
>>> 0.2381  89        0.2276  63        0.3752  85        0.3022 
>>> 114       0.3577  220       0.01
>>> 06  vmlinux                  vmlinux                  scheduler_tick
>>> 107       0.0788  13702     0.2257  16191     0.1706  102      
>>> 0.2007  28        0.0716  44        0.2620  41        0.1458 
>>> 66        0.2071  3890      0.18
>>> 66  vmlinux                  vmlinux                 
>>> tick_nohz_stop_idle
>>> 105       0.0773  4277      0.0704  7002      0.0738  53       
>>> 0.1043  26        0.0665  24        0.1429  24        0.0853 
>>> 38        0.1192  1561      0.07
>>> 49  vmlinux                  vmlinux                 
>>> ring_buffer_consume
>>> 103       0.0758  0              0  2        2.1e-05  5        
>>> 0.0098  0              0  0              0  20        0.0711 
>>> 13        0.0408  61        0.00
>>> 29  vmlinux                  vmlinux                  __find_get_block
>>> 103       0.0758  49251     0.8112  85999     0.9061  166      
>>> 0.3267  105       0.2685  97        0.5777  122       0.4337 
>>> 138       0.4330  17108     0.82
>>> 06  vmlinux                  vmlinux                  read_tsc
>>> 98        0.0722  17406     0.2867  40897     0.4309  15       
>>> 0.0295  2         0.0051  2         0.0119  62        0.2204 
>>> 51        0.1600  7913      0.37
>>> 95  vmlinux                  vmlinux                  kmem_cache_alloc
>>> 98        0.0722  14817     0.2440  21291     0.2243  41       
>>> 0.0807  49        0.1253  48        0.2859  62        0.2204 
>>> 64        0.2008  4582      0.21
>>> 98  vmlinux                  vmlinux                  ktime_get_ts
>>> 94        0.0692  0              0  1        1.1e-05  2        
>>> 0.0039  0              0  0              0  7         0.0249 
>>> 6         0.0188  16       7.7e-
>>> 04  vmlinux                  vmlinux                 
>>> __find_get_block_slow
>>> 92        0.0677  1161      0.0191  1411      0.0149  41       
>>> 0.0807  39        0.0997  40        0.2382  53        0.1884 
>>> 49        0.1537  408       0.01
>>> 96  vmlinux                  vmlinux                 
>>> select_nohz_load_balancer
>>> 88        0.0648  11160     0.1838  54990     0.5794  69       
>>> 0.1358  43        0.1100  35        0.2084  661       2.3499 
>>> 55        0.1726  4513      0.21
>>> 65  vmlinux                  vmlinux                 
>>> get_next_timer_interrupt
>>> 75        0.0552  0              0  0              0  0             
>>> 0  0              0  0              0  0              0 
>>> 0              0  6        2.9e-
>>> 04  vmlinux                  vmlinux                  make_request
>>> 69        0.0508  803       0.0132  1048      0.0110  35       
>>> 0.0689  26        0.0665  15        0.0893  24        0.0853 
>>> 58        0.1820  334       0.01
>>> --More--
>>>
>>> Paweł Staszewski pisze:
>>>> I dont know where to put this message so i start here because server
>>>> that i see this funny thing is make only networking
>>>> Routing (bgpd) / traffic management (iproute2)
>>>>
>>>> This starts on kernels >= 2.6.29
>>>>
>>>> On server i have almost constant traffic that is about 300Mbit/s RX
>>>> and 300Mbit/s TX
>>>> Server have two interfaces:
>>>> 04:00.0 Ethernet controller: Intel Corporation 82573E Gigabit
>>>> Ethernet Controller (Copper) (rev 03)
>>>> 05:00.0 Ethernet controller: Intel Corporation 82573L Gigabit
>>>> Ethernet Controller
>>>>
>>>> iBGP peer < - > [ eth0 - forward + shaping - eth1 ] < - >  eBGP
>>>>
>>>>
>>>> And what is weird is CPU Load
>>>> it fluctuate from 10% to almost 80% - but traffic is  constant :
>>>> bwm-ng v0.6 (probing every 10.000s), press 'h' for help
>>>>  input: /proc/net/dev type: rate
>>>>  |         iface                   Rx                  
>>>> Tx                Total
>>>>  
>>>> ==============================================================================
>>>>
>>>>               lo:           0.00  b/s            0.00 
>>>> b/s            0.00  b/s
>>>>             eth0:         200.22 Mb/s          266.74 Mb/s         
>>>> 466.96 Mb/s
>>>>             eth1:         264.00 Mb/s          192.59 Mb/s         
>>>> 456.59 Mb/s
>>>>  
>>>> ------------------------------------------------------------------------------
>>>>
>>>>            total:         464.22 Mb/s          459.34 Mb/s         
>>>> 923.56 Mb/s
>>>>
>>>>
>>>> bwm-ng v0.6 (probing every 10.000s), press 'h' for help
>>>>  input: /proc/net/dev type: rate
>>>>  /         iface                   Rx                  
>>>> Tx                Total
>>>>  
>>>> ==============================================================================
>>>>
>>>>               lo:            0.00 P/s             0.00
>>>> P/s             0.00 P/s
>>>>             eth0:        51314.95 P/s         49880.20 P/s       
>>>> 101195.14 P/s
>>>>             eth1:        43309.85 P/s         41885.79 P/s        
>>>> 85195.64 P/s
>>>>  
>>>> ------------------------------------------------------------------------------
>>>>
>>>>            total:        94624.80 P/s         91765.99 P/s       
>>>> 186390.79 P/s
>>>>
>>>>
>>>> mpstat -P ALL 1
>>>> 18:00:05     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:06     all    0.00    0.00    0.00    0.00    1.33   13.65   
>>>> 0.00    0.00   85.02
>>>> 18:00:06       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:06       1    0.00    0.00    0.00    0.00    3.77   39.62   
>>>> 0.00    0.00   56.60
>>>> 18:00:06       2    0.00    0.00    0.00    0.00    7.92   70.30   
>>>> 0.00    0.00   21.78
>>>> 18:00:06       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:06       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:06       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:06       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:06       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:06     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:07     all    0.12    0.00    0.12    0.00    1.19   13.35   
>>>> 0.00    0.00   85.22
>>>> 18:00:07       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:07       1    0.00    0.00    0.00    0.00    2.75   42.20   
>>>> 0.00    0.00   55.05
>>>> 18:00:07       2    0.00    0.00    0.99    0.00    5.94   65.35   
>>>> 0.00    0.00   27.72
>>>> 18:00:07       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:07       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:07       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:07       6    0.00    0.00    0.99    0.00    0.00    0.00   
>>>> 0.00    0.00   99.01
>>>> 18:00:07       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:07     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:08     all    0.00    0.00    0.12    0.00    0.85   13.42   
>>>> 0.00    0.00   85.61
>>>> 18:00:08       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:08       1    0.00    0.00    0.00    0.00    1.85   38.89   
>>>> 0.00    0.00   59.26
>>>> 18:00:08       2    0.00    0.00    0.00    0.00    5.94   68.32   
>>>> 0.00    0.00   25.74
>>>> 18:00:08       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:08       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:08       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:08       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:08       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:08     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:09     all    0.00    0.00    0.00    0.00    1.33   14.18   
>>>> 0.00    0.00   84.48
>>>> 18:00:09       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:09       1    0.00    0.00    0.99    0.00    2.97   50.50   
>>>> 0.00    0.00   45.54
>>>> 18:00:09       2    0.00    0.00    0.00    0.00    6.93   65.35   
>>>> 0.00    0.00   27.72
>>>> 18:00:09       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:09       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:09       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:09       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:09       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:09     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:10     all    0.00    0.00    0.24    0.00    1.20   11.81   
>>>> 0.00    0.00   86.75
>>>> 18:00:10       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:10       1    0.00    0.00    0.95    0.00    4.76   35.24   
>>>> 0.00    0.00   59.05
>>>> 18:00:10       2    0.00    0.00    0.00    0.00    4.95   59.41   
>>>> 0.00    0.00   35.64
>>>> 18:00:10       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:10       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:10       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:10       6    0.00    0.00    0.95    0.00    0.00    0.00   
>>>> 0.00    0.00   99.05
>>>> 18:00:10       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:10     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:11     all    0.00    0.00    0.24    0.00    0.36    5.97   
>>>> 0.00    0.00   93.43
>>>> 18:00:11       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:11       1    0.00    0.00    0.97    0.00    0.97   17.48   
>>>> 0.00    0.00   80.58
>>>> 18:00:11       2    0.00    0.00    0.00    0.00    2.88   31.73   
>>>> 0.00    0.00   65.38
>>>> 18:00:11       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:11       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:11       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:11       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:11       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:11     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:12     all    0.00    0.00    0.25    0.00    0.37    5.76   
>>>> 0.00    0.00   93.63
>>>> 18:00:12       0    0.00    0.00    0.00    1.06    0.00    0.00   
>>>> 0.00    0.00   98.94
>>>> 18:00:12       1    0.00    0.00    1.92    0.00    0.00   18.27   
>>>> 0.00    0.00   79.81
>>>> 18:00:12       2    0.00    0.00    0.96    0.00    1.92   26.92   
>>>> 0.00    0.00   70.19
>>>> 18:00:12       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:12       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:12       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:12       6    0.96    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00   99.04
>>>> 18:00:12       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:12     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:13     all    0.12    0.00    0.25    0.00    0.12    2.34   
>>>> 0.00    0.00   97.17
>>>> 18:00:13       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:13       1    0.00    0.00    0.00    0.00    0.97    7.77   
>>>> 0.00    0.00   91.26
>>>> 18:00:13       2    0.00    0.00    0.98    0.00    0.98   10.78   
>>>> 0.00    0.00   87.25
>>>> 18:00:13       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:13       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:13       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:13       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:13       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:13     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:14     all    0.00    0.00    0.37    0.00    0.37    2.33   
>>>> 0.00    0.00   96.94
>>>> 18:00:14       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:14       1    0.00    0.00    1.92    0.00    0.96    7.69   
>>>> 0.00    0.00   89.42
>>>> 18:00:14       2    0.00    0.00    0.97    0.00    0.97   10.68   
>>>> 0.00    0.00   87.38
>>>> 18:00:14       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:14       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:14       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:14       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:14       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:14     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:15     all    0.00    0.00    0.61    0.00    0.25    2.45   
>>>> 0.00    0.00   96.69
>>>> 18:00:15       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:15       1    0.00    0.00    3.85    0.00    0.96    7.69   
>>>> 0.00    0.00   87.50
>>>> 18:00:15       2    0.00    0.00    0.96    0.00    1.92   12.50   
>>>> 0.00    0.00   84.62
>>>> 18:00:15       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:15       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:15       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:15       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:15       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:15     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:16     all    0.00    0.00    0.37    0.00    0.37    1.83   
>>>> 0.00    0.00   97.43
>>>> 18:00:16       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:16       1    0.00    0.00    1.96    0.00    0.98    4.90   
>>>> 0.00    0.00   92.16
>>>> 18:00:16       2    0.00    0.00    1.00    0.00    1.00   10.00   
>>>> 0.00    0.00   88.00
>>>> 18:00:16       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:16       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:16       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:16       6    0.98    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00   99.02
>>>> 18:00:16       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:17     all    0.00    0.00    0.25    0.00    0.25    2.73   
>>>> 0.00    0.00   96.78
>>>> 18:00:17       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:17       1    0.00    0.00    1.96    0.00    0.00    8.82   
>>>> 0.00    0.00   89.22
>>>> 18:00:17       2    0.00    0.00    0.99    0.00    1.98   11.88   
>>>> 0.00    0.00   85.15
>>>> 18:00:17       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:17       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:17       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:17       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:17       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:17     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:18     all    0.12    0.00    0.12    0.00    0.95   11.51   
>>>> 0.00    0.00   87.31
>>>> 18:00:18       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:18       1    0.00    0.00    0.00    0.00    3.03   33.33   
>>>> 0.00    0.00   63.64
>>>> 18:00:18       2    0.00    0.00    0.00    0.00    5.00   64.00   
>>>> 0.00    0.00   31.00
>>>> 18:00:18       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:18       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:18       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:18       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:18       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:18     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:19     all    0.36    0.00    0.12    0.00    1.46   14.48   
>>>> 0.00    0.00   83.58
>>>> 18:00:19       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:19       1    0.00    0.00    0.00    0.00    4.00   46.00   
>>>> 0.00    0.00   50.00
>>>> 18:00:19       2    0.00    0.00    0.99    0.00    7.92   72.28   
>>>> 0.00    0.00   18.81
>>>> 18:00:19       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:19       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:19       5    2.86    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00   97.14
>>>> 18:00:19       6    0.95    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00   99.05
>>>> 18:00:19       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:19     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:20     all    0.00    0.00    0.00    0.00    1.58   14.86   
>>>> 0.00    0.00   83.56
>>>> 18:00:20       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:20       1    0.00    0.00    0.00    0.00    3.96   49.50   
>>>> 0.00    0.00   46.53
>>>> 18:00:20       2    0.00    0.00    0.00    0.00    9.00   73.00   
>>>> 0.00    0.00   18.00
>>>> 18:00:20       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:20       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:20       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:20       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:20       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:20     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:21     all    0.00    0.00    0.00    0.00    1.71   14.86   
>>>> 0.00    0.00   83.43
>>>> 18:00:21       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:21       1    0.00    0.00    0.00    0.00    5.05   51.52   
>>>> 0.00    0.00   43.43
>>>> 18:00:21       2    0.00    0.00    0.00    0.00   10.00   70.00   
>>>> 0.00    0.00   20.00
>>>> 18:00:21       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:21       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:21       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:21       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:21       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:21     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:22     all    0.00    0.00    0.24    0.00    1.70   13.97   
>>>> 0.00    0.00   84.08
>>>> 18:00:22       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:22       1    0.00    0.00    0.00    0.00    4.95   42.57   
>>>> 0.00    0.00   52.48
>>>> 18:00:22       2    0.00    0.00    1.01    0.00    8.08   72.73   
>>>> 0.00    0.00   18.18
>>>> 18:00:22       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:22       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:22       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:22       6    0.00    0.00    0.95    0.00    0.00    0.00   
>>>> 0.00    0.00   99.05
>>>> 18:00:22       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:22     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:23     all    0.12    0.00    0.00    0.00    1.58   14.13   
>>>> 0.00    0.00   84.17
>>>> 18:00:23       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:23       1    0.00    0.00    0.00    0.00    5.00   45.00   
>>>> 0.00    0.00   50.00
>>>> 18:00:23       2    0.99    0.00    0.00    0.00    8.91   70.30   
>>>> 0.00    0.00   19.80
>>>> 18:00:23       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:23       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:23       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:23       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:23       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:23     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:24     all    0.00    0.00    0.00    0.00    1.34   14.13   
>>>> 0.00    0.00   84.53
>>>> 18:00:24       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:24       1    0.00    0.00    0.00    0.00    4.04   42.42   
>>>> 0.00    0.00   53.54
>>>> 18:00:24       2    0.00    0.00    0.00    0.00    6.93   73.27   
>>>> 0.00    0.00   19.80
>>>> 18:00:24       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:24       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:24       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:24       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:24       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:24     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:25     all    0.12    0.00    0.12    0.00    1.45   14.67   
>>>> 0.00    0.00   83.64
>>>> 18:00:25       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:25       1    0.00    0.00    0.99    0.00    3.96   45.54   
>>>> 0.00    0.00   49.50
>>>> 18:00:25       2    0.00    0.00    0.00    0.00    8.00   75.00   
>>>> 0.00    0.00   17.00
>>>> 18:00:25       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:25       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:25       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:25       6    0.95    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00   99.05
>>>> 18:00:25       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:25     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:26     all    0.00    0.00    0.00    0.00    1.22   14.60   
>>>> 0.00    0.00   84.18
>>>> 18:00:26       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:26       1    0.99    0.00    0.00    0.00    2.97   50.50   
>>>> 0.00    0.00   45.54
>>>> 18:00:26       2    0.00    0.00    0.00    0.00    7.07   68.69   
>>>> 0.00    0.00   24.24
>>>> 18:00:26       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:26       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:26       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:26       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:26       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:26     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:27     all    0.00    0.00    0.00    0.00    1.59   15.02   
>>>> 0.00    0.00   83.39
>>>> 18:00:27       0    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:27       1    0.00    0.00    0.00    0.00    6.00   52.00   
>>>> 0.00    0.00   42.00
>>>> 18:00:27       2    0.00    0.00    0.00    0.00    6.93   71.29   
>>>> 0.00    0.00   21.78
>>>> 18:00:27       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:27       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:27       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:27       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:27       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> 18:00:27     CPU    %usr   %nice    %sys %iowait    %irq   %soft 
>>>> %steal  %guest   %idle
>>>> 18:00:28     all    0.00    0.00    0.00    0.00    1.33   14.56   
>>>> 0.00    0.00   84.10
>>>> 18:00:28       0    0.00    0.00    0.00    0.00    0.00    0.99   
>>>> 0.00    0.00   99.01
>>>> 18:00:28       1    0.00    0.00    0.00    0.00    4.00   50.00   
>>>> 0.00    0.00   46.00
>>>> 18:00:28       2    0.00    0.00    0.00    0.00    6.06   69.70   
>>>> 0.00    0.00   24.24
>>>> 18:00:28       3    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:28       4    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:28       5    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:28       6    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>> 18:00:28       7    0.00    0.00    0.00    0.00    0.00    0.00   
>>>> 0.00    0.00  100.00
>>>>
>>>> at the same time top shows:
>>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>   28 root      15  -5     0    0    0 S    2  0.0   0:16.60 events/1
>>>>   29 root      15  -5     0    0    0 S    2  0.0   0:15.62 events/2
>>>>   10 root      15  -5     0    0    0 S    2  0.0   0:41.01 ksoftirqd/2
>>>>    7 root      15  -5     0    0    0 S    1  0.0   0:31.17 ksoftirqd/1
>>>>   30 root      15  -5     0    0    0 S    1  0.0   0:03.00 events/3
>>>> 6545 root      20   0 11112 3244 1780 S    0  0.0   0:08.69 oprofiled
>>>> 6845 root      20   0 19084 1384 1008 R    0  0.0   0:00.07 top
>>>>    1 root      20   0  3812  612  520 S    0  0.0   0:01.00 init
>>>>    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd
>>>>    3 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/0
>>>>    4 root      15  -5     0    0    0 S    0  0.0   0:01.04 ksoftirqd/0
>>>>    5 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/0
>>>>    6 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/1
>>>>    8 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/1
>>>>    9 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/2
>>>>   11 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/2
>>>>   12 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/3
>>>>   13 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3
>>>>   14 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/3
>>>>   15 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/4
>>>>   16 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/4
>>>>   17 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/4
>>>>   18 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/5
>>>>   19 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/5
>>>>   20 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/5
>>>>   21 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/6
>>>>   22 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/6
>>>>   23 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/6
>>>>   24 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/7
>>>>   25 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/7
>>>>   26 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/7
>>>>   27 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/0
>>>>   31 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/4
>>>>   32 root      15  -5     0    0    0 S    0  0.0   0:00.01 events/5
>>>>   33 root      15  -5     0    0    0 S    0  0.0   0:00.02 events/6
>>>>   34 root      15  -5     0    0    0 S    0  0.0   0:00.01 events/7
>>>>   35 root      15  -5     0    0    0 S    0  0.0   0:00.01 khelper
>>>>   41 root      15  -5     0    0    0 S    0  0.0   0:00.00 async/mgr
>>>>  159 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/0
>>>>  160 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/1
>>>>  161 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/2
>>>>  162 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/3
>>>>  163 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/4
>>>>  164 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/5
>>>>  165 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/6
>>>>  166 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/7
>>>>  167 root      15  -5     0    0    0 S    0  0.0   0:00.00 kacpid
>>>>  168 root      15  -5     0    0    0 S    0  0.0   0:00.00
>>>> kacpi_notify
>>>>  246 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/0
>>>>  247 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/1
>>>>
>>>> and oprofile output:
>>>> CPU: Core 2, speed 3000.08 MHz (estimated)
>>>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with
>>>> a unit mask of 0x00 (Unhalted core cycles) count 100000
>>>> samples  %        image name               app name                
>>>> symbol name
>>>> 11138304 12.3598  vmlinux                  vmlinux                 
>>>> rb_next
>>>> 7714260   8.5602  vmlinux                  vmlinux                 
>>>> e1000_clean
>>>> 5380466   5.9705  vmlinux                  vmlinux                 
>>>> e1000_intr_msi
>>>> 4881445   5.4168  vmlinux                  vmlinux                 
>>>> u32_classify
>>>> 3977072   4.4132  vmlinux                  vmlinux                 
>>>> _raw_spin_lock
>>>> 3471611   3.8523  vmlinux                  vmlinux                 
>>>> hfsc_dequeue
>>>> 3103493   3.4438  vmlinux                  vmlinux                 
>>>> hfsc_enqueue
>>>> 2636824   2.9260  vmlinux                  vmlinux                 
>>>> delay_tsc
>>>> 2464293   2.7345  vmlinux                  vmlinux                 
>>>> e1000_xmit_frame
>>>> 2035779   2.2590  vmlinux                  vmlinux                 
>>>> e1000_clean_tx_irq
>>>> 1947231   2.1608  vmlinux                  vmlinux                 
>>>> mwait_idle
>>>> 1919540   2.1300  vmlinux                  vmlinux                 
>>>> ipt_do_table
>>>> 1908544   2.1178  vmlinux                  vmlinux                 
>>>> e1000_clean_rx_irq
>>>> 1865996   2.0706  vmlinux                  vmlinux                 
>>>> update_vf
>>>> 1829293   2.0299  vmlinux                  vmlinux                 
>>>> rb_erase
>>>> 1777847   1.9728  vmlinux                  vmlinux                 
>>>> ip_route_input
>>>> 1430773   1.5877  vmlinux                  vmlinux                 
>>>> rb_insert_color
>>>> 1329674   1.4755  vmlinux                  vmlinux                 
>>>> __alloc_skb
>>>> 1091678   1.2114  vmlinux                  vmlinux                 
>>>> sfq_enqueue
>>>> 985105    1.0931  vmlinux                  vmlinux                 
>>>> rb_first
>>>> 944928    1.0486  vmlinux                  vmlinux                 
>>>> sfq_dequeue
>>>> 756452    0.8394  vmlinux                  vmlinux                 
>>>> read_tsc
>>>> 728703    0.8086  vmlinux                  vmlinux                 
>>>> nf_iterate
>>>> 665551    0.7385  vmlinux                  vmlinux                 
>>>> ip_forward
>>>> 662616    0.7353  vmlinux                  vmlinux                 
>>>> dev_queue_xmit
>>>> 648919    0.7201  vmlinux                  vmlinux                 
>>>> __qdisc_run
>>>> 615592    0.6831  vmlinux                  vmlinux                 
>>>> ip_rcv
>>>> 589743    0.6544  vmlinux                  vmlinux                 
>>>> apic_timer_interrupt
>>>> 550581    0.6110  vmlinux                  vmlinux                 
>>>> irq_entries_start
>>>> 536172    0.5950  vmlinux                  vmlinux                 
>>>> getnstimeofday
>>>> 527467    0.5853  vmlinux                  vmlinux                 
>>>> rb_last
>>>> 508305    0.5640  vmlinux                  vmlinux                 
>>>> skb_release_data
>>>> 506763    0.5623  vmlinux                  vmlinux                 
>>>> kfree
>>>> 506269    0.5618  vmlinux                  vmlinux                 
>>>> dst_release
>>>> 464412    0.5153  vmlinux                  vmlinux                 
>>>> __hrtimer_start_range_ns
>>>> 456354    0.5064  vmlinux                  vmlinux                 
>>>> e1000_alloc_rx_buffers
>>>> 435339    0.4831  vmlinux                  vmlinux                 
>>>> dev_hard_start_xmit
>>>> 371439    0.4122  vmlinux                  vmlinux                 
>>>> local_bh_enable
>>>> 369266    0.4098  vmlinux                  vmlinux                 
>>>> __rb_rotate_left
>>>> 361627    0.4013  vmlinux                  vmlinux                 
>>>> net_tx_action
>>>> 357797    0.3970  vmlinux                  vmlinux                 
>>>> rtsc_y2x
>>>> 350595    0.3890  vmlinux                  vmlinux                 
>>>> _raw_spin_unlock
>>>> 349359    0.3877  vmlinux                  vmlinux                 
>>>> kmem_cache_alloc
>>>> 336255    0.3731  vmlinux                  vmlinux                 
>>>> skb_dma_map
>>>> 328809    0.3649  vmlinux                  vmlinux                 
>>>> netif_receive_skb
>>>> 315826    0.3505  vmlinux                  vmlinux                 
>>>> fn_trie_lookup
>>>> 313065    0.3474  vmlinux                  vmlinux                 
>>>> __kmalloc
>>>> 307684    0.3414  vmlinux                  vmlinux                 
>>>> __do_softirq
>>>> 298676    0.3314  vmlinux                  vmlinux                 
>>>> skb_pull
>>>> 292352    0.3244  vmlinux                  vmlinux                 
>>>> __napi_complete
>>>> 288878    0.3206  vmlinux                  vmlinux                 
>>>> get_next_timer_interrupt
>>>> 280449    0.3112  vmlinux                  vmlinux                 
>>>> ip_finish_output
>>>> 264482    0.2935  vmlinux                  vmlinux                 
>>>> consume_skb
>>>> 251622    0.2792  vmlinux                  vmlinux                 
>>>> __remove_hrtimer
>>>> 227178    0.2521  vmlinux                  vmlinux                 
>>>> tc_classify_compat
>>>> 226224    0.2510  vmlinux                  vmlinux                 
>>>> handle_edge_irq
>>>>
>>>>
>>>>
>>>> proces that this machine is running is only quagga (zebra + bgpd +
>>>> oprofiled)
>>>>
>>>>
>>>> Regards
>>>> Paweł Staszewski
>>>>
>>>>
>>>> -- 
>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>> the body of a message to [email-masked]
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>> the body of a message to [email-masked]
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to [email-masked]
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> 
> -- 
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to [email-masked]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>