Subject: PROBLEM: Only 2 of 4 cores used on IBM Cell blades and no threads shown in spufs

From: Dennis Schridde <[email-masked]>
Date: Mon, 22 Apr 2013 18:44:13 +0200

Hello!

[1.] One line summary of the problem:
Only 2 of 4 cores used on IBM Cell blades and no threads shown in spufs

[2.] Full description of the problem/report:
On my IBM Cell blades, only 2 out of the 4 CPU cores are being used, even when 
several threads are running. Also /spu is always empty, despite the SPUs being 
utilised. These are probably two unrelated problems, but I just wrote one 
email so I do not have to duplicate all the information. Please tell me if I 
should split the email up into two threads.

For testing, I started an instance of cellminer with --ppe 2 --spe 16 (2 
threads on the PPEs, 16 on the SPEs) and htop reports two CPU cores being used 
at 100% and two others at 0%, while 4 threads are running (each at 50% CPU 
utilisation -> they share a core).

During that test, spu-tools report:
---
# spu-top
spu-top: Context View
Cpu(s) load avg:   4.0%,  4.2%,  3.7%
Spu(s) load avg:  16.0%, 16.0%, 14.4%
Cpu(s): 50.0%us,  0.0%sys,  0.0%wait,  0.0%nice, 50.0%idle
Spu(s):100.0%us,  0.0%sys,  0.0%wait,  0.0%idle

   PID   TID USERNAME   S F  %SPU SPE     TIME BINARY
---
# spu-ps
   PID   TID USERNAME   S F SPE     TIME BINARY            
---

And the spufs is empty:
---
# ls -la /spu
total 0
# mount | grep spu
none on /spu type spufs (rw,relatime)
---

[3.] Keywords (i.e., modules, networking, kernel):
No idea what you want to know here...

[4.] Kernel information

[4.1.] Kernel version (from /proc/version):
I am using the Linux 3.8.8 kernel (vanilla-sources-3.8.8 on Gentoo/Linux):
# cat /proc/version 
Linux version 3.8.8 (root@blade00) (gcc version 4.7.2 (Gentoo 4.7.2-r1 p1.5, 
pie-0.5.5) ) #2 SMP Mon Apr 22 18:21:20 CEST 2013

[4.2.] Kernel .config file:
Please find the kernel config attached.

[5.] Most recent kernel version which did not have the bug:
N/A

[6.] Output of Oops.. message (if applicable) with symbolic information
     resolved (see Documentation/oops-tracing.txt)
N/A, but please find the output of dmesg attached.

[7.] A small shell script or example program which triggers the
     problem (if possible)
N/A

[8.] Environment

[8.1.] Software (add the output of the ver_linux script here)
# ./source/scripts/ver_linux 
If some fields are empty or look unusual you may have an old version.
Compare to the current minimal requirements in Documentation/Changes.
 
Linux blade00 3.8.8 #2 SMP Mon Apr 22 18:21:20 CEST 2013 ppc64 Cell Broadband 
Engine, altivec supported CHRP IBM,0793-38G GNU/Linux
 
Gnu C                  4.7.2
Gnu make               3.82
binutils               2.22
util-linux             2.22.2
mount                  debug
module-init-tools      12
e2fsprogs              1.42
Linux C Library        2.17
Dynamic linker (ldd)   2.17
Procps                 3.3.4
Net-tools              1.60_p20120127084908
Kbd                    1.15.3wip
Sh-utils               8.20
Modules Loaded         rdma_ucm rdma_cm iw_cm ib_addr ib_cm ib_uverbs ib_umad 
mlx4_ib ib_sa ib_mad ib_core mlx4_core rtc_generic nfsv3 nfs_acl nfsv4 
auth_rpcgss nfs lockd fscache af_packet ehci_pci ohci_hcd ehci_hcd usbcore 
usb_common nls_base tg3 hwmon ptp pps_core libphy ipv6 unix sunrpc

[8.2.] Processor information (from /proc/cpuinfo):
# cat /proc/cpuinfo 
processor       : 0
cpu             : Cell Broadband Engine, altivec supported
clock           : 3200.000000MHz
revision        : 48.0 (pvr 0070 3000)

processor       : 1
cpu             : Cell Broadband Engine, altivec supported
clock           : 3200.000000MHz
revision        : 48.0 (pvr 0070 3000)

processor       : 2
cpu             : Cell Broadband Engine, altivec supported
clock           : 3200.000000MHz
revision        : 48.0 (pvr 0070 3000)

processor       : 3
cpu             : Cell Broadband Engine, altivec supported
clock           : 3200.000000MHz
revision        : 48.0 (pvr 0070 3000)

timebase        : 26664380
platform        : Cell
model           : IBM,0793-38G
machine         : CHRP IBM,0793-38G

[8.3.] Module information (from /proc/modules):
# cat /proc/modules 
rdma_ucm 18293 0 - Live 0xd000000001870000
rdma_cm 41579 1 rdma_ucm, Live 0xd000000001855000
iw_cm 10924 1 rdma_cm, Live 0xd00000000183b000
ib_addr 7425 1 rdma_cm, Live 0xd00000000182e000
ib_cm 40938 1 rdma_cm, Live 0xd000000001818000
ib_uverbs 45864 1 rdma_ucm, Live 0xd0000000017f4000
ib_umad 17663 0 - Live 0xd0000000017d6000
mlx4_ib 132595 0 - Live 0xd00000000172c000
ib_sa 29508 4 rdma_ucm,rdma_cm,ib_cm,mlx4_ib, Live 0xd0000000016ec000
ib_mad 47187 4 ib_cm,ib_umad,mlx4_ib,ib_sa, Live 0xd0000000016cb000
ib_core 72200 9 
rdma_ucm,rdma_cm,iw_cm,ib_cm,ib_uverbs,ib_umad,mlx4_ib,ib_sa,ib_mad, Live 
0xd000000001697000
mlx4_core 208672 1 mlx4_ib, Live 0xd00000000162d000
rtc_generic 2325 0 - Live 0xd0000000015d4000
nfsv3 36741 1 - Live 0xd0000000015c2000
nfs_acl 3380 1 nfsv3, Live 0xd0000000015ac000
nfsv4 163536 1 - Live 0xd000000001571000
auth_rpcgss 44818 1 nfsv4, Live 0xd00000000151c000
nfs 183674 5 nfsv3,nfsv4, Live 0xd0000000014c3000
lockd 84130 2 nfsv3,nfs, Live 0xd000000001451000
fscache 49687 2 nfsv4,nfs, Live 0xd000000001417000
af_packet 38205 0 - Live 0xd0000000013ee000
ehci_pci 5600 0 - Live 0xd0000000013b0000
ohci_hcd 54842 0 - Live 0xd000000001398000
ehci_hcd 66682 1 ehci_pci, Live 0xd00000000136d000
usbcore 205456 3 ehci_pci,ohci_hcd,ehci_hcd, Live 0xd0000000011ca000
usb_common 987 1 usbcore, Live 0xd00000000116d000
nls_base 8541 1 usbcore, Live 0xd000000001164000
tg3 190708 0 - Live 0xd0000000010ad000
hwmon 2174 1 tg3, Live 0xd000000001069000
ptp 12774 1 tg3, Live 0xd00000000105e000
pps_core 11405 1 ptp, Live 0xd00000000104f000
libphy 30711 1 tg3, Live 0xd000000001038000
ipv6 348760 48 ib_addr, Live 0xd000000000fb8000
unix 41060 44 - Live 0xd000000000f20000
sunrpc 238337 23 nfsv3,nfs_acl,nfsv4,auth_rpcgss,nfs,lockd, Live 
0xd000000000eb6000

[8.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
# cat /proc/ioports 
# cat /proc/iomem 
00000000-ffffffff : System RAM
100000000-1ffffffff : System RAM
14540000200-14540000207 : serial
14540000300-14540000307 : serial
14780000000-1478000ffff : tg3
14780010000-1478001ffff : tg3
34540000200-34540000207 : serial
34540000300-34540000307 : serial
34780000000-34780000fff : ohci_hcd
34780001000-34780001fff : ohci_hcd
34780002000-347800020ff : ehci_hcd
3d080000000-3d0800fffff : mlx4_core
3d0c0000000-3d0c1ffffff : mlx4_core

[8.5.] PCI information ('lspci -vvv' as root)
Please find this attached.

[8.6.] SCSI information (from /proc/scsi/scsi)
The system is diskless and the file you request is not available. I also 
tested on my desktop machine - seems to be gone in newer kernel versions?

[8.7.] Other information that might be relevant to the problem
       (please look in /proc and include all information that you
       think to be relevant):
Please request anything that might be interesting.

[X.] Other notes, patches, fixes, workarounds:
I would like to know whether this a misconfiguration of my kernel, a hardware 
problem, or a bug in Linux.
You might find additional information (especially on the program being used 
for testing) in my support request for cellminer: 
https://github.com/verement/cellminer/issues/10

Best regards,
Dennis