L3 counters on AMD Opteron (Barcelona)

[cincaipatron]

I'm using papi-3.6.0 and quad-core Opteron 2.2GHz (8354 / Barcelona).

I keep getting PAPI_ESYS and errno = "Operation not permitted" when attempting to get L3-related events (either preset or native). However, papi_avail & papi_native_avail indicate the availability of such events on this platform.

Preset (only relevant portions are showed):
PAPI_L3_TCM 0x80000008 Yes No Level 3 cache misses
PAPI_L3_TCR 0x8000005d Yes No Level 3 total cache reads

Native (only relevant portions are showed):
x4000006f L3_CACHE_MISSES | L3 Cache Misses |
0x40000070 L3_FILLS_CAUSED_BY_L2_EVICTIONS | L3 Fills caused by L2 Eviction |
0x40000071 L3_EVICTIONS | L3 Evictions

Any clue what's going wrong?

[Dan Terpstra]

Have you tried the latest (3.7.0) release of PAPI? Which kernel patch are you using? Is it current? This should work.

[jdmccalpin]

The default configuration of the perfctr patch (at least through revision 2.6.31) disables access to any of the "shared" counters on Opteron systems (including Barcelona) when using the virtual interface that PAPI uses. This is not something under PAPI's control -- PAPI thinks that the L3 events are "available" because it knows how to create a native event code for them. It is only at run-time that the perfctr driver checks the hardware EventSelect value and aborts if the access is to a counter in the "shared" hardware (i.e., L3, crossbar, HyperTransport links, memory controllers, etc.).

On TACC's Ranger system, I modified the perfctr 2.6.31 patch to remove this test, so all Opteron preset and native events are available via PAPI. This makes it possible to get wrong answers (if you try to program the shared counters from more than one core), but it also makes it possible to get the right answers, so it seems like a reasonable tradeoff. The perfctr 2.6.39 and 2.6.40 versions have a different approach to this issue -- they bind any thread requesting a shared counter to core 0 on the chip that the thread is currently running on. I found that this did not play nicely with my own thread binding code (it hung the system), so I opted for a less restrictive approach.

[jakemp]

Howdy,

I am running Barcelona based Opterons, and am Trying to collect L3 information, but it locks up the machine on MKL work. I think the modifications you made might be what is needed. Is there a patch for the modifications available anywhere?