Counting hardware events for 16 threads simultaniously

[rmcilroy]

Hi,

I'm trying to count some native events (L3_CACHE_MISSES and HYPERTRANSPORT_LINK0) for a program throughout its execution and having some issues.
I'm using Papi 3.7.0 on Linux with perfctr. The machine I'm experimenting on is a 4x4 AMD Opteron machine, and the process I want to measure has 16 threads, with one thread pinned to each core.

I just want the count events for the whole process, so at first I tried to set the granularity to process level (PAPI_GRN_PROC), but it seems that the Linux substrate only supports thread level granularity. So instead I tried starting a counter for each of the 16 threads in the program. However, after stating 4 counters, the rest of the thread's fail with PAPI_ESYS.

Any idea's why this might be happening and what I could do to get around it?

[Dan Terpstra]

One thing that you're bumping up against is a fundamental limit in the Opteron hardware. The events you're monitoring are measuring shared resources at the chip level, even though you are using a core level counter interface to do the measurement. For most events, each core has complete and exclusive control of its 4 counters. For the events monitoring shared resources (L3 cache, Hypertransport) the cores on a chip share a set of 4 chip level counters, even though the core level programming looks identical. Think of it like having 4 remote controls for the same TV set. The AMD BKDG document says that by convention each chip should only allow one core to program these events. Otherwise core 2 (for example) could overwrite the settings of core 1. You are at an advantage by pinning threads to cores since thread migration won't be an issue, but you still need to be aware of this contention issue. Further, PAPI counts events only when a thread is active. Thus, even with the above constraints met, the value in the counters will be a lower bound on the number of actual events.
I hope this sheds at least some light on what you're seeing.

[rmcilroy]

Hi Dan,

Thanks for the reply. This is pretty much what I expected the problem was.

I tried getting around it by only measuring on four threads (one on each socket), then at regular intervals switching measurement to another thread on that chip (so only one thread on each chip is being measured simultaneously, but all threads are measured over the course of the program). I did this using Papi_stop on the thread which was currently being measuring, and Papi_start on the one I'd like to measure next, using mutexes and condition variables to ensure the previous counting has stopped before I start it on the new thread. However, even with this approach, I still get the same error when I call Papi_start on the new thread.

Is there anything I can do to ensure that the counters are reset / released by the original thread so they can be used by the next thread?

[Dan Terpstra]

Is there anything I can do to ensure that the counters are reset / released by the original thread so they can be used by the next thread?

This sounds like a bug in either PAPI or perfctr. Those counters should be released given the protocol you describe, unless perfctr somehow requires a shutdown and re-open to reassign those counters. This might be a question worth asking Mikael Pettersson on the perfctr mailing list: perfctr-devel@lists.sourceforge.net