Please see INSTALL.txt in the papi root directory.

SOme versions of GCC have a bug that is triggered by a statement in PAPI 3.0. This (one character) is fixed in the current tar ball, but may not be in the one you downloaded.

If you see an INTERNAL ERROR from GCC when compiling multiplex.c, do 2 things.

1) edit multiplex.c, line 1021 to have 2 equal signs instead of 1.

2 optional) send a message to your local gcc maintainer and complain.

The actual culprit is:

assert(retval = PAPI_OK) and it should be assert(retval == PAPI_OK)

Of course, both are legal C and nothing should trigger an internal compiler error, but hey...

P.S. If your current release compiled with GCC, you're still ok. As the statement above NEVER gets triggered. It is there as an artifact from the original multiplex.c implementation. So you don't need to change or upgrade your PAPI or gcc.

The warning message here is benign, but since it occurs on the last file to be compiled, it often looks like the build has been aborted. The reason the message occurs is that the compiler thinks it is trying to stuff too many bits into an integer value. You can fix it by rearranging the code a little bit. Or just download the latest copy of fmultiplex2.F from the cvs tree.

To compile with debugging, define CFLAGS to include -DDEBUG in the corresponding Makefile or Rules. file.

On most systems, overflow must be emulated in software by PAPI. Only on the UltraSparc III, Itanium and IRIX does the operating system support true interrupt on overflow. Therefore the user is advised on most platforms to make sure the overflow value is no more than 1/1000th the clock rate. The emulation handler in PAPI runs every millisecond, therefore the goal of the tool designer should be to pick an value that will overflow frequently but not too frequently. Not following these guidelines could result in either the overflows never occurring or overflows occurring on every interrupt and thus resulting in a flat profile.

Call PAPI_stop, and then call PAPI_overflow, PAPI_profile or PAPI_sprofil with a threshold value of 0. Since PAPI 3 can overflow and profile on multiple events, you must call the above routines for EACH event that had been previously enabled for overflow or profile.

PAPI only tracks 'hardware events', the occurrence of signals onboard the microprocessor. It does not count system calls, software interrupts or other software events. The user should remember that by default, PAPI only measures events that occur in User Space.

Currently, PAPI only supports thread level measurements with kernel or bound threads. Each thread must create, manipulate and read its own counters. When a thread is created, it inherits no PAPI events or information from the calling thread.

When a process is created, it inherits no PAPI information from the calling thread.

Yes, but the counts will reflect the total events for the process. Measurements done in other threads will all get the same values, namely those counts for the total process. For non-bound threads, it is not necessary to call PAPI_thread_init. But in most scenarios like with SMP or OpenMP compiler directives, bound threads will be the default. For those using Pthreads, the user should take care to set the scope of each thread to the PTHREAD_SCOPE_SYSTEM attribute, unless the system is known to have a non hybrid thread library implementation, like Linux.

In PAPI2.0: Unless otherwise stated in the FAQ section for your platform, the encoding is as follows:

event = ((reg_code & 0xffffff) << 8 | (reg_num & 0xff))

In PAPI3.0: Just find the native event name and then call PAPI_event_name_to_code. The code returned can be added directly to an event set. The native events can be listed with the test case 'native_avail' in the ctests directory.

This is not a question, but I'll help you. We the PAPI developers cannot be experts on the 1000's of events found across all supported platforms. However, if you are using a PAPI preset, the first thing to do is to look up the corresponding native event code using the test case 'avail'. Then the best bet is to always go to the vendor's technical documentation site and check the processor reference manual. If you're convinced everything is kosher, then please feel free to send a message to the mailing list and one of the members may be able to help you.

My program runs fine with 1 or 2 counters, but when I add more I get a -8, PAPI_ECNFLCT error code. The error text says, "Event exists. but cannot be counted due to hardware resource limitations". What does this mean?

Many systems have only a few hardware performance counter registers thus you can only measure a few metrics at once. Some platforms may support counter multiplexing, which gives the user the illusion of a larger number of registers by time sharing the performance registers. On the R10K series, the IRIX kernel supports multiplexing, allowing up to 32 events to be counted at once. Don't take fine grained measurements when multiplexing, unless you know what you're doing.

You have either exceeded the number of available hardware counters or two or more of the events you want to count need the same resources. This can be particularly annoying on machines like the Pentium 4. Although the P4 has 18 nominal counter registers, many events require resources that are restricted to 2 or 3 of these counters. In practice it is often difficult to count more than 4 or 5 simultaneous events on this platform. One way around limited counter resources is to use multiplexing.

Many systems have only a few hardware performance counter registers; thus you can only measure a few metrics at once. Some platforms may support counter multiplexing, which gives the user the illusion of a larger number of registers by time sharing the performance registers. On the MIPS R10K series, the IRIX kernel supports multiplexing, allowing up to 32 events to be counted at once. On other platforms PAPI does the multiplexing itself, swapping events in and out of the counters based on a timer interrupt. Don't take fine grained measurements when multiplexing, unless you know what you're doing.

PAPI multiplexing currently always uses one hardware counter for Total Cycles. If you are trying to multiplex a derived event on hardware with only two physical counters then you will get a PAPI_ECNFLCT error. This happens on the Intel Pentium IIIs for example.

Also, enabling multiplexing is a two-step process. You must call PAPI_multiplex_init() to initialize multiplexing system-wide. You must also call PAPI_set_multiplex() for *each* event set that you want to count in multiplexed mode. If you try to add too many events to an event set where multiplexing has not been set, a PAPI_ECNFLCT error will result.

Hardware counters count low level events that can be directly measured in the hardware. Often these low level events must be combined to form meaningful PAPI preset events. This linear combination of low level events is called a derived PAPI event. Derived events are usually formed by adding or subtracting 2 'native' events, but occasionally derived events can contain 4 or more terms.

Hardware counters are a limited resource. Some PAPI preset events are derived, and require the use of more than one hardware counter. For example, Solaris has 2 counters, both of which are needed to count Floating point instructions. Flops also uses total cycles to measure time. On Solaris this would mean using 3 counters, and those resources aren't available.
If you get this error on any platform, run the avail program in the ctests directory and see how many native events have to be monitored. PAPI_num_counters() can be used to determine how many counters exist on your platform. If there are more native events than counters, then this is the reason you are getting the error.

The Fortran header file you include has to be preprocessed before the Fortran file can use it. To have the cpp process the file before sending the file to the compiler, add the -F flag. For example:

f90 -F test.F -o test

According to the Intel documentation, the counts from this event are not intuitive relating to it's description. Older releases of PAPI had this preset available in the Intel ports, but no longer. It does appear to work on the AMD Athlon.

The EventSet MUST be set to PAPI_NULL before it is passed into PAPI_create_eventset.

TLS is thread local storage, a high performance mechanism in later GCC's/GLIBC/pthread to do constant time access to thread local storage. PAPI uses this if available.

However, many systems (especially IA64 running Debian or SuSE) provide very poor/buggy/non-existent support for this. If you're getting an error during compile (or seg faults on every program during the run), then please rebuild using ./configure.

Other systems don't bother to ship a gcc with this turned on, so you'll get the above error.

./configure has a test to make sure that the thread support is working on your platform.

If you find a case where configure did not detect a broken __thread implementation, please report it to us.

This question is meant for the CVS development team.

0) You must be in the group papi.
1) Login to nala.cs.utk.edu.
2) Make a temporary directory called foo.
3) Do the following:
 
> cd foo;
> cvs -d /cvs/homes/papi co CVSROOT
.
.
.
> cd CVSROOT
>
 
4) Add 'username' to the 'writers' file.
5) Add a password to the 'passwd' file with htpasswd program.
 
> /usr/local/apache/bin/htpasswd ~/tmp/CVSROOT/passwd 'username'
New password: <enter password>
Re-type new password: <again>
Added password for user 'username'
>
 
6) Commit the changes back to CVS.
                                                                               
> cvs commit -m "Added user 'username' to the cvs list"
.
.
.
cvs commit: Rebuilding administrative file database                                                

1) Check out the main trunk of the PAPI project:
        > cvs checkout -P papi
2) Update the main trunk by joining with the branch:
        > cvs update -j papi-2-3-3
3) Tag the branch for future updates:
        > cvs tag papi-2-3-3m1
4) Resolve any outstanding conflicts
                                                                               
5) Commit the modified files back to the main trunk:
        > cvs commit ...
...later...
6) To merge from the same branch again, repeat the steps above,
        but update from the tag created in step 3 rather than
        the branch point. This guarantees that only the changes
        made since the last merge are merged.

First you remove all the files in the directory (and all subdirectories).
This can be tedious, so do it like this. This assumes GNU grep.
                                                                               
> cd <dir_to_be_removed>
> find . -type f | grep --invert-match CVS | xargs > /tmp/removeme
                                                                               
Now you have a list of all files in all subdirectories to be removed,
not including anything under CVS. Now edit the file and put 'rm' at the
start of the file.
                                                                               
> vi /tmp/removeme
 
So the start of the file looks like:
"rm ./file1 ./dir2/file2 ...."
 
Now execute the remove.
 
> sh /tmp/removeme
 
Good. Now tell CVS to remove the files. We have to edit the /tmp/removeme
again to replace 'rm' with 'cvs delete'.
 
> vi /tmp/removeme
 
So the start of the file looks like:
"cvs delete ./file1 ./dir2/file2 ...."
 
> sh /tmp/removeme
 
[ lots of CVS log messages ]
 
Now commit the changes with an informative log message.
 
> cvs commit.

This is not something that needs to happen very often, and because of that, it's easy to forget how. We've done it often enough to want to record what we've learned. The cedarqvist manual describes the procedure, but it isn't completely clear. Here's what you do:

> cvs rtag -d <module>

- or -

> cvs rtag -d -B <branch> <module>

where <tag> or <branch> is the name of the label you want to remove, and <module> is usually 'papi'.

CAUTION: Removing tags is risky, but removing branches is downright dangerous. Do it only if you're really sure you need to, and as close to the creation date as possible!

Usually, branches or tags are added to the main trunk of a cvs repository. Occasionally it is desirable to add a branch to an existing branch. Here's how to do it:

> cvs rtag -b -r <oldtag> -- <newtag> <module>

Where <oldtag> is the name of the existing branch, <newtag> is the name of the new branch, and <module> is typically 'papi'.

Here's an example:

> cvs rtag -b -r papi-3-1-0 -- papi-3-2-0 papi

If you are running papi-3.0 on aix5.2 & power4 combo, and seeing failure. It is
most likely caused by the BUG in the KERNEL. You need look for efix for APAR IY57280, or
contact papi team at papi@cs.utk.edu for the fix. Here is the more precise info
from IBM:
the problam was introduced in 5.2 ML3, and fixed in 5.2 ML4 and 5.3.
 
To use PAPI in 64-bit mode on power4:
    make -f Makefile.aix-power4-64bit
        link your program with libpapi64.a or libpapi64.so
         
See: /usr/lpp/pmtoolkit/lib/<arch>.evs for POWER3;
     /usr/pmapi/lib/POWER4.evs and /POWER4.gps for POWER4
 
For threaded programs, you had better:
 
setenv AIXTHREAD_SCOPE S

AIX 4.3.x:
The current source and Makefile is for pmtoolkit 1.3.
If you have pmtoolkit 1.2 the test cases will fail. For example:
 
      ./tests/avail
      IOT trap
 
This can be remdied by recompiling the PAPI library with the option
-DPMTOOLKIT_1_2 set.
 
AIX 5.x:
The current source is for pmapi 1.4
 
The aix-power substrate is contained in a single source file, but targets
three different configurations.
Conditional compilation directed by three different make files determines
which configuration is targetted. Make sure you select the Makefile that
matches your configuration:
- Makefile.aix-power    for AIX 4.3.x on POWER3
- Makefile.aix5-power3  for AIX 5.x   on POWER3
- Makefile.aix-power4   for AIX 5.x   on POWER4

The POWER3 and POWER4 have a FMADD instruction. Although this instruction
performs two Floating Point operations, it is counted as one Floating Point
instruction. Because of this, there are situations where PAPI_FP_INS may
produce fewer Floating Point counts than expected.
Further, the Floating Point Instruction event on POWER3 and POWER4 also
counts Floating Point Stores, leading to higher Floating Point counts than
expected. There are occasions where these two effects can cancel each other
out, to produce the right result for the wrong reason!
Note that POWER3 and POWER4 also support an FMA counter (PAPI_FMA_INS).
Thus, a more accurate count of Floating Point Operations can be obtained
by PAPI_FP_INS + PAPI_FMA_INS.
Correcting for the overcount by Floating Point Stores is more difficult,
requiring the use of the native events: PM_FPU_LD_ST_ISSUES and PM_FPU_LD.
The complete expression for Floating Point Operations then becomes:
PAPI_FP_INS + PAPI_FMA_INS - (PM_FPU_LD_ST_ISSUES - PM_FPU_LD)

The POWER architecture supports up to 8 counters. However, in many cases
events are mutually exclusive and can't be counted simultaneously.
 
On POWER4, events are available only as members of predefined groups.
For more on these groups, see /usr/pmapi/lib/POWER4.gps.
 
The following table, submitted by Joel Malard, indicates
events that cannot be counted simultaneously on POWER3:

It is very important that you set the environment variable AIXTHREAD_SCOPE to "S", which disables user level threads.

This substrate works on all platforms. It is for testing purposes only. This substrate emulates hardware that returns a 2 counter history in hardware registers 0 and 1, 1 being the most recent. The values returned represent the number of register reads performed by the substrate on the counter hardware.

One of the PAPI substrates for HP Alpha Tru64 UNIX uses the Dynamic Access to DCPI Data (DADD) API from Hewlett-Packard.
DADD is assumed to be installed in
  /usr/lib/dcpi  (dadd.a)
  /usr/include/dcpi (dadd.h, virtual_counters.h)
  /usr/local/dadd  (dcpid)

DCPI uses instruction sampling to collect performance data, and the DADD counts for PAPI events are estimated from these samples.
For this reason, very short running programs instrumented with PAPI may show undercounts as low as zero. Best results are obtained on programs with long execution times.

Both n32-bit and 64-bit libraries are built for IRIX. To use the shared libraries, you will need to set LD_LIBRARYN32_PATH or LD_LIBRARY64_PATH to the location of the n32-bit libpapi.so or 64-bit libpapi64.so.

For *all* the native event names, run native_avail in the ctests subdirectory. For an example of how to use the native event names, see native.c

Hardware multiplexing is implemented and in use for IRIX-MIPS. You can count all 32 events simultaneously, but this must be enabled explicitly as with the other substrates.

There may be two bugs in IRIX64 about it's multiplexing:

First, It seems when multiplexing is in use, the counter result you get by calling ioctl is not multiplied by the number of events counted by the hardware counter.

Second, since there are only two hardware counters available in R10K/R12k, if you use multiple overflow and these overflow events are happen to be counted by the same hardware counter, the result is reasonable. However, if the overflow events are counted by different hardware counters, then one of the result from hardware counter 1 will be abnormal higher than the reasonable result. See: man r10k_counters

You have an older run-time system than what PAPI was developed on. The fix is to change the definition to 'struct Dl_info'.

IRIX is brain-dead, and subprocesses automatically inherit the counter state of the parent process. Solution: call PAPI_stop() just before you fork and PAPI_start() after you fork.

Make sure your /usr/src/linux points to the source tree.

The PAPI linux-alpha substrate uses the linux iprobe_4.3 driver and library from Compaq (now HP). This needs to be patched, because not all of the EV6 functions are in the library. Unpack the iprobe code in the usual way, and copy the patch in papi/src/alpha-linux/iprobe.patch to the top of the iprobe directory. Then
 
patch -Np1 < iprobe.patch
 
After that, build and install iprobe according to directions, and set IPROBE_HOME in papi/src/Makefile.linux-alpha.

This version of the substrate always scales PME_FP_OPS_RETIRED_HI, hex code 0xa, even if you are using it as a NATIVE event. Previous versions of PAPI did not scale this event and could produce erroneously low counts for PAPI_FP_OPS or PAPI_FP_INS.

PAPI_CA_SNP
PAPI_CA_INV
 Only counts snoops and invalidations from the local processor.
PAPI_TLB_TL
 Counts "real" TLB misses, i.e. misses that cause a VHPT walk or a TLB
 miss trap to the OS. Misses in the L1 TLBs are not counted.
PAPI_FP_STAL
 Counts stalls due to register dependencies and load latencies.
 If the FP pipeline can stall for some other reason (I don't know)
 then those stall cycles won't be counted.

Redhat broke the perfmon kernel interface in their kernels and thus only enabled it for root. In some kernels, its disabled entirely. You can test this by running your papi as root, if it then works, guess what, you have a broken kernel.

The fix is supposed to be in the latest update to RHEL3 and RHEL4. The best thing to do would be to download a kernel.org kernel, rebuild and go.

You are probably on an Altix or a system with a Redhat kernel. The solution for the later is replace the kernel you have with a patched kernel.org kernel, discussed in this section.

Please send us the kernel version if this happens to you. You'll notice it by running the profile_pthreads test case.

If you're an Altix user, then it's best to complain to SGI. But please let us know also.

The problem is not in PAPI, but in libpfm 3.x. When this library is built using icc, the file pfmlib_gen_ia64.c generates a series of errors. One workaround for this may be to make the libpfm library separately using gcc and then build PAPI with icc. Or just use gcc.

cd perfctr
more INSTALL
If you're getting compilation errors regarding not being able to find include files, then you're probably running a broken redhat installation.

Edit the path to your kernel include files at the top of either Makefile.linux-perfctr

If you have a properly functioning Perfctr patch from a previous release of PAPI, you will obviously not want to repatch your kernel. PAPI is compatible with PerfCtr 2.4.x and Perfctr 2.6.x.

The x86 Makefiles:
Makefile.linux-perfctr-p3
Makefile.linux-perfctr-p4
Makefile.linux-athlon
Makefile.linux-opteron

To recompile PAPI *not* using the included PerfCtr distribution, you simply pass the PERFCTR argument to the appropriate Makefile.

make -f Makefile.linux-perfctr-p3
PERFCTR=/usr/src/perfctr-2.4.x

To use Perfctr 2.6.x, simply type:
make -f Makefile.linux-perfctr-p3

To use the older version:
make -f Makefile.linux-perfctr-p3 VERSION=2.4.x

Easy huh?

See the INSTALL file in papi/src/perfctr-2.6.x. The instructions are very, very simple. Do not use perfctr-2.4.x unless you have to. There is no link of perfctr version to linux kernel version!

You are probably running udev, which is not smart enough to know the permissions of dynamically created devices. To fix this, find your udev/devices directory, often /lib/udev/devices or /etc/udev/devices and perform the following actions.

mknod perfctr c 10 182
chmod 644 perfctr

On Ubuntu 6.06 (and probably other debian distros), add a line to /etc/udev/rules.d/40-permissions.rules like this:

KERNEL=="perfctr", MODE="0666"

On SuSE, you may need to add something like the following to /etc/udev/rules.d/50-udev-default.rules: (SuSE does not have the 40-permissions.rules file in it.]

# cpu devices
KERNEL=="cpu[0-9]*", NAME="cpu/%n/cpuid"
KERNEL=="msr[0-9]*", NAME="cpu/%n/msr"
KERNEL=="microcode", NAME="cpu/microcode", MODE="0600"
KERNEL=="perfctr", NAME="perfctr", MODE="0644"

These lines tell udev to always create the device file with the appropriate permissions. Use 'perfex -i' from the perfctr distribution to test this fix.

YOU MUST COMPILE YOUR KERNEL WITH APIC SUPPORT IF YOU WANT INTERRUPT SUPPORT!
With Perfctr 2.3.3 or later it is possible to make the performance counters generate an interrupt when the counter reaches a certain count. This requires support in the Linux kernel, Perfctr, PAPI and the CPU to work properly.
The necessary kernel support is available if your kernel is compiled with SMP APIC support or uni-processor APIC support compiled in. This is true for 2.4-ac kernels and kernels 2.4.10 or later. This topic is discussed in more detail in Mikael Pettersson's installation instructions for PerfCtr.
Your CPU must be a Pentium 686/AMD K7 or similar which can generate APIC interrupts for performance counter events. This is _not_ true for some mobile Pentiums and early revisions of the AMD K7 or Athlon.
You can verify that all is working by running the perfctr/examples/perfex program with the -i flag. If you do not see "pcint" as one of the flags, you need to recompile your kernel or buy a real CPU. ;-)

Counting memory load and store instructions on the Pentium 4 is a two step process. First the desired events are tagged at the front of the pipeline. Then tagged events are counted as they graduate from the end of the pipeline. Unfortunately, the tags are all the same 'color' and can't be differentiated as they exit the pipe. Thus, you can correctly measure LD instructions, or correctly measure SR instructions, but if you try to measure them both at once, you will always get the sum of both operations in both counters. The same applies to PAPI_LST_INS.

This behavior is demonstrated in the test program ctests/p4_lst_ins.c.

The moral of the story is to always use these three events one-at-a-time on Pentium 4 machines.

The Pentium 4 can generate floating point instructions either through the x87 floating point unit or with SSE instructions.
Furthermore SSE can generate either packed (multiple operands in one 128-bit register) or unpacked (signal operand in one 128-bit register) instructions.
Depending on your compiler and settings you will get different instruction mixes.
 
PAPI provides 2 preset events to count floating point operations:
- PAPI_FP_INS counts intstructions passing through the floating point unit;
- PAPI_FP_OPS counts something closer to theoretical floating point operations.
 
To minimize the overlap and maximize the usefulness of these two events on Pentium 4, we have made the following choices:
- PAPI_FP_INS always counts only x87 floating point operations.
- PAPI_FP_OPS counts can be customized as discussed below.
 
Further complicating things is that the Pentium 4 hardware is too restrictive to count all these modes at once, so a decision must be made about what to count.
In order to enable PAPI to count these various mixes, we support 2 methods.
 
1) The PAPI_PENTIUM4_FP_xxx defines.
 
   Set these in the EVENTFLAGS of either the Makefile.linux-perfctr-p4 or
   Makefile.linux-perfctr-em64t.
 
   -DPAPI_PENTIUM4_FP_X87
   -DPAPI_PENTIUM4_FP_X87_SSE_SP
   -DPAPI_PENTIUM4_FP_X87_SSE_DP
   -DPAPI_PENTIUM4_FP_SSE_SP_DP
 
   The predefined value for Nocona/EM64T/Pentium 4 Model 3 is:
 
         -DPAPI_PENTIUM4_FP_X87_SSE_DP.
 
   The predefined value for anything else is:
 
         -DPAPI_PENTIUM4_FP_X87.
 
   If nothing is defined, the substrate defaults to:
 
         -DPAPI_PENTIUM4_FP_X87_SSE_DP.
 
2) The PAPI_PENTIUM4_FP environment variable.
 
   Set this to one or two of the following, and it will change the
   behavior of PAPI_FP_OPS.
 
   X87: count all x87 instructions
   SSE_SP: count all unpacked SSE single precision instructions
   SSE_DP: count all unpacked SSE double precision instructions
 
   Due to the design of the register set, only 2 of the three are countable
   at one time. Sorry folks.

PAPI can count 2 different types of vector instructions on the Pentium 4.
Either MMX instructions or packed SSE floating point instructions. These are supported with 2 methods, in a similar fashion to floating point events described above.
 
1) The PAPI_PENTIUM4_VEC_xxx defines.
 
   Set these in the EVENTFLAGS of either the Makefile.linux-perfctr-p4 or
   Makefile.linux-perfctr-em64t.
 
   -DPAPI_PENTIUM4_VEC_MMX
   -DPAPI_PENTIUM4_VEC_SSE
 
   The current default for all platforms is:
 
         -DPAPI_PENTIUM4_VEC_SSE.
 
   If nothing is defined, the substrate defaults to:
 
         -DPAPI_PENTIUM4_VEC_SSE.
 
2) The PAPI_PENTIUM4_VEC environment variable.
 
   Set this to either of the following, and it will change the
   behavior of PAPI_VEC_INS.
 
   SSE: count all packed SSE SP and DP instructions
   MMX: count all 64 and 128 bit MMX instructions

This is a known issue and we are looking in to the cause. Currently, we have no fix or work around.

(The following discussion does not apply to newer quad-core and higher Opteron processors)

The AMD Opteron is the first chip series from AMD that can measure and report floating point operations. Two native events measure floating point activity. One measures speculative operations that enter the FP units; the other measures operations that retire from the FP units.

The retired event generates precise event counts that scale with the amount of work done. However, it measures data movement as well as floating point operations, resulting in counts that are consistently significantly higher than the expected theoretical counts, often by factors of 2 or more.

The speculative event can be configured to generate counts of only the operations typically of interest. Since these counts are speculative, they tend to be higher by often widely variable amounts than expected theoretical counts, especially on complex production codes.

PAPI provides 2 preset events to count floating point operations:

- PAPI_FP_INS counts intstructions passing through the floating point unit;
- PAPI_FP_OPS is intended to count something closer to theoretical floating point operations.

To minimize the overlap and maximize the usefulness of these two events on AMD Opteron, we have made the following choices:

- PAPI_FP_INS always counts retired floating point operations. This value will be precise and accurate, but will include FP loads and stores as well as computations.

- PAPI_FP_OPS counts speculative computation operations by default, but can be customized as discussed below.

As an alternative to counting speculative computations, PAPI_FP_OPS can be configured to retired operations corrected for data movement. Unfortunately, the correction factors themselves are speculative, and can lead to undercounting errors similar in magnitude to those seen in the pure speculative counts.

Two methods are provided to allow customization of PAPI_FP_OPS:

1) The PAPI_OPTERON_FP_xxx defines.

Set these in the CFLAGS variable of Makefile.linux-perfctr-opteron.

-DPAPI_OPTERON_FP_RETIRED
-DPAPI_OPTERON_FP_SSE_SP
-DPAPI_OPTERON_FP_SSE_DP
-DPAPI_OPTERON_FP_SPECULATIVE

The default value is equivalent to:

-DPAPI_OPTERON_FP_SPECULATIVE.

2) The PAPI_OPTERON_FP environment variable.

Set this to one of the following, and it will change the behavior of PAPI_FP_OPS.

RETIRED: count all retired FP instructions
SSE_SP: correct retired counts optimized for single precision
SSE_DP: correct retired counts optimized for double precision
SPECULATIVE: count speculative computations (default)

1) Ultra I/II/III/III+ are currently supported;

2) Some of the cache events have documented bugs, see the Sun UltraSparc hardware reference manual.

3) WARNING FOR PEOPLE USING MULTITHREADED LIBRARIES ON SOLARIS 2.8: There is a bug that prevents setitimer() from being called after the process has called pthread() create at any point in time. Therefore if you suspect your communication library is multithreaded, you had better start the instrumentation before initializing it. See multiplex3_pthreads for details.

The PAPI alpha-tru64 substrate requires OSF 5.1 and a pfm device driver patch available from Bill Gray at Compaq (bgray@zk3.dec.com).

The EV6 has a very small number of countable events.  Only the following PAPI preset events are available:
 
 EV6
   - PAPI_TOT_CYC
   - PAPI_TOT_INS
   - PAPI_RES_STL
   - PAPI_BR_CN
 
The native envents in EV6 are
 {"cycles"},
 {"retinst"},
 {"retcondbranch"},
 {"retdtb1miss"},
 {"retdtb2miss"},
 {"retitbmiss"},
 {"retunaltrap"},
 {"replay"}

- OSF 5.1 and the pfm device driver do not support saving and restoring the counters on context switch. Whoever opens the driver owns the counters.

- Currently multiplexing is not working and as such the tests are skipped.

The Cray X-1 has 4 P-Chip counters per MSP (1 for each SSP), in addition there are 4 E-Chip counters and 16 M-Chip counters. Currently, PAPI only supports 1 valuefor each counter, and thus we send back an aggregate of all the counters. For P-Chips it may be desirable to see each SSP counter seperately, as is the case if a MPI job runs 4 processes on a single MSP and we are looking into ways of providing this for the user, but with the current implementation you can only obtain an aggregate. This of course only affects MSP applications. It is unclear which E-Chip and M-Chip counters should be read depending on the MSP the process is running on or if all should be aggregated. We are still examining the correlation between the process and MSP/ssp it is running on and the E-Chip/M-Chip counters. Thus the interface may end up changing to aggregatecertain counters or return all the values as we wish to do for the P-chip. But the user should be aware that all counters are aggregated currently.

Unlike most other platforms, the overhead on the Cray X1 is dependant on what events you are monitoring. Each chip type (P/M/E) requires its own ioctl call to start/stop and read the hardware counters. Because of this if you monitor events from multiple chips your overhead will double if monitoring events from 2 chip types or triple if monitoring from 3 chips types as most of the overhead comes from the ioctl calls.

The Cray X-1 has hardware overflow support for the P-Chip, however the M-Chip and E-Chip do not support hardware overflow. Because of this, when overflowing on events, if only P-Chip events are being overflowed on then we use hardware overflow. But if we are overflowing on M or E-Chip events (alone, together, or in combination with P-Chip events) then we use software overflowing. P-Chips: When in software mode the threshold being set is compared to the aggregate of all the SSP counters, but when in hardware the threshold is compared against each individual SSP. This means if you have a MSP application and 4 streams andyou want 100,000 threshold, you may want to set the threshold to 25,000 as you could possibly hit 399,997 before an overflow happens with a 100,000 overflow. But this depends on your application.

Hardware profiling is not available on the Cray X-1 and so we use overflows and return the pc to do profiling. Because of this we are either using hardware or software overflowing. Read the Overflow section to determine which will be used.

Only software multiplexing is available, currently software multiplexing only monitors 1 event per time slice, and so multiplexing is discouraged for the X1 since there are 64 possible counters.

For *all* available native event names, run native_avail under the ctests subdirectory. For more information on the native events SEE: man counters

Shared Objects are not supported on the Cray X1 and thus a shared library can not be built and we don't support any shared library information.

Currently this is none supported, but will be added in future releases.

The PAPI_get_real_usec uses the highest resolution timer available on the Cray X1.

The user should be aware that we do not stop the counters when reading the hardware counters as this would require 2 additional ioctl calls per chips (IE worse case scenario 9 ioctl calls to read all chips versus the 3 ioctl calls to read the chips if we don't stop/start the counters). Because of this the PAPI library will affect some of the counts. Since everything is aggregate we loop through the P-Chips 32 counters and add them up and then copy that information to the users array. Each is a long long and thus this will effect the instruction count. The E and M chip are already aggregated from the hardware and so the PAPI interface only has to copy the results.

The test pthreads_zero can possibly crash the X1 system if it is not running the most recent kernel.