TABLE OF CONTENTS

I. Preface

Intended Audience

Organization of This Document

Document Convention

II. Introduction to PAPI

What is PAPI?

PAPI Background/Motivation

PAPI Architecture (Internal Design)

III. How to install PAPI onto your system

IV. C and Fortran Calling Interfaces

V. Events

What are Events?

Native Events

What are Native Events?

Preset Events

What are Preset Events?

Preset Query

Preset Translation

VI. PAPI’S Counter Interfaces

High-Level API

What is a High-Level API?

Initialization of a High-Level API

Reading, Adding, and Stopping Counters

Mflops/s, Real Time, and Processor Time

Low-Level API

What is a Low-Level API?

Initialization of a Low-Level API

Event Sets

What are Event Sets?

Creating an Event Set

Adding events to an Event Set

Starting, Reading, Adding, and Stopping events in an Event Set

Resetting events in an Event Set

Removing events in an Event Set

Emptying and Destroying an Event Set

The State of an Event Set

Getting and Setting Options

Simple Code Examples

High-Level API

Low-Level API

VII. PAPI Timers

Real Time

Virtual Time

VIII. PAPI System Information

Executable Information

Hardware Information

IX. Advanced PAPI Features

Multiplexing

What is Multiplexing?

Using PAPI with Multiplexing

Initialization of Multiplex Support

Converting an Event Set into a Multiplexed Event Set

Issues of Multiplexing

Using PAPI with Parallel Programs

Threads

What are Threads?

Initialization of Thread Support

Thread ID

MPI

Overflow

What is an Overflow?

Beginning Overflows in Event Sets

Address of the Overflow

Statistical Profiling

What is Statistical Profiling?

Generating a PC Histogram

X. PAPI Error Handling

Error Codes

Converting Error Codes to Error Messages

XI. PAPI Mailing Lists

XII. Appendices

Appendix A. Table of Preset Events

Appendix B. High-Level API

Appendix C. Low-Level API

Appendix D. PAPI Supported Platforms

Appendix E. Table of Native Encoding for the Various Platforms

Appendix F. Table of Overhead for the Various Platforms

Appendix G. Table for Multiplexing

Appendix H. Table for Overflow

Appendix I. PAPI Supported Tools 

XIII. Bibliography

INTENDED AUDIENCE

This document is intended to provide the PAPI user with a discussion of how to use the different components and functions of PAPI . The intended users are atyle='font-size:12.0pt'>III. HOW TO INSTALL PAPI ONTO YOUR SYSTEM

This section provides an installation guide for PAPI. It states the necessary steps in order to install PAPI on the various supported operating systems.

IV. C AND FORTRAN CALLING INTERFACES

This section states the header files in which function calls are defined and the form of the function calls for both the C and Fortran calling interfaces. Also, it provides a table that shows the relation between certain pseudo-types and Fortran variable types.

V. EVENTS

This section provides an explanation of events as well as an explanation of native and preset events. The preset query and translation functions are also discussed in this section. There are code examples using native events, preset query, and preset translation with the corresponding output.

VI. PAPI COUNTER INTERFACES

This section discusses the high-level and low-level interfaces in detail. The initialization and functions of these interfaces are also discussed. Code examples along with the corresponding output are included as well.

VII. PAPI TIMERS

This section explains the PAPI functions associated with obtaining real and virtual time from the platform’s timers. Code examples along with the corresponding output are included as well.

VIII. PAPI SYSTEM INFORMATION

This section explains the PAPI functions associated with obtaining hardware and executable information. Code examples along with the corresponding output are included as well.

IX. ADVANCED PAPI FEATURES

This section discusses the advanced features of PAPI, which includes multiplexing, threads, MPI, overflows, and statistical profiling. The functions that are use to implement these features are also discussed. Code examples along with the corresponding output are included as well.

X. PAPI ERROR HANDLING

This section discusses the various negative error codes that are returned by the PAPI functions. A table with the names, values, and descriptions of the return codes are given as well as a discussion of the PAPI function that can be used to convert error codes to error messages along with a code example with the corresponding output.

XI. PAPI MAILING LISTS

This section provides information on PAPI two mailing lists for the users to ask various questions about the project.

XII. APPENDICES

These appendices provide various listings and tables, such as: a table of preset events and the platforms on which they are supported, a table of PAPI supported tools, more information on native events, multiplexing, overflow, and etc.

DOCUMENT CONVENTION

handle_error(1)

A function that passes the argument of 1 that the user should write to handle errors.

WHAT IS PAPI?

PAPI is an acronym for Performance Application Programming Interface. The PAPI Project is being developed at the University of Tennessee’s Innovative Computing Laboratory in the Computer Science Department. This project was created to design, standardize, and implement a portable and efficient API (Application Programming Interface) to access the hardware performance counters found on most modern microprocessors.

BACKGROUND

Hardware counters exist on every major processor today, such as Intel Pentium, IA-64, AMD Athlon, and IBM POWER series. These counters can provide performance tool developers with a basis for tool development and application developers with valuable information about sections of their code that can be improved. However, there are only a few APIs that allow access to these counters, and most of them are poorly documented, unstable, or unavailable. In addition, performance metrics may have different definitions and different programming interfaces on different platforms.

These considerations motivated the development of the PAPI Project. Some goals of the PAPI Project are as follows:

· To provide a solid foundation for cross platform performance analysis tools

· To present a set of standard definitions for performance metrics on all platforms

· To provide a standardize API among users, vendors, and academics

· To be easy to use, well documented, and freely available

ARCHITECTURE

The Figure below shows the internal design of the PAPI architecture. In this figure, we can see the two layers of the architecture:

The Portable Layer consists of the API (low level and high level) and machine independent support functions.

The Machine Specific Layer defines and exports a machine independent interface to machine dependent functions and data structures. These functions are defined in the substrate layer, which uses kernel extensions, operating system calls, or assembly language to access the hardware performance counters. PAPI uses the most efficient and flexible of the three, depending on what is available.

PAPI strives to provide a uniform environment across platforms. However, this is not always possible. Where hardware support for features, such as overflows and multiplexing is not supported, PAPI implements the features in software where possible. Also, processors do not support the same metrics, thus you can monitor different events depending on the processor in use. Therefore, the interface remains constant, but how it is implemented can vary. Throughout this guide, implementation decisions will be documented where it can make a difference to the user, such as overhead costs, sampling, and etc.

On some of the systems that PAPI supports (see Appendix D), you can install PAPI right out of the box without any additional setup. Others require drivers or patches to be installed first.

The general installation steps are below, but first find your particular Operating System’s section of the /papi/INSTALL file for current information on any additional steps that may be necessary.

General Installation
 
1. Pick the appropriate Makefile.<arch> for your system in the papi source distribution, edit it (if necessary) and compile.
 
 % make -f Makefile.<arch>
 
2. Check for errors. Look for the libpapi.a and libpapi.so in the current directory. Optionally, run the test programs in the ‘ftests’ and ‘tests’ directories. 
 Not all tests will succeed on all platforms.
 
 % ./run_tests.sh
 
This will run the tests in quiet mode, which will print PASSED, FAILED, or SKIPPED. Tests are SKIPPED if the functionality being tested is not supported by that platform.
  
3. Create a PAPI binary distribution or install PAPI directly.
 
 To directly install PAPI from the build tree:
 
 % make -f Makefile.<arch> DESTDIR=<install-dir> install
 
 Please use an absolute pathname for <install-dir>, not a relative pathname.
 
 To create a binary kit, papi-<arch>.tgz:
 
 % make -f Makefile.<arch> dist

PAPI is written in C. The function calls stops:.5in'>The function calls in the Fortran interface are defined in the header file, fpapi.h and consist of the following form:

PAPIF_function_name(arg1, arg2, …, check)

As you can probably see, the C function calls have equivalent Fortran function calls (PAPI_<call> becomes PAPIF_<call>). Well, this is true for most function calls, except for the functions that return C pointers to structures, such as PAPI_get_opt and PAPI_get_executable_info, which are either not implemented in the Fortran interface, or implemented with different calling semantics. In the function calls of the Fortran interface, the return code of the corresponding C routine is returned in the argument, check.

For most architectures, the following relation holds between the pseudo-types listed and Fortran variable types:

Pseudo-type	Fortran type	Description
C_INT	INTEGER	Default Integer type
C_FLOAT	REAL	Default Real type
C_LONG_LONG	INTEGER*8	Extended size integer
C_STRING	CHARACTER*(PAPI_MAX_STR_LEN)	Fortran string
C_INT FUNCTION	EXTERNAL INTEGER FUNCTION	Fortran function returning integer result

Array arguments must be of sufficient size to hold the input/output from/to the subroutine for predictable behavior. The array length is indicated either by the accompanying argument or by internal PAPI definitions.

Subroutines accepting C_STRING as an argument are on most implementations capable of reading the character string length as provided by Fortran. In these implementations, the string is truncated or space padded as necessary. For other implementations, the length of the character array is assumed to be of sufficient size. No character string longer than PAPI_MAX_STR_LEN is returned by the PAPIF interface.

For more information on all of the function calls and their job descriptions, see Appendix B for the high-level functions and Appendix C for the low-level functions.

WHAT ARE EVENTS?

Events are occurrences of specific signals related to a processor’s function. Hardware performance counters exist as a small set of registers that count events, such as cache misses and floating point operations while the program executes on the processor. Monitoring these events facilitates correlation between the structure of source/object code and the efficiency of the mapping of that code to the underlying architecture. Each processor has a number of events that are native to and often to that architecture. PAPI provides a software abstraction of these architecture-dependent native events into a collection of preset events that are accessible through the PAPI interface.

NATIVE EVENTS

WHAT ARE NATIVE EVENTS?

Native events comprise the set of all events that are countable by the CPU. In many cases, these events will be available through a matching preset PAPI event. Even if no preset event is available native events can still be accessed directly. These events are intended to be used by people who are very familiar with the particular platform in use. PAPI provides access to native events on all supported platforms through the low-level interface. Native events use the same interface as used when setting up a preset event, but a CPU-specific bit pattern is used instead of the PAPI event definition.

Native encoding is usually:

((register code & 0xffffff) << 8 | (register number & 0xff))

Native encodings are platform dependent, so the above native encoding may or may not work with your platform. To determine the native encoding for your platform, see Appendix F or the README file for your platform in the PAPI source distribution. In addition, the native event lists for the various platforms can be found in the processor architecture manual.

Native events are specified as arguments to the low-level function, PAPI_add_event. In the following code example, a native event is added by using PAPI_add_event with the register code = 0x800000 and the register number = 0x01:

For more code examples, see tests/native.c in the papi source distribution.

PRESET EVENTS

WHAT ARE PRESET EVENTS?

Preset events, also known as predefined events, are a common set of events deemed relevant and useful for application performance tuning. These events are typically found in many CPUs that provide performance counters and give access to the memory hierarchy, cache coherence protocol events, cycle and instruction counts, functional unit, and pipeline status. Furthermore, preset events are mappings from symbolic names (PAPI preset name) to machine specific definitions (native countable events) for a particular hardware resource. For example, Total Cycles (in user mode) is PAPI_TOT_CYC. Also, PAPI supports presets that may be derived from the underlying hardware metrics. For example, Floating Point Instructions per Second is PAPI_FLOPS. A preset can be either directly available as a single counter, derived using a combination of counters, or unavailable on any particular platform.

The PAPI library names approximately 100 preset events, which are defined in the header file, papiStdEventDefs.h. For a given platform, a subset of these preset events can be counted though either a simple high-level programming interface or a more complete C or Fortran low-level interface. For a list and a job description of all the preset events, see Appendix A.

The exact semantics of an event counter are platform dependent. PAPI preset names are mapped onto available events in a way, so it can count as many similar types of events as possible on different platforms. Due to hardware implementation differences, it is not necessarily feasible to directly compare the counts of a particular PAPI event obtained on different hardware platforms. To determine which preset events are available on a specific platform, see Appendix E or run tests/avail.c in the papi source distribution.

PRESET QUERY

The following low-level functions can be called to query about the existence of a preset (in other words, if the hardware supports that certain preset), to query details about a PAPI event, or to acquire details about all PAPI events, respectively:

C:

PAPI_query_event(EventCode)

PAPI_query_event_verbose(EventCode, info)

PAPI_query_all_events_verbose()

Fortran:

PAPIF_query_event(EventCode, check)

PAPIF_query_event_verbose(EventCode, EventName, EventDescr, EventLabel, avail, EventNote, flags, check)

ARGUMENTS

EventCode -- a defined event, such as PAPI_TOT_INS.

EventName -- the event name, such as the preset name, PAPI_BR_CN.

EventDescr -- a descriptive string for the event of length less than PAPI_MAX_STR_LEN.

EventLabel -- a short descriptive label for the event of length less than 18 characters.

avail -- zero if the event CANNOT be counted.

EventNote -- additional text information about an event (if available).

flags -- provides additional information about an event, e.g., PAPI_DERIVED for an event derived from 2 or more other events.

Note that PAPI_query_all_events_verbose is not implemented in Fortran because it returns a C pointer to an array of C structures.

PAPI_query_event asks the PAPI library if the PAPI Preset event can be counted on this architecture. If the event CAN be counted, the function returns PAPI_OK. If the event CANNOT be counted, the function returns an error code. On some platforms, this function also can be used to check the syntax of a native event.

PAPI_query_event_verbose asks the PAPI library for a copy of an event descriptor. This descriptor can then be used to investigate the details about the event. In Fortran, the individual fields in the descriptor are returned as parameters.

PAPI_query_all_events_verbose asks the PAPI library to return a pointer to an array of event descriptors. The number of objects in the array is PAPI_MAX_PRESET_EVENTS and each object is a descriptor as returned by PAPI_query_event_verbose().

OUTPUT (IF THE EVENT, PAPI_TOT_INS, IS AVAILABLE ON YOUR SYSTEM):

In the above code example, PAPI_query_event is used to see if a preset (“PAPI_TOT_INS”) exists, PAPI_query_event_verbose is used to query details about the event, and PAPI_query_all_events_verbose is used to acquire details about all PAPI events:

On success, PAPI_query_event and PAPI_query_event_verbose return PAPI_OK, and on error, a non-zero error code is returned.

On success, PAPI_query_all_events_verbose returns a pointer to an array of PAPI_preset_info_t structures and on error, a null pointer is returned.

For more information about the preset query functions, see Appendix C.

PRESET TRANSLATION

A preset event can be translated to a description, label, number, and string by calling the following low-level functions, respectively:

C:

PAPI_describe_event(EventName, EventCode, EventDescr)

PAPI_label_event(EventCode, EventLabel)

PAPI_event_name_to_code(EventName, EventCode)

PAPI_event_code_to_name(EventCode, EventName)

Fortran:

PAPIF_describe_event(EventName, EventCode, EventDesc, check)

PAPIF_label_event(EventCode, EventLabel, check)

PAPIF_event_name_to_code(EventName, EventCode, check)

PAPIF_event_code_to_name(EventCode, EventName, check)

ARGUMENTS

EventCode -- a defined event of integer type, such as PAPI_TOT_INS.

EventName -- the event name, such as the preset name, PAPI_BR_CN.

EventDescr -- a descriptive string for the event of length less than PAPI_MAX_STR_LEN.

EventLabel -- a short descriptive label for the event of length less than 18 characters.

Note that the preset does not actually have to exist to call these functions.

PAPI_describe_event is used to translate either an ASCII PAPI preset name or an integer PAPI preset event code into the corresponding event code or name as well as an ASCII description of that event. If the EventName argument is a string of length > 0 it is assumed to contain the name to look up and the corresponding event code is returned in the argument, EventCode. Otherwise, the EventCode argument is used to look up the event name, which is stored in the EventName argument. Finally, a descriptive string of length less than PAPI_MAX_STR_LEN is copied to the argument, EventDescr. Note that the functionality of this call is a superset of the PAPI_event_name_to_code and PAPI_event_code_to_name calls.

PAPI_label_event is used to translate an integer PAPI event code into a short (<=18 character) ASCII label that is more descriptive than the preset name but shorter than the description. These labels can be used as event identifiers in third party tools.

PAPI_event_name_to_code is used to translate an ASCII PAPI preset name into an integer PAPI event code.

PAPI_event_code_to_name is used to translate an integer PAPI event code into an ASCII PAPI preset name.

In the following code example, PAPI_event_name_to_code is used to translate a string into an integer (EventCode) and PAPI_label_event is used to label an event from an event code:

OUTPUT:

Note that the event code is in hexadecimal, which is consistent with all the preset translation functions. The hexadecimal values of each preset are specified in the header file, papiStdEventDefs.h.

On success, all the functions return PAPI_OK and on error, a non-zero error code is returned.

For more information about the preset translation functions, see Appendix C.

HIGH-LEVEL API

WHAT IS A HIGH-LEVEL API?

The high-level API (Application Programming Interface) simply provides the ability to start, stop, and read the counters for a specified list of events. It is meant for single thread applications and for programmers wanting simple and coarse-grained measurements. In addition, it is not thread safe and allows only PAPI preset events. Some of the benefits of using the high-level API rather than the low-level API are that it is easier to use and requires less setup (additional code).

It should also be noted that the high-level API could be used in conjunction with the low-level API and in fact does call the low-level API. However, the high-level API by itself is only able to access those events countable simultaneously by the underlying hardware.

There are six functions that represent the high-level API that allow the user to access and count specific hardware events. Note that these functions can be implemented in both C and Fortran. For a list and job description of all the high-level functions, see Appendix B. Also, for a code example of using the high-level interface, see Simple Code Examples: High Level API or tests/high-level.c in the PAPI source distribution.

INITIALIZATION OF A HIGH-LEVEL API

The PAPI library can be initialized implicitly by calling one of the following three high-level functions:

C:

PAPI_num_counters()

PAPI_start_counters(*events, array_length)

PAPI_flops(*real_time, *proc_time, *flpins, *mflops)

Fortran:

PAPIF_num_counters(check)

PAPIF_start_counters(*events, array_length, check)

PAPIF_flops(real_time, proc_time, flpins, mflops, check)

ARGUMENTS

*events -- an array of codes for events such as PAPI_INT_INS or a native event code.

array_length -- the number of items in the events array.

*real_time -- the total real time since the first PAPI_flops call.

*proc_time -- the total process time since the first PAPI_flops call.

*flpins -- the total floating point instructions since the first PAPI_flops call.

*mflops – Mflops/s achieved since the latest PAPI_flops call.

Note that one of the above functions must be called before calling any other PAPI function.

PAPI_num_counters returns the optimal length of the values array for high-level functions. This value corresponds to the number of hardware counters supported by the current substrate. PAPI_num_counters initializes the PAPI library using PAPI_library_init if necessary.

PAPI_start_counters initializes the PAPI library (if necessary) and starts counting the events named in the events array. This function implicitly stops and initializes any counters running as a result of a previous call to PAPI_start_counters. It is the user’s responsibility to choose events that can be counted simultaneously by reading the vendor’s documentation. The length of the events array should be no longer than the value returned by PAPI_num_counters.

The first call to PAPI_flops only initializes the library. For more information on PAPI_flops, see the following section:

Mflops/s, Real Time, and Processor Time.

In the following code example, PAPI_num_counters is used to initialize the library and to get the number of hardware counters available on the system. Also, PAPI_start_counters is used to start counting events:

POSSIBLE OUTPUT (VARIES ON DIFFERENT SYSTEMS):

On success, PAPI_num_counters returns the number of hardware counters available on the system and on error, a non-zero error code is returned.

Optionally, the PAPI library can be initialize explicitly by using PAPI_library_init.

For more information on these functions, see Appendix B.

READING, ADDING, AND STOPPING COUNTERS

Counters can be read, added, and stopped by calling the following high-level functions, respectively:

C:

PAPI_read_counters(*values, array_length)

PAPI_accum_counters(*values, array_length)

PAPI_stop_counters(*values, array_length)

Fortran:

PAPIF_read_counters(*values, array_length, check)

PAPIF_accum_counters(*values, array_length, check)

PAPIF_stop_counters(*values, array_length, check)

ARGUMENTS

*values -- an array where to put the counter values.

array_length -- the number of items in the *values array.

PAPI_read_counters and PAPI_accum_counters read (copy) and add the event counters into the array, values, respectively. The counters are reset and left running after the call of these functions.

PAPI_stop_counters stops the counters started by the function, PAPI_start_counters and return their values.

In the following code example, PAPI_read_counters and PAPI_stop_counters are used to copy and stop event counters in an array, respectively:

On success, all of these functions return PAPI_OK and on error, a non-zero error code is returned.

For more information on these functions, see Appendix B.

MFLOPS/S, REAL TIME, AND PROCESSOR TIME

Mflops/s, real time, and processor time can be obtained by calling the following high-level function:

C:

PAPI_flops(*real_time, *proc_time, *flpins, *mflops)

Fortran:

PAPIF_flops(real_time, proc_time, flpins, mflops, check)

ARGUMENTS

*real_time -- the total real time since the first PAPI_flops call.

*proc_time -- the total process time since the first PAPI_flops call.

*flpins -- the total floating point instructions since the first PAPI_flops call.

*mflops – Mflops/s achieved since the latest PAPI_flops call.

The first call to PAPI_flops initializes the PAPI library, set up the counters to monitor PAPI_FP_INS and PAPI_TOT_CYC events, and start the counters. Subsequent calls will read the counters and return total real time, total process time, total floating point instructions, and the Mflops/s rate since the last call to PAPI_flops. Any call with flpins = -1 will reinitialize all counters to 0.

Note that most platforms are only capable of counting the number of floating point instructions completed. This may or may not translate to your definition of floating point operations. The measured rate is thus Mflops/s, and will in some circumstances count FMA instructions as one operation. Consult the hardware documentation for your system for more details.

PAPI_flops may be called by the user’s application program and contains calls to the following functions:

PAPI_perror, PAPI_library_init, PAPI_get_hardware_info, PAPI_create_eventset, PAPI_add_event, PAPI_start, PAPI_get_real_usec, PAPI_accum, and PAPI_shutdown.

On success, it returns PAPI_OK and on error, a non-zero error code is returned.

For more information on this function, see Appendix B. Also, for a code example, see test/flops.c in the papi source distribution.

LOW-LEVEL API

WHAT IS A LOW-LEVEL API?

The low-level API (Application Programming Interface) manages hardware events in user-defined groups called Event Sets. It is meant for experienced application programmers and tool developers wanting more fine-grained measurements. Unlike the high-level interface, it is thread safe and allows both PAPI preset and native events. Another features of the low-level API are the ability to obtain information about the executable and the hardware as well as to set options for multiplexing and overflow handling. Some of the benefits of using the low-level API rather than the high-level API are that it increases efficiency and functionality.

It should also be noted that the low-level interface could be used in conjunction with the high-level interface, but the user would have to be careful about initialization and threads.

The low-level API is only as powerful as the substrate upon which it is built. Thus, some features may not be available on every platform. The converse may also be true, that more advanced features may be available on every platform and defined in the header file. Therefore, the user is encouraged to read the documentation for each platform carefully. There are approximately 40 functions that represent the low-level API, where some of these functions are implemented only in C or Fortran. For more information on these function and their job descriptions, see Appendix C. Also, for a code example of using the low-level interface, see Simple Code Examples: Low-Level API or tests/low_level.c in the PAPI source distribution.

INITIALIZATION OF A LOW-LEVEL API

The PAPI library can be initialized explicitly by calling the following low-level function:

C:

PAPI_library_init(version)

Fortran:

PAPIF_library_init(check)

ARGUMENT

version -- upon initialization, PAPI checks the argument against the internal value of PAPI_VER_CURRENT when the library was compiled. This guards against portability problems when updating the PAPI shared libraries on your system.

Note that this function must be called before calling any other PAPI function.

The following is a code example of using PAPI_library_init to initialize the PAPI library:

On success, this function returns PAPI_VER_CURRENT.

On error, a positive return code other than PAPI_VER_CURRENT indicates a library version mismatch and a negative return code indicates an initialization error.

For more information on this function, see Appendix C.

EVENT SETS

WHAT ARE EVENT SETS?

Event Sets are user-defined groups of hardware events (preset or native), which are used in conjunction with one another to provide meaningful information, such as: what low-level hardware counters to use, the most recently read counter values, the state of the Event Set (running/not running), and optional settings (e.g., overflow, profiling). Therefore, Event Sets allow a highly efficient implementation and as a result, users can have more detailed and accurate measurements. In addition, Event Sets are managed by the user through the use of integer handles, which helps simplify inter-language calling conventions. There are no real programming restrictions on the use of Event Sets. The user is free to allocate and use any number of them provided the substrate can provide the required resources. They may be used simultaneously and in fact may even share counter values.

CREATING AN EVENT SET

An event set can be created by calling the following the low-level function:

C:

PAPI_create_eventset (*EventSet)

Fortran:

PAPIF_create_eventset(EventSet, check)

ARGUMENT

EventSet -- Address of an integer location to store the new EventSet handle.

Note that EventSet must be initialized to PAPI_NULL before calling this function. Then, the user may add hardware events to the EventSet by calling PAPI_add_event or similar functions.

On success, this function returns PAPI_OK and on error, a non-zero error code is returned.

For more information on this function, see Appendix C. Also, for a code example, see the next section.

ADDING EVENTS TO AN EVENT SET

Hardware events can be added to an event set by calling the following the low-level functions:

C:

PAPI_add_event(*EventSet, EventCode)

PAPI_add_events(*EventSet, *EventCode, number)

Fortran:

PAPIF_add_event(EventSet, EventCode, check)

PAPIF_add_events(EventSet, EventCode, number, check)

ARGUMENTS

*EventSet -- an integer handle for a PAPI Event Set as created by PAPI_create_eventset.

EventCode -- a defined event such as PAPI_TOT_INS.

*EventCode -- an array of defined events.

number -- an integer indicating the number of events in the array *EventCode.

PAPI_add_event adds a single hardware event to a PAPI event set.

PAPI_add_events does the same as PAPI_add_event, but for an array of hardware event codes.

In the following code example, the preset event, PAPI_TOT_INS is added to an event set:

On success, both of these functions return PAPI_OK and on error, a non-zero error code is returned.

For more information on these functions, see Appendix C.

STARTING, READING, ADDING, AND STOPPING EVENTS IN AN EVENT SET

Hardware events in an event set can be started, read, added, and stopped by calling the following low-level functions, respectively:

C:

PAPI_start(EventSet)

PAPI_read(EventSet, *values)

PAPI_accum(EventSet, *values)

PAPI_stop(EventSet, *values)

Fortran:

PAPIF_start(EventSet, check)

PAPIF_read(EventSet, values, check)

PAPIF_accum(EventSet, values, check)

PAPIF_stop(EventSet, values, check)

ARGUMENTS

EventSet -- an integer handle for a PAPI Event Set as created by PAPI_create_eventset.

*values -- an array to hold the counter values of the counting events.

PAPI_start starts the counting events in a previously defined event set.

PAPI_read reads (copies) the counters of the indicated event set into the array, values. The counters are left counting after the read without resetting.

PAPI_accum adds the counters of the indicated event set into the array, values. The counters are reset and left counting after the call of this function.

PAPI_stop stops the counting events in a previously defined event set and return the current events.

The following is a code example of using PAPI_start to start the counting of events in an event set, PAPI_read to read the counters of the same event set into the array values, and PAPI_stop to stop the counting of events in the event set:

On success, these functions return PAPI_OK and on error, a non-zero error code is returned.

For more information on these functions, see Appendix C.

RESETTING EVENTS IN AN EVENT SET

The hardware event counts in an event set can be reset to zero by calling the following low-level function:

C:

PAPI_reset(EventSet)

Fortran:

PAPI_reset(EventSet, check)

ARGUMENT

EventSet -- an integer handle for a PAPI event set as created by PAPI_create_eventset.

Note that the event set must be running or stopped in order to call PAPI_reset.

For example, the EventSet in the code example of the previous section could have been reset to zero by adding the following lines:

if (PAPI_reset(EventSet) != PAPI_OK)

handle_error(1);

On success, this function returns PAPI_OK and on error, a non-zero error code is returned.

For more information on this function, see Appendix C.

REMOVING EVENTS IN AN EVENT SET

A hardware event and an array of hardware events can be removed from an event set by calling the following low-level functions, respectively:

C:

PAPI_rem_event(EventSet, EventCode)

PAPI_rem_events(EventSet, EventCode, number)

.0pt'>ARGUMENTS

EventSet -- an integer handle for a PAPI event set as created by PAPI_create_eventset.

EventCode -- a defined event such as PAPI_TOT_INS or a native event.

*EventCode -- an array of defined events.

number -- an integer indicating the number of events in the array *EventCode.

PAPI_rem_event removes a single hardware event from a PAPI event set.

PAPI_rem_events, does the same as PAPI_rem_event, but for an array of hardware event codes.

In the following code example, PAPI_rem_event is used to removed the event, PAPI_TOT_INS, from
an event set:

On success, these functions return PAPI_OK and on error, a non-zero error code is returned.

For more information on these functions, see Appendix C.

EMPTYING AND DESTROYING AN EVENT SET

All the events in an event set can be emptied and destroyed by calling the following low-level functions, respectively:

C:

PAPI_cleanup_eventset(EventSet)

PAPI_destroy_eventset(EventSet)

Fortran:

PAPIF_cleanup_eventset(EventSet, check)

PAPIF_destroy_eventset(EventSet, check)

ARGUMENT

EventSet -- an integer handle for a PAPI event set as created by PAPI_create_eventset.

Note that the event set must be empty in order to use PAPI_destroy_eventset.

In the following code example, PAPI_cleanup_eventset is used to empty all the events from an event set and PAPI_remove_eventset is used to deallocate the memory associated with the empty event set:

On success, these functions return PAPI_OK and on error, a non-zero error code is returned.

For more information on these functions, see Appendix C.

THE STATE OF AN EVENT SET

The counting state of an Event Set can be obtained by calling the following low-level function:

C:

PAPI_state(EventSet, *status)

Fortran:

PAPIF_state(EventSet, status, check)

ARGUMENTS

EventSet -- an integer handle for a PAPI event set as created by PAPI_create_eventset.

status -- an integer containing a Boolean combination of one or more of the following nonzero constants as defined in the PAPI header file, papi.h:

PAPI_STOPPED	EventSet is stopped
PAPI_RUNNING	EventSet is running
PAPI_PAUSED	EventSet temporarily disabled by the library
PAPI_NOT_INIT	EventSet defined, but not initialized
PAPI_OVERFLOWING	EventSet has overflow enabled
PAPI_PROFILING	PAPI_MULTIPLEXING	EventSet has multiplexing enabled
PAPI_ACCUMULATING	EventSet has accumulating enabled

In the following code example, PAPI_state is used to return the counting state of an EventSet:

OUTPUT:

On success, this function returns PAPI_OK and on error, a non-zero error code is returned.

For more information on this function, see Appendix C.

GETTING AND SETTING OPTIONS

The options of the PAPI library or a specific event set can be obtained and set by calling the following low-level functions, respectively:

C:

PAPI_get_opt(option, ptr)

PAPI_set_opt(option, ptr)

Fortran:

PAPIF_get_clockrate(clockrate)

PAPIF_get_domain(EventSet, domain, mode, check)

PAPIF_get_granularity(EventSet, granularity, mode, check)

PAPIF_get_preload(preload, check)

ARGUMENTS

option -- is an input parameter describing the course of action. The Fortran calls are implementations of specific options. Possible values are defined in papi.h and briefly described below:

Predefined name	Explanation
General information requests
PAPI_GET_CLOCKRATE	Return clockrate in MHz.
PAPI_GET_MAX_CPUS	Return number of CPUs.
PAPI_GET_MAX_HWCTRS	Return number of counters.
PAPI_GET_EXEINFO	Addresses for text/data/bss.
PAPI_GET_HWINFO	Info. about hardware.
PAPI_GET_PRELOAD	Get ‘‘LD_PRELOAD’’ environment equivalent.
Defaults for the global library
PAPI_GET_DEFDOM	Return the default counting domain for newly created event sets.
PAPI_SET_DEFDOM	Set the default counting domain.
PAPI_GET_DEFGRN	Return the default counting granularity.
PAPI_SET_DEFGRN	Set the default counting granularity.
PAPI_GET_DEBUG	Get the PAPI debug state. The available debug states are defined in papi.h. The debug state is available in ptr->debug
PAPI_SET_DEBUG	Set the PAPI debug state
Multiplexing control
PAPI_GET_MULTIPLEX	Get options for multiplexing. Currently not implemented.
PAPI_SET_MULTIPLEX	Set options for multiplexing
Manipulating individual event sets
PAPI_GET_DOMAIN	Get domain for a single event set. The event set is specified in ptr->domain.eventset
PAPI_SET_DOMAIN	Set the domain for a single event set.
PAPI_GET_GRANUL	Get granularity for a single event set. The event set is specified in ptr->granularity.eventset
PAPI_SET_GRANUL	Set the granularity for a single event set.

ptr -- is a pointer to a structure that acts as both an input and output parameter. It is defined in papi.h and below.

EventSet -- input; a reference to an EventSetInfo structure

clockrate -- output; cycle time of this CPU in MHz; *may* be an estimate generated at init time with a quick timing routine

domain -- output; execution domain for which events are counted

granularity -- output; execution granularity for which events are counted

mode -- input; determines if domain or granularity are default or for the current event set

preload -- output; environment variable string for preloading libraries

PAPI_get_opt and PAPI_set_opt query or change the options of the PAPI library or a specific event set created by PAPI_create_eventset. In C interface, these functions pass a pointer to the PAPI_option_t structure. Not all options require or return information in this structure. The Fortran interface is a series of calls implementing various subsets of the C interface. Not all options in C are available in Fortran. Note that some options, such as PAPI_SET_DOMAIN, are also available as separate entry points in both C and Fortran.

The file, papi.h, contains definitions for the structures combined in the PAPI_option_t structure. Users should use the definitions in papi.h that correspond with the library used.

In the following code example, PAPI_get_opt is used to acquire the option, PAPI_GET_MAX_HWCTRS, of an event set and PAPI_set_opt is used to set the option, PAPI_SET_DOMAIN, to the same event set:

POSSIBLE OUTPUT (VARIES ON DIFFERENT PLATFORMS):

On success, these functions return PAPI_OK and on error, a non-zero error code is returned.

For more information on these functions, see Appendix C and for more code examples, see tests/second.c or tests/third.c in the PAPI source distribution.

SIMPLE CODE EXAMPLES

HIGH-LEVEL API

The following is a simple code example of using the high-level API:

POSSIBLE OUTPUT:

Notice that on the second line (after adding the counters) the value is approximately twice as large as the first line (after reading the counters) because PAPI_read_counters resets and leaves the counters running, then PAPI_accum_counters adds the value of the current counter into the values array.

LOW-LEVEL API

The following is a simple code example that does the same technique as the above example, except it uses the Low-Level API:

POSSIBLE OUTPUT:

Notice that in order to get the desired results (the second line approximately twice as large as the first line), PAPI_reset was called to reset the counters, since PAPI_read did not reset the counters.

PAPI timers use the most accurate timers available on the platform in use. These timers can be use to obtain both real and virtual time on each supported platform. The real time clock runs all the time (e.g. a wall clock) and the virtual time clock runs only when the processor is running in user mode.

REAL TIME

Real time can be acquired in clock cycles and microseconds by calling the following low-level functions, respectively:

C:

PAPI_get_real_cyc()

PAPI_get_real_usec()

Fortran:

PAPIF_get_real_cyc(check)

PAPIF_get_real_usec(check)

Both of these functions return the total real time passed since some arbitrary starting point and are equivalent to wall clock time. Also, these functions always succeed (error-free) since they are guaranteed to exist on every PAPI supported platform.

In the following code example, PAPI_get_real_cyc() and PAPI_get_real_usec() are used to obtain the real time it takes to create an event set in clock cycles and microseconds, respectively:

POSSIBLE OUTPUT:

For more information on these functions, see Appendix C.

VI*st-template-ids:1989297510;} RTUAL TIME

Virtual time can be acquired in clock cycles and microseconds by calling the following low-level functions, respectively:

C:

PAPI_get_virt_cyc()

PAPI_get_virt_usec()

Fortran:

PAPIF_get_virt_cyc(check)

PAPIF_get_virt_usec(check)

Both of these functions return the total number of virtual units from some arbitrary starting point. Virtual units accrue every time a process is running in user-mode. Like the real time counters, these functions always succeed (error-free) since they are guaranteed to exist on every PAPI supported platform. However, the resolution can be as bad as 1/Hz as defined by the operating system on some platforms.

In the following code example, PAPI_get_virt_cyc() and PAPI_get_virt_usec() are used to obtain the virtual time it takes to create an event set in clock cycles and microseconds, respectively:

POSSIBLE OUTPUT:

For more information on these functions, see Appendix C.

EXECUTABLE INFORMATION

Information about the executable’s address space can be obtained by using the following low-level function:

C:

PAPI_get_executable_info()

Fortran:

PAPIF_get_exe_info(fullname, name, text_start, text_end, data_start, data_end, bss_start, bss_end, lib_preload_env, check)
 
ARGUMENTS

The following arguments are implicit in the structure returned by the C function, or explicitly returned by Fortran:

fullname -- fully qualified path + filename of the executable

name -- filename of the executable with no path information

text_start, text_end -- Start and End addresses of program text segment

data_start, data_end -- Start and End addresses of program data segment

bss_start, bss_end -- Start and End addresses of program bss segment

lib_preload_env -- environment variable for preloading libraries

Note that the arguments, text_start and text_end, are the only fields that are filled on every architecture.
 
In C, this function returns a pointer to a structure containing information about the current program, such as the start and end addresses of the text, data, and bss segments. 
In Fortran, the fields of the structure are returned explicitly.
 
In the following code example, PAPI_get_executable_info() is used to acquire information about the start and end addresses of the program’s text segment:
 


 

POSSIBLE OUTPUT:

In C, on success, the function returns a non-NULL pointer and on error, NULL is returned.
In Fortran, on success, the function returns PAPI_OK and on error, a non-zero error code is returned.
 
For more information on this function, see Appendix C.

HARDWARE INFORMATION

Information about the system hardware can be obtained by using the following low-level function:
 
C:
PAPI_get_hardware_info()
 
Fortan:
PAPIF_get_hardware_info (ncpu, nnodes, totalcpus, vendor, vendor_string, model, model_string, revision, mhz)
 
ARGUMENTS

The following arguments are implicit in the structure returned by the C function, or explicitly returned by Fortran. totalcpus -- total number of CPUs in the entire system

vendor -- vendor id number of CPU

vendor_string -- vendor id string of CPU

model -- model number of CPU

model_string -- model string of CPU

revision -- Revision number of CPU

mhz -- Cycle time of this CPU; *may* be an estimate generated at initial time with a quick timing routine

In C, this function returns a pointer to a structure containing information about the hardware on which the program runs, such as: the number of CPUs, CPU model information, and the cycle time of the CPU. 
In Fortran, the values of the structure are returned explicitly.
 
Note that if this function were called before PAPI_library_init, it would be undefined.
 
In the following code example, PAPI_get_hardware_info is used to acquire hardware information about the total number of CPUs and the cycle time of the CPU:
 


 

POSSIBLE OUTPUT:

In C, on success, this function returns a non-NULL pointer and on error, NULL is returned.
In Fortran, on success, this function returns PAPI_OK and on error, a non-zero error code is returned.
 
For more information on this function, see Appendix C.

MULTIPLEXING

WHAT IS MULTIPLEXING?

Multiplexing allows more counters to be used than what is supported by the hardware, thus allowing a larger number of events to be counted simultaneously. When a microprocessor has a very limited number of events that can be counted simultaneously, a large application with many hours of run time may require days or weeks of profiling in order to gather enough information to base a performance analysis. Therefore, multiplexing overcomes this limitation by subdividing the usage of the counter hardware over time (timesharing).

USING PAPI WITH MULTIPLEXING

INITIALIZATION OF MULTIPLEX SUPPORT

Multiplex support in the PAPI library can be enabled and initialized by calling the following low-level function:

C:

PAPI_muliplex_init()

Fortran:

PAPIF_multiplex_init(check)

The above function allows more events to be counted than there are physical counters by timesharing the existing counters at some loss in precision. This function should be used after calling PAPI_library_init. After this function is called, the user can proceed to use the normal PAPI routines. It should be also noted that applications that make no use of multiplexing should not call this function.

On success, this function returns PAPI_OK and on error, a non-zero error code is returned.

For more information on this function, see Appendix C and for a code example, see the next section.

CONVERTING AN EVENT SET INTO A MULTIPLEXED EVENT SET

In addition, a standard event set can be converted to a multiplexed event set by the calling the following low-level function:

C:

PAPI_set_multiplex(EventSet)

Fortran:

PAPIF_set_multiplex(EventSet)

ARGUMENT

*EventSet -- a pointer to an integer handle for a PAPI event set as created by PAPI_create_eventset.

The above function converts a standard PAPI event set created by a call to PAPI_create_eventset into an event set capable of handling multiplexed events. This function must be used after calling PAPI_multiplex_init and PAPI_create_eventset, but prior to calling PAPI_start. Events can be added to an event set either before or after converting it into a multiplexed set, but the conversion must be done prior to using it as a multiplexed set.

In the following code example, PAPI_set_multiplex is used to convert a standard event set into a multiplexed event set:

On success, both functions return PAPI_OK and on error, a non-zero error code is returned.

For more information on this function, see Appendix C. Also, for more code examples, see tests/multiplex1.c in the papi source distribution.

ISSUES OF MULTIPLEXING

The following are some issues concerning multiplexing that the PAPI user should be aware of:

· Multiplexing is not supported by all platforms and therefore, PAPI implements software multiplexing on those platforms that do not support multiplexing through the use of a high-resolution interval timer. For more information on which platforms support hardware or software multiplexing, see Appendix H.

· Multiplexing unavoidably incurs a small amount of overhead and can adversely affect the accuracy of reported counter values. In other words, the more events that are multiplexed, the more likely that the results will be incorrect. The granularity of the measured regions must be increased in order to get acceptable results.

· To prevent naïve use of multiplexing by the novice user, the high level API can only access those events countable simultaneously by the underlying hardware, unless a low level function has been called to explicitly enable multiplexing.

USING PAPI WITH PARALLEL PROGRAMS

THREADS

WHAT ARE THREADS?

A thread is an independent flow of instructions that can be scheduled to run by the operating system. Multi-threaded programming is a form of parallel programming where several controlled threads are executing concurrently in the program. All threads execute in the same memory space, and can therefore work concurrently on shared data. Threads can run parallel on several processors, allowing a single program to divide its work between several processors, thus running faster than a single-threaded program, which runs on only one processor at a time.

PAPI only supports thread level measurements with kernel or bound threads, which are threads that have a scheduling entity known and handled by the operating system’s kernel. In most cases like with SMP or OpenMP complier directives, bound threads will be the default. Each thread is responsible for the creation, start, stop, and read of its own counters. When a thread is created, it inherits no PAPI information from the calling thread. There are some threading packages or APIs that can be used to manipulate threads with PAPI, particularly Pthreads and OpenMP. For those using Pthreads, the user should take care to set the scope of each thread to PTHREAD_SCOPE_SYSTEM attribute, unless the system is known to have a non-hybrid thread library implementation.

In addition, PAPI does support unbound or non-kernel threads, but the counts will reflect the total events for the process. Measurements that are done in other threads will get all the same values, namely the counts for the total process. For unbound threads, it is not necessary to call PAPI_thread_init, which will be discussed in the next section.

When threads are in use, PAPI allows the user to provide a routine to its library that returns the thread ID of the currently running thread (for example, pthreads_self for Pthreads) and this thread ID is used as a lookup function for the internal data structures.

INITIALIZATION OF THREAD SUPPORT

Thread support in the PAPI library can be initialized by calling the following low-level function:

C:

PAPI_thread_init(handle, flag)

Fortran:

PAPIF_thread_init(handle, flag, check)

ARGUMENTS

handle -- Pointer to a routine that returns the current thread ID.

flag -- This is reserved for future use and should be set to zero.

This function should be called only once, just after PAPI_library_init, and before any other PAPI calls. If the function is called more than once, the application will exit. Also, applications that make no use of threads do not need to call this function.

The following example shows the correct syntax for using PAPI_thread_init with OpenMP:

On success, the function, PAPI_thread_init, returns PAPI_OK and on error, a non-zero error code is returned.

For more information on this function, see Appendix C and for a code example of using PAPI_thread_init with Pthreads, see the next section.

THREAD ID

The identifier of the current thread can be obtained by calling the following low-level function:

C:

PAPI_thread_id()

Fortran:

PAPIF_thread_id(check)

This function calls the thread id function registered by PAPI_thread_init and returns an unsigned long integer containing the thread identifier.

In the following code example, PAPI_thread_init and PAPI_thread_id are used to initialize thread support in the PAPI library and to acquire the identifier of the current thread, respectively, with Pthreads:

OUTPUT:

On success, this function returns a valid thread identifier and on error, (unsigned long int) –1 is returned.

More information on this function can be found in Appendix C.

For more code examples of using Pthreads and OpenMP with PAPI, see tests/zero_pthreads.c and tests/zero_omp.c in the papi source distribution, respectively. Also, for a code example of using SMP with PAPI, see tests/zero_smp.c in the papi source distribution.

MPI

MPI is an acronym for Message Passing Interface. MPI is a library specification for message-passing, proposed as a standard by a broadly based committee of vendors, implementers, and users. MPI was designed for high performance on both massively parallel machines and on workstation clusters. More information on MPI can be found at http://www-unix.mcs.anl.gov/mpi.

PAPI does support MPI. When using timers in applications that contain multiplexing, profiling, and overflow, MPI uses a default virtual timer and must be converted to a real timer in order to for the application to work properly. Otherwise, the application will exit.

Optionally, the supported tools, Tau and SvPablo, can be used to implement PAPI with MPI.

The following is a code example of using MPI’s PI program with PAPI:

POSSIBLE OUTPUT (AFTER ENTERING 50, 75, AND 100 AS INPUT):

OVERFLOW

WHAT IS AN OVERFLOW?

An overflow is when a particular hardware event exceeds a specified threshold. PAPI provides the ability to call user-defined handlers when an overflow occurs, which is accomplished by setting up a high-resolution interval timer and installing a timer interrupt handler. For the systems that do not support counter overflow at the operating system level, PAPI uses the signal, SIGPROF, by comparing the current counter value against the threshold. If the current value exceeds the threshold, then the user’s handler is called from within the signal context with some additional arguments. These arguments allow the user to determine which event overflowed, how much it overflowed, and at what location in the source code.

Using the same mechanism as for user programmable overflow, PAPI also guards against register precision overflow of counter values. Each counter can potentially be incremented multiple times in a single clock cycle. This fact combined with increasing clock speeds and the small precision of some of the physical counters means that an overflow is likely to occur on platforms where 64-bit counters are not supported in hardware or by the operating system. In those cases, the PAPI implements 64-bit counters in software using the very same mechanism that handles overflow dispatch.

For more information on which platforms support hardware or software overflow, see Appendix I.

BEGINNING OVERFLOWS IN EVENT SETS

An event set can begin registering overflows by calling the following low-level function:

C:

PAPI_overflow(EventSet, EventCode, threshold, flags*st-template-ids:1989297510;} ARGUMENTS

EventSet -- a reference to the event set to use

EventCode -- the counter to be used for overflow detection

threshold -- the overflow threshold value to use

flags -- bit map that controls the overflow mode of operation. This is currently not used and should be set to 0.

handler -- the handler function to call upon overflow

This function marks a specific EventCode in an EventSet to generate an overflow signal after every threshold events are counted. Only one event in an event set can be used as an overflow trigger. Subsequent calls to PAPI_overflow replace earlier calls. To turn off overflow, set the handler to NULL.

In the following code example, PAPI_overflow is used to mark PAPI_TOT_INS in order to generate an overflow signal after every 100,000 counted events:

On success, this function returns PAPI_OK and on error, a non-zero error code is returned.

For more information on this function, see Appendix C and for more code examples, see the tests/overflow.c or tests/overflow_pthreads.c in the papi source distribution.

ADDRESS OF THE OVERFLOW

The address where an overflow occurred can be obtained by calling the low-level function:

C:

PAPI_get_overflow_address(context)

Fortran:

NOT IMPLEMENTED

ARGUMENT

context -- a platform dependent structure containing information about the overflow event. Typically, the signal handler returns this structure automatically.

This function returns the instruction pointer where an overflow occurred and it is often used as part of the overflow handler routine. PAPI_get_overflow_address always returns the value at the offset in the context structure where the instruction pointer should be. No validity testing of this structure is done. If an invalid context pointer is passed to this function, the results will be undefined.

For more information on this function, see Appendix C and for code examples, see the above section as well as tests/overflow.c and tests/overflow_pthreads.c in the papi source distribution.

STATISTICAL PROFILING

WHAT IS STATISTICAL PROFILING?

Statistical Profiling is built upon the method of installing and emulating arbitrary callbacks on overflow. Profiling work as follows: when an event exceeds a threshold, the signal, SIGPROF, is delivered with a number of arguments. Among those arguments is the interrupted thread’s stack pointer and register set. The register set contains the program counter and the address at which the process was interrupted when the signal was delivered. Performance tools like UNIX prof extract this address and hashes the value into a histogram. At program completion, the histogram is analyzed and associated with symbolic information contained in the executable. GNU prof in conjunction with the –p option of the GCC compiler performs exactly this analysis using the process time as the overflow trigger. PAPI aims to generalize this functionality so that a histogram can be generated using any countable event as the basis for analysis.

GENERATING A PC HISTOGRAM

A PC histogram can be generated on any countable event by calling the following low-level functions:

C:

PAPI_profil(buf, bufsiz, offset, scale, EventSet, EventCode, threshold, flags)

PAPI_sprofil(prof, profcnt, EventSet, EventCode, threshold, flags)

Fortran:

PAPI_profil(buf, bufsiz, offset, scale, EventSet, EventCode, threshold, flags, check)

AGRUMENTS

*buf -- pointer to profile buffer array.

bufsiz -- number of entries in *buf.

offset -- starting value of lowest memory address to profile.

scale -- scaling factor for bin values.

EventSet -- The PAPI EventSet to profile when it is started.

EventCode -- Code of the Event in the EventSet to profile.

threshold -- threshold value for the Event triggers the handler.

flags -- bit pattern to control profiling behavior. The defined bit values for the flags variable are shown in the table below:

Defined bit	Description
PAPI_PROFIL_POSIX	Default type of profiling.
PAPI_PROFIL_RANDOM	Drop a random 25% of the samples.
PAPI_PROFIL_WEIGHTED	Weight the samples by their value.
PAPI_PROFIL_COMPRESS	Ignore samples if hash buckets get big.

*prof -- pointer to PAPI_sprofil_t structure.

profcnt -- number of buffers for hardware profiling (reserved).

PAPI_profil creates a histogram of overflow counts for a specified region of the application code by using its first four parameters to create the data structures needed by PAPI_sprofil and then calls PAPI_sprofil to do the work. PAPI_sprofil assumes a pre-initialized PAPI_sprofil_t structure and enables profiling for the EventSet based on its value. Note that the EventSet must be in the stopped state in order for both calls to succeed.

In the following code example, PAPI_profil is used to generate a PC histogram:

On success, these functions return PAPI_OK and on error, a non-zero error code is returned.

For more information on these functions, see Appendix C and for more code examples, see profile.c and sprofile.c in the PAPI source distribution.

ERROR CODES

All of the functions contained in the PAPI library return standardized error codes in which the values that are greater than or equal to zero indicate success and those that are less than zero indicate failure, as shown in the table below:

VALUE	SYMBOL	DEFINITION
0	PAPI_OK	No error
-1	PAPI_EINVAL	Invalid argument
-2	PAPI_ENOMEM	Insufficient memory
-3	PAPI_ESYS	A system or C library call failed, please check errno
-4	PAPI_ESBSTR	Substrate returned an error, usually the result of an unimplemented feature
-5	PAPI_ECLOST	Access to the counters was lost or interrupted
-6	PAPI_EBUG	Internal error, please send mail to the developers
-7	PAPI_ENOEVNT	Hardware event does not exist
-8	PAPI_ECNFLCT	Hardware event exists, but cannot be counted due to counter resource limitations
-9	PAPI_ENOTRUN	No events or event sets are currently not counting
-10	PAPI_EISRUN	Event Set is currently running
-11	PAPI_ENOEVST	No such event set available
-12	PAPI_ENOTPRESET	Event is not a valid preset
-13	PAPI_ENOCNTR	Hardware does not support performance counters
-14	PAPI_EMISC	‘Unknown error’ code

CONVERTING ERROR CODES TO ERROR MESSAGES

Error codes can be converted to error messages by calling the following low-level functions:

C:

PAPI_perror(code, destination, length)

PAPI_strerror(code)

Fortan:

PAPIF_perror(code, destination, check)

ARGUMENTS

code -- the error code to interpret

*destination -- "the error message in quotes"

length -- either 0 or strlen(destination)

PAPI_perror fills the string, destination, with the error message corresponding to the error code (code). The function copies length worth of the error description string corresponding to code into destination. The resulting string is always null terminated. If length is 0, then the string is printed to stderr.

PAPI_strerror returns a pointer to the error message corresponding to the error code (code). If the call fails, the function returns a NULL pointer. Otherwise, a non-NULL pointer is returned. Note that this function is not implemented in Fortran.

In the following code example, PAPI_perror is used to convert error codes to error messages:

OUTPUT:

Notice that the above output was generated from the last call to PAPI_perror.

On success, PAPI_perror returns PAPI_OK and on error, a non-zero error code is returned.

For more information on these functions, see Appendix C.

PAPI has the two following mailing lists for users to ask any questions about the project:

To contact a general users' discussion list for PAPI software:

Send mail to ptools-perfapi@ptools.org

This list is a good place for newbie questions and general conversation about how to use PAPI or tools that use PAPI.

To contact a list of developers of PAPI, performance tools and kernel patches:

Send mail to perfapi-devel@ptools.org

This list is intended for more technical discussions about PAPI. It is intended for developers of PAPI and other performance tools and kernel patches to share observations and insights. Interested hackers are welcomed. All the CVS log messages go here.

To subscribe to either of these mailing lists:

Send a message with blank subject to majordomo@ptools.org. In the body of the message, include 'subscribe <mailing_list>' without the single quotes. If you're having trouble, try sending 'help' in the body to the same address. Should you become hopelessly confused, send mail to the administrator.

APPENDIX A. TABLE OF PRESET EVENTS AND THEIR AVAILABILITY ON SOME PLATFORMS

The following is a table of hardware events that are defined in the header file, papiStdEventDefs.h, which are deemed relevant and useful in tuning application performance. These events have identical assignments in the header files on different platforms, however they may differ in their actual semantics. Therefore, all of these events are not guaranteed to be present on all platforms. The table indicates which events are available on some platforms. Please check your platform's documentation or run tests/avail.c in the papi source distribution to determine the preset events that are available on your platform. Note that these values should not be changed by the user.

PRESET NAME	DESCRIPTION	AMD ATHLON K7	IBM POWER3	INTEL/HP ITANIUM	INTEL PENTIUM III	MIPS R12K	ULTRA SPARC I
PAPI_L1_DCM	Level 1 data cache misses	·	·	·	·	·
PAPI_L1_ICM	Level 1 instruction cache misses	·	·	·	·	·	·
PAPI_L2_DCM	Level 2 data cache misses	·		·	·	·
PAPI_L2_ICM	Level 2 instruction cache misses	·		·	·	·
PAPI_L3_DCM	Level 3 data cache misses			·
PAPI_L3_ICM	Level 3 instruction cache misses			·
PAPI_L1_TCM	Level 1 total cache misses	·	·	·	·	·
PAPI_L2_TCM	Level 2 total cache misses	·		·	·	·
PAPI_L3_TCM	Level 3 total cache misses			·
PAPI_CA_SNP	Requests for a Snoop		·				·
PAPI_CA_SHR	Requests for access to shared cache line (SMP)		·		·	·
PAPI_CA_CLN	Requests for access to clean cache line (SMP)				·
PAPI_CA_INV	Cache Line Invalidation (SMP)				·	·	·
PAPI_CA_ITV	Cache Line Intervention (SMP)		·		·	·
PAPI_L3_LDM	Level 3 load misses			·
PAPI_L3_STM	Level 3 store misses			·
PAPI_BRU_IDL	Cycles branch units are idle		·
PAPI_FXU_IDL	Cycles integer units are idle		·
PAPI_FPU_IDL	Cycles floating point units are idle		·
PAPI_LSU_IDL	Cycles load/store units are idle		·
PAPI_TLB_DM	Data translation lookaside buffer misses	·		·
PAPI_TLB_IM	Instruction translation lookaside buffer misses	·		·	·
PAPI_TLB_TL	Total translation lookaside buffer misses	·	·			·
PAPI_L1_LDM	Level 1 load misses	·	·	·	·		·
PAPI_L1_STM	Level 1 store misses	·	·		·		·
PAPI_L2_LDM	Level 2 load misses	·	·	·
PAPI_L2_STM	Level 2 store misses	·	·	·
PAPI_BTAC_M	Branch target address cache (BTAC) misses		·		·
PAPI_PRF_DM	Pre-fetch data instruction caused a miss		·			·
PAPI_L3_DCH	Level 3 Data Cache Hit			·
PAPI_TLB_SD	Translation lookaside buffer shootdowns (SMP)		·
PAPI_CSR_FAL	Failed store conditional instructions		·			·
PAPI_CSR_SUC	Successful store conditional instructions		·			·
PAPI_CSR_TOT	Total store conditional instructions		·			·
PAPI_MEM_SCY	Cycles Stalled Waiting for Memory Access		·	·		·
PAPI_MEM_RCY	Cycles Stalled Waiting for Memory Read		·
PAPI_MEM_WCY	Cycles Stalled Waiting for Memory Write		·
PAPI_STL_ICY	Cycles with No Instruction Issue		·	·
PAPI_FUL_ICY	Cycles with Maximum Instruction Issue
PAPI_STL_CCY	Cycles with No Instruction Completion		·
PAPI_FUL_CCY	Cycles with Maximum Instruction Completion
PAPI_HW_INT	Hardware interrupts	·			·
PAPI_BR_UCN	Unconditional branch instructions executed	·
PAPI_BR_CN	Conditional branch instructions executed	·	·		·
PAPI_BR_TKN	Conditional branch instructions taken	·		·	·
PAPI_BR_NTK	Conditional branch instructions not taken	·		·	·
PAPI_BR_MSP	Conditional branch instructions mispredicted	·	·	·	·	·	·
PAPI_BR_PRC	Conditional branch instructions correctly predicted	·	·	·	·
PAPI_FMA_INS	FMA instructions completed		·
PAPI_TOT_IIS	Total instructions issued		·		·	·	·
PAPI_TOT_INS	Total instructions executed	·	·	·	·	·	·
PAPI_INT_INS	Integer instructions executed		·
PAPI_FP_INS	Floating point instructions executed		·	·	·	·	·
PAPI_LD_INS	Load instructions executed		·	·		·	·
PAPI_SR_INS	Store instructions executed		·	·		·	·
PAPI_BR_INS	Total branch instructions executed	·	·	·	·	·
PAPI_VEC_INS	Vector/SIMD instructions executed	·			·
PAPI_FLOPS	Floating Point Instructions executed per second		·	·	·	·
PAPI_RES_STL	Cycles processor is stalled on resource	·			·
PAPI_FP_STAL	Cycles any FP units are stalled
PAPI_TOT_CYC	Total cycles	·	·	·	·	·	·
PAPI_IPS	Instructions executed per second	·	·		·	·	·
PAPI_LST_INS	Total load/store instructions executed		·	·		·
PAPI_SYC_INS	Synchronization instructions executed		·
PAPI_L1_DCH	L1 data cache hits	·		·	·
PAPI_L2_DCH	L2 data cache hits	·
PAPI_L1_DCA	L1 data cache accesses	·		·	·
PAPI_L2_DCA	L2 data cache accesses	·		·	·
PAPI_L3_DCA	L3 data cache accesses			·
PAPI_L1_DCR	L1 data cache reads						·
PAPI_L2_DCR	L2 data cache reads	·		·	·
PAPI_L3_DCR	L3 data cache reads			·
PAPI_L1_DCW	L1 data cache writes						·
PAPI_L2_DCW	L2 data cache writes	·		·	·
PAPI_L3_DCW	L3 data cache writes			·
PAPI_L1_ICH	L1 instruction cache hits				·		·
PAPI_L2_ICH	L2 instruction cache hits				·		·
PAPI_L3_ICH	L3 instruction cache hits			·
PAPI_L1_ICA	L1 instruction cache accesses	·			·		·
PAPI_L2_ICA	L2 instruction cache accesses	·			·
PAPI_L3_ICA	L3 instruction cache accesses
PAPI_L1_ICR	L1 instruction cache reads	·		·	·
PAPI_L2_ICR	L2 instruction cache reads			·	·
PAPI_L3_ICR	L3 instruction cache reads			·
PAPI_L1_ICW	L1 instruction cache writes
PAPI_L2_ICW	L2 instruction cache writes
PAPI_L3_ICW	L3 instruction cache writes
PAPI_L1_TCH	L1 total cache hits
PAPI_L2_TCH	L2 total cache hits				·		PAPI_L3_TCH	L3 total cache hits
PAPI_L1_TCA	L1 total cache accesses	·			·
PAPI_L2_TCA	L2 total cache accesses				·		·
PAPI_L3_TCA	L3 total cache accesses
PAPI_L1_TCR	L1 total cache reads
PAPI_L2_TCR	L2 total cache reads				·
PAPI_L3_TCR	L3 total cache reads
PAPI_L1_TCW	L1 total cache writes
PAPI_L2_TCW	L2 total cache writes				·
PAPI_L3_TCW	L3 total cache writes
PAPI_FML_INS	Floating Multiply instructions				·
PAPI_FAD_INS	Floating Add instructions
PAPI_FDV_INS	Floating Divide instructions		·		·
PAPI_FSQ_INS	Floating Square Root instructions		·
PAPI_FNV_INS	Floating Inverse instructions

APPENDIX B. HIGH-LEVEL API

The simple interface implemented by the six routines of the High-Level PAPI API allows the user to access and count specific hardware events. It should be noted that this API could be used in conjunction with the low level API. However, the high level API by itself is only able to access those events countable simultaneously by the underlying hardware. Note that the high level interface performs initialization implicitly and is not thread safe. Under the covers it calls PAPI_library_init(PAPI_VER_CURRENT) and PAPI_thread_init(NULL, 0). Note that the High-Level API fully supports both C and Fortran. For full details on the calling semantics of these functions, please refer to the PAPI Programmer’s Reference.

APPENDIX C. LOW-LEVEL API

The functions of the Low Level PAPI API provide greatly increased efficiency and functionality over the high level API presented in the previous appendix. As mentioned in the introduction, the low level API is only as powerful as the substrate upon which it is built. Thus some features may not be available on every platform. The converse may also be true, that more advanced features may be available and defined in the header file. The user is encouraged to read the documentation for each platform carefully. Note that most functions are implemented in both C and Fortran, but some are implemented in only one of these two languages. For full details on the calling semantics of these functions, please refer to the PAPI Programmer’s Reference.

APPENDIX D. PAPI SUPPORTED PLATFORMS

HARDWARE	OPERATING SYSTEM	REQUIREMENTS
Alpha EV6 & EV67	Tru64 Unix	Contact dcpi@hp.com for required system software
Alpha EV6 & EV67	Linux	IProbe patch (included)
AMD Athlon	Linux 2.2, 2.4	Mikael Pettersson’s Perfctr kernel patch for Linux on web site
Cray SV1, SV2, & T3E	Unicos	None
IBM POWER3, 604, & 604e	AIX 4.3.3	Pmtoolkit from IBM alphaWorks(More information on web site)
IBM POWER4, POWER3, 604, & 604e	AIX 5.1	bos.pmapi must be installed
Intel/HP Itanium I & II	Linux 2.4	None
Intel Pentium Series through Pentium III	Linux 2.2, 2.4	Mikael Pettersson’s Perfctr kernel patch for Linux on web site
Intel Pentium Series through Pentium III	Windows NT, 2000, XP	Administrator privilege for installation
MIPS R10K & R12K	Irix 6.5	None
UltraSparc I, II, & III	Solaris 2.8 or newer	None

More information about the various supported platforms can be found at http://icl.cs.utk.edu/projects/papi/links.

APPENDIX E. TABLE OF NATIVE ENCODING FOR THE VARIOUS PLATFORMS

PLATFORM	NATIVE ENCODING
Alpha EV6 & EV67	((register_code & 0xffffff) << 8 | (register_number & 0xff))
AMD Athlon	(event_code << 8) | (hw_counter_num) event_code = 16 bit event selector code and unit mask. hw_counter_num = Event register number 0 through 1.
Cray SV1, SV2, & T3E	(mask & 0x7) (sel2 & 0xf) (sel1 & 0xf) (sel0 & 0x1) The mask indicates which of the three counters you want counted. If more than one bit is set in the mask, then the counters will be summed into a single event when read. The mask must be non-zero!
IBM POWER4, POWER3, 604, 604e	Low 8 bits indicate which counter number: 0 - 7 Bits 8-16 indicate which event number: 0 - 94
Intel/HP Itanium	((register_code & 0xffffff) << 8 | (register_number & 0xff))
Intel Pentium Series	((register_code & 0xffffff) << 8 | (register_number & 0xff))
MIPS R10K & R12K	Low 8 bits indicate which event number: 0 - 31
UltraSparc I, II, & III	8 bit event code in bits 8-16, counter number in bits 0-7

For more information on the native encoding for your platform, please see the README file for your platform in the papi source distribution.

APPENDIX F. TABLE OF OVERHEAD FOR THE VARIOUS PLATFORMS

< under development >

APPENDIX G. TABLE FOR MULTIPLEXING

< under development >

APPENDIX H. TABLE FOR OVERFLOW

< under development >

APPENDIX I. PAPI SUPPORTED TOOLS

TOOLS	LINKS
Cactus	http://www.cactuscode.org
DEEP/PAPI	http://www.crescentbaysoftware.com/deep_papi_top.html
DynaProf	http://www.cs.utk.edu/~mucci/dynaprof/
GProf	http://www.cs.utah.edu/dept/old/texinfo/as/gprof_toc.html
Paradyn	http://www.cs.wisc.edu/paradyn
Perfometer	http://icl.cs.utk.edu/projects/papi/tools
SCALEA	http://www.par.univie.ac.at/project/scalea/
SvPablo	http://vibes.cs.uiuc.edu/Software/SvPablo/svPablo.htm
TAU	http://www.cs.uoregon.edu/research/paracomp/tau
Vampir	http://www.pallas.com/e/products/vampir/index.htm
Vprof	http://aros.ca.sandia.gov/~cljanss/perf/vprof

Browne, S., J. Dongarra J., Garner N., London K., and Mucci, P., "A Portable Programming Interface for Performance Evaluation on Modern Processors,” University of Tennessee Technical Report, Knoxville, Tennessee, July 2000. http://icl.cs.utk.edu/papi/pubs

Browne, S., Dongarra J., Garner N., London K., and Mucci, P., "A Scalable Cross-Platform Infrastructure for Application Performance Tuning Using Hardware Counters,” Proc. SC'2000, November 2000. http://icl.cs.utk.edu/papi/pubs

Dongarra, J., London, K., Moore, S., Mucci, P., and Terpstra, D. "Using PAPI for Hardware Performance Monitoring on Linux Systems,” Conference on Linux Clusters: The HPC Revolution, Urbana, Illinois, June 25-27, 2001. http://icl.cs.utk.edu/papi/pubs

London, K., Moore, S., Mucci, P., Seymour, K., and Luczak, R. "The PAPI Cross-Platform Interface to Hardware Performance Counters,” Department of Defense Users' Group Conference Proceedings, Biloxi, Mississippi, June 18-21, 2001. http://icl.cs.utk.edu/papi/documens

London, K., Dongarra, J., Moore, S., Mucci, P., Seymour, K., and Spencer, T. "End-user Tools for Application Performance Analysis Using Hardware Counters,” International Conference on Parallel and Distributed Computing Systems, Dallas, TX, August 8-10, 2001. http://icl.cs.utk.edu/papi/pubs

Mucci, P., Moore, S., and Smeds, Nils. “Performance Tuning Using Hardware Counter Data,” Proc. SC’2001, November 2001. http://icl.cs.utk.edu/papi/pubs

Mucci, P. “The IA64 Hardware Performance Monitor and PAPI”, The 2001 International Conference on Parallel and Distributed Processing Techniques and Applications, June 2001. http://icl.cs.utk.edu/papi/pubs

Mucci, P. “PAPI -- The Performance Application Programming Interface,” April 2000. http://icl.cs.utk.edu/papi/pubs

“PAPI Programmer Reference.” December 2001. http://icl.cs.utk.edu/projects/papi/files/html_man/papi.html

“PAPI Software Specification.” December 2001. http://icl.cs.utk.edu/projects/papi/files/papispec21.html

“POSIX Threads Programming.” http://www.llnl.gov/computing/tutorials/workshops/workshop/pthreads/MAIN.html