---

Manual Reference Pages  - monitor (3)

---

monitor - user callbacks for library, process and thread initialization/creation/destruction

CONTENTS

Synopsis
Description
See Also
Bugs
Author
Copyright

#include <monitor.h> 


Callbacks from libmonitor.so to the user’s shared library:

void monitor_init_library(void); 

void monitor_fini_library(void); 

void monitor_init_process(char *process, int argc, char ** argv, unsigned tid); 

void monitor_fini_process(); 

void monitor_init_mpi(int rank, int size, int thread_support_level); 

voidmonitor_fini_mpi(); 

void monitor_init_thread_support(); 

void *monitor_init_thread(unsigned tid); 

void monitor_fini_thread(void *init_thread_data); 

void monitor_dlopen(const char *library); 

 

 
Functions/variables exported by libmonitor.so:

void monitor_real_dlopen(const char *filename, int flags); 

int monitor_real_execve(const char *filename, char *constargv[],char*constenvp[]); 

void monitor_real_exit(int); 

pid_t monitor_real_fork(void); 

long monitor_gettid(); 

int monitor_mpi_comm_size(); 

int monitor_mpi_comm_rank(); 

int monitor_mpi_thread_support(); 

int monitor_force_fini_process(); 

extern int monitor_opt_debug; 

extern int monitor_opt_error; 

The monitor library is designed to make the implementation of performance monitoring tools simple and easy. The functions listed above that are marked as optional are meant to be defined in a user provided shared library. Extra care is taken to appropriately handle important events such as all forms of fork(), exec() as well as variants of exit(), abort(), and assert(). Also, signal handlers for SIGABRT and SIGABRT are installed unless the user installs his own handlers inside monitor_init_process() or monitor_init_library(). Of course, the actual application could override these.

The library is intended to be used with LD_PRELOAD or the equivalent on your platform. This means that monitor DOES NOT WORK WITH STATIC BINARIES. The mechanism is to define LD_PRELOAD to point to the user’s library FIRST and then libmonitor.so. Then any process that runs (and associated threads, forks, and sub-shells) will execute any of these call backs that are defined. See the man page on ld.so or the example code in this distribution for more information how to use LD_PRELOAD.

On platforms that support weak symbols (like GNU/Linux), the above functions are marked as such. This means that if they are not defined by the user, dummy stubs are used and thus no symbols are reported as undefined at run-time. Thus, a very simple performance tool could simply just define one callback (like monitor_init_process()) and not worry about ld.so reporting undefined symbols and your tool crashing.

monitor_init_library() is called after monitor initializes itself but BEFORE anything else does (like the C library). This means that the user should be careful when installing code for this callback. Since this callback executes so early in the load process, various important functions may not work properly. It is highly recommended that the only thing that be done in this callback is to either dynamically load new libraries with dlopen() or load symbols of the current executable with dlsym() to be further with calls in the user’s tool. Please note that monitor_init_library() is not called when the target applications forks. It only runs when the run-time linker loads libmonitor.so.

monitor_fini_library() is there for completeness sake and it is called when the shared library is unloaded from the address space, upon a SIGINT, SIGABRT, abort(), assert() or bypasses normal termination by calling _exit(). The last five behaviors must be explicitly enabled with monitor_opt_error. The same warning applies, there is no guarantee that any part of the system or C library is functioning at this point. Really, don’t do anything here if you don’t have to, use monitor_fini_process() instead. monitor_fini_library() is also called before an exec() as part of the parent process’ cleanup.

monitor_init_process() is called just prior to entry of the main program. It is here that the user’s tool should perform the bulk of its initialization. It is also called after fork() inside the child process. Be careful with your handling of argc and argv, as any modifications will be passed along to main().

monitor_fini_process() is called after main returns and should be used instead of atexit(), which doesn’t always work if the process doesn’t finish cleanly. This routine is also called upon SIGINT, SIGABRT, assert(), abort() if the appropriate optioons are set. It is always called before exec() routines that look like they could succeed. The library does access() checks on the arguments to exec to defer calling this routine as long as possible.. As this routine runs before any libraries are unmapped, all routines in the C libraries and other dynamically loaded libraries are safe to use. If MONITOR_NONZERO_EXIT is set, this hook will also be called when exit() is called with a non-zero argument.

monitor_init_mpi() is called just after a successful call to MPI_Init() or MPI_Init_thread().

monitor_fini_mpi() is called just before a call to MPI_Finalize() after a successful call to either MPI_Init() or MPI_Init_thread().

monitor_init_thread_support() is called ONCE just before pthread_create() runs. It simply allows the user’s code to prepare for multithreaded operation, i.e.to initiatize thread safe data structures, mutexes etc. monitor_init_thread() is called from WITHIN the new thread’s context or ON the new thread’s LWP, just before the entry point specified to pthread_create(). This is very useful for doing accounting of metrics that need first person access to a kernel contexts (like hardware performance counters).

monitor_fini_thread() is handled by calls to pthread_cleanup_push() and pthread_cleanup_pop(). The calls monitor_init_thread() and monitor_fini_thread() are unique in that the first can return some thread specific data which is then passed to the second. This can be used to hold state without having to manage thread specific data or make use of native thread local storage constructs. This hook requires special attention when targeting the GCC 4/OpenMP implementation. This is because that implementation does not return from the function passed to pthread_create() and thus never calls the cleanup handler. For this case, we have implemented a workaround using GOMP_parallel_start(). The side effects here are that monitor_fini_thread() is called for all threads as specified by omp_get_max_threads(). Please ensure robustness in your software by handling this case gracefully. Note that ill-behaved applications that do not guarantee thread completion through pthread_join(), pthread_exit() or by returning from the function passed to pthread_create() do not currently receive the callbacks. A warning is issued to stderr should monitor detect unfinished threads. If you have any ideas how to handle these cases, please let us know. See the BUGS file in the distribution for more information.

monitor_dlopen() is called just after successful execution of dlopen(). It is called after so that the tool could inspect the proc filesystem for information on the new library.

External variables/functions defined by the monitor library:

monitor_force_fini_process() forces the monitor library to call the process finish handler monitor_fini_process() and update it’s internal state to reflect that fact. This may include forcing threads to run their monitor_fini_thread() callbacks. This is useful when using run-time systems that have an API call that cleans up the process before the traditional exit() call, like MPI and MPI_Finalize(). Monitor will guard against multiple invocations of the process finish handlers, even if the process subsequently calls exit().

monitor_real_exit() is a direct call to the system exit function, bypassing any and all monitor functionality. Use this in your error handlers in your tools if you want to avoid callbacks. The same is true for the rest of the monitor_real functions.

monitor_gettid() returns the thread identifier of the calling thread if threads have been enabled by the library. Note that calling this before one receives a monitor_init_thread_support() may result in unintentionally initializing the thread library. This can happen if the code links with the pthread library but doesn’t actually use it. If the executable is not linked with the thread library then monitor_gettid() returns 0.

monitor_mpi_comm_size(), monitor_mpi_comm_rank() and monitor_mpi_thread_support() all return their values from MPI and the MPI_COMM_WORLD communicator as obtained at the time of MPI_Init().

monitor_opt_debug simply enables debugging of the monitor library. By setting this variable to a non-zero value, debug messages are logged to stderr. The messages do not follow any consistent formatting rules. You can preempt the setting of this variable by setting the environment variable MONITOR_DEBUG to be any string.

monitor_opt_error controls whether or not the monitor_fini_library() and monitor_fini_process() callback routines are made upon abnormal program events. Values for this variable may be AND’ed together. The default is that the above routines are never called unless the program exits normally (with a zero exit status). The valid flags for this variable are MONITOR_NONZERO_EXIT, MONITOR_SIGINT and MONITOR_SIGABRT. The latter can be used to catch executions of assert() and abort() in the application code. You can preempt the setting of this variable by setting the environment variable MONITOR_OPTIONS to any combination of the above strings as well as the string MONITOR_DEBUG.

Both of the above variables should be set inside the monitor_init_library() callback to have the desired effect.

Note that debugging messages are generated to stderr if the symbol monitor_opt_debug has been set to non-zero.

ld.so(8), dlopen(3), dlsym(3), abort(3), assert(3), atexit(3), exit(2), exit(3), fork(2), execve(2), access(2), pthread_create(3), pthread_cleanup_push(3), pthread_cleanup_pop(3), monitor-config(1), MPI_Init(), MPI_Init_thread(), MPI_Finalize()

No known bugs.

monitor was written by Philip J. Mucci of the Innovative Computing Laboratory while at the Parallel Center for Computers at the Royal Institute of Technology in Stockholm, Sweden and at SiCortex Inc, in Maynard, MA. This library contains bits of code from HPCToolkit, written by Nathan Tallent, John Mellor-Crummey and Mark Krentel of Rice University. Please see http://www.cs.utk.edu/~mucci, http://icl.cs.utk.edu, and http://www.pdc.kth.se for more information.

This software is COMPLETELY OPEN SOURCE. If you incorporate any portion of this software, I would appreciate an acknowledgement in the appropriate places. Should you find monitor useful, please considering making a contribution in the form of hardware, software or plain old cash.