An Introduction to Distributed Computing

The efficient solution of large problems is an ongoing thread of research in scientific computing. An increasingly popular method of solving these types of problems is to harness disparate computational resources and use their aggregate power as if it were contained in a single machine. This mode of using computers that may be distributed in geography, as well as ownership, has been termed Distributed Computing. Some of the major issues concerned with Distributed Computing are resource discovery, resource allocation and resource management, fault-tolerance, security and access control, scalability, flexibility and performance. Various organizations have developed mechanisms that attempt to address these issues, each with their own perspectives of how to resolve them.

What is NetSolve?

NetSolve http://icl.cs.utk.edu/netsolve/ is an example of a Distributed Computing system that hopes to present functionalities and features that a wide variety of scientists will find highly useful and helpful.

Who is the NetSolve User?

There are two types of NetSolve users. The first type of user is one who installs and accesses only the client interface(s) and utilizes existing pools of resources (agent(s) and server(s)). The second type of NetSolve user installs and administrates his own NetSolve system (client, agent(s), server(s)), and potentially enables his software to be used by NetSolve. This Users' Guide addresses the needs of both types of users. If the user wishes to only install the client interface(s), he should follow instructions in Part II. The User's Manual. However, if the users wishes to install client, agent(s), and server(s), he should follow the instructions in Part III. The Administrator's Manual.

Note that the term "administrates" or "administrator" here simply refers to the person setting up and maintaining the NetSolve agent and server components -- NO ROOT PRIVILEGES ARE NEEDED TO INSTALL OR USE ANY COMPONENT OF THE NetSolve SYSTEM.

The Status of NetSolve

The official release of NetSolve-2.0 is July, 2003. New features implemented in this release include:

• Hardware/Software server, this new feature allows for the transparent movement of problems to take advantage of available hardware/software resources improving load balancing over your NetSolve system.

• Dynamic server, this feature improves the procedure required to add new problems to a NetSolve server by providing a mechanism to add new problems to a server without having to stop the server and recompile again, as was required previously.

• Octave support, Octave is now supported, providing an open source interface for numerical computations comparable to Matlab.

• Condor-G support, NetSolve can now use Condor-G, providing a transparent front-end to a Condor pool and the related functionality of a job queuing system.

• Distributed Storage Interface (DSI), NetSolve can now be integrated with DSI to provide enhanced data transfer and storage.

• GridRPC, this new feature provides a simple remote procedure call mechanism for grid computing using a common API.

• Interface Definition Language (IDL), provides library writers with an improved interface for easier integration of new problems into NetSolve.

• Cygwin/Mac OSX support, the NetSolve agent, server, and client can now be built and ran on Windows operating systems using Cygwin or Mac OSX.

• visPerf Monitor: Improved monitoring system to monitor NetSolve activities using a Java Webstart application.

Installation on Unix Systems using ns_install

The NetSolve distribution tar file is available from the NetSolve web site.http://icl.cs.utk.edu/netsolve/software/ Once the file, NetSolve-2.0.tgz, has been downloaded, the following UNIX commands will create the NetSolve-2.0 directory:

Installation on Unix Systems using ns_install

1.

Run ns_install from the new NetSolve-2.0 directory where the NetSolve package was uncompressed.

UNIX> ./ns_install

2.

The script will first prompt for the components to be installed.

    1. Standard (Client, Server, Agent, Testers, Tools)
    2. Client
    3. Server
    4. Agent
    5. Tools
    6. Testers
    7. GridRPC API
    8. Matlab interface
    9. Octave interface

3.

If your selection above included the Server component you will be asked whether you want to enable GPG for signing software when using the Hardware/Software server feature. It is recommended that you enter y here if you wish to use the Hardware/Software server.

4.

GPG options. If you selected to use GPG then you will be prompted to use version 1.2.3. If this version is present in your PATH you can choose to use it. Otherwise you may select to download GPG yourself ( see http://www.gnupg.org/ ), or let the script handle this download for you, or use a tarball already present. You may also skip GPG at this time.

5.

Finally you will be prompted for any additional arguements to be passed to configure. Options are listed below under 2.1 Command line arguements. For example, if you wanted to limit the number of ports used by NetSolve to 9001 thru 9204, simply enter;

--enable-port-restriction

6.

The script will now configure and build the components selected. If successful you should see, "NetSolve installation complete".

7.

Starting the Agent

bin/$NETSOLVE_ARCH/NS_agent

8.

Starting the Server. You should check the file $NETSOLVE_ROOT/server_config to be sure the AGENT parameter is set for the Agent that you intend to use, then run:

bin/$NETSOLVE_ARCH/NS_server

9.

To test your installation you can run:

bin/$NETSOLVE_ARCH/Test

Installation on Unix Systems

The NetSolve distribution tar file is available from the NetSolve homepage.http://icl.cs.utk.edu/netsolve/software/ Once the file has been downloaded, the following UNIX commands will create the NetSolve-2.0 directory:

gunzip -c NetSolve-2.0.tgz | tar xvf -

From this point forward, we assume that the UNIX SHELL is from the csh family.

The installation of NetSolve is configured for a given architecture using the GNU tool configure.

UNIX> cd NetSolve
UNIX> ./configure

UNIX> ./configure --help

Optional Features:

  --disable-FEATURE            do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]       include FEATURE [ARG=yes]
  --disable-debug              exclude debugging info when compiling
                                - (no effect when --with-cnoptflags and --with-coptflags are used
  --enable-port-restriction    server uses only ports 9001 thru 9204
  --enable-infoserver=alone    use InfoServer alone

  --with-cc                    determine which C compiler to use
  --with-cnooptflags           set compiler flags that don't deal with optimization
                                - (ONLY USE IN COMBINATION WITH --with-cc)
                                - CFLAGS will be set to C_OPT_FLAGS+C_NOOPT_FLAGS
  --with-coptflags             set compiler optimization flags
                                - (ONLY USE IN COMBINATION WITH --with-cc)
                                - CFLAGS will be set to C_OPT_FLAGS+C_NOOPT_FLAGS
  --with-fc                    determine which Fortran compiler to use
  --with-fnooptflags           set compiler flags that don't deal with optimization
                                - (ONLY USE IN COMBINATION WITH --with-fc)
                                - FFLAGS will be set to F_OPT_FLAGS+F_NOOPT_FLAGS
  --with-foptflags             set compiler optimization flags
                                - (ONLY USE IN COMBINATION WITH --with-fc)
                                - FFLAGS will be set to F_OPT_FLAGS+F_NOOPT_FLAGS
  --with-ldflags               set loader flags

  --with-nws=NWSDIR            location of NWS installation dir
  --with-ibp=IBPDIR            location of IBP installation dir
  --with-kerberos=KRBDIR       use Kerberos5 client authentication
  --with-proxy                 which Proxy? (netsolve, globus)
  --with-outputlevel           output level (debug,view,none)

  --with-petsc=PETSCDIR                 location of PETSc installation dir
  --with-petsclibdir=PETSC_LIB_DIR      location of PETSc library
  --with-aztec=AZTEC_DIR                location of Aztec installation dir
  --with-azteclib=AZTEC_LIB             Aztec link line
  --with-superlu=SUPERLU_DIR            location of SuperLU installation dir
  --with-superlulib=SUPERLU_LIB         SuperLU link line
  --with-ma28                           if ma28 is to be included in the NetSolve services
  --with-itpack                         if itpack is to be included in the NetSolve services
  --with-arpacklib=ARPACK_LIB           Arpack link line
  --with-mpi=MPI_DIR                    location of MPI Root Directory
  --with-lapacklib=LAPACK_LIB           LAPACK link line
  --with-scalapacklib=SCALAPACK_LIB     SCALAPACK link line
  --with-blaslib=BLAS_LIB               BLAS link line
  --with-blacslib=BLACS_LIB             BLACS link line
  --with-mldk=MLDK_PATH                 Path to MathLink Development Kit
  --with-rpclib=RPC_LIB                 Full path of RPC library
  --with-octave-include=OCTAVE_INC_DIR  location of Octave include directory
  --with-rpcinc=RPC_INC                 Directory containing RPC header files
  --with-gpg=GPGPATH                    Full path of gpg binary
  --with-buildgpg=BUILDGPG              Location of gpg tar.gz file

The configure script creates two main files, ./conf/Makefile.$NETSOLVE_ARCH.inc and ./conf/Makefile.inc. These files are created from the templates ./conf/Makefile.generic-arch and ./conf/Makefile.inc.in respectively. $NETSOLVE_ARCH is the string printed by the command ./conf/config.guess, with all '-' and '.' characters converted to '_' characters. The variable $NETSOLVE_ROOT is the complete path name to the installed NetSolve directory and is defined in ./conf/Makefile.inc. These *.inc files are included by the Makefiles that build the NetSolve system. Manually editing these configuration files is strongly discouraged. However, if the user prefers to edit this file, details of the $NETSOLVE_ROOT/conf/Makefile.$NETSOLVE_ARCH.inc file are explained in the Section called Details of the Makefile.NETSOLVE_ARCH.inc File in the chapter called Troubleshooting.

Typing make in the NetSolve directory will give instructions to complete the compilation. A typical client compilation includes:

UNIX> make C Fortran tools tester

UNIX> make matlab

UNIX> make mathematica

	Note
	It is recommended that you set the environment variables NETSOLVE_ROOT and NETSOLVE_ARCH prior to running configure.

Testing the Unix installation

Testing solely the client software means that a pre-existing NetSolve system will be contacted, possibly the default agent and servers running at the University of Tennessee. That system can be contacted via the host netsolve.cs.utk.edu which should always be running an agent. The step-by-step procedure to test your NetSolve client installation is as follows:

1. cd NetSolve

2. make tester

3. setenv NETSOLVE_AGENT netsolve.cs.utk.edu

4. $NETSOLVE_ROOT/bin/$NETSOLVE_ARCH/Test

While the tester is running, it prints messages about its execution. This test tests only the C and Fortran77 interfaces. Details of this process are explained in the following chapters. For more information on the C and Fortran77 interfaces, see the chapter called C and Fortran77 Interfaces. the chapter called Matlab Interface and the chapter called Mathematica Interface detail how to test the Matlab and Mathematica interfaces, respectively.

If an error is encountered during testing, refer to the Troubleshooting section of the Errata file for NetSolve. If the error experienced is not listed here you can email netsolve.cs.utk.edu for assistance.

Installation on Windows systems

This section describes the installation and testing of the Windows version of the NetSolve client software. At present, the software is distributed in the form of a self-extracting exe file. The Windows client only works with Windows 2000^TM and Windows XP^TM. It will not run on Windows 98 or earlier.

The contents of the self-extracting exe file are as follows, where NETSOLVE_DIR refers to the directory where you have unzipped the distribution.

The installation process is quite simple.

1. Run the exe you downloaded from the NetSolve webpage to extract the files to a directory.

2. Then run the executable netsolve_install.exe to set the registry keys for NetSolve.

To determine the agent host name, the user can issue the following command from a DOS prompt:

1. C:\>cd NETSOLVE_DIR\tools

2. C:\>getagent

To set a new agent host name, the user must issue the following command:

1. C:\>cd NETSOLVE_DIR\tools

2. C:\>setagent [agent host name]

   If the agent host name is not specified on the command line, you will be prompted for a host name. You will have the option of specifying a name or accepting the current agent name set in the registry.

The de-installation process is quite similar.

1. C:\>cd NETSOLVE_DIR

2. C:\>netsolve_install -uninstall

   The above program removes the keys from the Windows registry.

3. C:\>rmdir /s NETSOLVE_DIR

Testing the Windows installation

You can use the various programs in the NETSOLVE_DIR\testing directory to test your NetSolve installation. Remember that a valid NetSolve agent and server should already be running, and the required problems should be installed on the servers. Here is a list of test programs and the problems they make use of:

For example, to perform a sample run of c_test, the user must do the following:

1. Use setagent to point to the correct agent host. ( e.g. setagent netsolve.cs.utk.edu )

2. Run c_test.exe from the testing directory.

NetSolve Problem Specification

Solving a computational problem with NetSolve is a function evaluation:

<output> = <name>(<input>)

• <name> is a character string containing the name of the problem,

• <input> is a list of input objects, and

• <output> is a list of output objects.

An object is itself described by an object type and a data type. The types available in the current version of NetSolve are shown in Table 1 in the Section called NetSolve Objects in the chapter called The Problem Description File in Part III and Table 2 in the Section called NetSolve Objects in the chapter called The Problem Description File in Part III. Rather than giving examples for each object type, we refer the reader to the programs in $NETSOLVE_ROOT/src/Examples, $NETSOLVE_ROOT/src/Testing, and $NETSOLVE_ROOT/src/Tutorial. The user can also refer to the Section called Mnemonics in the chapter called The Problem Description File for a description of the requirements for each NetSolve object type as it relates to the problem description file.

Available Client Interfaces

NetSolve provides a variety of client interfaces:

• C, Fortran interfaces are detailed in the chapter called C and Fortran77 Interfaces.

• Matlab interface is detailed in the chapter called Matlab Interface.

• Mathematica interface is detailed in the chapter called Mathematica Interface.

• Octave interface is detailed in the chapter called Using Octave with NetSolve.

• An Excel interface called ActiveSheets is also being developed.

Problems that can be solved with NetSolve

In order for a problem to be solved (i.e., a function or library routine to be invoked) using NetSolve, there must exist a problem description file (PDF) corresponding to the problem/routine. A variety of PDFs are included with the NetSolve distribution. A user can also write his own PDF for his function, as described in the chapter called The Problem Description File.

The default NetSolve distribution only provides a subset of enabled software to test the various client interfaces. Interfaces have been written for a variety of software libraries (refer to $NETSOLVE_ROOT/problems/), but as the libraries themselves are not included in the NetSolve distribution, the library interfaces are not enabled. Therefore, the user can customize his installation to include these existing interfaces and/or new interfaces. Refer to the Section called Installation on Unix Systems using ns_install in the chapter called Downloading, Installing, and Testing the Agent and Server for further details.

It is possible to query a NetSolve agent to obtain a list and descriptions of the problems that can be solved by its respective servers. There are several ways of sending such queries.

1. From the NetSolve homepage, it is possible to specify an agent name and run CGI scripts to obtain detailed information about NetSolve problems, including C and Fortran calling sequence specifications.

2. Problem lists and descriptions are also directly available from the Matlab interface, the Mathematica interface, and the Octave interface.

3. The NetSolve management tools described in the chapter called NetSolve Management Tools for Administrators give access to that information from the UNIX prompt.

Naming Scheme for a NetSolve problem

The full name of a NetSolve problem has two parts:

1. the path, and

2. the nickname.

Let us demonstrate this with an example. The problem nicknamed ddot, which computes the inner product of two double-precision vectors, has the full name /BLAS/Level1/ddot. This problem can be found in $NETSOLVE_ROOT/problems/blas. This full name has two purposes. First, when we display a list of problems, they are sorted alphabetically by their full name, and the problems are grouped by "directory". Second, by convention, the first element of the full name (e.g., BLAS) is the name of the numerical library containing the operation (problem). This convention has proven to be useful, as seen in the Section called What is the Calling Sequence? in the chapter called C and Fortran77 Interfaces.

Introduction

As mentioned in the Section called Installation on Unix Systems using ns_install in the chapter called Downloading, Installing, and Testing the Client, the C/Fortran77 client interfaces for NetSolve are built by typing

UNIX> make C Fortran

• $NETSOLVE_ROOT/lib/$NETSOLVE_ARCH/libnetsolve.a : the C library

• $NETSOLVE_ROOT/lib/$NETSOLVE_ARCH/libfnetsolve.a : the Fortran77 library

Before linking to one of these libraries, the user must include the appropriate header file in his program:

• $NETSOLVE_ROOT/include/netsolve.h in C,

• $NETSOLVE_ROOT/include/fnetsolve.h in Fortran77.

The Fortran77 include file is not mandatory, but increases the source program readability by allowing calling subroutines to manipulate the NetSolve error codes by variable name rather than by integer value.

The Fortran77 interface is built on top of the C interface since all of the networking underneath NetSolve is written in C. However, we chose to write the Fortran77 interface with subroutines instead of functions (for reasons of compiler compatibilities). The C functions all return a NetSolve error code equal to 0 if the call was successful or to a negative value in case of error. the chapter called Error Handling in NetSolve contains the list of all possible error codes. The Fortran77 subroutines take an extra output integer argument (passed by reference) at the beginning of the calling sequence that contains the error code after completion of the call. The reference manuals for C and Fortran77 are in the chapter called C Reference Manual and the chapter called Fortran Reference Manual.

The basic concepts here are the same as the ones discussed in the chapter called Matlab Interface for the Matlab interface, especially the ability to call NetSolve in a blocking or nonblocking fashion.

We describe the C and Fortran77 interfaces by the means of an example. In the following section we start developing the example by demonstrating how a user can obtain information about the calling sequence to a given problem.

What is the Calling Sequence?

The C and Fortran77 interfaces, as they are not object-capable, need to use specific calling sequences that are more involved than the ones used from Matlab or Mathematica. In the Section called NetSolve Problem Specification in the chapter called Introduction to the NetSolve Client, we described the input and output arguments of a NetSolve problem as lists of objects. The Matlab and Mathematica interfaces to NetSolve can manipulate objects directly and it is therefore very easy to call NetSolve from their interfaces once problem descriptions are known. From interfaces that are not object-oriented (C and Fortran), it is necessary to use a calling sequence that describes the objects' features individually. For complete details, the user should refer to the chapter called C and Fortran77 Interfaces and the Section called Sparse Matrix Representation in NetSolve in the chapter called The Problem Description File.

Let us take a very simple example: the user wants to perform a dense linear system solve. The first thing to know, as stated in earlier chapters, is the name or IP address of a host running a NetSolve agent. The default NetSolve agent running at the University of Tennessee is aware of many servers that can perform the computation. In fact, a dense linear system solve is provided with the NetSolve distribution as default numerical software for the server. The user has now two possible courses of action to find out about the problem. Let us assume that the user chooses to use the UNIX command line management tools (see the chapter called NetSolve Management Tools for Administrators for a complete description of these tools). The alternative would be to use the CGI scripts on the NetSolve homepage.

the Section called Expanding the Server Capabilities in the chapter called Downloading, Installing, and Testing the Agent and Server shows how the servers specify the calling sequence to a given problem. It is usual for servers to enforce the same calling sequence as the original numerical software and to give a problem the name of the original library function. In the example, dgesv() is the name of an LAPACK subroutine and the user can therefore expect the calling sequence for the problem dgesv to match the one of the subroutine. One can see in the problem list returned by NS_problems a problem called linsol. In this example, linsol is a simplified version of dgesv and has a simplified calling sequence chosen by whomever started the first server that provides access to that problem. Since linsol is not the name of an LAPACK subroutine, its calling sequence can be arbitrary.

UNIX> NS_problems netsolve.cs.utk.edu
/LAPACK/LinearSystems/dgesv
/LAPACK/LinearSystems/linsol

Next, two situations are possible. First, the user already knows the numerical software (e.g., LAPACK) and may even have code already written in terms of this software. In this case, the switching to NetSolve is immediate. The second possibility is that the user does not know the software. If this is the case, he needs to pay close attention to the output given by NS_probdesc. The output from this command first gives the calling sequence as it would be invoked from Matlab, and then gives the calling sequence from C/Fortran.

UNIX> NS_probdesc netsolve.cs.utk.edu dgesv
-- dgesv -- From LAPACK -
Compute the solution to a real system of linear equations
  A * X = b
where A is an N-by-B matrix and X and B are N-by-NRHS matrices.
Matlab Example : [x y z info ] = netsolve('dgesv',a,b)
http://www.netlib.org/lapack/index.html
* 2 objects in INPUT
 - input 0: Matrix Double Precision Real.
 Matrix A
 - input 1: Matrix Double Precision Real.
 Right hand side
* 4 objects in OUTPUT
 - output 0: Matrix Double Precision Real.
 LU factors ( A = P*L*U)
 - output 1: Vector Integer.
 Vector of pivots (defines the P matrix)
 - output 2: Matrix Double Precision Real.
 Solution
 - output 3: Scalar Integer.
 INFO
  0  successful
  <0 error on calling ?
  >0 QR algorithm failed
* Calling sequence from C or Fortran
8 arguments
 - Argument #0:
   - number of rows of input object #0 (A)
   - number of columns of input object #0 (A)
   - number of rows of input object #1 (RHS)
 - Argument #1:
   - number of columns of input object #1 (RHS)
 - Argument #2:
   - pointer to input object #0 (A)
   - pointer to output object #0 (LU)
   - pointer to output object #0 (LU)
 - Argument #3:
   - leading dimension of input object #0 (A)
 - Argument #4:
   - pointer to output object #1 (PIVOT)
 - Argument #5:
   - pointer to input object #1 (RHS)
   - pointer to output object #1 (PIVOT)
   - pointer to output object #2 (SOLUTION)
 - Argument #6:
   - leading dimension of input object #1 (RHS)
 - Argument #7:
   - pointer to output object #3 (INFO)

This output can appear rather cryptic at first. Let us work through it step by step. First, the number of arguments in the calling sequence is 8. This means that the call from C will look like:

status = netsl('dgesv()',X0,X1,X2,X3,X4,X5,X6,X7);

CALL FNETSL('dgesv()',STATUS,X0,X1,X2,X3,X4,X5,X6,X7)

Blocking Call

The blocking call to NetSolve from C or Fortran77 is the easiest to implement. Specifically, if the main program is in C, one calls the function, netsl(), and if the main program is in Fortran77, one calls the function, FNETSL(). This C function returns an error code. It takes as arguments the name of a problem and the list of input data. These inputs are listed according to the calling sequence discussed in the Section called What is the Calling Sequence?. The C prototype of the function is

int netsl(char *problem_name, ... < argument list > ...)

 SUBROUTINE FNETSL( PROBLEM_NAME, STATUS, ... < argument list > ...)

Let us resume our example of the call to dgesv. In Fortran77, the direct call to LAPACK looks like

CALL DGESV( N, 1, A, LDA, IPIV, B, LDB, INFO )

 CALL FNETSL('DGESV()', STATUS, N, 1, A, LDA, IPIV, B, LDB, INFO )

The call in C is

status = netsl('dgesv()',n,1,a,lda,ipiv,b,ldb,&info);

From the user's point of view, the call to NetSolve is exactly equivalent to a call to LAPACK. One detail, however, needs to be mentioned. Most numerical software is written in Fortran77 and requires users to provide workspace arrays as well as data, since there is no possibility for dynamic memory allocation. Because we preserved the exact calling sequence of the numerical software, we require the user to pass those arrays. But, since the computation is performed remotely, workspace on the client side is meaningless. It will, in fact, be dynamically created on the server side. Therefore, when the numerical software would require workspace, the NetSolve user may provide a one-length array for workspace.

This is signaled in the output of NS_probdesc by an argument description such as:

 - Argument #6:
   - ignored

Nonblocking Call

The nonblocking call allows the user to have some sort of NetSolve-parallelism. The nonblocking version of netsl() is netslnb(). Similarly, the nonblocking version of FNETSL() is FNETSLNB(). The user calls it exactly as he would call netsl() or FNETSL(). If the call to netslnb() or FNETSLNB() is successful, it returns a request handler in the form of a (positive) integer. If it is not successful, it returns an error code. Continuing with our example:

 CALL FNETSLNB( 'DGESV()', REQUEST, N, 1, A, LDA, IPIV,
&               B, LDB, INFO )

request = netslnb('dgesv()',n,1,a,max,ipiv,b,max,&info);

In case of an error, the request handler actually contains the (negative) NetSolve error code.

The next step is to check the status of the request. As in the Matlab interface, the user can choose to probe or to wait for the request. Probing is done by calling netslpr() or FNETSLPR() which returns a NetSolve error code:

CALL FNETSLPR( REQUEST, INFO )

info = netslpr(request);

Typical error codes returned are NetSolveNotReady and NetSolveOK (see the chapter called Error Handling in NetSolve). Waiting is done by using netslwt() or FNETSLWT(). This function blocks until the computation is complete and the result is available. Here is the Fortran77 call:

CALL FNETSLWT( REQUEST, INFO )

info = netslwt(request);

Catching errors

Given a NetSolve error code, there is a function in the C and Fortran77 interface that prints explicit error messages to the standard error. The C call is :

netslerr(info);

CALL FNETSLERR( INFO )

Row- or column-major

To allow the NetSolve user to store her/his matrices either in row-wise or column-wise fashion, we also provide the function netslmajor() in C and FNETSLMAJOR() in Fortran77. This function can be called at any time in the user's program in C:

netslmajor("col");
netslmajor("row");

CALL FNETSLMAJOR('col')
CALL FNETSLMAJOR('row')

All of the subsequent calls to NetSolve will assume the corresponding major. The default values are of course row-wise for C and column-wise for Fortran77.

Limitations of the Fortran77 interface

Due to Fortran77's restrictions for the use of pointer and its inability to dynamically allocate memory, the Fortran77 interface to NetSolve does not support the PACKEDFILES and STRINGLIST object type. It also does not support output objects of type STRING.

Built-in examples

C and Fortran77 examples are included in the NetSolve distribution in $NETSOLVE_ROOT/src/Examples. To build them, modify the Makefiles as needed in each example directory. The examples use different problems that are available on servers at the University of Tennessee. They should help the user to understand how the system works. We also have full examples in C and Fortran in the appendix called Complete C Example and the appendix called Complete Fortran77 Example.

Introduction

Building the Matlab interface by typing

UNIX> make matlab

The ### part of the extension depends on the architecture (for instance, the extension is .mexsol for the Solaris Operating System). These four files alone are the Matlab interface to NetSolve. To make these four files accessible to Matlab, the user must modify the MATLABPATH environment variable as:

UNIX> setenv MATLAB_VERSION R13 ( or R12.1 )
UNIX> setenv MATLABPATH $NETSOLVE_ROOT/bin/$NETSOLVE_ARCH/MATLAB/$MATLAB_VERSION

What to Do First

Let us assume that the user has compiled the Matlab interface, set an agent name, started a Matlab session and is now ready to use NetSolve. In this section we describe those features of the interface that allow the user to obtain information about the currently available NetSolve system.

As stated briefly in the Section called Problems that can be solved with NetSolve in the chapter called Introduction to the NetSolve Client, it is possible to obtain the list of solvable problems from Matlab, as well as from the homepage CGI scripts or the management tools. In the case of Matlab, this information is obtained by typing the following command

>> netsolve
NetSolve - List of available problems -
/BLAS-wrappers/Level3/dmatmul
/BLAS-wrappers/Level3/zmatmul
/BLAS/Level1/daxpy
/BLAS/Level1/ddot
/BLAS/Level1/zaxpy
/BLAS/Level2/dgemv
/BLAS/Level3/dgemm
/BLAS/Level3/zgemm
/LAPACK-wrapper/Simple/Eig_and_Singular/eig
/LAPACK-wrapper/Simple/Linear_Equations/linsol
/LAPACK/Simple/Linear_Equations/dgesv
/LAPACK/Auxiliary/dlacpy
/Mandelbrot/mandelbrot
/QuickSort/DoublePrecision/dqsort
/QuickSort/Integer/iqsort
/SCALAPACK/LinearSystem/pdgesv
/SCALAPACK/LinearSystem/pdposv
/SCALAPACK/LinearSystem/plinsol
/SuperLU-MA28/sparse_direct_solve
-----------------------
[ output args ] = netsolve(problem name, input args)
-----------------------
Information on a specific problem : netsolve(problem name)
Information on the servers : netsolve('?')
-----------------------
>>

>> netsolve('eig')
-- eig -- Wrapper around the LAPACK routine DGEEV --
Simplified version of DGEEV.
Computes the eigenvalues of a double precision real
matrix A. Returns two double precision real
vectors containing respectively the real parts and
the imaginary parts of the eigenvalues.

MATLAB Example : [r i ] = netsolve('eig',a)

* 1 objects in INPUT
 - input 0: Matrix Double Precision Real.
 Matrix A
* 2 objects in OUTPUT
 - output 0: Vector Double Precision Real.
 Real parts of the eigen values
 - output 1: Vector Double Precision Real.
 Imaginary parts of the eigen values
--------------------------------------
Output Objects 0 and 1 can be merged.
>> 

Since Matlab provides a mechanism to manipulate complex objects, it is probable that the user would like to have eig return one single complex vector instead of two separate real vectors. Thus, in the Matlab interface it is possible to merge these two real output vectors into one complex vector. This point is further developed in the next section.

The Matlab interface has another feature that is concerned not with the actual problem solving, but with providing information about the NetSolve configuration itself. We have just seen how to get information about the problems handled by the NetSolve servers; it is also possible to obtain the physical locations of these servers. Let us assume that our NETSOLVE_AGENT environment variable is set to netsolve.cs.utk.edu (see the chapter called Running the NetSolve Agent). The command

>> netsolve('?')

NetSolve - List of available agents -
netsolve.cs.utk.edu(160.36.58.109)
NetSolve - List of available servers -
maruti.cs.berkeley.edu(128.32.36.223)
cupid.cs.utk.edu(160.36.58.63)
torc3.cs.utk.edu(160.36.56.202) (0 failures)

The same information can be obtained from the homepage CGI scripts or the management tools.

Calling netsolve() to perform computation

The easiest way to perform a numerical computation in NetSolve is to call the function netsolve(). With this function, the user sends a blocking request to NetSolve. By blocking we mean that after typing the command in the Matlab session, the user resumes control only when the computation has been successfully completed on a server. The other way to perform computation is to send a nonblocking request as described in the Section called Calling netsolve_nb().

Let us continue with the eig example we started to develop in the preceding section. The user now knows that he has to provide a double-precision square matrix to NetSolve, and he knows that he is going to get two real vectors back (or one single complex vector). He first creates a 300 × 300 matrix, for instance,

>> a = rand(300);

>> [x y] = netsolve('eig',a)

Let us see what happens when we type:

>> [x y] = netsolve('eig',a)
Sending Input to Server zoot.cs.utk.edu
Downloading Output from Server zoot.cs.utk.edu

x =          y =
    10.1204             0
    -0.9801        0.8991
    -0.9801       -0.8991
    -1.0195             0
    -0.6416        0.6511
      ...           ...
      ...           ...

As mentioned earlier, the user can decide to merge x and y into one single complex vector. Let us make it clear again that this possibility is a specificity only to functions that state in the end of their problem specification, "Output objects a and b can be merged." and is not available in general for all problems. To merge x and y, the user has to type:

>> [x] = netsolve('eig',a)
Sending Input to Server zoot.cs.utk.edu
Downloading Output from Server zoot.cs.utk.edu

x =  
    10.1204 
    -0.9801 + 0.8991i
    -0.9801 - 0.8991i
    -1.0195    
    -0.6416 + 0.6511i
        .........       
        .........      

Calling netsolve_nb()

The obvious drawback of the function netsolve() is that while the computation is being performed remotely, the user must wait to regain control of the prompt. To address this drawback, we provide a nonblocking function, netsolve_nb(). The user can then do work in parallel and check for the completion of the request later. He can even send multiple requests to NetSolve. Thanks to the load-balancing strategy implemented in the NetSolve agent, all these requests will be solved on different machines if possible, achieving some NetSolve-parallelism. Let us now describe this function with the eig example.

As in the Section called Calling netsolve() to perform computation, the user creates a 300 × 300 matrix and calls NetSolve:

>> a = rand(300);
>> [r] = netsolve_nb('send','eig',a)

Let us resume our example and see what NetSolve answers to the first call to netsolve_nb():

>> [r] = netsolve_nb('send','eig',a)
Sending Input to Server zoot.cs.utk.edu
rd->request_id = 0

r =

     0

>> 

netsolve_nb() returns a request handler: 0. This request handler will be used in the subsequent calls to the function. The request is being processed on zoot, and the result will eventually return. The user can obtain this result in one of two ways. The first one is to call netsolve_nb() with the 'probe' action:

>> [status] = netsolve_nb('probe',r)

netsolve_nb() returns the status of a pending request. The right-hand side contains the action, as is required for netsolve_nb(), and the request handler. This call returns immediately, and prints a message. Here are the two possible scenarios:

>> [status] = netsolve_nb('probe',r)
Not ready yet
status = -1
...
>> [status] = netsolve_nb('probe',r)
Result available
status = 1

To obtain the result of the computation one must call netsolve_nb() with the 'wait' action:

>> [x y] = netsolve_nb('wait',r)
Downloading Output from Server zoot.cs.utk.edu

x =          y =
    10.1204             0
    -0.9801        0.8991
    -0.9801       -0.8991
    -1.0195             0
    -0.6416        0.6511
      ...           ...
      ...           ...

As with the netsolve() function, one can merge the real part and the imaginary part into a single complex vector. The typical scenario is to call netsolve_nb() with the action 'send', then make repeated calls with the action 'probe' until there is nothing more to do than wait for the result. The user then calls netsolve_nb() with the action 'wait'. It is of course possible to call netsolve_nb() with the action 'wait' before making any call with the action 'probe'. One last action can be passed to netsolve_nb(), as shown here:

>> netsolve_nb('status')

>> a=rand(100); b = rand(150);
>> [r1] = netsolve_nb('send','eig',a)
Sending Input to Server zoot.cs.utk.edu
rd->request_id = 0

r1 =

     0

>> [r2] = netsolve_nb('send','eig',b)
Sending Input to Server zoot.cs.utk.edu
rd->request_id = 1

r2 =

     1

>> netsolve_nb('status')
--- NetSolve: pending requests ---
Requests #0: 'eig', submitted to zoot.cs.utk.edu (160.36.58.152)
        was started 24 seconds ago.
netsolveProbeRequest returned: 1, ns_errno = 0
        Completed
Requests #1: 'eig', submitted to zoot.cs.utk.edu (160.36.58.152)
        was started 7 seconds ago.
netsolveProbeRequest returned: 1, ns_errno = 0
        Completed

The user can check what requests he has sent so far and obtain an estimation of the completion times. By using the 'status' action, the user can also determine whether a request is still running or has been completed. By sending multiple non-blocking requests to NetSolve and relying on the agent for load balancing, the user can achieve parallelism.

What Can Go Wrong?

During a computation, two classes of error can occur: NetSolve failures and user mistakes. Let us demonstrate a few examples:

>> netsolve
NS:netsolveproxybasics.c:225: :  connection refused
 Cannot contact agent
...
>> [x] = netsolve('foo',a)
 unknown problem

x =

     []

...
>> [x y] = netsolve('eig',a,a)
'eig' requires 1 objects in input (2 provided)
 bad problem input/output

x =

     []


y =

     []
>>

In case of error, the different NetSolve functions print appropriate error messages. However, when the user writes Matlab scripts that call NetSolve, he/she needs ways to catch the errors while the script is running. Hence the functions described in the next section.

Catching NetSolve errors

There are two NetSolve functions that can be called from Matlab to catch errors. The first function, netsolve_err() takes no arguments and returns an integer that is the NetSolve error code returned by the last call to a NetSolve function (see the chapter called Error Handling in NetSolve for a list of the possible error codes). Here is a call:

>> e = netsolve_err
e = -11

The other function, netsolve_errmsg() takes an error code as an argument and returns a string that contains the corresponding error message. A typical call to netsolve_errmsg() is as follows:

>> [msg] = netsolve_errmsg(netsolve_err)

msg =

 bad problem input/output

Demo

A NetSolve-Matlab demo is available with the NetSolve distribution. It consists of a set of Matlab scripts that call NetSolve to compute parts of the Mandelbrot set. The main script is called mandel.m and is located in $NETSOLVE_ROOT/src/Demo/mandelbrot/. To run the demo, just type mandel at the Matlab prompt.

Optional: Testing the NetSolve BLAS interfaces

A NetSolve-Matlab BLAS test suite is available with the NetSolve distribution, and tests a subset of BLAS routines available in the NetSolve distribution. The user can test the reference implementation BLAS included in NetSolve or he could have enabled an optimized BLAS library during the configuration phase of NetSolve (./configure --with-blaslib=BLAS_LIB). The user must then enable the BLAS in the $NETSOLVE_ROOT/server_config file, and then he is ready to run this test suite. The test suite consists of a set of Matlab scripts that test each of the BLAS interfaces available in NetSolve. The main script is called blas_test.m and is located in $NETSOLVE_ROOT/src/Testing/matlab/. To run the BLAS test suite, type blas_test at the Matlab prompt.

Optional: Testing the NetSolve LAPACK interfaces

A NetSolve-Matlab LAPACK test suite is available with the NetSolve distribution. If the user enabled LAPACK and BLAS during the configuration phase of NetSolve as instructed in the Section called Enabling the LAPACK library in the chapter called Downloading, Installing, and Testing the Agent and Server, and has enabled LAPACK and BLAS in the $NETSOLVE_ROOT/server_config file, he/she may choose to run this test suite. Note that only a subset of LAPACK is included in the NetSolve distribution. The complete LAPACK library is not included as default numerical software for the server, and must be installed separately. The test suite consists of a set of Matlab scripts that test each of the LAPACK interfaces available in NetSolve. The main script is called lapack_test.m and is located in $NETSOLVE_ROOT/src/Testing/matlab/. To run the LAPACK test suite, type lapack_test at the Matlab prompt.

Optional: Testing the NetSolve ScaLAPACK interfaces

Likewise, a NetSolve-Matlab ScaLAPACK test suite is available with the NetSolve distribution. If the user enabled ScaLAPACK during the configuration phase of NetSolve as instructed in the Section called Enabling the ScaLAPACK library in the chapter called Downloading, Installing, and Testing the Agent and Server, and has enabled ScaLAPACK, MPIBLACS, BLAS, and MPI libraries, in the $NETSOLVE_ROOT/server_config file, then he may choose to run this test suite. The ScaLAPACK library is not included as default numerical software for the server, and must be installed separately (as well as MPI). The test suite consists of a set of Matlab scripts that test each of the ScaLAPACK interfaces available in NetSolve. The main script is called scalapack_test.m and is located in $NETSOLVE_ROOT/src/Testing/matlab/. To run the ScaLAPACK test suite, type scalapack_test at the Matlab prompt.

Optional: Testing the NetSolve 'sparse_iterative_solve' interface

The NetSolve 'sparse_iterative_solve' interface to PETSc, Aztec, and ITPACK can only be tested if the user has enabled sparse_iterative_solve in the $NETSOLVE_ROOT/server_config file and has configured NetSolve with the respective paths to the PETSc library, Aztec library, and MPI library. The PETSc, Aztec, and ITPACK libraries are not included as default numerical software for the server, and must be installed separately (as well as MPI). Refer to the Section called Enabling Sparse Iterative Solvers (PETSc, Aztec, and ITPACK) in the chapter called Downloading, Installing, and Testing the Agent and Server for further details.

This interface can be tested most effectively by using sparse matrices generated from collections such as the Harwell Boeing test collection on the Matrix Market homepage. Refer to the section on the webpage entitled Software, where the test matrices are available in C, Fortran, and Matlab. For ease of testing, several of the test matrices from this collection are included in the distribution of NetSolve.

After Matlab has been invoked, the user can then call the test scripts petsc_test.m, aztec_test.m, and itpack_test.m in the $NETSOLVE_ROOT/src/Testing/matlab/ directory, by typing

>> petsc_test

>> aztec_test

>> itpack_test

Alternatively, the user can generate a series of Harwell Boeing matrix types (1-5), using the generate.m script. To see a list of Harwell Boeing matrix types that can be generated, type

>> generate(0);

>> [A,rhs] = generate(1);
>> [x1,its1] = petsc(A,rhs);
>> [x2,its2] = aztec(A,rhs);

Note that the user can query for the list of arguments in the calling sequence to the routine by using the NetSolve tool routine.

>> netsolve('sparse_iterative_solve')

Optional: Testing the NetSolve 'sparse_direct_solve' interface

The NetSolve 'sparse_direct_solve' interface to MA28 and SuperLU can only be tested if the user has enabled sparse_direct_solve in the $NETSOLVE_ROOT/server_config file and has configured NetSolve with the respective paths to the SuperLU and MPI libraries. The MA28 library is distributed with NetSolve in $NETSOLVE_ROOT/src/SampleNumericalSoftware/MA28/ as a small modification to the library was necessary to enable its use in NetSolve. The SuperLU library is not included as default numerical software for the server, and must be installed separately (as well as MPI). Refer to the Section called Enabling Sparse Direct Solvers (SuperLU and MA28) in the chapter called Downloading, Installing, and Testing the Agent and Server for further details.

This interface can be tested most effectively by using sparse matrices generated from collections such as the Harwell Boeing test collection on the Matrix Market homepage. Refer to the section on the webpage entitled Software, where the test matrices are available in C, Fortran, and Matlab. For ease of testing, several of the test matrices from this collection are included in the distribution of NetSolve.

After Matlab has been invoked, the user can then call the test scripts ma28_test.m and superlu_test.m in the $NETSOLVE_ROOT/src/Testing/matlab/ directory, by typing

>> ma28_test

>> superlu_test

Alternatively, the user can generate a series of Harwell Boeing matrix types (1-5), using the generate.m script. To see a list of Harwell Boeing matrix types that can be generated, type

>> generate(0);

>> [A,rhs] = generate(1);
>> [x1] = ma28(A,rhs);
>> [x2] = superlu(A,rhs);

Note that the user can query for the list of arguments in the calling sequence to the routine by using the NetSolve tool routine.

>> netsolve('direct_solve_serial')

Introduction

The Mathematica client interface for NetSolve is built by typing

UNIX> make mathematica

Details of this interface can be found in [ns:mathematica] and quick instructions/requirements for building it are in the file: $NETSOLVE_ROOT/src/Mathematica/INSTALL

What to do first

Once the interface is successfully installed, the first thing to do is to start a Mathematica client by typing

    NetSolve[]

In[1]:= NetSolve[]
usage:
  NetSolve[FuncName[arg1, ...]]   - blocking problem call
  NetSolveNB[FuncName[arg1, ...]] - nonblocking problem call
  NetSolveProbe[request]          - checks if a request has been completed
  NetSolveWait[request]           - waits for a request to complete
  NetSolveGetAgent[]              - returns the current agent name
  NetSolveSetAgent[AgentName]     - changes the agent we are working with
  NetSolveError[]                 - returns the result code of the last
                                      executed NetSolve function
  NetSolveErrorMsg[rc]            - returns a string describing 
                                      the result code passed       
  NetSolve["?problems"]         - shows a list of available problems
  NetSolve["?servers"]          - shows a list of available servers
  NetSolve["?FuncName[]"]       - shows a problem description   

Let us review the possibilities:

Let us now assume that the user has started Mathematica and is ready to use NetSolve. We can check our agent by typing

    In[1]:= NetSolveGetAgent[]

    Out[1]= netsolve.cs.utk.edu

    In[2]:= NetSolveSetAgent["netsolve.cs.utk.edu"]

The agent can be changed at any time provided there is another NetSolve agent running on the host whose name has been passed as an argument. However, if the agent is changed, then the set of servers and possibly the set of solvable problems has also been changed.

A list of the solvable problems can be obtained by the function NetSolve["?problems"]. Here is a possible list (clipped to save space).

    In[3]:= NetSolve["?problems"]
    /BLAS/Matrices/dgemm
    /BLAS/Matrices/dmatmul
    /BLAS/Matrices/zgemm
    /BLAS/Matrices/zmatmul
    /BLAS/Vectors/daxpy
    /BLAS/Vectors/ddot
    /BLAS/Vectors/zaxpy
    /LAPACK/Matrices/EigenValues/eig
    /LAPACK/Matrices/LinearSystem/dgesv
    /LAPACK/Matrices/LinearSystem/linsol
    /MinPack/hybrd1
    /MinPack/lmdif1
    /QuickSort/DoublePrecision/dqsort
    /QuickSort/Integer/iqsort
    .  .  .

Similarly, a list of the servers can be printed by the function NetSolve["?servers"]

    In[4]:= NetSolve["?servers"]
    netsolve.cs.utk.edu (128.169.93.161)
    NetSolve Agent
    Host: Up  Server: Running
    cetus1a.cs.utk.edu (128.169.94.21)
    Handles 24 problems
    Host: Up  Server: Running
    cetus1b.cs.utk.edu (128.169.94.22)
    Handles 24 problems
    Host: Up  Server: Running
    cetus1c.cs.utk.edu (128.169.94.23)
    Handles 24 problems
    Host: Up  Server: Running
    .  .  .

For every server associated with a specific agent, the following information is given: its name, IP address, host and server status, and how many different problems it can solve.

The user can easily determine information about a specific problem, iqsort for instance, by typing

    NetSolve["?iqsort[]"]

The brackets after the problem name are required because every NetSolve problem is treated as a function defined in Mathematica.

The output of that command is as follows:

    In[5]:= NetSolve["?iqsort[]"]
    iqsort: Quicksort -
    Sorts a vector of integers

    Input:
      # 0 : Integer Vector
      Vector of integers to Sort
    
    Output:
      # 0 : Integer Vector
      Sorted Vector
    
    Mathematica example:
      rI0  = NetSolve[iqsort[I0]]

    examples for types:
    
             Char     Byte/Integer  Single/Double  Complex
    Scalar: "c"           42          66.32       4 - 7 I
    Vector: "vector"    {1,2,3}     {3,4.5,7}     {3, -5+3I, 8}
    Matrix: {"line 1", {{1,2,3},   {{6.4,2,1},    {{1+2I, 3+4I},
             "line 2"}  {4,5,6}}    {-7,1.2,4}}    {5-6I, 7}}

The first part of the output is a brief general description of the problem. The second part describes the input and output objects, their type and description. And lastly, an example is provided.

If the user does not provide the number, the type, and the sequence of arguments correctly, an error message message will be printed and the $Null symbol will be returned.

The arguments shown in the example are variables but the user may also choose to pass numerical values, symbols with assigned data or function calls.

Here are some rules the user must remember.

1. Characters are passed as strings (only the first character is used).

2. Integers can be passed instead of reals and vice versa (conversion is performed automatically).

3. Integers and reals can be passed instead of complex numbers.

4. Vectors of characters are passed as strings.

5. Matrices of characters are passed as vectors of strings.

Blocking call to NetSolve

In the previous section we explained how the user can obtain information about a problem and its calling sequence. For the call itself, the function NetSolve[] is invoked with the problem name and its arguments. For example,

    In[6]:= NetSolve[iqsort[{7,2,3,5,1}]]
    contacting server merlin.comlab ...
    
    Out[6]= {1, 2, 3, 5, 7} 

As stated earlier the user can pass not only numerical values, but also symbols that contain data of proper type or functions that return a result of this type. Indeed, Mathematica calculates these expressions and passes the arguments by value. For example

    In[7]:= v = -Range[5]

    Out[7]= {-1, -2, -3, -4, -5}

    In[8]:= NetSolve[iqsort[v]]
    contacting server merlin.comlab ...

    Out[8]= {-5, -4, -3, -2, -1}

    In[9]:= NetSolve[iqsort[Table[Ceiling[10*Random[]], {7}]]]
    contacting server merlin.comlab ...

    Out[9]= {1, 2, 2, 2, 4, 6, 7}  

Since NetSolve[] is a function defined in Mathematica, it can be used in expressions like:

    In[9]:= NetSolve[iqsort[Table[Ceiling[10*Random[]], {7}]]]
    contacting server merlin.comlab ...

    Out[9]= {1, 2, 2, 2, 4, 6, 7}

    In[10]:= Print["The minimal element of v is ", NetSolve[iqsort[v]][[1]]]
    contacting server merlin.comlab ...
    The minimal element of v is -5 

Let us consider a more complex problem such as the Level 3 BLAS subroutine dgemm[] which calculates where op(X) = X or op(X) = X'.

The routine dgemm[] requires the following 7 arguments.

Let us generate three random matrices.

    In[11]:= RandomMatrix[m_,n_] := Table[Ceiling[10*Random[]], {m}, {n}]

    In[12]:= a = RandomMatrix[2,3]

    Out[12]= {{9, 2, 3}, {6, 3, 9}}

    In[13]:= b = RandomMatrix[3,2]

    Out[13]= {{6, 4}, {4, 10}, {2, 9}}

    In[14]:= c = RandomMatrix[2,2]

    Out[14]= {{4, 7}, {4, 8}}

    In[15]:= NetSolve[dgemm["N", "N", 2, a, b, 3, c]]
    contacting server cetus2a.cs.utk.edu ...

    Out[15]= {{148., 187.}, {144., 294.}}
    
    In[16]:= 2 a . b + 3 c

    Out[16]= {{148, 187}, {144, 294}} 

Nonblocking Call to NetSolve

As in the Matlab interface (see the chapter called Matlab Interface), the Mathematica interface can be called in an asynchronous fashion. Nonblocking calls are performed by the function NetSolveNB[], and its calling sequence is the same as the blocking call NetSolve[]. The difference is in the result returned. NetSolveNB[] always returns a request handler.

NetSolveProbe[] returns an integer value to indicate if the problem has been completed. A value of 0 indicates that the result is available and a value of 1 indicates that the computation is still in progress. Other values are error codes (see the Section called Catching Errors).

Let us multiply two complex matrices using NetSolveNB[]. We generate the matrices ac and bc using already generated matrices a, b and c.

In[17]:= ac = a - 2 a I

Out[17]= {{9 - 18 I, 2 - 4 I, 3 - 6 I}, {6 - 12 I, 3 - 6 I, 9 - 18 I}}

In[18]:= bc = b - 3 b I

Out[18]= {{6 - 18 I, 4 - 12 I}, {4 - 12 I, 10 - 30 I}, {2 - 6 I, 9 - 27 I}}  

In[19]:= request = NetSolve[zmatmul[ac, bc]]
contacting server cetus2a.cs.utk.edu ...

Out[19]= 0

In[20]:= NetSolveProbe[request]

Out[20]= 0

As the computation is still in progress, the user can choose to perform other work, or wait for the request to complete:

In[21]:= NetSolveWait[request]

Out[21]= {{-340. - 340. I, -415. - 415. I}, {-330. - 330. I, -675. - 675. I}} 

Catching Errors

As in the Matlab interface, it is possible to detect errors with the functions NetSolveError[] and NetSolveErrorMsg[]. The first function returns an integer which is the error code of the last executed NetSolve function. NetSolveErrorMsg[] takes an error code as an input argument and returns a string describing the error.

With these two functions, it is possible to write Mathematica scripts that call NetSolve and handle all of the NetSolve errors at runtime.

Demo

A NetSolve-Mathematica demo is available with the NetSolve distribution. It invokes and explains the various NetSolve features available within Mathematica. The main script is called NSdemo.m and is located in $NETSOLVE_ROOT/src/Testing/mathematica/. To run the demo, just type <<NSdemo` at the Mathematica prompt.

Optional: Testing the NetSolve BLAS interfaces

A NetSolve-Mathematica BLAS test suite is available with the NetSolve distribution, and tests a subset of BLAS routines available in the NetSolve distribution. The user can test the reference implementation BLAS included in NetSolve, or he can enable an optimized BLAS library during the configuration phase of NetSolve (./configure --with-blaslib=BLAS_LIB). The user must then enable the BLAS in the $NETSOLVE_ROOT/server_config file, and then he is ready to run this test suite. The test suite consists of a set of Mathematica scripts that test each of the BLAS interfaces available in NetSolve. The main script is called NSblastest.m and is located in $NETSOLVE_ROOT/src/Testing/mathematica/. To run the BLAS test suite, type <<NSblastest` at the Mathematica prompt.

Optional: Testing the NetSolve LAPACK interfaces

A NetSolve-Mathematica LAPACK test suite is available with the NetSolve distribution. If the user enabled LAPACK during the configuration phase of NetSolve as instructed in the Section called Enabling the LAPACK library in the chapter called Downloading, Installing, and Testing the Agent and Server, and has enabled LAPACK and BLAS in the $NETSOLVE_ROOT/server_config file, he/she may choose to run this test suite. Note that only a subset of LAPACK is included in the NetSolve distribution. The complete LAPACK library is not included as default numerical software for the server, and must be installed separately. The test suite consists of a set of Mathematica scripts that test each of the LAPACK interfaces available in NetSolve. The main script is called NSlapacktest.m and is located in $NETSOLVE_ROOT/src/Testing/mathematica/. To run the LAPACK test suite, type <<NSlapacktest` at the Mathematica prompt.

Calling Farming in C

Like netsl() and netslnb(), the netsl_farm() function takes a variable number of arguments. Its first argument is a string that describes the iteration range. This string is of the form "i=%d,%d" (in C string format symbols). The second argument is a problem name appended with an opening and a closing parenthesis. The arguments following are similar in intent to the ones supplied to netsl(), but are iterators as opposed to integers or pointers. Where the user was passing, say an integer, to netsl(), he now needs to pass an array of integers and tell netsl_farm() which element of this array is to be used for which iteration. This information is encapsulated in an iterator and we provide three functions to generate iterators:

An example

Let us assume that the user wants to sort an array of integers with NetSolve using the C interface. The default NetSolve server comes with a default problem called iqsort that does a quicksort on an integer vector. The call looks like

status = netsl('iqsort()',size,ptr,sorted);

requests1 = netslnb('iqsort()',size1,ptr1,sorted1); 
requests2 = netslnb('iqsort()',size2,ptr2,sorted2); 
...
requests200 = netslnb('iqsort()',size200,array200,sorted200); 

With farming, one only needs to construct three arrays as:

int size_array[200];
void *ptr_array[200];
void *sorted_array[200];

size_array[0] = size1;
ptr_array[0] = ptr1;
sorted_array[0] = sorted1;
...

Then, netsl_farm() can be called as:

status_array = netsl_farm("i=0,199","iqsort()",
                                    ns_int_array(size_array,"$i"),
                                    ns_ptr_array(ptr_array,"$i"),
                                    ns_ptr_array(sorted_array,"$i"));

In short, netsl_farm() is a concise, convenient way of farming out groups of requests. Of course, it uses netslnb() underneath, thereby ensuring fault-tolerance and load-balancing.

Catching errors

netsl_farm() returns an integer array. That array is dynamically allocated and must be freed by the user after the call. The array is at least of size 1. The first element of the array is either 0 or -1. If it is 0, then the call was completed successfully and the array is of size 1. If first element of the array is -1, then at least one of the requests failed. The array is then of size one plus the number of requests and the (1+i)-th element of the array is the error code for the i-th request. Here is an example on how to print error messages:

status = netsl_farm("i=0,200",....);
if (status[0] == 0){
   fprintf(stderr,"Success\n");
   free(status);
} else {
  for (i=1;i<201;i++) {
    fprintf(stderr,"Request #%d:",i);
    netslerr(status[i]);
  }  
}
free(status);

Calling Farming in Matlab

NetSolve provides the Matlab user with a convenient interface whereby he can make multiple requests to the same NetSolve program possibly with different arguments. This facility is useful in task farming problems like Monte Carlo simulation where multiple simulation runs are executed across different machines and the results from the simulation runs are combined to form the final output.

The Matlab task farming uses "cell" function calls in Matlab. The NetSolve user needs to have Matlab version 6.x or higher to use this functionality.

The user calls the matlab task farm as

netsolve_farm(iterator string, problem name,
                  argument cell 0, argument cell 1, ...)

The iterator string represents the number of requests to be made to the netsolve problem whose name is represented by problem name argument. The value of the iterator string assumes the form "i=0,number of requests -1". The netsolve arguments needed by the netsolve problem are passed as cells. Each cell is a one-dimensional array and the size of the cell is equal to the number of requests in the farming. Each element in the cell corresponds to a single argument need by the NetSolve problem. The i-th element corresponds to the argument for the i-th request in the farm.

An example

In this example, the Matlab task farming interface is invoked to make multiple requests to a NetSolve problem, totalarea. The inputs to the problem are the name of a function which is a string, the starting point on the x-axis which is an integer and the ending point on the x-axis which is another integer value. The problem computes the area of the function under the curve bounded within a region and gives back the total area under the curve. Thus, using the normal NetSolve call, the user would invoke this problem as

area = netsolve('totalarea', xstart, xend);

One way to solve this problem is to divide the function into many sub areas and make multiple requests to the 'totalarea' problem passing different start and end points on the x-axis. The NetSolve program that uses the task farming interface for this method is given below.

function [total_area] =  totalarea ( func, a, b, num_strips )

strip_length = (b-a)/num_strips

x1 = cell(num_strips, 1);
x2 = cell(num_strips, 1);

x1{1} = a;
for i=1:num_strips
  funcs{i} = func;
  x2{i} = x1{i}+strip_length;
  if i < num_strips
    x1{i+1} = x2{i};
  end
end

str1 = 'i=0,';
str2 = int2str(num_strips-1);

str = [str1,str2];

area = netsolve_farm(str, 'area', funcs, x1, x2);


total_area = 0.0;
for i=1:num_strips
  total_area = total_area+area{i};
end
end

fprintf(2, 'total_area: %f\n', total_area);

Current Implementation

One of the advantages of farming is that the user does not have the responsibility of managing the requests. As it would be unreasonable to send all of the requests if there are not enough servers to perform the computations, the netsl_farm() farming algorithm avoids this problem by dynamically tuning the maximum number of pending requests to reflect changes in the computational server pool (size and load). This is done by constantly measuring the throughput of the computations.

Goals and Methodologies

Our aim in request sequencing is to decrease network traffic amongst NetSolve client and server components in order to decrease overall request response time. Our design ensures that i) no unnecessary data is transmitted and ii) all necessary data is transferred. As briefly discussed below, we also reduce execution time by executing computational modules simultaneously when possible. All this is accomplished by performing a detailed analysis of the input and output parameters of every request in the sequence to produce a directed acyclic graph (DAG) that represents the tasks and their execution dependences. This DAG is then sent to a server in the system where it is scheduled for execution.

In order to build the DAG or task graph, we need to analyze every input and output in the sequence of requests. We evaluate two parameters as the same if they share the same reference. We use the size fields and reference pointer of the input parameters to calculate when inputs overlap in the memory space. Only matrices and vectors are checked for recurrences on the premise that these are the only objects that tend to be large enough for the overhead of the analysis to pay dividends. Through this analysis we build a DAG in which the nodes represent computational modules or NetSolve services and the arcs represent data dependencies amongst these modules. The graph is acyclic because looping control structures are not allowed within the sequence, and therefore, a node can never be its own descendant.

The Application Programming Interface

For request sequencing, we add three functions to the NetSolve client API:

Figure 1 illustrates what a sequencing call might look like. Two points to note in this example: i)for all requests, only the last parameter is an output, and ii)the user is instructing the system not to return the intermediate results of command1 and command2.

      ...
      begin_sequence();
      submit_request("command1", A, B, C);
      submit_request("command2", A, C, D);
      submit_request("command3", D, E, F);
      begin_end(C, D, NS_NULL);
      ...

The other restriction is that statements that would change the value of any input parameter of any component of the sequence are forbidden within the sequence (with the exception of calls to the NetSolve API itself that the system can track.) This is because during the data analysis, only references to the data are stored. So if changed, the data transferred at the end of the sequence will not be the same as the data that was present when the request was originally made. We contemplated saving the entire data, rather than just the references, but this directly conflicts with one of our premises -- that the data sets are large; multiple copies of these data are not desirable.

Execution Scheduling at the Server

Once the entire DAG is constructed, it is transferred to a NetSolve computational server. In this first version of request sequencing, the NetSolve agent uses a large granularity and decides which server should execute the entire sequence. We execute a node if all its inputs are available and there are no conflicts with its output parameters. Currently the only mode of execution we support is on a single NetSolve server -- though, that server may be a symmetric multi-processor (SMP).

For data partitioning, we transfer the union of the input parameter sets to the selected server host. This makes input for all nodes, except those which are intermediate output from prior nodes, available for the execution of the sequence. Our scheduling algorithm can be summarized as follows:

  while(problems left to execute)
  {
    execute all problems with satisfied dependencies;
    wait for at least one problem to finish;
    update dependencies;
  }

netslpr

Checking for completion

To check whether a previously submitted asynchronous request has completed you should use the netslpr() call. It does not block, but returns a value signifying whether the call has completed or not. Below is a short code example.

Submit the asynchronous request:

status = netslnb("inttest()", &i);

status = netslpr(request);

• NetSolveNotReady - the request has not completed

• NetSolveOK - the request has completed

netslwt

Waiting for completion

To wait for a previously submitted asynchronous request to complete you should use the netslwt() call. It blocks until the request has completed. Below is a short code example.

Submit the asynchronous request:

status = netslnb("inttest()", &i);

status = netslwt(request);

• NetSolveOK - the request has completed successfully

• Otherwise the return status represents the specific error that occurred

netslkill

Cancelling a request

To cancel a previously submitted asynchronous request you should use the netslkill() call. It notifies the server that the request has been cancelled by the user. Below is a short code example.

Submit the asynchronous request

status = netslnb("inttest()", &i);

status = netslkill(request);

• NetSolveOK - the request has been successfully cancelled

• NetSolveInvalidRequestID - the request ID didn't correspond to a request that you previously submitted

• otherwise the return status represents the specific error that occurred

netsl_assignment

Assigned server requests

If you want to submit a request to a specific server you should use the netsl_assignment function (or netslnb_assignment for asynchronous requests). The server name is specified before the problem name in the call. for example:

status = netslnb_assignment("160.36.56.200:inttest()",&i);

Introduction

NetSolve components include clients, agents, and servers. Currently the only requests that require authentication are requests that the client makes to the server, and of those, only the ``run problem'' request. Other requests could be authenticated (an obvious one being ``kill server''), but drastic changes along these lines would probably require drastic restructuring of NetSolve. For instance, a client can currently inform an agent that a particular server is down, and the agent will not advertise that server for use in other problems. It seems of dubious value to require authentication for such requests until there is a mechanism for specifying the trust relationship between clients and agents.

An attempt has been made to allow Kerberized NetSolve clients to interoperate with both Kerberized and non-Kerberized NetSolve servers. In either case the client sends a request to the server. An ordinary server will return a status code indicating that he will accept the requested operation. By contrast, a Kerberized server will immediately return an ``authentication required'' error in response to the request. The client is then required to send Kerberos credentials to the server before the request will be processed. This allows the server to require authentication of the client. Currently there is no mechanism to allow the client to insist on authentication of the server. A Kerberized client will happily talk with either Kerberized or non-Kerberized servers.

The server implements access control via a simple list of Kerberos principal names. This list is kept in a text file which is consulted by the server. A request to a NetSolve server must be made on behalf of one of those principal names. If the principal name associated with the Kerberos credentials in the request appears in the list, and the credentials are otherwise valid, the request will be honored. Otherwise, the request will be denied.

Since the NetSolve server was not designed to run as a set-uid program, it is not currently feasible to have the NetSolve server run processes using the user-id of the particular UNIX user who submitted the request. NetSolve thus uses its own service principal name of ``netsolve'' rather than using the ``host'' principal. What this means (among other things) is that you need to generate service principals and keytabs for each of your NetSolve servers, even if you already have host principals in place.

The NetSolve server, by default, runs in non-Kerberized mode. To start up the server in Kerberized mode you need to add the -k option to the command-line, and also set environment variables NETSOLVE_KEYTAB (pointing to the keytab) and NETSOLVE_USERS pointing to the list of authorized users).

This version of Kerberized NetSolve performs no encryption of the data exchanged among NetSolve clients, servers, or agents. Nor is there any integrity protection for the data stream.

Compiling a Kerberized Server

1. Compile Kerberos. See the Kerberos V5 Installation Guide for instructions for how to do this.

2. Compile the NetSolve client libraries with Kerberos support. Refer to the instructions in the the Section called Installation on Unix Systems using ns_install in the chapter called Downloading, Installing, and Testing the Agent and Server section following the notes that talk about authentication and authentication libraries. In part, this involves editing the $NETSOLVE_ROOT/conf/Makefile.NETSOLVE_ARCH.inc and modifying the KLIBS field to point to the appropriate Kerberos libraries and setting the AUTHENTICATION field to KERBEROS5.

Running a Kerberized NetSolve Client

1. Set up the necessary environment variables:

   UNIX> setenv NETSOLVE_AGENT netsolve.agent.host

2. Run kinit to get a ticket-granting ticket for yourself. You don't have to do this if you already have a ticket and it has not expired.

3. Run your NetSolve program. If the server contacted requires authentication, the NetSolve client automatically contacts the Kerberos Key Distribution Center for a ticket and sends it to the server. If this client is authorized to utilize the NetSolve server services will be granted to the client, if not, an AUTHENTICATION_REJECTED error will be returned to the client.

Details of the Makefile.NETSOLVE_ARCH.inc File

Although suitable default options are provided for the compilation of the software, one may look in the NetSolve/conf directory to edit the Makefile.NETSOLVE_ARCH.inc file. This file contains parameters to customize the compilation process.

	Note:
	All of the parameters in this include file can (and should) be modified using command line arguments to configure.

Most of the contents of this file are straightforward, including definitions for compilers, linkers, etc., and will not be explained here. There are however a few entries that may need explanation.

An example Makefile.NETSOLVE_ARCH.inc for IRIX is listed below.

# Generated automatically from Makefile.generic-arch.in by configure.
# Never include this file directly!
#   Always include ./Makefile.inc and make sure it is appropriately
#   set to include the proper platform specific file.
# CUSTOMIZING CONFIGURATION
#

SHELL = /bin/sh

#############################
#### INSTALL DIRECTORIES ####
#############################

PLATFORM           = mips-sgi-irix6.5
NETSOLVE_VERSION   = 2.0
EXEC_PREFIX        = $(NETSOLVE_ROOT)/$(NETSOLVE_ARCH)
BINDIR             = $(NETSOLVE_ROOT)/bin/$(NETSOLVE_ARCH)
LIBDIR             = $(NETSOLVE_ROOT)/lib/$(NETSOLVE_ARCH)
OBJDIR             = $(NETSOLVE_ROOT)/obj/$(NETSOLVE_ARCH)
MATLABOBJDIR       = $(OBJDIR)/MATLAB
PDFGUICLASSDIR     = $(BINDIR)/PDFGUICLASSDIR

###############################
#### COMPILERS AND OPTIONS ####
###############################
CC               = /usr/bin/cc
C_OPT_FLAGS      = -O3
C_NOOPT_FLAGS    = -n32 -mips4 -r12000 -common
CFLAGS           = $(C_OPT_FLAGS) $(C_NOOPT_FLAGS)
NS_C_OPT_FLAGS   = $(C_OPT_FLAGS) $(HBMFLAG) $(F2CFLAG) $(OUTPUT_LEVEL) $(ARCHCFLAGS) \
                   $(INCDIR) $(PROXY) ${CPU_STAT} ${IBPFLAG} \
                   ${AUTHENTICATION} $(DSIFLAGS) 
NS_C_NOOPT_FLAGS = $(C_NOOPT_FLAGS) $(HBMFLAG) $(F2CFLAG) $(OUTPUT_LEVEL) $(ARCHCFLAGS) \
                   $(INCDIR) $(PROXY) ${CPU_STAT} ${IBPFLAG} \
                   ${AUTHENTICATION} $(DSIFLAGS) 
NS_CFLAGS        = $(CFLAGS) $(HBMFLAG) $(F2CFLAG) $(OUTPUT_LEVEL) $(ARCHCFLAGS) \
                   $(INCDIR) $(PROXY) ${CPU_STAT} ${IBPFLAG} \
                   ${AUTHENTICATION} $(DSIFLAGS)

FC               = /usr/bin/f77
F_OPT_FLAGS      = -O3
F_NOOPT_FLAGS    = -n32 -mips4 -r12000
FFLAGS           = $(F_OPT_FLAGS) $(F_NOOPT_FLAGS)
NS_FFLAGS        = $(FFLAGS) $(INCDIR) $(ARCHCFLAGS)
NS_F_OPT_FLAGS   = $(F_OPT_FLAGS) $(INCDIR) $(ARCHCFLAGS)
NS_F_NOOPT_FLAGS = $(F_NOOPT_FLAGS) $(INCDIR) $(ARCHCFLAGS)

LINKER       = $(FC)
LDFLAGS      = -LD_MSG:OFF=15,84 -n32 -mips4 -r12000

MEX          = /usr/local/matlab/bin/mex
MEXFLAGS     = -O
MEXEXT       = .mexsg
NS_MEXFLAGS  = $(MEXFLAGS) $(HBMFLAG) $(F2CFLAG) $(OUTPUT_LEVEL) $(ARCHMFLAGS) \
               $(INCDIR) $(PROXY) ${CPU_STAT} ${IBPFLAG} \
               ${AUTHENTICATION} $(DSIFLAGS) -g -DMATLAB

JAVAC        = 
NS_JAVAFLAGS = -classpath $(NETSOLVE_ROOT)/src/PDF_GUI/classes:$(PDFGUICLASSDIR) \
               -d $(PDFGUICLASSDIR)

##############################
### LIBS, DIRS AND DEFINES ###
##############################

LIBS         = -lm -lc
INCDIR       =  -I$(NETSOLVE_ROOT)/include \
                $(NWS_INCDIR) \
                $(IBP_INCDIR) \
                $(MPI_INCDIR)

ARCHCFLAGS      = -D$(NETSOLVE_OS) \
                  -D$(F2CSTR) -D$(F2CINT) -D$(F2CNAMES) -D$(RUSAGE) \
                  -DNETSOLVE_ROOT=\"$(NETSOLVE_ROOT)\" \
                  -DNETSOLVE_ARCH=\"$(NETSOLVE_ARCH)\" \
		  -DMPI_DIR=\"$(MPI_DIR)\"

ARCHMFLAGS      = -D$(NETSOLVE_OS) \
                  -D$(F2CSTR) -D$(F2CINT) -D$(F2CNAMES) -D$(RUSAGE) \
                  -D'NETSOLVE_ROOT=\"$(NETSOLVE_ROOT)\"' \
                  -D'NETSOLVE_ARCH=\"$(NETSOLVE_ARCH)\"'

#### $F2CINT options
#### FINT2CLONG  : F77 INTEGER -> C long
#### FINT2CINT   : F77 INTEGER -> C int    (default)
#### FINT2CSHORT : F77 INTEGER -> C short
F2CINT = FINT2CINT

#### $F2CNAMES options
#### F2CADD_     : F77 netsl( ) -> C netsl_( )  (default)
#### F2CADD__    : F77 netsl( ) -> C netsl__( )
#### F2CNOCHANGE : F77 netsl( ) -> C netsl( )
#### F2CUPCASE   : F77 netsl( ) -> C NETSL( )
F2CNAMES = F2CADD_

#### $F2CSTR options
#### F2CSTRSUNSTYLE    : Sun style of passing strings from f2c
#### F2CSTRCRAYSTYLE   : Cray style of passing strings from f2c
#### F2CSTRSTRUCTPTR   : Struct * style of passing strings from f2c
#### F2CSTRSTRUCTVAL   : Struct style of passing strings from f2c
F2CSTR = F2CSTRSUNSTYLE

##########################
### AUXILIARY PROGRAMS ###
##########################
FLEX         = /usr/bin/flex
BISON        = /usr/bin/bison
AR           = /usr/bin/ar
ARFLAGS      = cr
RANLIB       = :
RUSAGE       = HAVERUSAGE


###################################
#### NETSOLVE SPECIFIC OPTIONS ####	
###################################

#================#
# F2C
#================#
F2CFLAG = -DNOCHANGE

#================#
# Program Output #
#================#
#### 	DEBUG      : For really verbose debugging information
#### 	VIEW       : For smooth information during the execution
#### 	NO_OUTPUT  : no output
OUTPUT_LEVEL = -DVIEW

#==============#
# Client Proxy #
#==============#
####    Proxies are currently mutually exclusive
####    GLOBUS_PROXY    : build and enable globus proxy
####    NETSOLVE_PROXY  : build and enable netsolve proxy
PROXY = -DNETSOLVE_PROXY

#====================#
# Information Server #
#====================#

# options for INFOSERVERFLAGS
# INFOSERVERFLAGS =                                (blank means do not use)
# INFOSERVERFLAGS = -DINFOSERVER                   (use as part of agent)
# INFOSERVERFLAGS = -DINFOSERVER -DSTANDALONEISERV (use in standalone mode)
INFOSERVERFLAGS =  
INFOSERVER = 

#=================#
# Workload Prober #
#=================#
## Which probes? options are NWS, NS_WORKLOAD (NetSolve)
CPU_STAT = -DNS_WORKLOAD

#=====#
# DSI #
#=====#
DSIFLAGS =  

########################
## AUXILIARY PACKAGES ##
########################
#================#
# AUTHENTICATION #
#================#

## options are NO_AUTH, KERBEROS5
AUTHENTICATION = -DNO_AUTH
AUTH_LIBS = 

#=====#
# NWS #
#=====#
NWSDIR = 
NWS_INCDIR = 
NWSLIBS  = 
NWSEXECSSTUB = 

#=====#
# MPI #
#=====#
MPI_DIR = /usr/local/mpich
MPI_INCLUDE_DIR = $(MPI_DIR)/include
MPI_INCDIR = -I$(MPI_INCLUDE_DIR)

#=====#
# IBP #
#=====#
IBPDIR = 
IBPARCH = 
IBP_INCDIR = 
IBPLIB = 
IBPOBJS_STUB = 
IBPOBJS = 
IBPFLAG = 

#========#
# Globus #
#========#

#GLOBUS_DIR =
#include $(GLOBUS_DIR)/etc/makefile_header
#G_LIBS = -L$(GLOBUS_DIR)/lib $(GLOBUS_GRAM_CLIENT_LIBS) $(LIBS)
#G_CFLAGS = $(GLOBUS_GRAM_CLIENT_CFLAGS) -I$(GLOBUS_DIR)/include
#G_LDFLAGS = $(GLOBUS_GRAM_CLIENT_LDFLAGS)
#LDAP_DIR = /usr/local/ldap
#LDAP_LIBS = -L$(LDAP_DIR)/lib
#LDAP_CFLAGS = -I$(LDAP_DIR)/include
#LDAP_LDFLAGS =  -lldap -llber

#================#
# Auxiliary Libs #
#================#

HAVE_petsc = 0
PETSC_DIR  = /src/icl2/petsc/petsc-2.0.29/
PETSC_ARCH = linux
BOPT       = O
PETSC_LIB_DIR = $(PETSC_DIR)/lib/lib$(BOPT)/$(PETSC_ARCH)

HAVE_aztec = 0
AZTEC_DIR     = /src/icl2/Aztec/
AZTEC_LIB_DIR = /src/icl2/Aztec/lib/libg/linux

HAVE_superlu    = 0
SUPERLU_DIR     = /src/icl2/SuperLU/
SUPERLU_LIB_DIR = /src/icl2/SuperLU/lib/sequential/linux
USE_SUPERLU_SERIAL = -DUSE_SERIAL
USE_SUPERLU_DIST =

LAPACK_LIB_LINK = /usr/local/lib/liblapack-n32.a

SCALAPACK_LIB_LINK = /usr/local/lib/libscalapack.a

BLAS_LIB_LINK = /usr/lib32/mips4/libblas.a

BLACS_LIB_LINK = /usr/local/lib/libmpiblacsCinit-p4.a /usr/local/lib/libmpiblacs-p4.a /usr/local/lib/libmpiblacsCinit-p4.a

Installation on Unix Systems using ns_install

The NetSolve distribution tar file is available from the NetSolve web site.http://icl.cs.utk.edu/netsolve/software/ Once the file, NetSolve-2.0.tgz, has been downloaded, the following UNIX commands will create the NetSolve-2.0 directory:

gunzip -c NetSolve-2.0.tgz | tar xvf -

Installation on Unix Systems using ns_install

1.

Run ns_install from the new NetSolve-2.0 directory where the NetSolve package was uncompressed.

UNIX> ./ns_install

2.

The script will first prompt for the components to be installed.

    1. Standard (Client, Server, Agent, Testers, Tools)
    2. Client
    3. Server
    4. Agent
    5. Tools
    6. Testers
    7. GridRPC API
    8. Matlab interface
    9. Octave interface

3.

If your selection above included the Server component you will be asked whether you want to enable GPG for signing software when using the Hardware/Software server feature. It is recommended that you enter y here if you wish to use the Hardware/Software server.

4.

GPG options. If you selected to use GPG then you will be prompted to use version 1.2.3. If this version is present in your PATH you can choose to use it. Otherwise you may select to download GPG yourself ( see http://www.gnupg.org/ ), or let the script handle this download for you, or use a tarball already present. You may also skip GPG at this time.

5.

Finally you will be prompted for any additional arguements to be passed to configure. Options are listed below under 2.1 Command line arguements. For example, if you wanted to limit the number of ports used by NetSolve to 9001 thru 9204, simply enter;

--enable-port-restriction

6.

The script will now configure and build the components selected. If successful you should see, "NetSolve installation complete".

7.

Starting the Agent

bin/$NETSOLVE_ARCH/NS_agent

8.

Starting the Server. You should check the file $NETSOLVE_ROOT/server_config to be sure the AGENT parameter is set for the Agent that you intend to use, then run:

bin/$NETSOLVE_ARCH/NS_server

9.

To test your installation you can run:

bin/$NETSOLVE_ARCH/Test

Installation on Unix Systems using configure

From this point forward, we assume that the UNIX SHELL is from the csh family.

The installation of NetSolve is configured for a given architecture using the GNU tool configure.

UNIX> cd NetSolve
UNIX> ./configure

UNIX> ./configure --help

Optional Features:

  --disable-FEATURE            do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]       include FEATURE [ARG=yes]
  --disable-debug              exclude debugging info when compiling
                                - (no effect when --with-cnoptflags and --with-coptflags are used
  --enable-port-restriction    server uses only ports 9001 thru 9204
  --enable-infoserver=alone    use InfoServer alone

  --with-cc                    determine which C compiler to use
  --with-cnooptflags           set compiler flags that don't deal with optimization
                                - (ONLY USE IN COMBINATION WITH --with-cc)
                                - CFLAGS will be set to C_OPT_FLAGS+C_NOOPT_FLAGS
  --with-coptflags             set compiler optimization flags
                                - (ONLY USE IN COMBINATION WITH --with-cc)
                                - CFLAGS will be set to C_OPT_FLAGS+C_NOOPT_FLAGS
  --with-fc                    determine which Fortran compiler to use
  --with-fnooptflags           set compiler flags that don't deal with optimization
                                - (ONLY USE IN COMBINATION WITH --with-fc)
                                - FFLAGS will be set to F_OPT_FLAGS+F_NOOPT_FLAGS
  --with-foptflags             set compiler optimization flags
                                - (ONLY USE IN COMBINATION WITH --with-fc)
                                - FFLAGS will be set to F_OPT_FLAGS+F_NOOPT_FLAGS
  --with-ldflags               set loader flags

  --with-nws=NWSDIR            location of NWS installation dir
  --with-ibp=IBPDIR            location of IBP installation dir
  --with-kerberos=KRBDIR       use Kerberos5 client authentication
  --with-proxy                 which Proxy? (netsolve, globus)
  --with-outputlevel           output level (debug,view,none)

  --with-petsc=PETSCDIR                 location of PETSc installation dir
  --with-petsclibdir=PETSC_LIB_DIR      location of PETSc library
  --with-aztec=AZTEC_DIR                location of Aztec installation dir
  --with-azteclib=AZTEC_LIB             Aztec link line
  --with-superlu=SUPERLU_DIR            location of SuperLU installation dir
  --with-superlulib=SUPERLU_LIB         SuperLU link line
  --with-ma28                           if ma28 is to be included in the NetSolve services
  --with-itpack                         if itpack is to be included in the NetSolve services
  --with-arpacklib=ARPACK_LIB           Arpack link line
  --with-mpi=MPI_DIR                    location of MPI Root Directory
  --with-lapacklib=LAPACK_LIB           LAPACK link line
  --with-scalapacklib=SCALAPACK_LIB     SCALAPACK link line
  --with-blaslib=BLAS_LIB               BLAS link line
  --with-blacslib=BLACS_LIB             BLACS link line
  --with-mldk=MLDK_PATH                 Path to MathLink Development Kit
  --with-rpclib=RPC_LIB                 Full path of RPC library
  --with-octave-include=OCTAVE_INC_DIR  location of Octave include directory
  --with-rpcinc=RPC_INC                 Directory containing RPC header files
  --with-gpg=GPGPATH                    Full path of gpg binary
  --with-buildgpg=BUILDGPG              Location of gpg tar.gz file

	Note
	It is recommended that you set the environment variables NETSOLVE_ROOT and NETSOLVE_ARCH prior to running configure. If you wish to compile the Matlab interface to NetSolve, you must select a compiler flag setting for NetSolve that matches that used by Matlab. If you have the IRIX64 version of Matlab, you will need to overwrite NetSolve's flags to select the -64 during the configure step.

The configure script creates two main files, ./conf/Makefile.$NETSOLVE_ARCH.inc and ./conf/Makefile.inc. These files are created from the templates ./conf/Makefile.generic-arch and ./conf/Makefile.inc.in respectively. $NETSOLVE_ARCH is the string printed by the command ./conf/config.guess, with all '-' and '.' characters converted to '_' characters. The variable $NETSOLVE_ROOT is the complete path name to the installed NetSolve directory and defined in ./conf/Makefile.inc. These *.inc files are included by the Makefiles that build the NetSolve system. Manually editing these configuration files is strongly discouraged. However, details of the $NETSOLVE_ROOT/conf/Makefile.$NETSOLVE_ARCH.inc file are explained in the Section called Details of the Makefile.NETSOLVE_ARCH.inc File in the chapter called Troubleshooting.

Typing make in the NetSolve directory will give instructions to complete the compilation. A typical agent and server compilation includes:

UNIX> make standard

Testing the Software

Testing the software consists of starting an agent and a server and running a client test (the Section called Agent-Server-Client Test). Alternatively, the default agent and servers running at the University of Tennessee can be used to test the client only (see the Section called Testing the Unix installation in the chapter called Downloading, Installing, and Testing the Client). We describe here the step-by-step procedure that involves manipulations that will be detailed and explained in the following chapters.

UNIX> setenv NETSOLVE_AGENT my.machine.net
uNIX> $NETSOLVE_ROOT/bin/$NETSOLVE_ARCH/NS_agent
UNIX> $NETSOLVE_ROOT/bin/$NETSOLVE_ARCH/NS_server
UNIX> cd $NETSOLVE_ROOT/bin/$NETSOLVE_ARCH
UNIX>Test

Expanding the Server Capabilities

It is possible to add new functionalities to a NetSolve computational server by specifying additional problem description files in the server configuration file. In fact, a number of PDFs have been written for a variety of serial and parallel software packages: ARPACK, Aztec, BLAS, ITPACK, LAPACK, MA28, PETSc, ScaLAPACK, and SuperLU. These PDFs are available in the $NETSOLVE_ROOT/problems/ directory. If a user has one of these software libraries compiled on the architecture to which he is installing NetSolve, he can easily add this functionality to his server in three steps.

• During the configure phase of NetSolve, specify the configure option(s) for enabling the respective library. Refer to the Section called Installation on Unix Systems using ns_install for details. This step will set the required variables in the $NETSOLVE_ROOT/conf/Makefile.$NETSOLVE_ARCH.inc file.

• Uncomment the respective line in the section @PROBLEMS: section of the $NETSOLVE_ROOT/server_config file.

• Recompile the server by typing make server in the $NETSOLVE_ROOT/ directory. Or use the dynamic server feature described in the Section called Dynamic Servers in the chapter called Running the NetSolve Server, by sending kill -USR1 to the highest server PID.

	Note
	If you are enabling sparse_iterative_sovle or sparse_direct_solve, you will need to perform make wrappers first, followed by make server.

NetSolve's distributed memory services (e.g., ScaLAPACK, PETSc) are spawned using MPI (mpirun -machinefile MPImachines ...) and thus require an MPI machine file describing the parallel machine on which to run. The name of the file containing this list of homogeneous machines is called $NETSOLVE_ROOT/MPImachines and is referenced in the file $NETSOLVE_ROOT/server_config for configuring the server. Therefore, if you are enabling parallel services within a server, the user MUST edit this $NETSOLVE_ROOT/MPImachines file to list the specific machines to be used. The current implementation of NetSolve allows only one MPImachines file per server. This spawning file is tied to the server, and not to a specific service enabled. Therefore, if you wish to enable parallel services on different clusters, then you must enable the software on different servers -- i.e., maintain a separate NetSolve source code tree for each server enablement so that each parallel service can have its own MPImachines file from which to spawn. A future release of NetSolve should identify a separate MPImachines file with each parallel service that can be enabled.

Starting a Server

After compiling the server as explained in the Section called Installation on Unix Systems using ns_install in the chapter called Downloading, Installing, and Testing the Agent and Server, the executable of the NetSolve server is located in:

$NETSOLVE_ROOT/bin/$NETSOLVE_ARCH/

NS_server [-f config_file] [-l logfile] [-k] [-a IP_address] [-s]

UNIX> NS_server

UNIX> NS_server -f /path/to/my_config

The -l option specifies the name of a file to use for logging purposes.

UNIX> NS_server -l /path/to/agent_logfile

The -k starts the server in Kerberized mode.

UNIX> NS_server -k

UNIX> NS_server -s

	Note
	Also set the environment variables NETSOLVE_KEYTAB (pointing to the keytab) and NETSOLVE_USERS (pointing to the list of authorized users). The -k option is only useful when installed with KERBEROS libraries

The -a options assigns a specific IP address to the server. This allows the selection of which IP to use on servers with multiple network cards.

UNIX> NS_server -a <ip address>

Note

Multiple NetSolve servers can be running on a given machine if and only if they have a different NetSolve agent. When running multiple servers within the same directory tree, if a unique log file isn't specified, then the most recently started server will take over the log file. Log messages from other servers will be lost. Use the -l parameter to specify a unique log for each server to avoid this.

When the server has been compiled with the Kerberos libraries, the administrator has the option of having the server require clients to authenticate before rendering services. To mandate this authentication, the -k option must be used, otherwise no authentication will be asked for, and the server will be available to service requests to ANY client asking for services.

To terminate an existing server (or query an existing NetSolve system), the user should refer to the NetSolve management tools as outlined in the chapter called NetSolve Management Tools for Administrators.

The Server Configuration File

The server configuration file is used to customize the server. The default configuration file in $NETSOLVE_ROOT/server_config should be used as a template to create new configuration files. This configuration file is organized as follows. A line can start with a '#' in which case the line is ignored and can be used for comments. A line can also start with a keyword that is prefixed by a '@' typically followed by a single value or parameter. Let us review all of the possible keywords and how they can be used to precisely define a NetSolve server as it is done in the default configuration file.

• '@AGENT:<hostname>'[*] specifies the agent that the NetSolve server must contact to register into a NetSolve system. The agent is identified by the name of the host on which it is running and there can be only one such line in the configuration file.

• '@PROC:<number>' specifies the number of processors (=1 for a single processor, =2 for a dual processor, =4 for a quad processor) that can be used by the server to perform simultaneous computations on the local hosts. There can only be one such line in the configuration file.

• '@MPIHOSTS <filename> <number>' specifies the path to the file that contains the list of machines that can be used by MPI, and the maximum number of processors that can be spawned by MPI.

• '@WORKLOADMAX:<max>' specifies the value of the workload beyond which the server refuses new requests (e.g. '@WORKLOADMAX:100'). A value of -1 means that the server accepts requests regardless of the workload.

• '@SCRATCH:<path>' specifies where the NetSolve server can put temporary directories and files. The default is /tmp/.

• '@CONDOR:<path>' specifies that the NetSolve server is using a Condor [condor1] [condor2] pool as a computing resource. The path to the Condor base directory must be provided. There can be only one such line in the configuration file.

• '@CONDORG:<path>' specifies that the NetSolve server is using a Condor-G job management part of Condor. The path to the Condor-G base directory must be provided. There can be only one such line in the configuration file.

• '@PROBLEMS:' marks the beginning of the list of problem description file (PDF) names that are enabled in the NetSolve server installation. Each of these problem description files contains interfaces to a number of problems/subroutines from a particular software library. If a particular problem description file is enabled in the server configuration file, then the problems/subroutines contained therein become available on that server. A number of PDFs have been written for a variety of software packages, but the default NetSolve installation only enables a small subset, as there is only a limited amount of software included with the NetSolve distribution. Details of description files are given in the Section called Expanding the Server Capabilities in the chapter called Downloading, Installing, and Testing the Agent and Server.

• '@RESTRICTIONS:' marks the beginning of the list of access restrictions that are applicable to the NetSolve server. The list consists of lines formatted as:

  <domain name> <number of pending requests allowed>

  The symbol '*' is used as a wildcard in the domain name. For instance, the line:

  *.edu 10

  means that only 10 requests from clients residing on a .edu machine can be serviced simultaneously. When the server receives a request from some machine, it determines which line in the list must be used to accept or reject the request by taking the most refined domain name. For instance, if the list of the restrictions is:

  *.edu 5 *.utk.edu 10

  then the server accepts at most 5 simultaneous requests coming from .edu machines that are not in the .utk.edu sub-domain, and at most 10 requests that come from machines in the .utk.edu sub-domain for a total of 15 possible simultaneous requests.

Dynamic Servers

Previously, NetSolve servers had been designed such that a server has a specific set of problems which cannot be modified once the server has been started. If the server provider needed to make any modification, he had to bring down the server, make the modification and compile it again.

In any Grid environment, however this could be a big maintainance issue. However, now the server code has been modified in such a way that, the server provider can just modify the $NETSOLVE_ROOT/server_config file, adding or removing a problem, and then running the executable NS_updateserver. NS_updateserver takes two parameters, first the Agent name, then the name of the Server to be updated.

UNIX>NS_updateserver <agent name> <server name>

This would send a signal to the server, on receiving this signal from the user, the server would read the $NETSOLVE_ROOT/server_config file and make the neccessary changes dynamically. The new changes could be seen by using the $NETSOLVE_ROOT/bin/$NETSOLVE_ARCH/NS_problems tool.

Hardware/Software Servers

The current NetSolve has been modified in a way that servers can be registered as a Hardware/Software server, a Hardware server, or a Software server.

When a server is started it is by default a Hardware/Software server. It provides it's own problems to other machines of the same architecture as Software server but can also execute problems from other machines of the same architecture. But if a user wants to provide newly created functions without donating his hardware resources to a NetSolve pool he can install the new problem to a server on his machine and start it with the -s option. So the agent will add the problem to the repository and when someone wants to execute it the binary will be transfered to a Hardware server and executed there.

When a user starts a server without the -s option (the default), the machine becomes available to any NetSolve user for execution of his problem. The agent then adds this server to the database in the hardware server list. It also knows the architecture of the hardware server. Thus if the client submits a request, the agent can make a better decision whether to use this hardware server or not. If other servers are overloaded, it can select this hardware server as a potential server.

When the client makes a request, the agent creates a mapping of the software server containing the particular software with all the hardware servers with the same architecture. Thus, the return value from the agent contains a mapping of hardware and software servers.

The client then contacts the hardware server with the set of input data. The transfer of the software is done automatically. The hardware server contacts the software server and fetches the software.

The software is dynamically transferred from the software server to the hardware server. This however is done only once, when the software arrives at the hardware site, the hardware server can cache the software. The hardware server administrator decides the caching policy. Currently, the hardware server can cache a maximum of 10 problems, and after that it removes the problem on LRU basis.

To mark a problem as non-moveable, i.e., the problem cannot be moved by the Hardware/Software server feature to another server, use the:

@MOVEABLE 0

---

Hardware/Software Server Security

Due to the software transfer between the Software Server to Hardware Servers there exists a need to ensure the code being transfered can be trusted. Security is important to prevent any malicious code from being executed on the Hardware Server. Keeping this in mind, the NetSolve team has implemented a security mechanism in NetSolve using GPG.

GnuPG stands for GNU Privacy Guard and is GNU's tool for secure communication and data storage. It can be used to encrypt data and to create digital signatures. It includes an advanced key management facility and is compliant with the proposed OpenPGP Internet standard as described in RFC 2440. As such, it is aimed to be compatible with PGP from NAI, Inc. For more information on GPG, consult http://www.gnupg.org

Within the scope of NetSolve, Software providers are able to sign software using keys signed by the NetSolve Admin (or others if they choose to). The Hardware Server provider would be able to specify trustworthy parties, using GPG trust framework. By default, the hardware server trusts the NetSolve Admin. Thus to take advantage of software transfer, both the parties should have GPG and should enable the software transfer. NetSolve provides a good level of flexibility in the implementation of Hardware-Software feature and GPG.

There are 2 main cases:

CASE 1: GPG is disabled

In this case, the user has GPG disabled. Hence, the software (problem) will not be transfered by the Software Server nor will the software be transfered to the Hardware Server. This is the default NetSolve mechanism.

CASE 2: GPG is enabled

In this case, the default software, provided in the original NetSolve distribution will not be transfered. (See PDFs in $NETSOLVE_ROOT/problems directory). This is done by having MOVABLE set to 0 in the PDF. However, if the server admin decides to make new software, and have that software moveable (transferable), then he needs to set MOVABLE to 1 in the PDF.

@MOVABLE 1

He also needs to sign the software with a GPG key provided by someone whom the hardware server trusts. Typically this would be the NetSolve admin. The steps required for signing software for use with NetSolve and the Harfdware/Software server are described below.

	Note
	A problem with MOVABLE set to 1 must be signed to run, even on the server that it is located on.

UNIX>gpg --version

UNIX>NS_gen_key

UNIX>gpg --gen-key

   (1) DSA and ElGamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection?

What keysize do you want? (1024)

Requested keysize is 1024 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)

Real name:

Email address:

Comment:

Enter passphrase:

UNIX>NS_export_key username+netsolve@your.example.domain yourkey.gpg

UNIX>gpg --export username+netsolve@your.example.domain > yourkey.gpg

UNIX>NS_import_key signedkey.gpg

UNIX>gpg --import signedkey.gpg

UNIX>NS_sign_problem problem-filename key-id

UNIX>NS_sign_problem service-superlu username+netsolve@your.example.domain

---

Hardware Server Setup - Incorporating a key

By default, a NetSolve hardware server will not accept mobile code. In order for the server to accept mobile code it has to be configured to use gpg to verify the authenticity and integrity of code.

If you configure your hardware server to run gpg, by default the hardware server will accept any code which is either:

• signed by netsolve-master@netlib.org

• signed by a third party whose key is signed by netsolve-master@netlib.org

This section describes how to change which keys your server trusts. In order for your server to trust a signature, the following conditions are necessary

• that key is trusted by the server

• that key is signed by someone the server trusts

By default, NetSolve is set up to trust the following keys:

• "netsolve-hardware-server@example.org" is trusted ultimately

• "netsolve-master@netlib.org" is trusted fully

netsolve-hardware-server@example.org" is a dummy key that is used as the identity of the hardware server in the default configuration. If you wish to change the hardware server's trust parameters you will need to delete this key and create your own key.

NetSolve's copies of GPG files

The NetSolve server maintains its own copies of gpg files (pubring.gpg, secring.gpg, trustdb.gpg). It does this so that there is no conflict between NetSolve's use of gpg and ordinary personal uses of gpg. A user which is trusted to sign someone's key for email might not be trusted to sign netsolve code. These files are stored in a directory that the NetSolve server finds using the environment variable GPGHOME.

There is a program named ns-gpg which is used to affect keys used by the NetSolve server instead of the keys ordinarily used by gpg. It is just a shell script that calls gpg with additional options. You use it to mainipulate keys and trust parameters just like you would use gpg.

How to change the default trust parameters

If you wish to change the default trust parameters, you must go through the following steps:

• delete the netsolve-hardware-server key

• create your own key for the hardware server

• decide whether to trust netsolve-master key

1. Delete the netsolve-hardware-server key

UNIX>gpg --delete-key netsolve-hardware-server

2. Create your own key for the hardware server

UNIX>NS_gen_key

gpg will then prompt you for the name and size of the key to use(see above). We recommend RSA, sign-only keys. A key size of 1024 should be sufficient for most purposes but you may make it larger if necessary. (If you make it larger, signing and verifying will be slower).

To avoid conflicts, the name of the key should be an email address that you control. It's also occasionally handy if the person who runs the server can be reached by sending email to the address. However it might also be desirable for that address to NOT be the normal address of the user who runs the NetSolve hardware server. For example, it might be useful to set up an alias named ns-hw-server@mydomain.org that gets forwarded to joe.sysadmin@mydomain.org.

3. Decide how much to trust the netsolve-master key

If you want your hardware server to be able to accept code signed by other netsolve developers that we (the primary developers) trust, you will need to change the parameters to trust the netsolve-master key. (By default this key was trusted but that's because it was signed by the netsolve-hardware-server key, which you have deleted. Having created your own key, you now need to re-establish trust from that key in the netsolve-master key.)

If you want to trust the netsolve-master key you must do two things:

• sign the key

• change the trust parameters in the key to indicate that it is trusted

If you do not want to trust it, you should not sign the key. But you should change the parameters anyway (to indicate that you don't trust it) just to be sure. To sign the key, do:

UNIX>gpg --sign-key netsolve-master

it will prompt you for the duration of your signature and your passphrase.

To change the trust parameters, do:

UNIX>gpg --edit-key netsolve-master

At the prompt, type "trust" gpg will then display something like this:

----------------------------------------------------------------------- pub 2048R/AABD3F7C created: 2003-07-12 expires: never trust: f/f (1). NetSolve Master Key <netsolve-master@netlib.org> -----------------------------------------------------------------------

The end of the first line shows the current trust parameters:

pub 2048R/AABD3F7C created: 2003-07-12 expires: never trust: f/f ^ | this says how much you trust this key ------------------------------+

- means never set q means don't know n means do NOT trust m means trust marginally f means trust fully u means ultimate trust (i.e. this is you)

gpg goes on to print the following menu:

----------------------------------------------------------------------- Please decide how far you trust this user to correctly verify other users' keys (by looking at passports, checking fingerprints from different sources...)? 1 = Don't know 2 = I do NOT trust 3 = I trust marginally 4 = I trust fully 5 = I trust ultimately m = back to the main menu -----------------------------------------------------------------------

Select "2" if you do not wish to trust the netsolve-master key to sign other keys.

Select "4" if you wish to trust problems that have been signed by people whose keys are signed by netsolve-master. (i.e. people that the netsolve developers trust)

	Note
	Note that if you wish to trust code that is signed directly by netsolve-master, but NOT trust code that is signed by keys that were signed by netsolve-master, you should sign netsolve-master's key but NOT trust netsolve-master as an introducer.

How to list keys

To list the keys that the NetSolve server knows about, do:

UNIX>gpg --list-keys

This will return a list of keys along with their key-ids. You can list both keys and signatures by doing:

UNIX>gpg --list-sigs

How to add a key to NetSolve server's keyring

The person whose key you want to add needs to send it to you in a file. He can extract it into a file by doing:

UNIX>gpg --export key-id > key.gpg

Then (once you've verified that this key really belongs to that person) you can do:

UNIX>gpg --import key.gpg

Then you can sign the key and/or change its trust parameters, per the examples above.

How to delete a key

UNIX>gpg --delete-key key-id

See the gpg man page or other gpg documentation for more information.

NS_config

This executable takes one argument on the command line, the name of a host running a NetSolve agent:

UNIX> NS_config netsolve.cs.utk.edu

It prints the list of hosts participating in the NetSolve system:

AGENT: netsolve.cs.utk.edu (128.169.93.161)
SERVER: maruti.cs.berkeley.edu (128.32.36.83)
SERVER: cupid.cs.utk.edu (128.169.94.221)
SERVER: fsmat.htu.tuwien.ac.at (128.131.95.99) (0 failures, softwareserver)

NS_problems

This executable takes the name of a host running an agent as single argument on its command line. It prints the list of problems that can be solved by contacting that agent:

UNIX> NS_problems netsolve.cs.utk.edu
/BLAS/Matrices/matmul
/ItPack/jsi
/LAPACK/Matrices/EigenValues/eig
/LAPACK/Matrices/SingularValues/svd

NS_probdesc

This executable takes two arguments on its command line: the name of a host running a NetSolve agent and the nickname of a NetSolve problem. It prints the description of the problem:

UNIX> NS_probdesc netsolve.cs.utk.edu linsol
-- linsol -- From LAPACK -
Compute the solution to a real system of linear equations
  A * X = b
where A is an N-by-B matrix and X and B are N-by-NRHS matrices.
Matlab Example : [x] = netsolve('dgesv',a,b)
http://www.netlib.org/lapack/index.html
* 2 objects in INPUT
 - input 0: Matrix Double Precision Real.
 Matrix A
 - input 1: Matrix Double Precision Real.
 Right hand side
* 1 objects in OUTPUT
 - output 0: Matrix Double Precision Real.
 Solution
* Calling sequence from C or Fortran
6 arguments
 - Argument #0:
   - number of rows of input object #0 (A)
   - number of columns of input object #0 (A)
   - number of rows of input object #1 (RHS)
 - Argument #1:
   - number of columns of input object #1 (RHS)
 - Argument #2:
   - pointer to input object #0 (A)
 - Argument #3:
   - leading dimension of input object #0 (A)
 - Argument #4:
   - pointer to input object #1 (RHS)
   - pointer to output object #0 (SOLUTION)
 - Argument #5:
   - leading dimension of input object #1 (RHS)

-- povray -- allows the remote execution of PovRay
(this is a non-moveable problem)

NS_killagent

This executable takes one argument on its command line, the name of a host running a NetSolve agent. After a (basic) user authentication, the executable kills the agent.

UNIX> NS_killagent netsolve.cs.utk.edu
Agent on netsolve.cs.utk.edu :  killed

NS_killserver

This executable takes two arguments on its command line, the name of a host running a NetSolve agent and the name of a host running a NetSolve server. After a (basic) user authentication, the executable kills the server, using the agent as an entry-point into the system.

UNIX> NS_killserver netsolve.cs.utk.edu cupid.cs.utk.edu
Server on cupid.cs.utk.edu killed : killed

NS_killall

This Shell script takes one argument on its command line, the name of a host running a NetSolve agent. After a (basic) user authentication, the executable kills the agent, along with all other NetSolve processes (agents and servers) known to that agent:

UNIX> NS_killall netsolve.cs.utk.edu
Server on cupid.cs.utk.edu : killed
Server on maruti.cs.berkeley.edu : killed
Agent on netsolve.cs.utk.edu : killed

Contents of a Problem Description File

In what follows we describe the contents of a problem description file (PDF). We offer all of the details because it may be necessary or desirable to be aware of them, but we strongly recommend the use of the PDF Wizard described in the Section called PDF Wizard or the Interface Definition Language (IDL) described in the Section called NetSolve IDL - Simplified PDF, to assist in creating new PDFs.

The rationale for the syntax of the description files is explained in [ima]. Each description file is composed of several problem descriptions. Before explaining how to create a problem description, we reiterate the concept of objects in NetSolve, and then define the concept of mnemonics.

<output> = <name>(<input>)

         1 0 3 1
  A =    0 0 5 2
         6 1 0 8
         4 0 0 0

  then,

val:     1 3 1 5 2 6 1 8 4
col_ind: 0 2 3 2 3 0 1 3 0
row_ptr: 0 3 5 8 9

-- sm_prob --
* 1 object in INPUT
 - input 0: Sparse Matrix Double Precision Real.
 the sparse matrix
* Calling sequence from C or Fortran
11 arguments
 - Argument #0:
   - number of rows of input object #0 (sm)
   - number of columns of input object #0 (sm)
 - Argument #1:
   - number of non-zero values of input object #0 (sm)
 - Argument #2:
   - pointer to input object #0 (sm)
 - Argument #3:
   - column indices of non-zeros of input object #0 (sm)
 - Argument #4:
   - row pointers of the sparse matrix #0 (sm)

  >> netsolve('sm_prob', SM);

  double* val;
  int* col_index;
  int* row_ptr;

  int rows, num_nzeros;

  /* initialize the arrays and variables */
   ...
   ...
   ...

  status = netsl("sm_prob()", rows, num_nzeros, val, col_index, row_ptr);

	>> [b] = netsolve('toto',a)

        DOUBLE PRECISION A(M,N)
        DOUBLE PRECISION B(K,L)

        CALL FNETSL('toto()',A,B,M,N,K,L)
        CALL FNETSL('toto()',A,M,N,B,K,L)
        CALL FNETSL('toto()',M,N,A,K,L,B)
        etc.....

void matvec(float *a, float *b, int n, int l);

@CODE
if (*@mI1@ != *@nI0@)
  return NS_PROT_DIM_MISMATCH;

matvec(@I0@,@I1@,*@mI0@,*@mI0@);

@O0@ = @I1@;
*@mO0@ = *@mI1@;
@END_CODE

SUBROUTINE LINSOL( A, B, N, NRHS, LDA, LDB )

DOUBLE PRECISION A( LDA, * )  // Left-hand side (NxN)
DOUBLE PRECISION B( LDB, * )  // Right-hand side (NxNRHS),
                             // overwritten with the solution
INTEGER N
INTEGER NRHS
INTEGER LDA               // Leading Dimension of A
INTEGER LDB               // Leading Dimension of B

@PROBLEM linsol
@INCLUDE <math.h>
@INCLUDE "/home/me/my_header.h"
@LIB -L/home/lib/
@LIB -lstuff
@LIB /home/me/lib_$(NETSOLVE_ARCH).a
@LIB /home/stuff/add.o
@FUNCTION linsol
@LANGUAGE FORTRAN
@MAJOR COL
@PATH    LinearAlgebra/LinearSystems/
@DESCRIPTION
Solves the square linear system A*X = B. Where:
 A is a double-precision matrix of dimension NxN
 B is a double-precision matrix of dimension NxNRHS
 X is the solution
@INPUT 2
@OBJECT MATRIX D A 
Matrix A (NxN)
@OBJECT MATRIX D B
Matrix B (NxNRHS)
@OUTPUT 1
@OBJECT MATRIX D X
Solution X (NxNRHS)
@COMPLEXITY 3,3
@CALLINGSEQUENCE 
@ARG I0
@ARG I1,O0
@ARG nI0,mI0,mI1
@ARG nI1
@ARG lI0
@ARG lI1,lO0
@CODE

linsol(@I0@,@I1@,@mI0@,@nI1@,@lI0@,@lI1@);

@O0@ =@I1@;       /* Pointing to the overwritten input */
*@mO0@ = *@mI1@;  /* Setting the number of rows        */
*@nO0@ = *@nI1@;  /* Setting the number of columns     */

@END_CODE

UNIX> make pdfgui

NetSolve IDL - Simplified PDF

In an effort to help the library writers to easily integrate their problems into NetSolve, we are developing an IDL (Interface Definition Language) for NetSolve. The NetSolve IDL has a simpler format than the existing PDF (Problem Description File) mechanisms. Example 1: An example IDL is given below.

PROBLEM dgesv

Fortran ROUTINE dgesv(IN int N, IN int NRHS, INOUT double A[LDA][N],
                      IN int LDA, OUT int IPIV[N], INOUT double
                      B[LDB][NRHS], IN int LDB, OUT int INFO)

"From LAPACK -

Compute the solution to a real system of linear equations
  A * X = b
where A is an N-by-B matrix and X and B are N-by-NRHS matrices.

"
LIBS = "/usr/local/lib/liblapack.a
        /usr/local/lib/libf77blas.a
        /usr/local/lib/libatlas.a"

The above IDL describes a routine called dgesv written in Fortran. The routine accepts 8 parameters. Parameters 1, 2, 4 and 7 are input parameters of type integer. Parameter 8 is an output parameter of type integer. The fifth parameter, IPIV is an integer vector of N elements and is an output parameter. Parameters 3 and 6 are matrices of double precision numbers and act as both input and output parameters. The variables in the square brackets represent the rows and columns of the matrices respectively. For example, LDA represents the number of rows and N represents the number of columns of the matrix A.

The description of the function interface can be followed by comments about the problem in quotes. This is followed by a listing of the libraries needed for linking with the function. For example, in the above example, liblapack.a, libf77blas.a and libatlas.a are needed to link with the Fortran function dgesv.

The default type of the function is assumed to be sequential. Following is the Backus Normal Form (BNF) of the NetSolve IDL:

        <PROBLEMSTART>       -> <PROBLEMDESC> <FUNCTION>
    <PROBLEMDESC>        -> PROBLEM <PROBLEMNAME>
    <FUNCTION>           -> <LANGUAGE> FUNCTION <FUNCDEFN> <FUNCDESC>
                            <FUNCLIB> <FUNCMOVEABLE>
    <LANGUAGE>           -> C | FORTRAN
    <FUNCDEFN>           -> <FUNCNAME> ( <ARGLIST> )
    <FUNCDESC>           -> '' <STRING> ''
    <FUNCLIB>            -> LIBS = '' <STRING> ''
    <FUNCMOVEABLE>       -> empty | MOVEABLE | NONMOVEABLE
    <ARGLIST>            -> <ARGUMENT> | <ARGLIST> , <ARGUMENT>
    <ARGUMENT>           -> <INOUTSTRING> <DATATYPE> <VARNAME>
                            | <INOUTSTRING> <DATATYPE> <VARNAME>
                              <VACTORATTR>
                            | <INOUTSTRING> <DATATYPE> <VARNAME>
                              <MATRIXATTR>
                            | <INOUTSTRING> <DATATYPE> <VARNAME>
                              <SPARSEMATRIXATTR>
    <VECTORATTR>         -> [ <DIMENSIONEXPR> ]
    <MATRIXATTR>         -> [ <DIMENSIONEXPR> ] [ <DIMENSIONEXPR> ]
    <SPARSEMATRIXATTR>   -> <MATRIXATTR> <OPENANGLE> <NNZVAR> ,
                            <INDEXVAR> , <POINTERVAR> <CLOSEANGLE>
    <DIMENSIONEXPR>      -> <NUMBER> | <VARNAME>
    <OPENANGLE>          -> <
    <CLOSEANGLE>         -> >
    <NNZVAR>             -> IN int <VARNAME>
    <INDEXVAR>           -> IN int <VARNAME> <VECTORATTR>
    <POINTERVAR>         -> IN int <VARNAME> <VECTORATTR>
    <PROBLEMNAME>        -> <IDENTIFIER>
    <FUNCNAME>           -> <IDENTIFIER>
    <TYPESTRING>         -> sequential | parallel
    <INOUTSTRING>        -> IN | OUT | INOUT
    <DATATYPE>           -> int | float | double | char | scomplex |
                            dcomplex | string | file
    <VARNAME>            -> <IDENTIFIER>

Example 2: The following IDL file demonstrates the use of sparse matrix data structure in the IDL file.

PROBLEM sparse_direct_solve

C ROUTINE sparse_direct_solve(IN string package, IN int N, IN double
                              SM[N][N]<nnz, indices, pointer>, IN int
                              nnz, IN int indices[nnz], IN int
                              pointer[N], IN double rhs[N], IN double
                              pivot, IN int permutation, OUT double
                              sol[N])

"sparse direct solve - ma28 and superlu"
LIBS = "/home/vss/idltopdf/sparse_direct_wrapper.o
 -lm -L$(NETSOLVE_ROOT)/lib/$(NETSOLVE_ARCH) -lnetsolve_ma28
 -lnetsolve_superlu_serial -lnetsolve_aux -lnetsolve_direct_driver
 -lnetsolve_tester $(LIBDIR)/libma28.a $(SUPERLU_LIB_LINK)
 $(LAPACK_LIB_LINK) $(BLAS_LIB_LINK) -L$(MPI_DIR)/lib -lmpich"

In the above IDL file, the third parameter, SM, is a sparse matrix represented by a compressed-row format. N is the number of rows and columns of the sparse matrix, nnz is the number of non-zeros, indices is a vector of column indices and pointer is a vector containing a pointer to the first non-zero element in each row.

Limitations

The NetSolve IDL is in the early stages of development and does not support all the features supported by PDF. The number of elements in a vector or the number of rows and columns in a matrix can be either constants or one of the input integer parameters. It cannot be an arithmetic expression. Files and strings are not supported by the NetSolve IDL. The IDL does not contain a code section where the user can write program statements.

idltopdf Utility

As part of the standard make in NetSolve, an executable called "idltopdf" is created in the $NETSOLVE_ROOT/bin/$NETSOLVE_ARCH directory. After creating the IDL file, you can create the corresponding PDF by using the idltopdf utility. For example, after creating an IDL file by the name problem.idl, you can create problem.pdf by executing

UNIX> idltopdf problem.idl

./problems/problem.pdf

Introduction

This version of NetSolve has (rudimentary) Kerberos support. NetSolve components include clients, agents, and servers. Currently the only requests that require authentication are requests that the client makes to the server, and of those, only the ``run problem'' request. Other requests could be authenticated (an obvious one being ``kill server''), but drastic changes along these lines would probably require drastic restructuring of NetSolve. For instance, a client can currently inform an agent that a particular server is down, and the agent will not advertise that server for use in other problems. It seems of dubious value to require authentication for such requests until there is a mechanism for specifying the trust relationship between clients and agents.

An attempt has been made to allow Kerberized NetSolve clients to interoperate with both Kerberized and non-Kerberized NetSolve servers. In either case the client sends a request to the server.An ordinary server will return a status code indicating that he will accept the requested operation. By contrast, a Kerberized server will immediately return an ``authentication required'' error in response to the request. The client is then required to send Kerberos credentials to the server before the request will be processed. This allows the server to require authentication of the client. Currently there is no mechanism to allow the client to insist on authentication of the server - a Kerberized client will happily talk with either Kerberized or non-Kerberized servers.

The server implements access control via a simple list of Kerberos principal names. This list is kept in a text file which is consulted by the server. A request to a NetSolve server must be made on behalf of one of those principal names. If the principal name associated with the Kerberos credentials in the request appears in the list, and the credentials are otherwise valid, the request will be honored. Otherwise, the request will be denied.

Since the NetSolve server was not designed to run as a set-uid program, it is not currently feasible to have the NetSolve server run processes using the user-id of the particular UNIX user who submitted the request. NetSolve thus uses its own service principal name of ``netsolve'' rather than using the ``host'' principal. What this means (among other things) is that you need to generate service principals and keytabs for each of your NetSolve servers, even if you already have host principals in place.

The NetSolve server, by default, runs in non-Kerberized mode. To start up the server in Kerberized mode you need to add the -k option to the command-line, and also set environment variables NETSOLVE_KEYTAB (pointing to the keytab) and NETSOLVE_USERS pointing to the list of authorized users).

This version of Kerberized NetSolve performs no encryption of the data exchanged among NetSolve clients, servers, or agents. Nor is there any integrity protection for the data stream.

Compiling a Kerberized Server

1. Compile Kerberos. See the Kerberos V5 Installation Guide for instructions for how to do this.

2. Compile the NetSolve server with Kerberos support (./configure --with-kerberos).

Installing a Kerberized Server

1. Install Kerberos on the server machine. See Kerberos V5 Installation Guide for instructions for how to do this. You do not have to install all of the Kerberos clients just to run a NetSolve server, but you do need kadmin and components that deal with Kerberos tickets like kinit and kdestroy.

2. Define a Kerberos service principal for the NetSolve server. To define the principal for machine foo.bar.com:

   1. Get the name and the password of a Kerberos principal that is authorized to run kadmin and create principals.

   2. Log on to the machine where you want to install the Kerberized NetSolve server. Make sure you have a secure connection to the client machine (perhaps you're typing on the machine's keyboard, or perhaps you're using ssh to log in to that machine), so that your password will not be exposed on the net.

   3. Do a kinit to acquire a ticket that identifies you as someone who can create principals.

   4. Create a service principal for the NetSolve server on your host. If your host is named foo.bar.com, the service principal should be named netsolve/foo.bar.com:

      UNIX> kadmin

      (if you don't have a Kerberos ticket yet, kadmin will try to get one for you based on your UNIX username. If there is a Kerberos principal for that username, and that principal has the ability to create new principals, just type in your password when asked to do so. Otherwise run kinit to get a ticket for some other principal - one that has the ability to create new principals - and then run kadmin again.)

      UNIX> kadmin: addprincipal -randkey netsolve/foo.bar.com UNIX> kadmin: ktadd -k /etc/netsolve.keytab netsolve/foo.bar.com

      This will extract the key into the file /etc/netsolve.keytab. You can put this keytab any place you want it but it must be on a local filesystem. If you put the file on a NFS-mounted filesystem then (a) you will compromise the security of your server by exposing the key to eavesdroppers, and (b) there's a good chance that NFS file locking bugs will cause your NetSolve server to get wedged.

   5. While you're at it, you might want to define other service principals for the same host. For instance, a service principal of the form host/foo.bar.com is needed if you want to allow Kerberized logins to that host. This is straightforward:

      UNIX> kadmin: addprincipal -randkey host/foo.bar.com UNIX> kadmin: ktadd host/foo.bar.com

   6. Make sure that /etc/netsolve.keytab is readable only by the UNIX user-id that will run the NetSolve server. (Permissions should be 0600, -rw-------). The owner should not be root.

Running a Kerberized Server

1. You must have a NetSolve agent running somewhere first.

2. You must be logged into UNIX as the owner of the /etc/netsolve.keytab file, since the server needs to be able to read this file.

3. Set up the environment variables:

   UNIX> setenv NETSOLVE_AGENT netsolve.agent.host UNIX> setenv NETSOLVE_KEYTAB /etc/netsolve.keytab UNIX> setenv NETSOLVE_USERS /etc/netsolve.users

   The NETSOLVE_USERS file is a text file that contains a list of Kerberos principal names, one per line, who are authorized to use the server. It is reopened each time a user tries to authenticate to the server, so you can add users while the server is running.

4. Start the server

   UNIX> /path/to/netsolve/NS_server -k &

   If you do not use the -k flag, the server will not require authentication.

Introduction

NetSolve has provided an interface to the Condor system. Condor, developed at the University of Wisconsin, Madison, is a high throughput computing environment that can manage very large collections of distributed workstations. Condor-G is the job management part of Condor. It allows users to submit jobs into a queue, have a log detailing the life cycle of your jobs, manage input and output files, along with everything else expected from a job queuing system. It is a program that manages both a queue of jobs and the resources from one or more sites where those jobs can execute. Condor-G communicates with these resources and transfers files to and from these resources using Globus mechanisms.

It is possible to start a NetSolve server as a front-end to a Condor pool transparently to the NetSolve user. The user now can access grid resources without any interference.

To Use Condor-G with NetSolve

Before using Condor-G with NetSolve, Condor, Condor-G and Globus should be properly installed and configured. Condor and Globus can be downloaded at:

http://www.cs.wisc.edu/condor

http://www.globus.org

To use Condor-G with NetSolve, one should start a specialized Condor-G server in NetSolve. This is done by inserting; '@CONDORG:<path> 1' into $NETSOLVE_ROOT/server_config file. After making server by typing:

UNIX>make server

How it works

A job is submitted for execution to a Condor-G server. The server will then prepare a temporary submit description file similar to this:

executable      = /homes/shi/NetSolve/bin/sparc_sun_solaris2_8/service-lapack_subset
globusscheduler = msc03.cs.utk.edu/jobmanager
universe        = globus
output          = condor.out
error           = condor.error
arguments       = 0  /homes/shi/NetSolve/bin/sparc_sun_solaris2_8
queue

The globusscheduler command is dependent on the scheduling software available on the remote resource. This required command should be changed based on the Grid resource intended for execution of the job. All Condor-G jobs are submitted to the Globus universe. The NetSolve server then issues condor_submit to submit the job for execution on the Globus resources. After the job is finished on the remote machines, the server will collect the results and send them to the client.

DSI Introduction

The Distributed Storage Infrastructure (DSI) is an attempt towards achieving coscheduling of the computation and data movement over the NetSolve Grid. The DSI APIs help the user in controlling the placement of data that will be accessed by a NetSolve service. This is useful in situations where a given service accesses a single block of data a number of times. Instead of multiple transmissions of the same data from the client to the server, the DSI feature helps to transfer the data from the client to a storage server just once, and relatively cheap multiple transmissions from the storage server to the computational server. Thus the present DSI feature helps NetSolve to operate in a cache-like setting. Presently, only Internet Backplane Protocol (IBP) is used for providing the storage service. In the future, we hope to integrate other commonly available storage service systems.

Using DSI

To use DSI, one should enable the DSI feature both at the NetSolve client and the server. Type

UNIX> ./configure --with-ibp=IBP_DIR

	Note
	When using IBP in a server pool that has both IBP enabled servers and those that aren't IBP enabled, one should use the assigned server feature to ensure that the problem submission goes to a server with IBP enabled.

DSI APIs

The DSI APIs are modeled after the UNIX file manipulation commands (open, close etc.) with a few extra parameters that are specific to the concepts of DSI. This section provides the syntax and semantics of the different DSI APIs available to the NetSolve user.

ns_dsi_open() is used for allocating a chunk of storage in the IBP storage. On success, ns_dsi_open returns a pointer to the DSI file. On failure, returns NULL. Following are the various error values set in case of failure.

ns_dsi_close() is used for closing a DSI file.

On success returns 1. On failure, returns -1. Following are the various error values set in case of failure.

ns_dsi_write_vector() is used for writing a vector of a particular datatype to a DSI file.

On success, ns_dsi_write_vector() returns a pointer to the DSI object created for the vector. On failure, returns NULL. Following are the various error values set in case of failure.

Same functionality and return values as ns_dsi_write_vector() except ns_dsi_write_matrix() is used to write matrix of rows rows and cols columns.

On success, returns the number of elements read. On failure, returns -1. Following are the various error values set in case of failure.

Same functionality and return values as ns_dsi_read_vector() except ns_dsi_read_matrix() is used to read matrix of rows rows and cols columns.

DSI Example

This section shows two example programs. The first program solves quick sort without using the DSI feature. The second program solves the same quick sort, but with using the dsi feature.

int main(){
int i;
int length;
int* inputVec;
int* outputVec;
int status;

  printf("Enter the number of vector elements: \n");
  scanf("%d", &length);

  inputVec = (int*)malloc(sizeof(int)*length);
  outputVec = (int*)malloc(sizeof(int)*length);

  for(i=0; i<length; i++){
    printf("Element %d: ", i+1);
    scanf("%d", &inputVec[i]);
  }

  status = netsl("iqsort()", length, inputVec, outputVec);

  printf("\n\nSorted Elements: \n");
  for(i=0; i<length; i++)
    printf("%d ", outputVec[i]);
  printf("\n");

  return 0;
}

int main(){
int i;
int length;
int* inputVec;
int* outputVec;
int status;
DSI_FILE* dsi_file;
DSI_OBJECT* dvec;

  printf("Enter the number of vector elements: \n");

  scanf("%d", &length);

  inputVec = (int*)malloc(sizeof(int)*length);
  outputVec = (int*)malloc(sizeof(int)*length);

  for(i=0; i<length; i++){
    printf("Element %d: ", i+1);
    scanf("%d", &inputVec[i]);
  }


  dsi_file = ns_dsi_open("torc1.cs.utk.edu", O_CREAT|O_RDWR , 744 , 3000, IBP);
  if(dsi_file == NULL){
    printf("error in open\n");
  }

  dvec = ns_dsi_write_vector(dsi_file, inputVec, 10, NETSOLVE_D);
  if(dvec == NULL){
    printf("error in write\n");
  }

  status = netsl("iqsort()", length, dvec, outputVec);

  printf("\n\nSorted Elements: \n");
  for(i=0; i<length; i++)
    printf("%d ", outputVec[i]);
  printf("\n");

  ns_dsi_close(dsi_file);

  return 0;

}

Introduction

The GridRPC API represents ongoing work to standardize and implement a portable and simple remote procedure call (RPC) mechanism for grid computing. This standardization effort is being pursued through the Global Grid Forum Research Group on Programming Models [apme]. The initial work on GridRPC reported in [grpcapi] shows that client access to existing grid computing systems such as NetSolve and Ninf [ninf] can be unified via a common API, a task that has proven to be problematic in the past.

Starting with version 2.0, NetSolve includes an implementation of the GridRPC API. Use of this API is optional and has not superseded the original NetSolve API. For full details of the specification of the GridRPC API, see [grpcapi].

Compilation Instructions

You should configure NetSolve before attempting to compile the GridRPC library. Compiling all of NetSolve isn't necessary to compile the GridRPC client library, however the NetSolve client library will be automatically built first if it hasn't been built already. Therefore, to build the GridRPC API, from the NETSOLVE_ROOT, it should suffice to simply type:

UNIX> make gridrpc

This creates a file named $NETSOLVE_ROOT/lib/$NETSOLVE_ARCH/libnsgrpc.a as well as the NetSolve client library if necessary (named $NETSOLVE_ROOT/lib/$NETSOLVE_ARCH/libnetsolve.a).

Testing

Of course this requires that you have access to a NetSolve agent and server running somewhere. You can start your own or use the agent at netsolve.cs.utk.edu. In any case don't forget to set your NetSolve environment variables.

UNIX> $NETSOLVE_ROOT/bin/$NETSOLVE_ARCH/C_test_grpc

UNIX> cd $NETSOLVE_ROOT/src/GridRPC
UNIX> make tests

Additional Notes on Writing and Compiling GridRPC Programs

To use the GridRPC API, you should include the header file grpc.h in your C program. This header file defines the various constants and function prototypes you may need when writing GridRPC programs. Also, when compiling, append $NETSOLVE_ROOT/src/GridRPC to your include path. This is typically done with the "-I" flag.

At link time, you will need to link the GridRPC client library as well as the NetSolve client library with your program. This would normally be specified when linking as: "-lnsgrpc -lnetsolve". Also, the library path may need to be specified at this time using "-L$NETSOLVE_ROOT/lib/$NETSOLVE_ARCH".

Finally, on some platforms such as Solaris, you may need to link in additional libraries for certain system calls (e.g. socket). If you encounter linker errors, try adding "-lnsl -lsocket" to your link command.

Function Handles and Session IDs

Two fundamental objects in the GridRPC model are function handles and the session IDs. The function handle represents a mapping from a function name to an instance of that function on a particular server. The GridRPC API does not dictate the mechanics of resource discovery since different underlying GridRPC implementations may usevastly different protocols. Once a particular function-to-server mapping has been established by initializing a function handle, all RPC calls using that function handle will be executed on the server specified in that binding. A session ID is an identifier representing a particular non-blocking RPC call. The session ID is used throughout the API to allow users to obtain the status of a previously submitted non- blocking call, to wait for a call to complete, to cancel a call, or to check the error code of a call.

Initializing and Finalizing Functions

The initialize and finalize functions are similar to the MPI initialize and finalize calls. Client GridRPC calls before initialization or after finalization will fail.

• grpc_initialize reads the configuration file and initializes the required modules.

• grpc_finalize releases any resources being used by GridRPC.

Remote Function Handle Management Functions

The function handle management group of functions allows creating and destroying function handles.

• grpc_function_handle_default creates a new function handle using the default server. This could be a pre-determined server name or it could be a server that is dynamically chosen by the resource discovery mechanisms of the underlying GridRPC implementation, such as the NetSolve agent.

• grpc_function_handle_init creates a new function handle with a server explicitly specified by the user.

• grpc_function_handle_destruct releases the memory associated with the specified function handle.

• grpc_get_handle returns the function handle corresponding to the given session ID (that is, corresponding to that particular non-blocking request).

GridRPC Call Functions

The four GridRPC call functions may be categorized by a combination of two properties: block- ing behavior and calling sequence. A call may be either blocking (synchronous) or non-blocking (asynchronous) and it may use either a variable number of arguments (like printf) or anargument stack calling sequence. The argument stack calling sequence allows building the list of arguments to the function at runtime through elementary stack operations, such as push and pop.

• grpc_call makes a blocking remote procedure call with a variable number of arguments.

• grpc_call_async makes a non-blocking remote procedure call with a variable number of arguments.

• grpc_call_argstack makes a blocking call using the argument stack.

• grpc_call_argstack_async makes a non-blocking call using the argument stack.

Asynchronous GridRPC Control Functions

The following functions apply only to previously submitted non-blocking requests.

• grpc_probe checks whether the asynchronous GridRPC call has completed.

• grpc_cancel cancels the specified asynchronous GridRPC call.

Asynchronous GridRPC Wait Functions

The following five functions apply only to previously submitted non-blocking requests. These calls allow an application to express desired non-deterministic completion semantics to the underlying system, rather than repeatedly polling on a set of sessions IDs. (From an implementation standpoint, such information could be conveyed to the OS scheduler to reduce cycles wasted on polling.)

• grpc_wait blocks until the specified non-blocking requests to complete.

• grpc_wait_and blocks until all of the specified non-blocking requests in a given set have completed.

• grpc_wait_or blocks until any of the specified non-blocking requests in a given set has completed.

• grpc_wait_all blocks until all previously issued non-blocking requests have completed.

• grpc_wait_any blocks until any previously issued non-blocking request has completed.

Error Reporting Functions

Of course it is possible that some GridRPC calls can fail, so we need to provide the ability to check the error code of previously submitted requests. The following error reporting functions provide error codes and human-readable error descriptions.

• grpc_perror prints the error string associated with the last GridRPC call.

• grpc_error_string returns the error description string, given a numeric error code.

• grpc_get_error returns the error code associated with a given non-blocking request.

• grpc_get_last_error returns the error code for the last invoked GridRPC call.

Argument Stack Functions

When describing the GridRPC call functions, we mentioned that there is an alternate calling style that uses an argument stack. With the following functions it is possible to construct the arguments to a function call at run-time. When interpreted as a list of arguments, the stack is ordered from bottom up. That is, to emulate a function call f(a,b,c), the user would push the arguments in the same order: push(a); push(b); push(c);.

• newArgStack creates a new argument stack.

• pushArg pushes the specified argument onto the stack.

• popArg removes the top element from the stack.

• destructArgStack frees the memory associated with the specified argument stack.

Detailed GridRPC API Specification

Initializing and Finalizing Functions

int grpc_initialize( char * config_file_name);
int grpc_finalize();

Remote Function Handle Management Functions

int grpc_function_handle_default(grpc_function_handle_t * handle,
char * func_name);
int grpc_function_handle_init(grpc_function_handle_t * handle,
char * host_name, int port, char * func_name);
int grpc_function_handle_destruct(grpc_function_handle_t * handle);
grpc_function_handle_t * grpc_get_handle(int sessionId);

GridRPC Call Functions

int grpc_call(grpc_function_handle_t *handle, ...);
int grpc_call_async(grpc_function_handle_t *handle, ...);
int grpc_call_argstack(grpc_function_handle_t *handle, ArgStack *args);
int grpc_call_argstack_async(grpc_function_handle_t *handle, ArgStack *args);

Asynchronous GridRPC Control Functions

int grpc_probe(int sessionID);
int grpc_cancel(int sessionID);

Asynchronous GridRPC Wait Functions

int grpc_wait(int sessionID);
int grpc_wait_and(int * idArray, int length);
int grpc_wait_or(int * idArray, int length, int * idPtr);
int grpc_wait_all();
int grpc_wait_any(int * idPtr);

Error Reporting Functions

void grpc_perror(char * str);
char * grpc_error_string(int error_code);
int grpc_get_error(int sessionID);
int grpc_get_last_error();

Argument Stack Functions

ArgStack *newArgStack(int maxsize);
int pushArg(ArgStack *stack, void *arg);
void *popArg(ArgStack *stack);
int destructArgStack(ArgStack *stack);

Introduction

In NetSolve, as in other metacomputing systems, the scheduling of tasks to available resources is difficult. NetSolve uses a limited load-balancing strategy to improve the utilization of computational resources. This load-balancing strategy takes into account the current workload of the computational resources available in the NetSolve system. In scheduling the client's requests over a network, the workload estimate should be ``forecast'' for when the computation will execute, and not a workload estimate obtained at a time prior to the request. There are also other characteristics of distributed metacomputing resources such as the CPU speed of the resource, the amount of physical memory of the resource, as well as the latency/bandwidth from the client to the computational resource, that can be effectively utilized in scheduling decisions for the computational resources.

The Network Weather Service (NWS) is a system which provides a way of forecasting dynamically changing performance characteristics, such as the workload, from distributed metacomputing resources. Integrating NWS into NetSolve improves the load-balancing strategy by taking into account the future load instead of the current load of the computational resources.

Using NWS

To use NWS within NetSolve, one must enable the NWS feature by typing

UNIX> ./configure --with-nws=NWS_DIR

Note: $NWS_DIR/bin/ARCH should be in your PATH

NWS Components utilized in NetSolve

A nameserver must be started first in an NWS system, as all other NWS processes depend upon it. After starting the nameserver, memories can then register themselves, and sensor or forecaster processes can be initialized on any host.

The default port numbers reserved for the NWS processes (nameserver, memory, forecaster, and sensor) are specified in the file $NETSOLVE_ROOT/include/nwsutils.h.

The integration of NWS into NetSolve requires the startup of NWS processes, their management and the accurate use of the forecaster. The NWS processes (nameserver, memory, forecaster, and sensor) can be started in various places within NetSolve. We now present our design for the integration and motivate our choices.

NetSolve agent and the NWS nameserver, memory and forecast

As previously stated, only one NWS nameserver can exist in an NWS system, and this process must be placed in NetSolve where it will have full knowledge of the computational resources and be visible to all components of the NetSolve system. The Netsolve agent is the ``brain'' of the NetSolve system, knowing how many resources exist and where they are located, and making all decisions on the execution of requests in the system. Moreover, the NetSolve agent is known by all components of the NetSolve system. Thus, the logical choice for the placement of the NWS nameserver is on the NetSolve agent.

During the agents initialization, a nameserver and a memory are started. In fact the memory is started for the sake of simplicity. Indeed, the agent is known by the whole system. It enables each sensor to register and easily store its measurements. Furthermore this scheme avoids unnecessary communication costs. A forecaster process is then started by the agent. It generates information as soon as needed by the agent. Thus, the agent possesses its own forecaster and can deal with client requests. We shall now examine what happens on computational resources.

NetSolve server and the NWS sensor

As soon as a NetSolve server (computational resource) is added to the NetSolve system, it is necessary to start an NWS sensor. This sensor is started on the server after its registration with the agent to avoid any incoherency with the NetSolve system. The NWS sensor is totally independent from the NetSolve processes running on the server.

At present, the NWS sensor is only detecting the CPU speed of the computational resource. Future implementations will expand this functionality to include monitoring for the amount of physical memory available per computational resource, as well as the latency/bandwidth of the communication between each server and the client. These improvements will require an additional sensor to be started on the client.

Building the Octave Interface

First obtain and build Octave. Octave is available from: http://www.octave.org

The user should use the configure option --with-octave-include to specify the path for the Octave include directory. You may also need to include the configure option --enable-shared when building Octave. The user should set the environment variable NETSOLVE_ROOT to the top-level NetSolve directory before building the interface. For instance;

UNIX> setenv NETSOLVE_ROOT /path/to/NetSolve

UNIX> make octave

What to Do First

Let us assume that the user has compiled the Octave interface, set an agent name, started an Octave session and is now ready to use NetSolve. In this section we describe those features of the interface that allow to the user to obtain information about the currently available NetSolve system. To obtain a list of solvable problems from Octave type the following command;

octave>> netsolve
  NetSolve - List of available problems -
  /BLAS-wrappers/Level3/dmatmul
  /BLAS-wrappers/Level3/zmatmul
  /BLAS/Level1/daxpy
  /BLAS/Level1/ddot
  /BLAS/Level1/zaxpy
  /BLAS/Level2/dgemv
  /BLAS/Level3/dgemm
  /BLAS/Level3/zgemm
  /LAPACK-wrapper/Simple/Eig_and_Singular/eig
  >>

If the user would like more detailed information on a specific problem, for example, eig, he can type:

octave>> netsolve('eig')

  -- eig -- Wrapper around the LAPACK routine DGEEV --
  Simplified version of DGEEV.
  Computes the eigenvalues of a double precision real
  matrix A. Returns two double precision real
  vectors containing respectively the real parts and
  the imaginary parts of the eigenvalues.
  MATLAB Example : [r i ] = netsolve('eig',a)
  * 1 objects in INPUT
   - input 0: Matrix Double Precision Real.
   Matrix A
  * 2 objects in OUTPUT
   - output 0: Vector Double Precision Real.
   Real parts of the eigen values
   - output 1: Vector Double Precision Real.
   Imaginary parts of the eigen values
  --------------------------------------
  Output Objects 0 and 1 can be merged.

This output gives a short description of the problem, an example in Octave using netsolve(), the input objects that must be supplied by the user, and the output that will be returned to the user. The particular problem requires only one double-precision matrix on input. Notice that this matrix must be square (as stated in the description of the problem). If the user tries to call NetSolve for this problem with a rectangular matrix, he will receive an error message stating that the dimensions of the input are invalid. On output, the problems eig will return two vectors, the real and imaginary parts of the eigenvalues of the input matrix respectively.

Since Octave provides a mechanism to manipulate complex objects, it is probable that the user would like to have eig return one single complex vector instead of two separate real vectors. Thus in the Octave interface it is possible to merge these two real output vectors into one complex vector. This point is further developed in the next section.

The Octave interface has another feature that is concerned not with the actual problem solving but with providing information about the NetSolve configuration itself. We have just seen how to get information about the problems handled by the NetSolve servers; it is also possible to obtain the physical location of these servers. Let us assume that our NETSOLVE_AGENT environment variable is set to netsolve.cs.utk.edu. The command

octave>> netsolve('?')

NetSolve - List of available agents -
netsolve.cs.utk.edu(128.169.93.161)
NetSolve - List of available servers -
maruti.cs.berkeley.edu(128.32.36.83)
cupid.cs.utk.edu(128.169.94.221)
torc3.cs.utk.edu(128.169.93.74) (0 failures)

Calling netsolve() to perform computations

The easiet way to perform a numerical computation in NetSolve is to call the function netsolve(). With this function, the user sends a blocking request to NetSolve. By blocking we mean that after typing the command in the Octave session, the user resumes control only when the computation has been successfully completed on a server. The other way to perform computation is to send a nonblocking request as described in later sections.

Let us continue with the eig example we started to develop in the preceding section. The user now knows that he has to provide a double-precision square matrix to NetSolve, and he knows that he is going to get two real vectors back (or one single complex vector). He first creates a 300 X 300 matrix for instance;

octave>> a = rand(300);

>> [x y] = netsolve('eig',a)

All the calls to netsolve() will look the same, following a similar convention as those in the Matlab section. The left-hand side must contain the arguments, in the same order as listed in the output description. The first argument to netsolve() is always the name of the problem. After this first argument the input arguments are listed, in the same order as they are listed in the input description. This function does not have a fixed calling sequence, since the number of inputs and outputs depends on the problem that needs to be solved. Let us see what happens when we type:

octave>> [x y] = netsolve('eig',a)

  x =           y =

  10.1204            0
  -0.9801       0.8991
  -0.9801      -0.8991
  -1.0195            0
  -0.6416       0.6511
    ...          ...
    ...          ...

As mentioned earlier, the user can decide to regroup x and y into one single complex vector. Let us make it clear again that this possibility is a specificity of eig and is not available in general for all problems. To merge x and y, the user has to type:

octave>> [x] = netsolve('eig',a)

  x = 
      10.1204
      -0.9801 + 0.8991
      -0.9801 - 0.8991
      -1.0195
      -0.6416 + 0.6511
           ......
           ......

Calling netsolve_nb()

The obvious drawback of the function netsolve() is that while the computation is being performed remotely, the user must wait to regain control of the prompt. To address this drawback, we provide a nonblocking function, netsolve_nb(). The user can then do work in parallel and check for the completion of the request later. He can even send multiple requests to NetSolve. Thanks to the load-balancing strategy implemented in NetSolve agent, all these requests will be solved on different machines if possible, achieving some NetSolve-parallelism. Let us describe this function with the eig example.

As in the previous section, the user creates a 300 x 300 matrix and calls NetSolve:

octave>> a = rand(300);
octave>> [r] = netsolve_nb('send','eig',a)

Let us resume our example and see what NetSolve answers to the first call to netsolve_nb():

octave>> [r] = netsolve_nb('send','eig',a)
  rd->request_id = 0

  r =  
  
  0

octave>> [status] = netsolve_nb('probe',r)

octave>> [status] = netsolve_nb('probe',r)
  Not ready yet
  status = -1
  ...
octave>> [status] = netsolve_nb('probe',r)
  Result Available
  status = 0

To obtain the result of the computation one must call netsolve_nb() with the 'wait' action:

octave>> [x y] = netsolve_nb('wait',r)
  x =           y =
  
  10.1204            0
  -0.9801       0.8991
  -0.9801      -0.8991
  -1.0195            0
  -0.6416       0.6511
    ...          ...
    ...          ...

As with the netsolve() function, one can merge the real part and the imaginary part into a single complex vector. The typical scenario is to call netsolve_nb() with the action 'send', then make repeated calls with the action 'probe' until there is nothing more to do than wait for the result. The user then calls netsolve_nb() with the action 'wait'. It is of course possible to call netsolve_nb() with the action 'wait' before making any call with the action 'probe'. One last action can be passed to netsolve_nb() as shown here:

octave>> netsolve_nb('status')

octave>> a = rand(100); b = rand(150);
octave>> [r1] = netsolve_nb('send','eig',a)
  rd->request_id = 0

  r1 =
 

      0

octave>> [r2] = netsolve_nb('send','eig',b)
  rd->request_id = 1

  r2 =


      1

octave>> netsolve_nb('status')
  --- NetSolve: pending requests ---
  Requests #0: 'eig', submit-
  ted to torc3.cs.utk.edu (128.169.93.74)
          was started 41 seconds ago.
  netsolveProbeRequest returned: 1, ns_errno = 0
          Completed
  Requests #1: 'eig', submit-
  ted to torc3.cs.utk.edu (128.169.93.74)
          was started 23 seconds ago.
  netsolveProbeRequest returned: 1, ns_errno = 0
          Completed

Testing scripts

Test scripts are also included in NetSolve distribution and will be available in the $NETSOLVE_ROOT/bin/$NETSOLVE_ARCH/NS_octave directory after the Octave interface is built. These scripts help in testing the Octave interface and also serve as examples for using the Octave interface.