Zoltan User's Guide  |  Next  |  Previous

Using the Zoltan library

This section contains information needed to use the Zoltan library with applications:
System requirements.
Building the Zoltan library.
Testing the Zoltan library.
Reporting bugs in Zoltan.
Incorporating Zoltan into Applications
Building applications that use Zoltan.
Data types for global and local IDs.
C++ interface.
F90 interface.

System Requirements

Zoltan was designed to run on parallel computers and clusters of workstations. In order to build and use Zoltan, you will need: Zoltan has been tested on a variety of platforms, including Linux, Solaris, Irix, Mac OS X, and Sandia's ASC RedStorm machine. If you wish to use Zoltan on a non-Unix operating system, for example Windows NT or 2000, you will have to port Zoltan yourself.

Building the Zoltan Library

The Zoltan library is implemented in ANSI C and can be compiled with any ANSI C compiler. In Zoltan, there are two build environments currently supported: the native Zoltan build environment and an Autotools-based build environment used with the Trilinos framework. The native Zoltan build environment requires the user to specify paths to the desired compilers and libraries in a text file read by gmake. The Autotools-based environment can be used to build Zoltan only or to build Zoltan within Trilinos. User's can use either environment to build the Zoltan library, utilities and test drivers.

Using the Native Zoltan Build Environment

In the native Zoltan build environment, makefiles are distributed with the source code in directory zoltan/src; these makefiles require the GNU Make (gmake) utility. The top-level makefile defines targets for the Zoltan library, test driver programs in C, C++ and Fortran90, and two graphical utilities useful for visualization of geometric partitions. (The test drivers and utilities are primarily intended for use by developers.) This makefile need not be edited to build Zoltan. Instead, environment-specific definitions are specified in the configuration file, zoltan/src/Utilities/Config/Config.<platform>, where <platform> specifies the particular platform for which Zoltan is being built. Paths to compilers, include files, and libraries are defined in this file and are then read by the top-level makefile. Examples of configuration files for Solaris, Sandia's ASC RedStorm computer, SGI workstations, Macs running OS X, and PCs running Linux are included in the zoltan/src/Utilities/Config subdirectory. A well-commented version of the configuration file, zoltan/src/Utilities/Config/Config.generic, is also included; this file can be used as a template for new environment-specific files. The variables in these files should be edited to reflect the new system's environment.

The command for building Zoltan is shown below:

cd zoltan/src
gmake [options] zoltan
where the options that may be specified are listed below.
    ZOLTAN_ARCH=<platform> Specify the target architecture for the Zoltan library. A corresponding file, zoltan/src/Utilities/Config/Config.<platform>, containing environment definitions for <platform>, must be created in the zoltan/src/Utilities/Config directory.
    YES_FORTRAN=1 Include Fortran support in the Zoltan library.  By default, the Zoltan library is built without the interface that allows use from Fortran applications.  If this option is specified, the Fortran interface is compiled and included in the library.  Use of this option requires that a Fortran 90 (or 95, or later) compiler is available. Currently, the Fortran 90 interface can be built only with the native Zoltan build environment.
As an alternative to typing the options on the gmake command line, they may be set as environment variables; e.g., if you are using a C-shell (csh or tcsh), type
setenv ZOLTAN_ARCH <platform>
or if you are using a Bourne-type shell (e.g., sh or bash), type
ZOLTAN_ARCH = <platform>; export ZOLTAN_ARCH
The resulting library libzoltan.a, object files, and dependency files are stored in the directory Obj_<platform>.

Using Autotools to Build Zoltan

To build Zoltan using Autotools, users must first run the auto-configuration tool and then compile the library. The configuration tool allows paths to third-party libraries such as ParMETIS, PT-Scotch and PaToH to be specified through arguments to the configure command. These options can be seen with the following command issued in the top-level zoltan directory:
./configure --help
IMPORTANT! To comply with Trilinos practices, the default autotools configuration produces a serial build of Zoltan -- that is, a version of Zoltan that runs on only one processor. In most cases, Zoltan users prefer a parallel build that runs on multiple processors; users must link with MPI and may choose to compile with the MPI compilers. To enable the parallel build, users must use the --enable-mpi option; they can also require that MPI compilers (e.g., mpicc, mpic++) be used by specifying the --with-mpi-compiler option.

The following script is an example of configuration and build commands using Autotools. It specifies that Zoltan should be built with both the ParMETIS and PT-Scotch interfaces. Paths to both ParMETIS and PT-Scotch are given. The prefix option states where Zoltan should be installed; in this example, Zoltan's include files will be installed in /homes/username/zoltan_v3.1/OBJ_LINUX/include, and the libraries will be installed in /homes/username/zoltan_v3.1/OBJ_LINUX/lib. This examples assumes the script is run from /homes/username/zoltan_v3.1/OBJ_LINUX.

#
../configure \
--prefix=/homes/username/zoltan_v3.1/OBJ_LINUX \
--enable-mpi --with-mpi-compilers \
--enable-zoltan \
--with-gnumake \
--with-scotch \
--with-scotch-incdir="/Net/local/proj/zoltan/arch/all/src/Scotch5" \
--with-scotch-libdir="/Net/local/proj/zoltan/arch/linux64/lib/openmpi/Scotch5" \
--with-parmetis \
--with-parmetis-incdir="/Net/local/proj/zoltan/arch/all/src/ParMETIS3" \
--with-parmetis-libdir="/Net/local/proj/zoltan/arch/linux64/lib/openmpi/ParMETIS3"
make everything
make install

More examples are in the directory zoltan/SampleConfigurationScripts.

Note: Building Zoltan's Fortran 90 interface is not yet available with Autotools; use the native Zoltan build environment to build the Fortran 90 interface.


Testing the Zoltan Library

The examples directory contains simple C and C++ examples which use the Zoltan library. The makefile in this directory has three targets:
gmake ZOLTAN_ARCH=<platform> C_Examples
This builds simple C language examples that use the Zoltan library to perform load balancing.
gmake ZOLTAN_ARCH=<platform> CPP_Examples
This builds simple C++ language examples that use the Zoltan library to perform load balancing. To build C++ applications, define CPPC to point to your C++ compiler in the Config.<platform> file.
gmake ZOLTAN_ARCH=<platform> all
Build both C and C++ examples. Don't forget to define CPPC in your Config.<platform> file.
These examples are built automatically when the Autotools build environment is used.

Some of these examples make use of a small library of support routines found in the examples/lib directory. These routines create simple test meshes of varying sizes, perform error checking across the parallel application, and define Zoltan call backs.

The "right" answer for these tests depends on the number of processes with which you run the tests. In general, if they compile successfully, run quickly (in seconds), and produce reasonable looking output, then you have been successful in building Zoltan.


Reporting Bugs in Zoltan

Zoltan uses Bugzilla to collect bug reports. Please read the instructions for reporting bugs through the Zoltan Bugzilla database.


Incorporating Zoltan into Applications

Incorporating Zoltan into applications requires three basic steps: The set of query functions needed by an application depends on the particular tools (e.g., partitioning, ordering) used and on the algorithms selected within the tools. Not all query functions are needed by every application. See documentation on tools and algorithms to determine which query functions are needed.

Building Applications that use Zoltan

The C library interface is described in the include file include/zoltan.h; this file should be included in all C application source files that call Zoltan library routines.

The C++ interface to Zoltan is implemented in header files which define classes that wrap the Zoltan C library. The file include/zoltan_cpp.h defines the Zoltan class which encapsulates a load balancing data structure and the Zoltan load balancing functions which operate upon it. Include this header file instead in your C++ application. Note that C++ applications should call the C function Zoltan_Initialize before creating a Zoltan object.

Fortran applications must USE module zoltan and specify Zoltan/Obj_<platform> as a directory to be searched for module information files.

The C, C++ or Fortran application should then be linked with the Zoltan library (built with Fortran support in the Fortran case) and its utility libraries by including

-lzoltan
in the linking command for the application. Communication within Zoltan is performed through MPI, so appropriate MPI libraries must be linked with the application. Third-party libraries, such as ParMETIS, PT-Scotch and PaToH, must be also be linked with the application if they were included in compilation of the Zoltan library.  (A courtesy copy of ParMETIS is included with the Zoltan distribution. PaToH must be obtained directly from http://bmi.osu.edu/~umit/software.html. PT-Scotch is available at http://www.labri.fr/perso/pelegrin/scotch/.)
 

For applications that used versions of Zoltan before Zoltan v.1.3, only minor updates to the application build process are needed; see the section on backward compatibility of Zoltan.


Data Types for Object IDs

Application query functions and application callable library functions use global and local identifiers (IDs) for objects. All objects to be used in load balancing must have unique global IDs. Zoltan stores an ID as an array of unsigned integers. The number of entries in these arrays can be set using the NUM_GID_ENTRIES and NUM_LID_ENTRIES parameters; by default, one unsigned integer represents an ID. Applications may use whatever format is most convenient to store their IDs; the IDs can then be converted to and from Zoltan's ID format in the application-registered query functions.

The following type definitions are defined in include/zoltan_types.h; they can be used by an application for memory allocation, MPI communication, and as arguments to load-balancing interface functions and application-registered query functions.

typedef unsigned int ZOLTAN_ID_TYPE;
typedef ZOLTAN_ID_TYPE *ZOLTAN_ID_PTR;
#define ZOLTAN_ID_MPI_TYPE MPI_UNSIGNED
In the Fortran interface, IDs are passed as arrays of integers since unsigned integers are not supported in Fortran. See the description of the Fortran interface for more details.

The local IDs passed to Zoltan are not used by the library; they are provided for the convenience of the application and can contain any information desired by the application. For instance, local array indices for objects may be passed as local IDs, enabling direct access to object data in the query function routines. See the application-registered query functions for more details. The source code distribution contains an example application zdrive in which global IDs are integers and local IDs are local array indices. One may choose not to use local ids at all, in which case NUM_LID_ENTRIES may be set to zero.

Some Zoltan routines (e.g., Zoltan_LB_Partition and Zoltan_Invert_Lists) allocate arrays of type ZOLTAN_ID_PTR and return them to the application. Others (e.g., Zoltan_Order and Zoltan_DD_Find) require the application to allocate memory for IDs. Memory for IDs can be allocated as follows:

ZOLTAN_ID_PTR gids;
int num_gids, int num_gid_entries;
gids = (ZOLTAN_ID_PTR) ZOLTAN_MALLOC(num_gids * num_gid_entries * sizeof(ZOLTAN_ID_TYPE);
The system call malloc may be used instead of ZOLTAN_MALLOC.
[Table of Contents  | Next:  C++ Interface  |  Previous: Introduction]