TrilinosWrappers::MPI::Vector Class Reference
[TrilinosWrappersVector classes]

Inheritance diagram for TrilinosWrappers::MPI::Vector:
Inheritance graph
[legend]

List of all members.

Public Member Functions

 Vector ()
 Vector (const Epetra_Map &InputMap)
 Vector (const Vector &V)
 Vector (const Epetra_Map &InputMap, const VectorBase &v)
template<typename Number >
 Vector (const Epetra_Map &InputMap, const ::Vector< Number > &v)
void reinit (const Epetra_Map &input_map, const bool fast=false)
void reinit (const VectorBase &v, const bool fast=false, const bool allow_different_maps=false)
Vectoroperator= (const TrilinosScalar s)
Vectoroperator= (const Vector &V)
Vectoroperator= (const ::::TrilinosWrappers::Vector &V)
template<typename Number >
Vectoroperator= (const ::::Vector< Number > &v)
void import_nonlocal_data_for_fe (const ::TrilinosWrappers::SparseMatrix &matrix, const Vector &vector)

Private Attributes

Epetra_Map map

Detailed Description

This class implements a wrapper to use the Trilinos distributed vector class Epetra_FEVector. This class is derived from the TrilinosWrappers::VectorBase class and provides all functionality included there.

Note that Trilinos only guarantees that operations do what you expect if the function GlobalAssemble has been called after vector assembly in order to distribute the data. This is necessary since some processes might have accumulated data of elements that are not owned by themselves, but must be sent to the owning process. In order to avoid using the wrong data, you need to call Vector::compress() before you actually use the vectors.

Parallel communication model

The parallel functionality of Trilinos is built on top of the Message Passing Interface (MPI). MPI's communication model is built on collective communications: if one process wants something from another, that other process has to be willing to accept this communication. A process cannot query data from another process by calling a remote function, without that other process expecting such a transaction. The consequence is that most of the operations in the base class of this class have to be called collectively. For example, if you want to compute the l2 norm of a parallel vector, all processes across which this vector is shared have to call the l2_norm function. If you don't do this, but instead only call the l2_norm function on one process, then the following happens: This one process will call one of the collective MPI functions and wait for all the other processes to join in on this. Since the other processes don't call this function, you will either get a time-out on the first process, or, worse, by the time the next a callto a Trilinos function generates an MPI message on the other processes , you will get a cryptic message that only a subset of processes attempted a communication. These bugs can be very hard to figure out, unless you are well-acquainted with the communication model of MPI, and know which functions may generate MPI messages.

One particular case, where an MPI message may be generated unexpectedly is discussed below.

Accessing individual elements of a vector

Trilinos does allow read access to individual elements of a vector, but in the distributed case only to elements that are stored locally. We implement this through calls like d=vec(i). However, if you access an element outside the locally stored range, an exception is generated.

In contrast to read access, Trilinos (and the respective deal.II wrapper classes) allow to write (or add) to individual elements of vectors, even if they are stored on a different process. You can do this writing, for example, vec(i)=d or vec(i)+=d, or similar operations. There is one catch, however, that may lead to very confusing error messages: Trilinos requires application programs to call the compress() function when they switch from adding, to elements to writing to elements. The reasoning is that all processes might accumulate addition operations to elements, even if multiple processes write to the same elements. By the time we call compress() the next time, all these additions are executed. However, if one process adds to an element, and another overwrites to it, the order of execution would yield non-deterministic behavior if we don't make sure that a synchronisation with compress() happens in between.

In order to make sure these calls to compress() happen at the appropriate time, the deal.II wrappers keep a state variable that store which is the presently allowed operation: additions or writes. If it encounters an operation of the opposite kind, it calls compress() and flips the state. This can sometimes lead to very confusing behavior, in code that may for example look like this:

 * TrilinosWrappers::Vector vector;
 * // do some write operations on the vector 
 * for (unsigned int i=0; i<vector->size(); ++i) 
 *   vector(i) = i;
 *
 *                   // do some additions to vector elements, but
 *                   // only for some elements
 *   for (unsigned int i=0; i<vector->size(); ++i)
 *     if (some_condition(i) == true)
 *       vector(i) += 1;
 *
 *                   // do another collective operation
 *   const double norm = vector->l2_norm();
 * 

This code can run into trouble: by the time we see the first addition operation, we need to flush the overwrite buffers for the vector, and the deal.II library will do so by calling compress(). However, it will only do so for all processes that actually do an addition -- if the condition is never true for one of the processes, then this one will not get to the actual compress() call, whereas all the other ones do. This gets us into trouble, since all the other processes hang in the call to flush the write buffers, while the one other process advances to the call to compute the l2 norm. At this time, you will get an error that some operation was attempted by only a subset of processes. This behavior may seem surprising, unless you know that write/addition operations on single elements may trigger this behavior.

The problem described here may be avoided by placing additional calls to compress(), or making sure that all processes do the same type of operations at the same time, for example by placing zero additions if necessary.

See also:
Trilinos
Author:
Martin Kronbichler, Wolfgang Bangerth, 2008

Constructor & Destructor Documentation

TrilinosWrappers::MPI::Vector::Vector (  ) 

Default constructor that generates an empty (zero size) vector. The function reinit() will have to give the vector the correct size and distribution among processes in case of an MPI run.

TrilinosWrappers::MPI::Vector::Vector ( const Epetra_Map &  InputMap  ) 

This constructor takes an Epetra_Map that already knows how to distribute the individual components among the MPI processors. Since it also includes information about the size of the vector, this is all we need to generate a parallel vector.

TrilinosWrappers::MPI::Vector::Vector ( const Vector V  ) 

Copy constructor using the given vector.

TrilinosWrappers::MPI::Vector::Vector ( const Epetra_Map &  InputMap,
const VectorBase v 
) [explicit]

Copy constructor from the TrilinosWrappers vector class. Since a vector of this class does not necessarily need to be distributed among processes, the user needs to supply us with an Epetra_Map that sets the partitioning details.

template<typename Number >
TrilinosWrappers::MPI::Vector::Vector ( const Epetra_Map &  InputMap,
const ::Vector< Number > &  v 
) [inline, explicit]

Copy-constructor from deal.II vectors. Sets the dimension to that of the given vector, and copies all elements.


Member Function Documentation

void TrilinosWrappers::MPI::Vector::reinit ( const Epetra_Map &  input_map,
const bool  fast = false 
)

Reinit functionality. This function destroys the old vector content and generates a new one based on the input map.

void TrilinosWrappers::MPI::Vector::reinit ( const VectorBase v,
const bool  fast = false,
const bool  allow_different_maps = false 
)

Reinit functionality. This function sets the calling vector to the dimension and the parallel distribution of the input vector, but does not copy the elements in v. If fast is not true, the elements in the vector are initialized with zero, otherwise the content will be left unchanged and the user has to set all elements.

Reinit functionality. This class also has a third possible argument, allow_different_maps, that allows for an exchange of data between two equal-sized vectors (but being distributed differently among the processors). A trivial application of this function is to generate a replication of a whole vector on each machine, when the calling vector is build according to the localized vector class TrilinosWrappers::Vector, and v is a distributed vector. In this case, the variable fast needs to be set to false, since it does not make sense to exchange data between differently parallelized vectors without touching the elements.

Vector& TrilinosWrappers::MPI::Vector::operator= ( const TrilinosScalar  s  ) 

Set all components of the vector to the given number s. Simply pass this down to the base class, but we still need to declare this function to make the example given in the discussion about making the constructor explicit work.

Reimplemented from TrilinosWrappers::VectorBase.

Vector& TrilinosWrappers::MPI::Vector::operator= ( const Vector V  ) 

Copy the given vector. Resize the present vector if necessary. In this case, also the Epetra_Map that designs the parallel partitioning is taken from the input vector.

Reimplemented from TrilinosWrappers::VectorBase.

Vector& TrilinosWrappers::MPI::Vector::operator= ( const ::::TrilinosWrappers::Vector V  ) 

Copy operator from a given localized vector (present on all processes) in TrilinosWrappers format to the current distributed vector. This function assumes that the calling vector (left hand object) already is of the same size as the right hand side vector. Otherwise, an exception will be thrown.

template<typename Number >
Vector& TrilinosWrappers::MPI::Vector::operator= ( const ::::Vector< Number > &  v  )  [inline]

Another copy function. This one takes a deal.II vector and copies it into a TrilinosWrapper vector. Note that since we do not provide any Epetra_map that tells about the partitioning of the vector among the MPI processes, the size of the TrilinosWrapper vector has to be the same as the size of the input vector. In order to change the map, use the reinit(const Epetra_Map &input_map) function.

void TrilinosWrappers::MPI::Vector::import_nonlocal_data_for_fe ( const ::TrilinosWrappers::SparseMatrix matrix,
const Vector vector 
)

This reinit function is meant to be used for parallel calculations where some non-local data has to be used. The typical situation where one needs this function is the call of the FEValues<dim>::get_function_values function (or of some derivatives) in parallel. Since it is usually faster to retrieve the data in advance, this function can be called before the assembly forks out to the different processors. What this function does is the following: It takes the information in the columns of the given matrix and looks which data couples between the different processors. That data is then queried from the input vector. Note that you should not write to the resulting vector any more, since the some data can be stored several times on different processors, leading to unpredictable results. In particular, such a vector cannot be used for matrix-vector products as for example done during the solution of linear systems.


Member Data Documentation

Epetra_Map TrilinosWrappers::MPI::Vector::map [private]

The Epetra map is used to map (or rather, partition) vector data accross multiple processes. This is the communicator and data distribution object common to all Trilinos objects used by deal.II.


The documentation for this class was generated from the following file:

deal.II documentation generated on Mon Nov 23 22:58:41 2009 by doxygen 1.6.1