Torrus Installation Instructions

Note to Debian users: if you have installed torrus using Debian packages, all the needed dependencies outlined below have been satisfied and installation tasks carried out by the Debian package management system. The information below is provided for information purposes only and may not apply to the Debian torrus installations. Please see the README.Debian file in /usr/share/doc/torrus-common for more details.

Required Software

• Operating System
  
Any UNIX-like operation system where the required components are working. The author's primary development platform is Cygwin <http://www.cygwin.com>, and the primary testing environments are Sun Solaris/SPARC and FreeBSD.

Note to VmWare users: as recommended by Gord Philpott, Linux kernel for guest machine needs to be compiled with the following feature enabled: Character devices ---> Enhanced Real Time Clock Support. For other guest operating systems (e.g. FreeBSD), in order to let the Torrus collector and monitor run properly, you need to add the following lines to your torrus-siteconfig.pl:

  $Torrus::Scheduler::maxSleepTime = 15;
  $Torrus::Scheduler::ignoreClockSkew = 1;

# Debian package is maintained by Jurij Smakov and is available at # <http://pkg-torrus.alioth.debian.org>.

• Perl
  
Perl version 5.8.1 or higher is required. Version 5.8.0 is not recommended because of memory access bugs and unpredictable behaviour. Perl interpreter should be in PATH environment variable when running ./configure, otherwise use

  ./configure PERL=/path/to/perl

• RRDtool
  
Round Robin Database Tool is available at: <http://ee-staff.ethz.ch/~oetiker/webtools/rrdtool/>. Both stable version 1.0.x and the development version 1.1.x are supported.

As of this writing, rrdtool version 1.1.x is still in its development stage, but it is quite ready for using in production environments. There's a small glitch in legend text alignment, and it will hopefully be fixed.

• libxml2
  
libxml2 is the XML parser library from Gnome project <http://www.gnome.org>, available as a standalone package. Versions 2.4.23 and above were tested.
• Berkeley DB version 4.2.52 or later
  
There are two distinct packages with slightly conflicting names: The Berkeley DB from Sleepycat Software <http://www.sleepycat.com>, and BerkeleyDB Perl module available from CPAN. We will always refer the latter as BerkeleyDB Perl module.

The Berkeley DB database engine is used for all data storage. Versions 4.2.52 and above are recommended. Older versions of Berkeley DB had problems with database integrity in concurrent usage. By default, it installs into /usr/local/BerkeleyDB.4.2/.

Unfortunately, the BerkeleyDB Perl module cannot find the include and library files in that directory. Thus either configure Berkeley DB with the option --prefix=/usr/local, or you need to edit config.in in BerkeleyDB Perl module distribution.

• HTTP Server
  
Torrus requires Apache HTTP server (<http://httpd.apache.org>) and mod_perl (<http://perl.apache.org>).

Apache version 1.3 and mod_perl version 1.0 are supported by ApacheHandler.

Apache version 2.0 and mod_perl 2.0 are supported by Apache2Handler. Minimum supported mod_perl 2.0 version is 2.0.0. The handler also requires libapreq2 library, see the note below.

See Torrus Web Interface Reference for details on mod_perl configuration.

• Perl Modules
  
The Perl modules required are listed below. They are all available at CPAN <http://www.cpan.org>. Some of them require other modules to be installed. You can install all required modules with the following command, executed from within the distribution directory:

  perl -I `pwd`/setup_tools -MCPAN -e 'install Bundle::Torrus'

• XML::LibXML
  
This is a Perl interface for libxml2, providing DOM standards compliant interface. It is recommended that you use version 1.54_3 or higher.
• BerkeleyDB perl module
  
Perl interface to the database software. Version 0.19 or higher is required.
• Template-Toolkit
  
This is a powerful set of tools for text template processing. Torrus uses it for HTML files output. See also the project homepage: <http://www.template-toolkit.org>
• Proc::Daemon
  
Daemon invocation routines.
• Net::SNMP
  
SNMP queries and traps interface. Version 4.0.3 or higher is required.
• URI::Escape
  
Escape and unescape unsafe characters
• Apache (mod_perl version 1.0)
  
Mod_perl is a Perl runtime environment integrated into Apache HTTP server.
• Apache::Session
  
User session tracking helper
• Date::Parse and Date::Format
  
Module for parsing the user input for date and time.

• libapreq2 Perl library
  
Apache 2.0 handler requires an additional Perl library, libapreq2. It is available from CPAN <http://search.cpan.org/>, and currently only in development version. You need to download and install it manually in order for Apache2Handler to work. Minimum version required: libapreq2-2.05.
• HTTP Browser requirements
  
The HTTP browser must be compatible with HTML 4.01 Strict and CSS2 specifications. The tests were made with the following browsers: IE 5.5, Opera 6.12B1/Solaris, Opera 7.03/Win2K, Netscape 7.02.

Recommended Software

• XML::XUpdate::LibXML by Petr Pajas
  
This Perl module implements XUpdate specification <http://www.xmldb.org/xupdate/>. See Torrus User Guide for details.
• XSH by Petr Pajas
  
Available at <http://xsh.sourceforge.net>. This is a shell wrapper for a set of utilities for XML extraction, browsing, and editing.
• libxslt from Gnome project
  
libxslt is the XSLT stranslation library from Gnome project <http://www.gnome.org>, available as a standalone package. It includes a binary executable, xsltproc. See Torrus User Guide for examples of its usage.

Installation Procedure

Create the group torrus and a user torrus. The user should be a member of this group.

Add your Apache daemon user to the group torrus. Other users that will run any of Torrus processes must be included in this group.

For example, in Solaris (you may need to specify the GID and UID numbers according to your local administration policy):

  groupadd torrus
  useradd -c 'Torrus Daemon' -d /usr/local/torrus -g torrus torrus
  usermod -G www,torrus www

Further installation process is mostly as usual:

  ./configure
  make
  make install

The default directory layout is described below. Should you need to change it, there is a number of variables and configuration options that you may use as ./configure arguments. See ./configure --help for details. Alternatively you can utilize the script ./setup_tools/configure_fhs, which is designed to provide an FHS compliant setup. The script is equivalent to executing

  ./configure \
   --prefix=/opt \
   --mandir=/opt/share/man \
   pkghome=/opt/torrus \
   sitedir=/etc/opt/torrus

Do not try to change any paths by supplying them as make variables, this would most probably break the installation. The only make variable that is supported is DESTDIR. It may be used for preparing the package for further distribution. For example, the following command would install all Torrus files as if /tmp/stage were the root of the filesystem:

  make DESTDIR=/tmp/stage install

The presence of prerequisite Perl modules is checked during the execution of ./configure. You can disable this by giving --enable-pkgonly option.

The installer sets up the site configuration files, but only if they don't already exist.

Plugin modules should be installed separately, after the Torrus software is successfully installed. For each plugin, unpack the plugin distribution archive to some directory, and execute

  torrus install_plugin <UNPACKED_PLUGIN_DIR>

A number of directories is set up by default under the path /var, and they must be writable by all Torrus processes, including the Apache web server.

You can control these directories' access rights by setting the following environment variables: var_user, var_group, var_mode, like follows:

  ./configure var_group=wwwrun

Default values for operating systems other than Cygwin are: var_user=torrus, var_group=torrus, var_mode=775. Setgid bit is set by default for these directories.

If available, you may place these directories on a fast media, for example, a robust disk array. Changing the varprefix variable would be generally enough, for example:

  ./configure varprefix=/vol1/torrus_var

Torrus directory layout

  /usr/share/
        Home directory for Torrus distribution files

  /usr/share/torrus/conf_defaults/
        torrus-config.pl and other configuration files

  /usr/share/torrus/bin/
        Command-line executables

  /usr/share/doc/torrus-common/
        POD and TXT documentation files

  /usr/share/doc/torrus-common/examples/
        Miscelaneous example files

  /usr/share/perl5/
        Perl libraries

  /usr/share/torrus/plugins/
        Plugins configuration

  /usr/share/torrus/scripts/
        Scripts

  /usr/share/torrus/sup/
        Supplementary files, DTDs, MIBs, color schemas, web plain files

  /usr/share/torrus/templates/
        Renderer output templates

  /usr/share/torrus/xmlconfig/
        Distrubution XML files

  /etc/torrus/
        Site configurable files

  /etc/torrus/conf/
        Place for torrus-siteconfig.pl and other siteconfigs

  /etc/torrus/conf/discovery/
        Devdiscover input files

  /etc/torrus/templates/
        User-defined Renderer output templates

  /etc/torrus/xmlconfig/
        User XML configuration files

  /usr/share/man
        Place for man pages. All articles have the prefix C<torrus_>

  /var/log/torrus/
        Daemon logfiles

  /var/run/torrus
        Daemon PID files

  /var/cache/torrus
        Renderer cache

  /var/lib/torrus/db
        Berkeley DB home

  /var/lib/torrus/session_data/lock
  /var/lib/torrus/session_data/store
        Web interface session files

  /var/lib/torrus/collector_rrd
        Recommended directory for RRD files generated by collectors

Note: RRFW used the path /var/snmpcollector as the default path for storing the RRD files. According to FHS specification, the recommended path for service data files is recommended to be a subdirectory of /srv. The path /var/lib/torrus/collector_rrd is used as default by torrus genddx utility, and you can always change it in the command line or in your DDX files.

Configuring Torrus

The datasources are configured with %Torrus::Global::treeConfig hash in torrus-siteconfig.pl.

In this hash, the keys give the tree names. The values for each tree name are pointers to hashes, with the following keys and values: xmlfiles points to an array of source XML file names; run points to a hash with the names of the daemons that would be automatically launched for the tree; desription gives a short line describing the tree contents.

Two additional arrays: @Torrus::Global::xmlAlwaysIncludeFirst and @Torrus::Global::xmlAlwaysIncludeLast give a list of source XML files that are included in every tree, in the beginning or in the end of the XML files list. By default, this array consists of two files: defaults.xml and site-global.xml. The second one is resided in the user-configurable directory, and you can use it to override any default settings.

Example:

  @Torrus::Global::xmlAlwaysIncludeFirst =
      ('defaults.xml', 'site-global.xml');

  %Torrus::Global::treeConfig = (
    'tree_A' => {
      'description' => 'The First Tree',
      'xmlfiles' => [qw(a1.xml a2.xml a3.xml)],
      'run' => { 'collector' => 1, 'monitor' => 1 } },
    'tree_B' => {
      'description' => 'The Second Tree',
      'xmlfiles' => ['b1.xml', 'b2.xml'],
      'run' => {} }
   );

XML files are read additively within each tree, in the order as they are listed. The XML compiler searches for the files in several directories, listed in @Torrus::Global::xmlDirs. By default, this list conssists of two paths, one for Torrus distribution files, and the other for user files.

Files generated by devdiscover usually contain include statements which add the vendor-specific XML files to the compilation.

Below follows a short description of some XML files that come with Torrus distribution. Only site-global.xml is installed in the directory for user-configurable files, all others are placed in the distribution directory.

• defaults.xml
  
Default parameters for the root of the datasources tree. Default view definitions. Note: this file is automatically overwritten by make install.
• site-global.xml
  
Parameters that you want to change or define for your site. This file will be compiled for every tree after defaults.xml, and this is the place to override the defaults. The file that is supplied with the Torrus distribution has some useful parameters that you may simply uncomment. Note: this file is never overwritten by make install.
• snmp-defs.xml
  
SNMP collector defaults. The file defines several templates used for collecting SNMP data. Do not change this file. You may redefine the needed parameters in your site-specific files and templates.
• vendor/<vendor>.<product>[.<subsystem>].xml
  
SNMP collector definitions and templates for various hardware vendors and products. devdiscover includes some of these files automatically in the configuration.
• generic/*.xml
  
SNMP collector definitions and templates for vendor-independent MIBs. Most files are named after corresponding RFC numbers.

In addition, the distribution package contains several example files.

For more details about XML configuration, see Torrus User Guide and Torrus XML Configuration Guide.

Site configuration options

In addition to %Torrus::Global::treeConfig, you may wish to set some other parameters in your site configuration file (torrus-siteconfig.pl).

See torrus-config.pl for the complete list of varaibes that you may override in your site config. Among them, most interesting are:

• $Torrus::Renderer::companyName
  
The text that you specify here will appear in the top left corner of all HTML pages.
• $Torrus::Renderer::companyURL
  
The company name text will be clickable with the URL specified in this variable.

Apache HTTP server configuration

Torrus web interface is designed to run under mod_perl environment (<http://perl.apache.org>).

See the Torrus Web Interface Reference document for detailed instructions on Apache configuration.

In short, typical Apache configuration for mod_perl version 1.0 would look like follows:

  Alias /torrus/plain "/usr/share/torrus/sup/webplain"
  PerlRequire "/usr/share/torrus/conf_defaults/webmux.pl"
  <Location /torrus>
    SetHandler perl-script
    PerlHandler Torrus::ApacheHandler
  </Location>
  <Location /torrus/plain/>
    SetHandler default-handler
    Options None
  </Location>

Access Control Lists

By default, Torrus web interface requires user authentication. You can disable this by changing $Torrus::ApacheHandler::authorizeUsers to zero in your torrus-siteconfig.pl.

ACLs are controlled by acledit utility. See Torrus Manual pages for detailed description. Example:

  torrus acledit --addgroup=staff --permit=DisplayTree --for='*'
  torrus acledit --adduser=jsmith --password=mysecretpassword \
    --cn="John Smith" --addtogroup=staff
  torrus acledit --addgroup=admin \
    --permit=DisplayTree --permit=DisplayAdmInfo --for='*'

Cron Job

In order to clean old HTTP session data, it is recommended to run cleanup script in a cron job, once per day:

 #min hour mday month wday    who     command
 5    3    *    *     *       root    /usr/share/torrus/bin/cleanup

Startup script

The Torrus distriubution provides a startup script which you can install in the init scripts directory (/etc/init.d on most Unixes or /usr/local/etc/rc.d/ on FreeBSD). The script torrus is created during the installation process in the subdirectory <init.d> of the distribution directory.

The init script reads some parameters from /usr/share/torrus/conf_defaults/initscript.conf, and /etc/torrus/conf/initscript.siteconf if the latter exists. By default, it makes the monitor daemons sleep for 20 minutes when they are launched simultaneously with collector daemons.

Upgrade Procedure

Follow these instructions when upgrading from previous Torrus release.

In the previous distribution directory, look up the config.log file. It contains the configure options that you used in previous installation.

Unpack the new release distribution.

Run ./configure with the same options you used before.

Stop the collector and monitor processes (usually by /etc/init.d/torrus stop).

Stop Apache process.

Run make install.

Start the collector and monitor processes.

Start Apache process.

Also it is recommended to re-compile your XML configuration. If you prefer not to do this, it's recommended that you clear the Web interface cache both in your browser and in Torrus:

  torrus clearcache

Planning the disk space

In a typical Torrus setup, there are two disk space consuming entities: the RRDtool data storage and the Berkeley DB database.

RRDtool data files (or RRD's) are used to store the values that are gathered by the collector daemons. These are the most intensively updated files, so the disk speed is important here. If possible, it is better to spread the RRD files across several physical disks.

Berkeley DB database is used to keep all the configuratiuon data for your datasource trees, and it also keeps some live status information. It's intensively updated during XML compilation only. During normal Torrus working cycle, there are only few updates, not very critical in time. The database is read quite intensively during collector initialization, but usually it's the CPU speed that causes more delay.

For a typical Torrus installation with standard RRD creation parameters, the rough estimation is as follows: the Berkeley DB database occupies from 10 to 13 KB per collector datasource, and the RRD storage takes approximately 80 to 85 KB per datasource.

For a medium-sized system, few hundred megabytes for the Berkeley DB database and several gigabytes for RRD storage would be sufficient. Larger systems require a thorough design and staging work. See Torrus Scalability Guide for more details.

Network performance tuning

In large installations, the SNMP collector experiences quite intensive input traffic bursts. Thus the UDP socket receive buffers should be adapted to sustain the load. By default, Torus sets the UDP receiving buffer size, SO_RCVBUF, to 131071 bytes. This should fit most of installations. However, it's useful to check the network statistics if there are any UDP buffer overflows. On most systems, the commands netstat -s -p udp or netstat -s should show you these counters. The maximum buffer size is usually limited by a system kernel variable, and can be increased if needed. See $Torrus::Collector::SNMP::RxBuffer and its comments in torrus-config.pl for more details.

Author

Copyright (c) 2002-2005 Stanislav Sinyagin <ssinyagin@yahoo.com>