Remstats Home


  About remstats
  architecture
  Release Notes
  FAQ
  Conventions
  Prerequisites
  Installation
  Configuration
  Configuration Tools
  Servers
  Collectors
      cisco-access-server-collector
      dbi-collector
      dns-collector
      error-collector
      log-collector
      nt-status-collector
      ntop-collector
      ping-collector
      port-collector
      program-collector
      remoteping-collector
      snmp-collector
      snmp-route-collector
      unix-status-collector
  Updater
  Monitors
  pagemakers
  run-remstats2
  CGIs
  troubleshooting
  do-traceroutes
  Miscellany
  Thank-you
  Index


  Live Data

  Remstats was written by
  Thomas Erskine at the
  CRC in Canada and now
  at SourceWorks.

[Top] [Prev] [Next]

log-collector - get stats from remote log-files

Usage:

Can't locate RRDs.pm in @INC (@INC contains: . /usr/lib/remstats/lib /usr/lib/perl5/ /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl) at ../log-collector line 36.
BEGIN failed--compilation aborted at ../log-collector line 36.

Description:

The log-collector gets data from remote log-server's. This way the whole log-file doesn't have to be transfered. The "protocol", if it deserves that name, is very simple. The collector sends a request, which looks like (you can type it in via telnet):

	LOGFILE /wherever/the/logfile.is
	varname	function pattern
	...
	GO

The directives are all in UPPERCASE. They are LOGFILE, GO, DEBUG and TEST. The LOGFILE directive tells the log-server which file to read. The GO directive starts the request. DEBUG causes some extra remote debugging output, and TEST makes the log-server operate in test mode. In test mode it doesn't update the last-read position for that log-file, so you won't lose any data when testing.

The other lines are telling the log-server what data to collect. The first "word" is the variable name to be returned. The next is the function to be applied (from count, sum, average, min, max, first, last) and lastof. The rest of the line is a perl-style regex.

Note1: The lastof function is different from the others, in several respects. First, it takes multiple regex's. Second, it saves the current value of the variable in a context file with the same name as the context file for the log-file with '-' and the variable-name appended. The way it works is to return the number of the pattern which was last matched (starting with 1). If none of the patterns have ever matched, it will return zero, until one of them matches.

Note2: Except for the count function, the regex must contain a (parenthesized) expression which will match a number, to which the function will be applied.

For example, the line:

	rootlogins count ROOT LOGIN

would return data for a variable called rootlogins. The value would be the count of the records in the specified logfile which had the string 'ROOT LOGIN' in them.

The pattern can be much more complicated, for example (from the httpdlog rrd):

	bytes sum \sHTTP/\d\.\d"\s+2\d\d\s+(\d+)

This looks through a standard web-server log-file and extracts the bytes transferred and adds them up to produce the total number of bytes transferred for successes in that sample period.

How to make RRDs that use the log-collector

It's easiest to explain by example. Look at the beginning of the rrd httpdlog, copied here:

	source  log
	step    300
	data    requests        GAUGE:600:0:U   count (GET|POST)
	data    success         GAUGE:600:0:U   count \sHTTP/\d\.\d"\s+2\d\d
	data    bytes           GAUGE:600:0:U   sum \sHTTP/\d\.\d"\s+2\d\d\s+(\d+)

To form the requests to be sent to the log-server, the log-collector takes the DS name, e.g. success, and the last part of the line after all the DS definition count \sHTTP/\d\.\d"\s+2\d\d, combines the two and sends:

	success count \sHTTP/\d\.\d"\s+2\d\d

Note that the pattern can include magic cookies as of remstats version 0.12.2.

The line in the host config-file must specify the log-file name, like:

	httpdlog /var/log/httpd/access_log

This permits the same code to get information from the same log format, no matter where it's stored on various machines. Since log-file locations seem to vary a lot by host, I put the log-file specification in the host config-file, rather than in the rrd config-file.


[Top] [Remstats] [SourceWorks] [RRDtool] [SourceForge]
Last updated Fri Jun 22 13:37:45 GMT 2007 by <terskine@users.sourceforge.net>.