Previous Next Contents

3. NoSQL Operators

The NoSQL system comprises a set of programs called Operators.

Each operator is a separate program module that performs a unique function on the data. Operators can be grouped into data movers, report generators, and utilities.

The data movers are operators that extract or rearrange the data in some way. They each read an rdbtable via STDIN and write a rdbtable via STDOUT, and so are frequently connected using the UNIX pipe function to form a larger task. Each operator in such a "pipeline" style of operation gets its input from the output of the previous operator in the "pipeline". The data movers include:

nsq-col

Picks columns by name, outputs columns in listed order.

nsq-compute

Computes an arbitrary expression using column names.

nsq-headchg

Generates and replaces (or removes) the header of an rdbtable.

nsq-header

Extracts the header from an rdbtable.

nsq-join

Natural or "Master/Detail" join of two rdbtables.

nsq-listtotable

Converts files from /rdb list to /rdb table format.

nsq-merge

Merges two like rdbtables.

nsq-n2r

Converts tables from NoSQL to /rdb format.

nsq-r2n

Converts tables from /rdb to NoSQL format.

nsq-row

Selects rows based on arbitrary expressions.

nsq-search

Selects rows based on a multi-column key of a sorted or indexed rdbtable.

nsq-sort

Sorts a datafile by one or more columns.

nsq-subtot

Lists subtotals of specified columns.

nsq-tabletolist

Converts files from /rdb table to /rdb list format.

nsq-tee

Safely [over]writes an rdbtable.

nsq-uniq

Makes an rdbtable unique on specified columns.

The report generators each read an rdbtable via STDIN and produce a report on STDOUT, so when they are in a "pipeline" of operators they will be the operator at the end. The report generators are:

nsq-pr

Quick and easy printing of output formatted from information in the header.

nsq-report

Best form of output, with definable format.

nsq-summ

Summary/Statistical information about data values in an rdbtable.

nsq-valid

Verifies the structure of an rdbtable.

The utilities are used for manipulating the structure and content of rdbtables and are generally used as separate tasks,i.e. they do not read STDIN. The utilities are:

nsq

NoSQL interactive terminal interface.

nsq-ed

Uses an editor to allow modifications to an rdbtable.

nsq-index

Generate rdbtable index files to be used by 'nsq-search'.

nsq-lock

Trivial table locking program, for serializing access to rdbtables.

nsq-repair

Attempts to repair candidate NoSQL datafiles.

nsq-updseq

Updates/Creates a unique record ID on each row of an rdbtable. The field must be the first column in the table.

All operators take a '-h[elp]' option to show details of operation online. Following is a section for each operator, in alphabetic order.

3.1 nsq

Usage: nsq [options] [rdbtable]

Interactive analysis tool for NoSQL data tables (rdbtables). This is especially useful for rdbtables that were made from spreadsheet data.

Uses nine NoSQL modules: nsq-col, nsq-ed, nsq-pr, nsq-report, nsq-row, nsq-sort, nsq-t2l, nsq-summ, nsq-valid.

Specific module options are available using the '-help' option of individual modules. This utility uses the PAGER environment variable.

3.2 nsq-col

Usage: nsq-col [options] list

Selects ("projects") columns by name (and order) and outputs an rdbtable with these columns. Can effectively select, order, add, delete, or duplicate columns.

The value 'list' is normally a list of column names. If 'list' contains a triplicate of the form '-c NAME NEW' then column name 'NAME' will be changed to 'NEW'. If 'list' contains a triplicate of the form '-a NAME DEFN' then a new (null) column is added, at that point in the list of column names, with name 'NAME' and definition 'DEFN'.

This NoSQL operator reads an rdbtable from STDIN and writes an rdbtable to STDOUT. Options may be abbreviated.

As an example using the sample rdbtable from the DATA section (named sample), to select columns named 'NAME' and 'COUNT' the command would be:

      nsq-col  NAME  COUNT  <  sample
    

To select all columns except column 'NAME' the command would be:

      nsq-col  -v  NAME  <  sample
    

To add a new column named 'LENGTH' with a size of 10 the command would be:

      nsq-col  -v  -a  LENGTH  10  <  sample
    

Note that to include documentation with the new column definition the command would be:

      nsq-col  -v  -a  LENGTH  '10 length in meters'  <  sample
    

The '10 length in meters' must be quoted so that it will be treated as a single token.

3.3 nsq-compute

Usage: nsq-compute [options] [statements]

Computes values for data fields based on arbitrary statements using column names. Any characters that are special to the UNIX shell must be quoted.

Comparison operators may be of the form: gt, ge, lt, le, eq, ne. For example 'NAME eq Hobbs'. Logical constructors 'or' and 'and' may be used; as well as 'null' to indicate an empty data value. The supplied statements may be essentially any valid PERL statements.

All of the Comparison operators and Logical constructors are reserved and should not be used as column names (they are all lower case and four characters or less).

This operator reads a rdbtable via STDIN and writes a rdbtable via STDOUT. Options may be abbreviated.

If a file is used to contain the statements any line in the file that starts with a sharp sign (#) is treated as a comment and ignored. Also if there is a sharp sign preceded by a space character anywhere on the line the rest of the line is also treated as a comment.

Since column names and reserved words are parsed by the program, do not put the entire expression in a single pair of quotes as that will prevent the parsing. Also note that column names and reserved words need to be surrounded by blank spaces if they are not individually quoted. For example either form below is fine:

      nsq-row   NAME    eq   "L Brown"  <  sample

      nsq-row  "NAME"  "eq"  "L Brown"  <  sample
        

but do not use this form:

      nsq-row  "NAME  eq  L Brown"  <  sample
    

Example rdbtable (named cfile):

      name    count   type    amt
      6       5N      4       5N
      Bush    3       A       133
      Hansen  39      A       23
      Newton  8       E       8
      Hobbs   42      B       144
      Hart    2       C       55
      Jones   4       B       244
      Smith   5       D       1111
    

The command:

      nsq-compute  count  +=  100  if  type  lt  D  <  cfile  |  nsq-pr
    

gives the output:

      name    count  type    amt
      ------  -----  ----  -----
      Bush      103  A       133
      Hansen    139  A        23
      Newton      8  E         8
      Hobbs     142  B       144
      Hart      102  C        55
      Jones     104  B       244
      Smith       5  D      1111
    

Example file of commands named 'XXX':

      if( type eq A ){
          name = NEW ;
          amt = count * 2 ;
          type = 'AAA' ;
      }
      else{
          name = OLD ;
          amt = count + 1000 ;
          type = 'ZZZ' ;
      }
    

Output from command:

      nsq-compute  -fXXX  <  cfile  |  nsq-pr
    

would be:

      name    count  type    amt
      ------  -----  ----  -----
      NEW         3  AAA       6
      NEW        39  AAA      78
      OLD         8  ZZZ    1008
      OLD        42  ZZZ    1042
      OLD         2  ZZZ    1002
      OLD         4  ZZZ    1004
      OLD         5  ZZZ    1005
    

3.4 nsq-ed

Usage: nsq-ed [options] rdbtable [col_spec] [line_spec] [pat_spec]

This utility calls an editor to allow the editing of selected lines and/or columns of (or the entire) rdbtable. Options may be abbreviated.

A "col_spec" is a list of column names.

A "line_spec" is a list of line numbers, of increasing value, optionally separated by a dash to specify a range, e.g. "10-20". The form "N-" means from line N to end of file. The header is always included, so do not specify lines 1 or 2 (except as the first part of a larger group, e.g. "1-10").

A "pat_spec" is a single pattern (of the form: /pat/ ) optionally followed by one or more column names, and may be preceded with the reserved word 'ne' to negate the meaning (e.g. the pattern should NOT match).

The order of "col_spec", "line_spec", and "pat_spec" is significant only to the extent that "col_spec" must precede "pat_spec" in the command line if both are given.

If none of "col_spec", "line_spec", or "pat_spec" are given then the entire rdbtable will be edited. If one or more of the three above options are given then the selected subset of the rdbtable will be edited. The option "col_spec" identifies which columns of the rdbtable are to be edited, and options "line_spec" and "pat_spec" determine which lines will be selected for editing, either by direct reference ("line_spec" given "col_spec" not given) or by pattern matching ("col_spec" given "line_spec" not given). If both "line_spec" and "pat_spec" are given then only lines within the bounds of "line_spec" will be considered for selection by pattern matching.

If "pat_spec" does not include column names then the pattern (any PERL regular expression) is matched against each entire row; a row is selected if there is a match anywhere in the row. If column names are included the pattern is matched against only the specified columns. In this case a row is selected if a match is found in any specified column. If the "ne" option precedes the "pat_spec" without column names then an entire row is selected if the pattern does not match anywhere in the row, and if column names are given then the row is selected if the pattern does not match in any specified column.

The form of the file to be edited is either "column" with visible column delimiters (the default) or "list" format where the column names are on the left and the data is on the right. The default editor is specified by the environment variable EDITOR if set, otherwise the editor 'vi' is used.

In either form of editing the delimiter is a "pipe" symbol (|). Care should be taken when editing not to use any "pipe" symbols in the data, or to delete any existing pipe symbols in the file. Also, in the case of "list" form, one or more blank lines must separate each record.

The rbdtable may be an existing file, or it may be automatically checked out from RCS. In the latter case it will be checked back into RCS after the editing is complete. The default action is that if the rdbtable does not exist an attempt will be made to find the rdbtable under RCS (the '-RCS' option may be used to force the use of an RCS file).

Afterward, except in the RCS case, the original contents of the rdbtable will be left in a file of the same name preceded with a comma, e.g. "sample" will be ",sample".

Uses NoSQL operators: nsq-col, nsq-pr, nsq-mktbl, nsq-t2l, nsq-l2t.

WARNING: If line_spec is given the number of columns must not be changed by editing, or if col_spec and/or "pat_spec" is given the number of lines must not be changed by editing, otherwise the results may be unpredictable.

An example command to edit the rdbtable (named sample) from the DATA section would be:

          nsq-ed  sample
    

which would edit the entire rdbtable. The file as it is ready to edit is shown in Table 3. The pipe character '|' must not be removed during the editing process, although it may be moved left or right if necessary and the spaces around the pipe character may be deleted if desired.

This form of editing is fine if the rdbtable is not large. If it is large then editing only those parts that need changes is faster and less error prone. To edit only columns 'NAME', 'COUNT', and 'AMT', the command would be:

      nsq-ed  sample  NAME  COUNT  AMT
    

To edit only lines five thru seven the command would be:

      nsq-ed  sample  5-7
    
                            Table 3

           RDBTABLE (SAMPLE) READY TO EDIT, COLUMN FORM

      NAME   | COUNT | TYP  |   AMT | OTHER    |    RIGHT
      6      |    5N | 4    |    5N | 8        |       8>
      Bush   |    44 | A    |   133 | Another  |     This
      Hansen |    44 | A    |    23 | One      |       Is
      Jones  |    77 | X    |    77 | Here     |       On
      Perry  |    77 | B    |   244 | And      |      The
      Hart   |    77 | D    |  1111 | So       |    Right
      Holmes |    65 | D    |  1111 | On       |     Edge
    

To edit only lines five thru seven of only columns 'NAME', 'COUNT', and 'AMT' the command would be:

      nsq-ed  sample  NAME  COUNT  AMT  5-7
    

and the file to edit would look like:

      ..>>> 1 2 CONTROL LINE, DO NOT TOUCH <<<
      NAME   | COUNT |   AMT
      6      |    5N |    5N
      ..>>> 5 3 CONTROL LINE, DO NOT TOUCH <<<
      Jones  |    77 |    77
      Perry  |    77 |   244
      Hart   |    77 |  1111
    

Note that whenever a line_spec is given, control lines (starting with '..>>>' are inserted into the file to edit. They must not be modified during the editing process. They are used to reconstruct the rdbtable after editing.

If the rdbtable has data fields that are long, i.e. longer than convenient to edit in the column form shown above, the 'list' form is the preferred method. The usage of line_spec and col_spec are unchanged but the form of the file to edit is different. For example consider an rdbtable (named sample3) which is shown in Table 4, where the TAB characters are represented by '<T>and the newline characters are represented by <N>This small rdbtable looks very incoherent in raw form, and a file of any real size with long data fields is even more so. The command to edit the file sample3 in 'list' form would be:

                           Table 4
  
               RDBTABLE (SAMPLE3) ACTUAL CONTENT
  
name<T>datatype<T>agencysrc<T>dbms<T>contact<T>contents<T>notes<N>
46<T>15<T>60<T>15<T>21<T>530<T>600<N>
ACAS (Air Combat Assessment)<T>BDA<T>Bigplace AFB<T>File<T>Starr<T>Air
Combat Assessment BDA data.  Duplicates data under ACAS (Air Combat
Assessment) BDA Sorties, diskettes nr 1,2,3.<T>On two 3.5 inch
diskettes.<N>
ACAS (Air Combat Assessment) BDA Sorties<T>BDA<T>Sawyer AFB<T>File<T>
Hobbs/Emerson<T>85 files, 2 per day containing 12 and 24 hour reports.
This data is different from that under ACAS (Air Combat Assessment) BDA
Data, from diskettes 1,2,3.<T>Received 5/6/91.<N>
ATO (Air Tasking Orders) Original<T>ATO<T>HQ USAF, Universal AFB<T>
File<T>Marshall<T>Original ATO messages.  Both sets are incomplete.<T>
To be joined into single file and edited.  Missing sections not yet
ordered.  May be parsed completely, or only for key comments. Much data
to be processed.<N>
ABC Original<T>ABC<T>HQ USAF, Universal AFB<T>File<T>Marshall<T>Original
ATO messages.  Both sets are incomplete.<T>To be joined into single file
and edited.  Missing sections not yet ordered. May be parsed completely,
or only for key comments. Much data to be processed.<N>
    

The command to edit the file sample3 in 'list' form would be:

      nsq-ed  -list  sample3
    

which would produce a file to edit as shown in Table 5. Note that each section holds information relating to one row in the rdbtable and that the first section holds information relating to the header of the rdbtable. Also note that each section is separated by a blank line (it could be any number of blank lines).

Each row in a section relates to a single data value. The pipe character '|' must not be removed during the editing process, although it may be moved left or right if necessary. Only one pipe character is to be in the information relating to one data value, although that information may be physically on more than one line in the section if the data value is long. The spaces on both sides of the pipe character as well as the spaces around the column names are only for readability; they may be moved or even deleted if desired.

                           Table 5

           RDBTABLE (SAMPLE3) READY TO EDIT, LIST FORM

      name | 46
  datatype | 15
 agencysrc | 60
      dbms | 15
   contact | 21
  contents | 530
     notes | 600

      name | ACAS (Air Combat Assessment)
  datatype | BDA
 agencysrc | Bigplace AFB
      dbms | File
   contact | Starr
  contents | Air Combat Assessment BDA data.  Duplicates data under ACAS
             (Air Combat Assessment) BDA Sorties, diskettes nr 1,2,3.
     notes | On two 3.5 inch diskettes.

      name | ACAS (Air Combat Assessment) BDA Sorties
  datatype | BDA
 agencysrc | Sawyer AFB
      dbms | File
   contact | Hobbs/Emerson
  contents | 85 files, 2 per day containing 12 and 24 hour reports. This
             data is different from that under ACAS (Air Combat
             Assessment) BDA Data, from diskettes 1,2,3.
     notes | Received 5/6/91.
    
      name | ATO (Air Tasking Orders) Original
  datatype | ATO
 agencysrc | HQ USAF, Universal AFB
      dbms | File
   contact | Marshall
  contents | Original ATO messages.  Both sets are incomplete.
     notes | To be joined into single file and edited.  Missing sections
             not yet ordered.  May be parsed completely, or only for key
             comments. Much data to be processed.
    
      name | ABC Original
  datatype | ABC
 agencysrc | HQ USAF, Universal AFB
      dbms | File
   contact | Marshall
  contents | Original ATO messages.  Both sets are incomplete.
     notes | To be joined into single file and edited.  Missing sections
             not yet ordered.  May be parsed completely, or only for key
             comments.  Much data to be processed.
    

The advantage of this form of edit file is that even with very large data values most, if not all, of the information from each row of an rdbtable will be visible on the screen at once.

3.5 nsq-headchg

Usage: nsq-headchg [options] file.tpl

Replaces the header (first two rows) of an rdbtable with a header generated from information in the template file 'file.tpl'. Options are available to add, copy, or delete the header, or to generate a template file from an existing rdbtable.

Each line of the Template file contains info about a column, in order. The lines contain: (optional) index number (starting at 0 or 1), column name, definition, and (optional) comments or documentation, white space separated. If column name contains spaces it must be enclosed in double quotes. Names containing space characters are not recommended, however, as it is generally troublesome and error prone. A good substitute is the underscore character (_).

Lines that start with a sharp character '#' are skipped, as are blank lines. To start a column name with a sharp character '#' the name must be enclosed in double quotes. (but this is not recommended).

The number of columns in the header is normally reported on STDERR. This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated. This operator uses the NoSQL operator: nsq-valid.

As an example, to generate a template file named 'new.tpl' from the rdbtable (named sample) from the DATA section, the command would be:

      nsq-headchg  -templ  <  sample  >  new.tpl
    

The contents of file 'new.tpl' would then be:

     0                 NAME  6
     1                COUNT  5N
     2                  TYP  4
     3                  AMT  5N
     4                OTHER  8
     5                RIGHT  8>
    

To change the header of rdbtable 'sample', the procedure is to edit the file 'new.tpl', and then run 'nsq-headchg' using the modified file. For example, to change the names so that only the first letters are upper case and to make column 'OTHER' numeric, edit file 'new.tpl' so it looks like the following:

      0                 Name  6    All names are first letter upper case.
      1                Count  5N
      2                  Typ  4
      3                  Amt  5N
      4                Other  8N   Now numeric.
      5                Right  8>
    

Note the index in the zeroth column and the documentation in the fourth column, both of which are optional, but recommended. The command to change the header of rdbtable 'sample' and make a new rdbtable called 'new.sample' would be:

      nsq-headchg  new.tpl  <  sample  >  new.sample
    

3.6 nsq-header

Usage: nsq-header [options]

Section yet to be written. See 'nsq-header -help' in the meantime.

3.7 nsq-index

Usage:

nsq-index [options] rdbtbl column ...

or

nsq-index [options] -update [ index_file ... ]

The first form of usage of this utility generates an index file for the column(s) given, that refers to the specified rdbtbl. An index file is actually another (smaller) rdbtable containing only the column(s) given plus a column for index information. An index file can be used by the operator 'nsq-search' to quickly locate rows of data in the referenced rdbtable.

Index files are named by appending an 'x' and the column name(s) (seperated by a dots) to the base name of the rdbtable it refers to. For example an index file for the rdbtable "area.rdb" on column "strip" would be "area.x.strip". An index file that was also on column "depth" would be "area.x.strip.depth".

The second form of usage of this utility updates the index file(s) given. If no files are given all index files in the current directory are updated. An update of an index file is necessary when the rdbtable it refers to has been modified.

This utility writes or rewrites rdbtables with defined names in the current directory. Options may be abbreviated.

3.8 nsq-join

Usage: nsq-join [options] col.name[=col.name_2] rdbtable_2 < rdbtable_1

where rdbtable_2 is the "secondary" rdbtable, while rdbtable_1 is the the "primary" one.

Does a join of two rdbtables on the column(s) specified. The default is a "natural" join, with optional "Master/Detail" or cartesian (cross-product) type joins. Options may be abbreviated.

The Table from STDIN is the master. A natural join produces a new rdbtable that contains only rows from the input rdbtables that match on the specified columns (key columns). A master-detail join produces a new rdbtable that contains all rows from the master rdbtable and those rows from the secondary rdbtable that match. A cartesian join produces an rdbtable that contains all rows of both input rdbtables.

Each item in the list of column(s) may specify column names that are different in the two rdbtables, i.e. '=column_2', if given, refers to a name in rdbtable_2 that corresponds to 'column' in rdbtable_1. If '=column_2' is not given it means that the corresponding column name in both rdbtables is the same.

If different column names are specified, the name of the join columns in the output rdbtable will be from rdbtable_1.

Note that the two rdbtables must be sorted on the columns specified in order for a join operation to function correctly.

The order of columns in the output rdbtable will be: first the join columns, then the other columns from rdbtable_1, then the other columns from rdbtable_2.

This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT.

If we have the rdbtable (named samplej) here:

      name     nr     typ     amt
      6        2      4       4
      Bush     1      A       133
      Bush     2      A       134
      Hansen   3      A       143
      Hobbs    4      B       144
      Hobbs    5      B       144
      Jones    6      C       155
      Perry    7      D       244
      Perry    8      D       311
    

and the rdbtable (named samplej2) here:

      name     cnt    typ     amt
      6        5N     4       5N
      Hobbs    41     A       141
      Hobbs    42     BB      142
      Hobbs    51     BB      144
      Hobbs    43     CC      143
        

then the command to do a natural join of samplej and samplej2 on column name is:

      nsq-join  name  samplej2  <  samplej
        

and the result is shown in Table 6.

                          Table 6

         NATURAL JOIN OF RDBTABLES SAMPLEJ AND SAMPLEJ2

      name    nr      typ     amt     cnt     typ     amt
      6       2       4       4       5N      4       5N
      Hobbs   4       B       144     41      A       141
      Hobbs   4       B       144     42      BB      142
      Hobbs   4       B       144     51      BB      144
      Hobbs   4       B       144     43      CC      143
      Hobbs   5       B       144     41      A       141
      Hobbs   5       B       144     42      BB      142
      Hobbs   5       B       144     51      BB      144
      Hobbs   5       B       144     43      CC      143
        

The command to do a "masterdetail" join of the same two rdbtables on column name is:

      nsq-join  -md  name  samplej2  <  samplej
        

and the result is shown in Table 7.

                           Table 7

       MASTER-DETAIL JOIN OF RDBTABLES SAMPLEJ AND SAMPLEJ2

      name    nr      typ     amt     cnt     typ     amt
      6       2       4       4       5N      4       5N
      Bush    1       A       133
      Bush    2       A       134
      Hansen  3       A       143
      Hobbs   4       B       144     41      A       141
      Hobbs   4       B       144     42      BB      142
      Hobbs   4       B       144     51      BB      144
      Hobbs   4       B       144     43      CC      143
      Hobbs   5       B       144     41      A       141
      Hobbs   5       B       144     42      BB      142
      Hobbs   5       B       144     51      BB      144
      Hobbs   5       B       144     43      CC      143
      Jones   6       C       155
      Perry   7       D       244
      Perry   8       D       311
        

3.9 nsq-l2t

Usage: nsq-l2t [options]

Converts a file in "list" format to an rdbtable. Long data fields may be folded. This operator is mainly used by other operators. Options may be abbreviated.

3.10 nsq-listtotable

Usage: nsq-listtotable [options]

Section yet to be written. See 'nsq-listtotable -help' in the meantime.

3.11 nsq-lock

Usage: nsq-lock [options]

Section yet to be written. See 'nsq-lock -help' in the meantime.

3.12 nsq-merge

Usage: nsq-merge [options] < old_table column ... merge_table

This operator merges and/or deletes rows of 'old_table' based on data values in 'merge_table' in the specified column(s). Both tables should be sorted on the specified column(s).

In the normal case, one or more rows in 'merge_table' either replace one or more existing rows in 'old_table' if the key column(s) match, or are inserted in order if the key column(s) do NOT match.

If the delete option is specified on the command line, one or more existing rows in 'old_table' will be deleted if there is a key column(s) match and the data in the delete column is equal to the delete string, ">>DEL<<" (without the quotes) by default. The delete column is the first non-key column in 'merge_table'.

Both tables should have similar data structures. The header for the new rdbtable is taken from 'merge_table', thus allowing a change of header information to be made.

This operator writes an rdbtable via STDOUT. Options may be abbreviated.

3.13 nsq-mktbl

Usage: nsq-mktbl [options]

Makes a file of data in columns (with visible column delimiters) into an rdbtable. The column delimiter is the pipe symbol (|). This operator is mainly used by other operators.

This operator reads a file via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated.

3.14 nsq-n2r

Usage: nsq-n2r [options]

Section yet to be written. See 'nsq-n2r -help' in the meantime.

3.15 nsq-pr

Usage: nsq-pr [options]

This operator used for quick and easy printing of an rdbtable, in a simple but useful form. It prints an rdbtable using formatting information from the header.

The printing of each row of data will be on one line if possible, but when multiple lines are necessary the second and later lines are indented for readability. Also when multiple lines are necessary a simple space availability algorithm is used to minimize the number of lines printed for each row of data. This may result in the order of some data values being rearranged from their order in the rdbtable. The '-b0' option can override this algorithm and force the same printing order as in the rdbtable.

This NoSQL operator reads an rdbtable from STDIN and writes a formatted report on STDOUT. Options may be abbreviated.

As an example using the sample rdbtable from the DATA section (named sample), the command to view this rdbtable would be:

      nsq-pr < sample
    

which would produce the output shown in Table 8. The same command with a page heading for printing:

      nsq-pr  -PP  <  sample
    

produces the output as shown in Table 9. Using an rdbtable (named sample4) that has long data values, shown in Table 10, the command to print the rdbtable using the truncate option is:

      nsq-pr  -t  <  sample4
  
                         Table 8

          PRINTING RDBTABLE (SAMPLE) USING NSQ-PR

      NAME    COUNT  TYP     AMT  OTHER        RIGHT
      ------  -----  ----  -----  --------  --------
      Bush       44  A       133  Another       This
      Hansen     44  A        23  One             Is
      Jones      77  X        77  Here            On
      Perry      77  B       244  And            The
      Hart       77  D      1111  So           Right
      Holmes     65  D      1111  On            Edge
  

                         Table 9

  PRINTING RDBTABLE (SAMPLE) WITH PAGE HEADING USING NSQ-PR

      Page   1   Mon Dec  2 16:56:43 PST 1991

      NAME    COUNT  TYP     AMT  OTHER        RIGHT
      ------  -----  ----  -----  --------  --------
      Bush       44  A       133  Another       This
      Hansen     44  A        23  One             Is
      Jones      77  X        77  Here            On
      Perry      77  B       244  And            The
      Hart       77  D      1111  So           Right
      Holmes     65  D      1111  On            Edge

                          Table 10

    RDBTABLE WITH LONG DATA VALUES (SAMPLE4) ACTUAL CONTENT

  name<T>type<T>contact<T>contents<N>
  10<T>4<T>21<T>20<N>
  Hansen<T>AAA<T>R. Starr at the UCLA & USC<T>Duplicate data under
  processing order number 55-7.<N>
  Hart<T>CCC<T>Hobbs/Emerson at RAND Corporation<T>85 files, 2 per
  day containing 12 and 24 hour reports.<N>
  Hobbs<T>EEE<T>Marshall at Universal AFB<T>Original PAF messages.
  Both sets are incomplete.<N>
  Bush<T>KKK<T>General USAF personnel<T>Duplicate ATO messages,
  incomplete.<N>
  Lender<T>RRR<T>Army base in Nevada<T>Nothing.<N>
  Emerson<T>UUU<T>Navy at Washington DC<T>More than we thought at
  first.<N>
      

which will produce output with the data values truncated to the defined column width as in Table 11. Using the same rdbtable with the fold option:

      nsq-pr  -fold  <  sample4
    

produces output with the long data values 'folded' within their defined column widths as shown in Table 12. Note that each line is repeated until the entire data value for each column is completely shown. This makes this type of output variable length.

If you need a quick and easy way to look at the data in an rdbtable use the -win option. This option will cause nsq-pr to list as many columns as possible in single line records that will fit in the current window or terminal width. Note that you do not have to type the column names (or even know them) to use this option.

                           Table 11

        PRINTING RDBTABLE (SAMPLE4) WITH NSQ-PR -TRUNC OPTION

   name        type  contact                contents
   ----------  ----  ---------------------  --------------------
   Hansen      AAA   R. Starr at the UCLA   Duplicate data under
   Hart        CCC   Hobbs/Emerson at RAND  85 files, 2 per day
   Hobbs       EEE   Marshall at Universal  Original PAF message
   Bush        KKK   General USAF personne  Duplicate ATO messag
   Lender      RRR   Army base in Nevada    Nothing.
   Emerson     UUU   Navy at Washington DC  More than we thought
    


                           Table 12

         PRINTING RDBTABLE (SAMPLE4) WITH NSQ-PR -FOLD OPTION

   name        type  contact                contents
   ----------  ----  ---------------------  --------------------
   Hansen      AAA   R. Starr at the UCLA   Duplicate data under
                     & USC                  processing order 
                                            number 55-7.
   Hart        CCC   Hobbs/Emerson at RAND  85 files, 2 per day
                     Corporation            containing 12 and 24
                                            hour reports.
   Hobbs       EEE   Marshall at Universal  Original PAF
                     AFB                    messages.  Both sets
                                            are incomplete.
   Bush        KKK   General USAF           Duplicate ATO
                     personnel              messages,
                                            incomplete.
   Lender      RRR   Army base in Nevada    Nothing.
   Emerson     UUU   Navy at Washington DC  More than we thought
                                            at first.
        

It may be combined with the -t option to increase the number of columns of data shown on each line at the expense of some column width.

For example the command 'nsq-pr < d11c' on an 80 character wide window or terminal produces the following:

    name    count            type  amt    n1                        n3
    ------  ---------------  ----  -----  ------------------------  ------------
        n2                        n4                        n5
        ------------------------  ------------------------  ------------
        n6            n7
        ------------  ------------
    Bush    3                A     133    alpha22.307               117722
     baker                     DDBBx17                   other
        124567        8GGXXH17
    Hansen  39               A     23     beta222.307               117723
    charlie                   DDBBx18                   data
        1239870       GGXXH17
    Newton  8                E     8      gama22.333                117724
    dog                       DDBBx19                   exists
        1239870       GGXXH17
    Hobbs   42               B     144    delta3.3.118              117725
    echo                      DDBBx20                   here
        1239870       GGXXH17
    Hart    2                C     55     epslion33.118             117726
    foxtrot                   DDBBx21                   also
        1239870       GGXXH17
    

This is readable, but not very nice to look at, and even worse if there are more columns. The command

          nsq-pr -win < d11c
        

produces:

 name   count           type amt   n1                       n3
 ------ --------------- ---- ----- ------------------------ ------------
 Bush   3               A    133   alpha22.307              117722
 Hansen 39              A    23    beta222.307              117723
 Newton 8               E    8     gama22.333               117724
 Hobbs  42              B    144   delta3.3.118             117725
 Hart   2               C    55    epslion33.118            117726
        

Not all the data is listed, but the first few columns (sometimes the most important) are easier to view. The command

          nsq-pr -win -t6 < d11c
        

shows even more of the data, at the expense of some data width:

 name   count  type amt   n1     n2     n3     n4     n5     n6     n7
 ------ ------ ---- ----- ------ ------ ------ ------ ------ ------ ------
 Bush   3      A    133   alpha2 baker  117722 DDBBx1 other  124567 8GGXXH
 Hansen 39     A    23    beta22 charli 117723 DDBBx1 data   123987 GGXXH1
 Newton 8      E    8     gama22 dog    117724 DDBBx1 exists 123987 GGXXH1
 Hobbs  42     B    144   delta3 echo   117725 DDBBx2 here   123987 GGXXH1
 Hart   2      C    55    epslio foxtro 117726 DDBBx2 also   123987 GGXXH1
        

3.16 nsq-r2n

Usage: nsq-r2n [options]

Section yet to be written. See 'nsq-r2n -help' in the meantime.

3.17 nsq-repair

Usage: nsq-repair [options] file ...

Attempts to repair candidate NoSQL datafiles, e.g. files that have been ported from a MacIntosh or PC (MSDOS computer) in spreadsheet form but that do not yet have valid rdbtable structure. Generates definition lines (second line of header). The width of all data values is checked and the maximum width for a column is used as the column width in the definition line for that Table.

It also works with existing rdbtables ('-exist' option) and is convenient for removing leading and trailing space characters from data values (-blank option).

Adds fields as necessary to rows (null), or to header (DUM1, DUM2, ...) to make the Table structure valid.

The new rdbtables will be in the current directory (even if the input files are not) and will have the suffix changed (or added) to '.rdb' by default.

Options may be abbreviated.

3.18 nsq-report

Usage: nsq-report [options] file.frm

Formats and prints an arbitrary style report, with the format specified in the file "file.frm". A page header may be specified.

This NoSQL operator reads an rdbtable from STDIN and writes a formatted report on STDOUT. Options may be abbreviated.

The "file.frm" file (or form file) shows pictorially one 'record' of output, which will contain data from one row of an rdbtable. An optional page header may be defined as well.

The form file contains regular text, picture fields, and associated column names. Regular text prints just as given. Picture fields define the width and justification for printing a data value from a column. The names of the associated columns are listed on the line following the picture fields and in the same order. Note that this file should not contain any TAB characters; space characters should be used instead.

Picture fields start with either '@' or '^' and are followed by one of three primary characters to define the width of the field. The three characters are '<', '>', or '|' to specify left, right, or center justification respectively. There is also an alternate right justification character for printing numeric data, with optional decimal point. The character is the sharp sign '#', and a period specifies the decimal point placement, as in '@########.##'.

A numeric picture field has the following features:

Numeric Data may be in the form of integers, fixed point, or scientific notation' e.g. 12345, 4567.345, or 1.678E17.

Normally picture fields start with the '@' character. That means to put the referenced data value into the defined picture field, or as much of the data as will fit into the field, if the data is larger than the field. If the field starts with the '^' character it means to repeat the field on as many lines as necessary in order to print the entire data value. This is useful for large data fields, such as comments or free text.

Instead of a column name there are some special names that can be used to have other information inserted. This are especially useful if there is a page header. The special names and what they mean are:

An example of a form file for use with rdbtable 'sample' is shown in Table 13.

The first and last lines (that start with 'format' or a single period) define the pictorial records and must be as shown. The first record defines the header and is optional. If this form file (named sample.frm) were used in the command:

      nsq-report  sample.frm  <  sample
        

it would produce the one page report as in Table 14.

                            Table 13

                 FORM FILE FOR RDBTABLE (SAMPLE)

format top =
Page @>,            The Page Header     @<<<<<<<<<<<<<<<<<<<<<<<<<<<
   _pgnr_                                       _date_

.
format =
  Name:   @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<          Type:  @>>>>
               NAME                          TYP
                  Total: @<<<<<<<<     Other:  @<<<<<<<<<<<<<<<
                         AMT                        OTHER
.




                           Table 14

            PRINTING RDBTABLE (SAMPLE) WITH NSQ-REPORT

Page  1,            The Page Header     Mon Dec  2 16:21:18 PST 1991

Name:   Bush                                        Type:      A
                  Total: 133           Other:  Another
Name:   Hansen                                      Type:      A
                  Total: 23            Other:  One
Name:   Jones                                       Type:      X
                  Total: 77            Other:  Here
Name:   Perry                                       Type:      B
                  Total: 244           Other:  And
Name:   Hart                                        Type:      D
                  Total: 1111          Other:  So
Name:   Holmes                                      Type:      D
                  Total: 1111          Other:  On
        

For another example, one might want to have a date on a report in other than the standard date output format, and an idea of who executed the program, and have a reference count of the records being produced. The form file might be as in Table 15. which could produce:

                            Table 15

                        ANOTHER FORM FILE

format top =
Run By: @<<<<<<<        The Date/Time is    @<<<<<<<<<<<<<<<<<<<<<<<<
        _`whoami`_                           _`date "+%m/%d/%y %H:%M"`_
.
format =
    RecordNr:  @>>      @<<<<<<<<<<<<<<<<<<<<
              _rcnr_    OTHER
.



Run By: hobbs           The Date/Time is    10/15/91 09:43

RecordNr:        1      Other data here
RecordNr:        2      and here,
RecordNr:        3      and so on.
...             ...     ...
        

Another example shows how longer data values can be handled. If the form file (named sample2.frm) looks like Table 16, and if the following command is used:

      nsq-report  sample2.frm  <  sample2
    

then the output would be as shown in Table 17.

                            Table 16

                 FORM FILE FOR RDBTABLE (SAMPLE2)

format top =
Page @>,                Page Header here                 @<<<<<<<<<<<<<
_pgnr_                                      _`date "+%m/%d/%y %H:%M"`_
                Executed by @<<<<<<< on: @<<<<<<<<<
                            `whoami`     _`hostname`_

format =
    Name:   @<<<<<<<<<<<<<<<  Other: @<<<<<<<<<<        Type: @<<<<<<<<
            NAME              OTHER                            TYP
 Comment:   ^<<<<<<<<<<<<<<    Long: ^<<<<<<<<<<<<<<    Right: @>>>>>>>
            COMMENT                   LONG                     RIGHT
=======================================================================



                              Table 17

              PRINTING RDBTABLE (SAMPLE2) WITH NSQ-REPORT

Page  1,                Page Header here                 12/02/91 16:32
                Executed by hobbs    on: id

    Name:   Bush              Other: Another            Type: A
 Comment:   A comment          Long: This a long        Right:     This
            here.                    message for
                                     test.
=======================================================================
    Name:   Hansen            Other: One                Type: A
 Comment:   A longer           Long: This a long        Right:       Is
            comment here.            message for
                                     test.
=======================================================================
    Name:   Jones             Other: Here               Type: X
 Comment:   A longer,          Long: Short test.        Right:       On
            longer comment
            here.
=======================================================================
    Name:   Perry             Other: And                Type: B
 Comment:   A short comment    Long: This a long        Right:      The
            here.                    message for
                                     test.
=======================================================================
    Name:   Hart              Other: So                 Type: D
 Comment:   Little here.       Long: Here too.          Right:    Right
=======================================================================
    Name:   Holmes            Other: On                 Type: D
 Comment:   A comment here     Long: A short            Right:     Edge
            that is a                message.
            little ongoing,
            so to speak.
=======================================================================
        

Note that since there were two picture fields that started with the '^' character on one line the length of output records varies according to the length of the two associated data values.

3.19 nsq-row

Usage: nsq-row [options] expression

Selects rows from the input rdbtable based on an arbitrary expression using column names. Characters that are special to the UNIX shell must be quoted.

Logical constructors 'or' and 'and' may be used; as well as 'null' to indicate empty data values. Comparison operators may be of the form: gt, ge, lt, le, eq, ne, mat, nmat. The first six are the usual operators, E.g 'name eq Hobbs' or 'COUNT gt 100'. The last two stand for 'match' and 'non-match' and are used for pattern matching. They are exactally the same as using the PERL operators '= ' or '! ' respectively, except that pattern matching can be specified easier in expressions, as in:

      NAME  mat   /[Hh]obbs/  <<< First letter either case
      NAME  mat   /hobbs/i    <<< any combination of case
      NAME  nmat  /[aeiou]/i  <<< names without vowels
        

where 'NAME' and 'COUNT' are column names, of course. A warning message is produced on STDERR if either of 'mat' or 'nmat' is used with a numeric type column, but the execution continues. It does not check the '= ' or '! ' forms.

All of the Comparison operators and Logical constructors are reserved and should not be used as column names (they are all lower case and four characters or less).

Since column names and reserved words are parsed by the program, do not put the entire expression in a single pair of quotes as that will prevent the parsing. Also note that column names and reserved words need to be surrounded by blank spaces if they are not individually quoted. For example either form below is fine:

      nsq-row   NAME    eq   "L Brown"  <  sample
      nsq-row  "NAME"  "eq"  "L Brown"  <  sample
        

but do not use this form:

      nsq-row  "NAME  eq  L Brown"  <  sample
        

This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated.

As an example using the sample rdbtable from the DATA section (named sample), to select rows that have the NAME column equal to 'Hansen' the command would be:

      nsq-row  NAME  eq  Hansen  <  sample
        

which would produce:

          NAME    COUNT   TYP     AMT     OTHER   RIGHT
          6       5N      4       5N      8       8>
          Hansen  44      A       23      One     Is
        

to select rows that have the TYP column equal to 'A' or that have the AMT column greater than 1000 the command would be:

      nsq-row  TYP  eq  A  or  AMT  gt  1000  <  sample
        

producing:

          NAME    COUNT   TYP     AMT     OTHER   RIGHT
          6       5N      4       5N      8       8>
          Bush    44      A       133     Another This
          Hansen  44      A       23      One     Is
          Hart    77      D       1111    So      Right
          Holmes  65      D       1111    On      Edge
        

Note that in some rare cases there could be a column name that is identical to a data value specified in an expression using another column name that might cause a problem (this actually happened). For example if two column names are 'N' and 'T', and column 'N' has a data value of 'T', to select all rows where column 'N' is equal to 'T' the normal command would be:

      nsq-row  <  table  N  eq  T
        

Unfortunately the 'T' in the expression gets translated to 'column name T', not used as 'data value T'. That is, the expression askes for all rows where the data value of column N equals the data value of column T, legal, but not what was wanted. There is a simple workaround however. The 'T' in the expression can be escaped with a backslash to prevent the translation to a column name, as in the revised command:

      nsq-row  <  table  N  eq  '\T'
        

Thus either meaning can be specified, as desired.

3.20 nsq-search

Usage:

nsq-search [options] rdbtbl < keytbl

or:

nsq-search [options] -ind index_file [rdbtbl] < keytbl

This operator does a fast search of 'rdbtbl' (or index_file) using a binary search on a key of of one or more columns. The 'rdbtbl' (or index_file) must be sorted on the key columns. Each column in the key may be of type string or type numeric (but be carefull with numeric data and exact matches). In the second form of usage for this operator if 'rdbtbl' is not given its name will be inferred from the name of index_file. For example if index_file is 'skb.x.typ' then the rdbtbl name inferred will be 'skb.rdb'.

The column(s) in the file 'keytbl' specify both the key column name(s) and the argument values to search for. File 'Keytbl' is in rdbtable format.

Normally an argument value and a data field must compare exactally for a match to occur (exact match). If the paritial match otpion (-part) is selected, and if the argument value compares with the initial part of the data field it is considered a match. This applies to string type data only. Note that for numeric type data an exact match is always necessary.

Normally all rows that match will be written to the new rdbtable, in the same order as in the old rdbtable. If only a single row key match is appropriate some execution time can be saved by specifing the '-sgl' option.

This operator writes an rdbtable via STDOUT. Options may be abbreviated. Returns the number of non-finds at exit.

3.21 nsq-sort

Usage: nsq-sort [options] [-r] column [[-r] column] ...

Sorts an rdbtable on one or more columns. Each column may be sorted in normal (ascending) or reverse (descending) order. Also a column of monthnames (Jan, Apr, ...) in any case letters, may be sorted.

This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated. Uses the UNIX 'sort' routine.

For example, using the sample data file from the DATA section (named sample) in the following command:

      nsq-sort  COUNT  TYP  <  sample
        

would produce:

    
      NAME    COUNT   TYP     AMT     OTHER   RIGHT
      6       5N      4       5N      8       8>
      Bush    44      A       133     Another This
      Hansen  44      A       23      One     Is
      Holmes  65      D       1111    On      Edge
      Perry   77      B       244     And     The
      Hart    77      D       1111    So      Right
      Jones   77      X       77      Here    On

    

Of course it would look better if it was piped through 'nsq-pr'. The command:

      nsq-sort  COUNT  -r  AMT  <  sample
    

would produce:

    
      NAME    COUNT   TYP     AMT     OTHER   RIGHT
      6       5N      4       5N      8       8>
      Bush    44      A       133     Another This
      Hansen  44      A       23      One     Is
      Holmes  65      D       1111    On      Edge
      Hart    77      D       1111    So      Right
      Perry   77      B       244     And     The
      Jones   77      X       77      Here    On

        

3.22 nsq-subtot

Usage: nsq-subtot [options] B_column ... -s column ...

This operator lists subtotals of specified column(s) whenever the value of specified break columns(s) (B_column(s)) changes. A single break column will produce a sub-total of all specified columns on each line. If there is more than one break column given then in addition whenever the value of the first break column changes an additional line will be printed showing the sub-total for that group.

If no break column is given the first column is used; if no subtotal column is given then all columns of type numeric are sub-totaled.

This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated.

Example rdbtable (named small):


      name      amt    typ   count  n
      6         5N     4     5N     2
      Hansen    39     A     23     3
      Hansen    9      A     3      3
      Hansen    9      B     3      4
      Jones     42     B     144    5
      Jones     4      B     14     5
      Hart      9      C     3      5
      Hart      2      C     55     6
      Hart      2      D     55     6
      Hobbs     57     X     7      4
      Hobbs     5      X     57     4
    
        

The output from the command:

      nsq-subtot  name  -s  amt  <  small  |  nsq-pr
    

would be:


      name      amt
      ------  -----
      Hansen     57
      Jones      46
      Hart       13
      Hobbs      62

        

The output from the command:

      nsq-subtot  name  typ  -s  amt  count  <  small  |  nsq-pr
        

is shown in Table 18.


                             Table 18
  
                  OUTPUT FROM THE NSQ-SUBTOT OPERATOR

      name    typ     amt  count
      ------  ----  -----  -----
      Hansen  A        48     26
      Hansen  B         9      3
                       57     29
      
      Jones   B        46    158
                       46    158
      
      Hart    C        11     58
      Hart    D         2     55
                       13    113
      
      Hobbs   X        62     64
                       62     64
      
        

3.23 nsq-summ

Usage: nsq-summ [options] [column ...]

Produces "summary" information about the rdbtable. If no columns are given then information about all columns is produced. A Count of the data rows is always shown.

This operator reads an rdbtable via STDIN and writes a summary report via STDOUT. Options may be abbreviated.

3.24 nsq-t2l

Usage: nsq-t2l [options]

Converts an rdbtable to "list" format. Long data fields are folded. This operator is mainly used by other operators.

This NoSQL operator reads an rdbtable from STDIN and writes an rdbtable to STDOUT. Options may be abbreviated.

3.25 nsq-tabletolist

Usage: nsq-tabletolist [options]

Section yet to be written. See 'nsq-tabletolist -help' in the meantime.

3.26 nsq-tee

Usage: nsq-tee [options] rdbtable

Reads an rdbtable via STDIN and [over]writes the rdbtable given as a command line argument, abiding by the NoSQL table locking scheme. This operator is named after yhe UNIX utility tee.

This operator can be safely used in constructs like :

          nsq-compute ... < table.rdb | nsq-tee table.rdb | ...
    

without worring that the original input file table.rdb be destroyed before the first pipeline has completed.

If no output stream is desired on STDOUT then the latter can be directed to /dev/null, as usual:

      nsq-compute ... < table.rdb | nsq-tee table.rdb > /dev/null
    

This operator is especially meant to be used in programs that need to modify rdbtables 'in place'. It has been designed for the GNU Bourne Again Shell (Bash), but it should work just as well with other Bourne compatible shells.

Note: the output table name given on the command line may not start with the carachter "," , to prevent possible clobbering of the backup table.

3.27 nsq-uniq

Usage: nsq-uniq [options] column ...

Reads the input rdbtable and compares adjacent rows. The second and succeeding copies of repeated rows, considering only the selected columns, are removed. That is, adjacent rows are considered equal if the data values in all of the selected columns are equal. The remaining rows are written to the output rdbtable.

Note that repeated rows must be adjacent in order to be found. Normally this means that the input rdbtable should be sorted on the selected columns for this capability to work properly.

This NoSQL operator reads an rdbtable from STDIN and writes an rdbtable to STDOUT. Options may be abbreviated.

3.28 nsq-updseq

Usage: nsq-updseq [options]

Section yet to be written. See 'nsq-updseq -help' in the meantime.

3.29 nsq-valid

Usage: nsq-valid [options] [file ...]

Validates the structure of one or more rdbtables. Checks number of data fields per line, max width of column names and data values, and checks numeric data type values. Reports errors by line number and column name.

Reads from STDIN if filenames are not given. Writes diagnostic information on STDOUT. Options may be abbreviated.

If there is more than one file given each file will be identified on the output.

The '-size' option has proven very useful as it shows the actual size of the largest data value for each column, in addition to the template information. The command:

      nsq-valid  -size  sample
    

shows the following output:

    
    
          0                 NAME  6     6
          1                COUNT  5N    2
          2                  TYP  4     1
          3                  AMT  5N    4
          4                OTHER  8     7
          5                RIGHT  8>    5
          Columns: 6, Rows: 6, File format valid  sample

        

The last two columns above show the defined size of each column in the rdbtable, and the actual maximum size of the data values for each column in the rdbtable.


Previous Next Contents