Fundamental Specifications of QDBM Version 1

Copyright (C) 2000-2005 Mikio Hirabayashi
Last Update: Sat, 10 Sep 2005 07:50:50 +0900
[Japanese] [Home]

Table of Contents

  1. Overview
  2. Features
  3. Installation
  4. Depot: Basic API
  5. Commands for Depot
  6. Curia: Extended API
  7. Commands for Curia
  8. Relic: NDBM-compatible API
  9. Commands for Relic
  10. Hovel: GDBM-compatible API
  11. Commands for Hovel
  12. Cabin: Utility API
  13. Commands for Cabin
  14. Villa: Advanced API
  15. Commands for Villa
  16. Odeum: Inverted API
  17. Commands for Odeum
  18. File Format
  19. Porting
  20. Bugs
  21. Frequently Asked Questions
  22. Copying

Overview

QDBM is a library of routines for managing a database. The database is a simple data file containing records, each is a pair of a key and a value. Every key and value is serial bytes with variable length. Both binary data and character string can be used as a key and a value. There is neither concept of data tables nor data types. Records are organized in hash table or B+ tree.

As for database of hash table, each key must be unique within a database, so it is impossible to store two or more records with a key overlaps. The following access methods are provided to the database: storing a record with a key and a value, deleting a record by a key, retrieving a record by a key. Moreover, traversal access to every key are provided, although the order is arbitrary. These access methods are similar to ones of DBM (or its followers: NDBM and GDBM) library defined in the UNIX standard. QDBM is an alternative for DBM because of its higher performance.

As for database of B+ tree, records whose keys are duplicated can be stored. Access methods of storing, deleting, and retrieving are provided as with the database of hash table. Records are stored in order by a comparing function assigned by a user. It is possible to access each record with the cursor in ascending or descending order. According to this mechanism, forward matching search for strings and range search for integers are realized. Moreover, transaction is available in database of B+ tree.

QDBM is written in C, and provided as APIs of C, C++, Java, Perl, and Ruby. QDBM is available on platforms which have API conforming to POSIX. QDBM is a free software licensed under the GNU Lesser General Public License.

Features

Effective Implementation of Hash Database

QDBM is developed referring to GDBM for the purpose of the following three points: higher processing speed, smaller size of a database file, and simpler API. They have been achieved. Moreover, as with GDBM, the following three restrictions of traditional DBM: a process can handle only one database, the size of a key and a value is bounded, a database file is sparse, are cleared.

QDBM uses hash algorithm to retrieve records. If a bucket array has sufficient number of elements, the time complexity of retrieval is `O(1)'. That is, time required for retrieving a record is constant, regardless of the scale of a database. It is also the same about storing and deleting. Collision of hash values is managed by separate chaining. Data structure of the chains is binary search tree. Even if a bucket array has unusually scarce elements, the time complexity of retrieval is `O(log n)'.

QDBM attains improvement in retrieval by loading RAM with the whole of a bucket array. If a bucket array is on RAM, it is possible to access a region of a target record by about one path of file operations. A bucket array saved in a file is not read into RAM with the `read' call but directly mapped to RAM with the `mmap' call. Therefore, preparation time on connecting to a database is very short, and two or more processes can share the same memory map.

If the number of elements of a bucket array is about half of records stored within a database, although it depends on characteristic of the input, the probability of collision of hash values is about 56.7% (36.8% if the same, 21.3% if twice, 11.5% if four times, 6.0% if eight times). In such case, it is possible to retrieve a record by two or less paths of file operations. If it is made into a performance index, in order to handle a database containing one million of records, a bucket array with half a million of elements is needed. The size of each element is 4 bytes. That is, if 2M bytes of RAM is available, a database containing one million records can be handled.

QDBM provides two modes to connect to a database: `reader' and `writer'. A reader can perform retrieving but neither storing nor deleting. A writer can perform all access methods. Exclusion control between processes is performed when connecting to a database by file locking. While a writer is connected to a database, neither readers nor writers can be connected. While a reader is connected to a database, other readers can be connect, but writers can not. According to this mechanism, data consistency is guaranteed with simultaneous connections in multitasking environment.

Traditional DBM provides two modes of the storing operations: `insert' and `replace'. In the case a key overlaps an existing record, the insert mode keeps the existing value, while the replace mode transposes it to the specified value. In addition to the two modes, QDBM provides `concatenate' mode. In the mode, the specified value is concatenated at the end of the existing value and stored. This feature is useful when adding a element to a value as an array. Moreover, although DBM has a method to fetch out a value from a database only by reading the whole of a region of a record, QDBM has a method to fetch out a part of a region of a value. When a value is treated as an array, this feature is also useful.

Generally speaking, while succession of updating, fragmentation of available regions occurs, and the size of a database grows rapidly. QDBM deal with this problem by coalescence of dispensable regions and reuse of them, and featuring of optimization of a database. When overwriting a record with a value whose size is greater than the existing one, it is necessary to remove the region to another position of the file. Because the time complexity of the operation depends on the size of the region of a record, extending values successively is inefficient. However, QDBM deal with this problem by alignment. If increment can be put in padding, it is not necessary to remove the region.

As for many file systems, it is impossible to handle a file whose size is more than 2GB. To deal with this problem, QDBM provides a directory database containing multiple database files. Due to this feature, it is possible to handle a database whose total size is up to 1TB in theory. Moreover, because database files can be deployed on multiple disks, the speed of updating operations can be improved as with RAID-0 (striping). It is also possible for the database files to deploy on multiple file servers using NFS and so on.

Useful Implementation of B+ Tree Database

Although B+ tree database is slower than hash database, it features ordering access to each record. The order can be assigned by users. Records of B+ tree are sorted and arranged in logical pages. Sparse index organized in B tree that is multiway balanced tree are maintained for each page. Thus, the time complexity of retrieval and so on is `O(log n)'. Cursor is provided to access each record in order. The cursor can jump to position specified by a key and can step forward or backward from the current position. Because each page is arranged as double linked list, the time complexity of stepping cursor is `O(1)'.

B+ tree database is implemented, based on above hash database. Because each page of B+ tree is stored as each record of hash database, B+ tree database inherits efficiency of storage management of hash database. Because the header of each record is smaller and alignment of each page is calculated statistically, in most cases, the size of database file is cut by half compared to one of hash database. Although operation of many pages are required to update B+ tree, QDBM expedites the process by caching pages and reducing file operations. In most cases, because whole of the sparse index is cached on memory, it is possible to retrieve a record by one or less path of file operations.

B+ tree database features transaction mechanism. It is possible to commit a series of operations between the beginning and the end of the transaction in a lump, or to abort the transaction and perform rollback to the state before the transaction. Even if the process of an application is crashed while the transaction, the database file is not broken.

In case that QDBM was built with ZLIB enabled, a lossless data-compression library, the content of each page of B+ tree is compressed and stored in a file. Because each record in a page has similar patterns, high efficiency of compression is expected due to the LZ77 algorithm. In case handling text data, the size of a database is reduced to about 25%. If the scale of a database is large and disk I/O is the bottleneck, featuring compression makes the processing speed improved to a large extent.

Simple but Various Interfaces

QDBM provides very simple APIs. You can perform database I/O as usual file I/O with `FILE' pointer defined in ANSI C. In the basic API of QDBM, entity of a database is recorded as one file. In the extended API, entity of a database is recorded as several files in one directory. Because the two APIs are very similar with each other, porting an application from one to the other is easy.

APIs which are compatible with NDBM and GDBM are also provided. As there are a lot of applications using NDBM or GDBM, it is easy to port them onto QDBM. In most cases, it is completed only by replacement of header including (#include) and re-compiling. However, QDBM can not handle database files made by the original NDBM or GDBM.

In order to handle records on memory easily, the utility API is provided. It implements memory allocating functions, sorting functions, extensible datum, array list, hash map, and so on. Using them, you can handle records in C language cheaply as in such script languages as Perl or Ruby.

B+ tree database is used with the advanced API. The advanced API is implemented using the basic API and the utility API. Because the advanced API is also similar to the basic API and the extended API, it is easy to learn how to use it.

In order to handle an inverted index which is used by full-text search systems, the inverted API is provided. If it is easy to handle an inverted index of documents, an application can focus on text processing and natural language processing. Because this API does not depend on character codes nor languages, it is possible to implement a full-text search system which can respond to various requests from users.

Along with APIs for C, QDBM provides APIs for C++, Java, Perl, and Ruby. APIs for C are composed of seven kinds: the basic API, the extended API, the NDBM-compatible API, the GDBM-compatible API, the utility API, the advanced API, and the inverted API. Command line interfaces corresponding to each API are also provided. They are useful for prototyping, testing, debugging, and so on. The C++ API encapsulates database handling functions of the basic API, the extended API, and the advanced API with class mechanism of C++. The Java API has native methods calling the basic API, the extended API, and the advanced API with Java Native Interface. The Perl API has methods calling the basic API, the extended API, and the advanced API with XS language. The Ruby API has method calling the basic API, the extended API, and the advanced API as modules of Ruby. Moreover, CGI scripts for administration of databases, file uploading, and full-text search are provided.

Wide Portability

QDBM is implemented being based on syntax of ANSI C (C89) and using only APIs defined in ANSI C or POSIX. Thus, QDBM works on most UNIX and its compatible OSs. As for C API, checking operations have been done at least on the following platforms.

Although a database file created by QDBM depends on byte order of the processor, to do with it, utilities to dump data in format which is independent to byte orders are provided.

Installation

Preparation

To install QDBM from a source package, GCC of 2.8 or later version and `make' are required.

When an archive file of QDBM is extracted, change the current working directory to the generated directory and perform installation.

Usual Steps

Follow the procedures below on Linux, BSD, or SunOS.

Run the configuration script.

./configure

Build programs.

make

Perform self-diagnostic test.

make check

Install programs. This operation must be carried out by the root user.

make install

Using GNU Libtool

If above steps do not work, try the following steps. This way needs GNU Libtool of 1.5 or later version.

Run the configuration script.

./configure

Build programs.

make -f LTmakefile

Perform self-diagnostic test.

make -f LTmakefile check

Install programs. This operation must be carried out by the root user.

make -f LTmakefile install

Result

When a series of work finishes, the following files will be installed. As for the rest, manuals will be installed under `/usr/local/man/man1' and '/usr/local/man/man3', other documents will be installed under `/usr/local/share/qdbm'. A configuration file for `pkg-config' will be installed under `/usr/local/lib/pkgconfig'.

/usr/local/include/depot.h
/usr/local/include/curia.h
/usr/local/include/relic.h
/usr/local/include/hovel.h
/usr/local/include/cabin.h
/usr/local/include/villa.h
/usr/local/include/vista.h
/usr/local/include/odeum.h
/usr/local/lib/libqdbm.a
/usr/local/lib/libqdbm.so.11.5.0
/usr/local/lib/libqdbm.so.11
/usr/local/lib/libqdbm.so
/usr/local/bin/dpmgr
/usr/local/bin/dptest
/usr/local/bin/dptsv
/usr/local/bin/crmgr
/usr/local/bin/crtest
/usr/local/bin/crtsv
/usr/local/bin/rlmgr
/usr/local/bin/rltest
/usr/local/bin/hvmgr
/usr/local/bin/hvtest
/usr/local/bin/cbtest
/usr/local/bin/cbcodec
/usr/local/bin/vlmgr
/usr/local/bin/vltest
/usr/local/bin/vltsv
/usr/local/bin/odmgr
/usr/local/bin/odtest
/usr/local/bin/odidx

When you run a program linked dynamically to `libqdbm.so', the library search path should include `/usr/local/lib'. You can set the library search path with the environment variable `LD_LIBRARY_PATH'.

To uninstall QDBM, execute the following command after `./configure'. This operation must be carried out by the root user.

make uninstall

If an old version of QDBM is installed on your system, uninstall it before installation of a new one.

The other APIs except for C nor CGI scripts are not installed by default. Refer to `plus/xspex.html' to know how to install the C++ API. Refer to `java/jspex.html' to know how to install the Java API. Refer to `perl/plspex.html' to know how to install the Perl API. Refer to `ruby/rbspex.html' to know how to install the Ruby API. Refer to `cgi/cgispex.html' to know how to install the CGI script.

To install QDBM from such a binary package as RPM, refer to the manual of the package manager. For example, if you use RPM, execute like the following command by the root user.

rpm -ivh qdbm-1.x.x-x.i386.rpm

For Windows

On Windows (Cygwin), you should follow the procedures below for installation.

Run the configuration script.

./configure

Build programs.

make win

Perform self-diagnostic test.

make check-win

Install programs. As well, perform `make uninstall-win' to uninstall them.

make install-win

On Windows, the import library `libqdbm.dll.a' is created as well as the static library `libqdbm.a', and the dynamic linking library `qdbm.dll' is created instead of such shared libraries as `libqdbm.so'. `qdbm.dll' is installed into `/usr/local/bin'.

In order to build QDBM using MinGW on Cygwin, you should perform `make mingw' instead of `make win'. With the UNIX emulation layer of Cygwin, generated programs depend on `cygwin1.dll' (they come under GNU GPL). This problem is solved by linking them to the Win32 native DLL with MinGW.

In order to build QDBM using Visual C++, you should edit `VCmakefile' and set the search paths for libraries and headers. And perform `nmake /f VCMakefile'. However, how to install is not defined, and the DLL is not generated. Refer to `VCmakefile' for detail configurations.

For Mac OS X

On Mac OS X (Darwin), you should follow the procedures below for installation.

Run the configuration script.

./configure

Build programs.

make mac

Perform self-diagnostic test.

make check-mac

Install programs. As well, perform `make uninstall-mac' to uninstall them.

make install-mac

On Mac OS X, `libqdbm.dylib' and so on are created instead of `libqdbm.so' and so on. You can set the library search path with the environment variable `DYLD_LIBRARY_PATH'.

For HP-UX

On HP-UX, you should follow the procedures below for installation.

Run the configuration script.

./configure

Build programs.

make hpux

Perform self-diagnostic test.

make check-hpux

Install programs. As well, perform `make uninstall-hpux' to uninstall them.

make install-hpux

On HP-UX, `libqdbm.sl' is created instead of `libqdbm.so' and so on. You can set the library search path with the environment variable `SHLIB_PATH'.

For RISC OS

On RISC OS, you should follow the procedures below for installation.

Build programs. As `cc' is used for compilation by default, if you want to use `gcc', add the argument `CC=gcc'.

make -f RISCmakefile

When a series of work finishes, the library file `libqdbm' and such commands as `dpmgr' are generated. Because how to install them is not defined, copy them manually for installation. As with it, such header files as `depot.h' should be installed manually.

Detail Configurations

You can configure building processes by the following optional arguments of `./configure'.

Usually, QDBM and its applications can be built without any dependency on non-standard libraries except for `libqdbm.*'. However, they depend on `libpthread.*' if POSIX thread is enabled, and they depend on `libz.*' if ZLIB is enabled, and they depend on `libiconv.*' if ICONV is enabled.

Depot: Basic API

Overview

Depot is the basic API of QDBM. Almost all features for managing a database provided by QDBM are implemented by Depot. Other APIs are no more than wrappers of Depot. Depot is the fastest in all APIs of QDBM.

In order to use Depot, you should include `depot.h' and `stdlib.h' in the source files. Usually, the following description will be near the beginning of a source file.

#include <depot.h>
#include <stdlib.h>

A pointer to `DEPOT' is used as a database handle. It is like that some file I/O routines of `stdio.h' use a pointer to `FILE'. A database handle is opened with the function `dpopen' and closed with `dpclose'. You should not refer directly to any member of the handle. If a fatal error occurs in a database, any access method via the handle except `dpclose' will not work and return error status. Although a process is allowed to use multiple database handles at the same time, handles of the same database file should not be used.

API

The external variable `dpversion' is the string containing the version information.

extern const char *dpversion;

The external variable `dpecode' is assigned with the last happened error code. Refer to `depot.h' for details of the error codes.

extern int dpecode;
The initial value of this variable is `DP_ENOERR'. The other values are `DP_EFATAL', `DP_EMODE', `DP_EBROKEN', `DP_EKEEP', `DP_ENOITEM', `DP_EALLOC', `DP_EMAP', `DP_EOPEN', `DP_ECLOSE', `DP_ETRUNC', `DP_ESYNC', `DP_ESTAT', `DP_ESEEK', `DP_EREAD', `DP_EWRITE', `DP_ELOCK', `DP_EUNLINK', `DP_EMKDIR', `DP_ERMDIR', and `DP_EMISC'.

The function `dperrmsg' is used in order to get a message string corresponding to an error code.

const char *dperrmsg(int ecode);
`ecode' specifies an error code. The return value is the message string of the error code. The region of the return value is not writable.

The function `dpopen' is used in order to get a database handle.

DEPOT *dpopen(const char *name, int omode, int bnum);
`name' specifies the name of a database file. `omode' specifies the connection mode: `DP_OWRITER' as a writer, `DP_OREADER' as a reader. If the mode is `DP_OWRITER', the following may be added by bitwise or: `DP_OCREAT', which means it creates a new database if not exist, `DP_OTRUNC', which means it creates a new database regardless if one exists. Both of `DP_OREADER' and `DP_OWRITER' can be added to by bitwise or: `DP_ONOLCK', which means it opens a database file without file locking, or `DP_OLCKNB', which means locking is performed without blocking. `DP_OCREAT' can be added to by bitwise or: `DP_OSPARSE', which means it creates a database file as a sparse file. `bnum' specifies the number of elements of the bucket array. If it is not more than 0, the default value is specified. The size of a bucket array is determined on creating, and can not be changed except for by optimization of the database. Suggested size of a bucket array is about from 0.5 to 4 times of the number of all records to store. The return value is the database handle or `NULL' if it is not successful. While connecting as a writer, an exclusive lock is invoked to the database file. While connecting as a reader, a shared lock is invoked to the database file. The thread blocks until the lock is achieved. If `DP_ONOLCK' is used, the application is responsible for exclusion control.

The function `dpclose' is used in order to close a database handle.

int dpclose(DEPOT *depot);
`depot' specifies a database handle. If successful, the return value is true, else, it is false. Because the region of a closed handle is released, it becomes impossible to use the handle. Updating a database is assured to be written when the handle is closed. If a writer opens a database but does not close it appropriately, the database will be broken.

The function `dpput' is used in order to store a record.

int dpput(DEPOT *depot, const char *kbuf, int ksiz, const char *vbuf, int vsiz, int dmode);
`depot' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `vbuf' specifies the pointer to the region of a value. `vsiz' specifies the size of the region of the value. If it is negative, the size is assigned with `strlen(vbuf)'. `dmode' specifies behavior when the key overlaps, by the following values: `DP_DOVER', which means the specified value overwrites the existing one, `DP_DKEEP', which means the existing value is kept, `DP_DCAT', which means the specified value is concatenated at the end of the existing value. If successful, the return value is true, else, it is false.

The function `dpout' is used in order to delete a record.

int dpout(DEPOT *depot, const char *kbuf, int ksiz);
`depot' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is true, else, it is false. False is returned when no record corresponds to the specified key.

The function `dpget' is used in order to retrieve a record.

char *dpget(DEPOT *depot, const char *kbuf, int ksiz, int start, int max, int *sp);
`depot' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `start' specifies the offset address of the beginning of the region of the value to be read. `max' specifies the max size to be read. If it is negative, the size to read is unlimited. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the value of the corresponding record, else, it is `NULL'. `NULL' is returned when no record corresponds to the specified key or the size of the value of the corresponding record is less than `start'. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `dpgetwb' is used in order to retrieve a record and write the value into a buffer.

int dpgetwb(DEPOT *depot, const char *kbuf, int ksiz, int start, int max, char *vbuf);
`depot' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `start' specifies the offset address of the beginning of the region of the value to be read. `max' specifies the max size to be read. It shuld be equal to or less than the size of the writing buffer. `vbuf' specifies the pointer to a buffer into which the value of the corresponding record is written. If successful, the return value is the size of the written data, else, it is -1. -1 is returned when no record corresponds to the specified key or the size of the value of the corresponding record is less than `start'. Note that no additional zero code is appended at the end of the region of the writing buffer.

The function `dpvsiz' is used in order to get the size of the value of a record.

int dpvsiz(DEPOT *depot, const char *kbuf, int ksiz);
`depot' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is the size of the value of the corresponding record, else, it is -1. Because this function does not read the entity of a record, it is faster than `dpget'.

The function `dpiterinit' is used in order to initialize the iterator of a database handle.

int dpiterinit(DEPOT *depot);
`depot' specifies a database handle. If successful, the return value is true, else, it is false. The iterator is used in order to access the key of every record stored in a database.

The function `dpiternext' is used in order to get the next key of the iterator.

char *dpiternext(DEPOT *depot, int *sp);
`depot' specifies a database handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the next key, else, it is `NULL'. `NULL' is returned when no record is to be get out of the iterator. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. It is possible to access every record by iteration of calling this function. However, it is not assured if updating the database is occurred while the iteration. Besides, the order of this traversal access method is arbitrary, so it is not assured that the order of storing matches the one of the traversal access.

The function `dpsetalign' is used in order to set alignment of a database handle.

int dpsetalign(DEPOT *depot, int align);
`depot' specifies a database handle connected as a writer. `align' specifies the size of alignment. If successful, the return value is true, else, it is false. If alignment is set to a database, the efficiency of overwriting values are improved. The size of alignment is suggested to be average size of the values of the records to be stored. If alignment is positive, padding whose size is multiple number of the alignment is placed. If alignment is negative, as `vsiz' is the size of a value, the size of padding is calculated with `(vsiz / pow(2, abs(align) - 1))'. Because alignment setting is not saved in a database, you should specify alignment every opening a database.

The function `dpsync' is used in order to synchronize updating contents with the file and the device.

int dpsync(DEPOT *depot);
`depot' specifies a database handle connected as a writer. If successful, the return value is true, else, it is false. This function is useful when another process uses the connected database file.

The function `dpoptimize' is used in order to optimize a database.

int dpoptimize(DEPOT *depot, int bnum);
`depot' specifies a database handle connected as a writer. `bnum' specifies the number of the elements of the bucket array. If it is not more than 0, the default value is specified. If successful, the return value is true, else, it is false. In an alternating succession of deleting and storing with overwrite or concatenate, dispensable regions accumulate. This function is useful to do away with them.

The function `dpname' is used in order to get the name of a database.

char *dpname(DEPOT *depot);
`depot' specifies a database handle. If successful, the return value is the pointer to the region of the name of the database, else, it is `NULL'. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `dpfsiz' is used in order to get the size of a database file.

int dpfsiz(DEPOT *depot);
`depot' specifies a database handle. If successful, the return value is the size of the database file, else, it is -1.

The function `dpbnum' is used in order to get the number of the elements of the bucket array.

int dpbnum(DEPOT *depot);
`depot' specifies a database handle. If successful, the return value is the number of the elements of the bucket array, else, it is -1.

The function `dpbusenum' is used in order to get the number of the used elements of the bucket array.

int dpbusenum(DEPOT *depot);
`depot' specifies a database handle. If successful, the return value is the number of the used elements of the bucket array, else, it is -1. This function is inefficient because it accesses all elements of the bucket array.

The function `dprnum' is used in order to get the number of the records stored in a database.

int dprnum(DEPOT *depot);
`depot' specifies a database handle. If successful, the return value is the number of the records stored in the database, else, it is -1.

The function `dpwritable' is used in order to check whether a database handle is a writer or not.

int dpwritable(DEPOT *depot);
`depot' specifies a database handle. The return value is true if the handle is a writer, false if not.

The function `dpfatalerror' is used in order to check whether a database has a fatal error or not.

int dpfatalerror(DEPOT *depot);
`depot' specifies a database handle. The return value is true if the database has a fatal error, false if not.

The function `dpinode' is used in order to get the inode number of a database file.

int dpinode(DEPOT *depot);
`depot' specifies a database handle. The return value is the inode number of the database file.

The function `dpmtime' is used in order to get the last modified time of a database.

int dpmtime(DEPOT *depot);
`depot' specifies a database handle. The return value is the last modified time of the database.

The function `dpfdesc' is used in order to get the file descriptor of a database file.

int dpfdesc(DEPOT *depot);
`depot' specifies a database handle. The return value is the file descriptor of the database file. Handling the file descriptor of a database file directly is not suggested.

The function `dpremove' is used in order to remove a database file.

int dpremove(const char *name);
`name' specifies the name of a database file. If successful, the return value is true, else, it is false.

The function `dprepair' is used in order to repair a broken database file.

int dprepair(const char *name);
`name' specifies the name of a database file. If successful, the return value is true, else, it is false. There is no guarantee that all records in a repaired database file correspond to the original or expected state.

The function `dpexportdb' is used in order to dump all records as endian independent data.

int dpexportdb(DEPOT *depot, const char *name);
`depot' specifies a database handle. `name' specifies the name of an output file. If successful, the return value is true, else, it is false.

The function `dpimportdb' is used in order to load all records from endian independent data.

int dpimportdb(DEPOT *depot, const char *name);
`depot' specifies a database handle connected as a writer. The database of the handle must be empty. `name' specifies the name of an input file. If successful, the return value is true, else, it is false.

The function `dpinnerhash' is a hash function used inside Depot.

int dpinnerhash(const char *kbuf, int ksiz);
`kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. The return value is the hash value of 31 bits length computed from the key. This function is useful when an application calculates the state of the inside bucket array.

The function `dpouterhash' is a hash function which is independent from the hash functions used inside Depot.

int dpouterhash(const char *kbuf, int ksiz);
`kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. The return value is the hash value of 31 bits length computed from the key. This function is useful when an application uses its own hash algorithm outside Depot.

The function `dpprimenum' is used in order to get a natural prime number not less than a number.

int dpprimenum(int num);
`num' specified a natural number. The return value is a natural prime number not less than the specified number. This function is useful when an application determines the size of a bucket array of its own hash algorithm.

Examples

The following example stores and retrieves a phone number, using the name as the key.

#include <depot.h>
#include <stdlib.h>
#include <stdio.h>

#define NAME     "mikio"
#define NUMBER   "000-1234-5678"
#define DBNAME   "book"

int main(int argc, char **argv){
  DEPOT *depot;
  char *val;

  /* open the database */
  if(!(depot = dpopen(DBNAME, DP_OWRITER | DP_OCREAT, -1))){
    fprintf(stderr, "dpopen: %s\n", dperrmsg(dpecode));
    return 1;
  }

  /* store the record */
  if(!dpput(depot, NAME, -1, NUMBER, -1, DP_DOVER)){
    fprintf(stderr, "dpput: %s\n", dperrmsg(dpecode));
  }

  /* retrieve the record */
  if(!(val = dpget(depot, NAME, -1, 0, -1, NULL))){
    fprintf(stderr, "dpget: %s\n", dperrmsg(dpecode));
  } else {
    printf("Name: %s\n", NAME);
    printf("Number: %s\n", val);
    free(val);
  }

  /* close the database */
  if(!dpclose(depot)){
    fprintf(stderr, "dpclose: %s\n", dperrmsg(dpecode));
    return 1;
  }

  return 0;
}

The following example shows all records of the database.

#include <depot.h>
#include <stdlib.h>
#include <stdio.h>

#define DBNAME   "book"

int main(int argc, char **argv){
  DEPOT *depot;
  char *key, *val;

  /* open the database */
  if(!(depot = dpopen(DBNAME, DP_OREADER, -1))){
    fprintf(stderr, "dpopen: %s\n", dperrmsg(dpecode));
    return 1;
  }

  /* initialize the iterator */
  if(!dpiterinit(depot)){
    fprintf(stderr, "dpiterinit: %s\n", dperrmsg(dpecode));
  }

  /* scan with the iterator */
  while((key = dpiternext(depot, NULL)) != NULL){
    if(!(val = dpget(depot, key, -1, 0, -1, NULL))){
      fprintf(stderr, "dpget: %s\n", dperrmsg(dpecode));
      free(key);
      break;
    }
    printf("%s: %s\n", key, val);
    free(val);
    free(key);
  }

  /* close the database */
  if(!dpclose(depot)){
    fprintf(stderr, "dpclose: %s\n", dperrmsg(dpecode));
    return 1;
  }

  return 0;
}

Notes

For building a program using Depot, the program should be linked with a library file `libqdbm.a' or `libqdbm.so'. For example, the following command is executed to build `sample' from `sample.c'.

gcc -I/usr/local/include -o sample sample.c -L/usr/local/lib -lqdbm

If QDBM was built with POSIX thread enabled, the global variable `dpecode' is treated as thread specific data, and functions of Depot are reentrant. In that case, they are thread-safe as long as a handle is not accessed by threads at the same time, on the assumption that `errno', `malloc', and so on are thread-safe.

Commands for Depot

Depot has the following command line interfaces.

The command `dpmgr' is a utility for debugging Depot and its applications. It features editing and checking of a database. It can be used for database applications with shell scripts. This command is used in the following format. `name' specifies a database name. `key' specifies the key of a record. `val' specifies the value of a record.

dpmgr create [-s] [-bnum num] name
Create a database file.
dpmgr put [-kx|-ki] [-vx|-vi|-vf] [-keep|-cat] [-na] name key val
Store a record with a key and a value.
dpmgr out [-kx|-ki] name key
Delete a record with a key.
dpmgr get [-kx|-ki] [-start num] [-max num] [-ox] [-n] name key
Retrieve a record with a key and output it to the standard output.
dpmgr list [-ox] name
List all keys and values delimited with tab and line-feed to the standard output.
dpmgr optimize [-bnum num] [-na] name
Optimize a database.
dpmgr inform name
Output miscellaneous information to the standard output.
dpmgr remove name
Remove a database file.
dpmgr repair name
Repair a broken database file.
dpmgr exportdb name file
Dump all records as endian independent data.
dpmgr importdb [-bnum num] name file
Load all records from endian independent data.
dpmgr version
Output version information of QDBM to the standard output.

Options feature the following.

This command returns 0 on success, another on failure. The environment variable `QDBMDBGFD' specifies the file descriptor to output the history of updating the variable `dpecode'.

The command `dptest' is a utility for facility test and performance test. Check a database generated by the command or measure the execution time of the command. This command is used in the following format. `name' specifies a database name. `rnum' specifies the number of the records. `bnum' specifies the number of the elements of the bucket array. `pnum' specifies the number of patterns of the keys. `align' specifies the basic size of alignment.

dptest write [-s] name rnum bnum
Store records with keys of 8 bytes. They change as `00000001', `00000002'...
dptest read [-wb] name
Retrieve all records of the database above.
dptest rcat [-c] name rnum bnum pnum align
Store records with partway duplicated keys using concatenate mode.
dptest combo name
Perform combination test of various operations.
dptest wicked [-c] name rnum
Perform updating operations selected at random.

Options feature the following.

This command returns 0 on success, another on failure. The environment variable `QDBMDBGFD' specifies the file descriptor to output the history of updating the variable `dpecode'.

The command `dptsv' features mutual conversion between a database of Depot and a TSV text. This command is useful when data exchange with another version of QDBM or another DBM, or when data exchange between systems which have different byte orders. This command is used in the following format. `name' specifies a database name. The subcommand `export' reads TSV data from the standard input. If a key overlaps, the latter is adopted. `-bnum' specifies the number of the elements of the bucket array. The subcommand `import' writes TSV data to the standard output.

dptsv import [-bnum num] [-bin] name
Create a database from TSV.
dptsv export [-bin] name
Write TSV data of a database.

Options feature the following.

This command returns 0 on success, another on failure.

Commands of Depot realize a simple database system. For example, to make a database to search `/etc/password' by a user name, perform the following command.

cat /etc/passwd | tr ':' '\t' | dptsv import casket

Thus, to retrieve the information of a user `mikio', perform the following command.

dpmgr get casket mikio

It is easy to implement functions upsides with these commands, using the API of Depot.

Curia: Extended API

Overview

Curia is the extended API of QDBM. It provides routines for managing multiple database files in a directory. Restrictions of some file systems that the size of each file is limited are escaped by dividing a database file into two or more. If the database files deploy on multiple devices, the scalability is improved.

Although Depot creates a database with a file name, Curia creates a database with a directory name. A database file named as `depot' is placed in the specified directory. Although it keeps the attribute of the database, it does not keep the entities of the records. Besides, sub directories are created by the number of division of the database, named with 4 digits. The database files are placed in the subdirectories. The entities of the records are stored in the database file. For example, in the case that a database directory named as `casket' and the number of division is 3, `casket/depot', `casket/0001/depot', `casket/0002/depot' and `casket/0003/depot' are created. No error occurs even if the namesake directory exists when creating a database. So, if sub directories exists and some devices are mounted on the sub directories, the database files deploy on the multiple devices.

Curia features managing large objects. Although usual records are stored in some database files, records of large objects are stored in individual files. Because the files of large objects are deployed in different directories named with the hash values, the access speed is part-way robust although it is slower than the speed of usual records. Large and not often accessed data should be secluded as large objects. By doing this, the access speed of usual records are improved. The directory hierarchies of large objects are placed in the directory named as `lob' in the sub directories of the database. Because the key spaces of the usual records and the large objects are different, the operations keep out of each other.

In order to use Curia, you should include `depot.h', `curia.h' and `stdlib.h' in the source files. Usually, the following description will be near the beginning of a source file.

#include <depot.h>
#include <curia.h>
#include <stdlib.h>

A pointer to `CURIA' is used as a database handle. It is like that some file I/O routines of `stdio.h' use a pointer to `FILE'. A database handle is opened with the function `cropen' and closed with `crclose'. You should not refer directly to any member of the handle. If a fatal error occurs in a database, any access method via the handle except `crclose' will not work and return error status. Although a process is allowed to use multiple database handles at the same time, handles of the same database directory should not be used.

Curia also assign the external variable `dpecode' with the error code. The function `dperrmsg' is used in order to get the message of the error code.

API

The function `cropen' is used in order to get a database handle.

CURIA *cropen(const char *name, int omode, int bnum, int dnum);
`name' specifies the name of a database directory. `omode' specifies the connection mode: `CR_OWRITER' as a writer, `CR_OREADER' as a reader. If the mode is `CR_OWRITER', the following may be added by bitwise or: `CR_OCREAT', which means it creates a new database if not exist, `CR_OTRUNC', which means it creates a new database regardless if one exists. Both of `CR_OREADER' and `CR_OWRITER' can be added to by bitwise or: `CR_ONOLCK', which means it opens a database directory without file locking, or `CR_OLCKNB', which means locking is performed without blocking. `CR_OCREAT' can be added to by bitwise or: `CR_OSPARSE', which means it creates database files as sparse files. `bnum' specifies the number of elements of each bucket array. If it is not more than 0, the default value is specified. The size of each bucket array is determined on creating, and can not be changed except for by optimization of the database. Suggested size of each bucket array is about from 0.5 to 4 times of the number of all records to store. `dnum' specifies the number of division of the database. If it is not more than 0, the default value is specified. The number of division can not be changed from the initial value. The max number of division is 512. The return value is the database handle or `NULL' if it is not successful. While connecting as a writer, an exclusive lock is invoked to the database directory. While connecting as a reader, a shared lock is invoked to the database directory. The thread blocks until the lock is achieved. If `CR_ONOLCK' is used, the application is responsible for exclusion control.

The function `crclose' is used in order to close a database handle.

int crclose(CURIA *curia);
`curia' specifies a database handle. If successful, the return value is true, else, it is false. Because the region of a closed handle is released, it becomes impossible to use the handle. Updating a database is assured to be written when the handle is closed. If a writer opens a database but does not close it appropriately, the database will be broken.

The function `crput' is used in order to store a record.

int crput(CURIA *curia, const char *kbuf, int ksiz, const char *vbuf, int vsiz, int dmode);
`curia' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `vbuf' specifies the pointer to the region of a value. `vsiz' specifies the size of the region of the value. If it is negative, the size is assigned with `strlen(vbuf)'. `dmode' specifies behavior when the key overlaps, by the following values: `CR_DOVER', which means the specified value overwrites the existing one, `CR_DKEEP', which means the existing value is kept, `CR_DCAT', which means the specified value is concatenated at the end of the existing value. If successful, the return value is true, else, it is false.

The function `crout' is used in order to delete a record.

int crout(CURIA *curia, const char *kbuf, int ksiz);
`curia' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is true, else, it is false. False is returned when no record corresponds to the specified key.

The function `crget' is used in order to retrieve a record.

char *crget(CURIA *curia, const char *kbuf, int ksiz, int start, int max, int *sp);
`curia' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `start' specifies the offset address of the beginning of the region of the value to be read. `max' specifies the max size to be read. If it is negative, the size to read is unlimited. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the value of the corresponding record, else, it is `NULL'. `NULL' is returned when no record corresponds to the specified key or the size of the value of the corresponding record is less than `start'. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `crgetwb' is used in order to retrieve a record and write the value into a buffer.

int crgetwb(CURIA *curia, const char *kbuf, int ksiz, int start, int max, char *vbuf);
`curia' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `start' specifies the offset address of the beginning of the region of the value to be read. `max' specifies the max size to be read. It shuld be equal to or less than the size of the writing buffer. `vbuf' specifies the pointer to a buffer into which the value of the corresponding record is written. If successful, the return value is the size of the written data, else, it is -1. -1 is returned when no record corresponds to the specified key or the size of the value of the corresponding record is less than `start'. Note that no additional zero code is appended at the end of the region of the writing buffer.

The function `crvsiz' is used in order to get the size of the value of a record.

int crvsiz(CURIA *curia, const char *kbuf, int ksiz);
`curia' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is the size of the value of the corresponding record, else, it is -1. Because this function does not read the entity of a record, it is faster than `crget'.

The function `criterinit' is used in order to initialize the iterator of a database handle.

int criterinit(CURIA *curia);
`curia' specifies a database handle. If successful, the return value is true, else, it is false. The iterator is used in order to access the key of every record stored in a database.

The function `criternext' is used in order to get the next key of the iterator.

char *criternext(CURIA *curia, int *sp);
`curia' specifies a database handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the next key, else, it is `NULL'. `NULL' is returned when no record is to be get out of the iterator. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. It is possible to access every record by iteration of calling this function. However, it is not assured if updating the database is occurred while the iteration. Besides, the order of this traversal access method is arbitrary, so it is not assured that the order of storing matches the one of the traversal access.

The function `crsetalign' is used in order to set alignment of a database handle.

int crsetalign(CURIA *curia, int align);
`curia' specifies a database handle connected as a writer. `align' specifies the size of alignment. If successful, the return value is true, else, it is false. If alignment is set to a database, the efficiency of overwriting values are improved. The size of alignment is suggested to be average size of the values of the records to be stored. If alignment is positive, padding whose size is multiple number of the alignment is placed. If alignment is negative, as `vsiz' is the size of a value, the size of padding is calculated with `(vsiz / pow(2, abs(align) - 1))'. Because alignment setting is not saved in a database, you should specify alignment every opening a database.

The function `crsync' is used in order to synchronize updating contents with the files and the devices.

int crsync(CURIA *curia);
`curia' specifies a database handle connected as a writer. If successful, the return value is true, else, it is false. This function is useful when another process uses the connected database directory.

The function `croptimize' is used in order to optimize a database.

int croptimize(CURIA *curia, int bnum);
`curia' specifies a database handle connected as a writer. `bnum' specifies the number of the elements of each bucket array. If it is not more than 0, the default value is specified. In an alternating succession of deleting and storing with overwrite or concatenate, dispensable regions accumulate. This function is useful to do away with them.

The function `crname' is used in order to get the name of a database.

char *crname(CURIA *curia);
`curia' specifies a database handle. If successful, the return value is the pointer to the region of the name of the database, else, it is `NULL'. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `crfsiz' is used in order to get the total size of database files.

int crfsiz(CURIA *curia);
`curia' specifies a database handle. If successful, the return value is the total size of the database files, else, it is -1. If the total size is more than 2GB, the return value overflows.

The function `crfsizd' is used in order to get the total size of database files as double-precision floating-point number.

double crfsizd(CURIA *curia);
`curia' specifies a database handle. If successful, the return value is the total size of the database files, else, it is -1.0.

The function `crbnum' is used in order to get the total number of the elements of each bucket array.

int crbnum(CURIA *curia);
`curia' specifies a database handle. If successful, the return value is the total number of the elements of each bucket array, else, it is -1.

The function `crbusenum' is used in order to get the total number of the used elements of each bucket array.

int crbusenum(CURIA *curia);
`curia' specifies a database handle. If successful, the return value is the total number of the used elements of each bucket array, else, it is -1. This function is inefficient because it accesses all elements of each bucket array.

The function `crrnum' is used in order to get the number of the records stored in a database.

int crrnum(CURIA *curia);
`curia' specifies a database handle. If successful, the return value is the number of the records stored in the database, else, it is -1.

The function `crwritable' is used in order to check whether a database handle is a writer or not.

int crwritable(CURIA *curia);
`curia' specifies a database handle. The return value is true if the handle is a writer, false if not.

The function `crfatalerror' is used in order to check whether a database has a fatal error or not.

int crfatalerror(CURIA *curia);
`curia' specifies a database handle. The return value is true if the database has a fatal error, false if not.

The function `crinode' is used in order to get the inode number of a database directory.

int crinode(CURIA *curia);
`curia' specifies a database handle. The return value is the inode number of the database directory.

The function `crmtime' is used in order to get the last modified time of a database.

int crmtime(CURIA *curia);
`curia' specifies a database handle. The return value is the last modified time of the database.

The function `crremove' is used in order to remove a database directory.

int crremove(const char *name);
`name' specifies the name of a database directory. If successful, the return value is true, else, it is false.

The function `crrepair' is used in order to repair a broken database directory.

int crrepair(const char *name);
`name' specifies the name of a database directory. If successful, the return value is true, else, it is false. There is no guarantee that all records in a repaired database directory correspond to the original or expected state.

The function `crexportdb' is used in order to dump all records as endian independent data.

int crexportdb(CURIA *curia, const char *name);
`curia' specifies a database handle. `name' specifies the name of an output directory. If successful, the return value is true, else, it is false. Note that large objects are ignored.

The function `crimportdb' is used in order to load all records from endian independent data.

int crimportdb(CURIA *curia, const char *name);
`curia' specifies a database handle connected as a writer. The database of the handle must be empty. `name' specifies the name of an input directory. If successful, the return value is true, else, it is false. Note that large objects are ignored.

The function `crputlob' is used in order to store a large object.

int crputlob(CURIA *curia, const char *kbuf, int ksiz, const char *vbuf, int vsiz, int dmode);
`curia' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `vbuf' specifies the pointer to the region of a value. `vsiz' specifies the size of the region of the value. If it is negative, the size is assigned with `strlen(vbuf)'. `dmode' specifies behavior when the key overlaps, by the following values: `CR_DOVER', which means the specified value overwrites the existing one, `CR_DKEEP', which means the existing value is kept, `CR_DCAT', which means the specified value is concatenated at the end of the existing value. If successful, the return value is true, else, it is false.

The function `croutlob' is used in order to delete a large object.

int croutlob(CURIA *curia, const char *kbuf, int ksiz);
`curia' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is true, else, it is false. false is returned when no large object corresponds to the specified key.

The function `crgetlob' is used in order to retrieve a large object.

char *crgetlob(CURIA *curia, const char *kbuf, int ksiz, int start, int max, int *sp);
`curia' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `start' specifies the offset address of the beginning of the region of the value to be read. `max' specifies the max size to be read. If it is negative, the size to read is unlimited. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the value of the corresponding large object, else, it is `NULL'. `NULL' is returned when no large object corresponds to the specified key or the size of the value of the corresponding large object is less than `start'. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `crgetlobfd' is used in order to get the file descriptor of a large object.

int crgetlobfd(CURIA *curia, const char *kbuf, int ksiz);
`curia' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is the file descriptor of the corresponding large object, else, it is -1. -1 is returned when no large object corresponds to the specified key. The returned file descriptor is opened with the `open' call. If the database handle was opened as a writer, the descriptor is writable (O_RDWR), else, it is not writable (O_RDONLY). The descriptor should be closed with the `close' call if it is no longer in use.

The function `crvsizlob' is used in order to get the size of the value of a large object.

int crvsizlob(CURIA *curia, const char *kbuf, int ksiz);
`curia' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is the size of the value of the corresponding large object, else, it is -1. Because this function does not read the entity of a large object, it is faster than `crgetlob'.

The function `crrnumlob' is used in order to get the number of the large objects stored in a database.

int crrnumlob(CURIA *curia);
`curia' specifies a database handle. If successful, the return value is the number of the large objects stored in the database, else, it is -1.

Examples

The following example stores and retrieves a phone number, using the name as the key.

#include <depot.h>
#include <curia.h>
#include <stdlib.h>
#include <stdio.h>

#define NAME     "mikio"
#define NUMBER   "000-1234-5678"
#define DBNAME   "book"

int main(int argc, char **argv){
  CURIA *curia;
  char *val;

  /* open the database */
  if(!(curia = cropen(DBNAME, CR_OWRITER | CR_OCREAT, -1, -1))){
    fprintf(stderr, "cropen: %s\n", dperrmsg(dpecode));
    return 1;
  }

  /* store the record */
  if(!crput(curia, NAME, -1, NUMBER, -1, CR_DOVER)){
    fprintf(stderr, "crput: %s\n", dperrmsg(dpecode));
  }

  /* retrieve the record */
  if(!(val = crget(curia, NAME, -1, 0, -1, NULL))){
    fprintf(stderr, "crget: %s\n", dperrmsg(dpecode));
  } else {
    printf("Name: %s\n", NAME);
    printf("Number: %s\n", val);
    free(val);
  }

  /* close the database */
  if(!crclose(curia)){
    fprintf(stderr, "crclose: %s\n", dperrmsg(dpecode));
    return 1;
  }

  return 0;
}

The following example shows all records of the database.

#include <depot.h>
#include <curia.h>
#include <stdlib.h>
#include <stdio.h>

#define DBNAME   "book"

int main(int argc, char **argv){
  CURIA *curia;
  char *key, *val;

  /* open the database */
  if(!(curia = cropen(DBNAME, CR_OREADER, -1, -1))){
    fprintf(stderr, "cropen: %s\n", dperrmsg(dpecode));
    return 1;
  }

  /* initialize the iterator */
  if(!criterinit(curia)){
    fprintf(stderr, "criterinit: %s\n", dperrmsg(dpecode));
  }

  /* scan with the iterator */
  while((key = criternext(curia, NULL)) != NULL){
    if(!(val = crget(curia, key, -1, 0, -1, NULL))){
      fprintf(stderr, "crget: %s\n", dperrmsg(dpecode));
      free(key);
      break;
    }
    printf("%s: %s\n", key, val);
    free(val);
    free(key);
  }

  /* close the iterator */
  if(!crclose(curia)){
    fprintf(stderr, "crclose: %s\n", dperrmsg(dpecode));
    return 1;
  }

  return 0;
}

Notes

How to build programs using Curia is the same as the case of Depot.

gcc -I/usr/local/include -o sample sample.c -L/usr/local/lib -lqdbm

If QDBM was built with POSIX thread enabled, the global variable `dpecode' is treated as thread specific data, and functions of Curia are reentrant. In that case, they are thread-safe as long as a handle is not accessed by threads at the same time, on the assumption that `errno', `malloc', and so on are thread-safe.

Commands for Curia

Curia has the following command line interfaces.

The command `crmgr' is a utility for debugging Curia and its applications. It features editing and checking of a database. It can be used for the database applications with shell scripts. This command is used in the following format. `name' specifies a database name. `key' specifies the key of a record. `val' specifies the value of a record.

crmgr create [-s] [-bnum num] [-dnum num] name
Create a database file.
crmgr put [-kx|-ki] [-vx|-vi|-vf] [-keep|-cat] [-lob] [-na] name key val
Store a record with a key and a value.
crmgr out [-kx|-ki] [-lob] name key
Delete a record with a key.
crmgr get [-kx|-ki] [-start num] [-max num] [-ox] [-lob] [-n] name key
Retrieve a record with a key and output it to the standard output.
crmgr list [-ox] name
List all keys and values delimited with tab and line-feed to the standard output.
crmgr optimize [-bnum num] [-na] name
Optimize a database.
crmgr inform name
Output miscellaneous information to the standard output.
crmgr remove name
Remove a database directory.
crmgr repair name
Repair a broken database directory.
crmgr exportdb name dir
Dump all records as endian independent data.
crmgr importdb [-bnum num] [-dnum num] name dir
Load all records from endian independent data.
crmgr version
Output version information of QDBM to the standard output.

Options feature the following.

This command returns 0 on success, another on failure. The environment variable `QDBMDBGFD' specifies the file descriptor to output the history of updating the variable `dpecode'.

The command `crtest' is a utility for facility test and performance test. Check a database generated by the command or measure the execution time of the command. This command is used in the following format. `name' specifies a database name. `rnum' specifies the number of records. `bnum' specifies the number of elements of a bucket array. `dnum' specifies the number of division of a database. `pnum' specifies the number of patterns of the keys. `align' specifies the basic size of alignment.

crtest write [-s] [-lob] name rnum bnum dnum
Store records with keys of 8 bytes. They change as `00000001', `00000002'...
crtest read [-wb] [-lob] name
Retrieve all records of the database above.
crtest rcat [-c] name rnum bnum dnum pnum align
Store records with partway duplicated keys using concatenate mode.
crtest combo name
Perform combination test of various operations.
crtest wicked [-c] name rnum
Perform updating operations selected at random.

Options feature the following.

This command returns 0 on success, another on failure. The environment variable `QDBMDBGFD' specifies the file descriptor to output the history of updating the variable `dpecode'.

The command `crtsv' features mutual conversion between a database of Curia and a TSV text. This command is useful when data exchange with another version of QDBM or another DBM, or when data exchange between systems which have different byte orders. This command is used in the following format. `name' specifies a database name. The subcommand `export' reads TSV data from the standard input. If a key overlaps, the latter is adopted. `-bnum' specifies the number of the elements of the bucket array. `-dnum' specifies the number of division of the database. The subcommand `import' writes TSV data to the standard output.

crtsv import [-bnum num] [-dnum num] [-bin] name
Create a database from TSV.
crtsv export [-bin] name
Write TSV data of a database.

Options feature the following.

This command returns 0 on success, another on failure.

Commands of Curia realize a simple database system. For example, to make a database to search `/etc/password' by a user name, perform the following command.

cat /etc/passwd | tr ':' '\t' | crtsv import casket

Thus, to retrieve the information of a user `mikio', perform the following command.

crmgr get casket mikio

It is easy to implement functions upsides with these commands, using the API of Curia.

Relic: NDBM-compatible API

Overview

Relic is the API which is compatible with NDBM. So, Relic wraps functions of Depot as API of NDBM. It is easy to port an application from NDBM to QDBM. In most cases, you should only replace the includings of `ndbm.h' with `relic.h' and replace the linking option `-lndbm' with `-lqdbm'.

The original NDBM treats a database as a pair of files. One, `a directory file', has a name with suffix `.dir' and stores a bit map of keys. The other, `a data file', has a name with suffix `.pag' and stores entities of each records. Relic creates the directory file as a mere dummy file and creates the data file as a database. Relic has no restriction about the size of each record. Relic can not handle database files made by the original NDBM.

In order to use Relic, you should include `relic.h', `stdlib.h', `sys/types.h', `sys/stat.h' and `fcntl.h' in the source files. Usually, the following description will be near the beginning of a source file.

#include <relic.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

A pointer to `DBM' is used as a database handle. A database handle is opened with the function `dbm_open' and closed with `dbm_close'. You should not refer directly to any member of a handle.

API

Structures of `datum' type is used in order to give and receive data of keys and values with functions of Relic.

typedef struct { void *dptr; size_t dsize; } datum;
`dptr' specifies the pointer to the region of a key or a value. `dsize' specifies the size of the region.

The function `dbm_open' is used in order to get a database handle.

DBM *dbm_open(char *name, int flags, int mode);
`name' specifies the name of a database. The file names are concatenated with suffixes. `flags' is the same as the one of `open' call, although `O_WRONLY' is treated as `O_RDWR' and additional flags except for `O_CREAT' and `O_TRUNC' have no effect. `mode' specifies the mode of the database file as the one of `open' call does. The return value is the database handle or `NULL' if it is not successful.

The function `dbm_close' is used in order to close a database handle.

void dbm_close(DBM *db);
`db' specifies a database handle. Because the region of the closed handle is released, it becomes impossible to use the handle.

The function `dbm_store' is used in order to store a record.

int dbm_store(DBM *db, datum key, datum content, int flags);
`db' specifies a database handle. `key' specifies a structure of a key. `content' specifies a structure of a value. `flags' specifies behavior when the key overlaps, by the following values: `DBM_REPLACE', which means the specified value overwrites the existing one, `DBM_INSERT', which means the existing value is kept. The return value is 0 if it is successful, 1 if it gives up because of overlaps of the key, -1 if other error occurs.

The function `dbm_delete' is used in order to delete a record.

int dbm_delete(DBM *db, datum key);
`db' specifies a database handle. `key' specifies a structure of a key. The return value is 0 if it is successful, -1 if some errors occur.

The function `dbm_fetch' is used in order to retrieve a record.

datum dbm_fetch(DBM *db, datum key);
`db' specifies a database handle. `key' specifies a structure of a key. The return value is a structure of the result. If a record corresponds, the member `dptr' of the structure is the pointer to the region of the value. If no record corresponds or some errors occur, `dptr' is `NULL'. `dptr' points to the region related with the handle. The region is available until the next time of calling this function with the same handle.

The function `dbm_firstkey' is used in order to get the first key of a database.

datum dbm_firstkey(DBM *db);
`db' specifies a database handle. The return value is a structure of the result. If a record corresponds, the member `dptr' of the structure is the pointer to the region of the first key. If no record corresponds or some errors occur, `dptr' is `NULL'. `dptr' points to the region related with the handle. The region is available until the next time of calling this function or the function `dbm_nextkey' with the same handle.

The function `dbm_nextkey' is used in order to get the next key of a database.

datum dbm_nextkey(DBM *db);
`db' specifies a database handle. The return value is a structure of the result. If a record corresponds, the member `dptr' of the structure is the pointer to the region of the next key. If no record corresponds or some errors occur, `dptr' is `NULL'. `dptr' points to the region related with the handle. The region is available until the next time of calling this function or the function `dbm_firstkey' with the same handle.

The function `dbm_error' is used in order to check whether a database has a fatal error or not.

int dbm_error(DBM *db);
`db' specifies a database handle. The return value is true if the database has a fatal error, false if not.

The function `dbm_clearerr' has no effect.

int dbm_clearerr(DBM *db);
`db' specifies a database handle. The return value is 0. The function is only for compatibility.

The function `dbm_rdonly' is used in order to check whether a handle is read-only or not.

int dbm_rdonly(DBM *db);
`db' specifies a database handle. The return value is true if the handle is read-only, or false if not read-only.

The function `dbm_dirfno' is used in order to get the file descriptor of a directory file.

int dbm_dirfno(DBM *db);
`db' specifies a database handle. The return value is the file descriptor of the directory file.

The function `dbm_pagfno' is used in order to get the file descriptor of a data file.

int dbm_pagfno(DBM *db);
`db' specifies a database handle. The return value is the file descriptor of the data file.

Examples

The following example stores and retrieves a phone number, using the name as the key.

#include <relic.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
#include <string.h>

#define NAME     "mikio"
#define NUMBER   "000-1234-5678"
#define DBNAME   "book"

int main(int argc, char **argv){
  DBM *db;
  datum key, val;
  int i;

  /* open the database */
  if(!(db = dbm_open(DBNAME, O_RDWR | O_CREAT, 00644))){
    perror("dbm_open");
    return 1;
  }

  /* prepare the record */
  key.dptr = NAME;
  key.dsize = strlen(NAME);
  val.dptr = NUMBER;
  val.dsize = strlen(NUMBER);

  /* store the record */
  if(dbm_store(db, key, val, DBM_REPLACE) != 0){
    perror("dbm_store");
  }

  /* retrieve the record */
  val = dbm_fetch(db, key);
  if(val.dptr){
    printf("Name: %s\n", NAME);
    printf("Number: ");
    for(i = 0; i < val.dsize; i++){
      putchar(((char *)val.dptr)[i]);
    }
    putchar('\n');
  } else {
    perror("dbm_fetch");
  }

  /* close the database */
  dbm_close(db);

  return 0;
}

Notes

How to build programs using Relic is the same as the case of Depot. Note that an option to be given to a linker is not `-lndbm', but `-lqdbm'.

gcc -I/usr/local/include -o sample sample.c -L/usr/local/lib -lqdbm

Functions of Relic are thread-safe as long as a handle is not accessed by threads at the same time, on the assumption that `errno', `malloc', and so on are thread-safe.

Commands for Relic

Relic has the following command line interfaces.

The command `rlmgr' is a utility for debugging Relic and its applications. It features editing and checking of a database. It can be used for database applications with shell scripts. This command is used in the following format. `name' specifies a database name. `key' specifies the key of a record. `val' specifies the value of a record.

rlmgr create name
Create a database file.
rlmgr store [-kx] [-vx|-vf] [-insert] name key val
Store a record with a key and a value.
rlmgr delete [-kx] name key
Delete a record with a key.
rlmgr fetch [-kx] [-ox] [-n] name key
Retrieve a record with a key and output to the standard output.
rlmgr list [-ox] name
List all keys and values delimited with tab and line-feed to the standard output.

Options feature the following.

This command returns 0 on success, another on failure.

The command `rltest' is a utility for facility test and performance test. Check a database generated by the command or measure the execution time of the command. This command is used in the following format. `name' specifies a database name. `rnum' specifies the number of records.

rltest write name rnum
Store records with keys of 8 bytes. They change as `00000001', `00000002'...
rltest read name rnum
Retrieve records of the database above.

This command returns 0 on success, another on failure.

Hovel: GDBM-compatible API

Overview

Hovel is the API which is compatible with GDBM. So, Hovel wraps functions of Depot and Curia as API of GDBM. It is easy to port an application from GDBM to QDBM. In most cases, you should only replace the includings of `gdbm.h' with `hovel.h' and replace the linking option `-lgdbm' with `-lqdbm'. Hovel can not handle database files made by the original GDBM.

In order to use Hovel, you should include `hovel.h', `stdlib.h', `sys/types.h' and `sys/stat.h' in the source files. Usually, the following description will be near the beginning of a source file.

#include <hovel.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>

An object of `GDBM_FILE' is used as a database handle. A database handle is opened with the function `gdbm_open' and closed with `gdbm_close'. You should not refer directly to any member of a handle. Although Hovel works as a wrapper of Depot and handles a database file usually, if you use the function `gdbm_open2' to open the handle, it is possible to make behavior of a handle as a wrapper of Curia and treat a database directory.

API

Structures of `datum' type is used in order to give and receive data of keys and values with functions of Hovel.

typedef struct { char *dptr; size_t dsize; } datum;
`dptr' specifies the pointer to the region of a key or a value. `dsize' specifies the size of the region.

The external variable `gdbm_version' is the string containing the version information.

extern char *gdbm_version;

The external variable `gdbm_errno' is assigned with the last happened error code. Refer to `hovel.h' for details of the error codes.

extern gdbm_error gdbm_errno;
The initial value of this variable is `GDBM_NO_ERROR'. The other values are `GDBM_MALLOC_ERROR', `GDBM_BLOCK_SIZE_ERROR', `GDBM_FILE_OPEN_ERROR', `GDBM_FILE_WRITE_ERROR', `GDBM_FILE_SEEK_ERROR', `GDBM_FILE_READ_ERROR', `GDBM_BAD_MAGIC_NUMBER', `GDBM_EMPTY_DATABASE', `GDBM_CANT_BE_READER', `GDBM_CANT_BE_WRITER', `GDBM_READER_CANT_DELETE', `GDBM_READER_CANT_STORE', `GDBM_READER_CANT_REORGANIZE', `GDBM_UNKNOWN_UPDATE', `GDBM_ITEM_NOT_FOUND', `GDBM_REORGANIZE_FAILED', `GDBM_CANNOT_REPLACE', `GDBM_ILLEGAL_DATA', `GDBM_OPT_ALREADY_SET', and `GDBM_OPT_ILLEGAL'.

The function `gdbm_strerror' is used in order to get a message string corresponding to an error code.

char *gdbm_strerror(gdbm_error gdbmerrno);
`gdbmerrno' specifies an error code. The return value is the message string of the error code. The region of the return value is not writable.

The function `gdbm_open' is used in order to get a database handle after the fashion of GDBM.

GDBM_FILE gdbm_open(char *name, int block_size, int read_write, int mode, void (*fatal_func)(void));
`name' specifies the name of a database. `block_size' is ignored. `read_write' specifies the connection mode: `GDBM_READER' as a reader, `GDBM_WRITER', `GDBM_WRCREAT' and `GDBM_NEWDB' as a writer. `GDBM_WRCREAT' makes a database file or directory if it does not exist. `GDBM_NEWDB' makes a new database even if it exists. You can add the following to writer modes by bitwise or: `GDBM_SYNC', `GDBM_NOLOCK', `GDBM_LOCKNB', `GDBM_FAST', and `GDBM_SPARSE'. `GDBM_SYNC' means a database is synchronized after every updating method. `GDBM_NOLOCK' means a database is opened without file locking. `GDBM_LOCKNB' means file locking is performed without blocking. `GDBM_FAST' is ignored. `GDBM_SPARSE' is an original mode of QDBM and makes database a sparse file. `mode' specifies mode of a database file as the one of `open' call does. `fatal_func' is ignored. The return value is the database handle or `NULL' if it is not successful.

The function `gdbm_open2' is used in order to get a database handle after the fashion of QDBM.

GDBM_FILE gdbm_open2(char *name, int read_write, int mode, int bnum, int dnum, int align);
`name' specifies the name of a database. `read_write' specifies the connection mode: `GDBM_READER' as a reader, `GDBM_WRITER', `GDBM_WRCREAT' and `GDBM_NEWDB' as a writer. `GDBM_WRCREAT' makes a database file or directory if it does not exist. `GDBM_NEWDB' makes a new database even if it exists. You can add the following to writer modes by bitwise or: `GDBM_SYNC', `GDBM_NOLOCK', 'GDBM_LOCKNB', `GDBM_FAST', and `GDBM_SPARSE'. `GDBM_SYNC' means a database is synchronized after every updating method. `GDBM_NOLOCK' means a database is opened without file locking. `GDBM_LOCKNB' means file locking is performed without blocking. `GDBM_FAST' is ignored. `GDBM_SPARSE' is an original mode of QDBM and makes database sparse files. `mode' specifies a mode of a database file or a database directory as the one of `open' or `mkdir' call does. `bnum' specifies the number of elements of each bucket array. If it is not more than 0, the default value is specified. `dnum' specifies the number of division of the database. If it is not more than 0, the returning handle is created as a wrapper of Depot, else, it is as a wrapper of Curia. `align' specifies the basic size of alignment. The return value is the database handle or `NULL' if it is not successful. If the database already exists, whether it is one of Depot or Curia is measured automatically.

The function `gdbm_close' is used in order to close a database handle.

void gdbm_close(GDBM_FILE dbf);
`dbf' specifies a database handle. Because the region of the closed handle is released, it becomes impossible to use the handle.

The function `gdbm_store' is used in order to store a record.

int gdbm_store(GDBM_FILE dbf, datum key, datum content, int flag);
`dbf' specifies a database handle connected as a writer. `key' specifies a structure of a key. `content' specifies a structure of a value. `flag' specifies behavior when the key overlaps, by the following values: `GDBM_REPLACE', which means the specified value overwrites the existing one, `GDBM_INSERT', which means the existing value is kept. The return value is 0 if it is successful, 1 if it gives up because of overlaps of the key, -1 if other error occurs.

The function `gdbm_delete' is used in order to delete a record.

int gdbm_delete(GDBM_FILE dbf, datum key);
`dbf' specifies a database handle connected as a writer. `key' specifies a structure of a key. The return value is 0 if it is successful, -1 if some errors occur.

The function `gdbm_fetch' is used in order to retrieve a record.

datum gdbm_fetch(GDBM_FILE dbf, datum key);
`dbf' specifies a database handle. `key' specifies a structure of a key. The return value is a structure of the result. If a record corresponds, the member `dptr' of the structure is the pointer to the region of the value. If no record corresponds or some errors occur, `dptr' is `NULL'. Because the region pointed to by `dptr' is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `gdbm_exists' is used in order to check whether a record exists or not.

int gdbm_exists(GDBM_FILE dbf, datum key);
`dbf' specifies a database handle. `key' specifies a structure of a key. The return value is true if a record corresponds and no error occurs, or false, else, it is false.

The function `gdbm_firstkey' is used in order to get the first key of a database.

datum gdbm_firstkey(GDBM_FILE dbf);
`dbf' specifies a database handle. The return value is a structure of the result. If a record corresponds, the member `dptr' of the structure is the pointer to the region of the first key. If no record corresponds or some errors occur, `dptr' is `NULL'. Because the region pointed to by `dptr' is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `gdbm_nextkey' is used in order to get the next key of a database.

datum gdbm_nextkey(GDBM_FILE dbf, datum key);
`dbf' specifies a database handle. The return value is a structure of the result. If a record corresponds, the member `dptr' of the structure is the pointer to the region of the next key. If no record corresponds or some errors occur, `dptr' is `NULL'. Because the region pointed to by `dptr' is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `gdbm_sync' is used in order to synchronize updating contents with the file and the device.

void gdbm_sync(GDBM_FILE dbf);
`dbf' specifies a database handle connected as a writer.

The function `gdbm_reorganize' is used in order to reorganize a database.

int gdbm_reorganize(GDBM_FILE dbf);
`dbf' specifies a database handle connected as a writer. If successful, the return value is 0, else -1.

The function `gdbm_fdesc' is used in order to get the file descriptor of a database file.

int gdbm_fdesc(GDBM_FILE dbf);
`dbf' specifies a database handle connected as a writer. The return value is the file descriptor of the database file. If the database is a directory the return value is -1.

The function `gdbm_setopt' has no effect.

int gdbm_setopt(GDBM_FILE dbf, int option, int *value, int size);
`dbf' specifies a database handle. `option' is ignored. `size' is ignored. The return value is 0. The function is only for compatibility.

Examples

The following example stores and retrieves a phone number, using the name as the key.

#include <hovel.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
#include <string.h>

#define NAME     "mikio"
#define NUMBER   "000-1234-5678"
#define DBNAME   "book"

int main(int argc, char **argv){
  GDBM_FILE dbf;
  datum key, val;
  int i;

  /* open the database */
  if(!(dbf = gdbm_open(DBNAME, 0, GDBM_WRCREAT, 00644, NULL))){
    fprintf(stderr, "gdbm_open: %s\n", gdbm_strerror(gdbm_errno));
    return 1;
  }

  /* prepare the record */
  key.dptr = NAME;
  key.dsize = strlen(NAME);
  val.dptr = NUMBER;
  val.dsize = strlen(NUMBER);

  /* store the record */
  if(gdbm_store(dbf, key, val, GDBM_REPLACE) != 0){
    fprintf(stderr, "gdbm_store: %s\n", gdbm_strerror(gdbm_errno));
  }

  /* retrieve the record */
  val = gdbm_fetch(dbf, key);
  if(val.dptr){
    printf("Name: %s\n", NAME);
    printf("Number: ");
    for(i = 0; i < val.dsize; i++){
      putchar(val.dptr[i]);
    }
    putchar('\n');
    free(val.dptr);
  } else {
    fprintf(stderr, "gdbm_fetch: %s\n", gdbm_strerror(gdbm_errno));
  }

  /* close the database */
  gdbm_close(dbf);

  return 0;
}

Notes

How to build programs using Hovel is the same as the case of Depot. Note that an option to be given to a linker is not `-lgdbm', but `-lqdbm'.

gcc -I/usr/local/include -o sample sample.c -L/usr/local/lib -lqdbm

If QDBM was built with POSIX thread enabled, the global variable `gdbm_errno' is treated as thread specific data, and functions of Hovel are reentrant. In that case, they are thread-safe as long as a handle is not accessed by threads at the same time, on the assumption that `errno', `malloc', and so on are thread-safe.

Commands for Hovel

Hovel has the following command line interfaces.

The command `hvmgr' is a utility for debugging Hovel and its applications. It features editing and checking of a database. It can be used for database applications with shell scripts. This command is used in the following format. `name' specifies a database name. `key' specifies the key of a record. `val' specifies the value of a record.

hvmgr create [-qdbm bnum dnum] [-s] name
Create a database file.
hvmgr store [-qdbm] [-kx] [-vx|-vf] [-insert] name key val
Store a record with a key and a value.
hvmgr delete [-qdbm] [-kx] name key
Delete a record with a key.
hvmgr fetch [-qdbm] [-kx] [-ox] [-n] name key
Retrieve a record with a key and output to the standard output.
hvmgr list [-qdbm] [-ox] name
List all keys and values delimited with tab and line-feed to the standard output.
hvmgr optimize [-qdbm] name
Optimize a database.

Options feature the following.

This command returns 0 on success, another on failure.

The command `hvtest' is a utility for facility test and performance test. Check a database generated by the command or measure the execution time of the command. This command is used in the following format. `name' specifies a database name. `rnum' specifies the number of records.

hvtest write [-qdbm] [-s] name rnum
Store records with keys of 8 bytes. They changes as `00000001', `00000002'...
hvtest read [-qdbm] name rnum
Retrieve records of the database above.

Options feature the following.

This command returns 0 on success, another on failure.

Cabin: Utility API

Overview

Cabin is the utility API which provides memory allocating functions, sorting functions, extensible datum, array list, hash map, and so on for handling records easily on memory. This API features also parsing MIME, CSV, and XML, and features various types of encoding and decoding.

In order to use Cabin, you should include `cabin.h' and `stdlib.h' in the source files. Usually, the following description will be near the beginning of a source file.

#include <cabin.h>
#include <stdlib.h>

A pointer to `CBDATUM' is used as a handle of an extensible datum. A datum handle is opened with the function `cbdatumopen' and closed with `cbdatumclose'. A pointer to `CBLIST' is used as a handle of an array list. A list handle is opened with the function `cblistopen' and closed with `cblistclose'. A pointer to `CBMAP' is used as a handle of a hash map. A map handle is opened with the function `cbmapopen' and closed with `cbmapclose'. You should not refer directly to any member of each handles.

API

The external variable `cbfatalfunc' is the pointer to call back function for handling a fatal error.

extern void (*cbfatalfunc)(const char *);
The argument specifies the error message. The initial value of this variable is `NULL'. If the value is `NULL', the default function is called when a fatal error occurs. A fatal error occurs when memory allocation is failed.

The function `cbmalloc' is used in order to allocate a region on memory.

void *cbmalloc(size_t size);
`size' specifies the size of the region. The return value is the pointer to the allocated region. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbrealloc' is used in order to re-allocate a region on memory.

void *cbrealloc(void *ptr, size_t size);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. The return value is the pointer to the re-allocated region. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `memdup' is used in order to duplicate a region on memory.

char *cbmemdup(const char *ptr, int size);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. The return value is the pointer to the allocated region of the duplicate. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbglobalgc' is used in order to register the pointer or handle of an object to the global garbage collector.

void cbglobalgc(void *ptr, void (*func)(void *));
`ptr' specifies the pointer or handle of an object. `func' specifies the pointer to a function to release resources of the object. Its argument is the pointer or handle of the object to release. This function assures that resources of an object are released when the process exits normally by returning from the `main' function or calling the `exit' function.

The function `cbggcsweep' is used in order to exercise the global garbage collector explicitly.

void cbggcsweep(void);
Note that you should not use objects registered to the global garbage collector any longer after calling this function. Because the global garbage collector is initialized and you can register new objects into it.

The function `cbisort' is used in order to sort an array using insert sort.

void cbisort(void *base, int nmemb, int size, int(*compar)(const void *, const void *));
`base' specifies the pointer to an array. `nmemb' specifies the number of elements of the array. `size' specifies the size of each element. `compar' specifies the pointer to comparing function. The two arguments specify the pointers of elements. The comparing function should returns positive if the former is big, negative if the latter is big, 0 if both are equal. Insert sort is useful only if most elements have been sorted already.

The function `cbssort' is used in order to sort an array using shell sort.

void cbssort(void *base, int nmemb, int size, int(*compar)(const void *, const void *));
`base' specifies the pointer to an array. `nmemb' specifies the number of elements of the array. `size' specifies the size of each element. `compar' specifies the pointer to comparing function. The two arguments specify the pointers of elements. The comparing function should returns positive if the former is big, negative if the latter is big, 0 if both are equal. If most elements have been sorted, shell sort may be faster than heap sort or quick sort.

The function `cbhsort' is used in order to sort an array using heap sort.

void cbhsort(void *base, int nmemb, int size, int(*compar)(const void *, const void *));
`base' specifies the pointer to an array. `nmemb' specifies the number of elements of the array. `size' specifies the size of each element. `compar' specifies the pointer to comparing function. The two arguments specify the pointers of elements. The comparing function should returns positive if the former is big, negative if the latter is big, 0 if both are equal. Although heap sort is robust against bias of input, quick sort is faster in most cases.

The function `cbqsort' is used in order to sort an array using quick sort.

void cbqsort(void *base, int nmemb, int size, int(*compar)(const void *, const void *));
`base' specifies the pointer to an array. `nmemb' specifies the number of elements of the array. `size' specifies the size of each element. `compar' specifies the pointer to comparing function. The two arguments specify the pointers of elements. The comparing function should returns positive if the former is big, negative if the latter is big, 0 if both are equal. Being sensitive to bias of input, quick sort is the fastest sorting algorithm.

The function `cbstricmp' is used in order to compare two strings with case insensitive evaluation.

int cbstricmp(const char *astr, const char *bstr);
`astr' specifies the pointer of one string. `astr' specifies the pointer of the other string. The return value is positive if the former is big, negative if the latter is big, 0 if both are equivalent. Upper cases and lower cases of alphabets in ASCII code are not distinguished.

The function `cbstrfwmatch' is used in order to check whether a string begins with a key.

int cbstrfwmatch(const char *str, const char *key);
`str' specifies the pointer of a target string. `key' specifies the pointer of a forward matching key string. The return value is true if the target string begins with the key, else, it is false.

The function `cbstrfwimatch' is used in order to check whether a string begins with a key, with case insensitive evaluation.

int cbstrfwimatch(const char *str, const char *key);
`str' specifies the pointer of a target string. `key' specifies the pointer of a forward matching key string. The return value is true if the target string begins with the key, else, it is false. Upper cases and lower cases of alphabets in ASCII code are not distinguished.

The function `cbstrbwmatch' is used in order to check whether a string ends with a key.

int cbstrbwmatch(const char *str, const char *key);
`str' specifies the pointer of a target string. `key' specifies the pointer of a backward matching key string. The return value is true if the target string ends with the key, else, it is false.

The function `cbstrbwimatch' is used in order to check whether a string ends with a key, with case insensitive evaluation.

int cbstrbwimatch(const char *str, const char *key);
`str' specifies the pointer of a target string. `key' specifies the pointer of a backward matching key string. The return value is true if the target string ends with the key, else, it is false. Upper cases and lower cases of alphabets in ASCII code are not distinguished.

The function `cbstrstrkmp' is used in order to locate a substring in a string using KMP method.

char *cbstrstrkmp(const char *haystack, const char *needle);
`haystack' specifies the pointer of a target string. `needle' specifies the pointer of a substring to be found. The return value is the pointer to the beginning of the substring or `NULL' if the substring is not found. In most cases, `strstr' as a built-in function of the compiler is faster than this function.

The function `cbstrstrkmp' is used in order to locate a substring in a string using BM method.

char *cbstrstrbm(const char *haystack, const char *needle);
`haystack' specifies the pointer of a target string. `needle' specifies the pointer of a substring to be found. The return value is the pointer to the beginning of the substring or `NULL' if the substring is not found. In most cases, `strstr' as a built-in function of the compiler is faster than this function.

The function `cbstrtoupper' is used in order to convert the letters of a string to upper case.

char *cbstrtoupper(char *str);
`str' specifies the pointer of a string to convert. The return value is the pointer to the string.

The function `cbstrtolower' is used in order to convert the letters of a string to lower case.

char *cbstrtolower(char *str);
`str' specifies the pointer of a string to convert. The return value is the pointer to the string.

The function `cbstrtrim' is used in order to cut space characters at head or tail of a string.

char *cbstrtrim(char *str);
`str' specifies the pointer of a string to convert. The return value is the pointer to the string.

The function `cbstrsqzspc' is used in order to squeeze space characters in a string and trim it.

char *cbstrsqzspc(char *str);
`str' specifies the pointer of a string to convert. The return value is the pointer to the string.

The function `cbdatumopen' is used in order to get a datum handle.

CBDATUM *cbdatumopen(const char *ptr, int size);
`ptr' specifies the pointer to the region of the initial content. If it is `NULL', an empty datum is created. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. The return value is a datum handle.

The function `cbdatumdup' is used in order to copy a datum.

CBDATUM *cbdatumdup(const CBDATUM *datum);
`datum' specifies a datum handle. The return value is a new datum handle.

The function `cbdatumclose' is used in order to free a datum handle.

void cbdatumclose(CBDATUM *datum);
`datum' specifies a datum handle. Because the region of a closed handle is released, it becomes impossible to use the handle.

The function `cbdatumcat' is used in order to concatenate a datum and a region.

void cbdatumcat(CBDATUM *datum, const char *ptr, int size);
`datum' specifies a datum handle. `ptr' specifies the pointer to the region to be appended. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'.

The function `cbdatumptr' is used in order to get the pointer of the region of a datum.

const char *cbdatumptr(const CBDATUM *datum);
`datum' specifies a datum handle. The return value is the pointer of the region of a datum. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string.

The function `cbdatumsize' is used in order to get the size of the region of a datum.

int cbdatumsize(const CBDATUM *datum);
`datum' specifies a datum handle. The return value is the size of the region of a datum.

The function `cbdatumsetsize' is used in order to change the size of the region of a datum.

void cbdatumsetsize(CBDATUM *datum, int size);
`datum' specifies a datum handle. `size' specifies the new size of the region. If the new size is bigger than the one of old, the surplus region is filled with zero codes.

The function `cbdatumtomalloc' is used in order to convert a datum to an allocated region.

char *cbdatumtomalloc(CBDATUM *datum, int *sp);
`datum' specifies a datum handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer to the region of the datum. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. Because the region of the original datum is released, it should not be released again.

The function `cblistopen' is used in order to get a list handle.

CBLIST *cblistopen(void);
The return value is a list handle.

The function `cblistdup' is used in order to copy a list.

CBLIST *cblistdup(const CBLIST *list);
`list' specifies a list handle. The return value is a new list handle.

The function `cblistclose' is used in order to close a list handle.

void cblistclose(CBLIST *list);
`list' specifies a list handle. Because the region of a closed handle is released, it becomes impossible to use the handle.

The function `cblistnum' is used in order to get the number of elements of a list.

int cblistnum(const CBLIST *list);
`list' specifies a list handle. The return value is the number of elements of the list.

The function `cblistval' is used in order to get the pointer to the region of an element.

const char *cblistval(const CBLIST *list, int index, int *sp);
`list' specifies a list handle. `index' specifies the index of an element. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer to the region of the element. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. If `index' is equal to or more than the number of elements, the return value is `NULL'.

The function `cblistpush' is used in order to add an element at the end of a list.

void cblistpush(CBLIST *list, const char *ptr, int size);
`list' specifies a list handle. `ptr' specifies the pointer to the region of an element. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'.

The function `cblistpop' is used in order to remove an element of the end of a list.

char *cblistpop(CBLIST *list, int *sp);
`list' specifies a list handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer to the region of the value. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. If the list is empty, the return value is `NULL'.

The function `cblistunshift' is used in order to add an element at the top of a list.

void cblistunshift(CBLIST *list, const char *ptr, int size);
`list' specifies a list handle. `ptr' specifies the pointer to the region of an element. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'.

The function `cblistshift' is used in order to remove an element of the top of a list.

char *cblistshift(CBLIST *list, int *sp);
`list' specifies a list handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer to the region of the value. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. If the list is empty, the return value is `NULL'.

The function `cblistinsert' is used in order to add an element at the specified location of a list.

void cblistinsert(CBLIST *list, int index, const char *ptr, int size);
`list' specifies a list handle. `index' specifies the index of an element. `ptr' specifies the pointer to the region of the element. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'.

The function `cblistremove' is used in order to remove an element at the specified location of a list.

char *cblistremove(CBLIST *list, int index, int *sp);
`list' specifies a list handle. `index' specifies the index of an element. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer to the region of the value. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. If `index' is equal to or more than the number of elements, no element is removed and the return value is `NULL'.

The function `cblistover' is used in order to overwrite an element at the specified location of a list.

void cblistover(CBLIST *list, int index, const char *ptr, int size);
`list' specifies a list handle. `index' specifies the index of an element. `ptr' specifies the pointer to the region of the new content. `size' specifies the size of the new content. If it is negative, the size is assigned with `strlen(ptr)'. If `index' is equal to or more than the number of elements, this function has no effect.

The function `cblistsort' is used in order to sort elements of a list in lexical order.

void cblistsort(CBLIST *list);
`list' specifies a list handle. Quick sort is used for sorting.

The function `cblistlsearch' is used in order to search a list for an element using liner search.

int cblistlsearch(const CBLIST *list, const char *ptr, int size);
`list' specifies a list handle. `ptr' specifies the pointer to the region of a key. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. The return value is the index of a corresponding element or -1 if there is no corresponding element. If two or more elements corresponds, the former returns.

The function `cblistbsearch' is used in order to search a list for an element using binary search.

int cblistbsearch(const CBLIST *list, const char *ptr, int size);
`list' specifies a list handle. It should be sorted in lexical order. `ptr' specifies the pointer to the region of a key. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. The return value is the index of a corresponding element or -1 if there is no corresponding element. If two or more elements corresponds, which returns is not defined.

The function `cblistdump' is used in order to serialize a list into a byte array.

char *cblistdump(const CBLIST *list, int *sp);
`list' specifies a list handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. The return value is the pointer to the region of the result serial region. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cblistload' is used in order to redintegrate a serialized list.

CBLIST *cblistload(const char *ptr, int size);
`ptr' specifies the pointer to a byte array. `size' specifies the size of the region. The return value is a new list handle.

The function `cbmapopen' is used in order to get a map handle.

CBMAP *cbmapopen(void);
The return value is a map handle.

The function `cbmapdup' is used in order to copy a map.

CBMAP *cbmapdup(CBMAP *map);
`map' specifies a map handle. The return value is a new map handle. The iterator of the source map is initialized.

The function `cbmapclose' is used in order to close a map handle.

void cbmapclose(CBMAP *map);
`map' specifies a map handle. Because the region of a closed handle is released, it becomes impossible to use the handle.

The function `cbmapput' is used in order to store a record.

int cbmapput(CBMAP *map, const char *kbuf, int ksiz, const char *vbuf, int vsiz, int over);
`map' specifies a map handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `vbuf' specifies the pointer to the region of a value. `vsiz' specifies the size of the region of the value. If it is negative, the size is assigned with `strlen(vbuf)'. `over' specifies whether the value of the duplicated record is overwritten or not. If `over' is false and the key duplicated, the return value is false, else, it is true.

The function `cbmapputcat' is used in order to concatenate a value at the end of the value of the existing record.

void cbmapputcat(CBMAP *map, const char *kbuf, int ksiz, const char *vbuf, int vsiz);
`map' specifies a map handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `vbuf' specifies the pointer to the region of a value. `vsiz' specifies the size of the region of the value. If it is negative, the size is assigned with `strlen(vbuf)'. If there is no corresponding record, a new record is created.

The function `cbmapout' is used in order to delete a record.

int cbmapout(CBMAP *map, const char *kbuf, int ksiz);
`map' specifies a map handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is true. False is returned when no record corresponds to the specified key.

The function `cbmapget' is used in order to retrieve a record.

const char *cbmapget(const CBMAP *map, const char *kbuf, int ksiz, int *sp);
`map' specifies a map handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the value of the corresponding record. `NULL' is returned when no record corresponds. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string.

The function `cbmapmove' is used in order to move a record to the edge.

int cbmapmove(CBMAP *map, const char *kbuf, int ksiz, int head);
`map' specifies a map handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `head' specifies the destination which is head if it is true or tail if else. If successful, the return value is true. False is returned when no record corresponds to the specified key.

The function `cbmapiterinit' is used in order to initialize the iterator of a map handle.

void cbmapiterinit(CBMAP *map);
`map' specifies a map handle. The iterator is used in order to access the key of every record stored in a map.

The function `cbmapiternext' is used in order to get the next key of the iterator.

const char *cbmapiternext(CBMAP *map, int *sp);
`map' specifies a map handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the next key, else, it is `NULL'. `NULL' is returned when no record is to be get out of the iterator. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. The order of iteration is assured to be the same of the one of storing.

The function `cbmaprnum' is used in order to get the number of the records stored in a map.

int cbmaprnum(const CBMAP *map);
`map' specifies a map handle. The return value is the number of the records stored in the map.

The function `cbmapkeys' is used in order to get the list handle contains all keys in a map.

CBLIST *cbmapkeys(CBMAP *map);
`map' specifies a map handle. The return value is the list handle contains all keys in the map. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use.

The function `cbmapvals' is used in order to get the list handle contains all values in a map.

CBLIST *cbmapvals(CBMAP *map);
`map' specifies a map handle. The return value is the list handle contains all values in the map. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use.

The function `cbmapdump' is used in order to serialize a map into a byte array.

char *cbmapdump(const CBMAP *map, int *sp);
`map' specifies a map handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. The return value is the pointer to the region of the result serial region. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbmapload' is used in order to redintegrate a serialized map.

CBMAP *cbmapload(const char *ptr, int size);
`ptr' specifies the pointer to a byte array. `size' specifies the size of the region. The return value is a new map handle.

The function `cbmaploadone' is used in order to extract a record from a serialized map.

char *cbmaploadone(const char *ptr, int size, const char *kbuf, int ksiz, int *sp);
`ptr' specifies the pointer to a byte array. `size' specifies the size of the region. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the value of the corresponding record. `NULL' is returned when no record corresponds. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string.

The function `cbsprintf' is used in order to allocate a formatted string on memory.

char *cbsprintf(const char *format, ...);
`format' specifies a printf-like format string. The conversion character `%' can be used with such flag characters as `d', `o', `u', `x', `X', `e', `E', `f', `g', `G', `c', `s', and `%'. Specifiers of the field length and the precision can be put between the conversion characters and the flag characters. The specifiers consist of decimal characters, `.', `+', `-', and the space character. The other arguments are used according to the format string. The return value is the pointer to the allocated region of the result string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbreplace' is used in order to replace some patterns in a string.

char *cbreplace(const char *str, CBMAP *pairs);
`str' specifies the pointer to a source string. `pairs' specifies the handle of a map composed of pairs of replacement. The key of each pair specifies a pattern before replacement and its value specifies the pattern after replacement. The return value is the pointer to the allocated region of the result string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbsplit' is used in order to make a list by splitting a serial datum.

CBLIST *cbsplit(const char *ptr, int size, const char *delim);
`ptr' specifies the pointer to the region of the source content. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. `delim' specifies a string containing delimiting characters. If it is `NULL', zero code is used as a delimiter. The return value is a list handle. If two delimiters are successive, it is assumed that an empty element is between the two. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose'.

The function `cbreadfile' is used in order to read whole data of a file.

char *cbreadfile(const char *name, int *sp);
`name' specifies the name of a file. If it is `NULL', the standard input is specified. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer to the allocated region of the read data. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbwritefile' is used in order to write a serial datum into a file.

int cbwritefile(const char *name, const char *ptr, int size);
`name specifies the name of a file. If it is `NULL', the standard output is specified. `ptr' specifies the pointer to the region of the source content. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. If successful, the return value is true, else, it is false. If the file exists, it is overwritten. Else, a new file is created.

The function `cbreadlines' is used in order to read every line of a file.

CBLIST *cbreadlines(const char *name);
`name' specifies the name of a file. If it is `NULL', the standard input is specified. The return value is a list handle of the lines if successful, else it is NULL. Line separators are cut out. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use.

The function `cbdirlist' is used in order to read names of files in a directory.

CBLIST *cbdirlist(const char *name);
`name' specifies the name of a directory. The return value is a list handle of names if successful, else it is NULL. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use.

The function `cbfilestat' is used in order to get the status of a file or a directory.

int cbfilestat(const char *name, int *isdirp, int *sizep, int *mtimep);
`name' specifies the name of a file or a directory. `dirp' specifies the pointer to a variable to which whether the file is a directory is assigned. If it is `NULL', it is not used. `sizep' specifies the pointer to a variable to which the size of the file is assigned. If it is `NULL', it is not used. `mtimep' specifies the pointer to a variable to which the last modified time of the file is assigned. If it is `NULL', it is not used. If successful, the return value is true, else, false. False is returned when the file does not exist or the permission is denied.

The function `cburlbreak' is used in order to break up a URL into elements.

CBMAP *cburlbreak(const char *str);
`str' specifies the pointer to a string of URL. The return value is a map handle. Each key of the map is the name of an element. The key "self" specifies the URL itself. The key "scheme" specifies the scheme. The key "host" specifies the host of the server. The key "port" specifies the port number of the server. The key "authority" specifies the authority information. The key "path" specifies the path of the resource. The key "file" specifies the file name without the directory section. The key "query" specifies the query string. The key "fragment" specifies the fragment string. Supported schema are HTTP, HTTPS, FTP, and FILE. Absolute URL and relative URL are supported. Because the handle of the return value is opened with the function `cbmapopen', it should be closed with the function `cbmapclose' if it is no longer in use.

The function `cburlencode' is used in order to encode a serial object with URL encoding.

char *cburlencode(const char *ptr, int size);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. The return value is the pointer to the result string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cburldecode' is used in order to decode a string encoded with URL encoding.

char *cburldecode(const char *str, int *sp);
`str' specifies the pointer to a source string. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer to the region of the result. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbbaseencode' is used in order to encode a serial object with Base64 encoding.

char *cbbaseencode(const char *ptr, int size);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. The return value is the pointer to the result string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbbasedecode' is used in order to decode a string encoded with Base64 encoding.

char *cbbasedecode(const char *str, int *sp);
`str' specifies the pointer to a source string. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer to the region of the result. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbquoteencode' is used in order to encode a serial object with quoted-printable encoding.

char *cbquoteencode(const char *ptr, int size);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. The return value is the pointer to the result string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbquotedecode' is used in order to decode a string encoded with quoted-printable encoding.

char *cbquotedecode(const char *str, int *sp);
`str' specifies the pointer to a source string. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer to the region of the result. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbmimebreak' is used in order to split a string of MIME into headers and the body.

char *cbmimebreak(const char *ptr, int size, CBMAP *attrs, int *sp);
`ptr' specifies the pointer to the region of MIME data. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. `attrs' specifies a map handle to store attributes. If it is `NULL', it is not used. Each key of the map is an attribute name uncapitalized. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. The return value is the pointer of the body data. If the content type is defined, the attribute map has the key "TYPE" specifying the type. If the character encoding is defined, the key "CHARSET" specifies the encoding name. If the boundary string of multipart is defined, the key "BOUNDARY" specifies the string. If the content disposition is defined, the key "DISPOSITION" specifies the direction. If the file name is defined, the key "FILENAME" specifies the name. If the attribute name is defined, the key "NAME" specifies the name. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbmimeparts' is used in order to split multipart data of MIME into its parts.

CBLIST *cbmimeparts(const char *ptr, int size, const char *boundary);
`ptr' specifies the pointer to the region of multipart data of MIME. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. `boundary' specifies the pointer to the region of the boundary string. The return value is a list handle. Each element of the list is the string of a part. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use.

The function `cbmimeencode' is used in order to encode a string with MIME encoding.

char *cbmimeencode(const char *str, const char *encname, int base);
`str' specifies the pointer to a string. `encname' specifies a string of the name of the character encoding. The return value is the pointer to the result string. `base' specifies whether to use Base64 encoding. If it is false, quoted-printable is used. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbmimedecode' is used in order to decode a string encoded with MIME encoding.

char *cbmimedecode(const char *str, char *enp);
`str' specifies the pointer to an encoded string. `enp' specifies the pointer to a region into which the name of encoding is written. If it is `NULL', it is not used. The size of the buffer should be equal to or more than 32 bytes. The return value is the pointer to the result string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbcsvrows' is used in order to split a string of CSV into rows.

CBLIST *cbcsvrows(const char *str);
`str' specifies the pointer to the region of an CSV string. The return value is a list handle. Each element of the list is a string of a row. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use. The character encoding of the input string should be US-ASCII, UTF-8, ISO-8859-*, EUC-*, or Shift_JIS. Being compatible with MS-Excel, these functions for CSV can handle cells including such meta characters as comma, between double quotation marks.

The function `cbcsvcells' is used in order to split the string of a row of CSV into cells.

CBLIST *cbcsvcells(const char *str);
`str' specifies the pointer to the region of a row of CSV. The return value is a list handle. Each element of the list is the unescaped string of a cell of the row. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use.

The function `cbcsvescape' is used in order to escape a string with the meta characters of CSV.

char *cbcsvescape(const char *str);
`str' specifies the pointer to the region of a string. The return value is the pointer to the escaped string sanitized of meta characters. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbcsvunescape' is used in order to unescape a string with the escaped meta characters of CSV.

char *cbcsvunescape(const char *str);
`str' specifies the pointer to the region of a string with meta characters. The return value is the pointer to the unescaped string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbxmlbreak' is used in order to split a string of XML into tags and text sections.

CBLIST *cbxmlbreak(const char *str, int cr);
`str' specifies the pointer to the region of an XML string. `cr' specifies whether to remove comments. The return value is a list handle. Each element of the list is the string of a tag or a text section. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use. The character encoding of the input string should be US-ASCII, UTF-8, ISO-8859-*, EUC-*, or Shift_JIS. Because these functions for XML are not XML parser with validation check, it can handle also HTML and SGML.

The function `cbxmlattrs' is used in order to get the map of attributes of an XML tag.

CBMAP *cbxmlattrs(const char *str);
`str' specifies the pointer to the region of a tag string. The return value is a map handle. Each key of the map is the name of an attribute. Each value is unescaped. You can get the name of the tag with the key of an empty string. Because the handle of the return value is opened with the function `cbmapopen', it should be closed with the function `cbmapclose' if it is no longer in use.

The function `cbxmlescape' is used in order to escape a string with the meta characters of XML.

char *cbxmlescape(const char *str);
`str' specifies the pointer to the region of a string. The return value is the pointer to the escaped string sanitized of meta characters. This function converts only `&', `<', `>', and `"'. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbxmlunescape' is used in order to unescape a string with the entity references of XML.

char *cbxmlunescape(const char *str);
`str' specifies the pointer to the region of a string. The return value is the pointer to the unescaped string. This function restores only `&amp;', `&lt;', `&gt;', and `&quot;'. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbdeflate' is used in order to compress a serial object with ZLIB.

char *cbdeflate(const char *ptr, int size, int *sp);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If successful, the return value is the pointer to the result object, else, it is `NULL'. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. This function is available only if QDBM was built with ZLIB enabled.

The function `cbinflate' is used in order to decompress a serial object compressed with ZLIB.

char *cbinflate(const char *ptr, int size, int *sp);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the result object, else, it is `NULL'. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. This function is available only if QDBM was built with ZLIB enabled.

The function `cbgzencode' is used in order to compress a serial object with GZIP.

char *cbgzencode(const char *ptr, int size, int *sp);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If successful, the return value is the pointer to the result object, else, it is `NULL'. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. This function is available only if QDBM was built with ZLIB enabled.

The function `cbgzdecode' is used in order to decompress a serial object compressed with GZIP.

char *cbgzdecode(const char *ptr, int size, int *sp);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the result object, else, it is `NULL'. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. This function is available only if QDBM was built with ZLIB enabled.

The function `cbgetcrc' is used in order to get the CRC32 checksum of a serial object.

unsigned int cbgetcrc(const char *ptr, int size);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. The return value is the CRC32 checksum of the object. This function is available only if QDBM was built with ZLIB enabled.

The function `cbiconv' is used in order to convert the character encoding of a string.

char *cbiconv(const char *ptr, int size, const char *icode, const char *ocode, int *sp, int *mp);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. `icode' specifies the name of encoding of the input string. `ocode' specifies the name of encoding of the output string. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. `mp' specifies the pointer to a variable to which the number of missing characters by failure of conversion is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the result object, else, it is `NULL'. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. This function is available only if QDBM was built with ICONV enabled.

The function `cbencname' is used in order to detect the encoding of a string automatically.

const char *cbencname(const char *ptr, int size);
`ptr' specifies the pointer to a region. `size' specifies the size of the region. If it is negative, the size is assigned with `strlen(ptr)'. The return value is the string of the encoding name of the string. As it stands, US-ASCII, ISO-2022-JP, Shift_JIS, CP932, EUC-JP, UTF-8, UTF-16, UTF-16BE, and UTF-16LE are supported. If none of them matches, ISO-8859-1 is selected. This function is available only if QDBM was built with ICONV enabled.

The function `cbjetlag' is used in order to get the jet lag of the local time in seconds.

int cbjetlag(void);
The return value is the jet lag of the local time in seconds.

The function `cbcalendar' is used in order to get the Gregorian calendar of a time.

void cbcalendar(time_t t, int jl, int *yearp, int *monp, int *dayp, int *hourp, int *minp, int *secp);
`t' specifies a source time. If it is negative, the current time is specified. `jl' specifies the jet lag of a location in seconds. `yearp' specifies the pointer to a variable to which the year is assigned. If it is `NULL', it is not used. `monp' specifies the pointer to a variable to which the month is assigned. If it is `NULL', it is not used. 1 means January and 12 means December. `dayp' specifies the pointer to a variable to which the day of the month is assigned. If it is `NULL', it is not used. `hourp' specifies the pointer to a variable to which the hours is assigned. If it is `NULL', it is not used. `minp' specifies the pointer to a variable to which the minutes is assigned. If it is `NULL', it is not used. `secp' specifies the pointer to a variable to which the seconds is assigned. If it is `NULL', it is not used.

The function `cbdayofweek' is used in order to get the day of week of a date.

int cbdayofweek(int year, int mon, int day);
`year' specifies the year of a date. `mon' specifies the month of the date. `day' specifies the day of the date. The return value is the day of week of the date. 0 means Sunday and 6 means Saturday.

The function `cbdatestrwww' is used in order to get the string for a date in W3CDTF.

char *cbdatestrwww(time_t t, int jl);
`t' specifies a source time. If it is negative, the current time is specified. `jl' specifies the jet lag of a location in seconds. The return value is the string of the date in W3CDTF (YYYY-MM-DDThh:mm:ddTZD). Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbdatestrhttp' is used in order to get the string for a date in RFC 1123 format.

char *cbdatestrhttp(time_t t, int jl);
`t' specifies a source time. If it is negative, the current time is specified. `jl' specifies the jet lag of a location in seconds. The return value is the string of the date in RFC 1123 format (Wdy, DD-Mon-YYYY hh:mm:dd TZD). Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `cbstrmktime' is used in order to get the time value of a date string in decimal, hexadecimal, W3CDTF, or RFC 822 (1123).

time_t cbstrmktime(const char *str);
`str' specifies a date string in decimal, hexadecimal, W3CDTF, or RFC 822 (1123). The return value is the time value of the date or -1 if the format is invalid.

The function `cbproctime' is used in order to get user and system processing times.

void cbproctime(double *usrp, double *sysp);
`usrp' specifies the pointer to a variable to which the user processing time is assigned. If it is `NULL', it is not used. The unit of time is seconds. `sysp' specifies the pointer to a variable to which the system processing time is assigned. If it is `NULL', it is not used. The unit of time is seconds.

The function `cbstdiobin' is used in order to ensure that the standard I/O is binary mode.

void cbstdiobin(void);
This function is useful for applications on dosish file systems.

Examples

The following example is typical use.

#include <cabin.h>
#include <stdlib.h>
#include <stdio.h>

int main(int argc, char **argv){
  CBDATUM *datum;
  CBLIST *list;
  CBMAP *map;
  char *buf1, *buf2;
  int i;

  /* open the datum handle */
  datum = cbdatumopen("123", -1);
  /* concatenate the data */
  cbdatumcat(datum, "abc", -1);
  /* print the datum */
  printf("%s\n", cbdatumptr(datum));
  /* close the datum handle */
  cbdatumclose(datum);

  /* open the list handle */
  list = cblistopen();
  /* add elements into the list */
  cblistpush(list, "apple", -1);
  cblistpush(list, "orange", -1);
  /* print all elements */
  for(i = 0; i < cblistnum(list); i++){
    printf("%s\n", cblistval(list, i, NULL));
  }
  /* close the list handle */
  cblistclose(list);

  /* open the map handle */
  map = cbmapopen();
  /* add records into the map */
  cbmapput(map, "dog", -1, "bowwow", -1, 1);
  cbmapput(map, "cat", -1, "meow", -1, 1);
  /* search for values and print them */
  printf("%s\n", cbmapget(map, "dog", -1, NULL));
  printf("%s\n", cbmapget(map, "cat", -1, NULL));
  /* close the map */
  cbmapclose(map);

  /* Base64 encoding */
  buf1 = cbbaseencode("I miss you.", -1);
  printf("%s\n", buf1);
  /* Base64 decoding */
  buf2 = cbbasedecode(buf1, NULL);
  printf("%s\n", buf2);
  /* release the resources */
  free(buf2);
  free(buf1);

  /* register a plain pointer to the global garbage collector */
  buf1 = cbmemdup("Take it easy.", -1);
  cbglobalgc(buf1, free);
  /* the pointer is available but you don't have to release it */
  printf("%s\n", buf1);
  
  /* register a list to the global garbage collector */
  list = cblistopen();
  cbglobalgc(list, (void (*)(void *))cblistclose);
  /* the handle is available but you don't have to close it */
  cblistpush(list, "Don't hesitate.", -1);
  for(i = 0; i < cblistnum(list); i++){
    printf("%s\n", cblistval(list, i, NULL));    
  }

  return 0;
}

Notes

How to build programs using Cabin is the same as the case of Depot.

gcc -I/usr/local/include -o sample sample.c -L/usr/local/lib -lqdbm

Functions of Cabin except for `cbglobalgc' are thread-safe as long as a handle is not accessed by threads at the same time, on the assumption that `errno', `malloc', and so on are thread-safe.

Commands for Cabin

Cabin has the following command line interfaces.

The command `cbtest' is a utility for facility test and performance test. Measure the execution time of the command. This command is used in the following format. `rnum' specifies the number of records.

cbtest sort [-d] rnum
Perform test of sorting algorithms.
cbtest strstr [-d] rnum
Perform test of string locating algorithms.
cbtest list [-d] rnum
Perform writing test of list.
cbtest map [-d] rnum
Perform writing test of map.
cbtest wicked rnum
Perform updating operations of list and map selected at random.
cbtest misc
Perform test of miscellaneous routines.

Options feature the following.

This command returns 0 on success, another on failure.

The command `cbcodec' is a tool to use encoding and decoding features provided by Cabin. This command is used in the following format. `file' specifies a input file. If it is omitted, the standard input is read.

cbcodec url [-d] [-br] [-l] [-e expr] [file]
Perform URL encoding and its decoding.
cbcodec base [-d] [-l] [-c num] [-e expr] [file]
Perform Base64 encoding and its decoding.
cbcodec quote [-d] [-l] [-c num] [-e expr] [file]
Perform quoted-printable encoding and its decoding.
cbcodec mime [-d] [-hd] [-bd] [-part num] [-l] [-ec code] [-qp] [-dc] [-e expr] [file]
Perform MIME encoding and its decoding.
cbcodec csv [-d] [-t] [-l] [-e expr] [-html] [file]
Process CSV. By default, escape meta characters.
cbcodec xml [-d] [-p] [-l] [-e expr] [-tsv] [file]
Process XML. By default, escape meta characters.
cbcodec zlib [-d] [-gz] [-crc] [file]
Perform deflation and inflation with ZLIB. It is available only if QDBM was built with ZLIB enabled.
cbcodec iconv [-ic code] [-oc code] [-ol ltype] [-cn] [-um] [file]
Convert character encoding with ICONV. It is available only if QDBM was built with ICONV enabled.
cbcodec date [-wf] [-rf] [-utc] [str]
Convert a date string specified `str'. By default, UNIX time is output. If `str' is omitted, the current date is dealt.

Options feature the following.

This command returns 0 on success, another on failure.

Villa: Advanced API

Overview

Villa is the advanced API of QDBM. It provides routines for managing a database file of B+ tree. Each record is stored being sorted in order defined by a user. As for hash databases, retrieving method is provided only as complete accord. However, with Villa, it is possible to retrieve records specified by range. Cursor is used in order to access each record in order. It is possible to store records duplicating keys in a database. Moreover, according to the transaction mechanism, you can commit or abort operations of a database in a lump.

Villa is implemented, based on Depot and Cabin. A database file of Villa is actual one of Depot. Although processing speed of retrieving and storing is slower than Depot, the size of a database is smaller.

In order to use Villa, you should include `depot.h', `cabin.h', `villa.h' and `stdlib.h' in the source files. Usually, the following description will be near the beginning of a source file.

#include <depot.h>
#include <cabin.h>
#include <villa.h>
#include <stdlib.h>

A pointer to `VILLA' is used as a database handle. It is like that some file I/O routines of `stdio.h' use a pointer to `FILE'. A database handle is opened with the function `vlopen' and closed with `vlclose'. You should not refer directly to any member of the handle. If a fatal error occurs in a database, any access method via the handle except `vlclose' will not work and return error status. Although a process is allowed to use multiple database handles at the same time, handles of the same database file should not be used. Before the cursor is used, it should be initialized by one of `vlcurfirst', `vlcurlast' or `vlcurjump'. Also after storing or deleting a record, the cursor should be initialized.

Villa also assign the external variable `dpecode' with the error code. The function `dperrmsg' is used in order to get the message of the error code.

API

You can define a comparing function to specify the order of records. The function should be the following type.

typedef int(*VLCFUNC)(const char *aptr, int asiz, const char *bptr, int bsiz);
`aptr' specifies the pointer to the region of one key. `asiz' specifies the size of the region of one key. `bptr' specifies the pointer to the region of the other key. `bsiz' specifies the size of the region of the other key. The return value is positive if the former is big, negative if the latter is big, 0 if both are equivalent.

The function `vlopen' is used in order to get a database handle.

VILLA *vlopen(const char *name, int omode, VLCFUNC cmp);
`name' specifies the name of a database file. `omode' specifies the connection mode: `VL_OWRITER' as a writer, `VL_OREADER' as a reader. If the mode is `VL_OWRITER', the following may be added by bitwise or: `VL_OCREAT', which means it creates a new database if not exist, `VL_OTRUNC', which means it creates a new database regardless if one exists, `VL_OZCOMP', which means leaves in the database are compressed. Both of `VL_OREADER' and `VL_OWRITER' can be added to by bitwise or: `VL_ONOLCK', which means it opens a database file without file locking, or `VL_OLCKNB', which means locking is performed without blocking. `cmp' specifies the comparing function: `VL_CMPLEX' comparing keys in lexical order, `VL_CMPINT' comparing keys as objects of `int' in native byte order, `VL_CMPNUM' comparing keys as numbers of big endian, `VL_CMPDEC' comparing keys as decimal strings. Any function based on the declaration of the type `VLCFUNC' can be assigned to the comparing function. The comparing function should be kept same in the life of a database. The return value is the database handle or `NULL' if it is not successful. While connecting as a writer, an exclusive lock is invoked to the database file. While connecting as a reader, a shared lock is invoked to the database file. The thread blocks until the lock is achieved. `VL_OZCOMP' is available only if QDBM was built with ZLIB enabled. If `VL_ONOLCK' is used, the application is responsible for exclusion control.

The function `vlclose' is used in order to close a database handle.

int vlclose(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is true, else, it is false. Because the region of a closed handle is released, it becomes impossible to use the handle. Updating a database is assured to be written when the handle is closed. If a writer opens a database but does not close it appropriately, the database will be broken. If the transaction is activated and not committed, it is aborted.

The function `vlput' is used in order to store a record.

int vlput(VILLA *villa, const char *kbuf, int ksiz, const char *vbuf, int vsiz, int dmode);
`villa' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `vbuf' specifies the pointer to the region of a value. `vsiz' specifies the size of the region of the value. If it is negative, the size is assigned with `strlen(vbuf)'. `dmode' specifies behavior when the key overlaps, by the following values: `VL_DOVER', which means the specified value overwrites the existing one, `VL_DKEEP', which means the existing value is kept, `VL_DDUP', which means duplication of keys is allowed. If successful, the return value is true, else, it is false. A duplicated record is stored at the tail of the records of the same key. The cursor becomes unavailable due to updating database.

The function `vlout' is used in order to delete a record.

int vlout(VILLA *villa, const char *kbuf, int ksiz);
`villa' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is true, else, it is false. False is returned when no record corresponds to the specified key. When the key of duplicated records is specified, the first record of the same key is deleted. The cursor becomes unavailable due to updating database.

The function `vlget' is used in order to retrieve a record.

char *vlget(VILLA *villa, const char *kbuf, int ksiz, int *sp);
`villa' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the value of the corresponding record, else, it is `NULL'. `NULL' is returned when no record corresponds to the specified key. When the key of duplicated records is specified, the value of the first record of the same key is selected. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `vlvnum' is used in order to get the number of records corresponding a key.

int vlvnum(VILLA *villa, const char *kbuf, int ksiz);
`villa' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. The return value is the number of corresponding records. If no record corresponds, 0 is returned.

The function `vlputlist' is used in order to store plural records corresponding a key.

int vlputlist(VILLA *villa, const char *kbuf, int ksiz, const CBLIST *vals);
`villa' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `vals' specifies a list handle of values. The list should not be empty. If successful, the return value is true, else, it is false. The cursor becomes unavailable due to updating database.

The function `vloutlist' is used in order to delete all records corresponding a key.

int vloutlist(VILLA *villa, const char *kbuf, int ksiz);
`villa' specifies a database handle connected as a writer. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is true, else, it is false. False is returned when no record corresponds to the specified key. The cursor becomes unavailable due to updating database.

The function `vlgetlist' is used in order to retrieve values of all records corresponding a key.

CBLIST *vlgetlist(VILLA *villa, const char *kbuf, int ksiz);
`villa' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. If successful, the return value is a list handle of the values of the corresponding records, else, it is `NULL'. `NULL' is returned when no record corresponds to the specified key. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use.

The function `vlgetcat' is used in order to retrieve concatenated values of all records corresponding a key.

char *vlgetcat(VILLA *villa, const char *kbuf, int ksiz, int *sp);
`villa' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the concatenated values of the corresponding record, else, it is `NULL'. `NULL' is returned when no record corresponds to the specified key. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `vlcurfirst' is used in order to move the cursor to the first record.

int vlcurfirst(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is true, else, it is false. False is returned if there is no record in the database.

The function `vlcurlast' is used in order to move the cursor to the last record.

int vlcurlast(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is true, else, it is false. False is returned if there is no record in the database.

The function `vlcurprev' is used in order to move the cursor to the previous record.

int vlcurprev(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is true, else, it is false. False is returned if there is no previous record.

The function `vlcurnext' is used in order to move the cursor to the next record.

int vlcurnext(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is true, else, it is false. False is returned if there is no next record.

The function `vlcurjump' is used in order to move the cursor to position around a record.

int vlcurjump(VILLA *villa, const char *kbuf, int ksiz, int jmode);
`villa' specifies a database handle. `kbuf' specifies the pointer to the region of a key. `ksiz' specifies the size of the region of the key. If it is negative, the size is assigned with `strlen(kbuf)'. `jmode' specifies detail adjustment: `VL_JFORWARD', which means that the cursor is set to the first record of the same key and that the cursor is set to the next substitute if completely matching record does not exist, `VL_JBACKWARD', which means that the cursor is set to the last record of the same key and that the cursor is set to the previous substitute if completely matching record does not exist. If successful, the return value is true, else, it is false. False is returned if there is no record corresponding the condition.

The function `vlcurkey' is used in order to get the key of the record where the cursor is.

char *vlcurkey(VILLA *villa, int *sp);
`villa' specifies a database handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the key of the corresponding record, else, it is `NULL'. `NULL' is returned when no record corresponds to the cursor. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `vlcurval' is used in order to get the value of the record where the cursor is.

char *vlcurval(VILLA *villa, int *sp);
`villa' specifies a database handle. `sp' specifies the pointer to a variable to which the size of the region of the return value is assigned. If it is `NULL', it is not used. If successful, the return value is the pointer to the region of the value of the corresponding record, else, it is `NULL'. `NULL' is returned when no record corresponds to the cursor. Because an additional zero code is appended at the end of the region of the return value, the return value can be treated as a character string. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `vlsettuning' is used in order to set the tuning parameters for performance.

void vlsettuning(VILLA *villa, int lrecmax, int nidxmax, int lcnum, int ncnum);
`villa' specifies a database handle. `lrecmax' specifies the max number of records in a leaf node of B+ tree. If it is not more than 0, the default value is specified. `nidxmax' specifies the max number of indexes in a non-leaf node of B+ tree. If it is not more than 0, the default value is specified. `lcnum' specifies the max number of caching leaf nodes. If it is not more than 0, the default value is specified. `ncnum' specifies the max number of caching non-leaf nodes. If it is not more than 0, the default value is specified. The default setting is equivalent to `vlsettuning(49, 192, 1024, 512)'. Because tuning parameters are not saved in a database, you should specify them every opening a database.

The function `vlsync' is used in order to synchronize updating contents with the file and the device.

int vlsync(VILLA *villa);
`villa' specifies a database handle connected as a writer. If successful, the return value is true, else, it is false. This function is useful when another process uses the connected database file. This function should not be used while the transaction is activated.

The function `vloptimize' is used in order to optimize a database.

int vloptimize(VILLA *villa);
`villa' specifies a database handle connected as a writer. If successful, the return value is true, else, it is false. In an alternating succession of deleting and storing with overwrite or concatenate, dispensable regions accumulate. This function is useful to do away with them. This function should not be used while the transaction is activated.

The function `vlname' is used in order to get the name of a database.

char *vlname(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is the pointer to the region of the name of the database, else, it is `NULL'. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `vlfsiz' is used in order to get the size of a database file.

int vlfsiz(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is the size of the database file, else, it is -1. Because of the I/O buffer, the return value may be less than the hard size.

The function `vllnum' is used in order to get the number of the leaf nodes of B+ tree.

int vllnum(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is the number of the leaf nodes, else, it is -1.

The function `vlnnum' is used in order to get the number of the non-leaf nodes of B+ tree.

int vlnnum(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is the number of the non-leaf nodes, else, it is -1.

The function `vlrnum' is used in order to get the number of the records stored in a database.

int vlrnum(VILLA *villa);
`villa' specifies a database handle. If successful, the return value is the number of the records stored in the database, else, it is -1.

The function `vlwritable' is used in order to check whether a database handle is a writer or not.

int vlwritable(VILLA *villa);
`villa' specifies a database handle. The return value is true if the handle is a writer, false if not.

The function `vlfatalerror' is used in order to check whether a database has a fatal error or not.

int vlfatalerror(VILLA *villa);
`villa' specifies a database handle. The return value is true if the database has a fatal error, false if not.

The function `vlinode' is used in order to get the inode number of a database file.

int vlinode(VILLA *villa);
`villa' specifies a database handle. The return value is the inode number of the database file.

The function `vlmtime' is used in order to get the last modified time of a database.

int vlmtime(VILLA *villa);
`villa' specifies a database handle. The return value is the last modified time of the database.

The function `vltranbegin' is used in order to begin the transaction.

int vltranbegin(VILLA *villa);
`villa' specifies a database handle connected as a writer. If successful, the return value is true, else, it is false. Because this function does not perform mutual exclusion control in multi-thread, the application is responsible for it. Only one transaction can be activated with a database handle at the same time.

The function `vltrancommit' is used in order to commit the transaction.

int vltrancommit(VILLA *villa);
`villa' specifies a database handle connected as a writer. If successful, the return value is true, else, it is false. Updating a database in the transaction is fixed when it is committed successfully.

The function `vltranabort' is used in order to abort the transaction.

int vltranabort(VILLA *villa);
`villa' specifies a database handle connected as a writer. If successful, the return value is true, else, it is false. Updating a database in the transaction is discarded when it is aborted. The state of the database is rollbacked to before transaction.

The function `vlremove' is used in order to remove a database file.

int vlremove(const char *name);
`name' specifies the name of a database file. If successful, the return value is true, else, it is false.

The function `vlrepair' is used in order to repair a broken database file.

int vlrepair(const char *name, VLCFUNC cmp);
`name' specifies the name of a database file. `cmp' specifies the comparing function of the database file. If successful, the return value is true, else, it is false. There is no guarantee that all records in a repaired database file correspond to the original or expected state.

The function `vlexportdb' is used in order to dump all records as endian independent data.

int vlexportdb(VILLA *villa, const char *name);
`villa' specifies a database handle. `name' specifies the name of an output file. If successful, the return value is true, else, it is false.

The function `vlimportdb' is used in order to load all records from endian independent data.

int vlimportdb(VILLA *villa, const char *name);
`villa' specifies a database handle connected as a writer. The database of the handle must be empty. `name' specifies the name of an input file. If successful, the return value is true, else, it is false.

Examples

The following example stores and retrieves a phone number, using the name as the key.

#include <depot.h>
#include <cabin.h>
#include <villa.h>
#include <stdlib.h>
#include <stdio.h>

#define NAME     "mikio"
#define NUMBER   "000-1234-5678"
#define DBNAME   "book"

int main(int argc, char **argv){
  VILLA *villa;
  char *val;

  /* open the database */
  if(!(villa = vlopen(DBNAME, VL_OWRITER | VL_OCREAT, VL_CMPLEX))){
    fprintf(stderr, "vlopen: %s\n", dperrmsg(dpecode));
    return 1;
  }

  /* store the record */
  if(!vlput(villa, NAME, -1, NUMBER, -1, VL_DOVER)){
    fprintf(stderr, "vlput: %s\n", dperrmsg(dpecode));
  }

  /* retrieve the record */
  if(!(val = vlget(villa, NAME, -1, NULL))){
    fprintf(stderr, "vlget: %s\n", dperrmsg(dpecode));
  } else {
    printf("Name: %s\n", NAME);
    printf("Number: %s\n", val);
    free(val);
  }

  /* close the database */
  if(!vlclose(villa)){
    fprintf(stderr, "vlclose: %s\n", dperrmsg(dpecode));
    return 1;
  }

  return 0;
}

The following example performs forward matching search for strings.

#include <depot.h>
#include <cabin.h>
#include <villa.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#define DBNAME   "words"
#define PREFIX   "apple"

int main(int argc, char **argv){
  VILLA *villa;
  char *key, *val;

  /* open the database */
  if(!(villa = vlopen(DBNAME, VL_OWRITER | VL_OCREAT, VL_CMPLEX))){
    fprintf(stderr, "vlopen: %s\n", dperrmsg(dpecode));
    return 1;
  }

  /* store records */
  if(!vlput(villa, "applet", -1, "little application", -1, VL_DDUP) ||
     !vlput(villa, "aurora", -1, "polar wonderwork", -1, VL_DDUP) ||
     !vlput(villa, "apple", -1, "delicious fruit", -1, VL_DDUP) ||
     !vlput(villa, "amigo", -1, "good friend", -1, VL_DDUP) ||
     !vlput(villa, "apple", -1, "big city", -1, VL_DDUP)){
    fprintf(stderr, "vlput: %s\n", dperrmsg(dpecode));
  }

  /* set the cursor at the top of candidates */
  vlcurjump(villa, PREFIX, -1, VL_JFORWARD);

  /* scan with the cursor */
  while((key = vlcurkey(villa, NULL)) != NULL){
    if(strstr(key, PREFIX) != key){
      free(key);
      break;
    }
    if(!(val = vlcurval(villa, NULL))){
      fprintf(stderr, "vlcurval: %s\n", dperrmsg(dpecode));
      free(key);
      break;
    }
    printf("%s: %s\n", key, val);
    free(val);
    free(key);
    vlcurnext(villa);
  }

  /* close the database */
  if(!vlclose(villa)){
    fprintf(stderr, "vlclose: %s\n", dperrmsg(dpecode));
    return 1;
  }

  return 0;
}

Notes

How to build programs using Villa is the same as the case of Depot.

gcc -I/usr/local/include -o sample sample.c -L/usr/local/lib -lqdbm

If QDBM was built with POSIX thread enabled, the global variable `dpecode' is treated as thread specific data, and functions of Villa are reentrant. In that case, they are thread-safe as long as a handle is not accessed by threads at the same time, on the assumption that `errno', `malloc', and so on are thread-safe.

Vista: Extended Advanced API

Vista is the extended API of Villa. To compensate for the defect that Villa can not handle a file whose size is more than 2GB, Vista does not use Depot but Curia for handling its internal database. While Vista provides data structure and operations of B+ tree as with Villa, its database is realized as a directory.

In order to use Vista, you should include `vista.h' instead of `villa.h'. Because Vista is implemented by overriding symbols of Villa, it can be used as with Villa. That is, Signatures of Villa and Vista is all the same. However, as its adverse effect, modules (compilation unit) using Vista can not use Villa (do not include `villa.h').

Commands for Villa

Villa has the following command line interfaces.

The command `vlmgr' is a utility for debugging Villa and its applications. It features editing and checking of a database. It can be used for database applications with shell scripts. This command is used in the following format. `name' specifies a database name. `key' specifies the key of a record. `val' specifies the value of a record.

vlmgr create [-zc] name
Create a database file.
vlmgr put [-kx|-ki] [-vx|-vi|-vf] [-keep|-dup] name key val
Store a record with a key and a value.
vlmgr out [-l] [-kx|-ki] name key
Delete a record with a key.
vlmgr get [-l] [-kx|-ki] [-ox] [-n] name key
Retrieve a record with a key and output it to the standard output.
vlmgr list [-kx|-ki] [-ox] [-top key] [-bot key] [-gt] [-lt] [-max num] [-desc] name
List all keys and values delimited with tab and line-feed to the standard output.
vlmgr optimize name
Optimize a database.
vlmgr inform name
Output miscellaneous information to the standard output.
vlmgr remove name
Remove a database file.
vlmgr repair [-ki] name
Repair a broken database file.
vlmgr exportdb [-ki] name file
Dump all records as endian independent data.
vlmgr importdb [-ki] name file
Load all records from endian independent data.
vlmgr version
Output version information of QDBM to the standard output.

Options feature the following.

This command returns 0 on success, another on failure. The environment variable `QDBMDBGFD' specifies the file descriptor to output the history of updating the variable `dpecode'.

The command `vltest' is a utility for facility test and performance test. Check a database generated by the command or measure the execution time of the command. This command is used in the following format. `name' specifies a database name. `rnum' specifies the number of the records.

vltest write [-int] [-zc] [-tune lrecmax nidxmax lcnum ncnum] name rnum
Store records with keys of 8 bytes. They change as `00000001', `00000002'...
vltest read [-int] name
Retrieve all records of the database above.
vltest rdup [-int] [-zc] [-tune lrecmax nidxmax lcnum ncnum] name rnum pnum
Store records with partway duplicated keys using duplicate mode.
vltest combo [-zc] name
Perform combination test of various operations.
vltest wicked [-zc] name rnum
Perform updating operations selected at random.

Options feature the following.

This command returns 0 on success, another on failure. The environment variable `QDBMDBGFD' specifies the file descriptor to output the history of updating the variable `dpecode'.

The command `vltsv' features mutual conversion between a database of Villa and a TSV text. This command is useful when data exchange with another version of QDBM or another DBM, or when data exchange between systems which have different byte orders. This command is used in the following format. `name' specifies a database name. The subcommand `export' reads TSV data from the standard input. The subcommand `import' writes TSV data to the standard output.

vltsv import [-bin] name
Create a database from TSV.
vltsv export [-bin] name
Write TSV data of a database.

Options feature the following.

This command returns 0 on success, another on failure.

Commands of Villa realize a simple database system. For example, to make a database to search `/etc/password' by a user name, perform the following command.

cat /etc/passwd | tr ':' '\t' | vltsv import casket

Thus, to retrieve the information of a user `mikio', perform the following command.

vlmgr get casket mikio

It is easy to implement functions upsides with these commands, using the API of Villa.

Odeum: Inverted API

Overview

Odeum is the API which handles an inverted index. An inverted index is a data structure to retrieve a list of some documents that include one of words which were extracted from a population of documents. It is easy to realize a full-text search system with an inverted index. Odeum provides an abstract data structure which consists of words and attributes of a document. It is used when an application stores a document into a database and when an application retrieves some documents from a database.

Odeum does not provide methods to extract the text from the original data of a document. It should be implemented by applications. Although Odeum provides utilities to extract words from a text, it is oriented to such languages whose words are separated with space characters as English. If an application handles such languages which need morphological analysis or N-gram analysis as Japanese, or if an application perform more such rarefied analysis of natural languages as stemming, its own analyzing method can be adopted. Result of search is expressed as an array contains elements which are structures composed of the ID number of documents and its score. In order to search with two or more words, Odeum provides utilities of set operations.

Odeum is implemented, based on Curia, Cabin, and Villa. Odeum creates a database with a directory name. Some databases of Curia and Villa are placed in the specified directory. For example, `casket/docs', `casket/index', and `casket/rdocs' are created in the case that a database directory named as `casket'. `docs' is a database directory of Curia. The key of each record is the ID number of a document, and the value is such attributes as URI. `index' is a database directory of Curia. The key of each record is the normalized form of a word, and the value is an array whose element is a pair of the ID number of a document including the word and its score. `rdocs' is a database file of Villa. The key of each record is the URI of a document, and the value is its ID number.

In order to use Odeum, you should include `depot.h', `cabin.h', `odeum.h' and `stdlib.h' in the source files. Usually, the following description will be near the beginning of a source file.

#include <depot.h>
#include <cabin.h>
#include <odeum.h>
#include <stdlib.h>

A pointer to `ODEUM' is used as a database handle. A database handle is opened with the function `odopen' and closed with `odclose'. You should not refer directly to any member of the handle. If a fatal error occurs in a database, any access method via the handle except `odclose' will not work and return error status. Although a process is allowed to use multiple database handles at the same time, handles of the same database file should not be used.

A pointer to `ODDOC' is used as a document handle. A document handle is opened with the function `oddocopen' and closed with `oddocclose'. You should not refer directly to any member of the handle. A document consists of attributes and words. Each word is expressed as a pair of a normalized form and a appearance form.

Odeum also assign the external variable `dpecode' with the error code. The function `dperrmsg' is used in order to get the message of the error code.

API

Structures of `ODPAIR' type is used in order to handle results of search.

typedef struct { int id; int score; } ODPAIR;
`id' specifies the ID number of a document. `score' specifies the score calculated from the number of searching words in the document.

The function `odopen' is used in order to get a database handle.

ODEUM *odopen(const char *name, int omode);
`name' specifies the name of a database directory. `omode' specifies the connection mode: `OD_OWRITER' as a writer, `OD_OREADER' as a reader. If the mode is `OD_OWRITER', the following may be added by bitwise or: `OD_OCREAT', which means it creates a new database if not exist, `OD_OTRUNC', which means it creates a new database regardless if one exists. Both of `OD_OREADER' and `OD_OWRITER' can be added to by bitwise or: `OD_ONOLCK', which means it opens a database directory without file locking, or `OD_OLCKNB', which means locking is performed without blocking. The return value is the database handle or `NULL' if it is not successful. While connecting as a writer, an exclusive lock is invoked to the database directory. While connecting as a reader, a shared lock is invoked to the database directory. The thread blocks until the lock is achieved. If `OD_ONOLCK' is used, the application is responsible for exclusion control.

The function `odclose' is used in order to close a database handle.

int odclose(ODEUM *odeum);
`odeum' specifies a database handle. If successful, the return value is true, else, it is false. Because the region of a closed handle is released, it becomes impossible to use the handle. Updating a database is assured to be written when the handle is closed. If a writer opens a database but does not close it appropriately, the database will be broken.

The function `odput' is used in order to store a document.

int odput(ODEUM *odeum, const ODDOC *doc, int wmax, int over);
`odeum' specifies a database handle connected as a writer. `doc' specifies a document handle. `wmax' specifies the max number of words to be stored in the document database. If it is negative, the number is unlimited. `over' specifies whether the data of the duplicated document is overwritten or not. If it is false and the URI of the document is duplicated, the function returns as an error. If successful, the return value is true, else, it is false.

The function `odout' is used in order to delete a document specified by a URI.

int odout(ODEUM *odeum, const char *uri);
`odeum' specifies a database handle connected as a writer. `uri' specifies the string of the URI of a document. If successful, the return value is true, else, it is false. False is returned when no document corresponds to the specified URI.

The function `odoutbyid' is used in order to delete a document specified by an ID number.

int odoutbyid(ODEUM *odeum, int id);
`odeum' specifies a database handle connected as a writer. `id' specifies the ID number of a document. If successful, the return value is true, else, it is false. False is returned when no document corresponds to the specified ID number.

The function `odget' is used in order to retrieve a document specified by a URI.

ODDOC *odget(ODEUM *odeum, const char *uri);
`odeum' specifies a database handle. `uri' specifies the string of the URI of a document. If successful, the return value is the handle of the corresponding document, else, it is `NULL'. `NULL' is returned when no document corresponds to the specified URI. Because the handle of the return value is opened with the function `oddocopen', it should be closed with the function `oddocclose'.

The function `odgetbyid' is used in order to retrieve a document by an ID number.

ODDOC *odgetbyid(ODEUM *odeum, int id);
`odeum' specifies a database handle. `id' specifies the ID number of a document. If successful, the return value is the handle of the corresponding document, else, it is `NULL'. `NULL' is returned when no document corresponds to the specified ID number. Because the handle of the return value is opened with the function `oddocopen', it should be closed with the function `oddocclose'.

The function `odgetidbyuri' is used in order to retrieve the ID of the document specified by a URI.

int odgetidbyuri(ODEUM *odeum, const char *uri);
`odeum' specifies a database handle. `uri' specifies the string the URI of a document. If successful, the return value is the ID number of the document, else, it is -1. -1 is returned when no document corresponds to the specified URI.

The function `odcheck' is used in order to check whether the document specified by an ID number exists.

int odcheck(ODEUM *odeum, int id);
`odeum' specifies a database handle. `id' specifies the ID number of a document. The return value is true if the document exists, else, it is false.

The function `odsearch' is used in order to search the inverted index for documents including a particular word.

ODPAIR *odsearch(ODEUM *odeum, const char *word, int max, int *np);
`odeum' specifies a database handle. `word' specifies a searching word. `max' specifies the max number of documents to be retrieve. `np' specifies the pointer to a variable to which the number of the elements of the return value is assigned. If successful, the return value is the pointer to an array, else, it is `NULL'. Each element of the array is a pair of the ID number and the score of a document, and sorted in descending order of their scores. Even if no document corresponds to the specified word, it is not error but returns an dummy array. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. Note that each element of the array of the return value can be data of a deleted document.

The function `odsearchnum' is used in order to get the number of documents including a word.

int odsearchdnum(ODEUM *odeum, const char *word);
`odeum' specifies a database handle. `word' specifies a searching word. If successful, the return value is the number of documents including the word, else, it is -1. Because this function does not read the entity of the inverted index, it is faster than `odsearch'.

The function `oditerinit' is used in order to initialize the iterator of a database handle.

int oditerinit(ODEUM *odeum);
`odeum' specifies a database handle. If successful, the return value is true, else, it is false. The iterator is used in order to access every document stored in a database.

The function `oditernext' is used in order to get the next key of the iterator.

ODDOC *oditernext(ODEUM *odeum);
`odeum' specifies a database handle. If successful, the return value is the handle of the next document, else, it is `NULL'. `NULL' is returned when no document is to be get out of the iterator. It is possible to access every document by iteration of calling this function. However, it is not assured if updating the database is occurred while the iteration. Besides, the order of this traversal access method is arbitrary, so it is not assured that the order of string matches the one of the traversal access. Because the handle of the return value is opened with the function `oddocopen', it should be closed with the function `oddocclose'.

The function `odsync' is used in order to synchronize updating contents with the files and the devices.

int odsync(ODEUM *odeum);
`odeum' specifies a database handle connected as a writer. If successful, the return value is true, else, it is false. This function is useful when another process uses the connected database directory.

The function `odoptimize' is used in order to optimize a database.

int odoptimize(ODEUM *odeum);
`odeum' specifies a database handle connected as a writer. If successful, the return value is true, else, it is false. Elements of the deleted documents in the inverted index are purged.

The function `odname' is used in order to get the name of a database.

char *odname(ODEUM *odeum);
`odeum' specifies a database handle. If successful, the return value is the pointer to the region of the name of the database, else, it is `NULL'. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `odfsiz' is used in order to get the total size of database files.

double odfsiz(ODEUM *odeum);
`odeum' specifies a database handle. If successful, the return value is the total size of the database files, else, it is -1.0.

The function `odbnum' is used in order to get the total number of the elements of the bucket arrays in the inverted index.

int odbnum(ODEUM *odeum);
`odeum' specifies a database handle. If successful, the return value is the total number of the elements of the bucket arrays, else, it is -1.

The function `odbusenum' is used in order to get the total number of the used elements of the bucket arrays in the inverted index.

int odbusenum(ODEUM *odeum);
`odeum' specifies a database handle. If successful, the return value is the total number of the used elements of the bucket arrays, else, it is -1.

The function `oddnum' is used in order to get the number of the documents stored in a database.

int oddnum(ODEUM *odeum);
`odeum' specifies a database handle. If successful, the return value is the number of the documents stored in the database, else, it is -1.

The function `odwnum' is used in order to get the number of the words stored in a database.

int odwnum(ODEUM *odeum);
`odeum' specifies a database handle. If successful, the return value is the number of the words stored in the database, else, it is -1. Because of the I/O buffer, the return value may be less than the hard number.

The function `odwritable' is used in order to check whether a database handle is a writer or not.

int odwritable(ODEUM *odeum);
`odeum' specifies a database handle. The return value is true if the handle is a writer, false if not.

The function `odfatalerror' is used in order to check whether a database has a fatal error or not.

int odfatalerror(ODEUM *odeum);
`odeum' specifies a database handle. The return value is true if the database has a fatal error, false if not.

The function `odinode' is used in order to get the inode number of a database directory.

int odinode(ODEUM *odeum);
`odeum' specifies a database handle. The return value is the inode number of the database directory.

The function `odmtime' is used in order to get the last modified time of a database.

int odmtime(ODEUM *odeum);
`odeum' specifies a database handle. The return value is the last modified time of the database.

The function `odmerge' is used in order to merge plural database directories.

int odmerge(const char *name, const CBLIST *elemnames);
`name' specifies the name of a database directory to create. `elemnames' specifies a list of names of element databases. If successful, the return value is true, else, it is false. If two or more documents which have the same URL come in, the first one is adopted and the others are ignored.

The function `odremove' is used in order to remove a database directory.

int odremove(const char *name);
`name' specifies the name of a database directory. If successful, the return value is true, else, it is false. A database directory can contain databases of other APIs of QDBM, they are also removed by this function.

The function `oddocopen' is used in order to get a document handle.

ODDOC *oddocopen(const char *uri);
`uri' specifies the URI of a document. The return value is a document handle. The ID number of a new document is not defined. It is defined when the document is stored in a database.

The function `oddocclose' is used in order to close a document handle.

void oddocclose(ODDOC *doc);
`doc' specifies a document handle. Because the region of a closed handle is released, it becomes impossible to use the handle.

The function `oddocaddattr' is used in order to add an attribute to a document.

void oddocaddattr(ODDOC *doc, const char *name, const char *value);
`doc' specifies a document handle. `name' specifies the string of the name of an attribute. `value' specifies the string of the value of the attribute.

The function `oddocaddword' is used in order to add a word to a document.

void oddocaddword(ODDOC *doc, const char *normal, const char *asis);
`doc' specifies a document handle. `normal' specifies the string of the normalized form of a word. Normalized forms are treated as keys of the inverted index. If the normalized form of a word is an empty string, the word is not reflected in the inverted index. `asis' specifies the string of the appearance form of the word. Appearance forms are used after the document is retrieved by an application.

The function `oddocid' is used in order to get the ID number of a document.

int oddocid(const ODDOC *doc);
`doc' specifies a document handle. The return value is the ID number of a document.

The function `oddocuri' is used in order to get the URI of a document.

const char *oddocuri(const ODDOC *doc);
`doc' specifies a document handle. The return value is the string of the URI of a document.

The function `oddocgetattr' is used in order to get the value of an attribute of a document.

const char *oddocgetattr(const ODDOC *doc, const char *name);
`doc' specifies a document handle. `name' specifies the string of the name of an attribute. The return value is the string of the value of the attribute, or `NULL' if no attribute corresponds.

The function `oddocnwords' is used in order to get the list handle contains words in normalized form of a document.

const CBLIST *oddocnwords(const ODDOC *doc);
`doc' specifies a document handle. The return value is the list handle contains words in normalized form.

The function `oddocawords' is used in order to get the list handle contains words in appearance form of a document.

const CBLIST *oddocawords(const ODDOC *doc);
`doc' specifies a document handle. The return value is the list handle contains words in appearance form.

The function `oddocscores' is used in order to get the map handle contains keywords in normalized form and their scores.

CBMAP *oddocscores(const ODDOC *doc, int max, ODEUM *odeum);
`doc' specifies a document handle. `max' specifies the max number of keywords to get. `odeum' specifies a database handle with which the IDF for weighting is calculate. If it is `NULL', it is not used. The return value is the map handle contains keywords and their scores. Scores are expressed as decimal strings. Because the handle of the return value is opened with the function `cbmapopen', it should be closed with the function `cbmapclose' if it is no longer in use.

The function `odbreaktext' is used in order to break a text into words in appearance form.

CBLIST *odbreaktext(const char *text);
`text' specifies the string of a text. The return value is the list handle contains words in appearance form. Words are separated with space characters and such delimiters as period, comma and so on. Because the handle of the return value is opened with the function `cblistopen', it should be closed with the function `cblistclose' if it is no longer in use.

The function `odnormalizeword' is used in order to make the normalized form of a word.

char *odnormalizeword(const char *asis);
`asis' specifies the string of the appearance form of a word. The return value is is the string of the normalized form of the word. Alphabets of the ASCII code are unified into lower cases. Words composed of only delimiters are treated as empty strings. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `odpairsand' is used in order to get the common elements of two sets of documents.

ODPAIR *odpairsand(ODPAIR *apairs, int anum, ODPAIR *bpairs, int bnum, int *np);
`apairs' specifies the pointer to the former document array. `anum' specifies the number of the elements of the former document array. `bpairs' specifies the pointer to the latter document array. `bnum' specifies the number of the elements of the latter document array. `np' specifies the pointer to a variable to which the number of the elements of the return value is assigned. The return value is the pointer to a new document array whose elements commonly belong to the specified two sets. Elements of the array are sorted in descending order of their scores. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `odpairsor' is used in order to get the sum of elements of two sets of documents.

ODPAIR *odpairsor(ODPAIR *apairs, int anum, ODPAIR *bpairs, int bnum, int *np);
`apairs' specifies the pointer to the former document array. `anum' specifies the number of the elements of the former document array. `bpairs' specifies the pointer to the latter document array. `bnum' specifies the number of the elements of the latter document array. `np' specifies the pointer to a variable to which the number of the elements of the return value is assigned. The return value is the pointer to a new document array whose elements belong to both or either of the specified two sets. Elements of the array are sorted in descending order of their scores. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `odpairsnotand' is used in order to get the difference set of documents.

ODPAIR *odpairsnotand(ODPAIR *apairs, int anum, ODPAIR *bpairs, int bnum, int *np);
`apairs' specifies the pointer to the former document array. `anum' specifies the number of the elements of the former document array. `bpairs' specifies the pointer to the latter document array of the sum of elements. `bnum' specifies the number of the elements of the latter document array. `np' specifies the pointer to a variable to which the number of the elements of the return value is assigned. The return value is the pointer to a new document array whose elements belong to the former set but not to the latter set. Elements of the array are sorted in descending order of their scores. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use.

The function `odpairssort' is used in order to sort a set of documents in descending order of scores.

void odpairssort(ODPAIR *pairs, int pnum);
`pairs' specifies the pointer to a document array. `pnum' specifies the number of the elements of the document array.

The function `odlogarithm' is used in order to get the natural logarithm of a number.

double odlogarithm(double x);
`x' specifies a number. The return value is the natural logarithm of the number. If the number is equal to or less than 1.0, the return value is 0.0. This function is useful when an application calculates the IDF of search results.

The function `odvectorcosine' is used in order to get the cosine of the angle of two vectors.

double odvectorcosine(const int *avec, const int *bvec, int vnum);
`avec' specifies the pointer to one array of numbers. `bvec' specifies the pointer to the other array of numbers. `vnum' specifies the number of elements of each array. The return value is the cosine of the angle of two vectors. This function is useful when an application calculates similarity of documents.

The function `odsettuning' is used in order to set the global tuning parameters.

void odsettuning(int ibnum, int idnum, int cbnum, int csiz);
`ibnum' specifies the number of buckets for inverted indexes. `idnum' specifies the division number of inverted index. `cbnum' specifies the number of buckets for dirty buffers. `csiz' specifies the maximum bytes to use memory for dirty buffers. The default setting is equivalent to `odsettuning(32749, 7, 262139, 8388608)'. This function should be called before opening a handle.

The function `odanalyzetext' is used in order to break a text into words and store appearance forms and normalized form into lists.

void odanalyzetext(ODEUM *odeum, const char *text, CBLIST *awords, CBLIST *nwords);
`odeum' specifies a database handle. `text' specifies the string of a text. `awords' specifies a list handle into which appearance form is store. `nwords' specifies a list handle into which normalized form is store. If it is `NULL', it is ignored. Words are separated with space characters and such delimiters as period, comma and so on.

The function `odsetcharclass' is used in order to set the classes of characters used by `odanalyzetext'.

void odsetcharclass(ODEUM *odeum, const char *spacechars, const char *delimchars, const char *gluechars);
`odeum' specifies a database handle. `spacechars' spacifies a string contains space characters. `delimchars' spacifies a string contains delimiter characters. `gluechars' spacifies a string contains glue characters.

The function `odquery' is used in order to query a database using a small boolean query language.

ODPAIR *odquery(ODEUM *odeum, const char *query, int *np, CBLIST *errors);
`odeum' specifies a database handle. 'query' specifies the text of the query. `np' specifies the pointer to a variable to which the number of the elements of the return value is assigned. `errors' specifies a list handle into which error messages are stored. If it is `NULL', it is ignored. If successful, the return value is the pointer to an array, else, it is `NULL'. Each element of the array is a pair of the ID number and the score of a document, and sorted in descending order of their scores. Even if no document corresponds to the specified condition, it is not error but returns an dummy array. Because the region of the return value is allocated with the `malloc' call, it should be released with the `free' call if it is no longer in use. Note that each element of the array of the return value can be data of a deleted document.

Examples

The following example stores a document into the database.

#include <depot.h>
#include <cabin.h>
#include <odeum.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#define DBNAME   "index"

int main(int argc, char **argv){
  ODEUM *odeum;
  ODDOC *doc;
  CBLIST *awords;
  const char *asis;
  char *normal;
  int i;

  /* open the database */
  if(!(odeum = odopen(DBNAME, OD_OWRITER | OD_OCREAT))){
    fprintf(stderr, "odopen: %s\n", dperrmsg(dpecode));
    return 1;
  }

  /* get the document handle */
  doc = oddocopen("http://www.foo.bar/baz.txt");

  /* set attributes of the document */
  oddocaddattr(doc, "title", "Balcony Scene");
  oddocaddattr(doc, "author", "Shakespeare");

  /* break the text and get the word list */
  awords = odbreaktext("Parting is such sweet sorrow.");

  /* set each word into the document handle */
  for(i = 0; i < cblistnum(awords); i++){
    /* get one word of the list */
    asis = cblistval(awords, i, NULL);
    /* get the normalized form from the appearance form */
    normal = odnormalizeword(asis);
    /* add the word into the document handle */
    oddocaddword(doc, normal, asis);
    /* release the region of the normalized form */
    free(normal);
  }

  /* store the document into the database */
  if(!odput(odeum, doc, -1, 1)){
    fprintf(stderr, "odput: %s\n", dperrmsg(dpecode));
  }

  /* release the word list */
  cblistclose(awords);

  /* release the document handle */
  oddocclose(doc);

  /* close the database */
  if(!odclose(odeum)){
    fprintf(stderr, "odclose: %s\n", dperrmsg(dpecode));
    return 1;
  }

  return 0;
}

The following example retrieves documents.

#include <depot.h>
#include <cabin.h>
#include <odeum.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#define DBNAME   "index"

int main(int argc, char **argv){
  ODEUM *odeum;
  ODPAIR *pairs;
  ODDOC *doc;
  const CBLIST *words;
  const char *title, *author, *asis;
  int i, j, pnum;

  /* open the database */
  if(!(odeum = odopen(DBNAME, OD_OREADER))){
    fprintf(stderr, "odopen: %s\n", dperrmsg(dpecode));
    return 1;
  }

  /* retrieve documents */
  if((pairs = odsearch(odeum, "sorrow", -1, &pnum)) != NULL){

    /* scan each element of the document array */
    for(i = 0; i < pnum; i++){
      /* get the document handle */
      if(!(doc = odgetbyid(odeum, pairs[i].id))) continue;
      /* show the attributes */
      printf("URI: %s\n", oddocuri(doc));
      title = oddocgetattr(doc, "title");
      if(title) printf("TITLE: %s\n", title);
      author = oddocgetattr(doc, "author");
      if(author) printf("AUTHOR: %s\n", author);
      /* show words in appearance form */
      printf("WORDS:");
      words = oddocawords(doc);
      for(j = 0; j < cblistnum(words); j++){
        asis = cblistval(words, j, NULL);
        printf(" %s", asis);
      }
      putchar('\n');
      /* release the document handle */
      oddocclose(doc);
    }

    /* release the document array */
    free(pairs);

  } else {
    fprintf(stderr, "odsearch: %s\n", dperrmsg(dpecode));
  }

  /* close the database */
  if(!odclose(odeum)){
    fprintf(stderr, "odclose: %s\n", dperrmsg(dpecode));
    return 1;
  }

  return 0;
}

Notes

How to build programs using Odeum is the same as the case of Depot.

gcc -I/usr/local/include -o sample sample.c -L/usr/local/lib -lqdbm

If QDBM was built with POSIX thread enabled, the global variable `dpecode' is treated as thread specific data, and functions of Odeum are reentrant. In that case, they are thread-safe as long as a handle is not accessed by threads at the same time, on the assumption that `errno', `malloc', and so on are thread-safe.

If QDBM was built with ZLIB enabled, records in the database for document attributes are compressed. In that case, the size of the database is reduced to 30% or less. Thus, you should enable ZLIB if you use Odeum. A database of Odeum created without ZLIB enabled is not available on environment with ZLIB enabled, and vice versa.

Query Language

The query language of the function `odquery' is a basic language following this grammar:

expr ::= subexpr ( op subexpr )*
subexpr ::= WORD
subexpr ::= LPAREN expr RPAREN

Operators are "&" (AND), "|" (OR), and "!" (NOTAND). You can use parenthesis to group sub-expressions together in order to change order of operations. The given query is broken up using the function `odanalyzetext', so if you want to specify different text breaking rules, then make sure that you at least set "&", "|", "!", "(", and ")" to be delimiter characters. Consecutive words are treated as having an implicit "&" operator between them, so "zed shaw" is actually "zed & shaw".

The encoding of the query text should be the same with the encoding of target documents. Moreover, each of space characters, delimiter characters, and glue characters should be single byte.

Commands for Odeum

Odeum has the following command line interfaces.

The command `odmgr' is a utility for debugging Odeum and its applications. It features editing and checking of a database. It can be used for full-text search systems with shell scripts. This command is used in the following format. `name' specifies a database name. `file' specifies a file name, `expr' specifies the URI or the ID number of a document, `words' specifies searching words. `elems' specifies element databases.

odmgr create name
Create a database file.
odmgr put [-uri str] [-title str] [-author str] [-date str] [-wmax num] [-keep] name [file]
Add a document by reading a file. If `file' is omitted, the standard input is read and URI is needed.
odmgr out [-id] name expr
Delete a document specified by a URI.
odmgr get [-id] [-t|-h] name expr
Show a document specified by a URI. The output is the ID number and the URI of a document, in tab separated format.
odmgr search [-max num] [-or] [-idf] [-t|-h|-n] name words...
Retrieve documents including specified words. The first line of the output is the total number of hits and each word with its number of hits, in tab separated format. The second line and below are the ID numbers and the scores of documents, in tab separated format.
odmgr list [-t|-h] name
Show all documents in a database. Each line of the output is the ID number and the score of a document, in tab separated format.
odmgr optimize name
Optimize a database.
odmgr inform name
Output miscellaneous information.
odmgr merge name elems...
Merge plural databases.
odmgr remove name
Remove a database directory.
odmgr break [-h|-k|-s] [file]
Read a file and output words in the text. Each line of the output is the appearance form and the normalized form of a word, in tab separated format.
odmgr version
Output version information of QDBM.

Options feature the following.

This command returns 0 on success, another on failure. The environment variable `QDBMDBGFD' specifies the file descriptor to output the history of updating the variable `dpecode'.

The command `odtest' is a utility for facility test and performance test. Check a database generated by the command or measure the execution time of the command. This command is used in the following format. `name' specifies a database name. `dnum' specifies the number of the documents. `wnum' specifies the number of words per document. `pnum' specifies the number of patterns of words.

odtest write [-tune ibnum idnum cbnum csiz] name dnum wnum pnum
Store documents with random attributes and random words.
odtest read name
Retrieve all documents of the database above.
odtest combo name
Perform combination test of various operations.
odtest wicked name dnum
Perform updating operations selected at random.

Options feature the following.

This command returns 0 on success, another on failure. The environment variable `QDBMDBGFD' specifies the file descriptor to output the history of updating the variable `dpecode'.

The command `odidx' is a utility for indexing files on the local file system. This command is useful for a full-text search system of a Web site. Supported file format are plain text and HTML. Supported character encoding are US-ASCII and ISO-8859-1. The URI of each document is specified with the path of its file. Attributes named as `title' and `date' are given to each documents. When a document is already existing in the database, it is registered if its last modification time is newer, or it is ignored. Modification times are stored in the sub database `_mtime' in the main database directory. Score information are stored in the sub database `_score' in the main database directory. This command is used in the following format. `name' specifies a database name. `dir' specifies a directory name.

odidx register [-l file] [-wmax num] [-tsuf sufs] [-hsuf sufs] name [dir]
Register files in the specified directory. If `dir' is omitted, the current directory is specified.
odidx relate name
Add score information for relational document search to each documents in the database.
odidx purge name
Purge documents which are not existing on the local files system.

Options feature the following.

This command returns 0 on success, another on failure.

Commands of Odeum make it easy to realize a full-text search system. For example, to register files which are under `/home/mikio' and whose suffix are `.txt', `.c', or `.h', perform the following command.

odidx register -tsuf ".txt,.c,.h" -hsuf "" casket /home/mikio

Thus, to retrieve documents which include `unix' and `posix' and show the top 8 terms, perform the following command.

odmgr search -max 8 -h casket "unix posix"

A database generated by `odidx' is available with the CGI script which is included in QDBM for full-text search.

File Format

File Format of Depot

The contents of a database file managed by Depot is divided roughly into the following three sections: the header section, the bucket section and the record section.

The header section places at the beginning of the file and its length is constant 48 bytes. The following information are stored in the header section.

  1. magic number: from offset 0, contains "[DEPOT]\n\f" for big endian or "[depot]\n\f" for little endian.
  2. version number: decimal string of the version number of the library.
  3. flags for wrappers: from offset 16, type of `int'.
  4. file size: from offset 24, type of `int'.
  5. number of the bucket: from offset 32, type of `int'.
  6. number of records: from offset 40, type of `int'.

The bucket section places after the header section and its length is determined according to the number of the bucket. Each element of the bucket stores an offset of the root node of each separate chain.

The record section places after the bucket section and occupies to the end of the file. The element of the record section contains the following information.

  1. flags: type of `int'.
  2. second hash value: type of `int'.
  3. size of the key: type of `int'.
  4. size of the value: type of `int'.
  5. size of the padding: type of `int'.
  6. offset of the left child: type of `int'.
  7. offset of the right child: type of `int'.
  8. entity of the key: serial bytes with variable length.
  9. entity of the value: serial bytes with variable length.
  10. padding data: void serial bytes with variable length.

File Format of Villa

Every data handled by Villa is stored in a database of Depot. Storing data is divided into meta data and logical pages. Logical pages can be classified into leaf nodes and non-leaf nodes. Meta data are such managing information as the number of records. Both of its key and its value are type of `int'. Leaf nodes hold records. Non-leaf nodes hold sparse index referring to pages.

Villa uses variable length numeric format (BER compression) to handle small natural number with frugal storage. A variable length numeric object is parsed from the top of the region and parsing ends at the byte of positive value. Each byte are evaluated as absolute value and calculated as little endian number based on the radix 128.

Record is logical unit of user data. Some records overlapping keys are shaped into one physical record. A Physical record is serialized in the following format.

  1. size of the key: type of variable length number
  2. entity of the key: serial bytes with variable length
  3. number of values: type of variable length number
  4. list of values: serial bytes repeating the following expressions
    1. size: type of variable length number
    2. entity of the key: serial bytes with variable length

Leaf node is physical unit to store a set of records. The key of a leaf node is its ID whose type is `int'. A leaf node is stored in a database of Depot with the following values. Its records are sorted in ascending order of each key.

  1. ID of the previous leaf: type of variable length number
  2. ID of the next leaf: type of variable length number
  3. list of records: concatenation of serialized records

Index is logical unit of a pointer to search for pages. An index is serialized int the following format.

  1. ID of the referring page: type of variable length number
  2. size of the key: type of variable length number
  3. entity of the key: serial bytes with variable length

Non-leaf node is physical unit to store a set of indexes. The key of a non-leaf node is its ID whose type is `int'. A non-leaf node is stored in a database of Depot with the following values. Its indexes are sorted in ascending order of each key.

  1. ID of the first child node: type of variable length number
  2. list of indexes: concatenation of serialized indexes

Notes

Because the database file is not sparse, move, copy, unlink, ftp, and so on with the file are possible. Because Depot reads and writes data without normalization of byte order, it is impossible to share the same file between the environment with different byte order.

When you distribute a database file of Depot or Villa via network, the MIME type suggested to be `application/x-qdbm'. Suffix of the file name is suggested to be `.qdb'. When you distribute a database directory of Curia, you may convert the directory tree to an archive of such type as TAR.

For the command `file' to recognize database files, append the following expressions into `magic' file.

0       string          [DEPOT]\n\f     QDBM, big endian
>12     string          x               \b, version=%s
>16     belong          ^1              \b, Hash
>16     belong          &1              \b, B+ tree
>16     belong          &2              \b (deflated)
>24     belong          x               \b, filesize=%d
>32     belong          x               \b, buckets=%d
>40     belong          x               \b, records=%d
0       string          [depot]\n\f     QDBM, little endian
>12     string          x               \b, version=%s
>16     lelong          ^1              \b, Hash
>16     lelong          &1              \b, B+ tree
>16     lelong          &2              \b (deflated)
>24     lelong          x               \b, filesize=%d
>32     lelong          x               \b, buckets=%d
>40     lelong          x               \b, records=%d

Porting

One of the goal of QDBM is to work on all platforms which conform to POSIX. Even if some APIs are not implemented, QDBM should work. Moreover, it should be possible to build QDBM using compilers other than GCC. Porting to various platforms is performed to add a new `Makefile' or modify some parts of source files. As for APIs of C, some of the following files should be modified. Otherwise, you can create new files based on them.

On platforms which do not support file locking with `fcntl' call, you should append `-DMYNOLOCK' to the macro `CFLAGS' defined in `Makefile'. In that case, you should consider another exclusion control. As with it, on platforms without `mmap' call, you should append `-DMYNOMMAP' to `CFLAGS'. As for `mmap', its emulation using `malloc' and so on is provided. If other system calls are not implemented, you should define emulation by modification of `myconf.h' and `myconf.c'.

Because POSIX thread is used in C++ API, it is impossible to port C++ API to platforms without the package. Because JNI is used in Java API, you should pay attention to location of the headers and libraries. Moreover, you should consider such type definitions as `long long' or `int64'. Because APIs of Perl and Ruby use building commands provided with each language system, you should be knowledgeable about their specifications.

Bugs

Each document of QDBM should be calibrated by native English speakers.

There is no such bug which are found but not fixed, as crash by segmentation fault, unexpected data vanishing, memory leak and so on.

If you find any bug, report it to the author, with the information of the version of QDBM, the operating system and the compiler.

Databases created with QDBM version 1.7.13 or earlier are not compatible to ones of the later versions.

Frequently Asked Questions

Q. : Does QDBM support SQL?
A. : No, it does not. QDBM is not a RDBMS (Relational Database Management System). If you want an embedded RDBMS, use SQLite and so on.
Q. : After all, how different from GDBM (NDBM, SDBM, Berkeley DB)?
A. : Processing speed is higher, a database file is smaller, API is simpler. A highly important thing is that efficiency in time and space is very good when records are frequently overwritten, so, scalability in practical use is high. Moreover, even when constructing such a large database that the number of storing record is more than one million, processing speed does not slowdown deathly, filesize does not grow extremely. However, because other DBM or DBMS may be more suitable in some cases, comparing performance and functionality by yourself is suggested.
Q. : Which API should I use?
A. : If you search for records as complete accord, try Depot. If the scale is large, try Curia. If you access records in some order, try Villa. If the scale is large, try Vista. If you pursue the greatest number of records, build QDBM with ZLIB enabled and use Vista.
Q. : What is bibliography?
A. : Algorithms of QDBM are mainly based on the descriptions in `Data Structures and Algorithms' by Aho et al and `Algorithms in C' by Sedgewick.
Q. : Are there good sample codes for applications?
A. : Refer to the source code of commands of each API. `dptsv.c', `crtsv.c' and `vltsv.c' are simplest.
Q. : My database file is broken. Why?
A. : In most cases, the reason is that your application did not close the database on exit. No matter whether it is a demon process or a CGI script, any application should close handling databases when it exits. Moreover, we should remember that a process of CGI may be killed by SIGPIPE or SIGTERM.
Q. : How robust are databases of QDBM?
A. : QDBM does not assure absolute robustness. A database may be broken if your operating system crashes. Although transaction of Villa can save a database from crashes of applications, it is inadequate to crashes of operating systems. So, you should consider multiplexing of a database or backup system if you use QDBM for mission critical applications.
Q: How should I use alignment of Depot and Curia?
A: If your application repeats writing with overwrite or concatenate mode. Alignment saves the rapid growth of the size of the database file. Because the best suited size of alignment of each application is different, you should learn it by experiment. For the meantime, about 32 is suitable.
Q. : How should I tune performance parameters of Villa?
A. : If you perform mainly ordering access, `lrecmax' and `nidxmax' should be larger. If you perform mainly random access, they should be less. If RAM of your system is abundant, `lcnum' and `ncnum' should be increased in order to improve performance. If ZLIB is enabled, increase `lrecmax' and compression efficiency is improved.
Q. : What is `sparse file'?
A. : It is a file where some holes are. `Hole' means a block where any data has never written in. If a file system supports sparse file, holes are not allocated into any physical storage. As for QDBM, if a database is created with such flags as DP_OSPARSE, the bucket array is not initialized and its blocks become holes. According to that mechanism, you can use greatly huge hash tables. However, its performance is strongly depends on the setting of the file system.
Q. : Why Depot and Curia do not feature transaction?
A. : If an application implements its own transaction, inner transaction of database is superfluous. You can implement transaction for application easily with hash map provided by Cabin.
Q. : How should I tune the system for performance?
A. : Install more RAM on your machine than the size of a database. Then, enlarge I/O buffer and cut down on flushing dirty buffers. File system is also important. On Linux, although EXT2 is usually fastest, EXT3 is faster in some cases. ReiserFS is okey. The other modes of EXT3 are very slow. About other file systems, you should learn them by experiment.
Q. : Can I build QDBM using `cc' instead of `gcc'?
A. : Actually, yes. Try to build QDBM by `make unix' and install it by `make install-unix'.
Q. : Can I build QDBM using Visual C++?
A. : Yes. Use `VCmakefile' instead of `Makefile'.
Q. : Can I use QDBM with other languages?
A. : As for PHP, Scheme (Gauche), and OCaml, interfaces of QDBM have been released. If you need it for another language, try to turn it out.
Q. : What does `QDBM' mean?
A. : `QDBM' stands for `Quick Database Manager'. It means that processing speed is high, and that you can write applications quickly.

Copying

QDBM is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License or any later version.

QDBM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with QDBM (See the file `COPYING'); if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA.

QDBM was written by Mikio Hirabayashi. You can contact the author by e-mail to `mikio@users.sourceforge.net'. However, as for topics which can be shared among other users, please send it to the mailing list. To join the mailing list, refer to `http://lists.sourceforge.net/lists/listinfo/qdbm-users'.