Chapter 2: BOS Server Architecture
This chapter considers some of the architectual features of the AFS-3 BOS Server. First, the basic organizational and functional entity employed by the BOS Server, the bnode, is discussed. Next, the set of files with which the server interacts is examined. The notion of restart times is then explained in detail. Finally, the organization and components of the bosserver program itself, which implements the BOS Server, are presented.

Section 2.1: Bnodes

Section 2.1.1: Overview

The information required to manage each AFS-related program running on a File Server machine is encapsulated in a bnode object. These bnodes serve as the basic building blocks for BOS Server services. Bnodes have two forms of existence:
  • On-disk: The BosConfig file (see Section 2.3.4 below) defines the set of bnodes for which the BOS Server running on that machine will be responsible, along with specifying other information such as values for the two restart times. This file provides permanent storage (i.e., between bootups) for the desired list of programs for that server platform.
  • In-memory: The contents of the BosConfig file are parsed and internalized by the BOS Server when it starts execution. The basic data for a particular server program is placed into a struct bnode structure.
The initial contents of the BosConfig file are typically set up during system installation. The BOS Server can be directed, via its RPC interface, to alter existing bnode entries in the BosConfig file, add new ones, and delete old ones. Typically, this file is never edited directly.

Section 2.1.2: Bnode Classes

The descriptions of the members of the AFS server suite fall into three broad classes of programs:
  • Simple programs: This server class is populated by programs that simply need to be kept running, and do not depend on other programs for correctness or effectiveness. Examples of AFS servers falling into this category are the Volume Location Server, Authentication Server, and Protection Server. Since its members exhibit such straightforward behavior, this class of programs is referred to as the simple class.
  • Interrelated programs: The File Server program depends on two other programs, and requires that they be executed at the appropriate times and in the appropriate sequence, for correct operation. The first of these programs is the Volume Server, which must be run concurrently with the File Server. The second is the salvager, which repairs the AFS volume metadata on the server partitions should the metadata become damaged. The salvager must not be run at the same time as the File Server. In honor of the File Server trio that inspired it, the class of programs consisting of groups of interrelated processes is named the fs class.
  • Periodic programs: Some AFS servers, such as the BackupServer, only need to run every so often, but on a regular and well-defined basis. The name for this class is taken from the unix tool that is typically used to define such regular executions, namely the cron class.
The recognition and definition of these three server classes is exploited by the BOS Server. Since all of the programs in a given class share certain common characteristics, they may all utilize the same basic data structures to record and manage their special requirements. Thus, it is not necessary to reimplement all the operations required to satisfy the capabilities promised by the BOS Server RPC interface for each and every program the BOS Server manages. Implementing one set of operations for each server class is sufficient to handle any and all server binaries to be run on the platform.

Section 2.1.3: Per-Class Bnode Operations

As mentioned above, only one set of basic routines must be implemented for each AFS server class. Much like Sun's VFS/vnode interface [8], providing a common set of routines for interacting with a given file system, regardless of its underlying implementation and semantics, the BOS Server defines a common vector of operations for a class of programs to be run under the BOS Server's tutelage. In fact, it was this standardized file system interface that inspired the "bnode" name.
The BOS Server manipulates the process or processes that are described by each bnode by invoking the proper functions in the appropriate order from the operation vector for that server class. Thus, the BOS Server itself operates in a class-independent fashion. This allows each class to take care of the special circumstances associated with it, yet to have the BOS Server itself be totally unaware of what special actions (if any) are needed for the class. This abstraction also allows more server classes to be implemented without any significant change to the BOS Server code itself.
There are ten entries in this standardized class function array. The purpose and usage of each individual class function is described in detail in Section 3.3.5. Much like the VFS system, a collection of macros is also provided in order to simplify the invocation of these functions. These macros are described in Section 3.5. The ten function slots are named here for convenience:
  • create()
  • timeout()
  • getstat()
  • setstat()
  • delete()
  • procexit()
  • getstring()
  • getparm()
  • restartp()
  • hascore()

Section 2.2: BOS Server Directories

The BOS Server expects the existence of the following directories on the local disk of the platform on which it runs. These directories define where the system binaries, log files, ubik databases, and other files lie.
  • /usr/afs/bin: This directory houses the full set of AFS server binaries. Such executables as bosserver, fileserver, vlserver, volserver, kaserver, and ptserver reside here.
  • /usr/afs/db: This directory serves as the well-known location on the server's local disk for the ubik database replicas for this machine. Specifically, the Authentication Server, Protection Server, and the Volume Location Server maintain their local database images here.
  • /usr/afs/etc: This directory hosts the files containing the security, cell, and authorized system administrator list for the given machine. Specifically, the CellServDB, KeyFile, License, ThisCell, and UserList files are stored here.
  • /usr/afs/local: This directory houses the BosConfig file, which supplies the BOS Server with the permanent set of bnodes to support. Also contained in this directory are files used exclusively by the salvager.
  • /usr/afs/logs: All of the AFS server programs that maintain log files deposit them in this directory.

Section 2.3: BOS Server Files

Several files, some mentioned above, are maintained on the server's local disk and manipulated by the BOS Server. This section examines many of these files, and describes their formats.

Section 2.3.1: /usr/afs/etc/UserList

This file contains the names of individuals who are allowed to issue "restricted" BOS Server commands (e.g., creating & deleting bnodes, setting cell information, etc.) on the given hardware platform. The format is straightforward, with one administrator name per line. If a cell grants joe and schmoe these rights on a machine, that particular UserList will have the following two lines:
joe
schmoe

Section 2.3.2: /usr/afs/etc/CellServDB

This file identifies the name of the cell to which the given server machine belongs, along with the set of machines on which its ubik database servers are running. Unlike the CellServDB found in /usr/vice/etc on AFS client machines, this file contains only the entry for the home cell. It shares the formatting rules with the /usr/vice/etc version of CellServDB. The contents of the CellServDB file used by the BOS Server on host grand.central.org are:
>grand.central.org      #DARPA clearinghouse cell
192.54.226.100          #grand.central.org
192.54.226.101          #penn.central.org

Section 2.3.3: /usr/afs/etc/ThisCell

The BOS Server obtains its notion of cell membership from the ThisCell file in the specified directory. As with the version of ThisCell found in /usr/vice/etc on AFS client machines, this file simply contains the character string identifying the home cell name. For any server machine in the grand.central.org cell, this file contains the following:
grand.central.org 

Section 2.3.4: /usr/afs/local/BosConfig

The BosConfig file is the on-disk representation of the collection of bnodes this particular BOS Server manages. The BOS Server reads and writes to this file in the normal course of its affairs. The BOS Server itself, in fact, should be the only agent that modifies this file. Any changes to BosConfig should be carried out by issuing the proper RPCs to the BOS Server running on the desired machine.
The following is the text of the BosConfig file on grand.central.org. A discussion of the contents follows immediately afterwards.
restarttime 11 0 4 0 0 checkbintime 3 0 5 0 0 
bnode simple kaserver 1 parm /usr/afs/bin/kaserver 
end bnode simple ptserver 1 parm 
/usr/afs/bin/ptserver end bnode simple vlserver 1 
parm /usr/afs/bin/vlserver end bnode fs fs 1 parm 
/usr/afs/bin/fileserver parm /usr/afs/bin/volserver 
parm /usr/afs/bin/salvager end bnode simple runntp 
1 parm /usr/afs/bin/runntp -localclock transarc.com 
end bnode simple upserver 1 parm 
/usr/afs/bin/upserver end bnode simple 
budb_server 1 parm /usr/afs/bin/budb_server end 
bnode cron backup 1 parm 
/usr/afs/backup/clones/lib/backup.csh daily parm 
05:00 end 
The first two lines of this file set the system and new-binary restart times (see Section 2.4, below). They are optional, with the default system restart time being every Sunday at 4:00am and the new-binary restart time being 5:00am daily. Following the reserved words restarttime and checkbintime are five integers, providing the mask, day, hour, minute, and second values (in decimal) for the restart time, as diagrammed below:
restarttime <mask> <day> <hour> <minute> 
<second> checkbintime <mask> <day> <hour> 
<minute> <second> 
The range of acceptable values for these fields is presented in Section 3.3.1. In this example, the restart line specifies a (decimal) mask value of 11, selecting the KTIME HOUR, KTIME MIN, and KTIME DAY bits. This indicates that the hour, minute, and day values are the ones to be used when matching times. Thus, this line requests that system restarts occur on day 0 (Sunday), hour 4 (4:00am), and minute 0 within that hour.
The sets of lines that follow define the individual bnodes for the particular machine. The first line of the bnode definition set must begin with the reserved word bnode, followed by the type name, the instance name, and the initial bnode goal:
bnode <type_name> <instance_name> <goal_val> 
The <type name>=""> and <instance name>=""> fields are both character strings, and the <goal val>=""> field is an integer. Acceptable values for the <type name>=""> are simple, fs, and cron. Acceptable values for <goal val>=""> are defined in Section 3.2.3, and are normally restricted to the integer values representing BSTAT NORMAL and BSTAT SHUTDOWN. Thus, in the bnode line defining the Authentication Server, it is declared to be of type simple, have instance name kaserver, and have 1 (BSTAT NORMAL) as a goal (e.g., it should be brought up and kept running).
Following the bnode line in the BosConfig file may be one or more parm lines. These entries represent the command line parameters that will be used to invoke the proper related program or programs. The entire text of the line after the parm reserved word up to the terminating newline is stored as the command line string.
parm <command_line_text> 
After the parm lines, if any, the reserved word end must appear alone on a line, marking the end of an individual bnode definition.

Section 2.3.5: /usr/afs/local/NoAuth

The appearance of this file is used to mark whether the BOS Server is to insist on properly authenticated connections for its restricted operations or whether it will allow any caller to exercise these commands. Not only is the BOS Server affected by the presence of this file, but so are all of the bnodes objects the BOS Server starts up. If /usr/afs/local/NoAuth is present, the BOS Server will start all of its bnodes with the -noauth flag.
Completely unauthenticated AFS operation will result if this file is present when the BOS Server starts execution. The file itself is typically empty. If any data is put into the NoAuth file, it will be ignored by the system.

Section 2.3.6: /usr/afs/etc/KeyFile

This file stores the set of AFS encryption keys used for file service operations. The file follows the format defined by struct afsconf key and struct afsconf keys in include file afs/keys.h. For the reader's convenience, these structures are detailed below:
#define AFSCONF_MAXKEYS 8 
struct afsconf_key { 
        long kvno; 
        char key[8]; 
}; 
struct afsconf_keys { 
        long nkeys; 
        struct afsconf_key key[AFSCONF_MAXKEYS]; 
}; 
The first longword of the file reveals the number of keys that may be found there, with a maximum of AFSCONF MAXKEYS (8). The keys themselves follow, each preceded by its key version number.
All information in this file is stored in network byte order. Each BOS Server converts the data to the appropriate host byte order befor storing and manipulating it.

Section 2.4: Restart Times

It is possible to manually start or restart any server defined within the set of BOS Server bnodes from any AFS client machine, simply by making the appropriate call to the RPC interface while authenticated as a valid administrator (i.e., a principal listed in the UserList file on the given machine). However, two restart situations merit the implementation of special functionality within the BOS Server. There are two common occasions, occuring on a regular basis, where the entire system or individual server programs should be brought down and restarted:
Complete system restart: To guard against the reliability and performance problems caused by any core leaks in long-running programs, the entire AFS system should be occasionally shut down and restarted periodically. This action 'clears out' the memory system, and may result in smaller memory images for these servers, as internal data structures are reinitialized back to their starting sizes. It also allows AFS partitions to be regularly examined, and salvaged if necessary.
Another reason for performing a complete system restart is to commence execution of a new release of the BOS Server itself. The new-binary restarts described below do not restart the BOS Server if a new version of its software has been installed on the machine.
New-binary restarts: New server software may be installed at any time with the assistance of the BOS Server. However, it is often not the case that such software installations occur as a result of the discovery of an error in the program or programs requiring immediate restart. On these occasions, restarting the given processes in prime time so that the new binaries may begin execution is counterproductive, causing system downtime and interfering with user productivity. The system administrator may wish to set an off-peak time when the server binaries are automatically compared to the running program images, and restarts take place should the on-disk binary be more recent than the currently running program. These restarts would thus minimize the resulting service disruption.
Automatically performing these restart functions could be accomplished by creating cron-type bnodes that were defined to execute at the desired times. However, rather than force the system administrator to create and supervise such bnodes, the BOS Server supports the notion of an internal LWP thread with the same effect (see Section 2.5.2). As part of the BosConfig file defined above, the administrator may simply specify the values for the complete system restart and new-binary restart times, and a dedicated BOS Server thread will manage the restarts.
Unless otherwise instructed, the BOS Server selects default times for the two above restart times. A complete system restart is carried out every Sunday at 4:00am by default, and a new-binary restart is executed for each updated binary at 5:00am every day.

Section 2.5: The bosserver Process

Section 2.5.1: Introduction

The user-space bosserver process is in charge of managing the AFS server processes and software images, the local security and cell database files, and allowing administrators to execute arbitrary programs on the server machine on which it runs. It also implements the RPC interface defined in the bosint.xg Rxgen definition file.

Section 2.5.2: Threading

As with all the other AFS server agents, the BOS Server is a multithreaded program. It is configured so that a minimum of two lightweight threads are guaranteed to be allocated to handle incoming RPC calls to the BOS Server, and a maximum of four threads are commissioned for this task.
In addition to these threads assigned to RPC duties, there is one other thread employed by the BOS Server, the BozoDaemon(). This thread is responsible for keeping track of the two major restart events, namely the system restart and the new binary restart (see Section 2.4). Every 60 seconds, this thread is awakened, at which time it checks to see if either deadline has occurred. If the complete system restart is then due, it invokes internal BOS Server routines to shut down the entire suite of AFS agents on that machine and then reexecute the BOS Server binary, which results in the restart of all of the server processes. If the new-binary time has arrived, the BOS Server shuts down the bnodes for which binaries newer than those running are available, and then invokes the new software.
In general, the following procedure is used when stopping and restarting processes. First, the restart() operation defined for each bnode's class is invoked via the BOP RESTART() macro. This allows each server to take any special steps required before cycling its service. After that function completes, the standard mechanisms are used to shut down each bnode's process, wait until it has truly stopped its execution, and then start it back up again.

Section 2.5.3: Initialization Algorithm

This section describes the procedure followed by the BOS Server from the time when it is invoked to the time it has properly initialized the server machine upon which it is executing.
The first check performed by the BOS Server is whether or not it is running as root. It needs to manipulate local unix files and directories in which only root has been given access, so it immediately exits with an error message if this is not the case. The BOS Server's unix working directory is then set to be /usr/afs/logs in order to more easily service incoming RPC requests to fetch the contents of the various server log files at this location. Also, changing the working directory in this fashion results in any core images dumped by the BOS Server's wards will be left in /usr/afs/logs.
The command line is then inspected, and the BOS Server determines whether it will insist on authenticated RPC connections for secure administrative operations by setting up the /usr/afs/local/NoAuth file appropriately (see Section 2.3.5). It initializes the underlying bnode package and installs the three known bnode types (simple, fs, and cron).
After the bnode package is thus set up, the BOS Server ensures that the set of local directories on which it will depend are present; refer to Section 2.2 for more details on this matter. The license file in /usr/afs/etc/License is then read to determine the number of AFS server machines the site is allowed to operate, and whether the cell is allowed to run the NFS/AFS Translator software. This file is typically obtained in the initial system installation, taken from the installation tape. The BOS Server will exit unless this file exists and is properly formatted.
In order to record its actions, any existing /usr/afs/logs/BosLog file is moved to BosLog.old, and a new version is opened in append mode. The BOS Server immediately writes a log entry concerning the state of the above set of important directories.
At this point, the BOS Server reads the BosConfig file, which lists the set of bnodes for which it will be responsible. It starts up the processes associated with the given bnodes. Once accomplished, it invokes its internal system restart LWP thread (covered in Section 2.5.2 above).
Rx initialization begins at this point, setting up the RPC infrastructure to receive its packets on the AFSCONF NANNYPORT, UDP port 7007. The local cell database is then read and internalized, followed by acquisition of the AFS encryption keys.
After all of these steps have been carried out, the BOS Server has gleaned all of the necessary information from its environemnt and has also started up its wards. The final initialization action required is to start all of its listener LWP threads, which are devoted to executing incoming requests for the BOS Server RPC interface.

Section 2.5.4: Command Line Switches

The BOS Server recognizes exactly one command line argument: -noauth. By default, the BOS Server attempts to use authenticated RPC connections (unless the /usr/afs/local/NoAuth file is present; see Section 2.3.5). The appearance of the -noauth command line flag signals that this server incarnation is to use unauthenticated connections for even those operations that are normally restricted to system administrators. This switch is essential during the initial AFS system installation, where the procedures followed to bootstrap AFS onto a new machine require the BOS Server to run before some system databases have been created.