Debian Reference

Osamu Aoki

This book is free; you may redistribute it and/or modify it under the terms of the GNU General Public License of any version compliant to the Debian Free Software Guidelines (DFSG).

Abstract

This Debian Reference (v2) (2009-01-17 17:51:59 UTC) is intended to provide a broad overview of the Debian system as a post-installation user's guide. It covers many aspects of system administration through shell-command examples for non-developers.


Table of Contents

1. Preface
1.1. Disclaimer
1.2. What is Debian
1.3. About this document
1.3.1. Guiding rules
1.3.2. Prerequisites
1.3.3. Conventions
1.3.4. The popcon
1.3.5. The package size
1.3.6. Bug reports
1.4. Some quotes for new users
2. GNU/Linux tutorials
2.1. Console basics
2.1.1. The shell prompt
2.1.2. The shell prompt under X
2.1.3. The root account
2.1.4. The root shell prompt
2.1.5. GUI system administration tools
2.1.6. Virtual consoles
2.1.7. How to leave the command prompt
2.1.8. How to shutdown the system
2.1.9. Recovering a sane console
2.1.10. Additional package suggestions for the newbie
2.1.11. An extra user account
2.1.12. sudo configuration
2.1.13. Play time
2.2. Unix-like filesystem
2.2.1. Unix file basics
2.2.2. Filesystem internals
2.2.3. The filesystem permission system
2.2.4. Control of permissions for newly created files: umask
2.2.5. Permissions for groups of users (group)
2.2.6. Timestamps
2.2.7. Links
2.2.8. Named pipes (FIFOs)
2.2.9. Sockets
2.2.10. Device files
2.2.11. Special device files
2.2.12. procfs and sysfs
2.3. Midnight Commander (MC)
2.3.1. Customization of MC
2.3.2. Starting MC
2.3.3. File manager in MC
2.3.4. Command-line tricks in MC
2.3.5. The internal editor in MC
2.3.6. The internal viewer in MC
2.3.7. Auto-start features of MC
2.3.8. FTP virtual filesystem of MC
2.4. The basic Unix-like work environment
2.4.1. The login shell
2.4.2. Customizing bash
2.4.3. Special key strokes
2.4.4. Unix style mouse operations
2.4.5. The pager
2.4.6. The text editor
2.4.7. Setting a default text editor
2.4.8. Customizing vim
2.4.9. Recording the shell activities
2.4.10. Basic Unix commands
2.5. The simple shell command
2.5.1. Command execution and environment variable
2.5.1.1. LANG variable
2.5.1.2. PATH variable
2.5.1.3. HOME variable
2.5.2. Command line options
2.5.3. Shell glob
2.5.4. Return value of the command
2.5.5. Typical command sequences and shell redirection
2.5.6. Command alias
2.6. Unix-like text processing
2.6.1. Unix text tools
2.6.2. Regular expressions
2.6.3. Replacement expressions
2.6.4. Extract data from text file table
2.6.5. Script snippets for piping commands
2.6.6. Perl one liner for the regular-expression substitution
3. Debian package management
3.1. Debian package management prerequisites
3.1.1. Package configuration
3.1.2. Basic precautions
3.1.3. Life with eternal upgrades
3.1.4. Debian archive basics
3.1.5. Package dependencies
3.1.6. The event flow of the package management
3.1.7. First response to package management troubles
3.2. Basic package management operations
3.2.1. Basic package management operations with commandline
3.2.2. Interactive use of aptitude
3.2.3. Key bindings of aptitude
3.2.4. Package views under aptitude
3.2.5. Search method options with aptitude
3.2.6. The aptitude regex formula
3.2.7. Dependency resolution of aptitude
3.2.8. Package activity logs
3.2.9. Aptitude advantages
3.3. Examples of aptitude operations
3.3.1. List packages with regex matching package name
3.3.2. Browse with the regex matching
3.3.3. Purge removed packages for good
3.3.4. Tidy auto/manual install status
3.3.5. System wide upgrade with aptitude
3.4. Advanced package management operations
3.4.1. Advanced management operations with commandline
3.4.2. Verify installed package files
3.4.3. Safeguard for package problems
3.4.4. Search on the package meta data
3.5. Debian package management internals
3.5.1. Archive meta data
3.5.2. Top level Release file and authenticity
3.5.3. Archive level Release files
3.5.4. Fetching of the meta data for the package
3.5.5. The package state for APT
3.5.6. The package state for aptitude
3.5.7. The local copies of the fetched packages
3.5.8. The Debian package file name
3.5.9. The dpkg command
3.5.10. The update-alternative command
3.5.11. The dpkg-statoverride command
3.5.12. The dpkg-divert command
3.6. Recovery from a broken system
3.6.1. Incompatibility with old user configuration
3.6.2. Different packages with overlapped files
3.6.3. Fixing broken package script
3.6.4. Rescue using the dpkg command
3.6.5. Recover package selection data
3.7. Tips for the package management
3.7.1. How to pick Debian packages
3.7.2. Packages from mixed source of archives
3.7.3. Tweaking candidate version
3.7.4. Volatile and Backports.org
3.7.5. Emergency downgrading
3.7.6. Who uploaded the package?
3.7.7. The equivs package
3.7.8. Port a package to the stable system
3.7.9. Proxy server for APT
3.7.10. Small public package archive
3.7.11. Record/copy system configuration
3.7.12. Convert or install an alien binary package
3.7.13. Extract package without dpkg
3.7.14. More readings for the package management
4. The system initialization
4.1. An overview of the boot strap process
4.2. Stage 1: the BIOS
4.3. Stage 2: the boot loader
4.4. Stage 3: the mini-Debian system
4.5. Stage 4: the normal Debian system
4.5.1. The meaning of the runlevel
4.5.2. The configuration of the runlevel
4.5.3. The runlevel management example
4.5.4. The default parameter for each init script
4.5.5. The hostname
4.5.6. Network interface initialization
4.5.7. Network service initialization
4.5.8. The system message
4.5.9. The kernel message
4.5.10. The udev system
4.5.11. The kernel module initialization
5. Authentication
5.1. Normal Unix authentication
5.2. Manage account and password information
5.3. Good password
5.4. Creating encrypted password
5.5. PAM and NSS
5.5.1. Configuration files accessed by the PAM and NSS
5.5.2. The modern centralized system management
5.5.3. "Why GNU su does not support the wheel group"
5.5.4. Stricter password rule
5.6. Other access controls
5.6.1. sudo
5.6.2. SELinux
5.6.3. Restricting access to some server services
5.7. Security of authentication
5.7.1. Secure password over the Internet
5.7.2. Secure Shell
5.7.3. Extra security measures for the Internet
5.7.4. Securing the root password
6. Network setup
6.1. The basic network infrastructure
6.1.1. The domain name
6.1.2. The hostname resolution
6.1.3. The network interface name
6.1.4. The network address range for the LAN
6.1.5. The network configuration infrastructure
6.1.6. The network device support
6.2. The network connection method
6.2.1. The DHCP connection with the Ethernet
6.2.2. The static IP connection with the Ethernet
6.2.3. The PPP connection with pppconfig
6.2.4. The alternative PPP connection with wvdialconf
6.2.5. The PPPoE connection with pppoeconf
6.3. The basic network configuration with ifupdown
6.3.1. The command syntax simplified
6.3.2. The basic syntax of /etc/network/interfaces
6.3.3. The loopback network interface
6.3.4. The network interface served by the DHCP
6.3.5. The network interface with the static IP
6.3.6. The basics of wireless LAN interface
6.3.7. The wireless LAN interface with WEP
6.3.8. The wireless LAN interface with WPA/WPA2
6.3.9. The PPP connection
6.3.10. The alternative PPP connection
6.3.11. The PPPoE connection
6.3.12. The network configuration state of ifupdown
6.3.13. The basic network reconfiguration
6.3.14. The ifupdown-extra package
6.4. The advanced network configuration with ifupdown
6.4.1. The ifplugd
6.4.2. The ifmetric
6.4.3. The virtual interface
6.4.4. The advanced command syntax
6.4.5. The mapping stanza
6.4.6. The manually switchable network configuration
6.4.7. Scripting with the ifupdown
6.4.8. Mapping with guessnet
6.5. The network configuration for desktop
6.5.1. Automatic network configuration
6.5.2. GUI network configuration tools
6.6. The low level network configuration
6.6.1. Iproute2 commands
6.6.2. Safe lower level network operations
6.7. Network optimization
6.7.1. Finding optimal MTU
6.7.2. Setting MTU
6.7.3. WAN TCP optimization
6.8. Netfilter
7. Network applications
7.1. Web browsers
7.1.1. Browser configuration
7.2. The mail system
7.2.1. Modern mail service basics
7.2.2. Basic mail software choice
7.2.3. The mail configuration strategy for workstation
7.2.3.1. The configuration of exim4
7.2.3.2. The configuration of postfix with SASL
7.2.3.3. The mail address configuration
7.2.4. Tips for managing the mail
7.2.4.1. Basic MTA operations
7.2.4.2. Basic MUA -- Mutt
7.2.4.3. Redeliver mbox contents
7.2.5. Choices of software for the mail
7.2.5.1. MTA
7.2.5.2. MUA
7.2.5.3. The remote mail retrieval and forward utility
7.2.5.4. MDA
7.2.5.5. POP3/IMAP4 server
7.3. The print server and utility
7.4. The remote access server and utility (SSH)
7.4.1. Basics of SSH
7.4.2. Port forwarding for SMTP/POP3 tunneling
7.4.3. Connecting with fewer passwords -- RSA
7.4.4. Dealing with alien SSH clients
7.4.5. Setting up ssh-agent
7.4.6. Troubleshooting SSH
7.5. Other network application servers
7.6. Other network application clients
7.7. The diagnosis of the system daemons
8. The X window system
8.1. Key packages
8.2. Setting up desktop environment
8.2.1. Debian menu
8.2.2. Freedesktop.org menu
8.2.3. Debian menu under GNOME desktop environment
8.3. The server/client relationship
8.4. The X server
8.4.1. The (re)configuration of the X server
8.4.2. The connection methods to the X server
8.5. Starting the X window system
8.5.1. Starting X session with gdm
8.5.2. Customizing the X session (classic method)
8.5.3. Customizing the X session (new method)
8.5.4. Connecting a remote X client via SSH
8.5.5. Secure X terminal via Internet
8.6. Fonts in the X window
8.6.1. Basic fonts
8.6.2. Additional fonts
8.6.3. CJK fonts
8.7. X applications
8.7.1. X office applications
8.7.2. X utility applications
8.8. The X trivia
8.8.1. Keymaps and pointer button mappings in X
8.8.2. The classic X clients
8.8.3. The X terminal emulator -- xterm
8.8.4. Running X clients as root
9. The I18N and L10N
9.1. The keyboard input
9.1.1. The input method support with scim
9.1.2. An example for Japanese
9.1.3. Disabling the input method
9.2. The display output
9.3. The locale
9.3.1. Basics of encoding
9.3.2. Rationale for UTF-8 locale
9.3.3. The reconfiguration of the locale
9.3.4. Configuring system with LANG
9.3.5. Specific locale only under X Window
9.3.6. Filename encoding
9.3.7. Localized messages and translated documentation
9.3.8. Effects of locale
10. System tips
10.1. The screen program
10.1.1. The use scenario for the screen command
10.1.2. Key bindings for the screen command
10.1.3. A screen-like program for X window system
10.2. Data recording and presentation
10.2.1. The log daemon
10.2.2. Log analyzer
10.2.3. Recording the shell activities cleanly
10.2.4. Customized display of text data
10.2.5. Colorized shell echo
10.2.6. Colorized commands
10.2.7. Recording the graphic image of an X application
10.2.8. Recording changes in configuration files
10.3. Data storage tips
10.3.1. Partition configuration
10.3.2. Accessing partition using UUID
10.3.3. Filesystem configuration
10.3.4. Filesystem creation and integrity check
10.3.5. Optimization of filesystem by mount options
10.3.6. Optimization of filesystem via superblock
10.3.7. Optimization of harddisk
10.3.8. Expand usable storage space via LVM
10.3.9. Expand usable storage space by mounting another partition
10.3.10. Expand usable storage space using symlink
10.3.11. Expand usable storage space using aufs
10.4. Data encryption tips
10.4.1. Removable disk encryption with dm-crypt/LUKS
10.4.2. Encrypted swap partition with dm-crypt
10.4.3. Automatically encrypting files with eCryptfs
10.4.4. Automatically mounting eCryptfs
10.5. Monitoring, controlling, and starting program activities
10.5.1. Time a process
10.5.2. The scheduling priority
10.5.3. The ps command
10.5.4. The top command
10.5.5. List files opened by a process
10.5.6. Trace program activities
10.5.7. Identify processes using files or sockets
10.5.8. Repeating a command with a constant interval
10.5.9. Repeating a command looping over files
10.5.10. Starting a program from GUI
10.5.11. Customizing program to be started
10.5.12. Kill a process
10.5.13. Schedule tasks once
10.5.14. Schedule tasks regularly
10.5.15. Alt-SysRq
10.6. System maintenance tips
10.6.1. Who is logged on
10.6.2. Warn everyone
10.6.3. The hardware identification
10.6.4. The hardware configuration
10.6.5. System and hardware time
10.6.6. The terminal configuration
10.6.7. The sound infrastructure
10.6.8. Disable the screen saver
10.6.9. Disable the sound (beep)
10.6.10. Memory usage
10.6.11. System security and integrity check
10.7. The kernel
10.7.1. Linux kernel 2.6
10.7.2. Kernel headers
10.7.3. Kernel and module compile
10.7.4. Kernel source compile: Debian standard method
10.7.5. Module source compile: Debian standard method
10.7.6. Kernel source compile: classic method
10.7.7. Non-free hardware drivers
10.8. The chroot
10.8.1. Run a different Debian distribution with chroot
10.8.2. Setting up login for chroot
10.8.3. Setting up X for chroot
10.8.4. Run other distributions with chroot
10.8.5. Build packages under chroot
10.8.6. Other virtualization tools
11. Data management
11.1. Sharing, copying, and archiving
11.1.1. Archive and compression tools
11.1.2. Copy and synchronization tools
11.1.3. Idioms for the archive
11.1.4. Idioms for the copy
11.1.5. Idioms for the selection of files
11.1.6. Backup and recovery
11.1.7. Backup utility suites
11.1.8. An example script for the system backup
11.1.9. A copy script for the data backup
11.1.10. Removable mass storage device
11.1.11. Sharing data via network
11.1.12. Archive media
11.2. The binary data
11.2.1. Make the disk image file
11.2.2. Writing directly to the disk
11.2.3. View and edit binary data
11.2.4. Mount the disk image file
11.2.5. Manipulating files without mounting disk
11.2.6. Make the ISO9660 image file
11.2.7. Writing directly to the CD/DVD-R/RW
11.2.8. Mount the ISO9660 image file
11.2.9. Split a large file into small files
11.2.10. Clear file contents
11.2.11. Dummy files
11.2.12. Erase entire harddisk
11.2.13. Undelete deleted but still open file
11.2.14. Searching all hardlinks
11.2.15. Invisible disk space consumption
11.3. Data security infrastructure
11.3.1. Key management for Gnupg
11.3.2. Using GnuPG with files
11.3.3. Using GnuPG with Mutt
11.3.4. Using GnuPG with Vim
11.3.5. The MD5 sum
11.4. Source code merge tools
11.4.1. Extract differences for source files
11.4.2. Merge updates for source files
11.4.3. 3 way merge updates
11.5. Version control systems
11.5.1. Native VCS commands
11.5.2. CVS
11.5.2.1. Installing a CVS server
11.5.2.2. Use local CVS server
11.5.2.3. Use remote CVS pserver
11.5.2.4. Anonymous CVS (download only)
11.5.2.5. Use remote CVS through ssh
11.5.2.6. Create a new CVS archive
11.5.2.7. Work with CVS
11.5.2.8. Export files from CVS
11.5.2.9. Administer CVS
11.5.2.10. File permissions in repository
11.5.2.11. Execution bit
11.5.3. Subversion
11.5.3.1. Installing a Subversion server
11.5.3.2. Setting up a repository
11.5.3.3. Configuring Apache2
11.5.3.4. Subversion usage examples
11.5.3.5. Create a new Subversion archive
11.5.3.6. Working with Subversion
11.5.4. Git
11.5.4.1. Before using Git
11.5.4.2. Git references
11.5.4.3. Git commands
11.5.4.4. Git for recording configuration history
12. Data conversion
12.1. Text data conversion tools
12.1.1. To convert a text file with iconv
12.1.2. To convert file names with iconv
12.1.3. EOL conversion
12.1.4. TAB conversion
12.1.5. Editors with auto-conversion
12.1.6. Plain text extraction
12.1.7. Highlighting and formatting plain text data
12.2. XML data
12.2.1. Basic hints for XML
12.2.2. XML processing
12.2.3. The XML data extraction
12.3. Printable data
12.3.1. The Ghostscript
12.3.2. Merge two PS or PDF files
12.3.3. Printable data utilities
12.3.4. Printing with CUPS
12.4. Type setting
12.4.1. roff typesetting
12.4.2. TeX/LaTeX
12.4.3. Pretty print a manual page
12.4.4. Creating a manual page
12.5. The mail data conversion
12.5.1. Mail data basics
12.6. Graphic data tools
12.7. Miscellaneous data conversion
13. Programming
13.1. The shell script
13.1.1. POSIX shell compatibility
13.1.2. Shell parameters
13.1.3. Shell conditionals
13.1.4. Shell loops
13.1.5. The shell command-line processing sequence
13.1.6. Utility programs for shell script
13.1.7. Shell script dialog
13.1.8. Shell script example with zenity
13.2. Make
13.3. C
13.3.1. Simple C program (gcc)
13.3.2. Debugging with gdb
13.3.2.1. Basic gdb execution
13.3.2.2. Debugging the Debian package
13.3.2.3. Obtaining backtrace
13.3.2.4. Advanced gdb commands
13.3.2.5. Debugging X Errors
13.3.3. Check dependency on libraries
13.3.4. Debugging with memory leak detection tools
13.3.5. Disassemble binary
13.3.6. Flex -- a better Lex
13.3.7. Bison -- a better Yacc
13.4. Autoconf
13.4.1. Compile and install a program
13.4.2. Uninstall program
13.5. Perl short script madness
13.6. Web
13.7. Static code analysis tools
13.8. The source code translation
13.9. Making Debian package
A. Appendix
A.1. The Debian maze
A.2. Copyright history
A.3. Document format

List of Tables

2.1. List of interesting text-mode program packages.
2.2. List of informative documentation packages.
2.3. List of usage of key directories.
2.4. List of the first character of "ls -l" output
2.5. The numeric mode for file permissions in chmod(1) commands.
2.6. The umask value examples.
2.7. List of example system-provided groups for file access.
2.8. List of notable system provided groups for particular command executions.
2.9. List of types of timestamps.
2.10. The device types.
2.11. List of special device files.
2.12. The key bindings of MC.
2.13. The reaction to the enter key in MC.
2.14. List of shell programs.
2.15. List of key bindings for bash.
2.16. The Unix style mouse operations.
2.17. List of basic Unix commands.
2.18. The 3 parts of locale value.
2.19. List of locale recommendations.
2.20. List of HOME values.
2.21. The shell glob patterns.
2.22. Command exit code.
2.23. The shell command idioms.
2.24. The predefined file descriptors.
2.25. The metacharacters for BRE and ERE.
2.26. The replacement expression.
2.27. The script snippets for piping commands.
2.28. The comparison of ed vs perl for in-place editing.
3.1. List of Debian package management tools
3.2. Lists of Debian archive sites.
3.3. The lists of Debian archive components.
3.4. The relationship between suite and codename.
3.5. List of package dependencies.
3.6. List of key web site to resolving problems with a specific package.
3.7. Package management operations with commandline using aptitude and apt-get / apt-cache.
3.8. Notable command options for "aptitude".
3.9. List of key bindings for aptitude.
3.10. Views for aptitude.
3.11. The categorization of standard aptitude views.
3.12. List of the aptitude regex formula.
3.13. The log files for package activities.
3.14. List of advanced package management operations.
3.15. The content of the Debian archive meta data.
3.16. The name structures of the Debian packages.
3.17. The usable characters for each component in the Debian package names.
3.18. The notable files for dpkg.
3.19. List of essential default Pin-Priority values.
3.20. List of the proxy tools specially for Debian archive
4.1. List of boot loaders.
4.2. The meaning of GRUB parameters.
4.3. List of runlevels and meanings.
4.4. List of kernel error levels.
5.1. The 3 important configuration files for pam_unix.o.
5.2. The second entry content of /etc/passwd.
5.3. List of commands to manage account information.
5.4. List of tools to generate password.
5.5. List of notable PAM and NSS systems.
5.6. List of configuration files accessed by the PAM.
5.7. List of insecure and secure services and ports.
5.8. List of tools to provide extra security measures.
6.1. List of network address ranges.
6.2. List of network configuration tools.
6.3. List of network connection types and connection paths.
6.4. List of network connection configurations.
6.5. List of network connection acronyms.
6.6. List of configuration files for the PPP connection with pppconfig.
6.7. List of configuration files for the PPP connection with wvdialconf.
6.8. List of configuration files for the PPPoE connection with pppoeconf.
6.9. List of basic network configuration commands with ifupdown.
6.10. List of stanzas in /e/n/i
6.11. List of acronyms for WLAN.
6.12. List of terminology for network devices.
6.13. List of advanced network configuration commands with ifupdown.
6.14. List of environment variables passed by the ifupdown system
6.15. Translation table from obsolete net-tools commands to new iproute2 commands.
6.16. List of lower level network commands.
6.17. List of network optimization tools.
6.18. Basic guide lines of the optimal MTU value
6.19. List of firewall tools.
7.1. List of web browsers.
7.2. List of browser plugin packages.
7.3. List of popular mail system for workstation.
7.4. List of important postfix manual pages
7.5. List of mail address related configuration files.
7.6. List of basic MTA operation.
7.7. List of MTA.
7.8. List of MUA.
7.9. List of remote mail retrieval and forward utilities.
7.10. List of MDA.
7.11. List of POP3/IMAP4 servers.
7.12. List of print servers and utilities.
7.13. List of remote access server and utilities.
7.14. List of SSH authentication protocols and methods.
7.15. List of SSH configuration files.
7.16. List of SSH client startup examples.
7.17. List of free SSH clients for other platforms.
7.18. List of other network application servers.
7.19. List of network application clients.
7.20. List of popular RFCs.
8.1. List of key (meta)packages for X window.
8.2. List of server/client terminology.
8.3. List of connection methods to the X server.
8.4. Table of packages to support X window font systems.
8.5. Table of corresponding PostScript Type 1 fonts.
8.6. Table of corresponding TrueType fonts.
8.7. Table of key words used in CJK font names to indicate font types.
8.8. List of basic X office applications
8.9. List of basic X utility applications
9.1. List of keyboard reconfiguration methods.
9.2. List of input method supports with scim.
10.1. List of programs to support interrupted network connections.
10.2. List of key bindings for screen.
10.3. List of system log analyzers.
10.4. List of graphic image manipulation tools.
10.5. List of packages to record configuration history in VCS.
10.6. List of disk partition management packages
10.7. List of filesystem management packages
10.8. List of data encryption utilities.
10.9. List of tools for monitoring and controlling program activities
10.10. List of nice values for the scheduling priority.
10.11. List of ps command styles.
10.12. List of commands for top.
10.13. List of frequently used signals for kill command.
10.14. List of SAK command keys.
10.15. List of hardware identification tools.
10.16. List of hardware configuration tools.
10.17. List of sound packages
10.18. List of commands for disabling the screen saver.
10.19. List of memory sizes reported.
10.20. List of tools for system security and integrity check
10.21. List of key packages to be installed for the kernel recompilation on the Debian system
10.22. List of virtualization tools
11.1. List of archive and compression tools.
11.2. List of copy and synchronization tools.
11.3. List of backup suite utilities.
11.4. List of packages which permit normal users to mount removable devices without a matching /etc/fstab entry.
11.5. List of filesystem choices for removable storage devices with typical usage scenarios.
11.6. List of the network service to chose with the typical usage scenario.
11.7. List of packages which view and edit binary data.
11.8. List of packages to manipulate files without mounting.
11.9. List of data security infrastructure tools.
11.10. List of GNU Privacy Guard commands for the key management
11.11. List of the meaning of trust code.
11.12. List of gnu privacy guard commands on files
11.13. List of source code merge tools.
11.14. List of version control system tools.
11.15. Comparison of native VCS commands.
11.16. Assumption for the CVS archive.
11.17. Notable options for CVS commands (use as first argument(s) to cvs).
11.18. List of git packages and commands.
12.1. List of text data conversion tools.
12.2. List of encoding values and their usage.
12.3. List of EOL conversion tools.
12.4. List of TAB conversion commands from bsdmainutils and coreutils packages.
12.5. List of tools to extract plain text data.
12.6. List of tools to highlight plain text data.
12.7. List of predefined entities for XML.
12.8. List of XML tools.
12.9. List of DSSL tools.
12.10. List of XML data extraction tools.
12.11. List of XML pretty print tools.
12.12. List of Ghostscript PostScript interpreters.
12.13. List of printable data utilities.
12.14. List of type setting tools.
12.15. List of packages to help creating the manpage.
12.16. List of packages to help mail data conversion.
12.17. List of graphic data tools.
12.18. List of miscellaneous data conversion tools.
13.1. List of packages to help programing.
13.2. List of bashizms.
13.3. List of shell parameters.
13.4. List of shell parameter expansions.
13.5. List of key shell parameter substitutions.
13.6. List of file comparison operators in the conditional expression.
13.7. List of string comparison operators in the conditional expression.
13.8. List of user interface programs.
13.9. List of make automatic variables.
13.10. List of make variable expansions.
13.11. List of advanced gdb commands
13.12. List of memory leak detection tools
13.13. List of Yacc-compatible LALR parser generators
13.14. List of tools for static code analysis
13.15. List of source code translation tools.

Chapter 1. Preface

This Debian Reference (version 2) (2009-01-17 17:51:59 UTC) is intended to provide a broad overview of Debian system administration as a post-installation user guide.

The target reader is someone who is willing to learn shell scripts but who is not ready to read all the C sources to figure out how the GNU/Linux system works.

1.1. Disclaimer

All warranties are disclaimed. All trademarks are property of their respective trademark owners.

The Debian system itself is a moving target. This makes its documentation difficult to be current and correct. Although the current unstable version of Debian system was used as the basis for writing this, some contents may be already outdated by the time you read this.

Please treat this document as the secondary reference. This document does not replace any authoritative guides. The author and contributors do not take responsibility for consequences of errors, omissions or ambiguity in this document.

1.2. What is Debian

The Debian Project is an association of individuals who have made common cause to create a free operating system. It's distribution is characterized by:

Free Software pieces in Debian come from GNU, Linux, BSD, X, Berkeley DB, ISC, Apache, Ghostscript, Common Unix Printing System , Samba, GNOME, KDE, Mozilla, OpenOffice.org, Vim, TeX, LaTeX, DocBook, Perl, Python, Tcl, Java, Ruby, PHP, MySQL and many more independent free software projects. Debian integrates this diversity of Free Software into one system.

1.3. About this document

1.3.1. Guiding rules

Following guiding rules were followed while compiling this document:

  • don't explain in details what is well documented elsewhere (in the judgment of the author)

  • Provide overview and skip corner cases (Big Picture)

  • Keep It Short and Simple. (KISS)

  • Focus on non-GUI tools and consoles. (Use shell examples)

  • Do not reinvent the wheel. (Use pointers to the existing references)

  • Be objective. (Use popcon etc.)

I tried to elucidate hierarchical aspects and lower levels of the system.

1.3.2. Prerequisites

You are required to seek help from (in approximate order of importance, starting with the most important sources):

[Note] Note

For detailed documentation, you may need to install the corresponding documentation package named with "-doc" as its suffix.

1.3.3. Conventions

This document refers to some command names followed by the number in parenthesis , such as bash(1). This means that you are encouraged to obtain information from the manpage by typing the following on the shell command prompt:

$ man 1 bash 

This document provides information through the following simplified presentation style with bash(1) shell command examples and bullets:

# <command in root account>
$ <command in user account>
  • <description of action>

These shell prompts distinguish account used and correspond to PS1='\$' and PS2=' ' environment variables. These values are chosen for the sake of readability of this document and are not typical on actual installed system.

[Note] Note

See the meaning of the PS1 and PS2 environment variables in the manpage of bash(1) command.

1.3.4. The popcon

The popcon data was presented as the objective measure for the popularity of each package. It was downloaded on 2009-01-18 and contains the total submission of 79097 reports over 91823 binary packages and 20 architectures.

[Note] Note

Please note that the amd64 unstable archive contains only 24396 packages currently. The popcon data contains reports from many old system installations.

The popcon number preceded with "V:" for "votes" is calculated by 100 * (the popcon submissions for the package executed recently on the PC)/(the total popcon submissions).

The popcon number preceded with "I:" for "installs" is calculated by 100 * (the popcon submissions for the package installed on the PC)/(the total popcon submissions).

[Note] Note

The popcon figures should not be considered as absolute measures of the importance of packages. There are many factors which can skew statistics. For example, some system participating popcon may have mounted directories such as /bin with "noatime" option for system performance improvement and effectively disabled "vote" from such system.

1.3.5. The package size

The package size data was also presented as the objective measure for each package. It is based on the "Installed-Size:" reported by "apt-cache show " command (currently on amd64 architecture for the unstable release). The reported size is in KB (unit for 1024 bytes).

[Note] Note

A package with a small numerical package size may indicate that the package in the unstable release has a dummy package which installs other packages with significant contents by the dependency. The dummy package enables a smooth transition or split of the package.

1.3.6. Bug reports

Please file bug reports on "debian-reference" package using reportbug(1) if you find any issues. For simple spell errors and grammar corrections, you may alternatively edit the source text available as Debian Reference wiki page at http://wiki.debian.org/DebianReference and tell me to update the document.

1.4. Some quotes for new users

Here are some interesting quotes from the Debian mailing list which may help enlighten new users:

  • "This is Unix. It gives you enough rope to hang yourself." --- Miquel van Smoorenburg <miquels at cistron.nl>

  • "Unix IS user friendly... It's just selective about who its friends are." --- Tollef Fog Heen <tollef at add.no>

Chapter 2. GNU/Linux tutorials

I think learning a computer system is like learning a new foreign language. Although tutorial books and documentation are helpful, you have to practice it yourself. In order to help you get started smoothly, I will elaborate a few basic points.

The powerful design of Debian GNU/Linux comes from the Unix operating system, i.e., a multiuser, multitasking operating system. You must learn to take advantage of the power of these features and the similarities between Unix and GNU/Linux.

Don't shy away from Unix oriented texts and don't rely solely on GNU/Linux texts, as this will rob you of much useful information.

"Rute User's Tutorial and Exposition", in the Debian non-free archive as rutebook package (popcon: I:0.2), provides a good online resource to the generic system administration.

[Note] Note

If you have been using any Unix-like system for a while with command line tools, you probably know everything I explain here. Please use this as a reality check and refresher.

2.1. Console basics

2.1.1. The shell prompt

Upon starting the system, you are presented with the character based login screen if you did not install X Window System with the display manager such as gdm. Suppose your hostname is foo, the login prompt looks like:

foo login:

If you did install a GUI environment such as Gnome of KDE, then you can get to a prompt by Ctrl-Alt-F1, and you can return to the GUI screen via Alt-F7 (see Virtual Consoles below for more information).

Following what you selected during the installation process, you type your username, e.g. penguin, and press the Enter-key, then type your password and press the Enter-key again.

[Note] Note

Following the Unix tradition, the username and password of the Debian system are case sensitive. The username is usually chosen only from the lowercase.

Now you are in the shell. The shell interprets your commands. The system starts with the greeting message stored in /etc/motd (Message Of The Day) and with the command prompt as:

Debian GNU/Linux lenny/sid foo tty1
foo login: penguin
Password:
Last login: Sun Apr 22 09:29:34 2007 on tty1
Linux snoopy 2.6.20-1-amd64 #1 SMP Sun Apr 15 20:25:49 UTC 2007 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
foo:~$

Here, the main part of the greeting message can be customized by editing the /etc/motd.tail file. The first line is generated from the system information using "uname -snrvm".

2.1.2. The shell prompt under X

If you installed X Window System with a display manager such as Gnome's gdm by selecting "Desktop environment" task during the installation, you will be presented with the graphical login screen upon starting your system. You type your username and your password to login to the non-privileged user account. Use tab to navigate between username and password, or use the mouse and primary click.

You can gain the shell prompt under X by starting a x-terminal-emulator program such as gnome-terminal, rxvt or xterm. Under the Gnome Desktop environment, clicking "Applications" -> "Accessories" -> "Terminal" does the trick.

You can also see the section below Section 2.1.6, “Virtual consoles”.

Under some other Desktop systems (like fluxbox), there may be no obvious starting point for the menu. If this happens, just try (right) clicking the center of the screen and hope for a menu to pop-up.

2.1.3. The root account

The root account is also called superuser or privileged user. From this account, you can perform the following system administration activities:

  • read, write, and remove any files on the system irrespective of their file permissions

  • set file ownership and permission of any files on the system

  • set the password of any non-privileged users on the system

  • login to any accounts without their passwords

This unlimited power of root account requires you to be considerate and responsible when using it.

[Note] Note

The file permission of the file (including hardware devices such as CD-ROM etc. which are just another file for the Debian system) may render them unusable or inaccessible by non-root users. Although the use of root account is a quick way to test this kind of situation, the resolution of this situation should be done through proper setting of the access permission and the group.

[Warning] Warning

Never share the root password with others.

2.1.4. The root shell prompt

Here are a few basic methods to gain the root shell prompt by using the root password:

  • At the character based login prompt, you simply type root.

  • Under the Gnome Desktop environment, click "Applications" -> "Accessories" -> "Root Terminal".

  • From any user shell prompt, type "su -l". (This does not preserve the environment of the current user)

  • From any user shell prompt, type "su". (This preserves most of the environment of the current user)

2.1.5. GUI system administration tools

When your desktop menu does not start GUI system administration tools automatically with the appropriate privilege, you can start them from the root shell prompt of the X terminal emulator, such as gnome-terminal, rxvt, or xterm. See Section 2.1.4, “The root shell prompt”.

[Warning] Warning

Never start the X server under the root account by typing in root to the prompt of the display manager such as gdm because it is considered unsafe (insecure), even when you plan to perform administrative activities. The entire X architecture is considered insecure if run as root. You must always use the lowest privilege level possible, like a regular user account.

[Warning] Warning

Never run untrusted remote GUI program under X window when critical information is displayed since it may eavesdrop your X screen.

2.1.6. Virtual consoles

In the default Debian system, there are six switchable VT100-like character consoles available to start the command shell directly on the Linux host. Unless you are in a GUI environment, you can switch between the virtual consoles by pressing the Left-Alt-key and one of the F1--F6 keys simultaneously. Each character console allows independent login to the account and offers the multiuser environment. This multiuser environment is a great Unix feature, and very addictive.

If you are under the X Window System, you gain access to the character console 1 by pressing Ctrl-Alt-F1 key, i.e., the left-Ctrl-key, the left-Alt-key, and the F1-key are pressed together. You can get back to the X Window System, normally running on the virtual console 7, by pressing Alt-F7.

You can alternatively change to another virtual console, e.g. to the console 1, by the command:

# chvt 1

2.1.7. How to leave the command prompt

You type Ctrl-D, i.e., the left-Ctrl-key and the d-key pressed together, at the command prompt to close the shell activity. If you are at the character console, you will return to the login prompt with this. Even though these control characters are referred as "control D" with the upper case, you do not need to press the Shift-key. The short hand expression, ^D, is also used for Ctrl-D. Alternately, you can type "exit".

If you are at x-terminal-emulator, you can close x-terminal-emulator window with this.

2.1.8. How to shutdown the system

Just like any other modern OS where the file operation involves caching data in memory for improved performance, the Debian system needs the proper shutdown procedure before power can safely be turned off. This is to maintain the integrity of files, by forcing all changes in memory to be written to disk. If the software power control is available, the shutdown procedure automatically turns off power of the system. (Otherwise, you may have to press power button for few seconds after the shutdown procedure.)

Under the normal multiuser mode, use following from the root command prompt to shutdown the system:

# shutdown -h now

Under the single-user mode, use following from the root command prompt to shutdown the system:

# poweroff -i -f

Alternatively, you may type Ctrl-Alt-Delete (The left-Ctrl-key, the left-Alt-Key, and the Delete are pressed together) to shutdown if /etc/inittab contains "ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -h now" in it. See manpage for inittab(5) for details.

2.1.9. Recovering a sane console

When the screen goes berserk after doing some funny things such as "cat <some-binary-file>", type "reset" at the command prompt. You may not be able to see the command echoed as you type. You may also issue "clear" to clean up the screen.

2.1.10. Additional package suggestions for the newbie

Although even the minimal installation of the Debian system without any desktop environment tasks provides the basic Unix functionality, it is a good idea to install few additional commandline and curses based character terminal packages such as mc and vim with aptitude for beginners to get started. From the shell prompt as root:

# aptitude update
...
# aptitude install mc vim sudo
...

If you already had these packages installed, nothing will be installed.

Table 2.1.  List of interesting text-mode program packages.

package

popcon

size

description

mc

V:10, I:25

6328

A text-mode full-screen file manager

vim

V:12, I:29

1736

Unix text editor Vi IMproved, a programmers text editor (standard version)

vim-tiny

V:16, I:89

828

Unix text editor Vi IMproved, a programmers text editor (compact version)

emacs21

V:4, I:9

8176

GNU project Emacs, the Lisp based extensible text editor (version 21)

emacs22

V:2, I:4

10992

GNU project Emacs, the Lisp based extensible text editor (version 22)

w3m

V:20, I:85

1968

Text-mode WWW browsers

gpm

V:3, I:5

564

The Unix style cut-and-paste on the text console (daemon)


It may be a good idea to read some informative documentations.

Table 2.2.  List of informative documentation packages.

package

popcon

size

description

doc-debian

I:84

376

Debian Project documentation, (Debian FAQ) and other documents

debian-policy

I:3

2732

Debian Policy Manual and related documents

developers-reference

I:1.4

1308

Guidelines and information for Debian developers

maint-guide

I:1.0

644

Debian New Maintainers' Guide

debian-history

I:0.4

2544

History of the Debian Project

debian-faq

I:20

1190

Debian FAQ

doc-linux-text

I:83

8616

Linux HOWTOs and FAQ (text)

doc-linux-html

I:1.2

62564

Linux HOWTOs and FAQ (html)

sysadmin-guide

I:0.4

964

The Linux System Administrators' Guide

rutebook

I:0.2

8264

Linux: Rute User's Tutorial and Exposition (non-free)


You can install some of these packages by issuing the following command from the root shell prompt:

# aptitude install package_name

2.1.11. An extra user account

If you do not want to use your main user account for the following training activities, you can create a training user account, e.g. fish. Type at root shell prompt:

# adduser fish
  • answer all the questions

This will create a new account named as fish. After your practice, you can remove this user account and its home directory by:

# deluser --remove-home fish

2.1.12. sudo configuration

For the typical single user workstation such as the desktop Debian system on the laptop PC, it is common to deploy simple configuration of sudo(8) as follows to let the non-privileged user, e.g. penguin, to gain administrative privilege just with his user password (not with the root password).

# echo "penguin  ALL=(ALL) ALL" >> /etc/sudoers

Since the sudo(8) command normally use normal user's environment variables to execute commands and normal user's "$PATH" variable is set to "/usr/local/bin:/usr/bin:/bin:/usr/games" by the system default, you should also adjust "$PATH" variable as:

$ echo 'PATH=$PATH:/usr/sbin:/sbin' >> ~/.bashrc

This trick should only be used for the single user workstation which you administer and where you are the only user.

[Warning] Warning

Do not set up accounts of regular users on multiuser workstation like this because it would be very bad for system security.

[Caution] Caution

The password and the account of the penguin in the above example requires as much protection as the root password and the root account.

[Caution] Caution

Administrative privilege in this context belongs to someone authorized to perform the system administration task on the workstation. Never give some manager in the Admin department of your company or your boss such privilege unless they are authorized and capable.

[Note] Note

For providing access privilege to limited devices and limited files, you should consider to use group to provide limited access instead of using the root privilege via sudo(8).

[Note] Note

With more thoughtful and careful configuration, the sudo facility may provide means to grant limited administrative privileges to other users on a shared system without sharing the root password. This can help with accountability with hosts with multiple administrators so you can tell who did what. On the other hand, you might not want anyone else to have such privileges.

2.1.13. Play time

Now you are ready to play with the Debian system without risks as long as you use the non-privileged user account.

This is because the Debian system is, even after the default installation, configured with the proper file permissions which prevent non-privileged users from damaging the system. Of course, there may still be some holes which can be exploited but those who worry about these issues should not be reading this section but should be reading Securing Debian Manual.

We will review basic Unix filesystem theory first. Then we will learn the Debian system with the easy way using Midnight Commander (MC) and with the proper Unix-like ways.

2.2. Unix-like filesystem

In GNU/Linux and other Unix-like operating systems, files are organized into directories. All files and directories are arranged in one big tree, the file hierarchy, rooted at /. It's called a tree because if you draw the file system, it looks like a tree (upside down).

These files and directories can be spread out over several devices. The mount(8) command serves to attach the file system found on some device to the big file tree. Conversely, the umount(8) command will detach it again. On recent Linux kernels, mount(8) operation can bind part of the file hierarchy somewhere else or can mount filesystem as shared, private, slave, or unbindable. Supported mount options for each filesystem are available in /share/doc/linux-doc-2.6.*/Documentation/filesystems/.

Directories on Unix systems are called folders on some other systems. Please also note that there is no concept for drive such as A: on any Unix system. There is one file system, and everything is included. This is a huge advantage compared to Windows, so consider yourself lucky.

2.2.1. Unix file basics

Here are Unix file basics:

  • Filenames are case sensitive. That is, MYFILE and MyFile are different files.

  • The root directory is referred to as simply /. Don't confuse this with the root user or the home directory for the root user: /root. Note that both entities are commonly referred to as "root". The context of the usage should make it clear which is meant.

  • Every directory has a name which can contain any letters or symbols except /. The root directory is an exception; its name is / (pronounced "slash" or "the root directory") and it cannot be renamed.

  • Each file or directory is designated by a fully-qualified filename, absolute filename, or path, giving the sequence of directories which must be passed through to reach it. The three terms are synonymous.

  • All fully-qualified filenames begin with the / directory, and there's a / between each directory or file in the filename. The first / is the top level directory, and the other /'s separate successive subdirectories, until we reach the last entry which is the name of the actual file. The words used here can be confusing. Take the following fully-qualified filename as an example: /usr/share/keytables/us.map.gz. However, people will also refer to its basename us.map.gz alone as a filename.

  • The root directory has a number of branches, such as /etc/ and /usr/. These subdirectories in turn branch into still more subdirectories, such as /etc/init.d/ and /usr/local/. The whole thing viewed collectively is called the directory tree. You can think of an absolute filename as a route from the base of the tree (/) to the end of some branch (a file). You will also hear people talk about the directory tree as if it were a family tree: thus subdirectories have parents, and a path shows the complete ancestry of a file. There are also relative paths that begin somewhere other than the root directory. You should remember that the directory ../ refers to the parent directory. This also applies to other directory like structures, such as data structures or other tree hierarchical organized entities.

  • There's no special directory path name component that corresponds to a physical device, such as your hard disk. This differs from RT-11, CP/M, OpenVMS, MS-DOS, AmigaOS, and Microsoft Windows, where the path contains a device name such as C:\. (However, directory entries do exist that refer to physical devices as a part of the normal filesystem. See Section 2.2.2, “Filesystem internals”.)

[Note] Note

While you can use almost any letters or symbols in a file name, in practice it is a bad idea to do so. It is better to avoid any characters that often have special meanings on the command line, including spaces, tabs, newlines, and other special characters: { } ( ) [ ] ' ` " \ / > < | ; ! # & ^ * % @ $ . If you want to separate words in a name, good choices are the period, hyphen, and underscore. You could also capitalize each word, "LikeThis". Experienced Linux users tend to avoid spaces in filenames.

[Note] Note

The word path is used not only for fully-qualified filename as above but also for the command search path. The intended meaning is usually clear from the context.

The detailed best practices for the file hierarchy are described in the Filesystem Hierarchy Standard (/usr/share/doc/debian-policy/fhs/fhs-2.3.txt.gz and hier(7)). You should remember the following facts as the starter:

Table 2.3.  List of usage of key directories.

directory

usage

/

A simple / represents the root directory.

/etc/

This is the place for the system wide configuration files.

/var/log/

This is the place for the system log files.

/home/

This is the directory which contains all the home directories for all non-privileged users.


2.2.2. Filesystem internals

Following the Unix tradition, the Debian GNU/Linux system provides the filesystem under which physical data on harddisks and other storage devices reside, and the interaction with the hardware devices such as console screens and remote serial consoles are represented in an unified manner [under /dev/].

Each file, directory, named pipe (a way two programs can share data), or physical device on a Debian GNU/Linux system has a data structure called an inode which describes its associated attributes such as the user who owns it (owner), the group that it belongs to, the time last accessed, etc. If you are really interested, see /usr/include/linux/fs.h for the exact definition of struct inode in the Debian GNU/Linux system. The idea of representing just about everything in the file system was a Unix innovation, and modern Linux kernels have developed this idea ever further. Now, even information about processes running in the computer can be found in the file system.

This abstract and unified representation of physical entities and internal processes is very powerful since this allows us to use the same command for the same kind of operation on many totally different devices. It is even possible to change the way the kernel works by writing data to special files that are linked to running processes.

All of your files could be on one disk --- or you could have 20 disks, some of them connected to a different computer elsewhere on the network. You can't tell just by looking at the directory tree, and nearly all commands work just the same way no matter what physical device(s) your files are really on. [This is a good thing. Trust us.] Of course, methods do exist whereby you can tell what devices map to which physical or network devices. mount with no arguments will show how storage is mapped to physical or network devices.

2.2.3. The filesystem permission system

The filesystem permissions of Unix-like system are defined for three categories of affected users:

  • the user who owns the file (u),

  • other users in the group which the file belongs to (g), and

  • all other users (o) also referred to as "world" and "everyone".

For the file, each corresponding permission allows:

  • read (r): to examine contents of the file,

  • write (w): to modify the file, and

  • execute (x): to run the file as a command.

For the directory, each corresponding permission allows:

  • read (r): to list contents of the directory,

  • write (w): to add or remove files in the directory, and

  • execute (x): to access files in the directory.

Here, execute permission on the directory means not only to allow reading of files in its directory but also to allow viewing their attributes, such as the size and the modification time.

To display permission information (and more) for files and directories, ls(1) is used. When ls invoked with the -l option, it displays the following information in the order given:

  • the type of file (first character)

  • the file's access permissions (the next nine characters, consisting of three characters each for user, group, and other in this order)

  • the number of hard links to the file

  • the name of the user who owns the file

  • the name of the group which the file belongs to

  • the size of the file in characters (bytes)

  • the date and time of the file (mtime)

  • the name of the file.

Table 2.4.  List of the first character of "ls -l" output

character

meaning

-

normal file

d

directory

l

symlink

c

character device node

b

block device node

p

named pipe

s

socket


To change the owner of the file, chown(1) is used from the root account. To change the group of the file, chgrp(1) is used from the file's owner or root account. To change file and directory access permissions, chmod(1) is used from the file's owner or root account. Basic syntax to manipulate foo file is:

# chown <newowner> foo
# chgrp <newgroup> foo
# chmod  [ugoa][+-=][rwxXst][,...] foo

For example, in order to make a directory tree to be owned by a user foo and shared by a group bar, issue the following commands from the root account:

# cd /some/location/
# chown -R foo:bar .
# chmod -R ug+rwX,o=rX .

There are three more special permission bits:

  • set user ID (s or S instead of user's x),

  • set group ID (s or S instead of group's x), and

  • sticky bit (t or T instead of other's x).

Here the output of ls -l for these bits is capitalized if execution bits hidden by these outputs are unset.

Setting set user ID on an executable file allows a user to execute the executable file with the owner ID of the file (for example root). Similarly, setting set group ID on an executable file allows a user to execute the executable file with the group ID of the file (for example root). Because these settings can cause security risks, enabling them requires extra caution.

Setting set group ID on a directory enables the BSD-like file creation scheme where all files created in the directory belong to the group of the directory.

Setting the sticky bit on a directory prevents a file in the directory from being removed by a user who is not the owner of the file. In order to secure the contents of a file in world-writable directories such as /tmp or in group-writable directories, one must not only set write permission off for the file but also set the sticky bit on the directory. Otherwise, the file can be removed and a new file can be created with the same name by any user who has write access to the directory.

Here are a few interesting examples of the file permissions.

$ ls -l /etc/passwd /etc/shadow /dev/ppp /usr/sbin/exim4
crw------- 1 root root   108, 0 2007-04-29 07:00 /dev/ppp
-rw-r--r-- 1 root root     1427 2007-04-16 00:19 /etc/passwd
-rw-r----- 1 root shadow    943 2007-04-16 00:19 /etc/shadow
-rwsr-xr-x 1 root root   700056 2007-04-22 05:29 /usr/sbin/exim4
$ ls -ld /tmp /var/tmp /usr/local /var/mail /usr/src
drwxrwxrwt 10 root root  4096 2007-04-29 07:59 /tmp
drwxrwsr-x 10 root staff 4096 2007-03-24 18:48 /usr/local
drwxrwsr-x  4 root src   4096 2007-04-27 00:31 /usr/src
drwxrwsr-x  2 root mail  4096 2007-03-28 23:33 /var/mail
drwxrwxrwt  2 root root  4096 2007-04-29 07:11 /var/tmp

There is an alternative numeric mode to describe file permissions in chmod(1) commands. This numeric mode uses 3 to 4 digit wide octal (radix=8) numbers.

Table 2.5.  The numeric mode for file permissions in chmod(1) commands.

digit

meaning

1st optional digit

sum of set user ID (=4), set group ID (=2), and sticky bit (=1)

2nd digit

sum of read (=4), write (=2), and execute (=1) permissions for user

3rd digit

ditto for group

4th digit

ditto for other


This sounds complicated but it is actually quite simple. If you look at the first few (2-10) columns from "ls -l" command output and read it as a binary (radix=2) representation of file permissions ("-" being "0" and "rwx" being "1"), the last 3 digit of the numeric mode value should make sense as an octal (radix=8) representation of file permissions to you. For example, try:

$ touch foo bar
$ chmod u=rw,go=r foo
$ chmod 644 bar
$ ls -l foo bar
-rw-r--r-- 1 penguin penguin 17 2007-04-29 08:22 bar
-rw-r--r-- 1 penguin penguin 12 2007-04-29 08:22 foo
[Tip] Tip

If you need to access information displayed by "ls -l" in shell script, you should use pertinent commands such as test(1), stat(1) and readlink(1). The shell builtin such as "[" or "test" may be used too.

2.2.4. Control of permissions for newly created files: umask

What permissions are applied to a newly created file or directory is restricted by the umask shell built-in command. See dash(1), bash(1), and builtins(7).

 (file permission) = (requested file permission) & ~(umask value)

Table 2.6.  The umask value examples.

umask

usage

file permission created

directory permission created

0022

writable only by the user

-rw-r--r--

-rwxr-xr-x

0002

writable by the group

-rw-rw-r--

-rwxrwxr-x


The Debian system uses a user private group (UPG) scheme as its default. A UPG is created whenever a new user is added to the system. A UPG has the same name as the user for which it was created and that user is the only member of the UPG. UPGs makes it is safe to set umask to 0002 since every user has their own private group. (In some Unix variants, it is quite common to setup all normal users belonging to a single users group and is good idea to set umask to 0022 for security in such cases.)

2.2.5. Permissions for groups of users (group)

In order to make the group permission to be applied to a particular user, that user needs to be made a member of the group using "sudo vigr".

[Note] Note

Alternatively, you may dynamically add users to groups during the authentication process by adding "auth optional pam_group.so" line to /etc/pam.d/common-auth and setting /etc/security/group.conf. (See more: Chapter 5, Authentication.)

The hardware devices are just another kind of file on the Debian system. If you have problems accessing devices such as CD-ROM and USB memory stick from a user account, you should make that user a member of the relevant group.

Some notable system-provided groups allow their members to access particular files and devices without root privilege.

Table 2.7.  List of example system-provided groups for file access.

group

accessible files and devices

dialout

Full and direct access to serial ports. (reconfigure modem, dial anywhere, etc.)

dip

"Dialup IP", enough to run ppp or dip commands.

cdrom

CD-ROM, DVD+/-RW drives.

audio

An audio device.

video

A video device.

scanner

Scanner(s).

adm

System monitoring logs.

staff

Some directories for junior administrative work: /usr/local, /home .


Some notable system provided groups allow their members to execute particular commands without root privilege.

Table 2.8.  List of notable system provided groups for particular command executions.

group

accessible commands

sudo

execute sudo without their password.

lpadmin

execute commands to add, modify, and remove printers from printer databases.

plugdev

execute pmount(1) for removable devices such as USB memories.


For the full listing of the system provided users and groups, see the recent version of the "Users and Groups" (/usr/share/doc/base-passwd/users-and-groups.html) document provided by the base-passwd package.

See manpages of passwd(5), group(5), shadow(5), group(5), vipw(8), vigr(8), and pam_group(8) for the management commands of the user and group system.

2.2.6. Timestamps

There are three types of timestamps for a GNU/Linux file.

Table 2.9.  List of types of timestamps.

type

meaning

mtime

the file modification time (ls -l)

ctime

the file status change time (ls -lc)

atime

the last file access time (ls -lu)


Note that ctime is not file creation time.

  • Overwriting a file will change all of the mtime, ctime, and atime attributes of the file.

  • Changing permission or owner of a file will change the ctime and atime attributes of the file.

  • Reading a file will change the atime of the file.

Note that even simply reading a file on the Debian system will normally cause a file write operation to update atime information in the inode. Mounting a filesystem with the noatime option will let the system skip this operation and will result in faster file access for the read. This is often recommended for laptops, because it reduces hard drive activity and saves power. See mount(8).

Use touch(1) command to change timestamps of existing files.

For timestamps, the ls command outputs different strings under the modern English locale (en_US.UTF-8) from under the old one (C).

$ LANG=en_US.UTF-8  ls -l foo
-rw-r--r-- 1 penguin penguin 3 2008-03-05 00:47 foo
$ LANG=C  ls -l foo
-rw-r--r-- 1 penguin penguin 3 Mar  5 00:47 foo

2.2.7. Links

There are two methods of associating a file foo with a different filename bar.

  • a hard link is a duplicate name for an existing file ("ln foo bar"),

  • a symbolic link, or "symlink", is a special file that points to another file by name ("ln -s foo bar").

See the following example for the changes in link counts and the subtle differences in the result of the rm command.

$ echo "Original Content" > foo
$ ls -li foo
2398521 -rw-r--r-- 1 penguin penguin 17 2007-04-29 08:15 foo
$ ln foo bar     # hard link
$ ln -s foo baz  # symlink
$ ls -li foo bar baz
2398521 -rw-r--r-- 2 penguin penguin 17 2007-04-29 08:15 bar
2398538 lrwxrwxrwx 1 penguin penguin  3 2007-04-29 08:16 baz -> foo
2398521 -rw-r--r-- 2 penguin penguin 17 2007-04-29 08:15 foo
$ rm foo
$ echo "New Content" > foo
$ ls -li foo bar baz
2398521 -rw-r--r-- 1 penguin penguin 17 2007-04-29 08:15 bar
2398538 lrwxrwxrwx 1 penguin penguin  3 2007-04-29 08:16 baz -> foo
2398540 -rw-r--r-- 1 penguin penguin 12 2007-04-29 08:17 foo
$ cat bar
Original Content
$ cat baz
New Content

The hardlink can be made within the same file system and shares the same inode number which the "-i" option with ls command reveals.

The symlink always has nominal file access permissions of "rwxrwxrwx", as shown in the above example, with the effective access permissions dictated by the permissions of the file that it points to.

[Caution] Caution

It is generally good idea not to create complicated symbolic links or hardlinks at all unless you have a very good reason. It may cause nightmares where the logical combination of the symbolic links results in loops in the filesystem.

[Note] Note

It is generally preferable to use symbolic links rather than hardlinks unless you have a good reason for using a hardlink.

The "." directory links to the directory that it appears in, thus the link count of any new directory starts at 2. The ".." directory links to the parent directory, thus the link count of the directory increases with the addition of new subdirectories.

If you are just moving to Linux from Windows, it will soon become clear how well-designed the file-name linking of Unix is, compared with the nearest Windows equivalent of "shortcuts". Because it is implemented in the file system, applications can't see any difference between a linked file and the original. In the case of hardlinks, there really is no difference.

2.2.8. Named pipes (FIFOs)

A named pipe is a file that acts like a pipe. You put something into the file, and it comes out the other end. Thus it's called a FIFO, or First-In-First-Out: the first thing you put in the pipe is the first thing to come out the other end.

If you write to a named pipe, the process which is writing to the pipe doesn't terminate until the information being written is read from the pipe. If you read from a named pipe, the reading process waits until there is something to read before terminating. The size of the pipe is always zero --- it does not store data, it just links two processes like the shell "|". However, since this pipe has a name, the two processes don't have to be on the same command line or even be run by the same user. Pipes were a very influential innovation of Unix.

You can try it by doing the following:

$ cd; mkfifo mypipe
$ echo "hello" >mypipe & # put into background
[1] 8022
$ ls -l mypipe
prw-r--r-- 1 penguin penguin 0 2007-04-29 08:25 mypipe
$ cat mypipe
hello
[1]+  Done                    echo "hello" >mypipe
$ ls mypipe
mypipe
$ rm mypipe

2.2.9. Sockets

Sockets are used extensively by all the Internet communication, databases, and the operating system itself. It is similar to the named pipe (FIFO) and allows processes to exchange information even between different computers. For the socket, those processes do not need to be running at the same time nor to be running as the children of the same ancestor process. This is the endpoint for the inter process communication (IPC). The exchange of information may occur over the network between different hosts. The two most common ones are the Internet socket and the Unix domain socket.

[Tip] Tip

"netstat -an" will provide a very useful overview of the sockets that are open on a given system.

2.2.10. Device files

Device files refer to physical or virtual devices on your system, such as your hard disk, video card, screen, or keyboard. An example of a virtual device is the console, represented by /dev/console.

Table 2.10.  The device types.

device type

meaning

character device

This can be accessed one character at a time, that is, the smallest unit of data which can be written to or read from the device is a character (byte).

block device

This must be accessed in larger units called blocks, which contain a number of characters. Your hard disk is a block device.


You can read and write device files, though the file may well contain binary data which may be an incomprehensible-to-humans gibberish. Writing data directly to these files is sometimes useful for the troubleshooting of hardware connections. For example, you can dump a text file to the printer device /dev/lp0 or send modem commands to the appropriate serial port /dev/ttyS0. But, unless this is done carefully, it may cause a major disaster. So be cautious.

[Note] Note

For the normal access to a printer, use the lp(1) command.

The device node number are displayed by executing ls as:

$ ls -l /dev/hda /dev/ttyS0 /dev/zero
brw-rw---- 1 root cdrom   3,  0 2007-04-29 07:00 /dev/hda
crw-rw---- 1 root dialout 4, 64 2007-04-29 07:00 /dev/ttyS0
crw-rw-rw- 1 root root    1,  5 2007-04-29 07:00 /dev/zero

Here,

  • /dev/hda has the major device number 3 and the minor device number 0. This is read/write accessible by the user who belongs to disk group,

  • /dev/ttyS0 has the major device number 4 and the minor device number 64. This is read/write accessible by the user who belongs to dialout group, and

  • /dev/zero has the major device number 1 and the minor device number 5. This is read/write accessible by anyone.

In the Linux 2.6 system, the filesystem under /dev is automatically populated by the udev(7) mechanism.

2.2.11. Special device files

There are some special device files.

Table 2.11.  List of special device files.

device file

action

response

/dev/null

read

it returns "end-of-file (EOF) character".

/dev/null

write

it is a bottomless data dump pit.

/dev/zero

read

it returns "the \0 (NUL) character" (not the same as the number zero ASCII).

/dev/random

read

it returns random characters from a true random number generator, delivering real entropy. (slow)

/dev/urandom

read

it returns random characters from a cryptographically secure pseudorandom number generator.

/dev/full

write

it returns the disk-full (ENOSPC) error.


These are frequently used in conjunction with the shell redirection (see Section 2.5.5, “Typical command sequences and shell redirection”).

2.2.12. procfs and sysfs

The procfs and sysfs mounted on /proc and /sys are the pseudo-filesystem and expose internal data structures of the kernel to the userspace. In other word, these entries are virtual, meaning that they act as a convenient window into the operation of the operating system.

The directory /proc contains (among other things) one subdirectory for each process running on the system, which is named after the process ID (PID).

System utilities that access process information, such as ps, get their information from this directory structure.

The directories under /proc/sys/ contain interface to change certain kernel parameters at run time. (You may do the same through specialized command sysctl(8) or its preload/configuration file /etc/sysctrl.conf.)

[Note] Note

The Linux kernel may complain "Too many open files". You can fix avoid this by executing "echo "65536" > /proc/sys/fs/file-max" from the root shell to increase file-max value.

People frequently panic when they notice one file in particular - /proc/kcore - which is generally huge. This is (more or less) a copy of the contents of your computer's memory. It's used to debug the kernel. It is a virtual file that points to computer memory, so don't worry about its size.

The directory under /sys contains exported kernel data structures, their attributes, and the linkages between them. It also contains interface to change certain kernel parameters at run time.

See proc.txt(.gz), sysfs.txt(.gz) and other related documents in the Linux kernel documentation (/usr/share/doc/linux-doc-2.6.*/Documentation/filesystems/*) provided by the linux-doc-2.6.* package.

2.3. Midnight Commander (MC)

Midnight Commander (MC) is a GNU "Swiss army knife" for the Linux console and other terminal environments. This gives newbie a menu driven console experience which is much easier to learn than standard Unix commands.

You may need to install the Midnight Commander package which is titled mc.

 sudo aptitude install mc                                                                                                   

Use the mc command to explore the Debian system. This is the best way to learn. Please explore few interesting locations just using the cursor keys and Enter key:

  • /etc and its subdirectories.

  • /var/log and its subdirectories.

  • /usr/share/doc and its subdirectories.

  • /sbin and /bin

2.3.1. Customization of MC

In order to make MC to change working directory upon exit and cd to the frequently used directories, I suggest to modify ~/.bashrc to include:

. /usr/share/mc/bin/mc.sh                                                                                                        

See mc(1) (under the "-P" option) for the reason. (If you do not understand what exactly I am talking here, you can do this later.)

2.3.2. Starting MC

MC can be started by:

$ mc

MC takes care of all file operations through its menu, requiring minimal user effort. Just press F1 to get the help screen (see next paragraph if this doesn't work). You can play with MC just by pressing cursor-keys and function-keys.

In some consoles such as gnome-terminal, key strokes of function-keys may be stolen by the console program. You can disable these features by "Edit" -> "Keyboard Shortcuts" for gnome-terminal. In particular, consider removing the mapping of F1 to Gnome Terminal Help; doing so will allow F1 to show Midnight Commander help.

If you encounter character encoding problem which displays garbage characters, adding "-a" to MC's command line may help prevent problems.

If this doesn't clear up your display problems with MC, see Section 10.6.6, “The terminal configuration”.

2.3.3. File manager in MC

The default is two directory panels containing file lists. Another useful mode is to set the right window to "information" to see file access privilege information, etc. Following are some essential keystrokes. With the gpm daemon running, one can use a mouse, too. (Make sure to press the shift-key to obtain the normal behavior of cut and paste in MC.)

Table 2.12.  The key bindings of MC.

key

key binding

F1

Help menu

F3

Internal file viewer

F4

Internal editor

F9

Activate pull down menu

F10

Exit Midnight Commander

Tab

Move between two windows

Insert or Ctrl-T

Mark file for a multiple-file operation such as copy

Del

Delete file (be careful---set MC to safe delete mode)

Cursor keys

Self-explanatory


2.3.4. Command-line tricks in MC

  • Any cd command will change the directory shown on the selected screen.

  • Ctrl-Enter or Alt-Enter will copy a filename to the command line. Use this with the cp or mv command together with command-line editing.

  • Alt-Tab will show shell filename expansion choices.

  • One can specify the starting directory for both windows as arguments to MC; for example, mc /etc /root.

  • Esc + n-key == Fn (i.e., Esc + 1 = F1, etc.; Esc + 0 = F10)

  • Pressing Esc before the key has the same effect as pressing the Alt and the key together.; i.e., type Esc + c for Alt-C. Esc is called meta-key and sometimes noted as "M-"

2.3.5. The internal editor in MC

The internal editor has an interesting cut-and-paste scheme. Pressing F3 marks the start of a selection, a second F3 marks the end of selection and highlights the selection. Then you can move your cursor. If you press F6, the selected area will be moved to the cursor location. If you press F5, the selected area will be copied and inserted at the cursor location. F2 will save the file. F10 will get you out. Most cursor keys work intuitively.

This editor can be directly started on a file:

$ mc -e filename_to_edit
$ mcedit filename_to_edit

This is not a multi-window editor, but one can use multiple Linux consoles to achieve the same effect. To copy between windows, use Alt-F<n> keys to switch virtual consoles and use "File->Insert file" or "File->Copy to file" to move a portion of a file to another file.

This internal editor can be replaced with any external editor of choice.

Also, many programs use the environment variables EDITOR or VISUAL to decide which editor or viewer to use. If you are uncomfortable with vim or nano initially, you may set these to mcedit by adding these lines to ~/.bashrc:

...
export EDITOR=mcedit
export VISUAL=mcedit
...

I do recommend setting these to vim if possible.

If you are uncomfortable with vim, you can keep using mcedit for most system maintenance tasks.

2.3.6. The internal viewer in MC

Very smart viewer. This is a great tool for searching words in documents. I always use this for files in the /usr/share/doc directory. This is the fastest way to browse through masses of Linux information. This viewer can be directly started like so:

$ mc -v path/to/filename_to_view
$ mcview path/to/filename_to_view

2.3.7. Auto-start features of MC

Press Enter on a file, and the appropriate program will handle the content of the file (see Section 10.5.11, “Customizing program to be started”). This is a very convenient MC feature.

Table 2.13.  The reaction to the enter key in MC.

file type

reaction to enter key

executable file

Execute command

man file

Pipe content to viewer software

html file

Pipe content to web browser

tar.gz.deb file

Browse its contents as if subdirectory


In order to allow these viewer and virtual file features to function, viewable files should not be set as executable. Change their status using the chmod command or via the MC file menu.

2.3.8. FTP virtual filesystem of MC

MC can be used to access files over the Internet using FTP. Go to the menu by pressing F9, then type "p" to activate the FTP virtual filesystem. Enter a URL in the form "username:passwd@hostname.domainname", which will retrieve a remote directory that appears like a local one.

Try "http.us.debian.org/debian" as the URL and browse the Debian archive.

2.4. The basic Unix-like work environment

Although MC enables you to do almost everything, it is very important for you to learn how to use the command line tools invoked from the shell prompt and become familiar with the Unix-like work environment.

2.4.1. The login shell

You can select your login shell with the chsh command.

Table 2.14.  List of shell programs.

package

popcon

size

POSIX shell

description

bash

V:87, I:99

1300

Yes

The GNU Bourne Again SHell. (de facto standard)

tcsh

V:11, I:83

736

No

TENEX C Shell, an enhanced version of Berkeley csh.

dash

V:3, I:13

236

Yes

The Debian Almquist Shell. Good for shell script.

zsh

V:2, I:5

12504

Yes

The standard shell with many enhancements.

pdksh

V:0.3, I:1.3

464

Yes

A public domain version of the Korn shell.

csh

V:0.4, I:1.5

400

No

OpenBSD C Shell, a version of Berkeley csh.

sash

V:0.3, I:1.1

836

Yes

Stand-alone shell with built-in commands. (Not meant for standard /bin/sh.)

ksh

V:0.3, I:1.5

2540

Yes

The real, AT&T version of the Korn shell.

rc

V:0.08, I:0.5

204

No

An implementation of the AT&T Plan 9 shell.

posh

V:0.02, I:0.14

232

Yes

Policy-compliant Ordinary SHell. A pdksh derivative.

es

V:0.02, I:0.06

304

No

An extensible shell based on the AT&T Plan 9 shell: rc.


In this tutorial chapter, the interactive shell always means bash.

2.4.2. Customizing bash

You can customize bash behavior by ~/.bashrc. For example, I added followings to ~/.bashrc:

# CD upon exiting MC
. /usr/share/mc/bin/mc.sh

# set CDPATH to good one
CDPATH=.:/usr/share/doc:~/Desktop/src:~/Desktop:~
export CDPATH

PATH="${PATH}":/usr/sbin:/sbin
# set PATH so it includes user's private bin if it exists
if [ -d ~/bin ] ; then
  PATH=~/bin:"${PATH}"
fi
export PATH

EDITOR=vim
export EDITOR
[Tip] Tip

You can find more bash customization tips, such as Section 10.2.6, “Colorized commands”, in Chapter 10, System tips.

2.4.3. Special key strokes

In the Unix-like environment, there are few key strokes which have special meanings. Please note that on a normal Linux character console, only the left-hand Ctrl and Alt keys work as expected. Here are few notable key strokes to remember.

Table 2.15.  List of key bindings for bash.

key

key binding

Ctrl-U

Erase line before cursor.

Ctrl-H

Erase a character before cursor.

Ctrl-D

Terminate input. (exit shell if you are using shell)

Ctrl-C

Terminate a running program.

Ctrl-Z

Temporarily stop program by moving it to the background job

Ctrl-S

Halt output to screen.

Ctrl-Q

Reactivate output to screen.

Ctrl-Alt-Del

Reboot/halt the system, see manpage for inittab.

Left-Alt-key (optionally, Windows-key)

Meta-key for Emacs and the similar UI.

Up-arrow

Start command history search under bash.

Ctrl-R

Start incremental command history search under bash.

Tab

Complete input of the filename to the command line under bash.

Ctrl-V Tab

Input Tab without expansion to the command line under bash.


[Tip] Tip

The terminal feature of Ctrl-S can be disabled using stty(1) command.

2.4.4. Unix style mouse operations

The Unix style mouse operations are based on the 3 button mouse system.

Table 2.16.  The Unix style mouse operations.

action

response

Left-click-and-drag mouse

Select and copy to the clipboard.

Left-click

Select the start of selection.

Right-click

Select the end of selection and copy to the clipboard.

Middle-click

Paste clipboard at the cursor.


The center wheel on the modern wheel mouse is considered middle mouse button and can be used for middle-click. Clicking left and right mouse buttons together serves as the middle-click under the 2 button mouse system situation. In order to use a mouse in the Linux character console, you need to have gpm running as daemon.

2.4.5. The pager

The less program is the enhanced pager (file content browser). Hit "h" for help. It can do much more than more. This less command can be supercharged by executing eval $(lesspipe) or eval $(lessfile) in the shell startup script. See more in /usr/share/doc/lessf/LESSOPEN. The -R option allows raw character output and enables ANSI color escape sequences. See less(1).

2.4.6. The text editor

You should become proficient in one of the variants of Vim or Emacs programs which are popular in the Unix-like system.

I think getting used to Vim commands is the right thing to do, since Vi-editor is always there in the Linux/Unix world. (Actually, original vi or new nvi are the programs you find everywhere. I chose Vim instead for newbie since it offers you help through F1 key while it is similar enough and more powerful.)

If you chose either Emacs or XEmacs instead as your choice of the editor, that is another good choice indeed, particularly for programming. Emacs has a plethora of other features as well, including functioning as a newsreader, directory editor, mail program, etc.. When used for programming or editing shell scripts, it intelligently recognizes the format of what you are working on, and tries to provide assistance. Some people maintain that the only program they need on Linux is Emacs. Ten minutes learning Emacs now can save hours later. Having the gnu Emacs manual for reference when learning Emacs is highly recommended.

All these programs usually come with tutoring program for you to learn them by practice. Start Vim by typing "vim" and press F1-key. You should at least read the first 35 lines. Then do the online training course by moving cursor to |tutor| and pressing Ctrl-].

[Note] Note

Good editors, such as Vim and Emacs, can be used to handle UTF-8 and other exotic encoding texts correctly with proper option in the x-terminal-emulator on X under UTF-8 locale with proper font settings. Please refer to their documentation on multibyte text.

2.4.7. Setting a default text editor

Debian comes with a number of different editors. We recommend to install the vim package, as mentioned above.

Debian provides unified access to the system default editor via command /usr/bin/editor so other programs (e.g., reportbug(1)) can invoke it. You can change it by:

$ sudo update-alternatives --config editor

The choice /usr/bin/vim.basic is the recommendation for newbies by the author. This supports syntax highlighting.

[Tip] Tip

Many programs use the environment variables "EDITOR" or "VISUAL" to decide which editor to use (see Section 2.3.5, “The internal editor in MC” and Section 10.5.11, “Customizing program to be started”).

2.4.8. Customizing vim

You can customize vim behavior by ~/.vimrc. For example, I use:

" -------------------------------
" Local configuration
"
set nocompatible
set nopaste
set pastetoggle=<f2>
syn on
if $USER == "root"
 set nomodeline
 set noswapfile
else
 set modeline
 set swapfile
endif
" filler to avoid the line above being recognized as a modeline
" filler
" filler

2.4.9. Recording the shell activities

The output of the shell command may roll off your screen and may be lost forever. It is good practice to log shell activities into the file for you to review them later. This kind of record is essential when you perform any system administration tasks.

The basic method of recording the shell activity is to run it under the script(1) command.

$ script
Script started, file is typescript
  • do whatever shell commands ...

  • press Ctrl-D to exit script.

$ vim typescript

See Section 10.2.3, “Recording the shell activities cleanly” .

2.4.10. Basic Unix commands

Let's learn the basic Unix commands. Here I use "Unix" in its generic sense. Any Unix clone OSs usually offer the equivalent commands. The Debian system is no exception. Do not worry if some commands do not work as you wish now. If alias is used in the shell, its corresponding command outputs are different. These examples are not meant to be executed in this order.

Try all the following commands from the non-privileged user account:

Table 2.17.  List of basic Unix commands.

command

description

pwd

Display name of current/working directory.

whoami

Display current user name.

id

Display current user identity (name, uid, gid, and associated groups).

file <foo>

Display a type of file for the file <foo>.

type -p <commandname>

Display a file location of command <commandname>.

which <commandname>

, ,

type <commandname>

Display information on command <commandname>.

apropos <key-word>

Find commands related to <key-word>.

man -k <key-word>

, ,

whatis <commandname>

Display one line explanation on command <commandname>.

man -a <commandname>

Display explanation on command <commandname>. (Unix style)

info <commandname>

Display rather long explanation on command <commandname>. (GNU style)

ls

List contents of directory. (non-dot files and directories)

ls -a

List contents of directory. (all files and directories)

ls -A

List contents of directory. (almost all files and directories, i.e., skip ".." and ".")

ls -la

List all contents of directory with detail information.

ls -lai

List all contents of directory with inode number and detail information.

ls -d

List all directories under the current directory.

tree

Display file tree contents.

lsof <foo>

List open status of file <foo>.

lsof -p <pid>

List files opened by PID=<pid>.

mkdir <foo>

Make a new directory <foo> in the current directory.

rmdir <foo>

Remove a directory <foo> in the current directory.

cd <foo>

Change directory to the directory <foo> in the current directory or in the directory listed in the variable CDPATH.

cd /

Change directory to the root directory.

cd

Change directory to the current user's home directory.

cd /<foo>

Change directory to the absolute path directory /<foo>.

cd ..

Change directory to the parent directory.

cd ~<foo>

Change directory to the home directory of the user <foo>.

cd -

Change directory to the previous directory.

</etc/motd pager

Display contents of /etc/motd using the default pager.

touch <junkfile>

Create a empty file <junkfile>.

cp <foo> <bar>

Copy a existing file <foo> to a new file <bar>.

rm <junkfile>

Remove a file <junkfile>.

mv <foo> <bar>

Rename an existing file <foo> to a new name <bar>. The directory <bar> must not exist.

mv <foo> <bar>

Move an existing file <foo> to a new location <bar>/<foo>. The directory <bar> must exist.

mv <foo> <bar>/<baz>

Move an existing file <foo> to a new location with a new name <bar>/<baz>. The directory <bar> must exist but the directory <bar>/<baz> must not exist.

chmod 600 <foo>

Make an existing file <foo> to be non-readable and non-writable by the other people. (non-executable for all)

chmod 644 <foo>

Make an existing file <foo> to be readable but non-writable by the other people. (non-executable for all)

chmod 755 <foo>

Make an existing file <foo> to be readable but non-writable by the other people. (executable for all)

find . -name <pattern>

find matching filenames using shell <pattern>. (slower)

locate -d . <pattern>

find matching filenames using shell <pattern>. (quicker using regularly generated database)

grep -e "<pattern>" *.html

Find a "<pattern>" in all of the files ending with ".html" in current directory and display them all.

top

Display process information using full screen. Type "q" to quit.

ps aux | pager

Display information on all the running processes using BSD style output.

ps -ef | pager

Display information on all the running processes using Unix system-V style output.

ps aux | grep -e "[e]xim4*"

Display all processes running exim or exim4.

ps axf | pager

Display information on all the running processes with ASCII art output.

kill <1234>

Kill a process identified by the process ID: <1234>.

gzip <foo>

Compress <foo> to create <foo>.gz using the Lempel-Ziv coding (LZ77).

gunzip <foo>.gz

Decompress <foo>.gz to create <foo>.

bzip2 <foo>

Compress <foo> to create <foo>.bz2 using the Burrows-Wheeler block sorting text compression algorithm, and Huffman coding. (Better compression than gzip)

bunzip2 <foo>.bz2

Decompress <foo>.bz2 to create <foo>.

tar -xvf <foo.tar>

Extract files from <foo>.tar archive.

tar -xvzf <foo>.tar.gz

Extract files from gzipped <foo>.tar.gz archive.

tar -xvf -j <foo.tar.bz2>

Extract files from <foo>.tar.bz2 archive.

tar -cvf <foo>.tar <bar>/

Archive contents of folder <bar>/ in <foo>.tar archive.

tar -cvzf <foo>.tar.gz <bar>/

Archive contents of folder <bar>/ in compressed <foo>.tar.gz archive.

tar -cvjf <foo>.tar.bz2 <bar>/

Archive contents of folder <bar>/ in <foo>.tar.bz2 archive.

zcat README.gz | pager

Display contents of compressed README.gz using the default pager.

zcat README.gz > foo

Create a file foo with the decompressed content of README.gz.

zcat README.gz >> foo

Append the decompressed content of README.gz to the end of the file foo. (If it does not exist, create it first.)


[Note] Note

Unix has a tradition to hide filenames which start with ".". They are traditionally files that contain configuration information and user preferences.

[Note] Note

For cd command, see manpage of builtins(7).

[Note] Note

The default pager of the bare bone Debian system is more which cannot scroll back. By installing less package using command line "aptitude install less", less becomes default pager and you can scroll back with cursor keys.

[Note] Note

The "[" and "]" in the regular expression of the "ps aux | grep -e "[e]xim4*"" command above enable grep to avoid matching itself. The "4*" in the regular expression means 0 or more repeats of character "4" thus enables grep to match both "exim" and "exim4". Although "*" is used in the shell filename glob and the regular expression, their meanings are different in the regular expression. Learn the regular expression from the manpage of grep(1).

Please traverse directories and peek into the system using the above commands as training. If you have questions on any of the console commands, please make sure to read the manual page. For example, these commands are the good start:

$ man man
$ man bash
$ man builtins
$ man grep
$ man ls

The style of man pages may be a little hard to get used to, because they are rather terse, particularly the older, very traditional ones. But once you get used to it, you come to appreciate their succinctness.

Please note that many Unix-like commands including ones from GNU and BSD will display brief help information if you invoke them in one of the following ways (or without any arguments in some cases):

$ <commandname> --help
$ <commandname> -h

2.5. The simple shell command

Now you have some feel on how to use the Debian system. Let's look deep into the mechanism of the command execution in the Debian system. Here, I have simplified reality for the newbie. See manpages for bash(1) for the exact explanation.

A simple command is a sequence of

  1. variable assignments (optional)

  2. command name

  3. arguments (optional)

  4. redirections (optional: > , >> , < , << , etc.)

  5. control operator (optional: && , || ; <newline> , ; , & , ( , ) )

2.5.1. Command execution and environment variable

The values of some environment variables change the behavior of some Unix commands.

The default values of environment variables are initially set by the PAM system and then some of them may be reset by some application programs:

  • the display manager such as gdm, and

  • the shell in its start up codes bash_profile and .bashrc.

2.5.1.1. LANG variable

The full locale value given to LANG variable consists of 3 parts: xx_YY.ZZZZ.


For the language codes and country codes, see pertinent description in the info gettext.

For the codeset on the modern Debian system, you should always set it to UTF-8 unless you specifically want to use the historic one with good reason and background knowledge.

For fine details of the locale configuration, see: Section 9.3, “The locale” .

[Note] Note

The "LANG=en_US" is not "LANG=C" nor "LANG=en_US.UTF-8". It is "LANG=en_US.ISO-8859-1" (see: Section 9.3.1, “Basics of encoding”).

Table 2.19.  List of locale recommendations.

Language (area)

locale recommendation

English(USA)

en_US.UTF-8

English(Great_Britain)

en_GB.UTF-8

French(France)

fr_FR.UTF-8

German(Germany)

de_DE.UTF-8

Italian(Italy)

it_IT.UTF-8

Spanish(Spain)

es_ES.UTF-8

Catalan(Spain)

ca_ES.UTF-8

Swedish(Sweden)

sv_SE.UTF-8

Portuguese(Brasil)

pt_BR.UTF-8

Russian(Russia)

ru_RU.UTF-8

Chinese(P.R._of_China)

zh_CN.UTF-8

Chinese(Taiwan_R.O.C.)

zh_TW.UTF-8

Japanese(Japan)

ja_JP.UTF-8

Korean(Republic_of_Korea)

ko_KR.UTF-8

Vietnamese(Vietnam)

vi_VN.UTF-8


Typical command execution uses a shell line sequence like the following:

$ date
Sun Jun  3 10:27:39 JST 2007
$ LANG=fr_FR.UTF-8 date
dimanche 3 juin 2007, 10:27:33 (UTC+0900)

Here, the program date is executed in the foreground job. The environment variable "LANG" is:

  • system default locale (such as "en_US.UTF-8" depending on your configuration) for the first command

  • set to "fr_FR.UTF-8" (French UTF-8 locale assuming it is available on your system) for the second command

Most command executions usually do not have preceding environment variable definition. For the above example, you can alternatively execute:

$ LANG=fr_FR.UTF-8
$ date
dimanche 3 juin 2007, 10:27:33 (UTC+0900)

As you can see here, the output of command is affected by the environment variable to produce French output. If you want the environment variable to be inherited to the subprocesses (e.g., when calling shell script), you need to "export" it instead by using:

$ export LANG
[Tip] Tip

When filing a bug report, running the command under "LANG=en_US.UTF-8" is good idea if you use non-English environment.

See locale(5) and locale(7) for LANG and related environment variables.

[Note] Note

I recommend you to configure the system environment just by the LANG variable and to stay away from LC_* variables unless it is absolutely needed.

2.5.1.2. PATH variable

When you type a command into the shell, the shell searches the command in the list of directories contained in the PATH environment variable. The value of the PATH environment variable is also called the shell's search path.

In the default Debian installation, the PATH environment variable of user accounts may not include /sbin/. For example, The ifconfig command needs to be issued with full path as /sbin/ifconfig.

You can change the PATH environment variable by ~/.bash_profile or ~/.bashrc files.

2.5.1.3. HOME variable

Many commands stores user specific configuration in the home directory and changes their behavior by their contents. The home directory is identified by the environment variable: HOME:

Table 2.20.  List of HOME values.

situation

value of HOME

program run by the init process (daemon)

/

program run from the normal root shell

/root

program run from the normal user shell

/home/<normal_user>

program run from the normal user GUI desktop menu

/home/<normal_user>

program run as root with "sudo program"

/home/<normal_user>

program run as root with "sudo -H program"

/root


2.5.2. Command line options

Some commands take arguments. Arguments starting with "-" or "--" are called options and control the behavior of the command.

$ date
Mon Oct 27 23:02:09 CET 2003
$ date -R
Mon, 27 Oct 2003 23:02:40 +0100

Here the command-line argument "-R" changes the date command behavior to output RFC-2822 compliant date string.

2.5.3. Shell glob

Often you want a command to work with a group of files without typing all of them. The filename expansion pattern using the shell glob, (sometimes referred as wildcards), facilitate this need.

Table 2.21.  The shell glob patterns.

shell glob pattern

match

*

This matches filename (segment) not started with ".".

.*

This matches filename (segment) started with ".".

?

This matches exactly one character.

[...]

This matches exactly one character with any character enclosed in brackets.

[a-z]

This matches exactly one character with any character between "a" and "z".

[^...]

This matches exactly one character other than any character enclosed in brackets (excluding "^").


For example, try the following and think for yourself:

$ mkdir junk; cd junk; .[^.]*touch 1.txt 2.txt 3.c 4.h .5.txt ..6.txt
$ echo *.txt
1.txt 2.txt
$ echo *
1.txt 2.txt 3.c 4.h
$ echo *.[hc]
3.c 4.h
$ echo .*
. .. .5.txt ..6.txt
$ echo .*[^.]*
.5.txt ..6.txt
$ echo [^1-3]*
4.h
$ cd ..; rm -rf junk

See "man 7 glob" for more.

[Note] Note

Unlike normal filename expansion by the shell, the shell pattern "*" tested in the find command with -name test etc., matches the initial "." of the filename. (New POSIX feature)

[Note] Note

BASH can be tweaked to change its glob behavior with its shopt builtin options such as dotglob, noglob, nocaseglob, nullglob, nocaseglob, extglob, etc. See bash(1).

2.5.4. Return value of the command

Each command returns its exit status as the return value.

Table 2.22.  Command exit code.

command exit state

numeric return value

logical return value

command executed successfully.

$? = 0

TRUE

command exited with error.

$? != 0

FALSE


Thus:

$ [ 1 = 1 ] ; echo $?
0
$ [ 1 = 2 ] ; echo $?
1
[Note] Note

Please note that, in the logical context for the shell, success is treated as the logical TRUE which has 0 (zero) as its value. This is somewhat non-intuitive and needs to be reminded here.

2.5.5. Typical command sequences and shell redirection

Let's try to remember following shell command idioms.

Table 2.23.  The shell command idioms.

command idiom (type in one line)

effects

command &

The command is executed in the subshell in the background.

command1 | command2

The standard output of command1 is piped to the standard input of command2 . Both commands may be running concurrently.

command1 2>&1 | command2

Both standard output and standard error of command1 are piped to the standard input of command2. Both commands may be running concurrently.

command1 ; command2

The command1 and command2 are executed sequentially.

command1 && command2

The command1 is executed. If successful, command2 is also executed sequentially. Return success if both command1 and command2 are successful.

command1 || command2

The command1 is executed. If not successful, command2 is also executed sequentially. Return success if command1 or command2 are successful.

command > foo

Redirect standard output of command to a file foo. (overwrite)

command 2> foo

Redirect standard error of command to a file foo. (overwrite)

command >> foo

Redirect standard output of command to a file foo. (append)

command 2>> foo

Redirect standard error of command to a file foo. (append)

command > foo 2>&1

Redirect both standard output and standard error of command to a file foo.

command < foo

Redirect standard input of command to a file foo.

command << delimiter

Redirect standard input of command to the following lines until delimiter is met. (Here documents)

command <<- delimiter

Redirect standard input of command to the following lines until delimiter is met. The leading tab characters are stripped from input lines. (Here documents)


The Debian system is a multi-tasking system. Background jobs allow users to run multiple programs in a single shell. The management of the background process involves the shell built-ins: jobs, fg, bg, and kill. Please read the sections of the bash(1) manpage under "SIGNALS", and "JOB CONTROL", and the builtins(1) manpage.

Let's try simple examples of redirection:

$ </etc/motd pager

$ pager </etc/motd

$ pager /etc/motd

$ cat /etc/motd | pager

Although all 4 examples display the same thing, the last example runs an extra cat command and wastes resources with no reason.

The shell allows you to open files using the exec built-in with an arbitrary file descriptor.

$ echo Hello >foo
$ exec 3<foo 4>bar  # open files
$ cat <&3 >&4       # redirect stdin to 3, stdout to 4
$ exec 3<&- 4>&-    # close files
$ cat bar
Hello

Here, "n<&-" and "n>&-" mean to close the file descriptor "n".

The file descriptor 0-2 are predefined:

Table 2.24.  The predefined file descriptors.

device

description

file descriptor

stdin

standard input

0

stdout

standard output

1

stderr

standard error

2


2.5.6. Command alias

You can set an alias for the frequently used command. For example:

$ alias la='ls -la'

Now, la works as a short hand for "ls -la" which lists all files in the long listing format.

You can list any existing aliases:

$ alias

You can identity exact path or identity of the command using type command. For example:

$ type ls
ls is hashed (/bin/ls)
$ type la
la is aliased to ls -la
$ type echo
echo is a shell builtin
$ type file
file is /usr/bin/file

Here ls was recently searched while file was not, thus ls is "hashed", i.e., the shell has an internal record for the quick access to the location of the ls command.

2.6. Unix-like text processing

In Unix-like work environment, text processing is done by piping text through chains of standard text processing tools. This was another crucial Unix innovation.

2.6.1. Unix text tools

There are few standard text processing tools which are used very often on the Unix-like system.

  • No regular expression is used:
    • cat(1) concatenates files and outputs the whole content.

    • tac(1) concatenates files and outputs in reverse.

    • cut(1) selects parts of lines and outputs.

    • head(1) outputs the first part of files.

    • tail(1) outputs the last part of files.

    • sort(1) sorts lines of text files.

    • uniq(1) removes duplicate lines from a sorted file.

    • tr(1) translates or deletes characters.

    • diff(1) compares files line by line.

  • Basic regular expression (BRE) is used:

    • grep(1) matches text with the pattern.

    • ed(1) is a primitive line editor.

    • sed(1) is a stream editor.

    • vim(1) is a screen editor.

    • emacs(1) is a screen editor. (somewhat extended BRE)

  • Extended regular expression (ERE) is used:

    • egrep(1) matches text with pattern.

    • awk(1) does simple text processing.

    • tcl does every conceivable type of text processing: re_syntax(3). Often used with tk.

    • perl(1) does text processing, is capable of network programming, and much more. perlre(1).

    • pcregrep(1) from the pcregrep package matches text with Perl Compatible Regular Expressions (PCRE) pattern.

    • python with re module does every conceivable text processing. See /usr/share/doc/python/html/index.html.

If you are not sure what exactly these commands do, please use "man command" to figure it out by yourself.

[Note] Note

Sort order and range expression are locale dependent. If you wish to obtain traditional behavior for a command, use C locale instead of UTF-8 ones by prepnding command with "LANG=C" (see Section 2.5.1.1, “LANG variable” and Section 9.3, “The locale”).

[Note] Note

Perl regular expressions (perlre(1)), Perl Compatible Regular Expressions (PCRE), and Python regular expressions offered by re module have many common extensions to the normal ERE.

2.6.2. Regular expressions

Regular expressions are used in many text processing tools. They are analogous to the shell globs, but they are both more complicated and more powerful.

The regular expression describes the matching pattern and is made up of text characters and metacharacters.

The metacharacter is just a character with a special meaning. There are 2 major styles, BRE and ERE, depending on the text tools as described above.

Table 2.25.  The metacharacters for BRE and ERE.

BRE

ERE

The meaning of the regular expression

\ . [ ] ^ $ *

\ . [ ] ^ $ *

common metacharacters

\+ \? \( \) \{ \} \|

BRE only "\" quoted metacharacters

+ ? ( ) { } |

ERE only non-"\" quoted metacharacters

c

c

This matches the non-metacharacter "c".

\c

\c

This sequence matches the literal character "c" even if "c" is metacharacter by itself.

.

.

This matches any character including newline.

^

^

This matches the beginning of a string.

$

$

This matches the end of a string.

\<

\<

This matches the beginning of a word.

\>

\>

This matches the end of a word.

\[abc...\]

[abc...]

This character list matches any of the characters "abc...".

\[^abc...\]

[^abc...]

This negated character list matches any of the characters except "abc...".

r*

r*

This matches zero or more regular expressions identified by "r".

r\+

r+

This matches one or more regular expressions identified by "r".

r\?

r?

This matches zero or one regular expressions identified by "r".

r1\|r2

r1|r2

This matches one of the regular expressions identified by "r1" or "r2".

\(r1\|r2\)

(r1|r2)

This matches one of the regular expressions identified by "r1" or "r2" and treats it as a bracketed regular expression.


The regular expression of emacs is basically BRE but has been extended to treat "+"and "?" as the metacharacters as in ERE. Thus, there are no needs to quote them with "\" in the regular expression of emacs.

For example, grep can be used to perform the text search using the regular expression:

$ egrep 'GNU.*LICENSE|Yoyodyne' /usr/share/common-licenses/GPL
GNU GENERAL PUBLIC LICENSE
GNU GENERAL PUBLIC LICENSE
Yoyodyne, Inc., hereby disclaims all copyright interest in the program

2.6.3. Replacement expressions

For the replacement expression, following characters have special meanings:

Table 2.26.  The replacement expression.

character

meaning

&

This represents what the regular expression matched. (use \& in emacs)

\n

This represents what the n-th _bracketed_ regular expression matched. ("n" being number)


For Perl replacement string, $n is used instead of \n and & has no special meaning.

For example:

$ echo zzz1abc2efg3hij4 | \
sed -e 's/\(1[a-z]*\)[0-9]*\(.*\)$/=&=/'
zzz=1abc2efg3hij4=
$ echo zzz1abc2efg3hij4 | \
sed -e 's/\(1[a-z]*\)[0-9]*\(.*\)$/\2===\1/'
zzzefg3hij4===1abc
$ echo zzz1abc2efg3hij4 | \
perl -pe 's/(1[a-z]*)[0-9]*(.*)$/$2===$1/'
zzzefg3hij4===1abc
$ echo zzz1abc2efg3hij4 | \
perl -pe 's/(1[a-z]*)[0-9]*(.*)$/=&=/'
zzz=&=

Here please pay extra attention to the style of the bracketed regular expression and how the matched strings are used in the text replacement process on different tools.

These regular expressions can be used for the cursor movements and the text replacement actions in the editors too.

The back slash "\" at the end of line in the shell commandline escapes newline as a white space character and continues shell command line input to the next line.

Please read all the related manual pages to learn these commands.

2.6.4. Extract data from text file table

Let's consider a text file called DPL in which some pre-2004 Debian project leader's names and their initiation days are listed in a space-separated format.

Ian     Murdock   August  1993
Bruce   Perens    April   1996
Ian     Jackson   January 1998
Wichert Akkerman  January 1999
Ben     Collins   April   2001
Bdale   Garbee    April   2002
Martin  Michlmayr March   2003

Awk is frequently used to extract data from these types of files.

$ awk '{ print $3 }' <DPL                   # month started
August
April
January
January
April
April
March
$ awk '($1=="Ian") { print }' <DPL          # DPL called Ian
Ian     Murdock   August  1993
Ian     Jackson   January 1998
$ awk '($2=="Perens") { print $3,$4 }' <DPL # When Perens started
April 1996

Shells such as Bash can be also used to parse this kind of file:

$ while read first last month year; do
    echo $month
  done <DPL
  • same output as the first Awk example.

Here, read built-in command uses the characters in $IFS (internal field separators) to split lines into words.

If you change IFS to ":", you can parse /etc/passwd with shell nicely:

$ oldIFS="$IFS"   # save old value
$ IFS=":"
$ while read user password uid gid rest_of_line; do
    if [ "$user" = "osamu" ]; then
      echo "$user's ID is $uid"
    fi
  done < /etc/passwd
osamu's ID is 1000
$ IFS="$oldIFS"   # restore old value

(If Awk is used to do the equivalent, use "FS=":"" to set the field separator.)

IFS is also used by the shell to split results of parameter expansion, command substitution, and arithmetic expansion. These do not occur within double or single quoted words. The default value of IFS is <space>, <tab>, and <newline> combined.

Be careful about using this shell IFS tricks. Strange things may happen, when shell interprets some parts of the script as its input.

$ IFS=":,"                        # use ":" and "," as IFS
$ echo IFS=$IFS,   IFS="$IFS"     # echo is a Bash built-in
IFS=  , IFS=:,
$ date -R                         # just a command output
Sat, 23 Aug 2003 08:30:15 +0200
$ echo $(date -R)                 # sub shell --> input to main shell
Sat  23 Aug 2003 08 30 36 +0200
$ unset IFS                       # reset IFS to the default
$ echo $(date -R)
Sat, 23 Aug 2003 08:30:50 +0200

2.6.5. Script snippets for piping commands

The following scripts will do nice things as a part of a pipe.

Table 2.27.  The script snippets for piping commands.

script snippet (type in one line)

effect

find /usr -print

find all files under /usr (see Section 11.1.5, “Idioms for the selection of files”)

seq 1 100

print 1 to 100

| xargs -n 1 <command>

run command repeatedly with each item from pipe as its argument (see Section 10.5.9, “Repeating a command looping over files”)

| xargs -n 1 echo

split white-space-separated items from pipe into lines

| xargs echo

merge all lines from pipe into a line

| grep -e <regex_pattern>

extract lines from pipe containing <regex_pattern>

| grep -v -e <regex_pattern>

extract lines from pipe not containing <regex_pattern>

| cut -d: -f3 -

extract third field from pipe separated by ":" (passwd file etc.)

| awk '{ print $3 }'

extract third field from pipe separated by whitespaces

| awk -F'\t' '{ print $3 }'

extract third field from pipe separated by tab

| col -bx

remove backspace and expand tabs to spaces

| expand -

expand tabs

| sort| uniq

sort and remove duplicates

| tr 'A-Z' 'a-z'

convert uppercase to lowercase

| tr -d '\n'

concatenate lines into one line

| tr -d '\r'

remove CR

| sed 's/^/# /'

add "#" to the start of each line

| sed 's/\.ext//g'

remove ".ext"

| sed -n -e 2p

print the second line

| head -n 2 -

print the first 2 lines

| tail -n 2 -

print the last 2 lines


When using the shell interactive mode becomes too complicated, please consider to write a shell script (see: Section 13.1, “The shell script”).

2.6.6. Perl one liner for the regular-expression substitution

The following execution of perl(1) one liner command will replace all instances of FROM_REGEX with TO_TEXT in all of the files <target_file> ...:

$ perl -i -p -e 's/FROM_REGEX/TO_TEXT/g;' <target_file> ...

"-i" is for "in-place editing", "-p" is for "implicit loop over <target_file> ...". If the substitution is complex, you can make recovery from errors easier by using the parameter "-i.bak" instead of "-i"; this will keep each original file, adding ".bak" as a file extension.

[Note] Note

Although this is somewhat waste of the resource, this is used frequently to change file contents across the whole directly with minimal typing.

You can do the similar with ed(1) command too.

$ ed <target_file> <<EOF
,s/FROM_REGEX/TO_TEXT/g
w
q
EOF

Here, the ed commands are practically the same command as the vi command-mode command.

Table 2.28.  The comparison of ed vs perl for in-place editing.

command

type

argument

regex

script

ed

lighter and faster

works on one file

BRE

read from stdin

perl

heavier and slower

works on multiple files

ERE

can be as a part of the argument


Chapter 3. Debian package management

Debian is a volunteer organization which builds consistent distributions of pre-compiled binary packages of free software and distributes them from its archive.

The Debian archive is offered by many remote mirror sites for access through HTTP and FTP methods. It is also available as CD-ROM/DVD.

The Debian package management system, when used properly, offers the user to install consistent sets of binary packages to the system from the archive. Currently, there are 24396 packages available for the amd64 architecture.

The Debian package management system has a rich history and many choices for the front end user program and back end archive access method to be used. Currently, we recommend aptitude as the main front end program for the Debian package management activity.

Table 3.1.  List of Debian package management tools

package

popcon

size

description

aptitude

V:23, I:98

9808

terminal-based package manager (current standard, front-end for apt)

apt

V:85, I:99

5216

Advanced Packaging Tool, front-end for dpkg providing "http", "ftp", and "file" archive access methods (apt-get/apt-cache commands included)

tasksel

V:6, I:94

884

tool for selecting tasks for installation on Debian system (front-end for apt)

dselect

V:5, I:82

2192

terminal-based package manager (previous standard, front-end for apt and other old access methods)

dpkg

V:85, I:99

7340

package management system for Debian

dpkg-ftp

V:0.10, I:0.5

136

older ftp method for dselect

synaptic

V:20, I:52

5952

graphical package manager (Gnome front-end for apt)

gnome-apt

V:0.2, I:1.4

604

graphical package manager (Gnome front-end for apt)

kpackage

V:5, I:14

1496

graphical package manager (KDE front-end for apt)

apt-utils

V:49, I:99

456

APT utility programs: apt-extracttemplates(1), apt-ftparchive(1), and apt-sortpkgs(1)

apt-listchanges

V:3, I:6

264

package change history notification tool

apt-listbugs

V:1.6, I:2

436

lists critical bugs before each APT installation

apt-file

V:1.8, I:8

172

APT package searching utility -- command-line interface

apt-rdepends

V:0.18, I:0.9

92

recursively lists package dependencies


[Note] Note

The annoying bug #411123 for the mixed use of aptitude and apt-get commands has been resolved. If this kept you from using aptitude, please reconsider.

3.1. Debian package management prerequisites

3.1.1. Package configuration

Here are some key points for package configuration on the Debian system:

  • The manual configuration by the system administrator is respected. In other words, the package configuration system makes no intrusive configuration for the sake of convenience.

  • Each package comes with its own configuration script with standardized user interface called debconf(7) to help initial installation process of the package.

  • Debian Developers try their best to make your upgrade experience flawless with package configuration scripts.

  • Full functionalities of packaged software are available to the system administrator. But ones with security risks are disabled in the default installation.

  • If you manually activate a service with some security risks, you are responsible for the risk containment.

  • Esoteric configuration may be manually enabled by the system administrator. This may creates interference with popular generic helper programs for the system configuration.

3.1.2. Basic precautions

[Warning] Warning

Do not install packages from random mixture of suites. It will likely break the package consistency which requires deep system management knowledge, such as compiler ABI, library version, interpreter features, etc.

The newbie Debian system administrator should stay with the stable release of Debian while applying only security updates. I mean that some of the valid actions are better avoided, as a precaution, until you understand the Debian system very well:

  • Do not include testing or unstable in /etc/apt/sources.list,

  • Do not mix standard Debian with other non-Debian archives such as Ubuntu in /etc/apt/sources.list,

  • Do not create /etc/apt/preferences,

  • Do not change default behavior of package management tools through configuration files without knowing their full impacts,

  • Do not install random packages by "dpkg -i <random_package>",

  • Do not ever install random packages by "dpkg --force-all -i <random_package>",

  • Do not erase or alter files in /var/lib/dpkg/, or

  • Do not overwrite system files by installing software programs directly compiled from source. (Install them into /usr/local or /opt.)

The non-compatible effects caused by above actions to the Debian package management system may leave your system unusable.

The serious Debian system administrator who runs mission critical servers, should use extra precautions:

  • Do not install any packages including security updates from Debian without thoroughly testing them with your particular configuration under safe conditions. (Although Debian has been offering an extremely stable system for a long time, you as the system administrator are responsible for your system in the end.)

3.1.3. Life with eternal upgrades

Despite my warnings above, I know many readers of this document wish to run the "testing" or "unstable" suites of Debian as their main Desktop system since they work very well for self-administered Desktop environments. Because they are updated frequently, they offer the latest features.

[Caution] Caution

For your production server, the "stable" suite with the security updates is recommended. The same can be said for desktop PCs on which you can spend limited administration efforts, e.g. for your mother's PC.

It takes no more than simply setting the distribution string in the /etc/apt/sources.list to the suite: "testing" or "unstable"; or the codename: "squeeze" or "sid". This will let you live the life of eternal upgrades.

The use of "testing" or "unstable" is a lot of fun but comes with some risks. Even though the "unstable" suite of Debian system looks very stable for most of the times, there have been some package problems on the "testing" and "unstable" suite of Debian system and a few of them were not so trivial to resolve. It may be quite painful for you. Sometimes, you may have a broken package or missing functionality for a few weeks.

Here are some ideas to ensure quick and easy recovery from bugs in Debian packages:

  • make the system dual bootable by installing the "stable" suite of Debian system to another partition.

  • make the installation CD handy for the rescue boot.

  • consider installing apt-listbugs to check the Debian Bug Tracking System (BTS) information before the upgrade.

  • learn the package system infrastructure enough to work around the problem.

  • create a chroot or similar environment and run the latest system in it in advance. (optional)

(If you can not do any one of these precautionary actions, you are probably not ready for the "testing" and "unstable" suites.)

Enlightenment with the following will save a person from the eternal karmic struggle of upgrade hell and let him reach Debian nirvana.

3.1.4. Debian archive basics

Let's look into the Debian archive from a system user's perspective.

[Tip] Tip

Official policy of the Debian archive is defined at Debian Policy Manual, Chapter 2 - The Debian Archive.

For the typical HTTP access, the archive is specified in the /etc/apt/sources.list file as, e.g. for the current "stable" == "lenny" system:

deb http://ftp.XX.debian.org/debian/ lenny main contrib non-free
deb-src http://ftp.XX.debian.org/debian/ lenny main contrib non-free

deb http://security.debian.org/ lenny/updates main contrib
deb-src http://security.debian.org/ lenny/updates main contrib

Please note "ftp.XX.debian.org" must be replaced with appropriate mirror site URL for your location, for USA "ftp.us.debian.org", which can be found in the list of Debian worldwide mirror sites. The status of these servers can be checked at Debian Mirror Checker site.

Here, I tend to use codename "lenny" instead of suite name "stable" to avoid surprises when the next "stable" is released.

The meaning of this is described in "man 5 sources.list" and key points are:

  • The "deb" line defines for the binary packages.

  • The "deb-src" line defines for the source packages.

  • The 1st argument is the root URL of the Debian archive.

  • The 2nd argument is the distribution: either the suite name or the codename.

  • The 3rd and following arguments are the list of valid archive component names of the Debian archive.

The "deb-src" lines can safely be omitted (or commented out by placing "#" at the start of the line) if it is just for aptitude which does not access source related meta data. It will speed up the updates of the archive meta data. The URL can be "http://", "ftp://", "file://", ....

[Tip] Tip

If "sid" is used in the above example instead of "lenny", the "deb: http://security.debian.org/ ..." line for security updates in the /etc/apt/sources.list is not required. Security updates are only available for "stable" and "testing" (i.e., "lenny" and "squeeze").

Here are the lists of URL of the Debian archive sites and suite or codename used in the configuration file:

Table 3.2.  Lists of Debian archive sites.

archive URL

suite (codename)

purpose

http://ftp.XX.debian.org/debian/

stable (lenny)

stable (lenny) release

http://ftp.XX.debian.org/debian/

testing (squeeze)

testing (squeeze) release

http://ftp.XX.debian.org/debian/

unstable (sid)

unstable (sid) release

http://ftp.XX.debian.org/debian/

experimental

experimental pre-release (optional, only for developer)

http://ftp.XX.debian.org/debian/

stable-proposed-updates

Updates for the next stable point release (optional)

http://security.debian.org/

stable/updates

Security updates for stable release (important)

http://security.debian.org/

testing/updates

Security updates for testing release (important)

http://volatile.debian.org/debian-volatile/

volatile

Compatible updates for spam filter and IM clients, etc.

http://volatile.debian.org/debian-volatile/

volatile-sloppy

Non-compatible updates for spam filter, IM clients, etc.

http://backports.org/debian/

lenny-backports

Newer backported packages for lenny. (non-official, optional)


[Tip] Tip

For the Debian system with the "stable" and "testing" suites, it is a good idea to include lines with http://security.debian.org/ in the /etc/apt/sources.list to enable access to the archive for security updates as in the example above.

[Caution] Caution

Only pure stable release with security updates provides the best stability. Running mostly stable release mixed with some packages from testing or unstable release is riskier than running pure unstable release. If you really need the latest version of some programs under stable release, please use packages from the debian-volatile project and http://backports.org (see: Section 3.7.4, “Volatile and Backports.org”) services. These services must be used with extra care.

[Caution] Caution

You should basically list only one of "stable", "testing", or "stable" suites in the "deb" line. If you list any combination of "stable", "testing", and "stable" suites in the "deb" line, APT programs slow down while only the latest archive is effective. Multiple listing makes sense for these when the /etc/apt/preferences file is used with clear objectives (see: Section 3.7.3, “Tweaking candidate version”).

Each Debian archive consists of 3 components. Components are alternatively called categories in "Debian Policy" or areas in "Debian Social Contract". The component is grouped by the compliance to "The Debian Free Software Guidelines " (DFSG):

Table 3.3.  The lists of Debian archive components.

component

number of packages

criteria

main

23809

The package is fully compliant to DSFG and does not depend the non-free package.

contrib

211

The package is compliant to the DSFG but depends on the non-free package.

non-free

376

The package is not compliant to the DSFG but distributable and useful.


Here the number of packages in the above is for the amd64 architecture. Strictly speaking, only the main component archive shall be considered the Debian system.

The Debian archive organization can be studied best by pointing your browser to the each archive URL appended with dists or pool.

The distribution is referred by two ways, the suite or codename. The word distribution is alternatively used as the synonym to the suite in many documentations. The relationship between the suite and the codename can be summarized as:

Table 3.4.  The relationship between suite and codename.

Timing

suite = "stable"

suite ="testing"

suite ="unstable"

after the "lenny" release

codename = "lenny"

codename = "squeeze"

codename = "sid"

after the "squeeze" release

codename = "squeeze"

codename = "squeeze+1"

codename = "sid"


The history of codenames are described in Debian FAQ: 6.3.1 Which other codenames have been used in the past?

In the stricter Debian archive terminology, the word "section" is specifically used for the categorization of packages by the application area. (Although, the word "main section" may sometimes be used to describe the Debian archive section which provides the main component.)

Every time a new upload is done by the Debian developer (DD) to the "unstable" archive (via incoming processing), DD is required to ensure uploaded packages to be compatible with the latest set of packages in the latest "unstable" archive.

If DD breaks this compatibility intentionally for important library upgrade etc, there is usually announcement to the debian-devel mailing list etc.

Before a set of packages are moved by the Debian archive maintenance script from the "unstable" archive to the "testing" archive, the archive maintenance script not only checks the maturity (about 10 days old) and the status of the RC bug reports for the packages but also tries to ensure them to be compatible with the latest set of packages in the "testing" archive. This process makes the "testing" archive very current and usable.

Through the gradual archive freeze process led by the release team, the "testing" archive will be matured to make it completely consistent and bug free with some manual interventions. Then the new "stable" release is created by assigning the codename for the old "testing" archive to the new "stable" archive and creating the new codename for the new "testing" archive. The initial contents of the new "testing" archive is exactly the same as that of the newly released "stable" archive.

Both the "unstable" and the "testing" archives may suffer temporary glitches due to:

  • broken package upload to the archive (mostly for "unstable"),

  • delay of accepting the new packages to the archive (mostly for "unstable"),

  • archive synchronization timing issue (both for "testing" and "unstable"),

  • manual intervention to the archive such as package removal (more for "testing"), etc.

So if you ever decide to use these archives, you should be able to fix or work around these kinds of glitches.

[Tip] Tip

When tracking the "testing" archive, problem caused by a removed package is usually worked around by installing corresponding package from the "unstable" archive which is uploaded for bug fix.

See Debian Policy Manual for definition of:

3.1.5. Package dependencies

The Debian system offers a consistent set binary packages through its versioned binary dependency declaration mechanism through the control file fields. Here is a bit over simplified definition for them.

Table 3.5.  List of package dependencies.

dependency

meaning

Depends

This declares an absolute dependency and all of the packages listed in this field must be installed at the same time or in advance.

Pre-Depends

This is like Depends, except that it requires completed installation of the listed packages in advance.

Recommends

This declares a strong, but not absolute, dependency. Most users would not want the package unless all of the packages listed in this field are installed.

Suggests

This declares a weak dependency. Many users of this package may benefit from installing packages listed in this field but can have reasonable functions without them.

Enhances

This declares a week dependency like Suggests but works in the opposite direction.

Conflicts

This declares an absolute incompatibility. All of the packages listed in this field must be removed to install this package.

Replaces

This is declared when files installed by this package replace files in the listed packages.

Provides

This is declared when this package provide all of the files and functionality in the listed packages.


[Note] Note

Please note that defining, Provides, Conflicts and Replaces simultaneously to an virtual package is the sane configuration. This ensures that only one real package providing this virtual package can be installed at any one time.

The official definition including source dependency can be found in the Policy Manual: Chapter 7 - Declaring relationships between packages.

3.1.6. The event flow of the package management

The simplified event flow of the update is:

  • The local copy of package archive metadata is updated by the remote one(s) as: fetch -> reconstruct

The simplified event flow of the upgrade (safe-upgrade, full-upgrade, upgrade, and dist-upgrade) and the install are:

  1. APT system makes decision on candidate version which is usually the latest available version. (See Section 3.7.3, “Tweaking candidate version” for exception.)

  2. The system administrator makes choice such as upgrade of the entire system or install several new packages with candidate version.

  3. Selected binary packages are processed as: fetch -> unpack -> preinst -> install -> postinst

The package removal process has 2 distinct stages and the simplified event flow of them are:

  • remove : remove all installed files except configuration files as: prerm -> remove -> postrm

  • purge : purge all installed files completely including configuration files as: prerm -> purge -> postrm

Here, I intentionally skipped technical details for the sake of big picture.

3.1.7. First response to package management troubles

You should read the fine official documentation. The first document to read is the Debian specific /usr/share/doc/<package_name>/README.Debian. Other documentation in /usr/share/doc/<package_name>/ should be consulted too. If you set shell as previously discussed, type:

$ cd <package_name>
$ pager README.Debian
$ mc

You may need to install the corresponding documentation package named with "-doc" suffix for detailed information.

If you are experiencing problems with a specific package, make sure to check out these sites first:

Table 3.6.  List of key web site to resolving problems with a specific package.

site

command

Home page of the Debian bug tracking system (BTS)

$ sensible-browser http://bugs.debian.org/

The bug report of a known package name.

$ sensible-browser http://bugs.debian.org/<package_name>

The bug report of known bug number.

$ sensible-browser http://bugs.debian.org/<bug_number>


Search Google with search words including "site:debian.org".

When you file a bug report, please use reportbug command.

3.2. Basic package management operations

Aptitude is the current preferred package management tool for the Debian system. It can be used as the commandline alternative to apt-get / apt-cache and also as the full screen interactive package management tool.

For the package management operation which involves package installation or updates package metadata, you need to have root privilege.

3.2.1. Basic package management operations with commandline

Here are package management operations with commandline using aptitude(8) and apt-get(8) /apt-cache(8).

Table 3.7.  Package management operations with commandline using aptitude and apt-get / apt-cache.

aptitude syntax

apt-get/apt-cache syntax

description

"aptitude update"

"apt-get update"

Update package archive metadata.

"aptitude install foo"

"apt-get install foo"

Install candidate version of "foo" package with its dependencies.

"aptitude safe-upgrade"

"apt-get upgrade"

Install candidate version of installed packages without removing any other packages.

"aptitude full-upgrade"

"apt-get dist-upgrade <package>"

Install candidate version of installed packages while removing other packages if needed.

"aptitude remove foo

"apt-get remove foo"

Remove "foo" package while leaving its configuration files.

N/A

"apt-get autoremove"

Remove auto-installed packages which is no longer required.

"aptitude purge foo"

"apt-get purge foo"

Purge "foo" package with its configuration files.

"aptitude clean"

"apt-get clean"

Clear out the local repository of retrieved package files completely.

"aptitude autoclean"

"apt-get autoclean"

Clear out the local repository of retrieved package files for outdated packages.

"aptitude show foo"

"apt-cache show <package>"

Display detailed information about "foo" package.

"aptitude search <regex>"

"apt-cache search <regex>"

Search packages which match <regex>.

"aptitude why <regex>"

N/A

Explain the reason why <regex> matching packages should be installed.

"aptitude why-not <regex>"

N/A

Explain the reason why <regex> matching packages can not be installed.


Although it is now safe to mix different package tools on the Debian system, it is best to continue using aptitude as much as possible.

The difference between "safe-upgrade" and "full-upgrade" only appears when new versions of packages stand in different dependency relationships from old versions of those packages. The "aptitude safe-upgrade" command will never install new packages nor remove installed packages.

The "aptitude why <regex>" can list more information by "aptitude -v why <regex>". Similar information can be obtained by "apt-cache rdepends <package>".

When aptitude command is started in the commandline mode and faces some issues such as package conflicts, you can switch to the full screen interactive mode by pressing "e"-key later at the prompt.

You may provide command options right after "aptitude".

Table 3.8.  Notable command options for "aptitude".

command option

effects

-s

simulate the result of the command.

-d

download only but no install/upgrade.

-D

show brief explanations before the automatic installations and removals.


See aptitude(8) and the "User's Manual" /usr/share/doc/aptitude/README for more.

[Tip] Tip

The dselect package is still available and was the preferred full screen interactive package management tool in previous releases.

3.2.2. Interactive use of aptitude

For the interactive package management, you start aptitude in interactive mode from the console shell prompt as:

$ sudo aptitude -u
Password:

This will update the local copy of the archive information and display the package list in the full screen with menu. Aptitude places its configuration at $HOME/.aptitude/config.

[Tip] Tip

If you want to use root's configuration instead of user's one, use "sudo -H aptitude ..." instead of "sudo aptitude ..." in the above expression.

[Note] Note

Aptitude automatically sets pending actions as it is started interactively. If you do not like it, you can reset it from menu: "Action" -> "Cancel pending actions".

3.2.3. Key bindings of aptitude

Notable key strokes to browse status of packages and to set "planned action" on them in this full screen mode are:

Table 3.9.  List of key bindings for aptitude.

key

key binding

F10 or in a terminal window Ctrl-t

Menu

?

Display help for keystroke (more complete listing)

F10 -> Help -> User's Manual

Display User's Manual

u

Update package archive information

+

Mark the package for the upgrade or the install

-

Mark the package for the remove (keep conffiles)

_

Mark the package for the purge (remove conffiles)

=

Place the package on hold

U

Mark all upgradable packages (function as dist-upgrade)

g

Start downloading and installing selected packages

q

Quit current screen and save changes

x

Quit current screen and discard changes

Enter

View information about a package

C

View a package's changelog

l

Change the limit for the displayed packages

/

Search for the first match

\

Repeat the last search


The file name specification of the command line and the menu prompt after pressing "l" and "/" take the aptitude regex as described below. For the input to the menu prompt and argument to "aptitude search" command, "~n" is prepended to match the package name with the pattern if the input does not start with "~" character.

[Tip] Tip

You need to press "U" to get all the installed packages upgraded to the candidate version in the visual interface. Otherwise only the selected packages and certain packages with versioned dependency to them are upgraded to the candidate version.

3.2.4. Package views under aptitude

In the interactive full screen mode of aptitude(8), packages in the package list are displayed like this by default:

idA   libsmbclient                             -2220kB 3.0.25a-1  3.0.25a-2

Here, this line means from the left as:

  • The "current state" flag (the first letter)

  • The "planned action" flag (the second letter)

  • The "automatic" flag (the third letter)

  • The package name

  • The change in disk space usage attributed to "planned action".

  • The current version of the package.

  • The candidate version of the package.

[Tip] Tip

The full list of flags are given at the bottom of Help screen shown by pressing "?".

The candidate version is chosen according to the current local policy and preferences (see apt_preferences(5)).

Several types of package views are available under the menu "Views":

Table 3.10.  Views for aptitude.

view

categorization

status

Package View

See Table 3.11, “The categorization of standard aptitude views. ”. (default)

Good

Audit Recommendations

Packages which are recommended by some installed packages but not yet installed are listed.

Good.

Flat Package List

Packages are listed without categorization (for use with regex).

Good

Debtags Browser

Packages are categorized according to their debtags entries.

Very usable

Categorical Browser

Packages are categorized according to their category.

Deprecated (Use debtags!)


The standard "Package View" categorizes packages somewhat like dselect with few extra features. For switching distribution to a newer one can be achieved basically by

Table 3.11. The categorization of standard aptitude views.

category

organization

"Upgradable Packages"

Organized as section --> component --> package

"New Packages"

, ,

"Installed Packages"

, ,

"Not Installed Packages"

, ,

"Obsolete and Locally Created Packages"

, ,

"Virtual Packages"

You can pick a particular package from a set of packages with the same function.

"Tasks"

You can cherry pick particular packages from a set of packages of a task.


3.2.5. Search method options with aptitude

Aptitude offers several options for you to search packages using its regex formula:

  • "aptitude search '<aptitude_regex>'" to list their installation status, package name and short description.

  • "aptitude show '<package_name>'" to list their installation detailed description.

  • limit view to matching packages: Type "l" in the full screen mode.

  • search the first found package: type "/" in the full screen mode. "n" for find-next, "\" for backward search.

Here, the string for <package_name> is treated as the exact string match to the package name unless it is started explicitly with "~" to be the regex formula.

3.2.6. The aptitude regex formula

The aptitude regex formula is mutt-like extended ERE (see: Section 2.6.2, “Regular expressions”) and the meanings of the aptitude specific special match rule extensions are as below:

Table 3.12.  List of the aptitude regex formula.

meaning of the extended match rule

regex formula

match on package name

~n<regex_name>

match on description

~d<regex_description>

match on task name

~t<regex_task>

match on debtag

~G<regex_debtag>

match on maintainer

~m<regex_maintainer>

match on package section

~s<regex_section>

match on package version

~V<regex_version>

match archive

~A{sarge,etch,sid}

match origin

~O{debian,...}

match priority

~p{extra,important,optional,required,standard}

match essential packages

~E

match virtual packages

~v

match new packages

~N

match with pending action

~a{install,upgrade,downgrade,remove,purge,hold,keep}

match installed packages

~i

match installed packages with A-mark (auto installed package)

~M

match installed packages without A-mark (administrator selected package)

~i!~M

match installed and upgradable packages

~U

match removed but not purged packages

~c

match removed, purged or can-be-removed packages

~g

match with broken relation

~B<type>

match broken depends/predepends/conflict packages

~b

match packages whose control files define relation <type> to the <term> package

~D[<type>:]<term>

match packages whose control files define broken relation <type> to the <term> package

~DB[<type>:]<term>

match packages to which the <term> package defines relation <type>

~R[[<type>:]<term>

match packages to which the <term> package defines broken relation <type>

~RB[<type>:]<term>

match packages to which some other installed packages depend on

~R~i

match packages to which no other installed packages depend on

!~R~i

match packages to which some other installed packages depend or recommend on

~R~i|~Rrecommends:~i

match <term> package with filtered version

~S filter <term>

match all packages (true)

~T

match no packages (false)

~F


Here,

  • regex part is the same ERE as the one used in typical Unix-like text tools using "^", ".*", "$" etc. as in egrep(1), awk(1) and perl(1).

  • relation <type> is one of (depends, predepends, recommends, suggests, conflicts, replaces, provides).

  • the default relation type is "depends".

[Tip] Tip

When <regex_pattern> is a null string, place "~T" immediately after the command.

Short cuts:

  • "~P<term>" == "~Dprovides:<term>"

  • "~C<term>" == "~Dconflicts:<term>"

  • "...~W term" == "(...|term)"

Users familiar with mutt will pick up quickly, as mutt was the inspiration for the expression syntax. See "SEARCHING, LIMITING, AND EXPRESSIONS" in the "User's Manual" /usr/share/doc/aptitude/README.

[Note] Note

With the "lenny" version of aptitude(8), the new long form syntax such as "?broken" may be used for regex matching in place for its old short form equivalent "~b". Now space character " " is considered as one of the regex terminating character in addition to tilde character "~". See "User's Manual" for the new long form syntax.

3.2.7. Dependency resolution of aptitude

The selection of a package in aptitude not only pulls in packages which are defined in its "Depends:" list but also defined in the "Recommends:" list if the menu "F10 -> Options -> Dependency handling" is set accordingly. These auto installed packages are removed automatically if they are no longer needed under aptitude.

[Note] Note

Before the "lenny" release, apt-get and other standard APT tools did not offer the autoremove functionality.

3.2.8. Package activity logs

You can check package activity history in the log files.

Table 3.13.  The log files for package activities.

file

content

/var/log/dpkg.log

Log of dpkg level activity for all package activities.

/var/log/apt/term.log

Log of generic APT activity.

/var/log/aptitude

Log of aptitude command activity.


In reality, it is not so easy to get meaningful understanding quickly out from these logs. See Section 10.2.8, “Recording changes in configuration files” for easier way.

3.2.9. Aptitude advantages

Aptitude has advantages over other APT based packaging systems (apt-get, apt-cache, synaptic, ...):

  • aptitude removes unused auto installed packages automatically using its own extra layer of package state file (/var/lib/aptitude/pkgstates). (For new "lenny", other APT does the same.)

  • aptitude makes it easy to resolve package conflicts and to add recommended packages.

  • aptitude makes it easy to keep track of obsolete software by listing under "Obsolete and Locally Created Packages".

  • aptitude gives a log of its history in /var/log/aptitude.

  • aptitude offers access to all versions of the package if available.

  • aptitude includes a fairly powerful regex based system for searching particular packages and limiting the package display.

  • aptitude in the full screen mode has su functionality embedded and can be run from normal user until you really need administrative privileges.

For the old "etch" release version, synaptic also gives you the history log; apt-get did not but you can rely on the log of dpkg.

Anyway, aptitude is nice for interactive console use.

3.3. Examples of aptitude operations

Here are few examples of aptitude(8) operations.

3.3.1. List packages with regex matching package name

The following command lists packages with regex matching names.

$ aptitude search '~n(pam|nss).*ldap'
p libnss-ldap - NSS module for using LDAP as a naming service
p libpam-ldap - Pluggable Authentication Module allowing LDAP interfaces

This is quite handy for you to find the exact name of a package.

3.3.2. Browse with the regex matching

The regex "~dipv6" in the "New Flat Package List" view with "l" prompt, limits view to packages with the matching description and let you browse their information interactively.

3.3.3. Purge removed packages for good

You can purge all remaining configuration files of removed packages:

# aptitude search '~c'
  • check results

# aptitude purge '~c'

You may want to do the similar in the interactive mode for fine grained control.

You provide the regex "~c" in the "New Flat Package List" view with "l" prompt. This limits the package view only to regex matched packages, i.e., "removed but not purged". All these regex matched packages can be shown by pressing "[" at top level headings.

Then you press "_" at top level headings such as "Installed Packages". Only regex matched packages under the heading are marked to be purged by this. You can exclude some packages to be purged by pressing "=" interactively for each of them.

This technique is quite handy and works for many other command keys.

3.3.4. Tidy auto/manual install status

Here is how I tidy auto/manual install status for packages (after using non-aptitude package installer etc.):

  • Start aptitude in interactive mode as root.

  • Type "u", "U", "f" and "g" to update and upgrade package list and packages.

  • Type "l" to enter the package tree limit as "~i(~R~i|~Rrecommends:~i)" and type "M" over "Installed Packages" as auto installed.

  • Type "l" to enter the package tree limit as "~prequired|~pimportant|~pstandard|~E" and type "m" over "Installed Packages" as manual installed.

  • Type "l" to enter the package tree limit as "~i!~M" and remove unused package by typing "-" over each of them after exposing them by typing "[" over "Installed Packages".

  • Type "l" to enter the package tree limit as "~i" and type "m" over "Tasks" as manual installed.

  • Exit aptitude.

  • Start "apt-get -s autoremove|less" as root to check what are not used.

  • Restart aptitude in interactive mode and mark needed packages as "m".

  • Restart "apt-get -s autoremove|less" as root to recheck REMOVED contain only expected packages.

  • Start "apt-get autoremove|less" as root to autoremove unused packages.

The "m" action over "Tasks" is an optional one to prevent mass package removal situation in future.

3.3.5. System wide upgrade with aptitude

[Note] Note

When moving to a new release etc, you should consider to perform a clean installation of new system even though Debian is upgradable as described below. This provides you a chance to remove garbages collected and exposes you to the best combination of latest packages. Of course, you should make a full backup of system to a safe place (see Section 11.1.6, “Backup and recovery”) before doing this. I recommend to make a dual boot configuration using different partition to have the smoothest transition.

You can perform system wide upgrade to a newer release by changing contents of the /etc/apt/sources.list file pointing to a new release and running the "aptitude update; aptitude dist-upgrade" command.

To upgrade from "stable" to "testing" or "unstable", you replace "lenny" in the /etc/apt/sources.list example of Section 3.1.4, “Debian archive basics” with "squeeze" or "sid".

In reality, you may face some complications due to some package transition issues, mostly due to package dependencies. The larger the difference of the upgrade, the more likely you face larger troubles. For the transition from the old "stable" archive to the new "stable" after its release, you can read its new Release Notes and follow the exact procedure described in it to minimize troubles.

When you decide to move from "stable" to "testing" before its formal release, there are no Release Notes to help you. The difference between "stable" and "testing" could have grown quite large after the previous "stable" release and makes upgrade situation complicated.

You should make some precautionary moves while gathering latest information from mailing list and using common senses:

  • read previous "Release Notes".

  • back up entire system (especially data and configuration information).

  • have bootable media handy for broken bootloader.

  • inform users on the system well in advance.

  • record upgrade activity with the script(1) command.

  • apply "unmarkauto" to essential packages, e.g., "aptitude unmarkauto vim", to prevent removal.

  • minimize installed packages to reduce chance of package conflicts, e.g., remove desktop task packages.

  • remove the /etc/apt/preferences file. (disable apt-pinning)

  • try to upgrade step wise: "oldstable" --> "stable" --> "testing" --> "unstable".

  • update the /etc/apt/sources.list file to point to new archive only and run "aptitude update".

  • install, optionally, new core packages first, e.g., "aptitude install perl".

  • run the "aptitude dist-upgrade -s" command to assess impact.

  • run the "aptitude dist-upgrade" command.

[Caution] Caution

It is not wise to skip major Debian release when upgrading between "stable" releases.

[Caution] Caution

In previous "Release Notes", GCC, Linux Kernel, initrd-tools, Glibc, Perl, APT tool chain, etc. have required some special attention for system wide upgrade.

For daily upgrade in "unstable", see Section 3.4.3, “Safeguard for package problems”.

3.4. Advanced package management operations

3.4.1. Advanced management operations with commandline

Here are list of other package management operations for which aptitude is too high-level or lacks required functionalities.

Table 3.14.  List of advanced package management operations.

action

command

list status of an installed package for the bug report.

"COLUMNS=120 dpkg -l <package_name_pattern>"

list the contents of an installed package.

"dpkg -L <package_name>"

list the manpages for an installed package.

"dpkg -L <package_name> | egrep '/usr/share/man/man.*/.+'"

list installed packages which have matching file name.

"dpkg -S <file_name_pattern>"

list packages in archive which have matching file name.

"apt-file search <file_name_pattern>"

list the contents of matching packages in archive.

"apt-file list <package_name_pattern>"

reconfigure the exact package .

"dpkg-reconfigure <package_name>"

reconfigure the exact package with the most detailed question.

"dpkg-reconfigure -p=low <package_name>"

reconfigure packages from the full screen menu.

"configure-debian"

audit system for partially installed packages.

"dpkg --audit"

configures all partially installed packages.

"dpkg --configure -a"

show available version, priority, and archive information of a binary package.

"apt-cache policy <binary_package_name>"

show available version, archive information of a package.

"apt-cache madison <package_name>"

show source package information of a binary package.

"apt-cache showsrc <binary_package_name>"

install required packages to build package.

"apt-get build-dep <package_name>"

download a source. (from standard archive)

"apt-get source <package_name>"

download a source packages. (from other archive)

"dget <URL for dsc file>"

build a source tree from a set of source packages (*.tar.gz *.diff.gz).

"dpkg-source -x <package_name>_<version>-<debian_version>.dsc"

build package(s) from a local source tree.

"debuild binary"

build a kernel package from a kernel source tree.

"make-kpkg kernel_image"

build a kernel package from a kernel source tree with initramfs enabled.

"make-kpkg --initrd kernel_image"

install a local package to the system.

"dpkg -i <package_name>_<version>-<debian_version>_<arch>.deb"

install local package(s) to the system.

"debi <package_name>_<version>-<debian_version>_<arch>.dsc"

save dpkg level package selection state information

"dpkg --get-selection '*' >selection.txt"

set dpkg level package selection state information

"dpkg --set-selection <selection.txt"


[Caution] Caution

Use of lower level package tools such as "dpkg -i ..." and "debi ..." should be carefully used by the system administrator. It does not automatically take care required package dependencies. Dpkg's commandline options "--force-all" and similar (see dpkg(1)) are intended to be used by experts only. Using them without fully understanding their effects may break your whole system.

Please note:

[Tip] Tip

The source package format described here as a set of source packages (*.tar.gz *.diff.gz) is format 1.0 which is still popular. See more on dpkg-source(1) for other newer formats.

3.4.2. Verify installed package files

The installation of debsums package enables verification of installed package files against MD5sum values in the Packages file with debsums(1) command. See: Section 11.3.5, “The MD5 sum” for how MD5sum works.

[Note] Note

Because MD5sum database may be tampered by the intruder, debsums(1) is of limited use as a security tool. It is only good for checking local modifications by the administrator or damages due to media errors.

3.4.3. Safeguard for package problems

Many user prefer to follow the unstable release of the Debian system for its new features and packages. This makes the system more prone to be hit by the critical package bugs.

The installation of the apt-listbugs package will provide safeguard to the critical bugs by checking Debian BTS automatically for critical bugs when upgrading with APT system.

The installation of the apt-listchanges package will provide important news in NEWS.Debian when upgrading with APT system.

3.4.4. Search on the package meta data

Although visiting Debian site http://packages.debian.org/ facilitates easy ways to search on the package meta data these days, let's look into more traditional ways.

The grep-dctrl(1), grep-status(1), and grep-available(1) commands can be used to search any file which has the general format of a Debian package control file.

The "dpkg -S <file_name_pattern>" can be used search package names which contain files with the matching name installed by dpkg. But this overlooks files created by the maintainer scripts.

If you need to make more elaborate search on the dpkg meta data, you need to run "grep -e regex_pattern *" command in the /var/lib/dpkg/info/ directory. This will let you identify:

  • the package name which installs, creates or modifies particular file which match pattern.

  • the package name which asks the installation query words which match pattern.

If you wish to look up package dependency recursively, you should use apt-rdepends(8).

3.5. Debian package management internals

Let's learn how the Debian package management system works internally. This should help you to create your own solution to some package problems.

3.5.1. Archive meta data

The meta data files are stored under the dist/ on each Debian mirror sites, e.g., ftp://ftp.us.debian.org/debian/. Its archive structure can be browsed by the web browser. There are 6 types of key meta data:

Table 3.15.  The content of the Debian archive meta data.

file

location

content

Release

top of distribution

archive description and integrity information

Release.gpg

top of distribution

signature file for Release signed with the archive key

Contents-<architecture>

top of distribution

list of all files for all the packages in the pertinent archive

Release

top of each distribution/component/architecture combination

archive description

Packages

top of each distribution/component/binary-architecture combination

concatenated debian/control for binary packages

Sources

top of each distribution/component/source combination

concatenated debian/control for source packages


In the recent archive, these meta data are stored as the compressed and differential files to reduce network traffic.

3.5.2. Top level Release file and authenticity

Each suites of the Debian archive has a top level Release file, e.g., ftp://ftp.us.debian.org/debian/dists/unstable/Release:

Origin: Debian
Label: Debian
Suite: unstable
Codename: sid
Date: Sat, 26 Jan 2008 20:13:58 UTC
Architectures: alpha amd64 arm hppa hurd-i386 i386 ia64 m68k mips mipsel powerpc s390 sparc
Components: main contrib non-free
Description: Debian x.y Unstable - Not Released
MD5Sum:
 e9f11bc50b12af7927d6583de0a3bd06 22788722 main/binary-alpha/Packages
 43524d07f7fa21b10f472c426db66168  6561398 main/binary-alpha/Packages.gz
...
[Note] Note

Here, you can find my rationale to use the "suite", "codeneme", and "components" in Section 3.1.4, “Debian archive basics”. The "distribution" is used when referring to both "suite" and "codeneme".

The integrity of the top level Release file is verified by cryptographic infrastructure called the secure apt.

  • The cryptographic signature file Release.gpg is created from the authentic top level Release file and the secret Debian archive key.

  • The public Debian archive signing key can be seeded into /etc/apt/trusted.gpg:

  • The secure APT system verifies the integrity of the downloaded top level Release file cryptographically by this Release.gpg file and the public Debian archive key in /etc/apt/trusted.gpg.

The integrity of all the Packages and Sources files are verified by using MD5sum values in its top level Release file. The integrity of all package files are verified by using MD5sum values in the Packages and Sources files. See debsums(1) and Section 3.4.2, “Verify installed package files”.

Since the cryptographic signature verification is very CPU intensive process than the MD5sum value calculation, use of MD5sum value for each package while using cryptographic signature for the top level Release file provides the good security with the performance (see: Section 11.3, “Data security infrastructure”).

3.5.3. Archive level Release files

[Tip] Tip

The archive level Release files are used for the rule of apt_preferences(5).

There are archive level Release files for all archive locations specified by "deb:" line in /etc/apt/sources.list, such as "ftp://ftp.us.debian.org/debian/dists/unstable/main/binary-amd64/Release" or "ftp://ftp.us.debian.org/debian/dists/sid/main/binary-amd64/Release":

Archive: unstable
Component: main
Origin: Debian
Label: Debian
Architecture: amd64
[Caution] Caution

For Archive stanza, suite names ("stable", "testing", "unstable", ...) are used in the Debian archive while codenames ("dapper", "feisty", "gutsy", "hardy", "intrepid", ...) are used in the Ubuntu archive.

For some archives, such as "experimental", "volatile-sloppy", and "lenny-backports", which contain packages which should not be installed automatically, there is an extra line, e.g., "ftp://ftp.us.debian.org/debian/dists/experimental/main/binary-amd64/Release":

Archive: experimental
Component: main
Origin: Debian
Label: Debian
NotAutomatic: yes
Architecture: amd64

Please note that for normal archives without "NotAutomatic: yes", the default Pin-Priority value is 500, while for special archives with "NotAutomatic: yes", the default Pin-Priority value is 1 (see apt_preferences(5) and Section 3.7.3, “Tweaking candidate version”).

3.5.4. Fetching of the meta data for the package

When APT tools, such as aptitude, apt-get, synaptic, apt-file, auto-apt..., are used, we need to update the local copies of the meta data containing the Debian archive information. These local copies have file names corresponding to the specified distribution component and architecture names in the /etc/apt/sources.list (see: Section 3.1.4, “Debian archive basics”) under the /var/lib/apt/lists directory as:

  • ftp.us.debian.org_debian_dists_<distribution>_Release

  • ftp.us.debian.org_debian_dists_<distribution>_Release.gpg

  • ftp.us.debian.org_debian_dists_<distribution>_<section>_binary-<architecture>_Packages

  • ftp.us.debian.org_debian_dists_<distribution>_<section>_source_Sources

  • /var/cache/apt/apt-file/ftp.us.debian.org_debian_dists_<distribution>_Contents-<architecture>.gz (for apt-file)

First 4 are shared by all the pertinent APT commands and updated from command line by "apt-get update" and "aptitude update". The Packages meta data are updated if there is the deb line in /etc/apt/sources.list. The Sources meta data are updated if there is the deb-src line in /etc/apt/sources.list.

The Packages and Sources meta data contain "Filename:" stanza pointing to the file location of the binary and source packages. Currently, they are located under the pool/ directory tree for the improved transition over the releases.

The local copies of Packages meta data can be interactively searched with the help of aptitude. The specialized search command grep-dctrl(1) can search the local copies of Packages and Sources meta data.

The local copyies of Contents-<architecture> files can be updated by "apt-file update" and location is different from other 4 files. See apt-file(1). (The auto-apt uses different location for local caching of Contents-<architecture>.gz as default.)

3.5.5. The package state for APT

In addition to the remotely fetched meta data, the APT tool after "lenny" stores its locally generated installation state information in the /var/lib/apt/extended_states which is used only by all APT tools to track all auto installed packages.

3.5.6. The package state for aptitude

In addition to the remotely fetched meta data, the aptitude command stores its locally generated installation state information in the /var/lib/aptitude/pkgstates which is used only by it.

3.5.7. The local copies of the fetched packages

All the remotely fetched packages via APT mechanism are stored in the /var/cache/apt/packages until they are cleaned.

3.5.8. The Debian package file name

The Debian package files has particular name structures:

Table 3.16.  The name structures of the Debian packages.

entity

name structure

The binary package (a.k.a deb)

<package-name>_<epoch>:<upstream-version>-<debian.version>-<architecture>.deb

The binary package for the debian-installer (a.k.a udeb)

<package-name>_<epoch>:<upstream-version>-<debian.version>-<architecture>.udeb

The source package (upstream source)

<package-name>_<epoch>:<upstream-version>-<debian.version>.tar.gz

The source package (Debian changes)

<package-name>_<epoch>:<upstream-version>-<debian.version>.diff.gz

The source package (description)

<package-name>_<epoch>:<upstream-version>-<debian.version>.dsc


where,

Table 3.17.  The usable characters for each component in the Debian package names.

component

usable characters (regex)

required

<package-name>

[a-z,A-Z,0-9,.,+,-]+

required

<epoch>:

[0-9]+:

optional

<upstream-version>

[a-z,A-Z,0-9,.,+,-,:]+

required

<debian.version>

[a-z,A-Z,0-9,.,+,~]+

optional


[Note] Note

You can check package version order by dpkg(1), e.g., "dpkg --compare-versions 7.0 gt 7.~pre1 ; echo $?" .

[Note] Note

The debian-installer (d-i) uses udeb as the file extension for its binary package instead of normal deb. An udeb package is a stripped down deb package which removes few non-essential contents such as documentation to save space while relaxing the package policy requirements. Both deb and udeb package share the same package structure. The "u" stands for micro.

3.5.9. The dpkg command

The dpkg is the lowest level tool for the Debian package management. This is very powerful and needs to be used with care.

The fetched package is processed by dpkg in the following order:

  1. unpack the deb file ("ar -x" equivalent)

  2. preinst using debconf

  3. install the package content to the system ("tar -x" equivalent)

  4. postinst using debconf

The debconf system provides standardized user interaction with i18n and l17n supports.

Here dpkg creates following files under /var/lib/dpkg/info/ directory : While installing package called <package_name>, dpkg creates several files and execute scripts.

Table 3.18.  The notable files for dpkg.

file

contents

/var/lib/dpkg/info/<package_name>.conffiles

list of user modifiable files.

/var/lib/dpkg/info/<package_name>.list

list of files and directories installed by the package.

/var/lib/dpkg/info/<package_name>.md5sums

list of MD5 hash values for files installed by the package.

/var/lib/dpkg/info/<package_name>.preinst

package script run before the package installation.

/var/lib/dpkg/info/<package_name>.postinst

package script run after the package installation.

/var/lib/dpkg/info/<package_name>.prerm

package script run before the package removal.

/var/lib/dpkg/info/<package_name>.postrm

package script run after the package removal.

/var/lib/dpkg/info/<package_name>.config

package script for debconf system.

/var/lib/dpkg/alternatives/<package_name>

the alternative information used by the update-alternatives command.

/var/lib/dpkg/available

the availability information for all the package.

/var/lib/dpkg/diversions

the diversions information used by the dpkg-divert command.

/var/lib/dpkg/status

the status information for all the packages.

/var/lib/dpkg/status-old

the first-generation backup of the var/lib/dpkg/status file.

/var/backups/dpkg.status*

the second-generation backup and older ones of the var/lib/dpkg/status file.


The last file status is also used by the tools such as "dpkg", "select update" and "apt-get -u dselect-upgrade".

The specialized search command grep-dctrl(1) can search the local copies of status and available meta data.

[Tip] Tip

In the debian-installer environment, the udpkg command is used to open udeb packages. The udpkg is a stripped down version of dpkg command.

3.5.10. The update-alternative command

The Debian system has mechanism to install somewhat overlapping programs peacefully using update-alternatives(8). For example, to make the command vi select to run vim while installing both vim and nvi:

$ ls -l $(type -p vi)
lrwxrwxrwx 1 root root 20 2007-03-24 19:05 /usr/bin/vi -> /etc/alternatives/vi
$ sudo update-alternatives --display vi
...
$ sudo update-alternatives --config vi
  Selection    Command
-----------------------------------------------
      1        /usr/bin/vim
*+    2        /usr/bin/nvi

Enter to keep the default[*], or type selection number: 1

The Debian alternatives system keeps its selection as symlinks in /etc/alternatives/. The selection process uses corresponding file in /var/lib/dpkg/alternatives/.

3.5.11. The dpkg-statoverride command

Stat overrides provided by the dpkg-statoverride(8) command are a way to tell dpkg(1) to use a different owner or mode for a file when a package is installed. If "--update" is specified and file exists, it is immediately set to the new owner and mode.

[Note] Note

I use the word file here, but in reality this can be any filesystem object that dpkg handles, including directories, devices, etc.

[Caution] Caution

The direct alteration of owner or mode for a file owned by the package using chmod or chown commands by the system administrator will be reset by the next upgrade of the package.

3.5.12. The dpkg-divert command

File diversions provided by the dpkg-divert(8) command are a way of forcing dpkg(1) not to install a file into its default location, but to a diverted location. The use of dpkg-divert is meant for the package maintenance scripts. Its use by the system administrator is deprecated.

3.6. Recovery from a broken system

When running "unstable" system, the administrator is expected to recover from broken package management situation.

[Caution] Caution

Some methods described here are high risk actions. You have been warned!

3.6.1. Incompatibility with old user configuration

If a desktop GUI program experienced instability after significant upstream version upgrade, you should suspect interferences with old local configuration files created by it. If it is stable under newly created user account, this hypothesis is confirmed. (This is a bug of packaging and usually avoided by the packager.)

To recover stability, you should move corresponding local configuration files and restart the GUI program. You may need to read old configuration file contents to recover configuration information later. (Do not erase them too quickly.)

3.6.2. Different packages with overlapped files

Archive level package management systems, such as aptitude(8) or apt-get(1), will not even try to install packages with overlapped files using package dependencies (see Section 3.1.5, “Package dependencies”).

Errors by the package maintainer or deployment of inconsistently mixed source of archives (see Section 3.7.2, “Packages from mixed source of archives”) by the system administrator may create situation with incorrectly defined package dependencies. When you install a package with overlapped files using aptitude(8) or apt-get(1) under such situation, dpkg(1) which unpacks package ensures to return error to the calling program without overwriting existing files.

[Caution] Caution

The use of third party packages introduces significant system risks via maintainer scripts which are run with root privilege and can do anything to your system. The dpkg(1) command only protects against overwriting by the unpacking.

You can work around such broken installation by removing the old offending package, <old-package>, first:

$ sudo dpkg -P <old-package>

3.6.3. Fixing broken package script

When a command in the package script returns error for some reason and the script exits with error, the package management system aborts their action and ends up with partially installed packages. When a package contains bugs in its removal scripts, the package may become impossible to remove and quite nasty.

For the package script problem of "<package_name>", you should look for:

  • /var/lib/dpkg/info/<package_name>.preinst

  • /var/lib/dpkg/info/<package_name>.postinst

  • /var/lib/dpkg/info/<package_name>.prerm

  • /var/lib/dpkg/info/<package_name>.postrm

You edit the offending part of the script from the root:

  • to prepend with ": #" or,

  • to append with "|true".

Then configures all partially installed packages by:

# dpkg --configure -a

3.6.4. Rescue using the dpkg command

Since dpkg is very low level package tool, it can function under the very bad situation such as unbootable system without network connection. Let's assume foo package was broken and needs to be replaced.

You may still find cached copies of older version of foo package in /var/cache/apt/archives/ which is bug free. (If not, you can download it from archve of http://snapshot.debian.net/ or copy it from package cache of a functioning machine.)

If you can boot the system, you may install it by:

# dpkg -i /path/to/foo_<old_version>_<arch>.deb
[Tip] Tip

If system breakage is minor, you may alternatively downgrade the whole system as Section 3.7.5, “Emergency downgrading” using the higher level APT system.

If your system is unbootable from harddisk, you should seek other ways to boot it. For example, you can:

  • boot the system using the debian-installer CD in rescue mode,

  • mount the unbootable system on the harddisk to /target,

  • install older version of foo package by:

# dpkg --root /target -i /path/to/foo_<old_version>_<arch>.deb

This second example works even if the dpkg command on the harddisk is broken.

[Tip] Tip

Any linux system started by another system on harddisk, live linux CD, bootable USB-key drive, or netboot can be used similarly to rescue broken system.

If attempting to install a package this way fails due to some dependency violations and you really need to do this as the last resort, you can override dependency using dpkg's --ignore-depends, --force-depends and other options. If you do this, you need to make serious effort to restore proper dependency later. See dpkg(8) for details.

[Note] Note

When your system is seriously broken, you should make a full backup of system to a safe place (see Section 11.1.6, “Backup and recovery”) and should perform a clean installation. This is less time consuming and produces better results in the end.

3.6.5. Recover package selection data

If /var/lib/dpkg/status becomes corrupt for any reason, the Debian system loses package selection data and suffers severely. Look for the old /var/lib/dpkg/status file at /var/lib/dpkg/status-old or /var/backups/dpkg.status.*.

Keeping /var/backups/ in a separate partition may be a good idea since this directory contains lots of important system data.

If everything is gone, I recommend to make fresh re-install after making backup of the system in such serious breakage though. You can still recover some information from directories in /usr/share/doc/ to guide your new installation.

... reinstall minimal (desktop) system
... place old system at /path/to/old/system/
# cd /path/to/old/system/usr/share/doc
# ls -1 >~/ls1.txt
# cd /usr/share/doc
# ls -1 >>~/ls1.txt
# cd
# sort ls1.txt | uniq | less

Then you will be presented with missing package names. (There may be some non-package names such as "texmf".)

3.7. Tips for the package management

3.7.1. How to pick Debian packages

You can seek packages which satisfy your needs with aptitude from the package description or from the list under "Tasks".

When you encounter more than 2 similar packages and wonder which one to install without "trial and error" efforts, you can use some common sense. I consider following points are good indications of preferred packages.

  • essential: yes > no

  • component: main > contrib > non-free

  • priorities: required > important > standard > optional > extra

  • task: package listed in task such as desktop

  • package selected by the dependency package (e.g., python2.4 by python)

  • popcon: higher in the vote and install number

  • changelog: regular updates by the maintainer

  • BTS: No RC bugs (no critical, no grave, and no serious bugs)

  • BTS: responsive maintainer to bug reports

  • BTS: higher number of the recently fixed bugs

  • BTS: lower number of remaining non-wishlist bugs

Debian being a volunteer project with distributed development model, its archive contains many packages with different focus and quality. You must make your own decision what to do with them.

3.7.2. Packages from mixed source of archives

[Caution] Caution

Installing packages from mixed source of archives is not supported by the official Debian distribution except for officially supported particular combinations of archives such as "stable" with security updates and volatile updates.

Here is an example of operations to include specific newer upstream version packages found in "unstable" while tracking "testing" for single occasion:

  • change the /etc/apt/sources.list file temporarily to single "unstable" entry

  • run "aptitude update"

  • run "aptitude install <package-name>"

  • recover the original /etc/apt/sources.list file for "testing"

  • run "aptitude update"

You do not create the /etc/apt/preferences file nor need to worry about apt-pinning with this manual approach. But this is very cumbersome.

[Caution] Caution

When using mixed source of archives, you must ensure compatibility of packages by yourself since the Debian does not guarantee it. If package incompatibility exists, you may break system. You must be able to judge these technical requirements. The use of mixed source of random archives is completely optional operation and its use is not something I encourage you to use.

General rules for installing packages from different archives are:

  • non-binary packages ("Architecture: all") are safer to install.

    • documentation packages: no special requirements

    • interpreter program packages: compatible interpreter must be available

  • completely statically linked binary packages are safe to install.

  • binary packages (non "Architecture: all") usually face many road blocks and unsafe to install.

    • library version compatibility (including "libc")

    • related utility program version compatibility

    • Kernel ABI compatibility

    • C++ ABI compatibility

    • ...

[Note] Note

Except to avoid broken package for a short term, installing binary packages from officially unsupported archives is generally bad idea. This is true even if you use apt-pinning (see Section 3.7.3, “Tweaking candidate version”). You should consider chroot or similar techniques (see Section 10.8, “The chroot”) to run programs from different archives.

3.7.3. Tweaking candidate version

Without the /etc/apt/preferences file, APT system choses the latest available version as the candidate version using the version string. This is the normal state and most recommended usage of APT system. All officially supported combinations of archives do not require the /etc/apt/preferences file since some archives which should not be used as the automatic source of upgrades are marked as NotAutomatic and dealt properly.

[Tip] Tip

The version string comparison rule can be verified with, e.g., "dpkg --compare-versions ver1.1 gt ver1.1~1; echo $?" (see dpkg(1)).

When you install packages from mixed source of archives (see Section 3.7.2, “Packages from mixed source of archives”) regularly, you can automate these complicated operations by creating the /etc/apt/preferences file with proper entries and tweaking the package selection rule for candidate version as described in apt_preferences(5). This is called apt-pinning.

[Warning] Warning

Use of apt-pinning by a novice user is sure call for major troubles. You must avoid using apt-pinning except when you absolutely need it.

[Caution] Caution

When using apt-pinning, you must ensure compatibility of packages by yourself since the Debian does not guarantee it. The apt-pinning is completely optional operation and its use is not something I encourage you to use.

[Caution] Caution

Archive level Release files (see Section 3.5.3, “Archive level Release files”) are used for the rule of apt_preferences(5). Thus apt-pinning works only with "suite" name for normal Debian archives and security Debian archives. (This is different from Ubuntu archives). For example, you can do "Pin: release a=unstable" but can not do "Pin: release a=sid" in the /etc/apt/preferences file.

[Note] Note

Even if you do not create the /etc/apt/preferences file, you can do fairly complex system operations (see Section 3.6.4, “Rescue using the dpkg command” and Section 3.7.2, “Packages from mixed source of archives”) without apt-pinning.

Here is a simplified explanation of apt-pinning technique. Each package has its Pin-Priority value based on the entries in the /etc/apt/preferences file or the default values.

Table 3.19.  List of essential default Pin-Priority values.

Pin-Priority

description

990

default value for package from the target release archive

500

default value for package from the normal archive

100

default value for package from the installed package

1

default value for package from the NotAutomatic archive


APT system normally choses highest Pin-Priority upgrading package from the available resources defined in the /etc/apt/sources.list file as the candidate version. This version restriction for upgrading is dropped to enable downgrading if the Pin-Priority of package is larger than 1000 (see Section 3.7.5, “Emergency downgrading”).

The target release can be set:

  • by /etc/apt/apt.conf, e.g., "APT::Default-Release "stable";" line in it, or

  • by "-t" option argument, e.g., "apt-get install -t testing some-package".

The archive level Release file (see Section 3.5.3, “Archive level Release files”) of NotAutomatic archive contains "NotAutomatic: yes".

[Caution] Caution

Although cryptic to read, you should test Pin-Priority situation of your /etc/apt/preferences file using the output of "apt-cache policy <package>".

The Pin-Priority values of <package> from multiple sources are shown by the output of "apt-cache policy <package>":

  • a line started with "Package pin:" lists the package version of pin if association just with <package> is defined, e.g., "Package pin: 0.190",

  • no line with "Package pin:" exists if no association just with <package> is defined,

  • the Pin-Priority value associated just with <package> is listed right side of all version strings, e.g., "0.181 700",

  • 0 is listed right side of all version strings if no association just with <package> is defined, e.g., "0.181 0", and

  • the Pin-Priority values of archives (defined as "Package: *" in the /etc/apt/preferences file) are listed left side of all archive paths, e.g., "200 http://backports.org etch-backports/main Packages".

Here is an example of apt-pinning technique to include specific newer upstream version packages found in "unstable" regularly upgraded while tracking "testing". You list all required archives in the /etc/apt/sources.list file as:

deb http://ftp.us.debian.org/debian/ testing main contrib non-free
deb http://ftp.us.debian.org/debian/ unstable main contrib non-free
deb http://security.debian.org/ testing/updates main contrib

and set the /etc/apt/preferences file as:

Package: *
Pin: release a=testing
Pin-Priority: 500

Package: *
Pin: release a=unstable
Pin-Priority: 200

When you wish to install a package named "<package-name>" with its dependencies from "unstable" archive under this configuration, you issue the following command which switches target release with "-t" option (Pin-Priority of "unstable" becomes 990.):

$ sudo aptitude install -t unstable <package-name>

With this configuration, usual execution of "aptitude upgrade" and "aptitude dist-upgrade" will upgrade packages which were installed from "testing" archive using current "testing" archive and packages which were installed from "unstable" archive using current "unstable" archive.

[Caution] Caution

Be careful not to remove "testing" entry from the /etc/apt/sources.list file. Without "testing" entry in it, APT system will upgrade packages using newer "unstable" archive.

[Tip] Tip

I usually edit the /etc/apt/sources.list file to comment out "unstable" archive entry right after above operation. This avoids slow update process of having too many entries in the /etc/apt/sources.list file although this prevents upgrading packages which were installed from "unstable" archive using current "unstable" archive.

[Tip] Tip

If "Pin-Priority: 20" is used instead of "Pin-Priority: 200" for the /etc/apt/preferences file, already installed packages having Pin-Priority value of 100 will never be upgraded by "unstable" archive even if "testing" entry in the /etc/apt/sources.list file is removed.

If you wish to track particular packages in "unstable" automatically without initial "-t unstable" installation, you must create the /etc/apt/preferences file and explicitly lists all those packages at the top of it as:

Package: <package-1>
Pin: release a=unstable
Pin-Priority: 700

Package: <package-2>
Pin: release a=unstable
Pin-Priority: 700

...

These will set Pin-Priority value for each specific package. For example, in order to track the latest "unstable" version of this "Debian Reference" in English, you should have following entries in the /etc/apt/preferences file:

Package: debian-reference-en
Pin: release a=unstable
Pin-Priority: 700

Package: debian-reference-common
Pin: release a=unstable
Pin-Priority: 700
[Tip] Tip

This apt-pinning technique is valid even when you are tracking "stable" archive. Documentation packages have been always safe to install from "unstable" archive in my experience. (Reminder: This multi-archive tracking is not officially supported feature.)

Here is another example of apt-pinning technique to include specific newer upstream version packages found in "experimental" while tracking "unstable". You list all required archives in the /etc/apt/sources.list file as:

deb http://ftp.us.debian.org/debian/ unstable main contrib non-free
deb http://ftp.us.debian.org/debian/ experimental main contrib non-free
deb http://security.debian.org/ testing/updates main contrib

The default Pin-Priority value for "experimental" archive is always 1 since it is NotAutomatic archive (see Section 3.5.3, “Archive level Release files”). There is no need to set Pin-Priority value in the /etc/apt/preferences file to use experimental archive unless you wish to track particular packages automatically.

3.7.4. Volatile and Backports.org

There are The debian-volatile Project and http://backports.org archives which provide updgrade packages for "stable".

[Warning] Warning

Do not use all packages available in the NotAutomatic archives such as "lenny-backports" and "volatile-sloppy". Use only selected packages which fits your needs.

[Caution] Caution

http://backports.org is a non-Debian archive, although its packages are signed by Debian developers.

[Caution] Caution

Archive level Release files (see Section 3.5.3, “Archive level Release files”) are used for the rule of apt_preferences(5). Thus apt-pinning works only with "code" name for volatile Debian archives. This is different from other Debian archives. For example, you can do "Pin: release a=lenny" but can not do "Pin: release a=stable" in the /etc/apt/preferences file for volatile Debian archives.

Here is an example of apt-pinning technique to include specific newer upstream version packages found in "lenny-backports" while tracking "lenny" and "volatile". You list all required archives in the /etc/apt/sources.list file as:

deb http://ftp.us.debian.org/debian/ lenny main contrib non-free
deb http://security.debian.org/ lenny/updates main contrib
deb http://volatile.debian.org/debian-volatile/ lenny/volatile main contrib non-free
deb http://volatile.debian.org/debian-volatile/ lenny/volatile-sloppy main contrib non-free
deb http://backports.org/debian/ lenny-backports main contrib non-free

The default Pin-Priority value for http://backports.org archive and "volatile-sloppy" are always 1 since they are NotAutomatic archive (see Section 3.5.3, “Archive level Release files”). There is no need to set Pin-Priority value explicitly in the /etc/apt/preferences file just to use for http://backports.org and "volatile-sloppy" archive unless you wish to track packages automatically for next upgrading.

So whenever you wish to install a package named "<package-name>" with its dependency from "lenny-backports" archive, you use following command while switching target release with "-t" option:

$ sudo aptitude install -t lenny-backports <package-name>

If you wish to upgrade particular packages, you must create the /etc/apt/preferences file and explicitly lists all packages in it as:

Package: <package-1>
Pin: release o=Backports.org archive
Pin-Priority: 700

Package: <package-2>
Pin: release o=volatile.debian.org
Pin-Priority: 700

...

Alternatively, with the /etc/apt/preferences file as:

Package: *
Pin: release a=stable , o=Debian
Pin-Priority: 500

Package: *
Pin: release a=lenny, o=volatile.debian.org
Pin-Priority: 500

Package: *
Pin: release a=lenny-backports, o=Backports.org archive
Pin-Priority: 200

Package: *
Pin: release a=lenny-sloppy, o=volatile.debian.org
Pin-Priority: 200

execution of "aptitude upgrade" and "aptitude dist-upgrade" will upgrade packages which were installed from "stable" archive using current "stable" archive and packages which were installed from other archives using current corresponding archive for all archives in the /etc/apt/sources.list file.

3.7.5. Emergency downgrading

[Caution] Caution

Downgrading is not officially supported by the Debian by design. It should be done only as a part of emergency recovery process. Despite of this situation, it is known to work well in many incidents. For critical systems, You should backup all important data on the system after the recovery operation and re-install the new system from the scratch.

You may be lucky to downgrade from newer archive to older archive to recover from broken system upgrade by manipulating candidate version (see: Section 3.7.3, “Tweaking candidate version”). This is lazy alternative to tedious actions of many "dpkg -i <broken-package>_<old-version>.deb" commands (see Section 3.6.4, “Rescue using the dpkg command”).

For downgrading system tracking "unstable" to "testing", change the /etc/apt/sources.list file from:

deb http://ftp.us.debian.org/debian/ sid main contrib non-free

to:

deb http://ftp.us.debian.org/debian/ squeeze main contrib

and set the /etc/apt/preferences file as:

Package: *
Pin: release a=testing
Pin-Priority: 1010

Then run "aptitude dist-upgrade" to force downgrading of packages across the system. You should remove this special /etc/apt/preferences file after the downgrading.

3.7.6. Who uploaded the package?

Although the maintainer name listed in /var/lib/dpkg/available and /usr/share/doc/package_name/changelog provide some information on "who is behind the packaging activity", the actual uploader of the package is somewhat obscure. The who-uploads(1) in devscripts package identifies the actual uploader of Debian source packages.

3.7.7. The equivs package

If you are to compile a program from source to replace the Debian package, it is best to make it into a real local debianized package (*.deb) and use private archive.

If you chose to compile a program from source and to install them under /usr/local instead, you may need to use equivs as a last resort to satisfy the missing package dependency.

Package: equivs
Priority: extra
Section: admin
Description: Circumventing Debian package dependencies
 This is a dummy package which can be used to create Debian
 packages, which only contain dependency information.

3.7.8. Port a package to the stable system

For partial upgrades of the "stable" system, rebuilding a package within its environment using the source package is desirable. This avoids massive package upgrades due to their dependencies. First, add the following entries to the /etc/apt/sources.list of a "stable" system:

deb-src http://http.us.debian.org/debian unstable  main contrib non-free

Then get the required packages and the source to be downloaded and compiled by:

# aptitude update
# aptitude dist-upgrade
# aptitude install fakeroot devscripts build-essential
$ apt-get build-dep foo
$ apt-get source foo
$ cd foo*
  • adjust package if needed.

$ dch -i
  • bump package version, e.g. one appended with "+bp1".

$ debuild
$ cd ..
# debi foo*.changes 

3.7.9. Proxy server for APT

Since mirroring whole subsection of Debian archive wastes disk space and network bandwidth, deployment of a local proxy server for APT is desirable consideration when you administer many systems on LAN. APT can be configure to use web (http) proxy server such as squid (see Section 7.5, “Other network application servers”) as described in apt.conf(5) and in /usr/share/doc/apt/examples/configure-index.gz. The http_proxy environment variable can be used to override proxy server setting in the /etc/apt/apt.conf file.

There are proxy tools specially for Debian archive. You should check BTS before using them.

Table 3.20.  List of the proxy tools specially for Debian archive

package

popcon

size

description

URL links

approx

V:0.2, I:0.2

3632

caching proxy server for Debian archive files (compiled OCaml program)

BTS for approx

apt-proxy

V:0.5, I:0.6

428

Debian archive proxy and partial mirror builder (Python program)

BTS for apt-proxy

apt-cacher

V:0.3, I:0.6

312

Caching proxy for Debian package and source files (Perl program)

BTS for apt-cacher

apt-cacher-ng

V:0.09, I:0.13

680

aching proxy for distribution of software packages (compiled C++ program)

BTS for apt-cacher-ng

debtorrent

V:0.15, I:0.2

1173

bittorrent proxy for downloading Debian packages (Python program)

BTS for debtorrent


3.7.10. Small public package archive

Here is an example for creating a small public package archive compatible with the modern secure APT system (see Section 3.5.2, “Top level Release file and authenticity”). Let's assume few things:

  • Account name: foo

  • Host name: www.example.com

  • Required packages: apt-utils, gnupg, and other packages.

  • URL: http://www.example.com/~foo/ displays /home/foo/public_html/index.html

  • Architecture of packages: amd64

One time setup of APT archive on your server system:

  • Create Foo's archive key on server system:

$ ssh foo@www.example.com
$ gpg --gen-key
...
$ gpg -K
...
sec   1024D/3A3CB5A6 2008-08-14
uid                  Foo Bar (ARCHIVE KEY) <foo@www.example.com>
ssb   2048g/6856F4A7 2008-08-14
$ gpg --export -a 3A3CB5A6 >foo.public.key
  • New Foo's archive key is 3A3CB5A6

  • Publish foo.public.key file.

  • Create Foo's archive skeleton:

$ umask 022
$ mkdir -p ~/public_html/debian/pool/main
$ mkdir -p ~/public_html/debian/dists/unstable/main/binary-amd64
$ mkdir -p ~/public_html/debian/dists/unstable/main/source
$ cd ~/public_html/debian
$ cat > dists/unstable/main/binary-amd64/Release << EOF
Archive: unstable
Version: 4.0
Component: main
Origin: Foo
Label: Foo
Architecture: amd64
EOF
$ cat > dists/unstable/main/source/Release << EOF
Archive: unstable
Version: 4.0
Component: main
Origin: Foo
Label: Foo
Architecture: source
EOF
$ cat >aptftp.conf <<EOF
APT::FTPArchive::Release {
  Origin "Foo";
  Label "Foo";
  Suite "unstable";
  Codename "sid";
  Architectures "amd64";
  Components "main";
  Description "Foo's public archive";
};
EOF
$ cat >aptgenerate.conf <<EOF
Dir::ArchiveDir ".";
Dir::CacheDir ".";
TreeDefault::Directory "pool/";
TreeDefault::SrcDirectory "pool/";
Default::Packages::Extensions ".deb";
Default::Packages::Compress ". gzip bzip2";
Default::Sources::Compress "gzip bzip2";
Default::Contents::Compress "gzip bzip2";

BinDirectory "dists/unstable/main/binary-amd64" {
  Packages "dists/unstable/main/binary-amd64/Packages";
  Contents "dists/unstable/Contents-amd64";
  SrcPackages "dists/unstable/main/source/Sources";
};

Tree "dists/unstable" {
  Sections "main";
  Architectures "amd64 source";
};
EOF

Repetitive update of APT archive contents on your server system:

  • Place all package files into ~foo/public_html/debian/pool/main/ by executing "dupload -t foo changes_file" in client while having ~/.dupload.conf containing:

$cfg{'foo'} = {
  fqdn => "www.example.com",
  method => "scpb",
  incoming => "/home/foo/public_html/debian/pool/main",
  # The dinstall on ftp-master sends emails itself
  dinstall_runs => 1,
};

$cfg{'foo'}{postupload}{'changes'} = "
  echo 'cd public_html/debian ;
  apt-ftparchive generate -c=aptftp.conf aptgenerate.conf;
  apt-ftparchive release -c=aptftp.conf dists/unstable >dists/unstable/Release ;
  rm -f dists/unstable/Release.gpg ;
  gpg -u 3A3CB5A6 -bao dists/unstable/Release.gpg dists/unstable/Release'| 
  ssh foo@www.example.com  2>/dev/null ;
  echo 'Package archive created!'";

The postupload hook script initiated by dupload(1) creates updated archive files for each upload.

You can add this small public archive to the apt-line of your client system:

$ sudo bash
# echo "deb http://www.example.com/~foo/debian/ unstable main" \
   >> /etc/apt/sources.list
# apt-key add foo.public.key
[Tip] Tip

If the archive is located on the local file system, you can use "deb file:///home/foo/debian/ ..." instead.

3.7.11. Record/copy system configuration

To make a local copy of the package and debconf selection states:

# dpkg --get-selections '*' > selection.dpkg
# debconf-get-selections    > selection.debconf

Here, '*' makes selection.dpkg to include package entries for "purge" too.

You can transfer these 2 files to another computer, and install there with:

# dselect update
# debconf-set-selections < myselection.debconf
# dpkg --set-selections  < myselection.dpkg
# apt-get -u dselect-upgrade    # or dselect install

If you are thinking about managing many cluster of servers with practically the same configuration, you should consider to use specialized package such as fai to manage the whole system.

3.7.12. Convert or install an alien binary package

The alien command enables the conversion of binary packages provided in Red Hat rpm, Stampede slp, Slackware tgz, and Solaris pkg file formats into a Debian deb package. If you want to use a package from another Linux distribution than the one you have installed on your system, you can use alien to convert it to your preferred package format and install it. alien also supports LSB packages.

The original package needs to be statically linked or its library dependency needs to be satisfied manually. So use this command with great care.

3.7.13. Extract package without dpkg

The current .deb package contents can be extracted without using the dpkg command on any unix-like environment using standard ar and tar commands.

# ar x /path/to/dpkg_<version>_<arch>.deb
# ls
total 24
-rw-r--r-- 1 osamu osamu  1320 2007-05-07 00:11 control.tar.gz
-rw-r--r-- 1 osamu osamu 12837 2007-05-07 00:11 data.tar.gz
-rw-r--r-- 1 osamu osamu     4 2007-05-07 00:11 debian-binary
# mkdir control
# mkdir data
# tar xvzf control.tar.gz -C control
# tar xvzf data.tar.gz -C data

You can also browse package content using the mc command.

3.7.14. More readings for the package management

You should read:

  • manpages for aptitude(8), "dpkg", "man 8 tasksel", "man 8 apt-get", "man 8 apt-config", "man 8 apt-key", "man 5 sources.list", "man 5 apt.conf", and "man 5 apt_preferences";

  • "/usr/share/doc/apt-doc/guide.html/index.html" and "/usr/share/doc/apt-doc/offline.html/index.html" from the apt-doc package;

  • Read /usr/share/doc/aptitude/html/en/index.html from the aptitude-doc-en package.

The official and detailed secondary information on the Debian archive are given by:

The tutorial for building of a Debian package for the common Debian user is given by:

Chapter 4. The system initialization

It is wise for you as the system administrator to know roughly how the Debian system is started and configured. Although the exact details are in the source files of the packages installed and their documentations, it is a bit overwhelming for most of us.

I did my best to provide a quick overview of the key points of the Debian system and their configuration for your reference, based on the current and previous knowledge of mine and others. Since the Debian system is a moving target, the situation over the system may have been changed. Before making any changes to the system, you should refer to the latest documentation for each package.

4.1. An overview of the boot strap process

The computer system undergoes several phases of boot strap processes from the power-on event until it offers the fully functional operating system (OS) to the user.

For simplicity, I will limit discussion to the typical PC platform with the default installation.

The typical boot strap process is like a four-stage rocket. Each stage rocket hands over the system control to the next stage one. Here each stage corresponds to:

  • Stage 1: the BIOS

  • Stage 2: the boot loader

  • Stage 3, the mini-Debian system

  • Stage 4: the normal Debian system

Of course, these can be configured differently. For example, if you compiled your own kernel, you may be skipping the step with the mini-Debian system. So please do not assume this is the case for your system until you check it yourself.

[Note] Note

For non-legacy PC platform such as the SUN or the Macintosh system, the BIOS on ROM and the partition on the disk may be quite different (Section 10.3.1, “Partition configuration”). Please seek the platform specific documentations elsewhere for such a case.

4.2. Stage 1: the BIOS

The BIOS is the 1st stage of the boot process which is started by the power-on event. The BIOS resides on the read only memory (ROM) from the particular memory address to which the program counter of CPU is initialized by the power-on event.

This BIOS performs the basic initialization of the hardware (POST: power on self test) and hands the system control to the next step which you provide. The BIOS is usually provided with the hardware.

The BIOS startup screen usually indicates what key(s) to press to enter the BIOS setup screen to configure the BIOS behavior. Popular keys used are F1, F2, F10, Esc, Ins, and Del. If your BIOS startup screen is hidden by a nice graphics screen, you may press some keys such as Esc to disable this. These keys are highly dependent on the hardware.

The hardware location and the priority of the code started by the BIOS can be selected from the BIOS setup screen. Typically, the first few sectors of the first found selected device (hard disk, floppy disk, CD-ROM, ...) are loaded to the memory and this initial code is executed. This initial code can be:

  • the boot loader code,

  • the kernel code of the stepping stone OS such as FreeDOS, or

  • the kernel code of the target OS if it fits in this small space.

Typically, the system is booted from the specified partition of the primary hard disk partition. The first sector of the hard disk contain the master boot record (MBR). The disk partition information including the boot selection is recorded at the end of this MBR. The first boot loader code executed from the BIOS for the hard disk occupies the rest of this MBR.

4.3. Stage 2: the boot loader

The boot loader is the 2nd stage of the boot process which is started by the BIOS. It loads the system kernel image and the initrd image to the memory and hands control over to them. This initrd image is the root filesystem image and its support depends on the bootloader used.

The Debian system normally uses the Linux kernel as the default system kernel. The initrd image for the current 2.6 Linux kernel is technically the initramfs (initial RAM filesystem) image. The initramfs image is a gzipped cpio archive of files in the root filesystem.

The default install of the Debian system places first-stage GRUB boot loader code into the MBR for the PC platform. There are many boot loaders and configuration options available.

Table 4.1.  List of boot loaders.

bootloader

package

popcon

size

initrd

description

GRUB Legacy

grub

V:18, I:91

1884

Supported

This is smart enough to understand disk partitions and file systems such as vfat, ext3, .... (etch default)

GRUB 2

grub-pc

V:0.9, I:1.9

3952

Supported

This is smart enough to understand disk partitions and file systems such as vfat, ext3, .... (new for lenny)

GRUB 2

grub-rescue-pc

V:0.05, I:0.4

2476

Supported

This is GRUB 2 bootable rescue images (CD and floppy) (PC/BIOS version)

Lilo

lilo

V:1.0, I:4

1192

Supported

This relies on the sector locations of data on the hard disk. (Old)

Isolinux

syslinux

V:1.4, I:7

144

Supported

This understands the ISO9660 file system. This is used by the boot CD.

Syslinux

syslinux

V:1.4, I:7

144

Supported

This understands the MSDOS file system (FAT). This is used by the boot floppy.

Loadlin

loadlin

V:0.02, I:0.12

140

Supported

New system is started from the FreeDOS/MSDOS system.

Neil Turton's MBR

mbr

V:1.3, I:7

96

Not supported

This is free software which substitutes MSDOS MBR. This only understands disk partitions.


For GRUB Legacy, the menu configuration file is located at /boot/grub/menu.lst . For example, it has entries like:

title           Debian GNU/Linux
root            (hd0,2)
kernel          /vmlinuz root=/dev/hda3 ro
initrd          /initrd.img

For GRUB 2, the menu configuration file is located at /boot/grub/grub.cfg. It is automatically generated by /usr/sbin/update-grub using templates from /etc/grub.d/* and settings from /etc/default/grub. For example, it has entries like:

menuentry "Debian GNU/Linux" {
        set root=(hd0,3)
        linux /vmlinuz root=/dev/hda3
        initrd /initrd.img
}

For these examples, these GRUB parameters mean:

Table 4.2.  The meaning of GRUB parameters.

GRUB parameter

meaning

root

Use 3rd partition on boot disk by setting it as (hd0,2) in GRUB legacy or as (hd0,3) in GRUB 2.

kernel

Use kernel located at /vmlinuz with kernel parameter: "root=/dev/hda3 ro".

initrd

Use initrd/initramfs image located at "/initrd.img".


[Note] Note

The value of the partition number used by GRUB legacy program is one less than normal one used by Linux kernel and utility tools. GRUB 2 program fixes this problem.

[Tip] Tip

UUID (see Section 10.3.2, “Accessing partition using UUID”) may be used to identify a block special device instead of its file name such as "/dev/hda3", e.g."root=UUID=81b289d5-4341-4003-9602-e254a17ac232 ro".

[Tip] Tip

You can start a boot loader from another boot loader using techniques called chain loading.

[Note] Note

Having bootable rescue media (CD or floppy) created from images in the grub-rescue-pc package let you boot your system without functioning bootloader on the harddisk.

See "info grub" and grub-install(8).

4.4. Stage 3: the mini-Debian system

The mini-Debian system is the 3rd stage of the boot process which is started by the boot loader. It runs the system kernel with its root filesystem on the memory. It is an optional preparatory stage of the boot process.

[Note] Note

The term "the mini-Debian system" is coined by the author to describe this 3rd stage boot process for this document. This system is commonly referred as the initrd or initramfs system. Similar system on the memory is used by the Debian Installer.

The /init script is executed as the first program in this root filesystem on the memory. It is a shell script program which initializes the kernel in user space and hands control over to the next stage. This mini-Debian system offers flexibility to the boot process such as adding kernel modules before the main boot process or mounting the root file system as an encrypted one.

You can interrupt this part of the boot process to gain root shell by providing "break=init" etc. to the kernel boot parameter. See the /init script for more break conditions. This shell environment is sophisticated enough to make a good inspection of your machine's hardware.

Commands available in this mini-Debian system are stripped down ones and mainly provided by a GNU tool called busybox.

[Caution] Caution

You need to use "-n" option for mount command when you are on the readonly root file system.

4.5. Stage 4: the normal Debian system

The normal Debian system is the 4th stage of the boot process which is started by the mini-Debian system. The system kernel for the mini-Debian system continues to run in this environment. The root filesystem is switched from the one on the memory to the one on the real harddisk filesystem.

The /sbin/init program is executed as the first program and performs the main boot process. The Debian normally uses the traditional sysvinit scheme with the sysv-rc package. See man 8 init, man 5 inittab, and /usr/share/doc/sysv-rc/README.runlevels.gz for the exact explanation. Following is a simplified overview of this main boot process:

  1. The Debian system goes into runlevel N (none) to initialize the system by following the /etc/inittab description.

  2. The Debian system goes into runlevel S to initialize the system under the single-user mode to complete hardware initialization etc.

  3. The Debian system switches itself to one of the specified multi-user runlevels (2 to 5) to start the system services.

The initial runlevel used for multi-user mode is specified with the "init=" kernel boot parameter or in the "initdefault" line of this /etc/inittab. The Debian system as installed starts at the runlevel 2.

All scripts executed by the init system are located in the directory /etc/init.d/.

[Tip] Tip

For alternative boot mechanism to the sysv-rc package using a single configuration file, see the file-rc package. Both mechanisms are compatible through /etc/init.d/rc, /etc/init.d/rcS, /usr/sbin/update-rc.d, and /usr/sbin/invoke-rc.d scripts.

4.5.1. The meaning of the runlevel

Each runlevel uses a directory for its configuration and has specific meaning:

Table 4.3.  List of runlevels and meanings.

runlevel

directory

meaning

N

none

System bootup (NONE). There is no /etc/rcN.d/ directory.

0

/etc/rc0.d/

Halt the system.

S

/etc/rcS.d/

Single-user mode on boot. The lower case s can be used as alias.

1

/etc/rc1.d/

Single-user mode switched from multi-user mode.

2

/etc/rc2.d/

Multi-user mode.

3

/etc/rc3.d/

,,

4

/etc/rc4.d/

,,

5

/etc/rc5.d/

,,

6

/etc/rc6.d/

Reboot the system.

7

/etc/rc7.d/

Valid multi-user mode but not normally used.

8

/etc/rc8.d/

,,

9

/etc/rc9.d/

,,


You can change the runlevel from the console to, e.g., 4 by:

$ sudo telinit 4
[Caution] Caution

The Debian system does not pre-assign any special meaning differences among the runlevels between 2 and 5. The system administrator on the Debian system may change this. (I.e., Debian is not RedHat nor SOLARIS nor HP-UX nor ...)

[Caution] Caution

The Debian system does not populate directories for the runlevels between 7 and 9 when the package is installed. Traditional Unix variants don’t use these runlevels.

4.5.2. The configuration of the runlevel

The names of the symlinks in the runlevel directories have the form S<2-digit-number><original-name> or K<2-digit-number><original-name>. The 2-digit-number is used to determine the order in which to run the scripts. 'S' is for 'Start' and 'K' is for 'Kill'.

When init or telinit commands change the runlevel to <n>:

  1. the script names starting with a K in /etc/rc<n>.d/ are executed in alphabetical order with the single argument stop . (killing services)

  2. the script names starting with an S in /etc/rc<n>.d/ are executed in alphabetical order with the single argument start . (starting services)

For example, if you had the links S10sysklogd and S20exim4 in a runlevel directory, S10sysklogd would run before S20exim4.

[Warning] Warning

It is not advisable to make any changes to symlinks in /etc/rcS.d/ unless you know better than the maintainer.

4.5.3. The runlevel management example

For example, let's set up runlevel system somewhat like Redhat system, i.e.:

  • to start the system in runlevel=3 as the default,

  • not to start gdm in runlevel=(0,1,2,6), and

  • to start gdm in runlevel=(3,4,5).

The easy way is to use editor on the /etc/inittab file to change starting runlevel and use user friendly runlevel management tools such as sysv-rc-conf or bum to edit the runlevel. If you are to use command line only, here is how you do it (after the default installation of gdm package and selecting it to be the choice of display manager):

# cd /etc/rc2.d ; mv S21gdm K21gdm
# cd /etc ; perl -i -p -e 's/^id:.:/id:3:/' inittab

Please note the /etc/X11/default-display-manager file is checked when starting the display manager daemons: xdm, gdm, kdm, and wdm.

[Note] Note

You can still start X from any console shell with the startx command.

4.5.4. The default parameter for each init script

The default parameter for each init script in /etc/init.d/ is given by the corresponding file in /etc/default/ which contains environment variable assignments only. The choice of directory name is specific to the Debian system. It is roughly the equivalent of the /etc/sysconfig directory found in Red Hat and other distributions.

For example, /etc/default/hotplug can be used to control how /etc/init.d/hotplug works. The /etc/default/rcS file can be used to customize boot-time defaults for motd, sulogin, etc.

If you cannot get the behavior you want by changing such variables then you may modify the init scripts themselves: they are all configuration files.

4.5.5. The hostname

The kernel maintains the system hostname. The initscript /etc/init.d/hostname.sh sets the system hostname at boot time (using the hostname command) to the name stored in /etc/hostname. This file should contain only the system hostname, not a fully qualified domain name.

To print out the current hostname run hostname without an argument.

4.5.6. Network interface initialization

Network interfaces are initialized under single-user mode on boot by the initscript /etc/init.d/ifupdown-clean and /etc/init.d/ifupdown. See Chapter 6, Network setup for how to configure them.

4.5.7. Network service initialization

Many network services (see Chapter 7, Network applications) are started directly as daemon processes at boot time, e.g., /etc/rc2.d/S20exim4 (for RUNLEVEL=2) which is a symlink to /etc/init.d/exim4.

Some network services can be started on demand using the super-server, inetd (or its equivalents). The inetd is started at boot time by /etc/rc2.d/S20inetd (for RUNLEVEL=2) which is a symlink to /etc/init.d/inetd. Essentially, inetd allows one running daemon to invoke several others, reducing load on the system.

Whenever a request for service arrives, its protocol and service are identified by looking them up in the databases in /etc/protocols and /etc/services. inetd then looks up a normal Internet service in the /etc/inetd.conf database, or a Sun-RPC based service in /etc/rpc.conf.

For system security, make sure to disable unused services in /etc/inetd.conf. Sun-RPC services need to be active for NFS and other RPC-based programs.

Sometimes, inetd does not start the intended server directly but starts the TCP wrapper, tcpd, with the intended server name as its argument in /etc/inetd.conf. In this case, tcpd runs the appropriate server program after logging the request and doing some additional checks using /etc/hosts.deny and /etc/hosts.allow.

If you have problems with remote access in a recent Debian system, comment out "ALL: PARANOID" in /etc/hosts.deny if it exists. (But you must be careful on security risks involved with this kind of action.)

For details, see inetd(8), inetd.conf(5), protocols(5), services(5), tcpd(8), hosts_access(5), and hosts_options(5).

For more information on Sun-RPC, see rpcinfo(8), portmap(8), and /usr/share/doc/portmap/portmapper.txt.gz.

4.5.8. The system message

The system message can be customized by /etc/syslog.conf for both the log file and on-screen display. See syslogd(8) and syslog.conf(5). See also Section 10.2.2, “Log analyzer”.

4.5.9. The kernel message

The kernel message can be customized by /etc/init.d/klogd for both the log file and on-screen display. Set KLOGD="-c 3" in this script and run /etc/init.d/klogd restart. See klogd(8).

You may directly change the error message level by:

# dmesg -n3

Here:

Table 4.4.  List of kernel error levels.

error level value

error level name

meaning

0

KERN_EMERG

system is unusable

1

KERN_ALERT

action must be taken immediately

2

KERN_CRIT

critical conditions

3

KERN_ERR

error conditions

4

KERN_WARNING

warning conditions

5

KERN_NOTICE

normal but significant condition

6

KERN_INFO

informational

7

KERN_DEBUG

debug-level messages


4.5.10. The udev system

For Linux kernel 2.6, the udev system provides mechanism for the automatic hardware discovery and initialization (see udev(7)). Upon discovery of each device by the kernel, the udev system starts a user process which uses information from the sysfs filesystem (see Section 2.2.12, “procfs and sysfs”), loads required kernel modules supporting it using the modprobe(8) program (see Section 4.5.11, “The kernel module initialization”), and creates corresponding device nodes.

The name of device nodes can be configured by files in /etc/udev/rules.d/ (see /usr/share/doc/udev/writing_udev_rules/index.html).

Since the udev system is somewhat a moving target, I leave details to other documentations and describe the minimum information here.

4.5.11. The kernel module initialization

The modprobe(8) program enables us to configure running Linux kernel from user process by adding and removing kernel modules. The udev system (see Section 4.5.10, “The udev system”) automates its invocation to help the kernel module initialization.

Non-hardware modules and special hardware driver modules, such as:

need to be pre-loaded by listing them in the /etc/modules file (see modules(5)).

The configuration files for the modprobe(8) program are located under the /etc/modprobes.d/ directory as explained in modprobe.conf(5). (If you want to avoid some kernel modules to be auto-loaded, consider to blacklist them in the /etc/modprobes.d/blacklist file.)

The /lib/modules/<version>/modules.dep file generated by the depmod(8) program describes module dependencies used by the modprobe(8) program.

The modinfo(8) program shows information about a Linux Kernel module.

The lsmod(8) program nicely formats the contents of the /proc/modules, showing what kernel modules are currently loaded.

[Tip] Tip

You can identify exact hardware on your system. See Section 10.6.3, “The hardware identification”.

[Tip] Tip

You may configure hardware at boot time to activate expected hardware features. See Section 10.6.4, “The hardware configuration”.

[Tip] Tip

You can add support for your device by recompiling kernel. See Section 10.7, “The kernel”.

Chapter 5. Authentication

5.1. Normal Unix authentication

Normal Unix authentication is provided by the pam_unix.so method under the PAM (Pluggable Authentication Modules). Its 3 important configuration files, with ":" separated entries, are:

Table 5.1.  The 3 important configuration files for pam_unix.o.

file

permission

user

group

description

/etc/passwd

-rw-r--r--

root

root

The (sanitized) user account information.

/etc/shadow

-rw-r-----

root

shadow

The secure user account information.

/etc/group

-rw-r--r--

root

root

The group information.


The /etc/passwd file contains:

...
user1:x:1000:1000:User1 Name,,,:/home/user1:/bin/bash
user2:x:1001:1001:User2 Name,,,:/home/user2:/bin/bash
...

As explained in man 5 passwd, each ":" separated entry of this file means:

  • login name

  • password specification entry

  • numerical user ID

  • numerical group ID

  • user name or comment field

  • user home directory

  • optional user command interpreter

The second entry of /etc/passwd was used for the encrypted password entry. After the introduction of /etc/shadow, this entry is used for the password specification entry.

Table 5.2.  The second entry content of /etc/passwd.

content

meaning

(empty)

passwordless account

x

the encrypted password is in the /etc/shadow file

*

no login for this account

!

no login for this account


The /etc/shadow file contains:

...
user1:$1$Xop0FYH9$IfxyQwBe9b8tiyIkt2P4F/:13262:0:99999:7:::
user2:$1$vXGZLVbS$ElyErNf/agUDsm1DehJMS/:13261:0:99999:7:::
...

As explained in man 5 shadow, each ":" separated entry of this file means:

  • login name

  • encrypted password. The initial "$1$" indicates use of the MD5 encryption. The "*" indicate no login.

  • days since Jan 1, 1970 that password was last changed

  • days before password may be changed

  • days after which password must be changed

  • days before password is to expire that user is warned

The /etc/group file contains:

...
group1:x:20:user1,user2
...

As explained in man 5 shadow, each ":" separated entry of this file means:

  • group name

  • encrypted password (not really used)

  • numerical group ID.

  • "," separated list of user names.

The /etc/gshadow file provides the similar function as the /etc/shadow file for the /etc/group file but is not really used.

[Note] Note

The actual group membership of a user may be dynamically added if "auth optional pam_group.so" line is added to /etc/pam.d/common-auth and set it in /etc/security/group.conf. See pam_group(8).

[Note] Note

The base-passwd package contains an authoritative list of the user and the group: /usr/share/doc/base-passwd/users-and-groups.html.

5.2. Manage account and password information

Here are few notable commands to manage account information:

Table 5.3.  List of commands to manage account information.

command

function

getent passwd <user_name>

browse account information of <user_name>

getent shadow <user_name>

browse shadowed account information of <user_name>

getent group <group_name>

browse group information of <group_name>

passwd

manage password for the account

passwd -e

set one-time password for the account activation

chage

manage password aging information


You may need to have the root privilege for some functions to work. See man 3 crypt for the password and data encryption.

[Note] Note

On the system set up with PAM and NSS as the Debian alioth machine, the content of the local /etc/passwd, /etc/group and /etc/shadow files may not be actively used by the system thus requiring above commands to be used instead of the editor.

5.3. Good password

When creating an account during your system installation or with the passwd(1) command, you should choose a good password which consists of 6 to 8 characters including one or more characters from each of the following sets (per the passwd(1) manpage):

  • lower case alphabetics

  • digits 0 through 9

  • punctuation marks

[Warning] Warning

Do not chose guessable words for the password.

5.4. Creating encrypted password

There are independent tools to generate encrypted password with salt:

Table 5.4.  List of tools to generate password.

package

popcon

size

command

function

whois

V:9, I:89

336

mkpasswd

over-featured front end to the crypt(3) library

openssl

V:28, I:89

2348

openssl passwd

compute password hashes (OpenSSL). passwd(1ssl)


5.5. PAM and NSS

Modern Unix-like systems such as the Debian system provide PAM (Pluggable Authentication Modules) and NSS (Name Service Switch) mechanism to the local system administrator to configure his system. The role of these can be summarizes as:

  • PAM offers a flexible authentication mechanism used by the application software thus involves password data exchange.

  • NSS offers a flexible name service mechanism which is frequently used by the C library to obtain the user and group name for programs such as ls and id.

These PAM and NSS systems need to be configured consistently.

The notable packages of PAM and NSS systems are:

Table 5.5.  List of notable PAM and NSS systems.

package

popcon

size

description

libpam-modules

V:87, I:99

972

Pluggable Authentication Modules for PAM

libpam-ldap

V:0.7, I:4

388

Pluggable Authentication Module allowing LDAP interfaces

libpam-cracklib

V:0.17, I:1.3

116

Pluggable Authentication Module to enable cracklib support

libc6

V:88, I:99

11468

GNU C Library: Shared libraries which also provides "Name Service Switch" service

libnss-mdns

I:56

144

NSS module for Multicast DNS name resolution

libnss-ldap

I:4

304

NSS module for using LDAP as a naming service

libnss-ldapd

V:0.09, I:0.15

316

NSS module for using LDAP as a naming service (new folk of libnss-ldap)


[Note] Note

You can see more extensive and current list by "aptitude search 'libpam-|libnss-'" command. The acronym NSS may also mean "Network Security Service" which is different from "Name Service Switch".

[Note] Note

PAM is the most basic way to initialize environment variables for each program with the system wide default value.

5.5.1. Configuration files accessed by the PAM and NSS

Here are few notable configuration files accessed by the PAM:

Table 5.6.  List of configuration files accessed by the PAM.

configuration file

function

/etc/pam.d/<program_name>

set up PAM configuration for the <program_name>. See manpage of pam(7).

/etc/nsswitch.conf

set up NSS configuration with the entry for each service. See manpage of nsswitch.conf(5).

/etc/nologin

accessed by pam_nologin.so module to limit the user login.

/etc/securetty

accessed by pam_securetty.so module to limit the tty for the root access.

/etc/security/access.conf

accessed by pam_access.so module to set access limit.

/etc/security/group.conf

accessed by pam_group.so module to set group based restraint.

/etc/security/pam_env.conf

accessed by pam_env.so module to set environment variables.

/etc/environment

accessed by pam_env.so module with readenv=1 argument to set environment variables additionally.

/etc/default/locale

accessed by pam_env.so module with argument readenv=1 envfile=/etc/default/locale to set locale (Debian).

/etc/security/limits.conf

accessed by pam_linits.so module to set resource restraint (ulimit, core, ...).

/etc/security/time.conf

accessed by pam_time.so module to set time restraint.


The limitation of the password selection is implemented by the PAM modules, pam_unix.so and pam_cracklib.so and configuring them with arguments.

5.5.2. The modern centralized system management

The modern centralized system management can be deployed using the centralized Lightweight Directory Access Protocol (LDAP) server to administer many Unix-like and non-Unix-like systems on the network. The open source implementation of the Lightweight Directory Access Protocol is OpenLDAP Software.

The LDAP server provides the account information through the use of PAM and NSS with libpam-ldap and libnss-ldap packages for the Debian system. Several actions are required to enable this (I have not used this setup and based purely on secondary information. Please read this in this context.):

  • You set up a centralized LDAP server by running program such as stand-alone LDAP daemon, slapd.

  • You change the PAM configuration files in the /etc/pam.d/ directory to use pam_ldap.so instead of the default pam_unix.so.

  • You change the NSS configuration in the /etc/nsswitch.conf file to use ldap instead of the default (compat or file).

  • Debian uses /etc/pam_ldap.conf as libpam-ldap's configuration file and /etc/pam_ldap.secret as the file to store the password of the rootbinddn.

  • You must make libpam-ldap to use SSL (or TLS) connection for the security of password.

  • You may make libnss-ldap to use SSL (or TLS) connection using /etc/libnss-ldap.conf to ensure integrity of data at the cost of the LDAP network overhead.

  • You should run nscd locally to cache any LDAP search results in order to reduce the LDAP network traffic.

See documentations in /usr/share/doc/libpam-doc/html/ offered by libpam-doc package and "info libc 'Name Service Switch'" offered by glibc-doc package.

Similarly, you can set up alternative centralized systems with:

  • NIS or NIS+ with the traditional Unix systems.

  • winbind with Windows NT and SAMBA.

5.5.3. "Why GNU su does not support the wheel group"

This is the famous phrase at the bottom of the old "info su" page by Richard M. Stallman. Not to worry: the current su in Debian uses PAM, so that one can restrict the ability to use su to root group by enabling the line with pam_wheel.so in /etc/pam.d/su.

5.5.4. Stricter password rule

Installing the libpam-cracklib package will enable you to force stricter password rule, for example, by having active lines in /etc/pam.d/common-password as:

password required pam_cracklib.so retry=3 minlen=9 difok=3
password required pam_unix.so use_authtok nullok md5

5.6. Other access controls

[Note] Note

See Section 10.5.15, “Alt-SysRq” for restricting the kernel SAK feature.

5.6.1. sudo

sudo is a program designed to allow a sysadmin to give limited root privileges to users and log root activity. sudo requires only an ordinary user's password. Install sudo package and activate it by setting options in /etc/sudoers. See configuration example at /usr/share/doc/sudo/examples/sudoers.

My usage of sudo for the single user system (see Section 2.1.12, “sudo configuration”) is aimed to protect myself from my own stupidity. Personally, I consider using sudo a better alternative to using the system from the root account all the time. For example, following will change the owner of <some_file> to <my_name>:

$ sudo chown <my_name> <some_file>

Of course if you know the root password (as self-installed Debian users do), any command can be run under root from any user's account using "su -c".

5.6.2. SELinux

Security-Enhanced Linux (SELinux)is a framework to tighten privilege model tighter than the ordinary Unix-like security model with the mandatory access control (MAC) policies. The root power may be restricted under some conditions.

5.6.3. Restricting access to some server services

The Internet super-server, inetd, is started at boot time by /etc/rc2.d/S20inetd (for RUNLEVEL=2), which is a symlink to /etc/init.d/inetd. Essentially, inetd allows one running daemon to invoke several others, reducing load on the system.

Whenever a request for service arrives, its protocol and service are identified by looking them up in the databases in /etc/protocols and /etc/services. inetd then looks up a normal Internet service in the /etc/inetd.conf database, or a Sun-RPC based service in /etc/rpc.conf.

For system security, make sure to disable unused services in /etc/inetd.conf. Sun-RPC services need to be active for NFS and other RPC-based programs.

Sometimes, inetd does not start the intended server directly but starts the tcpd TCP/IP daemon wrapper program with the intended server name as its argument in /etc/inetd.conf. In this case, tcpd runs the appropriate server program after logging the request and doing some additional checks using /etc/hosts.deny and /etc/hosts.allow.

If you have problems with remote access in a recent Debian system, comment out "ALL: PARANOID" in /etc/hosts.deny if it exists.

For details, see inetd(8), inetd.conf(5), protocols(5), services(5), tcpd(8), hosts_access(5), and hosts_options(5).

For more information on Sun-RPC, see rpcinfo(8), portmap(8), and /usr/share/doc/portmap/portmapper.txt.gz.

There are also non-PAM based access control available for atd and cron.

5.7. Security of authentication

The information here may not be sufficient for your security needs but it should be a good start.

5.7.1. Secure password over the Internet

Many popular transportation layer services communicate messages including password authentication in the plain text. It is very bad idea to transmit password in the plain text over the wild Internet where it can be intercepted. You can run these services over "Transport Layer Security" (TLS) or its predecessor, "Secure Sockets Layer" (SSL) to secure entire communication including password by the encryption.

Table 5.7.  List of insecure and secure services and ports.

insecure service name

port

secure service name

port

www (http)

80

https

443

smtp (mail)

25

ssmtp (smtps)

465

ftp-data

20

ftps-data

989

ftp

21

ftps

990

telnet

23

telnets

992

imap2

143

imaps

993

pop3

110

pop3s

995

ldap

389

ldaps

636


The encryption costs CPU time. As a CPU friendly alternative, you can keep communication in plain text while securing just password with the secure authentication protocol such as "Authenticated Post Office Protocol" (APOP) for POP and "Challenge-Response Authentication Mechanism MD5" (CRAM-MD5) for SMTP and IMAP. (For sending mail messages over the Internet to your mail server from your mail client, it is recently popular to use new message submission port 587 instead of traditional SMTP port 25 to avoid port 25 blocking by the network provider while authenticating yourself with CRAM-MD5.)

5.7.2. Secure Shell

The Secure Shell (ssh) program (OpenSSH SSH client) and sshd program (OpenSSH SSH daemon) provide secure encrypted communications between two untrusted hosts over an insecure network with the secure authentication. This SSH can be used to tunnel the insecure protocol communication such as POP and X securely over the Internet with the port forwarding feature.

The client tries to authenticate itself using host-based authentication, public key authentication, challenge-response authentication, or password authentication. The use of public key authentication enables the remote password-less login. See man 1 ssh, man 8 sshd, and Section 7.4, “The remote access server and utility (SSH)”.

5.7.3. Extra security measures for the Internet

Even when you run secure services such as "Secure Shell" (SSH) and "Point-to-point tunneling protocol" (PPTP) servers, there are still chances for the break-ins using brute force password guessing attack etc. from the Internet. Use of the firewall policy (see Section 6.8, “Netfilter”) together with the following secure tools may improve the security situation.

Table 5.8.  List of tools to provide extra security measures.

package

popcon

description

knockd

V:0.13, I:0.3

small port-knock daemon knocked(1) and client konck(1)

denyhosts

V:1.3, I:1.7

an utility to help sysadmins thwart ssh hackers

fail2ban

V:2, I:2

bans IPs that cause multiple authentication errors

libpam-shield

V:0.01, I:0.05

locks out remote attackers trying password guessing


5.7.4. Securing the root password

To prevent people to access your machine with root privilege, you need to:

  • prevent physical access to the hard disk.

  • lock BIOS and prevent booting from the removable media.

  • set password for GRUB interactive session.

  • lock GRUB menu from editing.

With physical access to hard disk, resetting the password is relatively easy;

  • move the hard disk to a PC with CD bootable BIOS.

  • boot system with a rescue media (Debian boot disk, Knopix CD, GRUB CD, ...).

  • mount root partition with read-write access.

  • edit /etc/passwd in the root partition and make the second entry for root account empty.

If you have the edit access to the GRUB menu entry (see Section 4.3, “Stage 2: the boot loader”) for grub-rescue-pc) at the boot time, it is even easier:

  • boot system with the kernel parameter changed to something like "root=/dev/hda6 rw init=/bin/sh".

  • edit /etc/passwd and make the second entry for root account empty.

  • reboot system.

The root shell of the system is now accessible without password.

[Note] Note

Once you have root shell access, you can compromise password for all user accounts using brute force password cracking tools such as john and crack packages (see Section 10.6.11, “System security and integrity check”).

The only reasonable software solution to avoid all these concerns is to use software encrypted root partition (or /etc partition) using dm-crypt and initramfs (see Section 10.4, “Data encryption tips”). You always need password to boot the system, though.

Chapter 6. Network setup

This section will address TCP/IP network setup for the mobile PC which moves around different networks. (For the non-mobile PC, the debian-installer should have taken care your network setup and there are almost nothing for us to play with.)

You should install the resolvconf package for the mobile PC. This package provides framework to solve conflicts of the host address resolution between different network configuration scripts when network configuration changes. Read more on /usr/share/doc/resolvconf/README.Debian.

For the fixed location server machine, you can do without the resolvconf package and keep your system simple.

In this document, we focus on Debian-specific issues. For a general guide to GNU/Linux networking, read the Linux Network Administrators Guide.

6.1. The basic network infrastructure

Let's review the basic network infrastructure of the modern Debian system.

6.1.1. The domain name

The naming for the domain name is a tricky one for the normal workstation PC users. The PC workstation may be mobile one hopping around the network or located behind the NAT firewall inaccessible from the Internet. For such case, you may not want the domain name to be a valid domain name to avoid name collision.

According to rfc2606, "invalid" seems to be a choice for the top level domain (TLD) to construct domain names that are sure to be invalid from the Internet.

The mDNS network discovery protocol (Apple Bonjour / Apple Rendezvous, Avahi on Debian) uses "local" as the pseudo-top-level domain. Microsoft also seem to promote "local" for the TLD of local area network.

Other popular choices for the invalid TLD seem to be "localdomain", "lan", "localnet", or "home" according to my incoming mail analysis.

6.1.2. The hostname resolution

The hostname resolution is currently supported by the NSS (Name Service Switch) mechanism too. The flow of this resolution is:

  1. The /etc/nsswitch.conf file with stanza like "hosts: files dns" dictates the hostname resolution order. (This replaces the old functionality of the "order" stanza in /etc/host.conf.)

  2. The files method is invoked first. If the hostname is found in the /etc/hosts file, it returns all valid addresses for it and exits. (The /etc/host.conf file contains "multi on".)

  3. The dns method is invoked. If the hostname is found by the query to the Internet Domain Name System (DNS) identified by the /etc/resolv.conf file, it returns all valid addresses for it and exits.

The /etc/hosts file associates IP addresses with hostnames:

127.0.0.1 localhost
127.0.1.1 <host_name>.<domain_name> <host_name>

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Here the <host_name> in this matches the own hostname defined in the /etc/hostname. The <domain_name> in this is the fully qualified domain name (FQDN) of this host.

[Tip] Tip

I sometimes use bogus domain name for <domain_name>, such as "invalid" in this configuration for the mobile PC.

The /etc/resolv.conf is a static file if the resolvconf package is not installed. If installed, it is a symbolic link. Either way, it contains information that initialize the resolver routines. If the DNS is found at IP="192.168.11.1", it contains:

nameserver 192.168.11.1

The resolvconf package makes this /etc/resolv.conf into a symbolic link and manages its contents by the hook scripts automatically.

The hostname resolution via Multicast DNS (using Zeroconf, aka Apple Bonjour / Apple Rendezvous) which effectively allows name resolution by common Unix/Linux programs in the ad-hoc mDNS domain "local", can be provided by installing the libnss-mdns package. The /etc/nsswitch.conf file should have stanza like "hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4" to enable this functionality.

6.1.3. The network interface name

The network interface name, e.g. eth0, is assigned to each hardware in the Linux kernel through the user space configuration mechanism, udev (see: Section 4.5.10, “The udev system”), as it is found. The network interface name is referred as physical interface in the manpage of ifup(8) and interfaces(5).

In order to ensure each network interface to be named persistently for each reboot using MAC address etc., there is a record file "/etc/udev/rules.d/70-persistent-net.rules". This file is automatically generated by the "/lib/udev/write_net_rules" program, probably run by the "persistent-net-generator.rules" rules file. You can modify it to change naming rule.

[Caution] Caution

When editing the "/etc/udev/rules.d/70-persistent-net.rules" rules file, you must keep each rule on a single line and the MAC address in lowercase. For example, if you find "Firewire device" and "PCI device" in this file, you probably want to name "PCI device" as eth0 and configure it as the primary network interface.

6.1.4. The network address range for the LAN

Let us be reminded of the IPv4 32 bit address ranges in each class reserved for use on the local area networks (LANs) by rfc1918. These addresses are guaranteed not to conflict with any addresses on the Internet proper.

Table 6.1.  List of network address ranges.

Class

network addresses

net mask

net mask /bits

# of subnets

A

10.x.x.x

255.0.0.0

/8

1

B

172.16.x.x -- 172.31.x.x

255.255.0.0

/16

16

C

192.168.0.x -- 192.168.255.x

255.255.255.0

/24

256


[Note] Note

If one of these addresses is assigned to a host, then that host must not access the Internet directly but must access it through a gateway that acts as a proxy for individual services or else does Network Address Translation(NAT). The broadband router usually performs NAT for the consumer LAN environment.

6.1.5. The network configuration infrastructure

The ifupdown package and its associated packages are the de facto standard for the Debian networking infrastructure. Its configuration file is the /etc/network/interfaces (/e/n/i)file and its typical contents are:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
[Tip] Tip

There are independent automatic network configuration tools for mobile desktop users on laptops (see Section 6.5.1, “Automatic network configuration”).

Table 6.2.  List of network configuration tools.

packages

popcon

size

type

function

ifupdown

V:59, I:99

228

config::ifupdown

Standardized level to bring up and down the network (Debian specific)

ifplugd

V:0.5, I:1.1

332

, ,

Manage the wired network automatically

ifupdown-extra

V:0.06, I:0.2

124

, ,

Network testing script to enhance "ifupdown" package

ifmetric

V:0.01, I:0.10

100

, ,

Set routing metrics for a network interface.

guessnet

V:0.09, I:0.5

492

, ,

Mapping script to enhance "ifupdown" package via /e/n/i file

ifscheme

V:0.02, I:0.10

80

, ,

Mapping scripts to enhance "ifupdown" package

ifupdown-scripts-zg2

V:0.01, I:0.05

216

, ,

Zugschlus' interface scripts for ifupdown's manual method

network-manager

V:30, I:43

1540

config::NetworkManager

NetworkManager (daemon): Manage the network automatically

network-manager-gnome

V:21, I:39

1968

, ,

NetworkManager (GNOME frontend)

network-manager-kde

V:1.7, I:4

2380

, ,

NetworkManager (KDE frontend)

wicd

V:0.5, I:0.7

1828

config::wicd

Wired and wireless network manager

iptables

V:20, I:99

1356

config::Netfilter

Administration tools for packet filtering and NAT

iproute

V:42, I:69

984

config::iproute2

IPv6 and other advanced network configuration: ip(8), tc(8), etc.

ifrename

V:0.2, I:0.9

180

, ,

Rename network interfaces based on various static criteria: ifrename(8)

ethtool

V:2, I:10

228

, ,

Display or change Ethernet device settings:

iputils-ping

V:33, I:99

124

test::iproute2

Tools to test network reachability of a remote host by hostname or IP address

iputils-arping

V:2, I:35

72

, ,

Tools to test network reachability of a remote host specified by the ARP address

iputils-tracepath

V:0.5, I:2

108

, ,

Tools to trace the network path to a remote host

net-tools

V:68, I:99

888

config::net-tools

The NET-3 networking toolkit (IPv4 network configuration): ifconfig(8) etc.

inetutils-ping

V:0.05, I:0.14

192

test::net-tools

Tools to test network reachability of a remote host by hostname or IP address (legacy, GNU)

arping

V:0.5, I:2

64

, ,

Tools to test network reachability of a remote host specified by the ARP address (legacy)

traceroute

V:13, I:98

196

, ,

Tools to trace the network path to a remote host (legacy, console)

dhcp3-client

V:48, I:92

608

config::low-level

DHCP client

wpasupplicant

V:32, I:49

824

, ,

Client support for WPA and WPA2 (IEEE 802.11i)

wireless-tools

V:9, I:29

352

, ,

Tools for manipulating Linux Wireless Extensions

ppp

V:7, I:24

1100

, ,

PPP/PPPoE connection with chat

pppoeconf

V:0.5, I:5

200

config::helper

Configuration helper for PPPoE connection

pppconfig

V:0.4, I:3

900

, ,

Configuration helper for plain PPP connection with chat

wvdial

V:0.4, I:1.9

352

, ,

PPP connection with wvdial with ppp, configuration helper

mtr-tiny

V:6, I:82

120

test::low-level

Tools to trace the network path to a remote host (curses)

mtr

V:0.4, I:1.9

176

, ,

Tools to trace the network path to a remote host (curses and GTK+)

gnome-nettool

V:4, I:45

1766

, ,

Tools for common network information operations (GNOME)

nmap

V:6, I:29

3768

, ,

Network mapper / port scanner (console)

zenmap

V:0.19, I:0.7

1232

, ,

Network mapper / port scanner (GTK+)

knmap

V:0.15, I:0.9

1992

, ,

Network mapper / port scanner (KDE)

tcpdump

V:3, I:21

728

, ,

Network traffic analyzer (console)

wireshark

V:1.8, I:10

1604

, ,

Network traffic analyzer (GTK+)

tshark

V:0.6, I:3

284

, ,

Network traffic analyzer (console)

tcptrace

V:0.08, I:0.4

432

, ,

Tool to produce a summarization of the connections from tcpdump output

snort

V:0.7, I:1.0

1092

, ,

Flexible network intrusion detection system

ntop

V:1.1, I:2

15600

, ,

display network usage in web browser

dnsutils

V:12, I:91

388

, ,

Network clients provided with BIND: nslookup(8), nsupdate(8), dig(8)

dlint

V:0.5, I:8

96

, ,

Checks DNS zone information using nameserver lookups

dnstracer

V:0.13, I:0.6

88

, ,

Tool to trace a chain of DNS servers to the source


There are 2 types of low level networking programs for Linux networking system (see Section 6.6.1, “Iproute2 commands”).

  • Old net-tools programs (ifconfig(8), ...) are from the Linux NET-3 networking system. Most of these are obsolete now.

  • New Linux iproute2 programs (ip(8), ...) are the current Linux networking system.

6.1.6. The network device support

Although most hardware devices are supported by the Debian system, there are some network devices which require DSFG non-free external hardware drivers to support them. Please see Section 10.7.7, “Non-free hardware drivers”.

6.2. The network connection method

The typical network connection method and connection path for a PC can be summarized as:

Table 6.3.  List of network connection types and connection paths.

PC

connection method

connection path

Serial port (ppp0)

PPP

<=> modem <=> POTS <=> dial-up access point <=> ISP

Ethernet port (eth0)

PPPoE/DHCP/Static

<=> BB-modem <=> BB service <=> BB access point <=> ISP

Ethernet port (eth0)

DHCP/Static

<=> LAN <=> BB-router with network address translation (NAT) (<=> BB-modem ...)


Here is the summary of configuration script for each connection method:

Table 6.4.  List of network connection configurations.

connection method

configuration

backend package(s)

PPP

pppconfig to create deterministic chat

pppconfig, ppp

PPP (alternative)

wvdialconf to create heuristic chat

ppp, wvdial

PPPoE

pppoeconf to create deterministic chat

pppoeconf, ppp

DHCP

described in /etc/dhcp3/dhclient.conf

dhcp3-client

static IP (IPv4)

described in /e/n/i

net-tools

static IP (IPv6)

described in /e/n/i

iproute


The network connection acronyms mean:

Table 6.5.  List of network connection acronyms.

acronym

meaning

POTS

The plain old telephone service

BB

The broadband

BB-service

E.g., the digital subscriber line (DSL), the cable TV, or the fiber to the premises (FTTP).

BB-modem

E.g., the DSL modem, the cable modem, or the optical network terminal (ONT).

LAN

The local area network

WAN

The wide area network

DHCP

The dynamic host configuration protocol

PPP

The point-to-point protocol

PPPoE

The point-to-point protocol over Ethernet

ISP

The Internet service provider


[Note] Note

The WAN connection services via cable TV are generally served by DHCP or PPPoE. The ones by ADSL and FTTP are generally served by PPPoE. You have to consult your ISP for exact configuration requirements of the WAN connection.

[Note] Note

When BB-router is used to create home LAN environment, PCs on LAN are connected to the WAN via BB-router with network address translation (NAT). For such case, PC's network interfaces on the LAN are served by static IP or DHCP from the BB-router. BB-router must be configured to connect the WAN following the instruction by your ISP.

[Caution] Caution

The connection test method described in this section are meant for testing purposes. It is not meant to be used directly for the daily network connection. You are advised to use them via the ifupdown package (see Section 6.3, “The basic network configuration with ifupdown”).

6.2.1. The DHCP connection with the Ethernet

The typical modern home and small business network, i.e. LAN, are connected to the WAN(Internet) using some consumer grade broadband router. The LAN behind this router is usually served by the dynamic host configuration protocol (DHCP) server running on the router.

Just install the dhcp3-client package for the Ethernet served by the dynamic host configuration protocol (DHCP).

6.2.2. The static IP connection with the Ethernet

No special action is needed for the Ethernet served by the static IP.

6.2.3. The PPP connection with pppconfig

The configuration script pppconfig will configure the PPP connection interactively just by selecting:

  • the telephone number,

  • the ISP user name,

  • the ISP password,

  • the port speed,

  • the modem communication port, and

  • the authentication method.

The configuration files are:

Table 6.6.  List of configuration files for the PPP connection with pppconfig.

file

function

/etc/ppp/peers/<isp_name>

The pppconfig generated configuration file for pppd specific to <isp_name>

/etc/chatscripts/<isp_name>

The pppconfig generated configuration file for chat specific to <isp_name>

/etc/ppp/options

The general execution parameter for pppd

/etc/ppp/pap-secret

Authentication data for the PAP (still used)

/etc/ppp/chap-secret

Authentication data for the CHAP (more secure)


[Caution] Caution

The "<isp_name>" value of "provider" is assumed if pon and poff commands are invoked without arguments.

You can test configuration using lower level network configuration tools:

$ sudo pon <isp_name>
...
$ sudo poff <isp_name>

See /usr/share/doc/ppp/README.Debian.gz for more information.

6.2.4. The alternative PPP connection with wvdialconf

A different approach to using pppd is to run it from wvdial which comes in the wvdial package. Instead of pppd running chat to dial in and negotiate the connection, wvdial does the dialing and initial negotiating and then starts pppd to do the rest.

The configuration script wvdialconf will configure the PPP connection interactively just by selecting:

  • the telephone number,

  • the ISP user name, and

  • the ISP password.

The wvdial succeeds in making the connection in most cases and maintains authentication data list automatically.

The configuration files are:

Table 6.7.  List of configuration files for the PPP connection with wvdialconf.

file

function

/etc/ppp/peers/wvdial

The wvdialconf generated configuration file for pppd specific to wvdial

/etc/wvdial.conf

The wvdialconf generated configuration file

/etc/ppp/options

The general execution parameter for pppd

/etc/ppp/pap-secret

Authentication data for the PAP (still used)

/etc/ppp/chap-secret

Authentication data for the CHAP (more secure)


You can test configuration using lower level network configuration tools:

$ sudo wvdial
...
$ sudo killall wvdial

See manpages of wvdial(1), wvdial.conf(5) for more information.

6.2.5. The PPPoE connection with pppoeconf

When your ISP serves you with PPPoE connection and you decide to connect your PC directly to the WAN, the network of your PC must be configured with the PPPoE. The PPPoE stand for PPP over Ethernet. The configuration script pppoeconf will configure the PPPoE connection interactively.

The configuration files are:

Table 6.8.  List of configuration files for the PPPoE connection with pppoeconf.

file

function

/etc/ppp/peers/dsl-provider

The pppoeconf generated configuration file for pppd specific to pppoe

/etc/ppp/options

The general execution parameter for pppd

/etc/ppp/pap-secret

Authentication data for the PAP (still used)

/etc/ppp/chap-secret

Authentication data for the CHAP (more secure)


You can test configuration using lower level network configuration tools:

$ sudo /sbin/ifconfig eth0 up
$ sudo pon dsl-provider
...
$ sudo poff dsl-provider
$ sudo /sbin/ifconfig eth0 down

See /usr/share/doc/pppoeconf/README.Debian for more information.

6.3. The basic network configuration with ifupdown

The ifupdown package provides the standardized framework for the high level network configuration in the Debian system. In this section, we learn the basic network configuration with ifupdown with simplified introduction and many typical examples.

6.3.1. The command syntax simplified

The ifupdown package contains 2 commands: ifup(8) and ifdown(8). They offer high level network configuration dictated by the configuration file /e/n/i.

Table 6.9.  List of basic network configuration commands with ifupdown.

command

action

ifup eth0

To bring up a network interface eth0 with the configuration eth0 if "iface eth0" stanza exists.

ifdown eth0

To bring down a network interface eth0 with the configuration eth0 if "iface eth0" stanza exists.


[Note] Note

There is no command ifupdown.

[Warning] Warning

Do not use low level configuration tools such as ifconfig(8) and ip(8) commands to configure an interface in up state.

6.3.2. The basic syntax of /etc/network/interfaces

The key syntax of /etc/network/interfaces (/e/n/i as its acronym in the following text) as explained in the manpage interfaces(5) can be summarized as:

Table 6.10.  List of stanzas in /e/n/i

stanza

meaning

"auto <interface_name>"

To start interface <interface_name> upon start of the system.

"allow-auto <interface_name>"

, ,

"allow-hotplug <interface_name>"

To start interface <interface_name> when the kernel detects a hotplug event from the interface.

Lines started with "iface <config_name> ..."

To define the network configuration <config_name>.

Lines started with "mapping <interface_name_glob> "

To define mapping value of <config_name> for the matching <interface_name>.

A line starting with a hash "#"

To be ignored as comments. (end-of-line comments are not supported)

A line ending with a backslash "\"

To extend the configuration to the next line.


Lines started with iface stanza has the following syntax:

iface <config_name> <address_family> <method_name>
 <option1> <value1>
 <option2> <value2>
 ...

For the basic configuration, the mapping stanza is not used and you use the network interface name as the network configuration name. (See Section 6.4.5, “The mapping stanza”).

6.3.3. The loopback network interface

The following configuration entry in the /e/n/i file brings up the loopback network interface lo upon booting the system (via auto stanza).

auto lo
iface lo inet loopback

This one always exists in the /e/n/i file.

6.3.4. The network interface served by the DHCP

After prepairing the system by Section 6.2.1, “The DHCP connection with the Ethernet”, the network interface served by the DHCP is configured by creating the configuration entry in the /e/n/i file as:

allow-hotplug eth0
iface eth0 inet dhcp
 hostname "mymachine"

When the Linux kernel detects the physical interface eth0, the allow-hotplug stanza will cause ifup to bring up the interface and the iface stanza will cause ifup to use DHCP to configure the interface.

6.3.5. The network interface with the static IP

The network interface served by the static IP is configured by creating the configuration entry in the /e/n/i file as, e.g.,:

allow-hotplug eth0
iface eth0 inet static
 address 192.168.11.100
 netmask 255.255.255.0
 broadcast 192.168.11.255
 gateway 192.168.11.1
 dns-domain lan
 dns-nameservers 192.168.11.1

When the Linux kernel detects the physical interface eth0, the allow-hotplug stanza will cause ifup to bring up the interface and the iface stanza will cause ifup to use the static IP to configure the interface.

Here, I assumed:

  • IP address range of the LAN network: 192.168.11.0 - 192.168.11.255

  • IP address of the gateway: 192.168.11.1

  • IP address of the PC: 192.168.11.100

  • The resolvconf package is installed.

  • The domain name as "lan".

  • The DNS server at: 192.168.11.1

When the resolvconf package is not installed, DNS related configuration needs to be done manually by editing the /etc/resolv.conf as:

nameserver 192.168.11.1 
domain lan
[Caution] Caution

The IP addresses used in the above example are not meant to be copied literally. You have to adjust IP numbers to your actual network configuration.

6.3.6. The basics of wireless LAN interface

The wireless LAN (WLAN for short) provides the fast wireless connectivity through the spread-spectrum communication of unlicensed radio bands based on the set of standards called IEEE 802.11.

The WLAN interfaces are almost like normal Ethernet interfaces but require some network ID and encryption key data to be provided when they are initialized. The higher level network tools are exactly the same as the Ethernet except the interface names are a bit different like ath0, wlan0, ath0, wifi0, ... depending on the kernel drivers used.

Here are some keywords to remember for the WLAN:

Table 6.11.  List of acronyms for WLAN.

acronym

full word

meaning

NWID

Network ID

The 16 bit network ID used by pre-802.11 network. Very much deprecated.

(E)SSID

(Extended) Service Set Identifier

The network name of the Wireless Access Points (APs) interconnected to form an integrated 802.11 wireless LAN. Domain ID.

WEP, (WEP2)

Wired Equivalent Privacy

The 1st generation 64-bit (128-bit) wireless encryption standard with 40-bit key. Deprecated.

WPA

Wi-Fi Protected Access

The 2nd generation wireless encryption standard (most of 802.11i), compatible with WEP.

WPA2

Wi-Fi Protected Access 2

The 3rd generation wireless encryption standard (full 802.11i), non-compatible with WEP.


The actual choice of protocol is usually limited by the wireless router you deploy.

6.3.7. The wireless LAN interface with WEP

You need to install the wireless-tools package to support the WLAN with the old WEP. (Your consumer grade router may still be using. Insecure infrastructure but better than nothing.)

[Caution] Caution

Please note that your network traffic on WLAN may be sniffed by others.

In case of the DHCP served IP on WLAN connection, the /e/n/i file entry should be:

allow-hotplug eth0
iface eth0 inet dhcp
 wireless-essid Home
 wireless-key1 0123-4567-89ab-cdef
 wireless-key2 12345678
 wireless-key3 s:password
 wireless-defaultkey 2
 wireless-keymode open

See more on /usr/share/doc/wireless-tools/README.Debian.

6.3.8. The wireless LAN interface with WPA/WPA2

You need to install the wpasupplicant package to support the WLAN with the new WPA/WPA2.

In case of the DHCP served IP on WLAN connection, the /e/n/i file entry should be:

allow-hotplug ath0
iface ath0 inet dhcp
 wpa-ssid homezone
 # hexadecimal psk is encoded from a plaintext passphrase
 wpa-psk 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f

See more on /usr/share/doc/wpasupplicant/README.modes.gz.

6.3.9. The PPP connection

You need to configure the PPP connection first as described before (see Section 6.2.3, “The PPP connection with pppconfig”). Then, add the /e/n/i file entry for the primary PPP device ppp0 as:

iface ppp0 inet ppp
 provider <isp_name>

6.3.10. The alternative PPP connection

You need to configure the alternative PPP connection with wvdial first as described before (see Section 6.2.4, “The alternative PPP connection with wvdialconf”). Then, add the /e/n/i file entry for the primary PPP device ppp0 as:

iface ppp0 inet wvdial

6.3.11. The PPPoE connection

For PC connected directly to the WAN served by the PPPoE, you need to configure system with the PPPoE connection as described before (see Section 6.2.5, “The PPPoE connection with pppoeconf”). Then, add the /e/n/i file entry for the primary PPPoE device eth0 as:

allow-hotplug eth0
iface eth0 inet manual
 pre-up /sbin/ifconfig eth0 up
 up ifup ppp0=dsl
 down ifdown ppp0=dsl
 post-down /sbin/ifconfig eth0 down
# The following is used internally only
iface dsl inet ppp
 provider dsl-provider

6.3.12. The network configuration state of ifupdown

The /etc/network/run/ifstate file stores the intended network configuration states for all the currently active network interfaces managed by the ifupdown package are listed. Unfortunately, even if the ifupdown system fails to bring up the interface as intended, the /etc/network/run/ifstate file lists it active.

The output of the ifconfig(8) command without any arguments or with the interface name as its argument provides the actual network configuration state. If the interface does not have the second line as:

          inet addr:192.168.11.2  Bcast:192.168.11.255  Mask:255.255.255.0

it can not be used as a part of IPV4 network.

[Note] Note

For the Ethernet device connected to the PPPoE, the output of the ifconfig(8) command lacks above line.

6.3.13. The basic network reconfiguration

When you try to reconfigure the interface, e.g. eth0, you must disable it first with the "sudo ifdown eth0" command. This will remove the entry of eth0 from the /etc/network/run/ifstate file. (This may result in some error message if eth0 is not active or it is configured improperly previously. So far, it seems to be safe to do this for the simple single user work station at any time.)

You are now free to rewrite the /e/n/i contents as needed to reconfigure the network interface, eth0.

Then, you can reactivate eth0 with the "sudo ifup eth0" command.

[Tip] Tip

You can (re)initialize the network interface simply by "sudo ifdown eth0;sudo ifup eth0".

6.3.14. The ifupdown-extra package

The ifupdown-extra package provides the easy network connection test for use with the ifupdown package:

  • the network-test(1) command from the shell, and

  • the automatic scripts run for each ifup command execution.

The network-test command frees you from the execution of cumbersome low level commands to analyze the network problem.

The automatic scripts are installed in /etc/network/*/ and:

  • check the network cable connection,

  • check duplicate use of IP address,

  • setup system's static routes based on the /etc/network/routes definition,

  • check if network gateway is reachable, and

  • record results in the /var/log/syslog file.

This syslog record is quite useful for administration of the network problem on the remote system.

[Tip] Tip

The automatic behavior of the ifupdown-extra package is configurable with the /etc/default/network-test . Some of these automatic checks slow down the system bootup a little bit since it takes some time to listen for ARP replies.

6.4. The advanced network configuration with ifupdown

The functionality of the ifupdown package can be improved beyond what was described in Section 6.3, “The basic network configuration with ifupdown” with the advanced knowledge.

The functionalities described here are completely optional. I, being lazy and minimalist, rarely bother to use these.

[Caution] Caution

If you could not set up network connection by information in Section 6.3, “The basic network configuration with ifupdown”, you will make situation worse by using information below.

6.4.1. The ifplugd

The ifplugd package is older automatic network configuration tool which can manage only Ethernet connections. This solves unplugged/replugged Ethernet cable issues for mobile PC etc.. If you have NetworkManager or Wicd (see Section 6.5.1, “Automatic network configuration”) installled, you do not need this package.

This package runs daemon and replaces auto or allow-hotplug functionalities (see Table 6.10, “ List of stanzas in /e/n/i ”) and starts interfaces upon their connection to the network.

Here is how to use the ifplugd package for the internal Ethernet port, e.g. eth0:

  • Remove stanza in /e/n/i: "auto eth0" or "allow-hotplug eth0",

  • Keep stanza in /e/n/i: "iface eth0 inet ..." and "mapping ...",

  • Install the ifplugd package,

  • Run "sudo dpkg-reconfigure ifplugd", and

  • Put eth0 as the "static interfaces to be watched by ifplugd".

Now, the network reconfiguration works as you desire:

  • Upon power-on or upon hardware discovery, the interface is not brought up by itself.
  • Upon finding the Ethernet cable, the interface is brought up.

  • Upon some time after unplugging the Ethernet cable, the interface is brought down automatically.

  • Upon plugging in another Ethernet cable, the interface will be brought up under the new network environment.

[Tip] Tip

The arguments for the ifplugd(8) command can set its behaviors such as the delay for reconfiguring interfaces.

6.4.2. The ifmetric

The ifmeric package enables us to manipulate metrics of routes a posteriori even for DHCP.

The following will set the eth0 interface preferred over the wlan0 interface:

  • Install the ifmetric package, and

  • Add an option line with "metric 0" just below the "iface eth0 inet dhcp" line.

  • Add an option line with "metric 1" just below the "iface wlan0 inet dhcp" line.

The metric 0 means the highest priority route and is the default one. The larger metric value means lower priority routes. The IP address of the active interface with the lowest metric value becomes the originating one. See ifmetric(8).

6.4.3. The virtual interface

A single physical Ethernet interface can be configured as multiple virtual interfaces with different IP addresses. Usually the purpose is to connect an interface to several IP subnetworks. For example, IP address based virtual web hosting by a single network interface is one such application.

For example, let's suppose that

  • a single Ethernet interface on your host is connected to a Ethernet hub (not to the broadband router),

  • the Ethernet hub is connected to both the Internet and LAN network,

  • the Internet and LAN network by a single Ethernet interface with the Ethernet hub,

  • the LAN network uses subnet 192.168.0.x/24,

  • your host uses DHCP served IP address with the physical interface eth0 for the Internet, and

  • your host uses 192.168.0.1 with the virtual interface eth0:0 for the LAN,

then following stanzas in /e/n/i will configure your network:

iface eth0 inet dhcp
 metric 0
iface eth0:0 inet static
 address 192.168.0.1
 netmask 255.255.255.0
 network 192.168.0.0
 broadcast 192.168.0.255
 metric 1
[Caution] Caution

Although this configuration example with network address translation (NAT) using netfilter/iptables (see Section 6.8, “Netfilter”) can provide cheap router for the LAN with only single interface, there is no real firewall capability with such set up. You should use 2 physical interfaces with NAT to secure the local network from Internet.

6.4.4. The advanced command syntax

The ifupdown package offers advanced network configuration using the network configuration name and the network interface name. I use slightly different terminology from one used in the manpage of ifup(8) and interfaces(5).

Table 6.12.  List of terminology for network devices.

manpage terminology

my terminology

explanation

examples in the following text

physical interface name

network interface name

A name given by the Linux kernel (using udev mechanism).

lo, eth0, <interface_name>

logical interface name

network configuration name

A name token following iface in the /e/n/i.

config1, config2, <config_name>


Basic network configuration commands in Section 6.3.2, “The basic syntax of /etc/network/interfaces” require the network configuration name token of the iface stanza to match the network interface name in the /e/n/i.

Advanced network configuration commands enables separation of the network configuration name and the network interface name in the /e/n/i:

Table 6.13.  List of advanced network configuration commands with ifupdown.

command

action

ifup eth0=config1

To bring up a network interface eth0 with the configuration config1.

ifdown eth0=config1

To bring down a network interface eth0 with the configuration config1.

ifup eth0

To bring up a network interface eth0 with the configuration selected by mapping stanza.

ifdown eth0

To bring down a network interface eth0 with the configuration selected by mapping stanza.


6.4.5. The mapping stanza

We skipped explaining the mapping stanza in the /e/n/i in Section 6.3.2, “The basic syntax of /etc/network/interfaces” to avoid complication. This stanza has the following syntax:

mapping <interface_name_glob>
 script <script_name>
 map <script_input1>
 map <script_input2>
 map ...

This provides advanced feature to the /e/n/i file by automating the choice of the configuration with the mapping script specified by <script_name>.

When the "<interface_name_glob>" matches "eth0", the execution of

$ sudo ifup eth0

will produce the execution of:

$ sudo ifup eth0=$(echo -e '<script_input1> \n <script_input2> \n ...' | <script_name> eth0)

to configure eth0 automatically. Here, lines with "map" are optional and can be repeated.

[Note] Note

The glob for mapping stanza works like shell file name glob.

6.4.6. The manually switchable network configuration

Here is how to switch manually among several network configurations without rewriting the /e/n/i file as in Section 6.3.13, “The basic network reconfiguration” .

For all the network configuration you need to access, you create a single /e/n/i file, e.g,:

auto lo
iface lo inet loopback

iface config1 inet dhcp
 hostname "mymachine"

iface config2 inet static
 address 192.168.11.100
 netmask 255.255.255.0
 broadcast 192.168.11.255
 gateway 192.168.11.1
 dns-domain lan
 dns-nameservers 192.168.11.1

iface pppoe inet manual
 pre-up /sbin/ifconfig eth0 up
 up ifup ppp0=dsl
 down ifdown ppp0=dsl
 post-down /sbin/ifconfig eth0 down

# The following is used internally only
iface dsl inet ppp
 provider dsl-provider

iface pots inet ppp
 provider provider

Please note the network configuration name which is the token after iface does not use the token for the network interface name. Also, there are no auto stanza nor allow-hotplug stanza to start the network interface eth0 automatically upon events.

Now you are ready to switch the network configuration.

Let's move your PC to a LAN served by the DHCP. You bring up the network interface (the physical interface) eth0 by assigning the network configuration name (the logical interface name) config1 to it:

$ sudo ifup eth0=config1
Password:
...

The interface eth0 is up, configured by DHCP and connected to LAN.

$ sudo ifdown eth0=config1
...

The interface eth0 is down and disconnected from LAN.

Let's move your PC to a LAN served by the static IP. You bring up the network interface eth0 by assigning the network configuration name config2 to it:

$ sudo ifup eth0=config2
...

The interface eth0 is up, configured with static IP and connected to LAN. The additional parameters given as dns-* configures /etc/resolv.conf contents. This /etc/resolv.conf is better manged if the resolvconf package is installed.

$ sudo ifdown eth0=config2
...

The interface eth0 is down and disconnected from LAN, again.

Let's move your PC directly connected to BB-modem connected to the PPPoE served service. You bring up the network interface eth0 by assigning the network configuration name pppoe to it:

$ sudo ifup eth0=pppoe
...

The interface eth0 is up, configured with PPPoE connection directly to the ISP.

$ sudo ifdown eth0=pppoe
...

The interface eth0 is down and disconnected, again.

Let's move your PC to a location without LAN or BB-modem but with POTS and modem. You bring up the network interface ppp0 by assigning the network configuration name pots to it:

$ sudo ifup ppp0=pots
...

The interface ppp0 is up and connected to the Internet with PPP.

$ sudo ifdown ppp0=pots
...

The interface ppp0 is down and disconnected from the Internet.

You should check the /etc/network/run/ifstate file for the current network configuration state of the ifupdown system.

[Warning] Warning

You may need to adjust numbers at the end of eth*, ppp*, etc. if you have multiple network interfaces.

6.4.7. Scripting with the ifupdown

The ifupdown system automatically runs scripts installed in /etc/network/*/ while exporting environment variables to scripts:

Table 6.14.  List of environment variables passed by the ifupdown system

environment variable

value passed

IFACE

physical name (interface name) of the interface being processed.

LOGICAL

logical name (configuration name) of the interface being processed.

ADDRFAM

address family of the interface.

METHOD

method of the interface. (e.g., "static")

MODE

"start" if run from ifup, "stop" if run from ifdown.

PHASE

as per MODE, but with finer granularity, distinguishing the pre-up, post-up, pre-down and post-down phases.

VERBOSITY

indicates whether "--verbose" was used; set to 1 if so, 0 if not.

PATH

the command search path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

IF_<OPTION>

the value for the corresponding option under the iface stanza.


Here, each environment variable name, IF_<OPTION>, is created from the name for the corresponding option by prepending "IF_", converting the case to the upper case, replacing hyphens to underscores, and discarding non-alphanumeric characters.

The ifupdown-extra package (see Section 6.3.14, “The ifupdown-extra package”) uses these environment variables to extend the functionality of the ifupdown package. The ifmetric package (see Section 6.4.2, “The ifmetric”) installs the /etc/network/if-up.d/ifmetric script which sets the metric via the IF_METRIC variable. The guessnet package (see Section 6.4.8, “Mapping with guessnet”), which provides simple and powerful framework for the auto-selection of the network configuration via the mapping mechanism, also uses these.

[Note] Note

For more specific examples of custom network configuration scripts using these environment variables, you should check example scripts in /usr/share/doc/ifupdown/examples/* and scripts used in the ifscheme and ifupdown-scripts-zg2 packages. These additional scripts have some overlaps of functinalities with the basic ifupdown-extra and guessnet packages. If you install these additional scripts, you should customize these scripts to avoid interferences.

6.4.8. Mapping with guessnet

Instead of manually choosing configuration as described in Section 6.4.6, “The manually switchable network configuration”, you can use the mapping mechanism described in Section 6.4.5, “The mapping stanza” to select network configuration automatically with custom scripts.

The guessnet-ifupdown(8) command provided by the guessnet package is designed to be used as a mapping script and provides powerful framework to enhance the ifupdown system.

  • you list test condition as the value for guessnet options for each network configuration under iface stanza.

  • mapping will chose the iface with first non-ERROR result as the network configuration.

This dual usage of the /e/n/i file by the mapping script, guessnet-ifupdown, and the original network configuration infrastructure, ifupdown, does not cause negative impacts since guessnet options only export extra environment variables to scripts run by the ifupdown system. See details in guessnet-ifupdown(8).

[Note] Note

When multiple guessnet option lines are required in /e/n/i, use option lines started with guessnet1, guessnet2, and so on, since the ifupdown package does not allow starting strings of option lines to be repeated.

6.5. The network configuration for desktop

6.5.1. Automatic network configuration

There are independent automatic network configuration tools, such as NetworkManager (NM) (network-manager and associated packages) and Wicd (wicd package) which manage network connection via daemon independen of the ifupdown package. They allow easy management of wireless connections with nice GUI user interfaces.

[Caution] Caution

These automatic network configuration tools are aimed primarily for mobile desktop users on laptops and is not intended for usage on servers.

The configuration of NM is described in /usr/share/doc/network-manager/README.Debian. Essentially:

  1. Make desktop user, e.g. foo, belong to group "netdev".

    • "sudo adduser foo netdev"

  2. Keep configuration of /e/n/i as simple as:

    auto lo
    iface lo inet loopback
    
    auto eth0
    iface eth0 inet dhcp
    
  3. Restart NM.
    • "sudo /etc/init.d/network-manager restart"

[Note] Note

Only interfaces which are not listed in /e/n/i or which have been configured with "auto ..." or "allow-hotplug ..." and "iface ... inet dhcp" (with no other options) are managed by NM to avoid conflict with ifupdown.

[Caution] Caution

NM may not be compatible with esoteric configurations of ifupdown in /e/n/i such as ones described below. Having even "hostname" stanza for DHCP controlled interface as described in Section 6.3.4, “The network interface served by the DHCP” caused NM to ignore such interface in lenny. Check BTS of network-manager for current issues and limitations of NM.

The configuration of Wicd is described in /usr/share/doc/wicd/README.Debian. Essentially:

  1. Make configuration in /e/n/i only as:

    auto lo
    iface lo inet loopback
    

6.5.2. GUI network configuration tools

The capability of default GUI network configuration tools for each desktop tends to be limited to basic configurations such as static IP or DHCP. They actually overwrite contents of /e/n/i file behind you. Please check how they change e/n/i file by yourself.

[Caution] Caution

They may not understand complicated advanced configuration done manually in /e/n/i file.

6.6. The low level network configuration

6.6.1. Iproute2 commands

Iproute2 commands offer complete low-level network configuration capabilities. Here is a translation table from obsolete net-tools commands to new iproute2 commands.

Table 6.15.  Translation table from obsolete net-tools commands to new iproute2 commands.

net-tools

iproute2

manipulation

arp(8)

ip neigh

ARP or NDISC cache entry.

ifconfig(8)

ip addr

protocol (IP or IPv6) address on a device.

ipmaddr

ip maddr

multicast address.

iptunnel

ip tunnel

tunnel over IP.

route(8)

ip route

routing table entry.

nameif(8)

ifrename(8)

name network interfaces based on MAC addresses.

mii-tool(8)

ethtool(8)

Ethernet device settings.


See ip(8) and IPROUTE2 Utility Suite Howto.

6.6.2. Safe lower level network operations

You may use the lower level network commands as follows safely since they do not change network configuration:

Table 6.16.  List of lower level network commands.

command

effects

"ifconfig" without arguments

displays the status of active interfaces (IPv4)

"route -n"

display all the routing table in numerical addresses (IPv4)

"arp" without arguments

displays the current content of the ARP cache tables (IPv4)

"ip link show"

displays the link status of active interfaces (IPv4/v6)

"ip addr show"

displays the link and address status of active interfaces (IPv4/v6)

"ip route show"

displays the routing table (IPv4/v6)

"ip neigh"

displays the current content of the ARP cache tables (IPv4/v6)

"plog"

display ppp daemon log

"ping yahoo.com"

check Internet connection to yahoo.com

"whois yahoo.com"

check who registered yahoo.com in the domains database

"tracepath yahoo.com"

trace Internet connection to yahoo.com

"traceroute yahoo.com"

trace Internet connection to yahoo.com

"dig [@dns-server.com] example.com [{a|mx|any}]"

check example.com DNS records by dns-server.com for a {a|mx|any} record

"iptables -L -n"

check packet filter

"netstat -a"

find all open ports

"netstat -l --inet"

find listening ports

"netstat -ln --tcp"

find listening TCP ports (numeric)

"dlint example.com"

check DNS zone information of examle.org


[Tip] Tip

Some of these lower level network configuration tools reside in /sbin/. You may need to issue full command path such as /sbin/ifconfig or add /sbin to the PATH list in your ~/.bashrc.

6.7. Network optimization

Generic network optimization is beyond the scope of this documentation. I will touch only subjects pertinent to the consumer grade connection.

Table 6.17.  List of network optimization tools.

packages

popcon

size

description

iftop

V:0.8, I:5

108

displays bandwidth usage information on an network interface

iperf

V:0.3, I:2

208

Internet Protocol bandwidth measuring tool

apt-spy

V:0.2, I:1.8

204

writes a sources.list file based on bandwidth tests

ifstat

V:0.19, I:1.0

88

InterFace STATistics Monitoring

bmon

V:0.2, I:0.7

188

portable bandwidth monitor and rate estimator

ethstatus

V:0.14, I:0.8

84

script that quickly measures network device throughput

bing

V:0.13, I:0.8

96

Empirical stochastic bandwidth tester

bwm-ng

V:0.15, I:0.7

152

small and simple console-based bandwidth monitor

ethstats

V:0.07, I:0.4

52

console-based Ethernet statistics monitor

ipfm

V:0.03, I:0.14

156

a bandwidth analysis tool


6.7.1. Finding optimal MTU

The Maximum Transmission Unit (MTU) value can be determined experimentally with ping(8) with "-M do" option which sends ICMP packets with data size starting from 1500 (with offset of 28 bytes for the IP+ICMP header) and finding the largest size without IP fragmentation. For example:

$ ping -c 1 -s $((1500-28)) -M do www.debian.org
PING www.debian.org (194.109.137.218) 1472(1500) bytes of data.
From 192.168.11.2 icmp_seq=1 Frag needed and DF set (mtu = 1454)

--- www.debian.org ping statistics ---
0 packets transmitted, 0 received, +1 errors
  • ... try 1454 instead of 1500

  • The ping(8) command succeed

This process is Path MTU (PMTU) discovery (RFC1191)and the tracepath(8) command can automate this.

[Tip] Tip

The above example with PMTU value of 1454 is for my previous FTTH provider which used Asynchronous Transfer Mode (ATM) as its backbone network and served its clients with the PPPoE. The actual PMTU value depends on your environment, e.g., 1500 for the my new FTTH provider.

Table 6.18.  Basic guide lines of the optimal MTU value

network environment

MTU

rationale

Dial-up link (IP: PPP)

576

standard

Ethernet link (IP: DHCP or fixed)

1500

standard and default

Ethernet link (IP: PPPoE)

1492 (=1500-8)

2 bytes for PPP header and 6 bytes for PPPoE header

Ethernet link (ISP's backbone: ATM, IP: DHCP or fixed)

1462 (=48*31-18-8)

author's speculation: 18 for Ethernet header, 8 for SAR trailer.

Ethernet link (ISP's backbone: ATM, IP: PPPoE)

1454 (=48*31-8-18-8)

see "Optimal MTU configuration for PPPoE ADSL Connections" for rationale.


In addtion to these basic guide lines, you should know:

  • Any use of tunneling methods (VPN etc.) may reduce optimal MTU further by their overheads.

  • The MTU value should not exceed the experimentally determined PMTU value.

  • The bigger MTU value is generally better when other limitations are met.

6.7.2. Setting MTU

Here are examples for setting the MTU value from its default 1500 to 1454.

For the DHCP (see Section 6.3.4, “The network interface served by the DHCP”), you can replace pertinent iface stanza lines in the /e/n/i with, e.g.,:

iface eth0 inet dhcp
 hostname "mymachine"
 pre-up /sbin/ifconfig $IFACE mtu 1454

For the static IP (see Section 6.3.5, “The network interface with the static IP”), you can replace pertinent iface stanza lines in the /e/n/i with, e.g.,:

iface eth0 inet static
 address 192.168.11.100
 netmask 255.255.255.0
 broadcast 192.168.11.255
 gateway 192.168.11.1
 mtu 1454
 dns-domain lan
 dns-nameservers 192.168.11.1

For the direct PPPoE (see Section 6.2.5, “The PPPoE connection with pppoeconf”), you can replace pertinent mtu line in the /etc/ppp/peers/dsl-provider with:

mtu 1454

The maximum segment size (MSS) is used as an alternative measure of packet size. The relationship between MSS and MTU are:

  • MSS = MTU - 40 for IPv4

  • MSS = MTU - 60 for IPv6

[Note] Note

The iptables(8) (see Section 6.8, “Netfilter”) based optimization can clamp packet size by the MSS and is useful for the router.

6.7.3. WAN TCP optimization

The TCP throughput can be maximized by adjusting TCP buffer size parameters as described in "TCP Tuning Guide" and "TCP tuning" for the modern high-bandwidth and high-latency WAN. So far, the current Debian default settings serve well even for my LAN connected by the fast 100M bps FTTP service.

6.8. Netfilter

Netfilter provides infrastructure for stateful firewall and network address translation (NAT) with Linux kernel modules (see Section 4.5.11, “The kernel module initialization”).

Table 6.19.  List of firewall tools.

packages

popcon

size

description

iptables

V:20, I:99

1356

administration tools for netfilter

iptstate

V:0.15, I:0.9

156

Tool to continuously monitor netfilter state. (similar to top(1))

shorewall

V:0.8, I:2

40

Shoreline Firewall, netfilter configuration file generator (recommended for etch)

shorewall-perl

V:0.07, I:0.2

492

Shoreline Firewall, netfilter configuration file generator (Perl-based, recommended for lenny)

shorewall-shell

I:0.9

356

Shoreline Firewall, netfilter configuration file generator (shell-based, alternative for lenny)

ipmasq

V:0.3, I:0.5

612

Simple set of init script to configure netfilter (old)


Main user space program of netfilter is iptables(8). You can manually configure netfilter interactively from shell, save its state with iptables-save(8), and restore it via init script with iptables-restore(8) upon system reboot.

Configuration helper scripts such as shorewall ease this process.

See documentation at http://www.netfilter.org/documentation/ (or in /usr/share/doc/iptables/html/):

[Tip] Tip

Although these were written for Linux 2.4, both iptables(8) command and netfilter kernel function apply for current Linux 2.6.

Chapter 7. Network applications

7.1. Web browsers

There are many web browser packages to access remote contents with Hypertext Transfer Protocol (HTTP).

Table 7.1.  List of web browsers.

package

popcon

size

description

iceweasel

V:31, I:59

3960

Web browser (X) (unbranded Firefox)

iceape-browser

V:3, I:6

35436

Web browser (X) (unbranded Mozilla browser)

epiphany-browser

V:11, I:45

32

Web browser (X) (Gnome HIG compliant browser)

galeon

V:1.2, I:2

1732

Web browser (X) (Gnome browser)

konqueror

V:11, I:21

6056

Web browser (X) (KDE browser)

w3m

V:20, I:85

1968

Web browser (text)

lynx

V:4, I:25

44

, ,

elinks

V:2, I:6

1444

, ,

links

V:2, I:9

1372

, ,

links2

V:1.0, I:4

3280

, ,


7.1.1. Browser configuration

You may be able to use following special URL strings for some browsers to confirm their settings.

  • "about:"

  • "about:config"

  • "about:plugins"

Debian offers many free browser plugin packages in the main component which can handle not only Java (software platform) and Flash but also MPEG, MPEG2, MPEG4, DivX, Windows Media Video (.wmv), QuickTime (.mov), MP3 (.mp3), Ogg/Vorbis files, DVDs, VCDs, etc. Debian also offers helper programs to install non-free browser plugin packages as contrib or non-free components.

Table 7.2.  List of browser plugin packages.

package

popcon

size

component

description

icedtea-gcjwebplugin

V:0.6, I:0.8

204

main

Java plugin using Hotspot JIT

java-gcj-compat-plugin

V:0.6, I:1.8

104

main

Java plugin using the gij runtime

sun-java5-plugin

I:4

NOT_FOUND

non-free

Java plugin for Sun's Java SE 5.0 (i386 only)

sun-java6-plugin

I:6

NOT_FOUND

non-free

Java plugin for Sun's Java SE 6 (i386 only)

swfdec-mozilla

V:5, I:9

244

main

Flash plugin based on libswfdec

mozilla-plugin-gnash

V:0.6, I:1.7

180

main

Flash plugin based on Gnash

flashplugin-nonfree

V:1.5, I:9

128

contrib

Flash plugin helper to install Adobe Flash Player (i386, amd64 only)

mozilla-bonobo

V:0.19, I:0.4

168

main

Mozilla plugin support for Gnome Bonobo components

mozilla-plugin-vlc

V:2, I:5

160

main

Multimedia plugin based on VLC media player

totem-mozilla

V:13, I:45

241

main

Multimedia plugin based on Gnome's Totem media player

gecko-mediaplayer

V:0.10, I:0.16

688

main

Multimedia plugin based on (GNOME) MPlayer

nspluginwrapper

V:2, I:3

372

contrib

A wrapper to run i386 Netscape plugins on amd64 architecture


[Tip] Tip

Although use of above Debian packages are much easier, browser plugins can be still manually enabled by installing "*.so" into plugin directories (e.g., /usr/lib/iceweasel/plugins/) and restarting browsers.

Some web sites refuse to be connected based on the user-agent string of your browser. You can work around this situation by spoofing the user-agent string. For exaple, you can do this by adding:

user_pref{"general.useragent.override","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"};

into user configuration files such as ~/.gnome2/epiphany/mozilla/epiphany/user.js or ~/.mozilla/firefox/*.default/user.js. Alternatively, you can add and reset this variable by typing "about:config" into URL and right clicking its display contents.

[Caution] Caution

Spoofed user-agent string may cause bad side effects with Java.

7.2. The mail system

[Caution] Caution

If you are to set up the mail server to exchange mail directly with the Internet, you should be better than reading this elementary document.

7.2.1. Modern mail service basics

In order to contain spam (unwanted and unsolicited e-mail) problems, many ISPs which provide consumer grade Internet connection are implementing counter measures:

  • The smarthost service for their customers to send message uses the message submission port (587) specified in rfc4409 with the password (SMTP AUTH service) specified in rfc4954.

  • The SMTP port (25) connection from their internal network hosts (except ISP's own outgoing mail server) to the Internet are blocked.

  • The SMTP port (25) connection to the ISP's incoming mail server from some suspicious external network hosts are blocked. (The connection from hosts on the dynamic IP address range used by the dial-up and other consumer grade Internet connections are the first ones to be blocked.)

When configuring your mail system or resolving mail delivery problems, you must consider these new limitations.

In light of these hostile Internet situation and limitations, some independent Internet mail ISPs such as Yahoo.com and Gmail.com offer the secure mail service which can be connected from anywhere on the Internet using Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL) :

  • The smarthost service for their customers to send message uses the SMTP/SSL port (465) or the message submission port (587) with the password (SMTP AUTH service).

  • The incoming mail is accessible at the TLS/POP3 port (995) with POP3.

[Caution] Caution

It is not realistic to run SMTP server on consumer grade network to send mail directly to the remote host reliably. They are very likely to be rejected. You must use some smarthost services offered by your connection ISP or independent mail ISPs. For the simplicity, I will assume that the smarthost is located at "smtp.hostname.dom", requires SMTP AUTH, and uses the message submission port 587 in the following text.

7.2.2. Basic mail software choice

Table 7.3.  List of popular mail system for workstation.

package

popcon

size

function

exim4-daemon-light

V:57, I:67

928

Exim4 mail transport agent (MTA: Debian etch default)

exim4-base

V:59, I:70

1656

Exim4 documentation (text) and common files

exim4-doc-html

I:0.9

5756

Exim4 documentation (html)

exim4-doc-info

I:0.5

596

Exim4 documentation (info)

postfix

V:15, I:17

3196

Postfix mail transport agent (MTA: alternative)

postfix-doc

I:2

3124

Postfix documentation (html+text)

sasl2-bin

V:2, I:5

444

Cyrus SASL API implementation (supplement postfix for SMTP-AUTH)

cyrus-sasl2-doc

I:4

288

Cyrus SASL - documentation

fetchmail

V:3, I:7

1812

Remote mail retrieval and forwarding utility

procmail

V:16, I:87

360

Mail filter utility

mutt

V:21, I:84

5396

Mail user agent (MUA) to read/write the mail usually used with vim


The choice between exim4-* and postfix packages is really up to you.

Although the popcon vote count of exim4 looks several times popular than that of postfix, this does not mean postfix is not popular with Debian developers. The Debian server system uses both exim4 and postfix. The mail header analysis of mailing list postings from prominent Debian developers also indicate both of these MTAs are as popular.

The exim4-* are known to have very small memory consumption and very flexible for its configuration. The postfix is known to be compact, fast, simple, and secure. Both come with ample documentation and are as good in quality and license.

7.2.3. The mail configuration strategy for workstation

The most simple mail configuration is that the mail is sent to the ISP's smarthost and received from ISP's POP3 server by the MUA itself. This type of configuration is popular with full featured GUI based mail user agent (MUA) such as icedove, evolution, etc.. If you need to filter mail by their types, you use MUA's filtering function. For this case, the local mail transport agent (MTA) need to do local delivery only.

The alternative mail configuration is that the mail is sent via local MTA to the ISP's smarthost and received from ISP's POP3 by fetchmail(1) to the local mailbox. If you need to filter mail by their types, you use procmail(1) to filter mail into separate mailboxes. This type of configuration is popular with simple console based MUA such as mutt, gnus, etc., although this is possible with any MUAs. For this case, the local MTA need to do both smarthost delivery and local delivery.

7.2.3.1. The configuration of exim4

For Internet via smarthost, you (re)configure exim4-* packages as follows:

$ sudo /etc/init.d/exim4 stop
$ sudo dpkg-reconfigure exim4-conf
  • Chose "mail sent by smarthost; received via SMTP or fetchmail".

  • Set "IP address or host name of the outgoing smarthost:" to "smtp.hostname.dom:587".

$ sudo vim /etc/exim4/passwd.client
  • Create password entries for the smarthost.

$ cat /etc/exim4/passwd.client
^smtp.*\.hostname\.dom:username@hostname.dom:password
$ sudo /etc/init.d/exim4 start

The host name in /etc/exim4/passwd.client should not be the alias. You check the real host name with:

$ host smtp.hostname.dom
smtp.hostname.dom is an alias for smtp99.hostname.dom.
smtp99.hostname.dom has address 123.234.123.89

I use regex in /etc/exim4/passwd.client to work around the alias issue so even if the ISP moves host pointed by the alias, SMTP AUTH will likely be working.

[Tip] Tip

Local customization file /etc/exim4/exim4.conf.localmacros may be created to set MACROs. For example, Yahoo's mail service is said to require "MAIN_TLS_ENABLE = true" and "AUTH_CLIENT_ALLOW_NOTLS_PASSWORDS=yes" in it.

[Note] Note

Please read the official guide at: /usr/share/doc/exim4-base/README.Debian.gz and update-exim4.conf(8).

[Caution] Caution

You should execute update-exim4.conf(8) after updating exim4 configuration files in /etc/exim4.

7.2.3.2. The configuration of postfix with SASL

For Internet via smarthost, you should first read postfix documentation and key manual pages:

Table 7.4.  List of important postfix manual pages

command

function

postfix(1)

Postfix control program

postconf(1)

Postfix configuration utility

postconf(5)

Postfix configuration parameters

postmap(1)

Postfix lookup table maintenance

postalias(1)

Postfix alias database maintenance


You (re)configure postfix and sasl2-bin packages as follows:

$ sudo /etc/init.d/postfix stop
$ sudo dpkg-reconfigure postfix
  • Chose "Internet with smarthost"

  • Set "SMTP relay host (blank for none):" to "[smtp.hostname.dom]:587"

$ sudo postconf -e 'smtp_sender_dependent_authentication = yes'
$ sudo postconf -e 'smtp_sasl_auth_enable = yes'
$ sudo postconf -e 'smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd'
$ sudo postconf -e 'smtp_sasl_type = cyrus'
$ sudo vim /etc/postfix/sasl_passwd
  • Create password entries for the smarthost.

$ cat /etc/postfix/sasl_passwd
[smtp.hostname.dom]:587     username:password
$ sudo postmap hush:/etc/postfix/sasl_passwd
$ sudo /etc/init.d/postfix start

Here the use of [ ] in the dpkg-reconfigure dialogue and /etc/postfix/sasl_passwd ensures not to check MX record but directly use exact hostname specified. Read more for "Enabling SASL authentication in the Postfix SMTP client" in usr/share/doc/postfix/html/SASL_README.html.

7.2.3.3. The mail address configuration

There are a few mail address configuration files for mail transport, delivery and user agents.

Table 7.5.  List of mail address related configuration files.

file

function

application

/etc/mailname

default host name for (outgoing) mail

Debian specific, mailname(5)

/etc/email-addresses

host name spoofing for outgoing mail

exim(8) specific, exim4-config_files(5)

/etc/postfix/generic

host name spoofing for outgoing mail

postfix(1) specific, activated after postmap(1) command execution.

/etc/aliases

account name alias for incoming mail

general, activated after newaliases(1) command execution.


The mailname in the /etc/mailname file is usually a fully qualified domain name (FQDN) that resolves to one of the host's IP addresses. The mobile workstation which does not have a hostname with resolvable IP address, set this mailname to the value of "hostname -f". (This is safe choice and works for both exim4-* and postfix.)

[Tip] Tip

The contents of /etc/mailname is used by many non-MTA programs for their default behavior. For mutt, set "hostname" and "from" variables in ~/muttrc file to override the mailname value. For devscripts package programs such as bts and dch, export environment variables "DEBFULLNAME" and "DEBEMAIL" to override it.

When setting the mailname to "hostname -f", the spoofing of the source mail address via MTA can be realized by:

  • /etc/email-addresses file for exim4(8) as explained in the exim4-config_files(5), and

  • /etc/postfix/generic file for postfix(1) as explained in the generic(5).

For postfix, the following extra steps are needed:

# postmap hash:/etc/postfix/generic
# postconf -e 'smtp_generic_maps = hash:/etc/postfix/generic'
# postfix reload

You check filters using:

  • exim(8) with -brw, -bf, -bF, -bV, ... options.

  • postmap(1) with -q option.

[Tip] Tip

Exim comes with several utility programs such as exiqgrep(8) and exipick(8). See "dpkg -L exim4-base|grep man8/" for available commands.

7.2.4. Tips for managing the mail

7.2.4.1. Basic MTA operations

There are several basic MTA operations. Some may be performed via sendmail(1) compatibility interface.

Table 7.6.  List of basic MTA operation.

exim command

postfix command

description

sendmail

sendmail

Read mail from standard input and arrange for delivery. (-bm)

mailq

mailq

List the mail queue with status and queue ID. (-bp)

newaliases

newaliases

Initialize alias database. (-I)

exim4 -q

postqueue -f

flush waiting mail (-q)

exim4 -qf

postsuper -r ALL deferred; postqueue -f

flush all mail

exim4 -qff

postsuper -r ALL; postqueue -f

flush even frozen mail

exim4 -Mg queue_id

postsuper -h queue_id

freeze one message by its queue ID

exim4 -Mrm queue_id

postsuper -d queue_id

remove one message by its queue ID

---

postsuper -d ALL

remove all messages


For the script in /etc/ppp/ip-up.d/*, "flush all mail" may be good idea.

7.2.4.2. Basic MUA -- Mutt

Use mutt as the mail user agent (MUA) in combination with vim. Customize with ~/.muttrc; for example:

# use visual mode and "gq" to reformat quotes
set editor="vim -c 'set tw=72 et ft=mail'"
#
# header weeding taken from the manual (Sven's Draconian header weeding)
#
ignore *
unignore from: date subject to cc
unignore user-agent x-mailer
hdr_order from subject to cc date user-agent x-mailer
set hostname=spoof.example.org
set from="First Last <username@example.org>"
....

Add the following to /etc/mailcap or $HOME/.mailcap to display HTML mail and MS Word attachments inline:

text/html; lynx -force_html %s; needsterminal;
application/msword; /usr/bin/antiword '%s'; copiousoutput; description="Microsoft Word Text"; nametemplate=%s.doc

7.2.4.3. Redeliver mbox contents

You need to manually deliver mails to the sorted mailboxes in your home directory from /var/mail/<username> if your home directory became full and procmail failed. After making disk space in the home directory, run:

# /etc/init.d/${MAILDAEMON} stop
# formail -s procmail </var/mail/<username>
# /etc/init.d/${MAILDAEMON} start

7.2.5. Choices of software for the mail

For mail system programs, there are many alternatives developed with different priority. Here is the overview.

7.2.5.1. MTA

Table 7.7.  List of MTA.

package

popcon

size

capability

exim4-daemon-light

V:57, I:67

928

full

postfix

V:15, I:17

3196

full (security)

exim4-daemon-heavy

V:1.8, I:2

1040

full (flexible)

sendmail-bin

V:2, I:2

2080

full (only if you are already familiar)

nullmailer

V:0.6, I:0.8

452

strip down, no local mail

ssmtp

V:0.8, I:1.2

8

strip down, no local mail

nbsmtp

V:0.3, I:0.4

120

?

courier-mta

V:0.2, I:0.2

4004

very full (web interface etc.)

xmail

V:0.15, I:0.19

824

light

masqmail

V:0.05, I:0.07

568

light

esmtp

V:0.08, I:0.2

156

light

esmtp-run

V:0.06, I:0.09

8

light (sendmail compatibility extension to esmtp)

msmtp

V:0.2, I:0.6

336

light

msmtp-mta

V:0.06, I:0.10

52

light (sendmail compatibility extension to msmtp)


7.2.5.2. MUA

If you subscribe to Debian related mailing list, it may be a good idea to use such MUA as mutt and gnus which are the de facto standard for the participant and known to behave as expected.

Table 7.8.  List of MUA.

package

popcon

size

type

iceweasel

V:31, I:59

3960

X GUI (unbranded Firefox)

evolution

V:17, I:45

9324

X GUI (part of a groupware suite)

icedove

V:9, I:14

38108

X GUI (unbranded Thunderbird)

mutt

V:21, I:84

5396

character terminal probably with vim

gnus

V:0.04, I:0.7

6272

character terminal under (x)emacs


7.2.5.3. The remote mail retrieval and forward utility

Table 7.9.  List of remote mail retrieval and forward utilities.

package

popcon

size

capability

fetchmail

V:3, I:7

1812

mail retriever (POP3, APOP, IMAP) (de facto)

getmail4

V:0.2, I:0.6

632

mail retriever (POP3, IMAP4, and SDPS)

mailfilter

V:0.02, I:0.08

352

mail retriever (POP3) with with regex filtering capability

mpop

V:0.02, I:0.06

360

mail retriever (POP3) and MDA with filtering capability


The fetchmail is the current de facto standard for the remote mail retrieval utility. The SysV init script in /etc/init.d/fetchmail will start a fetchmail daemon running as the user fetchmail to fetch mail from multiple POP3 accounts on multiple ISPs, if the configuration file /etc/fetchmailrc is present in the system. If the configuration file is not present, nothing is started.

[Note] Note

If your email headers are contaminated by ^M due to your ISP's mailer, add "stripcr" to your options in $HOME/.fetchmailrc:

options fetchall no keep stripcr

7.2.5.4. MDA

Table 7.10.  List of MDA.

package

popcon

size

description

procmail

V:16, I:87

360

MDA with filter (de facto)

mailagent

V:0.4, I:6

1688

MDA with perl filter

maildrop

V:0.4, I:0.8

1040

MDA with structured filtering language


The procmail is the current de facto standard for the mail filter utility. One needs to create $HOME/.procmailrc for each account that uses it. For example:

# All delivery to Qmail style Maildir.  i.e. followed by /
# No lock needed
MAILDIR=$HOME/Mail
DEFAULT=$MAILDIR/Inbox/
LOGFILE=$MAILDIR/Maillog

:0
* ^Resent-Sender.*debian-devel-request@lists.debian.org
debian-devel/

:0
Inbox/

7.2.5.5. POP3/IMAP4 server

If you are to run a private server on LAN, you may consider to run POP3 / IMAP4 server for delivering mail to LAN clients.

Table 7.11.  List of POP3/IMAP4 servers.

package

popcon

size

type

description

qpopper

V:1.2, I:5

644

POP3

Qualcomm enhanced version

courier-pop

V:1.0, I:2

232

POP3

support only the maildir format

ipopd

V:0.15, I:0.2

204

POP3

formerly part of the University of Washington IMAP package

cyrus-pop3d-2.2

V:0.16, I:0.3

856

POP3

part of the Cyrus IMAPd suite

xmail

V:0.15, I:0.19

824

POP3

ESMTP/POP3 mail server

courier-imap

V:3, I:4

1604

IMAP

This provides access to email stored in Maildirs

uw-imapd

V:1.0, I:5

272

IMAP

the University of Washington IMAP

cyrus-imapd-2.2

V:0.5, I:0.7

2636

IMAP

part of the Cyrus IMAPd suite


7.3. The print server and utility

In the old Unix-like system, the BSD Line printer daemon was the standard. Since the standard print out format of the free software is PostScript on the Unix like system, some filter system was used along with Ghostscript to enable printing to the non-PostScript printer.

Recently, Common UNIX Printing System (CUPS) is the new de facto standard. The CUPS uses Internet Printing Protocol (IPP). The IPP is now supported by other OSs such as Windows XP and Mac OS X and has became new cross-platform de facto standard for remote printing with bi-directional communication capability.

The standard printable data format for the application on the Debian system is the PostScript (PS) which is a page description language. The data in PS format is fed into the Ghostscript PostScript interpreter to produce the printable data specific to the printer. See: Section 12.3.1, “The Ghostscript” .

Thanks to the file format dependent auto-conversion feature of the CUPS system, simply feeding any data to the lpr command should generate the expected print output. (In CUPS, lpr can be enabled by installing the cups-bsd package.)

The Debian system has few notable packages for the print servers and utilities:

Table 7.12.  List of print servers and utilities.

package

popcon

size

function

port

lpr

V:3, I:4

440

BSD lpr/lpd (Line printer daemon)

printer (515)

lprng

V:1.2, I:1.4

3016

, , (Enhanced)

, ,

cups

V:18, I:23

10556

Internet Printing CUPS server

IPP (631)

cups-client

V:7, I:24

412

System V printer commands for CUPS: lp(1), lpstat(1), lpoptions(1), cancel(1), lpmove(8), lpinfo(8), lpadmin(8), ...

, ,

cups-bsd

V:5, I:21

176

BSD printer commands for CUPS: lpr(1), lpq(1), lprm(1), lpc(8)

, ,

cups-driver-gutenprint

V:4, I:15

1348

printer drivers for CUPS

Not applicable


[Tip] Tip

You can configure CUPS system by pointing your web browser to "http://localhost:631/" .

7.4. The remote access server and utility (SSH)

The Secure SHell (SSH) is the secure way to connect over the Internet. A free version of SSH called OpenSSH is available as the ssh package in Debian.

Table 7.13.  List of remote access server and utilities.

package

popcon

size

tool

comment

openssh-client

V:54, I:98

2080

ssh

Secure shell client

openssh-server

V:60, I:72

812

sshd

Secure shell server

ssh-askpass-fullscreen

V:0.11, I:0.5

92

ssh-askpass-fullscreen

asks user for a pass phrase for ssh-add (GNOME2)

ssh-askpass

V:0.6, I:4

156

ssh-askpass

asks user for a pass phrase for ssh-add (plain X)


[Tip] Tip

Please use the screen(1) program to enable remote shell process to survive the interrupted connection (see Section 10.1, “The screen program”).

[Caution] Caution

See Section 5.7.3, “Extra security measures for the Internet” if your SSH is accessible from Internet.

7.4.1. Basics of SSH

/etc/ssh/sshd_not_to_be_run must not be present if one wishes to run the OpenSSH server.

SSH has two authentication protocols:

Table 7.14.  List of SSH authentication protocols and methods.

SSH protocol

SSH method

description

SSH-1

RSAAuthentication

RSA identity key based user authentication

, ,

RhostsAuthentication

.rhosts based host authentication (insecure, disabled)

, ,

RhostsRSAAuthentication

.rhosts authentication combined with RSA host key (disabled)

, ,

ChallengeResponseAuthentication

RSA challenge-response authentication

, ,

PasswordAuthentication

password based authentication

SSH-2

PubkeyAuthentication

public key based user authentication

, ,

HostbasedAuthentication

.rhosts or /etc/hosts.equiv authentication combined with public key client host authentication (disabled)

, ,

ChallengeResponseAuthentication

challenge-response authentication

, ,

PasswordAuthentication

password based authentication


Be careful about these differences if you are using a non-Debian system.

See /usr/share/doc/ssh/README.Debian.gz, ssh(1), sshd(8), ssh-agent(1), and ssh-keygen(1) for details.

Following are the key configuration files:

Table 7.15.  List of SSH configuration files.

configuration file

function

/etc/ssh/ssh_config

SSH client defaults. See ssh_config(5).

/etc/ssh/sshd_config

SSH server defaults. See sshd_config(5).

$HOME/.ssh/authorized_keys

the lists of the default public SSH keys that clients use to connect to this account on this host.

$HOME/.ssh/identity

secret SSH-1 RSA key of the user.

$HOME/.ssh/id_rsa

secret SSH-2 RSA key of the user.

$HOME/.ssh/id_dsa

secret SSH-2 DSA key of the user.


[Tip] Tip

See ssh-keygen(1), ssh-add(1) and ssh-agent(1) for how to use public and secret SSH keys.

The following will start an ssh(1) connection from a client.

Table 7.16.  List of SSH client startup examples.

command

description

ssh username@hostname.domain.ext

connect with default mode

ssh -v username@hostname.domain.ext

connect with default mode with debugging messages

ssh -1 username@hostname.domain.ext

force to connect with SSH version 1

ssh -1 -o RSAAuthentication=no -l username hostname.domain.ext

force to use password with SSH version 1

ssh -o PreferredAuthentications=password -l username hostname.domain.ext

force to use password with SSH version 2


If you use the same user name on the local and the remote host, you can eliminate typing "username@". Even if you use different user name on the local and the remote host, you can eliminate it using "~/.ssh/config". For Debian Alioth service with account name "foo-guest", you set "~/.ssh/config" to contain:

Host alioth.debian.org svn.debian.org git.debian.org
    User foo-guest

For the user, ssh(1) functions as a smarter and more secure telnet(1). Unlike telnet command, ssh command does not bomb on the telnet escape character (initial default CTRL-]).

7.4.2. Port forwarding for SMTP/POP3 tunneling

To establish a pipe to connect to port 25 of remote-server from port 4025 of localhost, and to port 110 of remote-server from port 4110 of localhost through ssh, execute on the local machine:

# ssh -q -L 4025:remote-server:25 4110:remote-server:110 username@remote-server

This is a secure way to make connections to SMTP/POP3 servers over the Internet. Set the AllowTcpForwarding entry to yes in /etc/ssh/sshd_config of the remote host.

7.4.3. Connecting with fewer passwords -- RSA

One can avoid having to remember a password for each remote system by using RSAAuthentication (SSH-1 protocol) or PubkeyAuthentication (SSH-2 protocol).

On the remote system, set the respective entries, "RSAAuthentication yes" or "PubkeyAuthentication yes", in /etc/ssh/sshd_config.

Then generate authentication keys locally and install the public key on the remote system:

  • RSAAuthentication: RSA1 key for SSH-1 (deprecated because superseded.)

$ ssh-keygen
$ cat .ssh/identity.pub | ssh user1@remote "cat - >>.ssh/authorized_keys"
  • PubkeyAuthentication: RSA key for SSH-2

$ ssh-keygen -t rsa
$ cat .ssh/id_rsa.pub | ssh user1@remote "cat - >>.ssh/authorized_keys"
  • PubkeyAuthentication: DSA key for SSH-2 (deprecated because key is smaller and slow. Also see DSA-1571-1.)

$ ssh-keygen -t dsa
$ cat .ssh/id_dsa.pub | ssh user1@remote "cat - >>.ssh/authorized_keys"
[Note] Note

There are no more reasons to work around RSA patent using DSA since it has been expired. DSA stands for Digital Signature Algorithm and slow.

One can change the pass phrase later with "ssh-keygen -p". Make sure to verify settings by testing the connection. In case of any problem, use "ssh -v".

You can add options to the entries in authorized_keys to limit hosts and to run specific commands. See sshd(8) for details.

Note that SSH-2 has HostbasedAuthentication. For this to work, you must adjust the settings of HostbasedAuthentication to yes in both /etc/ssh/sshd_config on the server machine and /etc/ssh/ssh_config or $HOME/.ssh/config on the client machine.

7.4.4. Dealing with alien SSH clients

There are a few free SSH clients available for other platforms.

Table 7.17.  List of free SSH clients for other platforms.

environment

free SSH program

Windows

puTTY (http://www.chiark.greenend.org.uk/~sgtatham/putty/) (GPL)

Windows (cygwin)

SSH in cygwin (http://www.cygwin.com/) (GPL)

Macintosh Classic

macSSH (http://www.macssh.com/) (GPL)

Mac OS X

OpenSSH; use ssh in the Terminal application (GPL)


7.4.5. Setting up ssh-agent

It is safer to protect your SSH authentication key with a pass phrase. If it was not set, use ssh-keygen -p to set it.

Place your public key (e.g. ~/.ssh/id_rsa.pub) into ~/.ssh/authorized_keys on a remote host using a password-based connection to the remote host as described above.

$ ssh-agent bash
$ ssh-add ~/.ssh/id_rsa
Enter passphrase for /home/osamu/.ssh/id_rsa:
Identity added: /home/osamu/.ssh/id_rsa (/home/osamu/.ssh/id_rsa)
  • No passphrase needed from here on, e.g.:

$ scp foo user@remote.host:foo
  • No password requested.

  • Press ^D to terminating ssh-agent session.

For the X server, the normal Debian startup script executes ssh-agent as the parent process. So you only need to execute ssh-add once. For more, read ssh-agent(1)and ssh-add(1).

7.4.6. Troubleshooting SSH

If you have problems, check the permissions of configuration files and run ssh with the "-v" option.

Use the "-P" option if you are root and have trouble with a firewall; this avoids the use of server ports 1--1023.

If ssh connections to a remote site suddenly stop working, it may be the result of tinkering by the sysadmin, most likely a change in host_key during system maintenance. After making sure this is the case and nobody is trying to fake the remote host by some clever hack, one can regain a connection by removing the host_key entry from $HOME/.ssh/known_hosts on the local machine.

7.5. Other network application servers

Table 7.18.  List of other network application servers.

package

popcon

size

protocol

focus

telnetd

V:0.5, I:1.4

156

TELNET

TELNET server

telnetd-ssl

V:0.19, I:0.5

204

, ,

, , (SSL support)

nfs-kernel-server

V:13, I:23

396

NFS

Unix file sharing

nfs-user-server

V:0.6, I:1.0

292

, ,

, ,

samba

V:19, I:34

12380

SMB

windows file and printer sharing

netatalk

V:5, I:11

2304

ATP

apple/mac file and printer sharing (AppleTalk)

proftpd

V:5, I:9

196

FTP

general file download

wu-ftpd

V:0.6, I:0.8

820

, ,

, ,

apache2-mpm-prefork

V:34, I:40

572

HTTP

general web server

apache2-mpm-worker

V:5, I:6

580

, ,

, ,

squid

V:5, I:6

1816

, ,

general web proxy server

squid3

V:0.8, I:1.0

2444

, ,

, ,

slpd

V:0.2, I:0.4

228

SLP

OpenSLP Server as LDAP server

bind9

V:10, I:17

768

DNS

IP address for other hosts

dhcp3-server

V:4, I:8

804

DHCP

IP address of client itself


Common Internet File System Protocol (CIFS) is the same protocol as Server Message Block (SMB).

[Tip] Tip

Use of proxy server such as squid is much more efficient for saving bandwidth than use of local mirror server with the full Debian archive contents.

7.6. Other network application clients

Table 7.19.  List of network application clients.

package

popcon

size

protocol

focus

netcat

V:4, I:83

36

TCP/IP

TCP/IP swiss army knife

stunnel4

V:0.5, I:1.6

504

SSL

Universal SSL Wrapper

telnet

V:13, I:90

200

TELNET

TELNET client

telnet-ssl

V:0.3, I:1.3

244

, ,

, , (SSL support)

nfs-common

V:50, I:82

564

NFS

Unix file sharing

smbclient

V:5, I:40

18876

SMB

MS windows file and printer sharing client

smbfs

V:4, I:27

4168

, ,

Mount and umount commands for remote MS windows file

ftp

V:10, I:87

160

FTP

FTP client

lftp

V:1.3, I:6

1712

, ,

, ,

ncftp

V:1.7, I:8

1164

, ,

Full screen FTP client

wget

V:28, I:99

1944

HTTP and FTP

Web downloader

curl

V:4, I:18

320

, ,

, ,

dog

V:0.08, I:0.3

76

HTTP

Web uploader (cat with URL support)

bind9-host

V:42, I:90

172

DNS

The host command from bind9, priority standard

dnsutils

V:12, I:91

388

, ,

The dig command from bind, priority standard

host

V:1.3, I:3

180

, ,

The host command from dnsutils, priority extra

dhcp3-client

V:48, I:92

608

DHCP

Obtain IP address

ldap-utils

V:1.5, I:7

588

LDAP

Obtain data from LDAP server


7.7. The diagnosis of the system daemons

The telnet program enables manual connection and diagnosis of the system daemons. E.g.:

$ telnet mail.ispname.net pop3

The following RFCs provide required knowledge to text each system daemon.

Table 7.20.  List of popular RFCs.

RFC

description

rfc1939 and rfc2449

POP3 service

rfc3501

IMAP4 service

rfc2821 (rfc821)

SMTP service

rfc2822 (rfc822)

Mail file format

rfc2045

Multipurpose Internet Mail Extensions (MIME)

rfc819

DNS service

rfc2616

HTTP service

rfc2396

URI definition


The port usage is described in /etc/services.

[Note] Note

For testing TLS/SSL services such as HTTPS, you need TLS/SSL enabled telnet program.

Chapter 8. The X window system

The X window system on the Debian system is based on the source from X.Org. As of January 2009, they are X11R7.1(etch), X11R7.3(lenny) and X11R7.3(sid).

8.1. Key packages

There are a few (meta)packages provided to ease installation.

Table 8.1.  List of key (meta)packages for X window.

(meta)package

popcon

size

description

xorg

I:56

52

This metapackage provides the X libraries, an X server, a set of fonts, and a group of basic X clients and utilities.

xserver-xorg

V:20, I:62

696

This package provides the full suits of the X server and its configuration.

xbase-clients

V:25, I:64

84

This package provides a miscellaneous assortment of the X clients.

x11-common

V:48, I:88

896

This package contains the filesystem infrastructure for the X window system.

xorg-docs

I:12

5068

This package contains miscellaneous documentation for the X.Org software suite.

xspecs

I:1.8

6504

This package contains X protocol, extension, and library technical specifications.

menu

V:32, I:62

1956

This package generates the Debian menu for all menu-aware applications.

menu-xdg

I:60

76

This package converts the Debian menu structure to the freedesktop.org xdg menu structure.

xdg-utils

V:8, I:53

256

This package provides utilities to integrate desktop environment provided by the freedesktop.org.

gnome-desktop-environment

I:38

20

metapackage for the stadard GNOME desktop environment.

kde-core

I:14

40

metapackage for the core KDE desktop environment.

xfce4

I:5

48

metapackage for the Xfce lightweight desktop environment.

lxde-core

I:0.5

40

metapackage for the LXDE lightweight desktop environment.

fluxbox

V:1.3, I:3

4332

Fluxbox: package for highly configurable and low resource X window manager.


For the basics of X, refer to X(7), the LDP XWindow-User-HOWTO.

8.2. Setting up desktop environment

A desktop environment is usually a combination of a X window manager, a file manager, and a suite of compatible utility programs.

You can setup a full desktop environment such as GNOME, KDE, Xfce, or LXDE, from the aptitude under the task menu.

[Tip] Tip

Task menu may be out of sync with the latest package transition state under Debian unstable/testing environment. In such situation, you need to deselect some (meta)packages listed under aptitude(8) task menu to avoid package conflicts. When deselecting (meta)packages, you must select certain packages providing their dependencies if you need to keep them.

You may alternatively setup a simple environment manually just with a X window manager such as Fluxbox.

See Window Managers for X for the guide to the X window manager and the desktop environment.

8.2.1. Debian menu

Debian menu system provides a general interface for both text- and X-oriented programs with the update-menus(1) command from the menu package. Each package installs its menu data in the /usr/share/menu/ directory. See /usr/share/menu/README.

8.2.2. Freedesktop.org menu

Each package which is compliant to Freedesktop.org's xdg menu system installs its menu data provided by "*.desktop" under the /usr/share/applications/ directory. Modern desktop environments which are compliant to Freedesktop.org standard use these data to generate their menu using the xdg-utils package. See /usr/share/doc/xdg-utils/README.

8.2.3. Debian menu under GNOME desktop environment

In order to obtain access to the traditional Debian menu under GNOME desktop environment, you must install the menu-xdg package, click "System" -> "Preference" -> "Main Menu", and check the box for "Debian".

[Tip] Tip

You may need to do the similar for other modern desktop environments which are compliant to Freedesktop.org standard.

8.3. The server/client relationship

The X window system is activated as a combination of the server and client programs. The meaning for the words server and client with respect to the words local and remote requires attention here:

Table 8.2.  List of server/client terminology.

type

description

X server

a program run on a local host connected to the user's display and input devices.

X client

a program run on a remote host that processes data and talks to the X server.

application server

a program run on a remote host that processes data and talks to the clients.

application client

a program run on a local host connected to the user's display and input devices.


8.4. The X server

See xorg(1) for X server information.

8.4.1. The (re)configuration of the X server

To (re)configure an X server,

# dpkg-reconfigure --priority=low x11-common
# dpkg-reconfigure --priority=low xserver-xorg

will generate a new /etc/X11/xorg.conf file using dexconf(1).

If you have manually edited this /etc/X11/xorg.conf file but would like it to be automatically updated again, run the following command:

# sudo dpkg-reconfigure -phigh xserver-xorg                               

Please check your X configuration with respect to the specification of your monitor carefully. For the large high resolution CRT monitor, it is a good idea to set the refresh rate as high as your monitor can handle (85 Hz is great, 75 Hz is OK) to reduce flicker. For the LCD monitor, slower standard refresh rate (60Hz) is usually fine due to its slow response.

[Note] Note

Be careful not to use too high refresh rate which may cause fatal hardware failure of your monitor system.

8.4.2. The connection methods to the X server

There are several ways of getting the "X server" (display side) to accept connections from an "X client" (application side):

Table 8.3.  List of connection methods to the X server.

method

package

popcon

size

user

encryption

pertinent use

xhost command

xbase-clients

V:25, I:64

84

unchecked

no

deprecated

xauth command

xbase-clients

V:25, I:64

84

checked

no

for local connection via pipe

ssh -X command

openssh-client

V:54, I:98

2080

checked

yes

for remote network connection

Gnome display manager

gdm

V:32, I:49

15215

checked

no(XDMCP)

for local connection via pipe

KDE display manager

kdm

V:10, I:13

1668

checked

no(XDMCP)

for local connection via pipe

X display manager

xdm

V:0.9, I:2

812

checked

no(XDMCP)

for local connection via pipe

WindowMaker display manager

wdm

V:20, I:85

1968

checked

no(XDMCP)

for local connection via pipe

Secure display manager

sdm

V:0.03, I:0.19

124

checked

yes

for remote SSH network connection (thin client)

Linux Terminal Server Project display manager

ldm

V:0.00, I:0.09

500

checked

yes

for remote SSH network connection (thin client)


[Warning] Warning

Do not use remote TCP/IP connection over unsecured network unless you have very good reason. A remote TCP/IP socket connection without encryption is prone to the eavesdropping attack and is disabled by default on the Debian system. Use ssh -X.

[Tip] Tip

You can dare to enable remote TCP/IP connection by setting DisallowTCP=false in /etc/gdm/gdm.conf and by removing -nolisten from lines found by "find /etc/X11 -type f -print0 | xargs -0 grep nolisten", if you are in the fully secured environment.

[Warning] Warning

Do not use XDMCP connection over unsecured network either. It sends data via UDP without encryption and prone to the eavesdropping attack.

8.5. Starting the X window system

The X Window system is usually started as an X session which is the combination of an X server and connecting X clients. For normal desktop system, both of them are executed on the workstation.

To start the X Window system,

  • startx command started from the command line, or

  • one of the X display manager daemon programs *dm started from the end of the start up script in /etc/rc?.d/ directory (? corresponding to the runlevel)

are used to start the X session. (The start up script for the display manager daemons checks the content of the /etc/X11/default-display-manager file before actually executing themselves.)

[Tip] Tip

See Section 9.3.5, “Specific locale only under X Window” for initial environment variables of the X display manager.

Essentially, all these programs execute the /etc/X11/Xsession script. Then the /etc/X11/Xsession script performs run-parts like action to execute scripts in the /etc/X11/Xsession.d/ directory. This is essentially an execution of a program which was first found in the following order with the exec command:

  1. The script specified as the argument of /etc/X11/Xsession by the X display manager, if it is defined.

  2. The ~/.xsession or ~/.Xsession script, if it is defined.

  3. The /usr/bin/x-session-manager command, if it is defined.

  4. The /usr/bin/x-window-manager command, if it is defined.

  5. The /usr/bin/x-terminal-emulator command, if it is defined.

This process is affected by the content of the /etc/X11/Xsession.options file. The exact programs to which these usr/bin/x-* commands point, are determined by the Debian alternative system and changed by update-alternatives --config x-session-manager etc.

8.5.1. Starting X session with gdm

The gdm program lets you select the session type (or desktop environment: Section 8.2, “Setting up desktop environment”), and language (or locale: Section 9.3, “The locale”) of the X session from its menu. It keeps the selected default value in ~/.dmrc as, e.g.:

[Desktop]
Session=default
Language=ja_JP.UTF-8

8.5.2. Customizing the X session (classic method)

On a system where /etc/X11/Xsession.options contains a line allow-user-xsession without preceding # characters, any user who defines ~/.xsession or ~/.Xsession will be able to customize the action of /etc/X11/Xsession by completely overiding the system code. The last command in the ~/.xsession file should use form of exec some-window/session-manager to start your favorite X window/session managers.

8.5.3. Customizing the X session (new method)

Here are new methods to customize the X session without completely overiding the system code as above.

  • The display manager gdm can select a specific session and set it as the argument of /etc/X11/Xsession.

  • The ~/.xsessionrc file is executed as a part of start up process (desktop independent.)

  • The ~/.gnomerc file is executed as a part of start up process. (Gnome desktop only)

  • The GUI program based session management software can use the ~/.gnome2/session file etc..

8.5.4. Connecting a remote X client via SSH

The use of ssh -X enables a secure connection from a local X server to a remote application server.

  • Set X11Forwarding entries to yes in /etc/ssh/sshd_config of the remote host, if you want to avoid -X command-line option.

  • Start the X server on the local host.

  • Open an xterm in the local host.

  • Run ssh to establish a connection with the remote site.

    localname @ localhost $ ssh -q -X loginname@remotehost.domain
    Password:
    .....
    
  • Run an X application command, e.g. gimp, on the remote site.

    loginname @ remotehost $ gimp &
    

This method allows the display of the remote X client output as if it were locally connected through a local UNIX domain socket.

8.5.5. Secure X terminal via Internet

Secure X terminal via Internet, which displays remotely run entire X desktop environment, can easily achieved by using specialized package such as sdm and ldm. Your local machine becomes a secure thin client to the remote application server connected via SSH.

If you want to add similar feature to your normal display manager gdm, create executable shell script at /usr/local/bin/ssh-session as:

#!/bin/sh -e
# Based on gdm-ssh-session in gdm source (GPL)
ZENITY=$(type -p zenity)
TARGETHOST=$($ZENITY --width=600 \
--title "Host to connect to" --entry \
--text "Enter the name of the host you want to log in to as user@host.dom:")
TARGETSESSION=$($ZENITY --width=600 --height=400 \
--title "Remote session name" --list --radiolist --text "Select one" \
--column " " --column "Session" --column "description" --print-column 2 \
TRUE "/etc/X11/Xsession" "Debian" \
FALSE "/etc/X11/xinit/Xclients" "RH variants" \
FALSE "gnome-session" "Gnome session" \
FALSE "xterm" "Safe choice" \
FALSE "rxvt" "Safe choice" \
FALSE "gnome-terminal" "Safe choice")
echo "Connecting to "$TARGETHOST" with $TARGETSESSION"
/usr/bin/ssh -A -X -T -n "$TARGETHOST" "$TARGETSESSION"
#SSH_ASKPASS=/usr/bin/ssh-askpass /usr/bin/ssh -A -X -T -n "$TARGETHOST" "$TARGETSESSION"

Then add followings to /etc/dm/Sessions/ssh.desktop:

[Desktop Entry]
Encoding=UTF-8
Name=SSH
Comment=This session logs you into a remote host using ssh
Exec=/usr/local/bin/ssh-session
Type=Application

8.6. Fonts in the X window

X window on the Debian system support two mechanisms for font management:

  • server side font management: the original core X11 font system and

  • client side font management: the new Xft 2 font system.

The core X11 font system provides backward compatibility with older applications such as Xterm with bitmap fonts. It is supported by installing pertinent font packages which trigger defoma scripts to generate required files such as "fonts.dir".

The Xft2 font system is used by all modern applications such as ones from GNOME, KDE, OpenOffice.org, etc.. It supports all fonts listed below (Section 8.6.1, “Basic fonts”, Section 8.6.2, “Additional fonts”, and Section 8.6.3, “CJK fonts”) with advanced features such as anti-aliasing. It has no configuration mechanism itself, rather it relies upon the fontconfig library to configure and customize fonts as described in fonts.conf(5). Actual rasterization is supported by the FreeType 2 font engine. These new X clients using Xft2 font system can talk to modern X server via the X Rendering Extension.

Table 8.4.  Table of packages to support X window font systems.

package

popcon

size

description

xfonts-utils

V:38, I:70

456

X Window System font utility programs

libxft2

V:40, I:76

144

Xft: FreeType-based font drawing library for X

libfreetype6

V:51, I:88

740

FreeType 2 font engine, shared library files

fontconfig

V:34, I:75

272

generic font configuration library - support binaries

fontconfig-config

V:24, I:82

250

generic font configuration library - configuration

defoma

V:22, I:85

449

Debian Font Manager -- automatic font configuration framework

x-ttcidfont-conf

I:30

156

TrueType configuration for X (for CJK support)


You can check actual font path for:

  • core X11 font path with "xset q"

  • fontconfig font default: "fc-match"

[Tip] Tip

"The Penguin and Unicode" is a good overview of modern X Window system. Other documentations at http://unifont.org/ should provide good information on Unicode fonts, Unicode-enabled software, internationalization, and Unicode usability issues on free/libre/open source (FLOSS) operating systems.

[Tip] Tip

You should rely on fontconfig infrastructure to configure fonts on the Debian system. Debian Font Manager (defoma) is only useful as font installation such as X logical font description (XLFD) data.

8.6.1. Basic fonts

There are 2 major types of computer fonts:

  • bitmap fonts (good for low resolution rasterization)

  • outline/stroke fonts (good for high resolution rasterization)

While scaling of bitmap fonts causes jugged image, scaling of outline/stroke fonts produces smooth image.

Bitmap fonts on the Debian system are provided by compressed X11 pcf bitmap font files having their file extension ".pcf.gz".

Outline fonts on the Debian system are provided by:

  • PostScript Type 1 font files having their file extension ".pfb" (binary font file) and ".afm" (font metrics file).

  • TrueType font files having their file extension ".ttf".

Table 8.5.  Table of corresponding PostScript Type 1 fonts.

font package

popcon

size

sans-serif font

serif font

monospace font

source of font

(PostScript)

Not applicable

NOT_FOUND

Helvetica

Times

Courier

Adobe

gsfonts

I:71

4792

Nimbus Sans L

Nimbus Roman No9 L

Nimbus Mono L

URW (Adobe compatible size)

gsfonts-x11

I:26

116

Nimbus Sans L

Nimbus Roman No9 L

Nimbus Mono L

X font support with PostScript Type 1 fonts.

t1-cyrillic

I:1.7

4996

Free Helvetian

Free Times

Free Courier

URW extended (Adobe compatible size)

lmodern

V:4, I:13

46180

LMSans*

LMRoman*

LMTypewriter*

scalable PostScript and OpenType fonts based on Computer Modern (from TeX)


Table 8.6.  Table of corresponding TrueType fonts.

font package

popcon

size

sans-serif font

serif font

monospace font

source of font

ttf-mscorefonts-installer

I:8

188

Arial

Times New Roman

Courier New

Microsoft (Adobe compatible size) (This installs non-free data)

ttf-liberation

I:23

1696

Liberation Sans

Liberation Serif

Liberation Mono

Liberation Fonts project (Microsoft compatible size)

ttf-freefont

I:15

3412

FreeSans

FreeSerif

FreeMono

GNU freefont (Microsoft compatible size)

ttf-bitstream-vera

I:20

732

Bitstream Vera Sans

Bitstream Vera Serif

Bitstream Vera Sans Mono

Bitstream, Inc.

ttf-dejavu

I:82

68

DejaVu Sans

DejaVu Serif

DejaVu Sans Mono

http://dejavu.sourceforge.net/, Bitstream with extended character code support

ttf-dejavu-core

I:31

2488

DejaVu Sans

DejaVu Serif

DejaVu Sans Mono

http://dejavu.sourceforge.net/ basic font style variants

ttf-dejavu-extra

I:31

5504

DejaVu Sans

DejaVu Serif

DejaVu Sans Mono

http://dejavu.sourceforge.net/ extra font style variants

ttf-unifont

I:2

16060

N/A

N/A

unifont

http://Unifoundry.com, GNU Unifont, with all printable character code in Unicode 5.1 Basic Multilingual Plane (BMP)


8.6.2. Additional fonts

The aptitude will help you find additional fonts easily:

  • by the short package list under "Packages which depend on defoma" under the defoma package list,

  • by the short package list under "Tasks" -> "Localization",

  • by the filtered flat package list of font data with regex on debtag: "~Gmade-of::data:font",

  • by the filtered flat package list of the BDF (bitmap) font packages with regex on package name: "~nxfonts-", or

  • by the filtered flat package list of the TrueType (outline) font packages with regex on package name: "~nttf-".

Since Free fonts are sometimes limited, installing or sharing some commercial TrueType fonts is an option for a Debian users. In order to make this process easy for the user, some convenience packages have been created:

  • ttf-mathematica4.1

  • ttf-mscorefonts-installer

You'll have a really good selection of TrueType fonts at the expense of contaminating your Free system with non-Free fonts.

8.6.3. CJK fonts

Here are some key points focused on CJK issues.

Table 8.7.  Table of key words used in CJK font names to indicate font types.

font type

Japanese font name

Chinese font name

Korean font name

sans-serif

gothic, ゴチック

hei, gothic

dodum, gulim, gothic

serif

mincho, 明朝

song, ming

batang


Font name such as "VL PGothic" with "P" is a proportional font which corresponds to the fixed width "VL Gothic" font.

For example, Shift_JIS code table comprises 7070 characters. They can be grouped into:

  • JIS X 0201 single-byte characters (191 characters),

  • JIS X 0208 double-byte characters (6879 characters).

Double-byte characters occupy double width on console terminals which uses CJK fixed width fonts. In order to cope with such situation, Hanzi Bitmap Font (HBF) File with file extension ".hbf" may be deployed for fonts containing single-byte and double-byte characters.

In order to save space for TrueType font files, TrueType font collection file with file extension ".ttc" may be used.

I order to cover complicated code space of characters, CID keyed Type 1 PostScript font is used with CMap files starting themselves with "%!PS-Adobe-3.0 Resource-CMap". This is rarely used for normal X display but used for PDF rendering etc. (see Section 8.7.2, “X utility applications”).

[Tip] Tip

The multiple glyphs are expected for some Unicode code points due to Han unification. One of the most annoying ones are "U+3001 IDEOGRAPHIC COMMA" and "U+3002 IDEOGRAPHIC FULL STOP" whose character positions differ among CJK countries. Configuring priority of Japanese centric fonts over Chinese ones using "~/.fonts.conf" should give peace of minds to Japanese.

8.7. X applications

8.7.1. X office applications

Here is a list of basic office applications (OO is OpenOffice.org):

Table 8.8.  List of basic X office applications

package

popcon

package size

description

type

openoffice.org-writer

V:21, I:52

22560

word processor

OO

openoffice.org-calc

V:21, I:51

16412

spreadsheet

OO

openoffice.org-impress

V:20, I:51

2940

presentation

OO

openoffice.org-base

V:19, I:50

8916

database management

OO

openoffice.org-draw

V:20, I:51

8476

vector graphics editor (draw)

OO

openoffice.org-math

V:19, I:50

1580

mathematical equation/formula editor

OO

abiword

V:3, I:5

8296

word processor

GNOME

gnumeric

V:4, I:9

7072

spreadsheet

GNOME

gimp

V:13, I:55

11928

bitmap graphics editor (paint)

GTK

inkscape

V:7, I:18

61376

vector graphics editor (draw)

GNOME

dia-gnome

V:2, I:5

596

flowchart and diagram editor

GNOME

mergeant

V:0.15, I:0.3

1412

database management

GNOME

planner

V:0.9, I:6

7388

project management

GNOME

kword

V:1.1, I:2

8528

word processor

KDE

kspread

V:1.1, I:2

9064

spreadsheet

KDE

kpresenter

V:0.9, I:2

4492

presentation

KDE

kexi

V:0.4, I:2

12092

database management

KDE

kivio

V:1.0, I:2

2224

flowchart and diagram editor

KDE

karbon

V:0.9, I:2

3956

vector graphics editor (draw)

KDE

krita

V:1.2, I:2

11404

bitmap graphics editor (paint)

KDE

kplato

V:0.2, I:2

3420

project management

KDE

kchart

V:0.8, I:2

4028

graph and chart drawing program

KDE

kformula

V:0.8, I:2

2284

mathematical equation/formula editor

KDE

kugar

V:0.8, I:2

1956

business quality report generator

KDE


8.7.2. X utility applications

Here is a list of basic utility applications which caught my eyes:

Table 8.9.  List of basic X utility applications

package

popcon

package size

description

type

evince

V:23, I:48

5264

document(pdf) viewer

GNOME

kpdf

V:8, I:16

2784

document(pdf) viewer

KDE

okular

V:1.0, I:1.7

2872

document(pdf) viewer

KDE4

evolution

V:17, I:45

9324

Personal information Management (groupware and email)

GNOME

kontact

V:2, I:14

4052

Personal information Management (groupware and email)

KDE

scribus

V:0.6, I:3

26708

desktop page layout editor

KDE

glabels

V:0.2, I:0.9

1045

label editor

GNOME

kbarcode

V:0.08, I:0.5

2180

barcode and label printing application

KDE

gnucash

V:0.9, I:3

5860

personal accounting

GNOME

homebank

V:0.07, I:0.3

840

personal accounting

GTK

kmymoney2

V:0.2, I:1.0

8960

personal accounting

KDE

xsane

V:7, I:45

744

scanner frontend

GTK

kooka

V:1.5, I:14

1284

scanner frontend

KDE


[Caution] Caution

The poppler-data package (non-free, see Section 12.3.1, “The Ghostscript”) needs to be installed for evince and okular to display CJK PDF documents using Cmap data (Section 8.6.3, “CJK fonts”).

[Note] Note

Installing softwares such as scribus (KDE) on GNOME desktop environment are quite acceptable since corresponding functionality is not available under GNOME desktop environment. But installing too many packages with duplicated functionalities will clutter your menu.

8.8. The X trivia

8.8.1. Keymaps and pointer button mappings in X

The xmodmap(1) program is a utility for modifying keymaps and pointer button mappings in the X window system. To get the keycode, run the xev program in the X and press keys. To get the meaning of keysym, look into the MACRO definition in /usr/include/X11/keysymdef.h file. All #define statements in this file are named as XK_ prepended to keysym names.

8.8.2. The classic X clients

Most traditional X client programs, such as xterm, can be started with a set of standard command line options to specify geometry, font, and display.

They also use the X resource database to configure their appearance. The system-wide defaults of X resources are stored in /etc/X11/Xresources/* and application defaults of them are stored in /etc/X11/app-defaults/*. Use these settings as the starting points.

The file ~/.Xresources is used to store user resource specifications. This file is automatically merged into the default X resources upon login. To make changes to these settings and make them effective immediately, merge them into the database using the command:

$ xrdb -merge ~/.Xresources

See x(7) and xrdb(1).

8.8.3. The X terminal emulator -- xterm

Learn everything about xterm at http://dickey.his.com/xterm/xterm.faq.html.

8.8.4. Running X clients as root

[Warning] Warning

Never attempt to start an X display/session manager as root to have entire X desktop environment running as root. This is considered security risk.

The easiest way to run a particular X client, e.g. "foo" as root is to use sudo:

$ sudo foo &

or

$ sudo -s
# foo &

or

$ gksu foo &

or

$ ssh -X root@localhost
# foo &
[Caution] Caution

Use of ssh just for this purpose as above is waste of resource.

Please note, in order for the X client to connect to the X server,

  • values of the old user's "$XAUTHORITY" and "$DISPLAY" environment variables must be copied to the new user's ones, and

  • the file pointed by value of the "$XAUTHORITY" environment variable must be readable by the new user.

The gksu package (popcon: V:24, I:56) is a specialized GTK+ GUI package for gaining the root privileges. It can be configured to use su(1) or sudo(8) as its backend depending on the /apps/gksu/sudo-mode gconf key. You can edit gconf key using the gconf-editor(1) command (menu: "Applications" -> "System Tools" -> "Configuration Editor").

Chapter 9. The I18N and L10N

The international support for the application software is done in 2 steps:

  • Multilingualization (m17n): To make software independent of the particular locale.

  • Localization (l10n): To configure multilingualized software to the particular locale.

Here, I18N stands for "internationalization" and L10N stands for "localization". (There are 18 and 10 letters between I and N or L and N.)

The modern software such as Gnome and KDE which are written to handle UTF-8 data and to provide their message through the gettext(1) infrastructure are multilingualized and can be localized simply by setting pertinent environment variables to the appropriate locale if the corresponding translated message packages are installed.

The simplest representation of the text data is ASCII which is sufficient for English and uses less than 127 characters (representable with 7 bits). In order to support much more characters for the international support, many character encoding systems have been invented. The modern and sensible encoding system is UTF-8 which can handle practically all the characters known to the human (see: Section 9.3.1, “Basics of encoding”).

See Introduction to i18n for details.

The international hardware support is enabled with localized hardware configuration data.

9.1. The keyboard input

The Debian system can be configured to work with many international keyboard arrangements:

Table 9.1.  List of keyboard reconfiguration methods.

environment

command

Linux console

dpkg-reconfigure --priority=low console-data

X Window

dpkg-reconfigure --priority=low xserver-xorg


This will support keyboard input for accented characters of many European languages with its dead-key function. For Asian languages, you need more complicated input method support such as scim discussed next.

9.1.1. The input method support with scim

Setup of multilingual input for the Debian system is simplified by using the SCIM family of packages with the im-switch package. The list of SCIM packages are:

Table 9.2.  List of input method supports with scim.

package

popcon

locale

scim-anthy

V:0.2, I:0.7

Japanese

scim-canna

V:0.05, I:0.3

, ,

scim-skk

V:0.04, I:0.3

, ,

scim-prime

V:0.04, I:0.3

, ,

scim-tables-ja

I:0.6

, , (not useful)

scim-tables-zh

I:1.2

Chinese (for zh_*)

scim-pinyin

V:0.2, I:1.1

, , (for zh_CN)

scim-chewing

V:0.09, I:0.8

, , (for zh_TW)

scim-hangul

V:0.04, I:0.10

Korean

scim-tables-ko

I:0.08

, ,

scim-thai

V:0.01, I:0.04

Thai

scim-m17n

V:0.06, I:0.17

Multilingual: Indic, Arabic and others

scim-tables-additional

I:0.2

, ,

scim-uim

V:0.04, I:0.3

, ,


The kinput2 method and other locale dependent Asian classic input methods still exist but are not recommended for the modern UTF-8 X environment. The uim tool chain is an alternative approach for the international input method for the modern UTF-8 X environment which is also capable for non-X environment.

9.1.2. An example for Japanese

I find the Japanese input method started under English environment (en_US.UTF-8) very useful. Here is how I did it with SCIM.

  • Install the Japanese input tool package scim-anthy with its recommended packages such as im-switch.

  • Execute "im-switch -c" from user's shell and select "scim".

  • Relogin to user's account.

  • Verify setting by "im-switch -l".

  • Setup input method and mode by right clicking GUI toolbar. (You can reduce menu choice of input method)

  • Start SCIM input method by CTRL-SPACE

[Note] Note

In order to start SCIM under the non-CJK and non-en_US locale, you need to add list of those locales in UTF-8 to the ~/.scim/global or /etc/scim/global file as:

/SupportedUnicodeLocales = en_US.UTF-8,en_GB.UTF_8,fr_FR.UTF-8

Please note:

  • The im-switch command behaves differently if command is executed from root or not.

  • Input method started by im-switch depends on the locale.

  • Use of new immodule mechanism (via GTK_IM_MODULE) may cause instability during the library transition in unstable.

For the detail of setup, see /usr/share/doc/im-switch/README.Debian.gz, /usr/share/doc/scim/README.Debian.gz or /usr/share/doc/uim/README.Debian.gz. Here key points are described.

9.1.3. Disabling the input method

If you wish to input without going through XIM, set XMODIFIERS value to "none" while starting a program. This may be the case if you use Japanese input infrastructure egg on emacs. From shell, execute as:

$ XMODIFIERS=none emacs

In order to adjust the command executed by the Debian menu, place customized configuration in /etc/menu following method described in /usr/share/doc/menu/html .

9.2. The display output

Linux console can only display limited characters. (You need to use special terminal program such as jfbterm to display non-European languages on the non-X console.)

X Window can display any characters in the UTF-8 as long as required font data exists. (The encoding of the original font data is taken care by the X Window system and transparent to the user.)

9.3. The locale

The following will focus on the locale for applications run under X Window environment started from the gdm.

9.3.1. Basics of encoding

The environment variable "LANG=xx_YY.ZZZZ" sets the locale to language code "xx", country code "yy", and encoding "ZZZZ" (see: Section 2.5.1.1, “LANG variable”).

Current Debian system normally sets the locale as "LANG=xx_YY.UTF-8". This uses the UTF-8 encoding with the Unicode character set. This UTF-8 encoding system is a multibyte code system and uses code points smartly. The ASCII data, which consist only with 7-bit range codes, are always valid UTF-8 data consisting only with 1 byte per character.

Previous Debian system used to set the locale as "LANG=C" or "LANG=xx_YY".

Actual traditional encoding system used for "LANG=xx_YY" can be identified by checking "/usr/share/i18n/SUPPORTED". For example, "en_US" uses "ISO-8859-1" encoding and "fr_FR@euro" uses "ISO-8859-15" encoding.

[Tip] Tip

For meaning of encoding values, see: Table 12.2, “ List of encoding values and their usage. ” .

9.3.2. Rationale for UTF-8 locale

The UTF-8 encoding is the modern and sensible text encoding system for I18N and enables to represent Unicode characters, i.e., practically all characters known to human. UTF stands for Unicode Transformation Format (UTF) schemes.

I recommend to use UTF-8 locale for your desktop, e.g., "LANG=en_US.UTF-8". The first part of the locale determines messages presented by applications. For example, gedit(1) (text editor for the GNOME Desktop) under "LANG=fr_FR.UTF-8" locale can display and edit Chinese character text data while presenting menus in French, as long as required fonts and input methods are installed.

I also recommend to set the locale only using LANG environment variable. I do not see much benefit of setting a complicated combination of LC_* variables (see locale(1) manpage) under UTF-8 locale.

Even plain English text may contain non-ASCII characters, e.g. left and right quotation marks are not available in ASCII:

“double quoted text”
‘single quoted text’

When ASCII plain text data is converted to UTF-8 one, it has exactly the same content and size as the original ASCII one. So you loose nothing by deploying UTF-8 locale.

Some programs consume more memory after supporting I18N. This is because they are coded to use UTF-32(UCS4) internally to support Unicode for speed optimization and consume 4 bytes per each ASCII character data independent of locale selected. Again, you loose nothing by deploying UTF-8 locale.

The vendor specific old non-UTF-8 encoding systems tend to have minor but annoying differences on some characters such as graphic ones for many countries. The deployment of the UTF-8 system by the modern OSs practically solved these conflicting encoding issues.

9.3.3. The reconfiguration of the locale

In order for the system to access a particular locale, the locale data must be compiled from the locale database. (The Debian system does not come with all available locales pre-compiled unless you installed the locales-all package.) The full list of supported locales available for compiling are listed in /usr/share/i18n/SUPPORTED. This lists all the proper locale names. The following will list all the available UTF-8 locales already compiled to the binary form:

$ locale -a

The following command execution will reconfigure locale package:

  • to update the list of available locales,

  • to compile them into the binary form, and

  • to set the system wide default locale value in the /etc/defaults/locale for use by PAM (see: Section 5.5, “PAM and NSS”).

# dpkg-reconfigure locales

The list of available locale should include "en_US.UTF-8" and all the interesting languages with "UTF-8".

The recommended default locale is "en_US.UTF-8" for US English. For other languages, please make sure to chose locale with "UTF-8". Any one of these settings can handle any international characters.

[Note] Note

Although setting locale to "C" will use US English message, it handles only ASCII characters.

9.3.4. Configuring system with LANG

The environment variable LANG is:

  • set initially by some display manager such as gdm for all X programs,

  • changed by the X session startup code via $HOME/.xsessionrc for all X programs (lenny feature),

  • set initially by the PAM mechanism of the login for the local Linux console programs,

  • set initially by the PAM mechanism of the ssh for the remote console programs, or

  • changed by the shell startup code, e.g. $HOME/.bashrc, for all console programs.

[Tip] Tip

It is good idea to install system wide default locale as "en_US.UTF-8" for maximum compatibility.

9.3.5. Specific locale only under X Window

You can chose specific locale only under X Window irrespective of your system wide default locale. This should provide your best desktop experience with stability.

This way, you can always access functioning character terminal with readable messages even when X Window system is not working. This becomes essential for languages which use non-roman characters such as Chinese, Japanese, and Korean.

For gdm, you can select different locale for the X session from its menu independent of the system default locale value in the /etc/defaults/locale.

You can set the locale of the X session manager and the value for the default locale permanently using PAM customization (see: Section 5.5, “PAM and NSS”) as follows. (There may be another way available as the improvement of X session manager package but please read following as the generic and basic method of setting the locale.)

First, change the following line defining language environment variable in its PAM configuration file, such as /etc/pam.d/gdm:

auth    required        pam_env.so read_env=1 envfile=/etc/default/locale

into

auth    required        pam_env.so read_env=1 envfile=/etc/default/locale-x

Then create a /etc/defaults/locale-gdm file with "-rw-r--r-- 1 root root" permission containing, eg. for Japanese message:

LANG="ja_JP.UTF-8"

and keep the default /etc/defaults/locale file for other programs being:

LANG="en_US.UTF-8"

This is the most generic technique to customize locale.

Alternatively for this case, you may simply change locale using the $HOME/.xsessionrc file.

9.3.6. Filename encoding

For cross platform data exchanges (see Section 11.1.10, “Removable mass storage device”), you may need to mount some file system with particular encodings. For example, the mount(8) command for vfat filesystem assumes CP437 if used without option. You need to provide explicit mount option to use UTF-8 or CP932 for filenames.

[Note] Note

When auto-mounting a hot-pluggable USB memory stick under modern desktop environment such as Gnome, you may provide such mount option by right clicking the icon on the desktop, click "Drive" tab, click to expand "Setting", and entering "utf8" to "Mount options:". The next time this memory stick is mounted, mount with UTF-8 is enabled.

[Note] Note

If you are upgrading system or moving disk drives from older non-UTF-8 system, file names with non-ASCII characters may be encoded in the historic and deprecated encodings such as ISO-8859-1 or eucJP. Please seek help of text conversion tools to convert them to UTF-8. See: Section 12.1, “Text data conversion tools” .

Samba uses Unicode for newer clients (Windows NT, 200x, XP) but uses CP850 as default for older clients (DOS and Windows 9x/Me clients). This default for older clients can be changed using "dos charset" in the /etc/samba/smb.conf file, e.g., to CP932 for Japanese.

9.3.7. Localized messages and translated documentation

Translations exist for many of the text messages and documents that are displayed in the Debian system, such as error messages, standard program output, menus, and manual pages. GNU gettext(1) command tool chain is used as the backend tool for most translation activities.

The aptitude list under "Tasks" -> "Localization" provide extensive list of useful binary packages which add localized messages to applications and provide translated documentation.

For example, you can obtain the localized message for manpage by installing the manpages-<LANG>. To read the Italian-language manpage for <programname>, execute

LANG=it_IT.UTF-8 man <programname>

to read it from /usr/share/man/it/.

9.3.8. Effects of locale

The sort order of characters with sort(1) is affected by the language choice of the locale. Spanish and English locale sort differently.

The date format of ls(1) is affected by the locale. The date format of "LANG=C ls -l" and "LANG=en_US.UTF-8" are different.

Number punctuation are different for locale. For example, in English locale, one thousand one point one is displayed as "1,000.1" while in German locale, it is displayed as "1.000,1". You see this difference in spreadsheet program.

Chapter 10. System tips

Here, I will describe basic tips to configure and manage systems, mostly from the console.

10.1. The screen program

The screen program is a very useful tool for people to access remote sites via unreliable or intermittent connections since it support interrupted network connections.

Table 10.1.  List of programs to support interrupted network connections.

package

popcon

size

description

screen

V:10, I:29

1036

terminal multiplexer with VT100/ANSI terminal emulation

xmove

V:0.05, I:0.5

372

allows you to move programs between X Window System displays


10.1.1. The use scenario for the screen command

The screen(1) program not only allows one terminal window to work with multiple processes, but also allows remote shell process to survive interrupted connections. Here is a typical use scenario of the screen(1) program.

  1. You login to a remote machine.

  2. You start the screen command on a single console.

  3. You execute multiple programs in screen windows created with ^A c ("Control-A" followed by "c").

  4. You switch among the multiple screen windows by ^A n ("Control-A" followed by "n").

  5. Suddenly you need to leave your terminal, but you don't want to lose your active work by keeping the connection.

  6. You detach the screen session by any methods such as:

    • brutally unplugging your network connection,

    • typing ^A d ("Control-A" followed by "d") and manually logging out from the remote connection, or

    • typing ^A DD ("Control-A" followed by "DD") to have screen detach and log you out.

  7. You log in again to the same remote machine (even from a different terminal).

  8. You enter the "screen -r" command.

  9. The screen program will magically reattach all previous screen windows with all actively running programs.

[Tip] Tip

You can save connection fees for metered network connections such as dial-up and packet ones, because you can leave a process active while disconnected, and then re-attach it later when you connect again.

10.1.2. Key bindings for the screen command

In screen session, all keyboard inputs are sent to your current window except for the command keystroke, by default ^A ("Control-A"). All screen commands are entered by typing ^A plus a single key [plus any parameters]. Here are important ones to remember:

Table 10.2.  List of key bindings for screen.

key binding

meaning

^A ?

show a help screen (display key bindings)

^A c

create a new window and switch to it

^A n

go to next window

^A p

go to previous window

^A 0

go to window number 0

^A 1

go to window number 1

^A w

show a list of windows

^A a

send a Ctrl-A to current window as keyboard input

^A h

write a hardcopy of current window to file

^A H

begin/end logging current window to file

^A ^X

lock the terminal (password protected)

^A d

detach screen session from the terminal

^A DD

detach screen session and log out


See screen(1) for details.

10.1.3. A screen-like program for X window system

The xmove package enables support for mobile X clients; that is, X clients can move between displays. See xmove(1).

10.2. Data recording and presentation

10.2.1. The log daemon

Many programs record their activities under the /var/log/ directory.

  • The kernel log daemon: klogd(8)

  • The system log daemon: syslogd(8)

See Section 4.5.8, “The system message” and Section 4.5.9, “The kernel message”.

10.2.2. Log analyzer

Here are notable log analyzers ("~Gsecurity::log-analyzer" in aptitude).

Table 10.3.  List of system log analyzers.

package

popcon

size

description

logwatch

V:2, I:2

2312

log analyser with nice output written in Perl

fail2ban

V:2, I:2

616

bans IPs that cause multiple authentication errors

analog

V:1.2, I:17

4608

web server log analyzer

awstats

V:1.5, I:3

4896

powerful and featureful web server log analyzer

sarg

V:1.5, I:1.5

1448

squid analysis report generator

pflogsumm

V:0.3, I:0.7

164

Postfix log entry summarizer

syslog-summary

V:0.2, I:1.0

80

summarize the contents of a syslog log file

lire

V:0.18, I:0.2

5056

full-featured log analyzer and report generator

fwlogwatch

V:0.14, I:0.2

432

Firewall log analyzer

squidview

V:0.09, I:0.6

260

monitors and analyses squid access.log files

visitors

V:0.08, I:0.3

224

fast web server log analyzer

swatch

V:0.06, I:0.2

112

Log file viewer with regexp matching, highlighting, & hooks

crm114

V:0.08, I:0.2

1120

The Controllable Regex Mutilator and Spam Filter (CRM114)

icmpinfo

V:0.07, I:0.4

84

Interpret ICMP messages


[Note] Note

CRM114 provides language infrastructure to write fuzzy filters with the TRE regex library. Its popular use is spam mail filter but it can be used as log analyzer.

10.2.3. Recording the shell activities cleanly

The simple use of the script(1) command (see: Section 2.4.9, “Recording the shell activities”) to record shell activity produces a file with control characters. This can be avoided by using the col(1) command:

$ script
Script started, file is typescript
  • do whatever ...

  • Press Ctrl-D to exit script

$ col -bx <typescript >cleanedfile
$ vim cleanedfile

If you don't have the script command (for example, during the boot process in the initramfs), you can use following instead:

$ sh -i 2>&1 | tee typescript
[Tip] Tip

Some x-terminal-emulator such as gnome-terminal can record. You may wish to extend line buffer for scrollback.

[Tip] Tip

You may use screen command with "^A H" (see Section 10.1.2, “Key bindings for the screen command”) to perform recording of console.

[Tip] Tip

You may use emacs with "M-x shell", "M-x eshell", or "M-x term" to perform recording of console. You may later use "C-x C-w" to write the buffer to a file.

10.2.4. Customized display of text data

Although pager tools such as more(1) and less(1) (see Section 2.4.5, “The pager”) and custom tools for highlighting and formatting Section 12.1.7, “Highlighting and formatting plain text data” can display text data nicely, general purpose editors (see Section 2.4.6, “The text editor”) are most versatile and customizable.

[Tip] Tip

For vim(1) and its pager mode alias view(1), ":set hls" will enable highlighted search.

10.2.5. Colorized shell echo

Shell echo to most modern terminals can be colorized using ANSI escape code (see /usr/share/doc/xterm/ctlseqs.txt.gz). E.g.:

$ RED=$(printf "\x1b[31m")
$ NORMAL=$(printf "\x1b[0m")
$ REVERSE=$(printf "\x1b[7m")
$ echo "${RED}RED-TEXT${NORMAL} ${REVERSE}REVERSE-TEXT${NORMAL}"

10.2.6. Colorized commands

Colorized commands are handy for inspecting their output in the interactive environment. I include following in my ~/.bashrc.

if [ "$TERM" != "dumb" ]; then
    eval "`dircolors -b`"
    alias ls='ls --color=always'
    alias ll='ls --color=always -l'
    alias la='ls --color=always -A'
    alias less='less -R'
    alias ls='ls --color=always'
    alias grep='grep --color=always'
    alias egrep='egrep --color=always'
    alias fgrep='fgrep --color=always'
    alias zgrep='zgrep --color=always'
else
    alias ll='ls -l'
    alias la='ls -A'
fi

The use of alias limits color effects to the interactive command usage. It has advantage over exporting environment variable "export GREP_OPTIONS='--color=auto'" since color can be seen under pager programs such as "less".

[Tip] Tip

You can turn off these colorizing aliases in the interactive environment by invoking shell with "TERM=dumb bash".

10.2.7. Recording the graphic image of an X application

There are few ways to record the graphic image of an X application, including an xterm display.

Table 10.4.  List of graphic image manipulation tools.

package

popcon

size

command

xbase-clients

V:25, I:64

84

xwd(1)

gimp

V:13, I:55

11928

GUI menu

imagemagick

V:13, I:31

4464

import(1)

scrot

V:0.2, I:1.1

76

scrot(1)


10.2.8. Recording changes in configuration files

There are specialized tools to record changes in configuration files with help of DVCS system.

Table 10.5.  List of packages to record configuration history in VCS.

package

popcon

size

description

etckeeper

V:0.2, I:0.5

296

store configuration files and its metadata with Git (default), Mercurial, or Bazaar. (new)

changetrack

V:0.07, I:0.09

152

store configuration files with RCS. (old)


I recommend to use the etckeeper package with git(1) which put entire "/etc" under VCS control. Its installation guide and tutorial are found in "/usr/share/doc/etckeeper/README.gz".

Essentially, running "sudo etckeeper init" initializes the git repository for "/etc" just like the process explained in Section 11.5.4.4, “Git for recording configuration history”) but with special hook scripts for more thorough setups.

As you change your configuration, you can use git(1) normally to record them. It will automatically record changes nicely every time you run package management commands, too.

You can browse history of "/etc" and package upgrade by executing "cd /etc; sudo gitk".

10.3. Data storage tips

Booting your system with Linux live CDs or debian-installer CDs in rescue mode make it easy for you to reconfigure data storage on your boot device. See also Section 11.2, “The binary data”.

10.3.1. Partition configuration

For partition configuration, although fdisk(8) has been considered standard, parted(8) deserves some attention. "Disk partitioning data", "partition table", "partition map", and "disk label" are all synonyms.

Most PCs use the classic Master Boot Record (MBR) scheme to hold disk partitioning data in the first sector, i.e., LBA sector 0 (512 bytes).

[Note] Note

Some new PCs with Extensible Firmware Interface (EFI), including Intel-based Macs, use GUID Partition Table (GPT) scheme to hold disk partitioning data not in the first sector.

Although fdisk(8) has been standard for the disk partitioning tool, parted(8) is replacing it.

Table 10.6.  List of disk partition management packages

package

pocon

size

description

GUID Partition Table

util-linux

V:85, I:99

1744

Miscellaneous system utilities including fdisk(8) and cfdisk(8)

Not supported

parted

V:1.1, I:8

180

The GNU Parted disk partition resizing program

Supported

gparted

V:4, I:46

1940

GNOME partition editor based on libparted

Supported

qtparted

V:0.14, I:1.1

764

KDE partition editor based on libparted

Supported

gptsync

V:0.00, I:0.11

72

Synchronize classic MBR partition table with the GPT one

Supported


[Caution] Caution

Although parted(8) claims to create and to resize filesystem too, it is safer to do such things using best maintained specialized tools such as mkfs(8) (mkfs.msdos(8), mkfs.ext2(8), mkfs.ext3(8), ...) and resize2fs(8).

[Note] Note

In order to switch between GPT and MBR, you need to erase first few blocks of disk contents directly (see Section 11.2.10, “Clear file contents”) and use "parted /dev/sdx mklabel gpt" or "parted /dev/sdx mklabel msdos" to set it. Please note "msdos" is use here for MBR.

10.3.2. Accessing partition using UUID

Although reconfiguration of your partition may yield different names for partitions, you can access them consistently. This is also helpful if you have multiple disks and your BIOS doesn't give them consistent device names.

[Tip] Tip

You can probe UUID of a block special device with the vol_id(8) command.

10.3.3. Filesystem configuration

For ext3 filesystem, the e2fsprogs package provides:

  • mkfs.ext3(8) to create new ext3 filesystem,

  • fsck.ext3(8) to check and to repair existing ext3 filesystem, and

  • tune2fs(8) to configure superblock of ext3 filesystem.

The mkfs(8) and fsck(8) commans are provided by the e2fsprogs package as front-ends to various filesystem dependent programs (mkfs.fstype and fsck.fstype). For ext3 filesystem, they are mkfs.ext3(8) and fsck.ext3(8) (they are hardlinked to mke2fs(8) and e2fsck(8)).

Similar commands are available for each filesystem supported by Linux.

Table 10.7.  List of filesystem management packages

package

popcon

size

description

e2fsprogs

V:61, I:99

2088

Utilities for the ext2/ext3/ext4 filesystems.

reiserfsprogs

V:3, I:10

1156

Utilities for the Reiserfs filesystem.

dosfstools

V:3, I:24

224

Utilities for the FAT filesystem. (Microsoft: MS-DOS, Windows)

xfsprogs

V:2, I:9

3132

Utilities for the XFS filesystem. (SGI: IRIX)

ntfsprogs

V:1.3, I:6

632

Utilities for the NTFS filesystem. (Microsoft: Windows NT, ...)

jfsutils

V:0.6, I:2

1116

Utilities for the JFS filesystem. (IBM: AIX, OS/2)

reiser4progs

V:0.08, I:0.7

1248

Utilities for the Reiser4 filesystem.

hfsprogs

V:0.03, I:0.4

324

Utilities for HFS and HFS+ filesystem. (Apple: Mac OS)

btrfs-tools

V:0.01, I:0.05

576

Utilities for the btrfs filesystem.


[Tip] Tip

Ext3 filesystem is the default filesystem for the Linux system and strongly recommended to use it unless you have some specific reasons not to. After Linux kernel 2.6.28 (Debian squeeze), ext4 filesystem will be available and expected to be the default filesystem for the Linux system. btrfs filesystem is expected to be the next default filesystem after ext4 filesystem for the Linux system.

[Tip] Tip

Some tools allow access to filesystem without Linux kernel support (see Section 11.2.5, “Manipulating files without mounting disk”).

10.3.4. Filesystem creation and integrity check

The mkfs(8) command creates the filesystem on a Linux system. The fsck(8) command provides the filesystem integrity check and repair on a Linux system.

[Tip] Tip

Check files in /var/log/fsck/ for the result of the fsck(8) command run from the boot script.

[Caution] Caution

It is generally not safe to run fsck on mounted filesystems.

[Tip] Tip

Use "shutdown -F -r now" to force to run the fsck(8) command safely on all filesystems including root file system on reboot. See the shutdown(8) manpage for more.

10.3.5. Optimization of filesystem by mount options

Performance and characteristics of a filesystem can be optimized by mount options used on it (see fstab(5) and mount(8)). For example:

  • "defaults" option implies default options: "rw,suid,dev,exec,auto,nouser,async". (general)

  • "noatime" or "relatime" option is very effective for speeding up the read access. (general)

  • "user" option allows an ordinary user to mount the file system. This option implies "noexec,nosuid,nodev" option combination. (general, used for CD and floppy)

  • "noexec,nodev,nosuid" option combination is used to enhance security. (general)

  • "noauto" option limits mounting by explicit operation only. (general)

  • "data=journal" option for ext3fs can enhance data integrity against power failure with some loss of write speed.

[Tip] Tip

You need to provide kernel boot parameter "rootflags=data=journal" to deploy "data=journal" option for the root file system formatted with ext3fs.

10.3.6. Optimization of filesystem via superblock

Characteristics of a filesystem can be optimized via its superblock using the tune2fs(8) command. For example on /dev/hda1:

  • Execution of "sudo tune2fs -l /dev/hda1" will display the contents of its filesystem superblock.

  • Execution of "sudo tune2fs -c 50 /dev/hda1" will change frequency of filesystem checks (fsck execution during boot-up) to every 50 boots.

  • Execution of "sudo tune2fs -j /dev/hda1" will add journaling capability to the filesystem, i.e. filesystem conversion from ext2 to ext3. (Do this on the unmounted filesystem.)

  • Execution of "sudo tune2fs -O extents,uninit_bg,dir_index /dev/hda1 && fsck -pf /dev/hda1" will convert it from ext3 to ext4. (Do this on the unmounted filesystem.)

[Warning] Warning

Filesystem conversion for the boot device to the ext4 filesystem should be avoided until GRUB boot loader supports the ext4 filesystem well and installed Linux Kernel version is newer than 2.6.28.

[Tip] Tip

Despite its name, tune2fs(8) works not only on the ext2 filesystem but also on the ext3 and ext4 filesystems.

10.3.7. Optimization of harddisk

[Warning] Warning

Please check your hardware and read manpage of hdparam(8) before playing with harddisk configuration because this may be quite dangerous for the data integrity.

You can test disk access speed of a harddisk, e.g. /dev/hda, by "hdparm -tT /dev/hda". For some harddisk connected with (E)IDE, you can speed it up with "hdparm -q -c3 -d1 -u1 -m16 /dev/hda" by enabling the "(E)IDE 32-bit I/O support", enabling the "using_dma flag", setting "interrupt-unmask flag", and setting the "multiple 16 sector I/O" (dangerous!).

You can test write cache feature of a harddisk, e.g. /dev/sda, by "hdparm -W /dev/sda". You can disable its write cache feature with "hdparm -W 0 /dev/sda".

You may be able to read badly pressed CDROMs on modern high head CD-ROM drive by slowing it down with "setcd -x 2.

10.3.8. Expand usable storage space via LVM

For partitions created on Logical Volume Manager (Linux) at install time, they can be resized easily by concatenating extents onto them or truncating extents from them over multiple storage devices without major system reconfiguration.

[Caution] Caution

Deployment of the current LVM system may degrade guarantee against filesystem corruption offered by journaled file systems such as ext3fs unless their system performance is sacrificed by disabling write cache of harddisk.

10.3.9. Expand usable storage space by mounting another partition

If you have an empty partition (e.g., /dev/sdx), you can format it with mkfs.ext3(1) and mount(8) it to a directory where you need more space. (You need to copy original data contents.)

$ sudo mv work-dir old-dir
$ sudo mkfs.ext3 /dev/sdx
$ sudo mount -t ext3 /dev/sdx work-dir
$ sudo cp -a old-dir/* work-dir
$ sudo rm -rf old-dir

10.3.10. Expand usable storage space using symlink

If you have an empty directory (e.g., /path/to/emp-dir) in another partition with usable space, you can create a symlink to the directory with ln(8).

$ sudo mv work-dir old-dir
$ sudo mkdir -p /path/to/emp-dir
$ sudo ln -sf /path/to/emp-dir work-dir
$ sudo cp -a old-dir/* work-dir
$ sudo rm -rf old-dir
[Caution] Caution

Some software may not function well with "symlink to a directory".

10.3.11. Expand usable storage space using aufs

If you have usable space in another partition (e.g., /path/to/), you can create a directory in it and stack that on to a directory where you need space with aufs.

$ sudo mv work-dir old-dir
$ sudo mkdir -p /path/to/emp-dir
$ sudo mount -t aufs -o br:/path/to/emp-dir:old-dir none work-dir
[Caution] Caution

Use of aufs for long term data storage is not good idea since it is under development and its design change may introduce issues.

[Tip] Tip

In order to use aufs, its utility package aufs-tools and kernel module package for aufs such as aufs-modules-2.6-amd64 need to be installed.

[Note] Note

aufs is used to provide writable root filesystem by many modern live CD projects.

10.4. Data encryption tips

Since gaining root privilege is relatively easy with physical access (see Section 5.7.4, “Securing the root password”), it can not secure your private and sensitive data against possible theft of your PC. You must deploy data encryption technology to do it. Although GNU privacy guard (see Section 11.3, “Data security infrastructure”) can encrypt files, it takes some user efforts.

dm-crypt and eCryptfs facilitates automatic data encryption natively via Linux kernel modules with minimal user efforts.

Table 10.8.  List of data encryption utilities.

package

popcon

size

function

cryptsetup

V:2, I:4

840

Utilities for encrypted block device (dm-crypt / LUKS)

cryptmount

V:0.11, I:0.5

292

Utilities forencrypted block device (dm-crypt / LUKS) with focus on mount/unmount by normal users

ecryptfs-utils

V:0.03, I:0.17

396

Utilities for encrypted stacked filesystem (eCryptfs)


Dm-crypt is a cryptographic filesystem using device-mapper. Device-mapper maps one block device to another.

eCryptfs is another cryptographic filesystem using stacked filesystem. Stacked filesystem stacks itself on top of an existing directory of a mounted filesystem.

[Note] Note

Entire Debian system can be installed on a encrypted disk by the debian installer (lenny or newer) using dm-crypt/LUKS and initramfs.

[Caution] Caution

Data encryption costs CPU time etc. Please weigh its benefits and costs.

[Tip] Tip

See Section 11.3, “Data security infrastructure” for user space encryption utility: GNU Privacy Guard.

10.4.1. Removable disk encryption with dm-crypt/LUKS

You can encrypt contents of removable mass storage devices, e.g. USB memory stick on /dev/sdx, using dm-crypt/LUKS. You simply formatting it as:

# badblocks -c 10240 -s -w -t random -v /dev/sdx
# shred -v -n 1 /dev/sdx
# fdisk /dev/sdx
... "n" "p" "1" "return" "return" "w"
# cryptsetup luksFormat /dev/sdx1
...
# cryptsetup luksOpen /dev/sdx1 sdx1
...
# ls -l /dev/mapper/
total 0
crw-rw---- 1 root root  10, 60 2008-10-04 18:44 control
brw-rw---- 1 root disk 254,  0 2008-10-04 23:55 sdx1
# mkfs.vfat /dev/mapper/sdx1
...
# cryptsetup luksClose sdx1

Then, it can be mounted just like normal one on to /media/<disk_label>, except for asking password (see Section 11.1.10, “Removable mass storage device”) under modern desktop environment, such as Gnome using gnome-mount(1). The difference is that every data written to it is encrypted. You may alternatively format media in different file format, e.g., ext3 with "mkfs.ext3 /dev/sdx1".

[Note] Note

If you are really paranoid for the security of data, you may need to overwrite multiple times in the above example. This operation is very time consuming though.

10.4.2. Encrypted swap partition with dm-crypt

If your original /etc/fstab contains:

/dev/sda7 swap sw 0 0

then you can enable encrypted swap partition using dm-crypt as

# swapoff -a
# echo "cswap /dev/sda7 /dev/urandom swap" >> /etc/crypttab
# perl -i -p -e "s/\/dev\/sda7/\/dev\/mapper\/cswap/" /etc/fstab
# swapon -a

10.4.3. Automatically encrypting files with eCryptfs

You can encrypt files written under "~/Private/" automatically using eCryptfs and the ecryptfs-utils package.

  • run "ecryptfs-setup-private" and set up "~/Private/" by following prompts.

  • activate "~/Private/" by issuing "ecryptfs-mount-private".

  • move sensitive data files to "~/Private/" and make symlinks.

  • move sensitive data directories to "~/Private/" and make symlinks.

  • do the same for "~/.gnupg" and other directories containing sensitive data.

  • create symlink from "~/.ssh" to "~/Private/.ssh"

  • deactivate "~/Private/" by issuing "ecryptfs-umount-private".

  • activate "~/Private/" by issuing "ecryptfs-mount-private" as you need encrypted data.

[Tip] Tip

Files and directories with "go-r" permission such as "~/.cvspass", "~/.fetchmailrc", "~/.ssh/identity", "~/.ssh/id_rsa", "~/.ssh/id_dsa", "~/.gnupg/", "~/.gnome2/", ... can be considered sensitive data.

[Tip] Tip

Since eCryptfs selectively encrypt only the sensitive files, its system cost is much less than using dm-crypt on the entire root or home device. It does not require any special on-disk storage allocation effort but cannot keep all filesystem metadata confidential.

10.4.4. Automatically mounting eCryptfs

If you use your login password for wrapping encryption keys, you can automate mounting eCryptfs via Pluggable Authentication Module by having active lines in /etc/pam.d/common-auth as:

auth required pam_unix.so nullok_secure
auth required pam_ecryptfs.so unwrap

and active lines in /etc/pam.d/common-session as:

session required pam_unix.so
session optional pam_ecryptfs.so unwrap

This is quite convienient.

[Caution] Caution

If you use your login password for wrapping encryption keys, your encrypted data are as secure as your user login password (see Section 5.3, “Good password”). Unless you are careful to set up a strong password, your data will be at risk when someone runs password cracking software after stealing your laptop (see Section 5.7.4, “Securing the root password”). The squeeze version of the ecryptfs-utils package comes with option to have independent password for wrapping and to set up user's entire home directory for encryption. This is an actively developed package.

10.5. Monitoring, controlling, and starting program activities

Program activities can be monitored and controlled using specialized tools.

Table 10.9.  List of tools for monitoring and controlling program activities

package

popcon

size

description

time

V:6, I:86

152

The time(1) command runs a program to report system resource usages with respect to time.

coreutils

V:87, I:99

11448

The nice(1) command runs a program with modified scheduling priority.

bsdutils

V:61, I:99

168

The renice(1) command modifies the scheduling priority of a running process.

powertop

V:0.5, I:7

424

On Intel-based laptops powertop(1) gives information about system power use.

procps

V:82, I:99

748

The /proc file system utilities: ps(1), top(1), kill(1), watch(1), ...

psmisc

V:51, I:89

508

The /proc file system utilities: killall(1), fuser(1), pstree(1)

cron

V:86, I:99

320

This package run processes according to a schedule (in background).

at

V:51, I:84

220

The at(1) or batch(1) commands run a job at a specified time or below certain load level.

lsof

V:16, I:92

424

The lsof(8) command lists open files by a running process using "-p" option.

strace

V:9, I:88

420

The strace(1) command traces system calls and signals.

ltrace

V:0.3, I:2

220

The ltrace(1) command traces library calls.

xtrace

V:0.02, I:0.15

216

The xtrace(1) command traces communication between X11 client and server.


10.5.1. Time a process

Display time used by the process invoked by the command.

# time some_command >/dev/null
real    0m0.035s       # time on wall clock (elapsed real time)
user    0m0.000s       # time in user mode
sys     0m0.020s       # time in kernel mode

10.5.2. The scheduling priority

A nice value is used to control the scheduling priority for the process.

Table 10.10.  List of nice values for the scheduling priority.

nice value

scheduling priority

19

lowest priority process (nice)

0

very high priority process for user.

-20

very high priority process for root. (not-nice)


# nice  -19 top                                      # very nice
# nice --20 wodim -v -eject speed=2 dev=0,0 disk.img # very fast

Sometimes an extreme nice value does more harm than good to the system. Use this command carefully.

10.5.3. The ps command

The ps(1) command on the Debian support both BSD and SystemV features and helps to identify the process activity statically.

Table 10.11.  List of ps command styles.

style

typical command

feature

BSD

ps aux

display %CPU %MEM

System V

ps -efH

display PPID


For the zombie (defunct) children process, you can kill them by the parent process ID identified in the (PPID) field.

The pstree(1) command display a tree of processes.

10.5.4. The top command

The top(1) command on the Debian has rich features and helps to identify what process is acting funny dynamically.

Table 10.12.  List of commands for top.

command key

response

h or ?

To show help.

f

To set/reset display field.

o

To reorder display field.

F

To set sort key field.

k

To kill a process.

r

To renice a process.

q

To quit the top command.


10.5.5. List files opened by a process

You can list all files opened by a process with a process ID (PID), e.g. 1 as:

$ sudo lsof -p 1

PID=1 is usually init program.

10.5.6. Trace program activities

You can trace program activity with strace(1), ltrace(1), or xtrace(1) commands for system calls and signals, library calls, or communication between X11 client and server. For example:

$ sudo strace ls
...

10.5.7. Identify processes using files or sockets

You can also identify processes using files or sockets by fuser(1). For example:

$ sudo fuser -v /var/log/mail.log
                     USER        PID ACCESS COMMAND
/var/log/mail.log:   root       2946 F.... syslogd

You see that file /var/log/mail.log is open for writing by the syslogd(8) command.

$ sudo fuser -v smtp/tcp
                     USER        PID ACCESS COMMAND
smtp/tcp:            Debian-exim   3379 F.... exim4

Now you know your system runs exim4(8) to handle TCP connections to SMTP port (25).

10.5.8. Repeating a command with a constant interval

The watch(1) command executes a program repeatedly with a constant interval while showing its output in fullscreen.

$ watch w

This will display who is logged on to the system updated every 2 seconds.

10.5.9. Repeating a command looping over files

There are several ways to repeat a command looping over files matching some condition, e.g. matching glob pattern "*.ext".

  • Shell for-loop method (see Section 13.1.4, “Shell loops”):
    for x in *.ext; do if [ -f "$x"]; then command "$x" ; fi; done
    
  • find(1) and xargs(1) combination:

    find . -type f -maxdepth 1 -name '*.ext' -print0 | xargs -0 -n 1 command
    
  • find(1) with "-exec" option with a command:

    find . -type f -maxdepth 1 -name '*.ext' -exec command '{}' \;
    
  • find(1) with "-exec" option with a short shell script:

    find . -type f -maxdepth 1 -name '*.ext' -exec sh -c "command '{}' && echo 'successful'" \;
    

The above examples are written to ensure proper handling of funny file names such as ones containing spaces. See Section 11.1.5, “Idioms for the selection of files” for more advance uses of find(1).

10.5.10. Starting a program from GUI

You can set up to start a process from graphical user interface (GUI).

Under Gnome desktop environment, a program program can be started with proper argument by drag-and-drop of an icon to the launcher icon or by "Open with ..." menu with right clicking. KDE can do the equivalent, too. Here is an example for Gnome to set up mc program started in gnome-terminal:

  • create an executable program "mc-term" as:

# cat >/usr/local/mc-term <<EOF
#!/bin/sh
gnome-terminal -e "mc $1"
EOF
# chmod 755 /usr/local/mc-term
  • create a desktop launcher
    • right clicking desktop space to select "Create Launcher ..."

      • set "Type" to "Application"

      • set "Name" to "mc"

      • set "Command" to "mc-term %f"

      • click "OK"

  • create an open-with association
    • right click folder to select "Open with Other Application ..."

      • click open "Use a custom command" dialog and enter "mc-term %f"

      • click "Open".

[Tip] Tip

Launcher is a file at "~/Desktop" with "desktop" as its extension.

10.5.11. Customizing program to be started

Some programs start another program automatically. Here are check points for customizing this process:

  • configuration file of the parent program such as "/etc/mc/mc.ext".

  • system configuration menu such as "System" -> "Preferences" -> "Preferred Application" for Gnome.

  • environment variables such as "BROWSER", "EDITOR", "VISUAL", and "PAGER" (see eviron(7)).

  • the update-alternatives(8) system for programs such as "editor", "view", "x-www-browser", "gnome-www-browser", and "www-browser" (see Section 2.4.7, “Setting a default text editor”).

  • the "$HOME/.mailcap" and "/etc/mailcap" file contents which associate MIME type with program (see mailcap(5)).

  • the the "$HOME/.mime.types" and "/etc/mime.types" file contents which associate file name extension with MIME type (see run-mailcap(1)).

[Tip] Tip

The update-mime(8) command updates the "/etc/mailcap" file using "/etc/mailcap.order" file (see mailcap.order(5)).

[Tip] Tip

The debianutils package provides sensible-browser(1), sensible-editor(1), and sensible-pager(1) commands which make sensible decisions on which editor, pager, and web browser to call, respectively. I recommend you to read these shell commands.

10.5.12. Kill a process

Use the kill(1) command to kill (or send a signal to) a process by the process ID.

Use killall(1) or pkill(1) commands to do the same by the process command name and other attributes.

Table 10.13.  List of frequently used signals for kill command.

signal value

signal name

function

1

HUP

restart daemon

15

TERM

normal kill

9

KILL

kill hard


10.5.13. Schedule tasks once

Run the at(1) command to schedule a one-time job:

$ echo 'command -args'| at 3:40 monday

10.5.14. Schedule tasks regularly

Use cron(8) to schedule tasks regularly. See crontab(1) and crontab(5).

Run the command "crontab -e" to create or edit a crontab file to set up regularly scheduled events.

Example of a crontab file:

# use /bin/sh to run commands, no matter what /etc/passwd says
SHELL=/bin/sh
# mail any output to paul, no matter whose crontab this is
MAILTO=paul
# Min Hour DayOfMonth Month DayOfWeek command (Day... are OR'ed)
# run at 00:05, every day
5  0  *  * *   $HOME/bin/daily.job >> $HOME/tmp/out 2>&1
# run at 14:15 on the first of every month -- output mailed to paul
15 14 1  * *   $HOME/bin/monthly
# run at 22:00 on weekdays(1-5), annoy Joe. % for newline, last % for cc:
0 22 *   * 1-5 mail -s "It's 10pm" joe%Joe,%%Where are your kids?%.%%
23 */2 1 2 *   echo "run 23 minutes after 0am, 2am, 4am ..., on Feb 1"
5  4 *   * sun echo "run at 04:05 every sunday"
# run at 03:40 on the first Monday of each month
40 3 1-7 * *   [ "$(date +%a)" == "Mon" ] && command -args
[Tip] Tip

For the system not running continuously, install the anacron package to schedule periodic command at the specified intervals as closely as machine-uptime permits.

10.5.15. Alt-SysRq

Insurance against system malfunction is provided by the kernel compile option "Magic SysRq key" (SAK key) which is now the default for the Debian kernel. Pressing Alt-SysRq followed by one of the following keys does the magic of rescuing control of the system:

Table 10.14.  List of SAK command keys.

key following Alt-SysRq

function

r

Unraw restores the keyboard after things like X crashes.

0

Changing the console loglevel to 0 reduces error messages.

k

SAK (system attention key) kills all processes on the current virtual console.

e

Send a SIGTERM to all processes, except for init.

i

Send a SIGKILL to all processes, except for init.

s

Sync all mounted filesystems.

u

Remount all mounted filesystems read-only (umount).

b

Reboot the system without syncing or unmounting.


The combination of "Alt-SysRq s", "Alt-SysRq u", and "Alt-SysRq r" is good for getting out of really bad situations.

See /usr/share/doc/linux-doc-2.6.*/Documentation/sysrq.txt.gz .

[Caution] Caution

The Alt-SysRq feature may be considered a security risk by allowing users access to root-privileged functions. Placing "echo 0 >/proc/sys/kernel/sysrq" in /etc/rc.local or "kernel.sysrq = 0" in /etc/sysctl.conf will disable the Alt-SysRq feature.

[Tip] Tip

From SSH terminal etc., you can use the Alt-SysRq feature by writing to the /proc/sysrq-trigger. For example, "echo s > /proc/sysrq-trigger; echo u > /proc/sysrq-trigger" from the root shell prompt will sync and umount all mounted filesystems.

10.6. System maintenance tips

10.6.1. Who is logged on

You can check who is logged on to the system with w(1) or who(1) commands.

10.6.2. Warn everyone

You can send message to everyone who is logged on to the system with the wall(1) command:

$ echo "We are shutting down in 1 hour" | wall

10.6.3. The hardware identification

For the PCI-like devices (AGP, PCI-Express, CardBus, ExpressCard, etc.), lspci(8) command (probably with "-nn" option) is a good start for the hardware identification

Alternatively, you can identify the hardware by reading contents of /proc/bus/pci/devices or browsing directory tree under /sys/bus/pci (see Section 2.2.12, “procfs and sysfs”).

Table 10.15.  List of hardware identification tools.

package

popcon

size

description

pciutils

V:43, I:92

708

Linux PCI Utilities, lspci(8)

usbutils

V:27, I:97

520

Linux USB utilities, lsusb(8)

pcmciautils

V:1.0, I:16

172

PCMCIA utilities for Linux 2.6, pccardctl(8)

scsitools

V:0.2, I:1.5

484

Collection of tools for SCSI hardware management, lsscsi(8)

pnputils

V:0.02, I:0.2

108

Plug and Play BIOS utilities, lspnp(8)

procinfo

V:0.7, I:5

116

Displays system information from /proc, lsdev(8)

lshw

V:1.1, I:5

804

Information about hardware configuration, lshw(1)

discover

V:5, I:16

928

Hardware identification system, discover(8)


10.6.4. The hardware configuration

Although most of the hardware configuration on modern GUI desktop systems such as Gnome and KDE can be managed through accompanying GUI configuration tools, it is a good idea to know some basics methods to configure them.

Table 10.16.  List of hardware configuration tools.

package

popcon

size

description

hal

V:42, I:60

1980

Hardware Abstraction Layer, lshal(1)

console-tools

V:56, I:96

948

Linux console font and keytable utilities.

x11-xserver-utils

V:21, I:28

616

X server utilities. xset(1) and xmodmap(1) commands.

acpid

V:52, I:89

196

Daemon to manage events delivered by the Advanced Configuration and Power Interface (ACPI)

acpi

V:3, I:34

92

Utilities for ACPI devices

apmd

V:1.3, I:13

144

Daemon to manage events delivered by the Advanced Power Management (APM)

powersaved

V:1.0, I:1.3

1800

Daemon to manage battery, temperature, ac, cpufreq (SpeedStep, Powernow!) control and monitor with ACPI and APM supports.

noflushd

V:0.09, I:0.17

244

Allow idle hard disks to spin down

sleepd

V:0.09, I:0.12

140

Puts a laptop to sleep during inactivity

hdparm

V:10, I:34

272

Hard disk access optimization. Very effective but dangerous. You must read hdparm(8) first.

smartmontools

V:5, I:14

828

Control and monitor storage systems using S.M.A.R.T.

setserial

V:2, I:5

176

Collection of tools for serial port management.

memtest86+

V:0.5, I:4

248

Collection of tools for memory hardware management.

scsitools

V:0.2, I:1.5

484

Collection of tools for SCSI hardware management.

tpconfig

V:0.3, I:0.5

208

A program to configure touchpad devices

setcd

V:0.11, I:0.5

64

Compact disc drive access optimization.

big-cursor

I:0.2

68

Larger mouse cursors for X

lspowertweak

V:0.02, I:0.10

88

Simple front end to powertweak, lspowertweak(8)


Here, ACPI is a newer framework for the power management system than APM.

10.6.5. System and hardware time

The following will set system and hardware time to MM/DD hh:mm, CCYY.

# date MMDDhhmmCCYY
# hwclock --utc --systohc
# hwclock --show

Times are normally displayed in the local time on the Debian system but the hardware and system time usually use UTC.

If the hardware (BIOS) time is set to GMT, change the setting to UTC=yes in the /etc/default/rcS.

If you wish to update system time via network, consider to use the NTP service with the packages such as ntp, ntpdate, and chrony. See:

[Tip] Tip

The ntptrace(8) command in the ntp package can trace a chain of NTP servers back to the primary source.

10.6.6. The terminal configuration

There are several components to configure character console and ncurses(3) system features:

  • the terminfo(5) file

  • the TERM(7) environment variable

  • the setterm(1) command

  • the stty(1) command

  • the tic(1) command

  • the toe(1) command

If the terminfo entry for xterm doesn't work with a non-Debian xterm, change your terminal type from xterm to one of the feature-limited versions such as xterm-r6 when you log in to a Debian system remotely. See /usr/share/doc/libncurses5/FAQ for more. dumb is the lowest common denominator for terminfo.

10.6.7. The sound infrastructure

Device drivers for sound cards for current Linux 2.6 are provided by Advanced Linux Sound Architecture (ALSA). ALSA provides emulation mode for previous Open Sound System (OSS) for compatibility.

Run "dpkg-reconfigure linux-sound-base" to select the sound system to use ALSA via blacklisting of kernel modules. Unless you have very new sound hardware, udev infrastructure should configure your sound system.

[Tip] Tip

Use "cat /dev/urandom > /dev/audio" or the speaker-test(1) command to test speaker. (^C to stop)

[Tip] Tip

If you can not get sound, your speaker may be connected to a muted output. Modern sound system has many outputs. The alsamixer(1) command in the alsa-utils package is useful to configure volume and mute settings.

Application softwares may be configured not only to access sound devices directly but also to access them via some standardized sound server system.

Table 10.17.  List of sound packages

package

pocon

size

description

linux-sound-base

V:20, I:58

136

Base package for ALSA and OSS sound systems

alsa-base

I:58

456

ALSA driver configuration files

alsa-utils

V:38, I:58

1984

Utilities for configuring and using ALSA

oss-compat

V:17, I:25

60

OSS compatibility under ALSA preventing "/dev/dsp not found" errors

esound-common

V:27, I:67

68

Enlightened Sound Daemon (ESD) common (Enlightenment and GNOME)

esound

V:4, I:35

84

Enlightened Sound Daemon (ESD) server (Enlightenment and GNOME)

esound-clients

V:10, I:19

168

Enlightened Sound Daemon (ESD) client (Enlightenment and GNOME)

libesd-alsa0

V:2, I:2

96

Enlightened Sound Daemon (ESD) library Enlightenment and GNOME)

libesd0

V:39, I:64

92

Enlightened Sound Daemon (ESD) library (Enlightenment and GNOME) - OSS

arts

I:15

36

aRts server (KDE)

libarts1c2a

V:22, I:45

6180

aRts library (KDE)

libartsc0

V:21, I:50

72

aRts library (KDE)

jackd

V:1.5, I:6

400

JACK Audio Connection Kit. (JACK) server (low latency)

libjack0

V:14, I:26

540

JACK Audio Connection Kit. (JACK) library (low latency)

libjack0.100.0-0

I:32

48

JACK Audio Connection Kit. (JACK) library (low latency)

nas

V:0.18, I:0.3

328

Network Audio System (NAS) server

libaudio2

V:24, I:63

204

Network Audio System (NAS) library

pulseaudio

V:0.8, I:2

1264

PulseAudio server, replacement for ESD

libpulse0

V:3, I:27

508

PulseAudio client library, replacement for ESD

libpulsecore5

V:0.7, I:1.8

612

PulseAudio server library, replacement for ESD

libgstreamer0.10-0

V:18, I:61

2644

GStreamer: Gnome sound engine

libxine1

V:6, I:51

24

xine: KDE older sound engine

libphonon4

I:5

472

Phonon: KDE new sound engine


There is usually a common sound engine for each popular desktop environment. Each sound engine used by the application can choose to connect to different sound servers.

10.6.8. Disable the screen saver

For disabling the screen saver, use following commands.

Table 10.18.  List of commands for disabling the screen saver.

environment

command

The Linux console

setterm -powersave off

The X Window by turning off screensaver

xset s off

The X Window by disabling dpms

xset -dpms

The X Window by GUI configuration of screen saver

xscreensaver-command -prefs


10.6.9. Disable the sound (beep)

One can always unplug the PC speaker. Removing pcspkr kernel module does this for you.

The following will prevent the readline program used by the bash to beep when encountering "\a" (ASCII=7):

$ echo "set bell-style none">> ~/.inputrc

10.6.10. Memory usage

The kernel boot message in the /var/log/dmesg contains the total exact size of available memory.

The free(1) and top(1) commands display information on memory resources on the running system.

$ grep '^Memory' /var/log/dmesg
Memory: 990528k/1016784k available (1975k kernel code, 25868k reserved, 931k data, 296k init)
$ free -k
             total       used       free     shared    buffers     cached
Mem:        997184     976928      20256          0     129592     171932
-/+ buffers/cache:     675404     321780
Swap:      4545576          4    4545572

For my MacBook with 1GB=1048576k DRAM (video system steals some of this):

Table 10.19.  List of memory sizes reported.

report

size

Total size in dmesg

1016784k = 1GB - 31792k

Free in dmesg

990528k

Total under shell

997184k

Free under shell

20256k


Do not worry about the large size of "used" and the small size of "free" in the "Mem:" line, but read the one under them (675404 and 321780 in the example below) and relax.

10.6.11. System security and integrity check

Poor system maintenance may expose your system to external exploitation.

For system security and integrity check, you should start with:

Table 10.20.  List of tools for system security and integrity check

package

popcon

size

description

logcheck

V:3, I:4

264

This mails anomalies in the system logfiles to the administrator

debsums

V:2, I:3

236

This verifies installed package files against MD5 checksums.

chkrootkit

V:2, I:6

852

Rootkit detector.

clamav

V:2, I:10

420

Anti-virus utility for Unix - command-line interface.

tiger

V:0.8, I:1.0

3072

Report system security vulnerabilities

tripwire

V:0.5, I:0.7

5020

File and directory integrity checker

john

V:0.4, I:1.8

476

Active password cracking tool

aide

V:0.3, I:0.5

1112

Advanced Intrusion Detection Environment - static binary

bastille

V:0.2, I:0.6

1484

Security hardening tool

integrit

V:0.13, I:0.2

440

A file integrity verification program

crack

V:0.05, I:0.3

204

Password guessing program


Here is a simple script to check for typical world writable incorrect file permissions.

# find / -perm 777 -a \! -type s -a \! -type l -a \! \( -type d -a -perm 1777 \)
[Caution] Caution

Since the debsums package uses MD5 checksums stored locally, it can not be fully trusted as the system security audit tool against malicious attacks.

10.7. The kernel

Debian distributes modularized Linux kernel as packages for supported architectures.

10.7.1. Linux kernel 2.6

There are few notable features on Linux kernel 2.6 compared to 2.4.

  • Devices are created by the udev system (see Section 4.5.10, “The udev system”).

  • Read/write accesses to IDE CD/DVD devices do not use the ide-scsi module.

  • Network packet filtering functions use iptable kernel modules.

10.7.2. Kernel headers

Most normal programs don't need kernel headers and in fact may break if you use them directly for compiling. They should be compiled against the headers in /usr/include/linux and /usr/include/asm provided by the libc6-dev package (created from the glibc source package) on the Debian system.

[Note] Note

For compiling some kernel-specific programs such as the kernel modules from the external source and the automounter daemon (amd), you must include path to the corresponding kernel headers, e.g. -I/usr/src/linux-particular-version/include/ , to your command line. The module-assistant package helps users to build and install module package(s) easily for one or more custom kernels with the m-a(8) command.

10.7.3. Kernel and module compile

Debian has its own method of compiling the kernel and related modules.

Table 10.21.  List of key packages to be installed for the kernel recompilation on the Debian system

package

popcon

size

description

build-essential

I:40

48

essential packages for building Debian packages: make, gcc, ...

bzip2

V:51, I:81

132

compress and decompress utilities for bz2 files

libncurses5-dev

V:4, I:28

6812

developer's libraries and docs for ncurses

git-core

V:4, I:8

7164

git: distributed revision control system used by the Linux kernel

fakeroot

V:4, I:26

456

provide fakeroot environment for building package as non-root

initramfs-tools

V:29, I:96

412

tool to build an initramfs (Debian specific)

kernel-package

V:3, I:18

2304

tool to build Linux kernel packages (Debian specific)

module-assistant

V:3, I:22

512

tool to help build module packages (Debian specific)

devscripts

V:2, I:13

1592

helper scripts for a Debian Package maintainer (Debian specific)

linux-tree-2.6.*

I:0.15

128

Linux kernel source tree for building Debian kernel images (Debian specific)


If you use initrd in Section 4.3, “Stage 2: the boot loader”, make sure to read the related information in initramfs-tools(8), update-initramfs(8), mkinitramfs(8) and initramfs.conf(5).

[Warning] Warning

Do not put symlinks to the directories in the source tree (e.g. /usr/src/linux*) from /usr/include/linux and /usr/include/asm when compiling the Linux kernel source. (Some outdated documents suggest this.)

[Note] Note

When compiling the latest Linux kernel on the Debian stable system, the use of backported latest tools from the Debian unstable may be needed.

10.7.4. Kernel source compile: Debian standard method

The Debian standard method for compiling kernel source to create a custom kernel package uses make-kpkg(1) command. The official documentation is in (the bottom of) /usr/share/doc/kernel-package/README.gz. See kernel-pkg.conf(5) and kernel-img.conf(5) for customization.

Here is an example for amd64 system:

# aptitude install linux-tree-<version>
$ cd /usr/src
$ tar -xjvf linux-source-<version>.tar.bz2
$ cd linux-source-<version>
$ cp /boot/config-<oldversion> .config
$ make menuconfig
 ...
$ make-kpkg clean
$ fakeroot make-kpkg --append_to_version -amd64 --initrd --revision=rev.01 kernel_image modules_image
$ cd ..
# dpkg -i linux-image*.deb
  • reboot to new kernel with "shutdown -r now" .

[Caution] Caution

When you intend to create a non-modularized kernel compiled only for one machine, invoke make-kpkg command without "--initrd" option since initrd is not used. Invocation of "make oldconfig" and "make dep" are not required since "make-kpkg kernel_image" invokes them.

10.7.5. Module source compile: Debian standard method

The Debian standard method for creating and installing a custom module package for a custom kernel package uses module-assistant(8) command and module-source packages. For example, following will build the unionfs kernel module package and installs it.

$ sudo aptitude install module-assistant
...
$ sudo aptitude install unionfs-source unionfs-tools unionfs-utils
$ sudo m-a update
$ sudo m-a prepare
$ sudo m-a auto-install unionfs
...
$ sudo apt-get autoremove

10.7.6. Kernel source compile: classic method

You can still build Linux kernel from the pristine sources with the classic method. You must take care the details of the system configuration manually.

$ cd /usr/src
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-<version>.tar.bz2
$ tar -xjvf linux-<version>.tar.bz2
$ cd linux-<version>
$ cp /boot/config-<version> .config
$ make menuconfig
 ...
$ make dep; make bzImage
$ make modules
# cp ./arch/x86_64/boot/bzImage /boot/vmlinuz-<version>
# make modules_install
# depmod -a
# update-initramfs -c -k <version>
  • set up bootloader
    • edit /etc/lilo.conf and run /sbin/lilo, if you use lilo .

    • edit /boot/grub/menu.lst, if you use grub .

  • reboot to new kernel with "shutdown -r now" .

10.7.7. Non-free hardware drivers

Although most of hardware drivers are available as free software and as a part of the Debian system, you may need to load some non-free external drivers to support some hardwares, such as Winmodem, on your system.

Check pertinent resources:

10.8. The chroot

The chroot(8) program is most basic way to run different instances of the GNU/Linux environment on a single system simultaneously without rebooting. I will explain simple chroot systems in the following as examples.

For serious chroot setup with the detail configuration, please consider to use the specialized schroot package.

10.8.1. Run a different Debian distribution with chroot

A chroot Debian environment can easily be created by the debootstrap or cdebootstrap command.

For example, the following will create a sid chroot on /sid-root while having fast Internet access:

main # debootstrap sid /sid-root http://ftp.debian.org/debian/
  • watch it download the whole system

main # echo "proc-sid    /sid-root/proc     proc   none 0 0" >> /etc/fstab
main # echo "devpts-sid  /sid-root/dev/pts  devpts defaults 0 0" >> /etc/fstab
main # mount -a
main # cp -f /etc/passwd /sid-root/etc/passwd
main # cp -f /etc/shadow /sid-root/etc/shadow
main # cp -f /etc/group  /sid-root/etc/group
main # cp -f /etc/hosts  /sid-root/etc/hosts
main # chroot /sid-root /bin/bash
chroot # cd /dev; /sbin/MAKEDEV generic ; cd -
chroot # vi /etc/apt/sources.list
  • point the source to unstable

chroot # aptitude update
...
chroot # aptitude install locales
...
  • add "en_US.UTF-8" as locale and make it default

Do you want to continue? [Y/n/?] y
chroot # aptitude install mc vim
...
Do you want to continue? [Y/n/?] y
...
chroot # exit
main #

At this point you should have a fully working Debian sid system, where you can play around without fear of affecting your main Debian installation.

[Caution] Caution

If you use bind mount for directories such as home directory in the chroot, you must be careful for its side effects. I heard people lost their home directory after executing "rm -rf /sid-root" without unbinding their home directory in the chroot. A bind mount is not normally visible with "df", you need to execute "df -a" to see it.

This debootstrap trick can also be used to install Debian to a system without using a Debian install disk, but instead from another GNU/Linux distribution.

10.8.2. Setting up login for chroot

Typing "chroot /sid-root /bin/bash" is easy, but it retains all sorts of environment variables that you may not want, and has other issues. A much better approach is to run another login process on a separate virtual terminal where you can log in to the chroot directly.

Since on default Debian systems tty1 to tty6 run Linux consoles and tty7 runs the X Window System, let's set up tty8 for a chrooted console as an example. After creating a chroot system, type from the root shell of the main system:

main # echo "8:23:respawn:/usr/sbin/chroot /sid-root /sbin/getty 38400 tty8"  >> /etc/inittab
main # init q
  • reload init

10.8.3. Setting up X for chroot

You want to run the latest X and GNOME safely in your chroot? That's entirely possible! The following example will make GDM run on virtual terminal vt9.

First install a chroot system. From the root of the main system, copy key configuration files to the chroot system.

main # cp /etc/X11/xorg.conf /sid-root/etc/X11/xorg.conf
main # chroot /sid-root
chroot # cd /dev; /sbin/MAKEDEV generic ; cd -
chroot # aptitude install gdm gnome x-window-system
chroot # vim /etc/gdm/gdm.conf
  • change "[servers]" section with "s/vt7/vt9/" to make the first virtual console in the chroot from vt7 to vt9.

chroot # /etc/init.d/gdm start

Now you can easily switch back and forth between full X environments in your chroot and your main system just by switching between Linux virtual terminals; e.g. by using Ctrl-Alt-F7 and Ctrl-Alt-F9. Have fun!

10.8.4. Run other distributions with chroot

A chroot environment for another Linux distribution can easily be created. You install a system into separate partitions using the installer of the other distribution. If its root partition is in /dev/hda9:

main # cd / ; mkdir /other-dist
main # mount -t ext3 /dev/hda9 /other-dist
main # chroot /other-dist /bin/bash

10.8.5. Build packages under chroot

There is a more specialized chroot package, pbuilder, which constructs a chroot system and builds a package inside the chroot. It is an ideal system to use to check that a package's build-dependencies are correct, and to be sure that unnecessary and wrong build dependencies will not exist in the resulting package.

10.8.6. Other virtualization tools

There are several system virtualization and emulation related packages in Debian beyond simple chroot.

Table 10.22.  List of virtualization tools

package

pocon

size

description

schroot

V:1.0, I:1.7

1900

Specialized tool for executing Debian binary packages in chroot

sbuild

V:0.08, I:0.3

452

Tool for building Debian binary packages from Debian sources

pbuilder

V:0.5, I:2

1116

Personal package builder for Debian packages

debootstrap

V:1.5, I:12

260

Bootstrap a basic Debian system (written in sh)

cdebootstrap

V:0.6, I:3

156

Bootstrap a Debian system (written in C)

rootstrap

V:0.04, I:0.2

156

A tool for building complete Linux filesystem images

user-mode-linux

V:0.11, I:0.5

17816

User-mode Linux (kernel)

xen-tools

V:0.2, I:1.9

996

Tools to manage debian XEN virtual server

bochs

V:0.08, I:0.6

3072

Bochs: IA-32 PC emulator

qemu

V:0.8, I:6

30752

Qemu: fast generic processor emulator

virtualbox-ose

V:2, I:2

22272

VirtualBox: x86 virtualization solution on i386 and amd64

wine

V:2, I:15

96

Wine: Windows API Implementation (standard suite)

dosbox

V:0.6, I:3

2240

DOSBox: x86 emulator with Tandy/Herc/CGA/EGA/VGA/SVGA graphics, sound and DOS

util-vserver

V:0.6, I:1.0

2928

Linux-VServer virtual private servers - user-space tools

vzctl

V:0.3, I:0.7

1120

OpenVZ server virtualization solution - control tools

vzquota

V:0.3, I:0.7

272

OpenVZ server virtualization solution - quota tools


See Wikipedia article http://en.wikipedia.org/wiki/Comparison_of_virtual_machines for detail comparison of different virtualization solutions.

Chapter 11. Data management

11.1. Sharing, copying, and archiving

The security of the data and its controlled sharing have several aspects:

  • the creation of data archive,

  • the remote storage access,

  • the duplication,

  • the tracking of the modification history,

  • the facilitation of data sharing,

  • the prevention of unauthorized file access, and

  • the detection of unauthorized file modification.

These can be realized by using some combination of:

  • the archive and compression tools,

  • the copy and synchronization tools,

  • the network file system,

  • the removable storage media,

  • the secure shell,

  • the authentication system,

  • the version control system tools, and

  • hash and cryptographic encryption tools.

11.1.1. Archive and compression tools

Here is a summary of archive and compression tools available on the Debian system:

Table 11.1.  List of archive and compression tools.

package

popcon

size

command

comment

extension

tar

V:57, I:99

2320

tar(1)

the standard archiver (de facto standard)

.tar

cpio

V:24, I:99

696

cpio(1)

Unix System V style archiver, use with find command

.cpio

binutils

V:36, I:85

7912

ar(1)

archiver for the creation of static libraries

.ar

fastjar

V:11, I:27

220

fastjar(1)

archiver for Java (zip like)

.jar

pax

V:0.7, I:5

156

pax(1)

new POSIX standard archiver, compromise between tar and cpio

.pax

afio

V:0.4, I:1.9

240

afio(1)

extended cpio with per-file compression etc.

.afio

gzip

V:86, I:99

288

gzip(1), zcat(1), ...

GNU LZ77 compression utility (de facto standard)

.gz

bzip2

V:51, I:81

132

bzip2(1), bzcat(1), ...

Burrows-Wheeler block-sorting compression utility with higher compression ratio than gzip(1) (slower than gzip with similar syntax)

.bz2

lzma

V:4, I:34

172

lzma(1)

LZMA compression utility with higher compression ratio than gzip(1) (slower than gzip with similar syntax)

.lzma

p7zip

V:2, I:16

988

7zr(1), p7zip(1)

7-Zip file archiver with high compression ratio (LZMA compression)

.7z

p7zip-full

V:2, I:15

3180

7z(1), 7za(1)

7-Zip file archiver with high compression ratio (LZMA compression and others)

.7z

lzop

V:1.2, I:8

144

lzop(1)

LZO compression utility with higher compression and decompression speed than gzip(1) (lower compression ratio than gzip with similar syntax)

.lzo

zip

V:9, I:65

276

zip(1)

InfoZIP: DOS archive and compression tool

.zip

unzip

V:22, I:74

348

unzip(1)

InfoZIP: DOS unarchive and decompression tool

.zip


[Note] Note

The gzipped .tar archive sometimes uses the file extension .tgz.

[Note] Note

The cp, scp and tar may have some limitation for special files. The cpio and afio are most versatile.

[Note] Note

The cpio and afio commands are designed to be used with the find and other commands and suitable for creating backup scripts since the file selection part of the script can be tested independently.

[Note] Note

afio compresses each file in the archive. This makes afio to be much safer for the file corruption than the globally compressed tar or cpio archives and to be the best archive engine for the backup script.

[Note] Note

Internal structure of OpenOffice data files are .jar file.

11.1.2. Copy and synchronization tools

Here is a summary of simple copy and backup tools available on the Debian system:

Table 11.2.  List of copy and synchronization tools.

package

popcon

size

tool

function

coreutils

V:87, I:99

11448

GNU cp

Locally copy files and directories ("-a" for recursive).

openssh-client

V:54, I:98

2080

scp

Remotely copy files and directories (client). "-r" for recursive.

openssh-server

V:60, I:72

812

sshd

Remotely copy files and directories (remote server).

rsync

V:14, I:37

616

-

1-way remote synchronization and backup.

unison

V:0.9, I:3

1432

-

2-way remote synchronization and backup.

pdumpfs

V:0.08, I:0.19

148

-

Daily local backup using hardlinks, similar to Plan9's dumpfs.


[Tip] Tip

Execution of the bkup script mentioned in Section 11.1.9, “A copy script for the data backup” with the "-gl" option under cron(8) should provide very similar functionality as pdumpfs for the static data archive.

[Tip] Tip

Version control system (VCS) tools in Table 11.14, “ List of version control system tools. ” can function as the multi-way copy and synchronization tools.

11.1.3. Idioms for the archive

Here are several ways to archive and unarchive the entire contents of the directory /source.

With GNU tar:

$ tar cvzf archive.tar.gz /source
$ tar xvzf archive.tar.gz

With cpio:

$ find /source -xdev -print0 | cpio -ov --null > archive.cpio; gzip archive.cpio
$ zcat archive.cpio.gz | cpio -i

With afio:

$ find /source -xdev -print0 | afio -ovZ0 archive.afio
$ afio -ivZ archive.afio

11.1.4. Idioms for the copy

Here are several ways to copy the entire contents of the directory

  • from /source to /dest , and

  • from /source at local to /dest at user@host.dom.

With GNU cp and openSSH scp:

# cp -a /source /dest
# scp -pr /source user@host.dom:/dest

With GNU tar:

# (cd /source && tar cf - . ) | (cd /dest && tar xvfp - )
# (cd /source && tar cf - . ) | ssh user@host.dom '(cd /dest && tar xvfp - )'

With cpio:

# cd /source; find . -print0 | cpio -pvdm --null --sparse /dest

With afio:

# cd /source; find . -print0 | afio -pv0a /dest

The scp command can even copy files between remote hosts:

# scp -pr user1@host1.dom:/source user2@host2.dom:/dest

11.1.5. Idioms for the selection of files

The find(1) command is used to select files for archive and copy commands (see Section 11.1.3, “Idioms for the archive” and Section 11.1.4, “Idioms for the copy”) or for the xargs(1) command (see Section 10.5.9, “Repeating a command looping over files”). This can be enhanced by using its command arguments.

Basic syntax of find(1) can be summarized as:

  • Its conditional arguments are evaluated from left to right.

  • This evaluation stops once its outcome is determined.

  • "Logical OR" (specified by "-o" between conditionals) has lower precedence than "logical AND" (specified by "-a" or nothing between conditionals).

  • "Logical NOT" (specified by "!" before a conditional) has higher precedence than "logical AND".

  • "-prune" always returns logical TRUE and, if it is a directory, searching of file is stopped beyond this point.

  • "-name" matches the base of the filename with shell glob (see Section 2.5.3, “Shell glob”) but it also matches its initial "." with metacharacters such as "*" and "?". (New POSIX feature)

  • "-regex" matches the full path with emacs style BRE (see Section 2.6.2, “Regular expressions”) as default.

  • "-size" matches the file based on the file size (value precedented with "+" for larger, precedented with "-" for smaller)

  • "-newer" matches the file newer than the one specified in its argument.

  • "-print0" always returns logical TRUE and print the full filename (null-terminated) on the standard output.

This find(1) command is often used with an idiomatic style. For example:

# find /path/to \
    -xdev -regextype posix-extended \
    -type f -regex ".*\.afio|.*~" -prune -o \
    -type d -regex ".*/\.git" -prune -o \
    -type f -size +99M -prune -o \
    -type f -newer /path/to/timestamp -print0

This means to do following actions:

  • search all files starting from "/path/to"

  • globally limit its search within its starting filesystem and uses ERE (see Section 2.6.2, “Regular expressions”) instead,

  • exclude files matching regex of ".*\.afio" or ".*~" from search by stop processing,

  • exclude directories matching regex of ".*/\.git" from search by stop processing,

  • exclude files larger than 99 Megabytes (units of 1048576 bytes) from search by stop processing, and

  • print filenames which satisfy above search conditions and newer than "/path/to/timestamp".

Please note the idiomatic use of "-prune -o" to exclude files in the above example.

[Note] Note

For non-Debian unix-like system, some options may not be supported for find(1). In such a case, please consider to adjust matching methods and replace "-print0" with "-print". You may need to adjust related commands too.

11.1.6. Backup and recovery

We all know that computers fail sometime or human errors cause system and data damages. Backup and recovery operations are the essential part of successful system administration. All possible failure modes will hit you some day.

There are 3 key factors which determine actual backup and recovery policy:

  1. Knowing what to backup and recover.
    • Data files directly created by you: data in /$HOME/

    • Data files created by applications used by you: data in /var/ (except /var/cache/, /var/run/, and /var/tmp/).

    • System configuration files: data in /etc/

    • Local softwares: data in /usr/local/ or /opt/

    • System installation information: a memo in plain text on key steps (partition, ...).

    • Proven set of data: experimenting with recovery operations in advance.

  2. Knowing how to backup and recover.
    • Secure storage of data: protection from overwrite and system failure.

    • Frequent backup: scheduled backup.

    • Redundant backup: data mirroring.

    • Fool proof process: easy single command backup.

  3. Assessing risks and costs involved.
    • Failure mode and their possibility.

    • Value of data when lost.

    • Required resources for backup: human, hardware, software, ...

As for secure storage of data, data should be at least on different disk partitions preferably on different disks and machines to withstand the filesystem corruption. Important data are best stored on a write-once media such as CD/DVD-R to prevent overwrite accidents. (See Section 11.2, “The binary data” for how to write to the storage media from the shell commandline. Gnome desktop GUI environment gives you easy access via menu: "Places->CD/DVD Creator".)

[Note] Note

You may wish to stop some application daemons such as MTA (see Section 7.2.5.1, “MTA”) while backing up data.

[Note] Note

You should pay extra care to the backup and restoration of identity related data files such as /etc/ssh/ssh_host_dsa_key, /etc/ssh/ssh_host_rsa_key, $HOME/.gnupg/*, $HOME/.ssh/*, /etc/passwd, /etc/shadow, /etc/fetchmailrc, popularity-contest.conf, /etc/ppp/pap-secrets, and /etc/exim4/passwd.client. Some of these data can not be regenerated by entering the same input string to the system.

[Note] Note

If you run a cron job as a user process, you need to restart it after the system restoration. See Section 10.5.14, “Schedule tasks regularly” for cron(8) and crontab(1).

11.1.7. Backup utility suites

Here is a select list of notable backup utility suites available on the Debian system:

Table 11.3.  List of backup suite utilities.

package

popcon

size

description

sbackup

V:0.11, I:0.3

488

Simple Backup Suite for Gnome desktop

keep

V:0.2, I:0.7

1196

backup system for KDE

rdiff-backup

V:1.2, I:3

764

remote incremental backup

backupninja

V:0.3, I:0.4

408

lightweight, extensible meta-backup system

mondo

V:0.14, I:0.9

1172

Mondo Rescue: disaster recovery backup suite

bacula-common

V:0.9, I:1.9

832

Bacula: network backup, recovery and verification - common support files

bacula-client

I:0.9

60

Bacula: network backup, recovery and verification - client meta-package

bacula-console

V:0.3, I:1.1

340

Bacula: network backup, recovery and verification - text console

bacula-server

I:0.5

60

Bacula: network backup, recovery and verification - server meta-package

amanda-common

V:0.4, I:1.0

3132

Amanda: Advanced Maryland Automatic Network Disk Archiver (Libs)

amanda-client

V:0.3, I:0.9

560

Amanda: Advanced Maryland Automatic Network Disk Archiver (Client)

amanda-server

V:0.14, I:0.4

1264

Amanda: Advanced Maryland Automatic Network Disk Archiver (Server)

cdrw-taper

V:0.00, I:0.08

172

taper replacement for Amanda to support backups to CD-RW or DVD+RW

backuppc

V:0.6, I:0.8

2082

BackupPC is a high-performance, enterprise-grade system for backing up PCs (disk based)

backup-manager

V:0.4, I:0.5

508

command-line backup tool

backup2l

V:0.2, I:0.3

140

low-maintenance backup/restore tool for mountable media (disk based)

faubackup

V:0.2, I:0.2

156

backup system using a filesystem for storage (disk based)


The sbackup and keep packages provide easy GUI access to regular backups of user data for desktop users. An equivalent function can be realized by a simple script (Section 11.1.8, “An example script for the system backup”) and cron(8).

Mondo Rescue facilitates restoration of complete system from backup CD/DVD etc. without going through normal system installation processes.

Bacula, Amanda, and BackupPC are full featured backup suite utilities which are focused on regular backups over network.

11.1.8. An example script for the system backup

For a personal Debian desktop system running unstable suite, I only need to protect personal and critical data. I reinstall system once a year anyway. Thus I see no reason to backup the whole system or to install a full featured backup utility.

I use a simple script to make a backup archive and burn it into CD/DVD using GUI. Here is an example script for this.

#!/bin/sh -e
# Copyright (C) 2007-2008 Osamu Aoki <osamu@debian.org>, Public Domain
BUUID=1000; USER=osamu # UID and name of a user who accesses backup files
BUDIR="/var/backups"
XDIR0=".+/Mail|.+/Desktop"
XDIR1=".+/\.thumbnails|.+/\.?Trash|.+/\.?[cC]ache|.+/\.gvfs|.+/sessions"
XDIR2=".+/CVS|.+/\.git|.+/\.svn|.+/Downloads|.+/Archive|.+/Checkout|.+/tmp"
XSFX=".+\.iso|.+\.tgz|.+\.tar\.gz|.+\.tar\.bz2|.+\.afio|.+\.tmp|.+\.swp|.+~"
SIZE="+99M"
DATE=$(date --utc +"%Y%m%d-%H%M")
[ -d "$BUDIR" ] || mkdir -p "BUDIR"
umask 077
dpkg --get-selections \* > /var/lib/dpkg/dpkg-selections.list
debconf-get-selections > /var/cache/debconf/debconf-selections

{
find /etc /usr/local /opt /var/lib/dpkg/dpkg-selections.list \
     /var/cache/debconf/debconf-selections -xdev -print0
find /home/$USER /root -xdev -regextype posix-extended \
  -type d -regex "$XDIR0|$XDIR1" -prune -o -type f -regex "$XSFX" -prune -o \
  -type f -size  "$SIZE" -prune -o -print0
find /home/$USER/Mail/Inbox /home/$USER/Mail/Outbox -print0
find /home/$USER/Desktop  -xdev -regextype posix-extended \
  -type d -regex "$XDIR2" -prune -o -type f -regex "$XSFX" -prune -o \
  -type f -size  "$SIZE" -prune -o -print0
} | cpio -ov --null -O $BUDIR/BU$DATE.cpio
chown $BUUID $BUDIR/BU$DATE.cpio
touch $BUDIR/backup.stamp

This is meant to be a script example executed from root:

[Tip] Tip

You can recover debconf configuration data with "debconf-set-selections debconf-selections" and dpkg selection data with "dpkg --set-selection <dpkg-selections.list".

11.1.9. A copy script for the data backup

For the set of data under a directory tree, the copy with "cp -a" provides the normal backup.

For the set of large non-overwritten static data under a directory tree such as the data under the /var/cache/apt/packages/ directory, hardlinks with "cp -al" provide an alternative to the normal backup with efficient use of the disk space.

Here is a copy script, which I named as bkup, for the data backup. This script copies all (non-VCS) files under the current directory to the dated directory on the parent directory or on a remote host.

#!/bin/sh -e
# Copyright (C) 2007-2008 Osamu Aoki <osamu@debian.org>, Public Domain
function fdot(){ find . -type d \( -iname ".?*" -o -iname "CVS" \) -prune -o -print0;}
function fall(){ find . -print0;}
function mkdircd(){ mkdir -p "$1";chmod 700 "$1";cd "$1">/dev/null;}
FIND="fdot";OPT="-a";MODE="CPIOP";HOST="localhost";EXTP="$(hostname -f)"
BKUP="$(basename $(pwd)).bkup";TIME="$(date  +%Y%m%d-%H%M%S)";BU="$BKUP/$TIME"
while getopts gcCsStrlLaAxe:h:T f; do case $f in
g)  MODE="GNUCP";; # cp (GNU)
c)  MODE="CPIOP";; # cpio -p
C)  MODE="CPIOI";; # cpio -i
s)  MODE="CPIOSSH";; # cpio/ssh
S)  MODE="AFIOSSH";; # afio/ssh
t)  MODE="TARSSH";; # tar/ssh
r)  MODE="RSYNCSSH";; # rsync/ssh
l)  OPT="-alv";; # hardlink (GNU cp)
L)  OPT="-av";;  # copy (GNU cp)
a)  FIND="fall";; # find all
A)  FIND="fdot";; # find non CVS/ .???/
x)  set -x;; # trace
e)  EXTP="${OPTARG}";; # hostname -f
h)  HOST="${OPTARG}";; # user@remotehost.example.com
T)  MODE="TEST";; # test find mode
\?) echo "use -x for trace."
esac; done
shift $(expr $OPTIND - 1)
if [ $# -gt 0 ]; then
  for x in $@; do cp $OPT $x $x.$TIME; done
elif [ $MODE = GNUCP ]; then
  mkdir -p "../$BU";chmod 700 "../$BU";cp $OPT . "../$BU/"
elif [ $MODE = CPIOP ]; then
  mkdir -p "../$BU";chmod 700 "../$BU"
  $FIND|cpio --null --sparse -pvd ../$BU
elif [ $MODE = CPIOI ]; then
  $FIND|cpio -ov --null | ( mkdircd "../$BU"&&cpio -i )
elif [ $MODE = CPIOSSH ]; then
  $FIND|cpio -ov --null|ssh -C $HOST "( mkdircd \"$EXTP/$BU\"&&cpio -i )"
elif [ $MODE = AFIOSSH ]; then
  $FIND|afio -ov -0 -|ssh -C $HOST "( mkdircd \"$EXTP/$BU\"&&afio -i - )"
elif [ $MODE = TARSSH ]; then
  (tar cvf - . )|ssh -C $HOST "( mkdircd \"$EXTP/$BU\"&& tar xvfp - )"
elif [ $MODE = RSYNCSSH ]; then
  rsync -rlpt ./ "${HOST}:${EXTP}-${BKUP}-${TIME}"
else
  echo "Any other idea to backup?"
  $FIND |xargs -0 -n 1 echo
fi

This is meant to be command examples. Please read script and test it by yourself.

[Tip] Tip

I keep this bkup in my /usr/local/bin/ directory. I issue bkup command without any option in the working directory whenever I need a temporary snapshot backup.

[Tip] Tip

For making snapshot history of a source file tree or a configuration file tree, it is easier and space efficient to use git(7) (see Section 11.5.4.4, “Git for recording configuration history”).

11.1.10. Removable mass storage device

Removable mass storage devices may be any one of

  • harddisk,

  • any format of flash memory devices, or

  • digital camera,

which are connected via USB, IEEE 1394 / Firewire, PC Card, etc.

These removable mass storage devices can be automatically mounted as a user under modern desktop environment, such as Gnome using gnome-mount(1).

  • Mount point under Gnome is chosen as /media/<disk_label> which can be customized

    • by the mlabel(1) command for FAT filesystem,

    • by the genisoimage(1) command with "-V" option for ISO9660 filesystem, and

    • by the tune2fs(1) command with "-L" option for ext2/ext3 filesystem.

  • The choice of encoding may need to be provided as mount option (see Section 9.3.6, “Filename encoding”).

  • The ownership of the mounted filesystem may need to be adjusted for use by the normal user.

[Tip] Tip

When providing wrong mount option causes problem, erase its corresponding setting under /system/storage/ via gconf-editor(1).

[Note] Note

Automounting under modern desktop environment happens only when those removable media devices are not listed in /etc/fstab.

Table 11.4.  List of packages which permit normal users to mount removable devices without a matching /etc/fstab entry.

package

popcon

size

description

gnome-mount

V:14, I:24

844

wrapper for (un)mounting and ejecting storage devices (used by Gnome)

pmount

V:19, I:50

856

mount removable devices as normal user (used by KDE)

cryptmount

V:0.11, I:0.5

292

Management and user-mode mounting of encrypted file systems

usbmount

I:1.9

116

automatically mount and unmount USB mass storage devices


When sharing data with other system via removable mass storage device, you should format it with common filesystem supported by both systems. Here is a list of filesystem choices.

Table 11.5.  List of filesystem choices for removable storage devices with typical usage scenarios.

filesystem

typical usage scenario

FAT12

Cross platform sharing of data on the floppy disk. (<=32MiB)

FAT16

Cross platform sharing of data on the small harddisk like device. (<=2GiB)

FAT32

Cross platform sharing of data on the large harddisk like device. (<=8TiB, supported by newer than MS Windows95 OSR2)

NTFS

Cross platform sharing of data on the large harddisk like device. (supported natively on MS Windows NT and later version, and supported by NTFS-3G via FUSE on Linux)

ISO9660

Cross platform sharing of static data on CD-R and DVD+/-R

UDF

Incremental data writing on CD-R and DVD+/-R (new)

MINIX filesystem

Space efficient unix file data storage on the floppy disk.

ext2 filesystem

Sharing of data on the harddisk like device with older Linux systems.

ext3 filesystem

Sharing of data on the harddisk like device with current Linux systems. (Journaling file system)


[Tip] Tip

See Section 10.4.1, “Removable disk encryption with dm-crypt/LUKS” for cross platform sharing of data using device level encryption.

The FAT filesystem is supported by almost all modern operating systems and is quite useful for the data exchange purpose via removable harddisk like media (.

When formatting removable harddisk like devices for cross platform sharing of data with the FAT filesystem, the following should be safe choices:

  • Partitioning them with fdisk, cfdisk or parted command (see Section 10.3.1, “Partition configuration”) into a single primary partition and to mark it as:

    • type-"6" for FAT16 for media smaller than 2GB or

    • type-"c" for FAT32 (LBA) for larger media.

  • Formatting the primary partition with the mkfs.vfat command

    • with just its device name, e.g. "/dev/sda1" for FAT16, or

    • with the explicit option and its device name, e.g. "-F 32 /dev/sda1" for FAT32.

When using the FAT or ISO9660 filesystems for sharing data, the following should be the safe considerations:

  • Archiving files into an archive file first using the tar(1), cpio(1), or afio(1) command to retain the long filename, the symbolic link, the original Unix file permission and the owner information.

  • Splitting the archive file size into less than 2 GiB chunks with the "split(1)" command to protect it from the file size limitation.

  • Encrypting the archive file to secure its contents from the unauthorized access.

[Note] Note

For FAT filesystems by its design, the maximum file size is (2^32 - 1) bytes = (4GiB - 1 byte). For some applications on the older 32 bit OSs, the maximum file size was even smaller (2^31 - 1) bytes = (42GiB - 1 byte). Debian does not suffer the latter problem.

[Note] Note

Microsoft itself does not recommend to use FAT for drives or partitions of over 200 MB. Microsoft highlights its short comings such as inefficient disk space usage in their "Overview of FAT, HPFS, and NTFS File Systems". Of course for the Linux, we should normally use the ext3 filesystem.

[Tip] Tip

For more on filesystems and accessing filesystems, please read "Filesystems HOWTO".

11.1.11. Sharing data via network

When sharing data with other system via network, you should use common service. Here are some hints.

Table 11.6.  List of the network service to chose with the typical usage scenario.

network service

typical usage scenario

SMB/CIFS network mounted filesystem with Samba

Sharing files via "Microsoft Windows Network". See smb.conf(5) and The Official Samba 3.2.x HOWTO and Reference Guide or the samba-doc package.

NFS network mounted filesystem with the Linux kernel

Sharing files via "Unix/Linux Network". See exports(5) and Linux NFS-HOWTO.

HTTP service

Sharing file between the web server/client.

HTTPS service

Sharing file between the web server/client with encrypted Secure Sockets Layer (SSL) or Transport Layer Security (TLS).

FTP service

Sharing file between the FTP server/client.


Although these filesystems mounted over network or file transfer methods over network are quite convenient for sharing data, these may be insecure. Their network connection must be secured by:

  • encrypting it with SSL/TLS,

  • tunneling it via SSH,

  • tunneling it via VPN or,

  • limiting it behind the secure firewall.

See also Section 7.5, “Other network application servers” and Section 7.6, “Other network application clients”.

11.1.12. Archive media

When choosing computer data storage media for important data archive, you should be careful about their limitations. For small personal data back up, I use CD-R and DVD-R by the brand name company and store in a cool, dry, clean environment. (Tape archive media seem to be popular for professional use.)

[Note] Note

A fire-resistant safe are usually meant for paper documents. Most of the computer data storage media have less temperature tolerance than paper. I usually rely on multiple secure encrypted copies stored in multiple secure locations.

Optimistic storage life of archive media seen on the net (mostly from vendor info):

  • 100+ years : acid free paper with ink

  • 100 years : optical storage (CD/DVD, CD/DVD-R)

  • 30 years : magnetic storage (tape, floppy)

  • 20 years : phase change optical storage (CD-RW)

These do not count on the mechanical failures due to handling etc.

Optimistic write cycle of archive media seen on the net (mostly from vendor info):

  • 250,000+ cycles : Harddisk drive

  • 10,000+ cycles : Flash memory

  • 1,000 cycles : CD/DVD-RW

  • 1 cycles : CD/DVD-R, paper

[Caution] Caution

Figures of storage life and write cycle here should not be used for decisions on any critical data storage. Please consult the specific product information provided by the manufacture.

[Tip] Tip

Since CD/DVD-R and paper have only 1 write cycle, they inherently prevent accidental data loss by overwriting. This is advantage!

[Tip] Tip

If you need fast and frequent backup of large amount of data, a harddisk on a remote host linked by a fast network connection, may be the only realistic option.

11.2. The binary data

Here, we discuss direct manipulation of the binary data on storage media. See Section 10.3, “Data storage tips”, too.

11.2.1. Make the disk image file

The disk image file, disk.img, of an unmounted device, e.g., the second SCSI drive /dev/sdb, can be made using cp(1) or dd(1):

# cp /dev/sda disk.img
# dd if=/dev/sda of=disk.img

The disk image of the traditional PC's master boot record (MBR) (see Section 10.3.1, “Partition configuration”) which reside on the first sector on the primary IDE disk partial disk can be made by using dd(1):

# dd if=/dev/hda of=mbr.img bs=512 count=1
# dd if=/dev/hda of=mbr-nopart.img bs=446 count=1
# dd if=/dev/hda of=mbr-part.img skip=446 bs=1 count=66
  • mbr.img : the MBR with the partition table.

  • mbr-nopart.img : the MBR without the partition table.

  • part.img : the partition table of the MBR only..

If you have a SCSI device (including the new serial ATA drive) as the boot disk, substitute "/dev/hda" with "/dev/sda".

If you are making an image of a disk partition of the original disk, substitute "/dev/hda" with "/dev/hda1" etc.

11.2.2. Writing directly to the disk

The disk image file, disk.img can be written to an unmounted device, e.g., the second SCSI drive /dev/sdb with matching size, by dd(1):

# dd if=disk.img of=/dev/sda

Similarly, the disk partition image file, disk.img can be written to an unmounted partition, e.g., the first partition of the second SCSI drive /dev/sdb1 with matching size, by dd(1):

# dd if=disk.img of=/dev/sda1

11.2.3. View and edit binary data

The most basic viewing method of binary data is to use "od -t x1" command.

Table 11.7.  List of packages which view and edit binary data.

package

popcon

size

description

coreutils

V:87, I:99

11448

This basic package has od(1) command to dump files in octal and other formats.

bsdmainutils

V:47, I:99

644

This utility package has hd(1) command to dump files in ASCII, decimal, hexadecimal, and octal formats.

hexedit

V:0.3, I:2

108

View and edit files in hexadecimal or in ASCII

bless

V:0.04, I:0.2

1200

Full featured hexadecimal editor (Gnome)

khexedit

V:1.8, I:14

1404

Full featured hexadecimal editor (KDE).

ncurses-hexedit

V:0.10, I:0.6

196

Edit files/disks in HEX, ASCII and EBCDIC

lde

V:0.05, I:0.5

992

Linux Disk Editor

beav

V:0.05, I:0.3

164

Binary editor and viewer for HEX, ASCII, EBCDIC, OCTAL, DECIMAL, and BINARY formats.

hexcat

V:0.05, I:0.2

56

Hexadecimal dumping utility

hex

V:0.01, I:0.10

84

Hexadecimal dumping tool for Japanese


[Tip] Tip

HEX is used as an acronym for hexadecimal format.

11.2.4. Mount the disk image file

If disk.img contains an image of the disk contents and the original disk had a disk configuration which gives xxxx = (bytes/sector) * (sectors/cylinder), then the following will mount it to /mnt:

# mount -o loop,offset=xxxx disk.img /mnt

Note that most hard disks have 512 bytes/sector. This offset is to skip MBR of the hard disk. You can skip offset in the above example, if disk.img contains

  • only an image of a disk partition of the original hard disk, or

  • only an image of the original floppy disk.

11.2.5. Manipulating files without mounting disk

There are tools to write files without mounting disk.

Table 11.8.  List of packages to manipulate files without mounting.

package

popcon

size

description

mtools

V:7, I:84

476

Utilities for MSDOS files without mounting them.

hfsutils

V:0.2, I:1.9

236

Utilities for HFS and HFS+ files without mounting them.


11.2.6. Make the ISO9660 image file

The ISO9660 image file, cd.iso, from the source directory tree at source_directory can be made using genisoimage(1) command:

#  genisoimage -r -J -T -V volume_id -o cd.iso source_directory

Similary, the bootable ISO9660 image file, cdboot.iso, can be made from debian-installer like directory tree at source_directory:

#  genisoimage -r -o cdboot.iso -V volume_id \
   -b isolinux/isolinux.bin -c isolinux/boot.cat \
   -no-emul-boot -boot-load-size 4 -boot-info-table source_directory

Here Isolinux boot loader (see Section 4.3, “Stage 2: the boot loader”) is used for booting.

To make the disk image directly from the CD-ROM device using cp(1) or dd(1) has a few problems. The first run of the dd(1) command may cause an error message and may yield a shorter disk image with a lost tail-end. The second run of the dd(1) command may yield a larger disk image with garbage data attached at the end on some systems if the data size is not specified. Only the second run of the dd(1) command with the correct data size specified, and without ejecting the CD after an error message, seems to avoid these problems. If for example the image size displayed by df(1) is 46301184 blocks, use the following command twice to get the right image (this is my empirical information):

# dd if=/dev/cdrom of=cd.iso bs=2048 count=$((46301184/2))

11.2.7. Writing directly to the CD/DVD-R/RW

[Tip] Tip

DVD is only a large CD to wodim(1).

You can find a usable device by:

# wodim --devices

Then the blank CD-R is inserted to the device, and the ISO9660 image file, cd.iso is written to this device, e.g., /dev/hda, by wodim(1):

# wodim -v -eject dev=/dev/hda cd.iso

If CD-RW is used instead of CD-R, do this instead:

# wodim -v -eject blank=fast dev=/dev/hda cd.iso
[Tip] Tip

If your desktop system mounts CD automatically, unmount it before issuing the wodim(1) command by "sudo unmount /dev/hda".

11.2.8. Mount the ISO9660 image file

If cd.iso contains an ISO9660 image, then the following will manually mount it to /cdrom:

# mount -t iso9660 -o ro,loop cd.iso /cdrom
[Tip] Tip

Modern desktop system mounts removable media automatically (see Section 11.1.10, “Removable mass storage device”).

11.2.9. Split a large file into small files

When a data is too big to backup, you can back up a large file into, e.g. 2000MiB chunks and merge those files into a large file.

$ split -b 2000m large_file
$ cat x* >large_file
[Caution] Caution

Please make sure you do not have any file starting with "x" to avoid the file name crash.

11.2.10. Clear file contents

In order to clear the contents of a file such as a log file, do not use rm to delete the file and then create a new empty file, because the file may still be accessed in the interval between commands. The following is the safe way to clear the contents of the file.

$ :>file_to_be_cleared

11.2.11. Dummy files

The following commands will create dummy or empty files:

$ dd if=/dev/zero    of=5kb.file bs=1k count=5
$ dd if=/dev/urandom of=7mb.file bs=1M count=7
$ touch zero.file
$ : > alwayszero.file
  • 5kb.file is 5KB of zeros.

  • 7mb.file is 7MB of random data.

  • zero.file is 0 byte file (if file exists, the file contents are kept while updating mtime.)

  • alwayszero.file is always 0 byte file (if file exists, the file contents are not kept while updating mtime.)

11.2.12. Erase entire harddisk

There are several ways to completely erase data from an entire harddisk-like device, e.g., USB memory stick at /dev/sda.

[Caution] Caution

Check your USB memory stick location with the "mount" command first before executing commands here. The device pointed by /dev/sda may be SCSI harddisk or serial-ATA harddisk where your entire system resides.

  • Erase all by resetting data to 0:

dd if=/dev/zero of=/dev/sda
  • Erase all by overwriting random data:

# dd if=/dev/urandom of=/dev/sda
  • Erase all by overwriting random data very efficiently (fast):

# shred -v -n 1 /dev/sda

Since the dd command is available from the shell of many bootable Linux CDs such as Debian installer CD, you can erase your installed system completely by running an erase command from such media on the system hard disk, e.g., /dev/hda, /dev/sda, etc.

11.2.13. Undelete deleted but still open file

Even if you have accidentally deleted a file, as long as that file is still being used by some application (read or write mode), it is possible to recover such a file.

  • On one terminal:

$ echo foo > bar
$ less bar
  • Then on another terminal:

$ ps aux | grep ' less[ ]'
osamu    4775  0.0  0.0  92200   884 pts/8    S+   00:18   0:00 less bar
$ rm bar
$ ls -l /proc/4775/fd | grep bar
lr-x------ 1 osamu osamu 64 2008-05-09 00:19 4 -> /home/osamu/bar (deleted)
$ cat /proc/4775/fd/4 >bar
$ ls -l
-rw-r--r-- 1 osamu osamu 4 2008-05-09 00:25 bar
$ cat bar
foo
  • Alternatively, when you have the lsof command installed, on another terminal:

$ ls -li bar
2228329 -rw-r--r-- 1 osamu osamu 4 2008-05-11 11:02 bar
$ lsof |grep bar|grep less
less 4775 osamu 4r REG 8,3 4 2228329 /home/osamu/bar
$ rm bar
$ lsof |grep bar|grep less
less 4775 osamu 4r REG 8,3 4 2228329 /home/osamu/bar (deleted)
$ cat /proc/4775/fd/4 >bar
$ ls -li bar
2228302 -rw-r--r-- 1 osamu osamu 4 2008-05-11 11:05 bar
$ cat bar
foo

11.2.14. Searching all hardlinks

Files with hardlinks can be identified by "ls -li", e.g.:

$ ls -li
total 0
2738405 -rw-r--r-- 1 root root 0 2008-09-15 20:21 bar
2738404 -rw-r--r-- 2 root root 0 2008-09-15 20:21 baz
2738404 -rw-r--r-- 2 root root 0 2008-09-15 20:21 foo

Both "baz" and "foo" have link count of "2" (>1) showing them to have hardlinks. Their inode numbers are common "2738404". This means they are the same hardlinked file. If you do not happen to find all hardlinked files by chance, you can search it by the inode, e.g., "2738404":

# find /path/to/mount/point -xdev -inum 2738404 

11.2.15. Invisible disk space consumption

All deleted but open files consumes disk space although they are not visible from normal du(1). They can be listed with their size by:

# lsof -s -X / |grep deleted

11.3. Data security infrastructure

The data security infrastructure is provided by the combination of data encryption tool, message digest tool, and signature tool.

Table 11.9.  List of data security infrastructure tools.

package

popcon

size

function

gnupg

V:30, I:99

4972

GNU privacy guard - OpenPGP encryption and signing tool. gpg(1)

gnupg-doc

I:1.7

3976

GNU Privacy Guard documentation

gpgv

V:60, I:98

332

GNU privacy guard - signature verification tool

cryptsetup

V:2, I:4

840

Utilities for dm-crypto block device encryption supporting LUKS

ecryptfs-utils

V:0.03, I:0.17

396

Utilities for ecryptfs stacked filesystem encryption

coreutils

V:87, I:99

11448

The md5sum command computes and checks MD5 message digest

coreutils

V:87, I:99

11448

The sha1sum command computes and checks SHA1 message digest

openssl

V:28, I:89

2348

The "openssl dgst" command computes message digest (OpenSSL). dgst(1ssl)


See Section 10.4, “Data encryption tips” on dm-crypto and ecryptfs which implement automatic data encryption infrastructure via Linux kernel modules.

11.3.1. Key management for Gnupg

Here are GNU Privacy Guard commands for the basic key management:

Table 11.10.  List of GNU Privacy Guard commands for the key management

command

effects

gpg --gen-key

generate a new key

gpg --gen-revoke my_user_ID

generate revoke key for my_user_ID

gpg --edit-key user_ID

"help" for help, interactive

gpg -o file --exports

export all keys to file

gpg --imports file

import all keys from file

gpg --send-keys user_ID

send key of user_ID to keyserver

gpg --recv-keys user_ID

recv. key of user_ID from keyserver

gpg --list-keys user_ID

list keys of user_ID

gpg --list-sigs user_ID

list sig. of user_ID

gpg --check-sigs user_ID

check sig. of user_ID

gpg --fingerprint user_ID

check fingerprint of "user_ID"

gpg --refresh-keys

update local keyring


Here is the meaning of trust code:

Table 11.11.  List of the meaning of trust code.

code

trust

-

No owner trust assigned / not yet calculated.

e

Trust calculation has failed.

q

Not enough information for calculation.

n

Never trust this key.

m

Marginally trusted.

f

Fully trusted.

u

Ultimately trusted.


The following will upload my key "A8061F32" to the popular keyserver hkp://subkeys.pgp.net:

$ gpg --keyserver hkp://subkeys.pgp.net --send-keys A8061F32

A good default keyserver set up in $HOME/.gnupg/gpg.conf (or old location $HOME/.gnupg/options) contains:

keyserver hkp://subkeys.pgp.net

The following will obtain unknown keys from the keyserver:

$ gpg --list-sigs | \
  sed -n '/^sig.*\[User ID not found\]/s/^sig..........\(\w\w*\)\W.*/\1/p' |\
  sort | uniq | xargs gpg --recv-keys

There was a bug in OpenPGP Public Key Server (pre version 0.9.6) which corrupted key with more than 2 sub-keys. The newer gnupg (>1.2.1-2) can handle these corrupted subkeys. See gpg(1) manpage under --repair-pks-subkey-bug option.

11.3.2. Using GnuPG with files

File handling:

Table 11.12.  List of gnu privacy guard commands on files

command

effects

gpg -a -s file

sign file into ascii armored file.asc

gpg --armor --sign file

, ,

gpg --clearsign file

clear-sign message

gpg --clearsign --not-dash-escaped patchfile

clear-sign patchfile

gpg --verify file

verify clear-signed file

gpg -o file.sig -b file

create detached signature

gpg -o file.sig --detach-sig file

, ,

gpg --verify file.sig file

verify file with file.sig

gpg -o crypt_file.gpg -r name -e file

public-key encryption intended for name from file to binary crypt_file.gpg

gpg -o crypt_file.gpg --recipient name --encrypt file

, ,

gpg -o crypt_file.asc -a -r name -e file

public-key encryption intended for name from file to ASCII armored crypt_file.asc

gpg -o crypt_file.gpg -c file

symmetric encryption from file to crypt_file.gpg

gpg -o crypt_file.gpg --symmetric file

, ,

gpg -o crypt_file.asc -a -c file

symmetric encryption intended for name from file to ASCII armored crypt_file.asc

gpg -o file -d crypt_file.gpg -r name

decryption

gpg -o file --decrypt crypt_file.gpg

, ,


11.3.3. Using GnuPG with Mutt

Add the following to ~/.muttrc to keep a slow GnuPG from automatically starting, while allowing it to be used by typing "S" at the index menu.

macro index S ":toggle pgp_verify_sig\n"
set pgp_verify_sig=no

11.3.4. Using GnuPG with Vim

The gnupg plugin let you run GnuPG transparently for files with extension .gpg, .asc, and .ppg.

# aptitude install vim-scripts vim-addon-manager
$ vim-addons install gnupg

11.3.5. The MD5 sum

The md5sum program provides utility to make a digest file using the method in rfc1321 and verifying each file with it.

$ md5sum foo bar >baz.md5
$ cat baz.md5
d3b07384d113edec49eaa6238ad5ff00  foo
c157a79031e1c40f85931829bc5fc552  bar
$ md5sum -c baz.md5
foo: OK
bar: OK
[Note] Note

The computation for the MD5 sum is less CPU intensive than the one for the cryptographic signature by the Gnupg. Usually, only the top level digest file is cryptographically signed to ensure data integrity.

11.4. Source code merge tools

There are many merge tools for the source code. Following commands caught my eyes.:

Table 11.13.  List of source code merge tools.

command

package

popcon

size

description

diff(1)

diff

V:86, I:99

764

This compares files line by line.

diff3(1)

diff

V:86, I:99

764

This compares and merges three files line by line.

vimdiff(1)

vim

V:12, I:29

1736

This compares 2 files side by side in vim.

patch(1)

patch

V:9, I:94

204

This applies a diff file to an original.

dpatch(1)

dpatch

V:2, I:15

344

This manage series of patches for Debian package.

diffstat(1)

diffstat

V:1.8, I:12

84

This produces a histogram of changes by the diff.

combinediff(1)

patchutils

V:2, I:12

288

This creates a cumulative patch from two incremental patches.

dehtmldiff(1)

patchutils

V:2, I:12

288

This extracts a diff from an HTML page.

filterdiff(1)

patchutils

V:2, I:12

288

This extracts or excludes diffs from a diff file.

fixcvsdiff(1)

patchutils

V:2, I:12

288

This fixes diff files created by CVS that "patch" mis-interprets.

flipdiff(1)

patchutils

V:2, I:12

288

This exchanges the order of two patches.

grepdiff(1)

patchutils

V:2, I:12

288

This shows which files are modified by a patch matching a regex.

interdiff(1)

patchutils

V:2, I:12

288

This shows differences between two unified diff files.

lsdiff(1)

patchutils

V:2, I:12

288

This shows which files are modified by a patch.

recountdiff(1)

patchutils

V:2, I:12

288

This recomputes counts and offsets in unified context diffs.

rediff(1)

patchutils

V:2, I:12

288

This fixes offsets and counts of a hand-edited diff.

splitdiff(1)

patchutils

V:2, I:12

288

This separates out incremental patches.

unwrapdiff(1)

patchutils

V:2, I:12

288

This demangles patches that have been word-wrapped.

wiggle(1)

wiggle

V:0.03, I:0.12

204

This applies rejected patches.

quilt(1)

quilt

V:0.9, I:6

824

This manage series of patches.

meld(1)

meld

V:0.5, I:2

2140

This is a GTK graphical file comparator and merge tool.

xxdiff(1)

xxdiff

V:0.2, I:1.1

1360

This is a plain X graphical file comparator and merge tool.

dirdiff(1)

dirdiff

V:0.09, I:0.5

212

This displays and merges changes between directory trees.

docdiff(1)

docdiff

V:0.03, I:0.19

688

This compares two files word by word / char by char.

imediff2(1)

imediff2

V:0.03, I:0.12

76

This is an interactive full screen 2-way merge tool.

makepatch(1)

makepatch

V:0.03, I:0.2

148

This generates extended patch files.

applypatch(1)

makepatch

V:0.03, I:0.2

148

This applies extended patch files.

wdiff(1)

wdiff

V:1.5, I:11

124

This displays word differences between text files.


11.4.1. Extract differences for source files

Following one of these procedures will extract differences between two source files and create unified diff files file.patch0 or file.patch1 depending on the file location:

$ diff -u file.old file.new > file.patch0
$ diff -u old/file new/file > file.patch1

11.4.2. Merge updates for source files

The diff file (alternatively called patch file) is used to send a program update. The receiving party will apply this update to another file by:

$ patch -p0 file < file.patch0
$ patch -p1 file < file.patch1

11.4.3. 3 way merge updates

If you have three versions of source code, you can merge them more effectively using diff3:

$ diff3 -m file.mine file.old file.yours > file

11.5. Version control systems

Here is a summary of the version control systems (VCS) on the Debian system:

[Note] Note

If you are new to VCS systems, you should start learning with Git, which is growing fast in popularity.

Table 11.14.  List of version control system tools.

package

popcon

size

tool

VCS type

comment

cssc

V:0.00, I:0.04

2168

CSSC

local

Clone of the Unix SCCS (deprecated)

rcs

V:1.8, I:10

780

RCS

local

"Unix SCCS done right"

cvs

V:4, I:27

3660

CVS

remote

The previous standard remote VCS

subversion

V:9, I:29

4168

Subversion

remote

"CVS done right", the new de facto standard remote VCS

git-core

V:4, I:8

7164

Git

distributed

fast DVCS in C (used by the Linux kernel and others)

mercurial

V:0.9, I:3

376

Mercurial

distributed

DVCS in python and some C.

bzr

V:0.4, I:2

14964

Bazaar

distributed

DVCS influenced by tla written in python (used by Ubuntu)

darcs

V:0.3, I:1.7

7996

Darcs

distributed

DVCS with smart algebra of patches (slow).

tla

V:0.19, I:1.3

1104

GNU arch

distributed

DVCS mainly by Tom Lord. (Historic)

monotone

V:0.07, I:0.4

6016

Monotone

distributed

DVCS in C++


VCS is sometimes known as revision control system (RCS), or software configuration management (SCM).

Distributed VCS such as Git is the tool of choice these days. CVS and Subversion may still be useful to join some existing open source program activities.

Debian provides free VCS services via Debian Alioth service. It supports practically all VCSs. Its documentation can be found at http://wiki.debian.org/Alioth .

[Caution] Caution

The git package is "GNU Interactive Tools" which is not the DVCS.

11.5.1. Native VCS commands

Here is an oversimplified comparison of native VCS commands to provide the big picture. The typical command sequence may require options and arguments.

Table 11.15.  Comparison of native VCS commands.

CVS

Subversion

Git

function

cvs init

svn create

git init

create the (local) repository

cvs login

-

-

login to the remote repository

cvs co

svn co

git clone

check out the remote repository as the working tree

cvs up

svn up

git pull

update the working tree by merging the remote repository

cvs add

svn add

git add .

add file(s) in the working tree to the VCS

cvs rm

svn rm

git rm

remove file(s) in working tree from the VCS

cvs ci

svn ci

-

commit changes to the remote repository

-

-

git commit -a

commit changes to the local repository

-

-

git push

update the remote repository by the local repository

cvs status

svn status

git status

display the working tree status from the VCS

cvs diff

svn diff

git diff

diff <reference_repository> <working_tree>

-

-

git repack -a -d; git prune

repack the local repository into single pack.


[Caution] Caution

Invoking a git subcommand as "git-xyzzy" from the command line has been deprecated since early 2006.

[Tip] Tip

Git can work directly with different VCS repositories such as ones provided by CVS and Subversion, and provides the local repository for local changes with the git-cvs and git-svn packages. See git for CVS users, Git for GNOME developers and Section 11.5.4, “Git”.

[Note] Note

Git has commands which have no equivalents in CVS and Subversion. "Fetch", "Rebase", "Cherrypick", ...

11.5.2. CVS

Check

  • "sensible-browser file:///usr/share/doc/cvs/html-cvsclient",

  • "sensible-browser file:///usr/share/doc/cvs/html-info",

  • "sensible-browser file:///usr/share/doc/cvsbook",

  • "info cvs", and

  • "man cvs"

for detailed information.

11.5.2.1. Installing a CVS server

The following setup will allow commits to the CVS repository only by a member of the "src" group, and administration of CVS only by a member of the "staff" group, thus reducing the chance of shooting oneself.

# cd /var/lib; umask 002; mkdir cvs
# export CVSROOT=/var/lib/cvs
# cd $CVSROOT
# chown root:src .
# chmod 2775 .
# cvs -d $CVSROOT init
# cd CVSROOT
# chown -R root:staff .
# chmod 2775 .
# touch val-tags
# chmod 664 history val-tags
# chown root:src history val-tags

You may restrict creation of new project by changing the owner of $CVSROOT directory to "root:staff and its permission to "3775".

11.5.2.2. Use local CVS server

The following will set up shell environments for the local access to the CVS repository:

$ export CVSROOT=/var/lib/cvs

11.5.2.3. Use remote CVS pserver

The following will set up shell environments for the read-only remote access to the CVS repository without SSH (use RSH protocol capability in cvs):

$ export CVSROOT=:pserver:account@cvs.foobar.com:/var/lib/cvs
$ cvs login

This is prone to eavesdropping attack.

11.5.2.4. Anonymous CVS (download only)

The following will set up shell environments for the read-only remote access to the CVS repository:

$ export CVSROOT=:pserver:anonymous@cvs.sf.net:/cvsroot/qref
$ cvs login
$ cvs -z3 co qref

11.5.2.5. Use remote CVS through ssh

The following will set up shell environments for the read-only remote access to the CVS repository with SSH:

$ export CVSROOT=:ext:account@cvs.foobar.com:/var/lib/cvs

or for SourceForge:

$ export CVSROOT=:ext:account@cvs.sf.net:/cvsroot/qref

You can also use public key authentication for SSH which eliminates the password prompt.

11.5.2.6. Create a new CVS archive

For,

Table 11.16.  Assumption for the CVS archive.

ITEM

VALUE

MEANING

source tree

~/project-x

All source codes

Project name

project-x

Name for this project

Vendor Tag

Main-branch

Tag for the entire branch

Release Tag

Release-initial

Tag for a specific release


Then,

$ cd ~/project-x
  • create a source tree ...

$ cvs import -m "Start project-x" project-x Main-branch Release-initial
$ cd ..; rm -R ~/project-x

11.5.2.7. Work with CVS

To work with project-x using the local CVS repository:

$ mkdir -p /path/to; cd /path/to
$ cvs co project-x
  • get sources from CVS to local

$ cd project-x
  • make changes to the content ...

$ cvs diff -u
  • similar to "diff -u repository/ local/"

$ cvs up -C modified_file
  • undo changes to a file

$ cvs ci -m "Describe change"
  • save local sources to CVS

$ vi newfile_added
$ cvs add newfile_added
$ cvs ci -m "Added newfile_added"
$ cvs up
  • merge latest version from CVS.

  • To create all newly created subdirectories from CVS, use "cvs up -d -P" instead.

  • Watch out for lines starting with "C filename" which indicates conflicting changes.

  • unmodified code is moved to .#filename.version .

  • search for "<<<<<<<" and ">>>>>>>" in the files for conflicting changes.

  • edit file to fix conflicts.

$ cvs tag Release-1
  • add release tag

  • edit further ...

$ cvs tag -d Release-1
  • remove release tag

$ cvs ci -m "more comments"
$ cvs tag Release-1

* re-add release tag {{ $ cd /path/to $ cvs co -r Release-initial -d old project-x }}}

  • get original version to "/path/to/old" directory

$ cd old
$ cvs tag -b Release-initial-bugfixes
  • create branch (-b) tag "Release-initial-bugfixes"

  • now you can work on the old version (Tag is sticky)

$ cvs update -d -P
  • don't create empty directories

  • source tree now has sticky tag "Release-initial-bugfixes"

  • work on this branch ... while someone else making changes too

$ cvs up -d -P
  • sync with files modified by others on this branch

$ cvs ci -m "check into this branch"
$ cvs update -kk -A -d -P
  • remove sticky tag and forget contents

  • update from main trunk without keyword expansion

$ cvs update -kk -d -P -j Release-initial-bugfixes
  • merge from Release-initial-bugfixes branch into the main

  • trunk without keyword expansion. Fix conflicts with editor.

$ cvs ci -m "merge Release-initial-bugfixes"
$ cd
$ tar -cvzf old-project-x.tar.gz old
  • make archive. use "-j" if you want .tar.bz2 .

$ cvs release -d old
  • remove local source (optional)

Table 11.17.  Notable options for CVS commands (use as first argument(s) to cvs).

option

meaning

-n

dry run, no effect

-t

display messages showing steps of cvs activity


11.5.2.8. Export files from CVS

To get the latest version from CVS, use "tomorrow":

$ cvs ex -D tomorrow module_name

11.5.2.9. Administer CVS

Add alias to a project (local server):

$ export CVSROOT=/var/lib/cvs
$ cvs co CVSROOT/modules
$ cd CVSROOT
$ echo "px -a project-x" >>modules
$ cvs ci -m "Now px is an alias for project-x"
$ cvs release -d .
$ cvs co -d project px
  • check out project-x (alias:px) from CVS to directory project

$ cd project
  • make changes to the content ...

In order to perform above procedure, you should have the appropriate file permission.

11.5.2.10. File permissions in repository

CVS will not overwrite the current repository file but replaces it with another one. Thus, write permission to the repository directory is critical. For every new repository creation, run the following to ensure this condition if needed.

# cd /var/lib/cvs
# chown -R root:src repository
# chmod -R ug+rwX   repository
# chmod    2775     repository

11.5.2.11. Execution bit

A file's execution bit is retained when checked out. Whenever you see execution permission problems in checked-out files, change permissions of the file in the CVS repository with the following command.

# chmod ugo-x filename

11.5.3. Subversion

Subversion is a "next-generation" version control system, intended to replace CVS, so it has most of CVS's features. Generally, Subversion's interface to a particular feature is similar to CVS's, except where there's a compelling reason to do otherwise.

11.5.3.1. Installing a Subversion server

You need to install the subversion, libapache2-svn and subversion-tools packages to set up a server.

11.5.3.2. Setting up a repository

Currently, the subversion package does not set up a repository, so one must be set up manually. One possible location for a repository is in /var/local/repos.

Create the directory:

# mkdir -p /var/local/repos

Create the repository database:

# svnadmin create /var/local/repos

Make the repository writable by the WWW server:

# chown -R www-data:www-data /var/local/repos

11.5.3.3. Configuring Apache2

To allow access to the repository via user authentication, add (or uncomment) the following in /etc/apache2/mods-available/dav_svn.conf:

<Location /repos>
  DAV svn
  SVNPath /var/local/repos
  AuthType Basic
  AuthName "Subversion repository"
  AuthUserFile /etc/subversion/passwd
<LimitExcept GET PROPFIND OPTIONS REPORT>
    Require valid-user
</LimitExcept>
</Location>

Then, create a user authentication file with the command:

htpasswd2 -c /etc/subversion/passwd some-username

Restart Apache2, and your new Subversion repository will be accessible with the URL http://hostname/repos.

11.5.3.4. Subversion usage examples

The following sections teach you how to use different commands in Subversion.

11.5.3.5. Create a new Subversion archive

To create a new Subversion archive, type the following:

$ cd ~/your-project         # go to your source directory
$ svn import http://localhost/repos your-project project-name -m "initial project import"

This creates a directory named project-name in your Subversion repository which contains your project files. Look at http://localhost/repos/ to see if it's there.

11.5.3.6. Working with Subversion

Working with project-y using Subversion:

$ mkdir -p /path/to ;cd  /path/to
$ svn co http://localhost/repos/project-y
  • Check out sources

$ cd project-y
  • do some work ...

$ svn diff
  • similar to "diff -u repository/ local/"

$ svn revert modified_file
  • undo changes to a file

$ svn ci -m "Describe changes"
  • check in your changes to the repository

$ vi newfile_added
$ svn add newfile_added
$ svn add new_dir
  • recursively add all files in new_dir

$ svn add -N new_dir2
  • non recursively add the directory

$ svn ci -m "Added newfile_added, new_dir, new_dir2"
$ svn up
  • merge in latest version from repository

$ svn log
  • shows all changes committed

$ svn copy http://localhost/repos/project-y \
      http://localhost/repos/project-y-branch \
      -m "creating my branch of project-y"
  • branching project-y

$ svn copy http://localhost/repos/project-y \
      http://localhost/repos/projct-y-release1.0 \
      -m "project-y 1.0 release"
  • added release tag.

  • note that branching and tagging are the same. The only difference is that branches get committed whereas tags do not.

  • make changes to branch ...

$ svn merge http://localhost/repos/project-y \
   http://localhost/repos/project-y-branch
  • merge branched copy back to main copy

$ svn co -r 4 http://localhost/repos/project-y
  • get revision 4

11.5.4. Git

Git can do everything for both local and remote source code management. This means that you can record the source code changes without needing network connectivity to the remote repository.

11.5.4.1. Before using Git

You may wish to set several global configuration in ~/.gitconfig such as your name and email address used by Git:

$ git config --global user.name "Name Surname"
$ git config --global user.email yourname@example.com

If you are too used to CVS or Subversion commands, you may wish to set several command aliases;

$ git config --global alias.ci "commit -a"
$ git config --global alias.co checkout

You can check your global configuration by:

$ git config --global --list

11.5.4.2. Git references

There are good references for Git.

The git-gui and gitk commands make using Git very easy.

[Warning] Warning

Do not use the tag string with spaces in it even if some tools such as gitk allow you to use it. It will choke some other git commands.

11.5.4.3. Git commands

Even if your upstream uses different VCS, it is good idea to use git(1) for local activity since you can manage your local copy of source tree without the network connection to the upstream. Here are commands used with git(1).

Table 11.18.  List of git packages and commands.

command

package

popcon

size

description

N/A

git-doc

I:2

5428

This provide the documentation for Git.

git(7)

git-core

V:4, I:8

7164

The main command for Git.

gitk(1)

gitk

V:0.5, I:2

612

The GUI Git repository browser with history.

git-gui(1)

git-gui

V:0.2, I:1.6

1128

The GUI for Git. (No history)

git-svnimport(1)

git-svn

V:0.3, I:2

416

This import the data out of Subversion into Git.

git-svn(1)

git-svn

V:0.3, I:2

416

This provides bidirectional operation between the Subversion and Git.

git-cvsimport(1)

git-cvs

V:0.15, I:1.0

436

This import the data out of CVS into Git.

git-cvsexportcommit(1)

git-cvs

V:0.15, I:1.0

436

This exports a commit to a CVS checkout from Git.

git-cvsserver(1)

git-cvs

V:0.15, I:1.0

436

A CVS server emulator for Git.

git-send-email(1)

git-email

V:0.10, I:1.0

292

This sends a collection of patches as email from the Git.

stg(1)

stgit

V:0.08, I:0.6

844

This is quilt on top of git. (Python)

git-buildpackage(1)

git-buildpackage

V:0.10, I:0.6

420

This automates the Debian packaging with the Git.

guilt(7)

guilt

V:0.01, I:0.09

320

This is quilt on top of git. (SH/AWK/SED/...)


11.5.4.4. Git for recording configuration history

You can manually record chronological history of configuration using Git tools. Here is a simple example for your practice to record /etc/apt/ contents.:

$ cd /etc/apt/
$ sudo git init
$ sudo chmod 700 .git
$ sudo git add .
$ sudo git commit -a
  • commit configuration with description.

  • make modification to the configuration files

$ cd /etc/apt/
$ sudo git commit -a
  • commit configuration with description.

  • ... continue your life ...

$ cd /etc/apt/
$ sudo gitk --all
  • you have full configuration history with you.

[Note] Note

The sudo(8) command is needed to work with permissions of configuration data. For user configuration data, you may skip the sudo(8) command.

[Note] Note

The "chmod 700 .git" command in the above example is needed to protect archive data from unauthorized read access.

[Tip] Tip

For more complete setup for recording configuration history, please look for the etckeeper package: Section 10.2.8, “Recording changes in configuration files”.

Chapter 12. Data conversion

Standard based tools are in very good shape but support for proprietary data formats are limited.

12.1. Text data conversion tools

Following packages for the text data conversion caught my eyes:

Table 12.1.  List of text data conversion tools.

package

popcon

size

keyword

function

libc6

V:88, I:99

11468

charset

The text encoding conversion between locales with iconv command. (fundamental)

recode

V:1.7, I:7

780

charset+eol

The text encoding conversion between locales. (versatile, more aliases and features)

konwert

V:0.4, I:4

192

charset

The text encoding conversion between locales. (fancy)

nkf

V:0.4, I:2

300

charset

The character set translator for Japanese.

tcs

V:0.04, I:0.2

544

charset

The character set translator.

unaccent

V:0.03, I:0.12

60

charset

Replace accented letters by their unaccented equivalent.

tofrodos

V:1.3, I:8

80

eol

The text format converter between DOS and Unix: fromdos and todos

macutils

V:0.13, I:0.8

356

eol

The text format converter between Macintosh and Unix: frommac and tomac


12.1.1. To convert a text file with iconv

The iconv command is provided as a part of the libc6 package and it is always available on all system to convert the encoding of characters:

$ iconv -f encoding1 -t encoding2 input.txt >output.txt

Encoding values are case insensitive and ignore "-" and "_" for matching. The supported encodings can be checked by the "iconv -l" command.

Table 12.2.  List of encoding values and their usage.

encoding value

usage

ASCII.

American Standard Code for Information Interchange. 7 bit code w/o accented characters.

UTF-8

Standard multilingual compatibility] for all modern OSs.

ISO-8859-1

Old standard for western European languages, ASCII + accented characters.

ISO-8859-2

Old standard for eastern European languages, ASCII + accented characters.

ISO-8859-15

Old standard for western European languages, ISO-8859-1 with euro sign.

CP850

Code page 850, Microsoft DOS characters with graphics for western European languages. ISO-8859-1 variant.

CP932

Code page 932, Microsoft Windows style Shift-JIS variant, for Japanese.

CP936

Code page 936, Microsoft Windows style GB2312, GBK, or GB18030 variant, for Simplified Chinese.

CP949

Code page 949, Microsoft Windows style EUC-KR or Unified Hangul Code variant, for Korean.

CP950

Code page 950, Microsoft Windows style Big5 variant, for Traditional Chinese.

CP1251

Code page 1251, Microsoft Windows style encoding for the Cyrillic alphabet.

CP1252

Code page 1252, Microsoft Windows style ISO-8859-15 variant for western European languages.

KOI8-R

Old Russian UNIX standard for the Cyrillic alphabet.

ISO-2022-JP

Standard encoding for Japanese e-mail which uses only 7 bit codes.

eucJP

Old Japanese UNIX standard 8 bit code and completely different from Shift-JIS.

Shift-JIS

JIS X 0208 Appendix 1 standard, for Japanese. See CP932 above.


[Note] Note

Some encodings are only supported for the data conversion and are not used as locale values (Section 9.3.1, “Basics of encoding”).

For character sets which fit in single byte such as ASCII and ISO-8859 character sets, the character encoding means almost the same thing as the character set.

For character sets with many characters such as JIS X 0213 for Japanese or Universal Character Set (UCS, Unicode, ISO-10646-1) for practically all languages, there are many encoding schemes to fit them into the sequence of the byte data:

For these, there is clear differentiation between the character set and the character encoding.

The code page is used as the synonym to the character encoding tables for some vendor specific ones.

[Note] Note

Please note most encoding systems share the same code with ASCII for the 7 bit characters. But there are some exceptions. If you are converting old Japanese C programs and URLs data from the casually-called shift-JIS encoding format to UTF-8 format, use "CP932" as the encoding name instead of "shift-JIS" to get the expected results: 0x5C -> "\" and 0x7E -> "~" . Otherwise, these are converted to wrong characters.

[Tip] Tip

The recode command may be used too and offers more than the combined functionality of the iconv, fromdos, todos, frommac, and tomac commands. For more, see pertinent description in the "info recode".

12.1.2. To convert file names with iconv

Here is an example script to convert encoding of the file name from ones created under older OS to modern UTF-8 ones for the simple case.

#!/bin/sh
ENCDN=iso-8859-1
for x in *;
 do
 mv "$x" $(echo "$x" | iconv -f $ENCDN -t utf-8)
done

The "$ENCDN" variable should be set by the encoding values from Table 12.2, “ List of encoding values and their usage. ” .

For more complicated case, please mount disk drive containing such file names with proper encoding as the mount(8) option (see Section 9.3.6, “Filename encoding”) and copy entire disk to another disk drive mounted as UTF-8 with "cp -a" command.

12.1.3. EOL conversion

The text file format, specifically the end-of-line (EOL) code, is dependent on the platform:

Table 12.3.  List of EOL conversion tools.

platform

EOL code

EOL control sequence

EOL ASCII value

Debian (unix)

LF

^J

10

MSDOS and Windows

CR-LF

^M^J

13, 10

Apple's Macintosh

CR

^M

13


The EOL format conversion programs, fromdos(1), todos(1), frommac(1), and tomac(1), are quite handy. The recode command is also useful.

[Tip] Tip

The use of "sed -e '/\r$/!s/$/\r/'" instead of "todos" is better when you want to unify the EOL style to the MSDOS style from the mixed MSDOS and Unix style. (e.g., after merging 2 MSDOS style files with diff3.) This is because "todos" adds CR to all lines.

[Note] Note

Some data on the Debian system, such as the wiki page data for the python-moinmoin, use MSDOS style CR-LF as the EOL code. So the above rule is just general rule.

[Note] Note

Most editors (eg. vim, emacs, gedit, ...) can handle files in MSDOS style EOL transparently.

12.1.4. TAB conversion

You can expand the tab code in the text to the multiple spaces in vim using the ":retab" command.

There are few popular specialized programs to convert the tab codes:

Table 12.4.  List of TAB conversion commands from bsdmainutils and coreutils packages.

function

bsdmainutils

coreutils

expand tab to spaces

"col -x"

expand

unexpand tab from spaces

"col -h"

unexpand


The indent(1) from the indent package completely reformats whitespaces in the C program.

12.1.5. Editors with auto-conversion

Intelligent modern editors such as the vim program are quite smart and copes well with any encoding systems and any file formats. You should use these editors under the UTF-8 locale in the UTF-8 capable console for the best compatibility.

An old western European Unix text file, "u-file.txt", stored in the latin1 encoding can be edited simply with vim as:

$ vim u-file.txt

This is possible since the auto detection mechanism of the file encoding in vim assumes the UTF-8 encoding first and, if it fails, assumes it to be latin1.

An old Polish Unix text file, "pu-file.txt", stored in the latin2 encoding can be edited with vim as:

$ vim '+e ++enc=latin2 pu-file.txt'

An old Japanese unix text file, "ju-file.txt", stored in the eucJP encoding can be edited with vim as:

$ vim '+e ++enc=eucJP ju-file.txt'

An old Japanese MS-Windows text file, "jw-file.txt", stored in the so called shift-JIS encoding (more precisely: CP932) can be edited with vim as:

$ vim '+e ++enc=CP932 ++ff=dos jw-file.txt'

When a file is opened with "++enc" and "++ff" options, the "w" in the Vim command line stores it in the original format and overwrite the original file. You can also specify the saving format and the file name in the Vim command line, e.g., "w ++enc=utf8 new.txt".

Please refer to the mbyte.txt "multi-byte text support" in vim on-line help.

The emacs family of programs can perform the equivalent functions.

12.1.6. Plain text extraction

Following will read a web page into a text file. This is very useful when copying configurations off the Web or applying basic Unix text tools such as grep on the web page.

$ lynx -dump http://www.remote-site.com/help-info.html >textfile

Similarly, you can extract plain text data from other formats using followings:

Table 12.5.  List of tools to extract plain text data.

package

popcon

size

keyword

function

html2text

V:15, I:40

288

html->text

An advanced HTML to text converter. (Better than "lynx -dump")

w3m

V:20, I:85

1968

html->text

An HTML to text converter with the "w3m -dump" command.

lynx

V:4, I:25

44

html->text

An HTML to text converter with the "lynx -dump" command.

elinks

V:2, I:6

1444

html->text

An HTML to text converter with the "elinks -dump" command.

links

V:2, I:9

1372

html->text

An HTML to text converter with the "links -dump" command.

links2

V:1.0, I:4

3280

html->text

An HTML to text converter with the "links2 -dump" command.

antiword

V:1.1, I:2

780

MSWord->text,ps

This converts !MSWord files to plain text or ps.

catdoc

V:0.9, I:2

2664

MSWord->text,TeX

This converts !MSWord files to plain text or TeX.

pstotext

V:0.8, I:1.5

160

ps/pdf->text

Extract text from PostScript and PDF files.

unhtml

V:0.04, I:0.2

76

html->text

Remove the markup tags from an HTML file.

odt2txt

V:0.5, I:0.9

104

odt->text

The converter from OpenDocument Text to text.

wpd2sxw

V:0.03, I:0.16

156

WordPerfect->sxw

WordPerfect to OpenOffice.org/StarOffice writer document converter.


12.1.7. Highlighting and formatting plain text data

Table 12.6.  List of tools to highlight plain text data.

package

popcon

size

keyword

function

vim-runtime

V:3, I:33

24504

highlight

Vim can convert source code to HTML with :source $VIMRUNTIME/syntax/html.vim (vim MACRO)

cxref

V:0.13, I:0.7

1104

c->html

The converter for the C program to latex and HTML. (C)

src2tex

V:0.06, I:0.3

1968

highlight

This convert many source codes to TeX. (C)

source-highlight

V:0.12, I:0.7

1980

highlight

This convert many source codes to HTML, XHTML, LaTeX, Texinfo, ANSI color escape sequences and DocBook files with highlight. (C++)

highlight

V:0.06, I:0.4

1664

highlight

This convert many source codes to HTML, XHTML, RTF, LaTeX, TeX or XSL-FO files with highlight. (C++)

grc

V:0.03, I:0.12

164

text->color

The generic colouriser for everything. (Python)

txt2html

V:0.09, I:0.5

296

text->html

Text to HTML converter. (Perl)

markdown

V:0.10, I:0.5

96

text->html

The converter from text to (X)HTML. (Perl)

asciidoc

V:0.14, I:0.8

2756

text->any

A text document formatter to XML. (Python)

txt2tags

V:0.07, I:0.3

1556

text->any

The document conversion from text to HTML, SGML, LaTeX, man page, MoinMoin, Magic Point and PageMaker. (Python)

udo

V:0.02, I:0.08

556

text->any

universal document - text processing utility. (C)

stx2any

V:0.01, I:0.07

484

text->any

The document converter from structured plain text to other formats. (m4)

rest2web

V:0.01, I:0.09

644

text->html

The document converter from ReStructured Text to html. (Python)

aft

V:0.01, I:0.09

336

text->any

The "free form" document preparation system. (Perl)

yodl

V:0.01, I:0.07

1756

text->any

A pre-document language and tools to process it. (C)

sdf

V:0.01, I:0.10

1940

text->any

The simple document parser. (Perl)

sisu

V:0.01, I:0.07

5504

text->any

The document structuring, publishing and search framework. (Ruby)


12.2. XML data

The Extensible Markup Language (XML) is a markup language for documents containing structured information.

XML.COM has good introductory information:

12.2.1. Basic hints for XML

XML text looks somewhat like HTML. It enables us to manage multiple formats of output for a document. One easy XML system is docbook-xsl, which is used here.

Each XML file starts with standard XML declaration:

<?xml version="1.0" encoding="UTF-8"?>

The basic syntax for one XML element is marked up as:

<name attribute="value">content</name>

XML element with empty content is marked up in the short form as:

<name attribute="value"/>

The "attribute="value"" in the above examples are optional.

The comment section in XML is marked up as:

<!-- comment -->

Other than adding markups, XML requires minor conversion to the content using predefined entities for the following character:

Table 12.7.  List of predefined entities for XML.

predefined entity

character to be converted from

&quot;

" : quote

&apos;

' : apostrophe

&lt;

< : less-than

&gt;

> : greater-than

&amp;

& : ampersand


[Caution] Caution

"<" or "&" can not be used in attributes or elements.

[Note] Note

When SGML style user defined entities, e.g. "&some-tag:", are used, the first definition wins over others. The entity definition is expressed in "<!ENTITY some-tag "entity value">".

[Note] Note

As long as the XML markup are done consistently with certain set of the tag name (either some data as content or attribute value), conversion to another XML is trivial task using XSLT.

12.2.2. XML processing

There are many tools available to process XML files such as the Extensible Stylesheet Language (XSL).

Basically, once you create well formed XML file, you can convert it to any format using Extensible Stylesheet Language for Transformation (XSLT).

Although the Extensible Stylesheet Language for Formatting Object (XSL-FO) is supposed to be solution for formatting, FOP program is not in the Debian main (yet?). So the LaTeX code is usually generated from XML using XSLT and the LaTeX system is used to create printable file such as DVI, PostScript, and PDF.

Table 12.8.  List of XML tools.

package

popcon

size

keyword

function

docbook-xml

V:35, I:58

2488

xml

This package contains the XML document type definition (DTD) for DocBook.

xsltproc

V:5, I:56

180

xslt

XSLT command line processor. (XML-> XML, HTML, plain text, etc.)

docbook-xsl

V:0.6, I:6

11064

xml/xslt

This contains XSL stylesheets for processing DocBook XML to various output formats with XSLT.

xmlto

V:0.3, I:2

232

xml/xslt

XML-to-any converter with XSLT.

dblatex

V:0.05, I:0.3

5284

xml/xslt

This converts Docbook files to DVI, PostScript, PDF documents with XSLT.


Since XML is subset of Standard Generalized Markup Language (SGML), it can be processed by the extensive tools available for SGML, such as Document Style Semantics and Specification Language (DSSSL).

Table 12.9.  List of DSSL tools.

package

popcon

size

keyword

function

openjade

V:0.6, I:4

1212

dsssl

Implementation of the DSSSL language based on James Clark's Jade software.

jade

V:0.5, I:3

1056

dsssl

James lark's DSSSL language.

docbook-dsssl

V:0.8, I:5

3100

xml/dsssl

This contains DSSSL stylesheets for processing DocBook XML to various output formats with DSSSL.

docbook-utils

V:0.3, I:2

440

xml/dsssl

The utilities for Docbook files including conversion to other formats (HTML, RTF, PS, man, PDF) with docbook2* commands with DSSSL.

sgml2x

V:0.00, I:0.11

216

SGML/dsssl

The converter from SGML and XML using DSSSL stylesheets.


12.2.3. The XML data extraction

You can extract HTML or XML data from other formats using followings:

Table 12.10.  List of XML data extraction tools.

package

popcon

size

keyword

function

wv

V:1.6, I:3

2136

MSWord->any

The document converter from Microsoft Word to HTML, LaTeX, etc..

texi2html

V:0.5, I:3

1752

texi->html

The converter from Texinfo to HTML.

man2html

V:0.3, I:1.9

372

manpage->html

The converter from manpage to HTML. (CGI support)

tex4ht

V:0.2, I:1.9

936

tex<->html

The converter between (La)TeX and HTML.

xlhtml

V:0.6, I:1.7

184

MSExcel->html

The converter from !MSExcel .xls to HTML.

ppthtml

V:0.6, I:1.6

120

MSPowerPoint->html

The converter from !MSPowerPoint to HTML.

unrtf

V:0.4, I:1.1

276

rtf->html

The document converter from RTF to HTML, etc..

info2www

V:0.5, I:1.5

156

info->html

The converter from GNU info to HTML. (CGI support)

ooo2dbk

V:0.04, I:0.2

941

sxw->xml

The converter from OpenOffice.org SXW documents to DocBook XML.

wp2x

V:0.02, I:0.13

240

WordPerfect->any

WordPerfect 5.0 and 5.1 files to TeX, LaTeX, troff, GML and HTML.

doclifter

V:0.00, I:0.05

420

troff->xml

The converter from troff to DocBook XML.


For non-XML HTML files, you can convert them to XHTML which is an instance of well formed XML and can be processed by XML tools.

Table 12.11.  List of XML pretty print tools.

package

popcon

size

keyword

function

libxml2-utils

V:5, I:56

116

xml<->html<->xhtml

The command line XML tool with "xmllint" command. (syntax check, reformat, lint, ...)

tidy

V:1.9, I:17

108

xml<->html<->xhtml

HTML syntax checker and reformatter.


Once proper XML is generated, you can use XSLT technology to extract data based on the mark-up context etc.

12.3. Printable data

Printable data is expressed in the PostScript format on the Debian system. The Common Unix Printing System (CUPS) uses the Ghostscript as its rasterizer backend for non-PostScript printers.

12.3.1. The Ghostscript

The core of printable data manipulation is the Ghostscript PostScript interpreter which generates raster image.

The latest upstream Ghostscript from Artifex was re-licensed from AFPL to GPL and merged all the latest ESP version changes such as CUPS related ones at 8.60 release as unified release.

Table 12.12.  List of Ghostscript PostScript interpreters.

package

popcon

size

description

ghostscript

V:19, I:28

3560

The GPL Ghostscript PostScript/PDF interpreter

ghostscript-x

V:9, I:23

188

The GPL Ghostscript PostScript/PDF interpreter - X Display support

gs-cjk-resource

I:0.5

4652

Resource files for gs-cjk, ghostscript CJK-TrueType extension

cmap-adobe-cns1

I:0.4

1588

CMaps for Adobe-CNS1 (for traditional Chinese support)

cmap-adobe-gb1

I:0.4

1580

CMaps for Adobe-GB1 (for simplified Chinese support)

cmap-adobe-japan1

I:0.9

2476

CMaps for Adobe-Japan1 (for Japanese standard support)

cmap-adobe-japan2

I:0.5

440

CMaps for Adobe-Japan2 (for Japanese extra support)

cmap-adobe-korea1

I:0.3

912

CMaps for Adobe-Korea1 (for Korean support)

libpoppler3

I:26

2196

PDF rendering library based on xpdf PDF viewer

libpoppler-glib3

V:8, I:24

396

PDF rendering library based (GLib-based shared library)

poppler-data

I:0.2

12240

Encoding data for the poppler PDF rendering library (for CJK support)


[Tip] Tip

"gs -h" to display Ghostscript configuration.

12.3.2. Merge two PS or PDF files

You can merge two PS or PDF files using the gs(1) command of the Ghostscript.

$ gs -q -dNOPAUSE -dBATCH -sDEVICE=pswrite -sOutputFile=bla.ps -f foo1.ps foo2.ps
$ gs -q -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=bla.pdf -f foo1.pdf foo2.pdf
[Note] Note

The Portable Document Format (PDF), which is widely used cross-platform printable data format, is essentially the compressed PS format with few additional features and extensions.

[Tip] Tip

From command line, psmerge(1) and other commands from the psutils package are useful for manipulating PostScript documents. Commands in the pdfjam package work similarly for manipulating PDF documents. pdftk(1) from the pdftk package is useful for manipulating PDF documents, too.

12.3.3. Printable data utilities

The following packages for the printable data utilities caught my eyes:

Table 12.13.  List of printable data utilities.

package

popcon

size

keyword

function

poppler-utils

V:18, I:53

452

pdf->ps,text,...

PDF utilities. (pdftops, pdfinfo, pdfimages, pdftotext, and pdffonts)

psutils

V:3, I:27

408

ps->ps

PostScript document conversion tools

poster

V:2, I:18

80

ps->ps

Create large posters out of PostScript pages.

xpdf-utils

V:3, I:10

4680

pdf->ps,text,...

PDF utilities. (pdftops, pdfinfo, pdfimages, pdftotext, and pdffonts)

enscript

V:2, I:23

2432

text->ps, html, rtf

Converts ASCII text to Postscript, HTML, RTF or Pretty-Print.

a2ps

V:1.5, I:8

4288

text->ps

'Anything to PostScript' converter and pretty-printer.

pdftk

V:0.8, I:4

3292

pdf->pdf

PDF document conversion tool: (pdftk)

mpage

V:0.3, I:2

224

text,ps->ps

Print multiple pages per sheet.

html2ps

V:0.3, I:2

260

html->ps

The converter from HTML to PostScript.

pdfjam

V:0.3, I:1.8

112

pdf->pdf

PDF document conversion tools: pdf90, pdfjoin, and pdfnup

gnuhtml2latex

V:0.16, I:1.0

24

html->latex

The converter from html to latex.

latex2rtf

V:0.17, I:1.1

544

latex->rtf

This converts documents from LaTeX to RTF which can be read by MS Word.

ps2eps

V:0.9, I:7

116

ps->eps

The converter from PostScript to EPS (Encapsulated PostScript).

e2ps

V:0.03, I:0.2

188

text->ps

Text to PostScript converter with Japanese encoding support.

impose+

V:0.04, I:0.19

180

ps->ps

Postscript utilities.

trueprint

V:0.03, I:0.19

188

text->ps

This pretty print many source codes (C, C++, Java, Pascal, Perl, Pike, Sh, and Verilog) to PostScript. (C)

pdf2svg

V:0.08, I:0.3

60

ps->svg

Converter from PDF to Scalable vector graphics format.

pdftoipe

V:0.03, I:0.19

648

ps->ipe

Converter from PDF to IPE's XML format.


12.3.4. Printing with CUPS

Both lp and lpr commands offered by Common Unix Printing System (CUPS) provides options for customized printing the printable data.

For printing 3 copies of a file collated:

$ lp -n 3 -o Collate=True filename

, or

$ lpr -#3 -o Collate=True filename

You can further customize printer operation by using printer option such as "-o number-up=2", "-o page-set=even", "-o page-set=odd", "-o scaling=200", "-o natural-scaling=200", etc., documented at Command-Line Printing and Options.

12.4. Type setting

The Unix troff originally developed by AT&T can be used for simple type setting. It is usually used to create manpages.

TeX created by Donald Knuth is very powerful type setting tool and is the de facto standard . LaTeX originally written by Leslie Lamport enables a high-level access to the power of TeX.

Table 12.14.  List of type setting tools.

package

popcon

size

keyword

function

texlive-base

V:6, I:16

17372

(La)TeX

TeX system for typesetting, previewing and printing.

groff

V:1.3, I:6

5540

troff

GNU troff text-formatting system.


12.4.1. roff typesetting

Traditionally, roff is the main Unix text processing system.

See roff(7), groff(7), groff(1), grotty(1), troff(1), groff_mdoc(7), groff_man(7), groff_ms(7), groff_me(7), groff_mm(7), and info groff.

A good tutorial on -me macros exists. If you have groff (1.18 or newer), find /usr/share/doc/groff/meintro.me.gz and do the following:

$ zcat /usr/share/doc/groff/meintro.me.gz | \
     groff -Tascii -me - | less -R

The following will make a completely plain text file:

$ zcat /usr/share/doc/groff/meintro.me.gz | \
    GROFF_NO_SGR=1 groff -Tascii -me - | col -b -x > meintro.txt

For printing, use PostScript output.

$ groff -Tps meintro.txt | lpr
$ groff -Tps meintro.txt | mpage -2 | lpr

12.4.2. TeX/LaTeX

Preparation:

# aptitude install texlive

References for LaTeX:

  • The teTeX HOWTO: The Linux-teTeX Local Guide (http://www.tldp.org/HOWTO/TeTeX-HOWTO.html)

  • tex(1)

  • latex(1)

  • "The TeXbook", by Donald E. Knuth, (Addison-Wesley)

  • LaTeX - A Document Preparation System, by Leslie Lamport, (Addison-Wesley)

  • The LaTeX Companion, by Goossens, Mittelbach, Samarin, (Addison-Wesley)

This is the most powerful typesetting environment. Many SGML processors use this as their back end text processor. Lyx provided by lyx, lyx-xforms, or lyx-qt and GNU TeXmacs provided by texmacs package offers nice WYSIWYG editing environment for LaTeX while many use Emacs and Vim as the choice for the source editor.

There are many online resources available:

When documents become bigger, sometimes TeX may cause errors. You must increase pool size in /etc/texmf/texmf.cnf (or more appropriately edit /etc/texmf/texmf.d/95NonPath and run update-texmf) to fix this.

[Note] Note

The TeX source of "The TeXbook" is available at http://tug.ctan.org/tex-archive/systems/knuth/dist/tex/texbook.tex .

This file contains most of the required macros. I heard that you can process this document with tex after commenting lines 7 to 10 and adding "\input manmac \proofmodefalse". It's strongly recommended to buy this book (and all other books from Donald E. Knuth) instead of using the online version but the source is a great example of TeX input!

12.4.3. Pretty print a manual page

The following will print a manual page into a PostScript file/printer.

$ man -Tps some_manpage | lpr
$ man -Tps some_manpage | mpage -2 | lpr

12.4.4. Creating a manual page

Although writing manpage in plain troff is possible, there are few helper packages to create the manpage.

Table 12.15.  List of packages to help creating the manpage.

package

popcon

size

keyword

function

docbook-to-man

V:0.5, I:3

248

SGML->manpage

The converter from DocBook SGML into roff man macros.

help2man

V:0.15, I:1.1

232

text->manpage

Automatic manpage generator from --help.

info2man

V:0.03, I:0.18

204

info->manpage

The converter from GNU info to POD or man pages.

txt2man

V:0.04, I:0.19

88

text->manpage

Converts flat ASCII text to man page format.


12.5. The mail data conversion

The following packages for the mail data conversion caught my eyes:

Table 12.16.  List of packages to help mail data conversion.

package

popcon

size

keyword

function

sharutils

V:6, I:86

976

mail

shar, unshar, uuencode, uudecode

mpack

V:4, I:76

84

mail

The encoder and decoder MIME messages: mpack and munpack.

tnef

V:0.5, I:1.6

160

mail

unpacking MIME attachments of type "application/ms-tnef" which is a Microsoft only format.

uudeview

V:0.2, I:1.6

128

mail

The encoder and decoder for the following formats: uuencode, xxencode, BASE64, quoted printable, and BinHex

mimedecode

V:0.15, I:0.9

76

mail

This decodes transfer encoded text type mime messages.

readpst

V:0.06, I:0.3

228

windows/mail

This converts Outlook PST files to mbox format.


[Tip] Tip

The Internet Message Access Protocol version 4 (IMAP4) server (see: Section 7.2.5.5, “POP3/IMAP4 server”) may be used to move mails out from the proprietary mail system if the mail client software can be configured to use IMAP4 server too.

12.5.1. Mail data basics

Mail (SMTP) data should be limited to 7 bit. So binary data and 8 bit text data are encoded into 7 bit format with the Multipurpose Internet Mail Extensions (MIME) and the selection of the charset (see: Section 9.3.1, “Basics of encoding”).

The standard mail storage format is mbox formatted according to RFC2822 (updated RFC822). See man 5 mbox (provided by the mutt package).

For European languages, "Content-Transfer-Encoding: quoted-printable" with the ISO-8859-1 charset is usually used since there are no much 8 bit characters. If the text is in UTF-8, "Content-Transfer-Encoding: quoted-printable" is also used since it is mostly 7 bit data.

For Japanese, traditionally "Content-Type: text/plain; charset=ISO-2022-JP" should be used to keep text in 7 bits. But mails from older Microsoft systems may use in Shift-JIS without proper declaration. For Japanese, if the text is in UTF-8, it contains many 8 bit data and is encoded into 7 bit data by Base64. The situation of other Asian languages is similar.

[Note] Note

If your non-Unix mail data is accessible by a non-Debian client software which can talk to the IMAP4 server, you may be able to move them out by running your own IMAP4 server (see: Section 7.2.5.5, “POP3/IMAP4 server”).

[Note] Note

If you use other mail storage formats, moving them to mbox format is the good first step. The versatile client program such as mutt may be handy for this.

You can split mailbox contents to each message using procmail(1) and formail(1).

Each mail message can be unpacked using the munpack(1) command from the mpack package (or other specialized tools) to obtain the MIME encoded contents.

12.6. Graphic data tools

The following packages for the graphic data conversion, editing, and organization tools caught my eyes:

Table 12.17.  List of graphic data tools.

package

popcon

size

keyword

function

gimp

V:13, I:55

11928

image(bitmap)

The GNU Image Manipulation Program.

imagemagick

V:13, I:31

4464

image(bitmap)

Image manipulation programs.

graphicsmagick

V:1.4, I:2

3432

image(bitmap)

Image manipulation programs. (folk of imagemagick)

xsane

V:7, I:45

744

image(bitmap)

GTK+-based X11 frontend for SANE (Scanner Access Now Easy).

netpbm

V:4, I:21

4408

image(bitmap)

Graphics conversion tools.

icoutils

V:0.07, I:0.5

196

png<->ico(bitmap)

Converts MS Windows icons and cursors to and from PNG formats (favicon.ico)

xpm2wico

V:0.04, I:0.16

80

xpm->ico(bitmap)

Converts XPM to MS Windows icon formats

scribus

V:0.6, I:3

26708

ps/pdf/SVG/...

The Scribus DTP editor.

openoffice.org-draw

V:20, I:51

8476

image(vector)

OpenOffice.org office suite - drawing

inkscape

V:7, I:18

61376

image(vector)

The SVG (Scalable Vector Graphics) editor.

dia-gnome

V:2, I:5

596

image(vector)

Diagram editor (Gnome)

dia

V:2, I:5

596

image(vector)

Diagram editor (Gtk)

xfig

V:2, I:5

1624

image(vector)

Facility for Interactive Generation of figures under X11

pstoedit

V:1.1, I:8

768

ps/pdf->image(vector)

PostScript and PDF files to editable vector graphics converter. (SVG)

libwmf-bin

V:1.0, I:8

84

Windows/image(vector)

Windows metafile (vector graphic data) conversion tools.

fig2sxd

V:0.08, I:0.3

200

fig->sxd(vector)

Convert XFig files to OpenOffice.org Draw format

unpaper

V:0.2, I:1.3

736

image->image

Post-processing tool for scanned pages for OCR.

tesseract-ocr

V:0.2, I:1.3

2072

image->text

Free OCR software based on the HP's commercial OCR engine.

tesseract-ocr-eng

I:0.5

1760

image->text

OCR engine data: tesseract-ocr language files for English text.

clara

V:0.09, I:0.4

1000

image->text

Free OCR software.

gocr

V:1.1, I:6

484

image->text

Free OCR software.

ocrad

V:0.9, I:7

364

image->text

Free OCR software.

gtkam

V:0.4, I:2

1348

image(Exif)

Manipulates digital camera photo files (GNOME) - GUI

gphoto2

V:0.6, I:3

944

image(Exif)

Manipulates digital camera photo files (GNOME) - command line

kamera

V:1.0, I:20

392

image(Exif)

Manipulates digital camera photo files (KDE)

jhead

V:0.8, I:4

128

image(Exif)

Manipulates the non-image part of Exif compliant JPEG (digital camera photo) files

exif

V:0.3, I:1.8

164

image(Exif)

Command-line utility to show EXIF information in JPEG files

exiftags

V:0.2, I:1.1

248

image(Exif)

Utility to read Exif tags from a digital camera JPEG file

exiftran

V:0.2, I:1.4

92

image(Exif)

Transforms digital camera jpeg images

exifprobe

V:0.07, I:0.3

484

image(Exif)

Reads metadata from digital pictures

dcraw

V:1.4, I:6

408

image(Raw)->ppm

Decodes raw digital camera images

findimagedupes

V:0.10, I:0.5

136

image->fingerprint

Finds visually similar or duplicate images

ale

V:0.05, I:0.2

772

image->image

Merges images to increase fidelity or create mosaics

imageindex

V:0.04, I:0.3

192

image(Exif)->html

Generates static HTML galleries from images

f-spot

V:0.6, I:1.8

9320

image(Exif)

Personal photo management application (GNOME)

bins

V:0.04, I:0.2

2008

image(Exif)->html

Generates static HTML photo albums using XML and EXIF tags

galrey

V:0.03, I:0.17

116

image(Exif)->html

Generates browsable HTML photo albums with thumbnails

outguess

V:0.04, I:0.17

248

jpeg,png

Universal Steganographic tool

qcad

V:1.2, I:2

3824

DXF

CAD data editor (KDE)

blender

V:0.7, I:3

22508

blend, TIFF, VRML, ...

3D content editor for animation etc.

open-font-design-toolkit

I:0.02

12

ttf, ps, ...

Metapackage for open font design

fontforge

V:0.3, I:2

14236

ttf, ps, ...

Font editor for PS, TrueType and OpenType fonts

xgridfit

V:0.01, I:0.05

536

ttf

a program for gridfitting, or "hinting," TrueType fonts

gbdfed

V:0.03, I:0.18

520

bdf

Editor for BDF fonts


[Tip] Tip

Search more image tools using regex "~Gworks-with::image" in aptitude (see Section 3.2.5, “Search method options with aptitude”).

Although GUI programs such as gimp are very powerful, command line tools such as imagemagik are quite useful for automating image manipulation with the script.

The de facto image file format of the digital camera is the Exchangeable Image File Format (EXIF) which is the JPEG image file format with additional metadata tags. It can hold information such as date, time, and camera settings.

The Lempel-Ziv-Welch (LZW) lossless data compression patent has been expired. Graphics Interchange Format (GIF) utilities which use the LZW compression method are now freely available on the Debian system.

[Tip] Tip

Any digital camera or scanner with removable recording media will work with Linux through USB Mass Storage readers.

12.7. Miscellaneous data conversion

There are many other programs for converting data. Following packages caught my eyes using regex "~Guse::converting" in aptitude (see Section 3.2.5, “Search method options with aptitude”):

Table 12.18.  List of miscellaneous data conversion tools.

package

popcon

size

keyword

function

alien

V:1.9, I:13

276

rpm/tgz->deb

The converter for the foreign package into the Debian package.

freepwing

V:0.00, I:0.04

568

EB->EPWING

The converter from "Electric Book" (popular in Japan) to a single JIS X 4081 format (a subset of the EPWING V1).


You can also extract data from RPM format with:

$ rpm2cpio file.src.rpm | cpio --extract

Chapter 13. Programming

I provide some pointers for people to learn programming on the Debian system enough to trace the packaged source code. Here are notable packages and corresponding documentation packages for programing.

Table 13.1.  List of packages to help programing.

package

popcon

size

documentation

autoconf

V:4, I:26

1676

"info autoconf" provided by autoconf-doc

automake

V:2, I:16

1700

"info automake" provided by automake1.10-doc

bash

V:87, I:99

1300

"info bash" provided by bash-doc

bison

V:2, I:18

1392

"info bison" provided by bison-doc

cpp

V:43, I:86

72

"info cpp" provided by cpp-doc

ddd

V:0.4, I:3

4104

"info ddd" provided by ddd-doc

exuberant-ctags

V:1.3, I:6

288

exuberant-ctags(1)

flex

V:2, I:18

1004

"info flex" provided by flex-doc

gawk

V:23, I:26

1794

"info gawk" provided by gawk-doc

gcc

V:15, I:66

64

"info gcc" provided by gcc-doc

gdb

V:7, I:50

7128

"info gdb" provided by gdb-doc

gettext

V:9, I:50

7856

"info gettext" provided by gettext-doc

gfortran

V:0.8, I:6

40

"info gfortran" provided by gfortran-doc

glade

V:0.4, I:2

1172

Help provided via menu

glade-gnome

V:0.2, I:1.6

488

Help provided via menu

libc6

V:88, I:99

11468

"info libc" provided by glibc-doc and glibc-doc-reference

make

V:20, I:83

1220

"info make" provided by make-doc

mawk

V:68, I:99

248

"man mawk"

perl

V:84, I:99

18892

"man 1 perl" and html pages provided by perl-doc and perl-doc-html

python

V:59, I:96

612

"man python" and html pages provided by python-doc

tcl8.4

V:8, I:43

3308

"man tcl" and detail manual pages provided by tcl8.4-doc

tk8.4

V:6, I:35

2800

"man tk" and detail manual pages provided by tk8.4-doc

ruby

V:8, I:23

100

"man ruby" and interactive reference provided by ri

vim

V:12, I:29

1736

Help(F1) menu provided by vim-doc

susv2

I:0.04

48

Fetch "The Single Unix Specifications v2"

susv3

I:0.09

48

Fetch "The Single Unix Specifications v3"


Online references are available by typing man name after installing manpages and manpages-dev packages. Online references for the GNU tools are available by typing info program_name after installing the pertinent documentation packages. You may need to include the contrib and non-free archives in addition to the main archive since some GFDL documentations are not considered to be DSFG compliant.

[Tip] Tip

Code examples of creating "Song 99 Bottles of Beer" should give you good idea of practically all the programming languages.

[Caution] Caution

You should install software programs directly compiled from source into /usr/local or /opt to avoid collision with system programs.

[Warning] Warning

Do not use "test" as the name of an executable test file. "test" is a shell built-in.

13.1. The shell script

The shell script is a text file with the execution bit set and contains the commands in the following format.

#!/bin/sh
... command lines ...

The first line specifies the shell interpreter which read and execute this file contents.

Reading shell scripts is the best way to understand how a Unix-like system works. Here, I give some pointers and reminders for shell programming. See "Shell Mistakes" (http://www.greenend.org.uk/rjk/2001/04/shell.html) to learn from mistakes.

Unlike shell interactive mode (see: Section 2.5, “The simple shell command” and Section 2.6, “Unix-like text processing”), parameters, conditionals, and loops are used frequently.

13.1.1. POSIX shell compatibility

Many system scripts may be interpreted by any one of POSIX shells (see Table 2.14, “ List of shell programs. ”). For this shell bash may be used, but for the sake of speed, dash is recommended. You can switch actual shell program to be used by changing symlink for /bin/sh.

Avoid writing a shell script with "bashisms" or "zshisms" to make it portable among all POSIX shells:

Table 13.2.  List of bashizms.

Good: POSIX

Avoid: bashism

if [ "$foo" = "$bar" ] ; then ...

if [ "$foo" == "$bar" ] ; then ...

diff -u file.c.orig file.c

diff -u file.c{.orig,}

mkdir /foobar /foobaz

mkdir /foo{bar,baz}

octal format: "\377"

hexadecimal format: "\xff"


The "echo" command must be used with care since its implementation differs among shell built-in commands and external command.

  • Avoid using any command options except "-n". (Notably avoid "-e" and "-E")

  • Avoid using escape sequences in the string since their handling varies.

[Note] Note

Although "-n option is not really POSIX syntax, it is generally accepted.

[Tip] Tip

Use the "printf" command instead of the "echo" command if you need to embed escape sequences in the output string.

13.1.2. Shell parameters

Special shell parameters are frequently used in the shell script:

Table 13.3.  List of shell parameters.

shell parameter

value

$0

name of the shell or shell script

$1

first(1) shell argument

$9

ninth(9) shell argument

$#

number of positional parameters

"$*"

"$1 $2 $3 $4 ... "

"$@"

"$1" "$2" "$3" "$4" ...

$?

exit status of the most recent command

$$

PID of this shell script

$!

PID of most recently started background job


Basic parameter expansions to remember:

Table 13.4.  List of shell parameter expansions.

parameter expression form

If var is set

If var is not set

${var:-string}

$var

string

${var:+string}

string

null

${var:=string}

$var

string (and run var=string)

${var:?string}

$var

(echo string to stderr and exit with error)


Here, the colon ':' in all of these operators is actually optional.

  • With ':' = operator test for "exist" and "not null".

  • Without ':' = operator test for "exist" only.

Table 13.5.  List of key shell parameter substitutions.

parameter substitution form

Result

${var%suffix}

Remove smallest suffix pattern

${var%%suffix}

Remove largest suffix pattern

${var#prefix}

Remove smallest prefix pattern

${var##prefix}

Remove largest prefix pattern


13.1.3. Shell conditionals

Each command returns an exit status which can be used for conditional expressions:

  • Success: 0 (True)

  • Error: 1--255 (False)

Note that the use here of a 0 value to mean "true" differs from the usual convention in some other areas of computing. Also, `[' is the equivalent of the test command, which evaluates its arguments up to `]' as a conditional expression.

Basic conditional idioms to remember are:

  • <command> && <if_success_run_this_command_too> || true

  • <command> || <if_not_success_run_this_command_too> || true

  • if [ <conditional_expression> ]; then
    <if_success_run_this_command>
    else
    <if_not_success_run_this_command>
    fi
    

Here trailing "|| true" was needed to ensure this shell script will not exit at this line accidentally when shell is invoked with -e flag.

Table 13.6.  List of file comparison operators in the conditional expression.

equation

value

-e <file>

True if <file> exists.

-d <file>

True if <file> exists and is a directory.

-f <file>

True if <file> exists and is a regular file.

-w <file>

True if <file> exists and is writable.

-x <file>

True if <file> exists and is executable.

<file1> -nt <file2>

True if <file1> is newer than <file2>. (modification).

<file1> -ot <file2>

True if <file1> is older than <file2>. (modification).

<file1> -ef <file2>

True if they are the same device and inode number.


Table 13.7.  List of string comparison operators in the conditional expression.

equation

value

-z <str>

True if the length of <str> is zero.

-n <str>

True if the length of <str> is non-zero.

<str1> = <str2>

True if <str1> and <str2> are equal.

<str1> != <str2>

True if <str1> and <str2> are not equal.

<str1> < <str2>

True if <str1> sorts before <str2>. (locale dependent)

<str1> > <str2>

True if <str1> sorts after <str2>. (locale dependent)


Arithmetic integer comparison operators in the conditional expression are -eq, -ne, -lt, -le, -gt, and -ge.

13.1.4. Shell loops

There are several loop idioms to use in POSIX shell:

  • "for name in word ; do list ; done": loops over list of words.

  • "while list; do list; done": repeats while true.

  • "until list; do list; done": repeats while not true.

  • "break": enables to exit from the loop.

  • "continue" enables to resume the next iteration of the loop.

[Note] Note

The C-language like numeric iteration can be realized by using the seq(1) command as the "word" generator.

13.1.5. The shell command-line processing sequence

The shell processes a script as following sequence:

  • split into tokens by the metacharacters: SPACE TAB NEWLINE ; ( ) < > | &

  • check keyword if not within "..." or '...' (loop)

  • expand alias if not within "..." or '...' (loop)

  • expand tilde, ~<user> -> <user>'s home directory, if not within "..." or '...'

  • expand parameter, $PARAMETER, if not within '...'

  • expand command substitution, $(command), if not within '...'

  • split into words with $IFS if not within "..." or '...'

  • expand * ? [ ] in pathname if not within "..." or '...'

  • look up command

  • function

  • built-in

  • file in $PATH

  • loop

Single quotes within double quotes have no effect.

Executing set -x in the shell or invoking the shell with -x option make the shell to print all of commands executed. This is quite handy for debugging.

13.1.6. Utility programs for shell script

In order to make your shell program as portable as possible across Debian system, it is good idea to limit utility programs used within Essential programs listed by "aptitude search ~E" as much as possible.

The coreutils, bsdutils, and debianutils packages contain many useful small utilities.

13.1.7. Shell script dialog

The user interface of a simple shell program can be improved from dull interaction by the "echo" and "read" commands to more interactive one by using one of the so-called dialog program etc.

Table 13.8.  List of user interface programs.

package

popcon

size

function

xbase-clients

V:25, I:64

84

The xmessage(1) displays a message or query in a window (X)(etch)

x11-utils

V:13, I:28

640

The xmessage(1) , , (lenny)

whiptail

V:48, I:99

96

displays user-friendly dialog boxes from shell scripts. (newt)

dialog

V:5, I:22

1508

displays user-friendly dialog boxes from shell scripts. (ncurses)

zenity

V:5, I:50

4336

display graphical dialog boxes from shell scripts. (gtk2.0)

xdialog

V:0.2, I:1.1

1176

X11 replacement for the text utility dialog. (gtk1.2)

gtkdialog

V:0.05, I:0.3

488

GUI-creation command-line utility based on GTK+ library. (gtk2.0+glade2)

ssft

V:0.01, I:0.12

152

Shell Scripts Frontend Tool. (wrapper for zenity, kdialog, and dialog with gettext)

gettext

V:9, I:50

7856

The gettext.sh for translate message


13.1.8. Shell script example with zenity

Here is a simple script which creates ISO image with RS02 data supplemented by dvdisaster(1):

#!/bin/sh -e
# gmkrs02 : Copyright (C) 2007 Osamu Aoki <osamu@debian.org>, Public Domain
#set -x
error_exit()
{
  echo "$1" >&2
  exit 1
}
# Initialize variables
DATA_ISO="$HOME/Desktop/iso-$$.img"
LABEL=$(date +%Y%m%d-%H%M%S-%Z)
if [ $# != 0 ] && [ -d "$1" ]; then
  DATA_SRC="$1"
else
  # Select directory for creating ISO image from folder on desktop
  DATA_SRC=$(zenity --file-selection --directory  \
    --title="Select the directory tree root to create ISO image") \
    || error_exit "Exit on directory selection"
fi
# Check size of archive
xterm -T "Check size $DATA_SRC" -e du -s $DATA_SRC/*
SIZE=$(($(du -s $DATA_SRC | awk '{print $1}')/1024))
if [ $SIZE -le 520 ] ; then
  zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
    --text="The data size is good for CD backup:\\n $SIZE MB"
elif [ $SIZE -le 3500 ]; then
  zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
    --text="The data size is good for DVD backup :\\n $SIZE MB"
else
  zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
    --text="The data size is too big to backup : $SIZE MB"
  error_exit "The data size is too big to backup :\\n $SIZE MB"
fi
# only xterm is sure to have working -e option
# Create raw ISO image
rm -f "$DATA_ISO" || true
xterm -T "genisoimage $DATA_ISO" \
  -e genisoimage -r -J -V "$LABEL" -o "$DATA_ISO" "$DATA_SRC"
# Create RS02 supplemental redundancy
xterm -T "dvdisaster $DATA_ISO" -e  dvdisaster -i "$DATA_ISO" -mRS02 -c
zenity --info --title="Dvdisaster RS02" --width 640  --height 400 \
  --text="ISO/RS02 data ($SIZE MB) \\n created at: $DATA_ISO"
# EOF

You may wish to create launcher on the desktop with command set something like "/usr/local/bin/gmkrs02 %d".

13.2. Make

Make is a utility to maintain groups of programs. Upon execution of make(1), make read the rule file, Makefile, and updates a target if it depends on prerequisite files that have been modified since the target was last modified, or if the target does not exist. The execution of these updates may occur concurrently.

The rule file syntax is :

target: [ prerequisites ... ]
 [TAB]  command1
 [TAB]  -command2 # ignore errors
 [TAB]  @command3 # suppress echoing

Here " [TAB] " is a TAB code. Each line is interpreted by the shell after make variable substitution. Use \ at the end of a line to continue the script. Use "$$" to enter "$" for environment values for a shell script.

Implicit rules for the target and prerequisites can be written, for example, as:

%.o: %.c header.h

Here, the target contains the character "%" (exactly one of them). The "%" can match any nonempty substring in the actual target filenames. The prerequisites likewise use "%" to show how their names relate to the actual target name.

Table 13.9.  List of make automatic variables.

automatic variable

value

$@

target

$<

first prerequisite

$?

all newer prerequisites

$^

all prerequisites

$*

% matched stem in the target pattern


Table 13.10.  List of make variable expansions.

variable expansion

effects

foo1 := bar

One-time expansion

foo2 = bar

Recursive expansion

foo3 += bar

Append


Run make -p -f/dev/null to see automatic internal rules.

13.3. C

You can set up proper environment to compile programs written in the C programming language by:

# aptitude install glibc-doc manpages-dev libc6-dev gcc build-essential

The libc6-dev package, i.e., GNU C Library, provides C standard library which is collection of header files and library routines used by the C programming language.

References for C:

  • info libc (C library function reference)

  • gcc(1) and info gcc

  • each_C_library_function_name(3)

  • Kernighan & Ritchie, "The C Programming Language", 2nd edition (Prentice Hall).

13.3.1. Simple C program (gcc)

A simple example to compile example.c with a library libm into an executable run_example:

$ cat > example.c << EOF
#include <stdio.h>
#include <math.h>
#include <string.h>

int main(int argc, char **argv, char **envp){
        double x;
        char y[11];
        x=sqrt(argc+7.5);
        strncpy(y, argv[0], 10); /* prevent buffer overflow */
        y[10] = '\0'; /* fill to make sure string ends with '\0' */
        printf("%5i, %5.3f, %10s, %10s\n", argc, x, y, argv[1]);
        return 0;
}
EOF
$ gcc -Wall -g -o run_example example.c -lm
$ ./run_example
        1, 2.915, ./run_exam,     (null)
$ ./run_example 1234567890qwerty
        2, 3.082, ./run_exam, 1234567890qwerty

Here, -lm is needed to link library libm for sqrt(). The actual library is in /lib/ with filename libm.so.6, which is a symlink to libm-2.1.3.so.

Look at the last parameter in the output text. There are more than 10 characters even though %10s is specified.

The use of pointer memory operation functions without boundary checks, such as sprintf and strcpy, is deprecated to prevent buffer overflow exploits that leverage the above overrun effects. Instead, use snprintf and strncpy.

13.3.2. Debugging with gdb

In order to be a good Debian user, you must be able to produce meaningful bug report using debugger. The fist step is to install gdb:

# aptitude install gdb gdb-doc build-essential devscripts

Good tutorial of gdb is provided by "info gdb" or found elsewhere on the web.

13.3.2.1. Basic gdb execution

Here is a simple example of using gdb(1) on a program compiled with the -g option to produce debugging information.

$ gdb program
(gdb) b 1                # set break point at line 1
(gdb) run args           # run program with args
(gdb) next               # next line
...
(gdb) step               # step forward
...
(gdb) p parm             # print parm
...
(gdb) p parm=12          # set value to 12
...
(gdb) quit
[Tip] Tip

Many gdb commands can be abbreviated. Tab expansion works as in the shell.

13.3.2.2. Debugging the Debian package

Since all installed binaries should be stripped on the Debian system by default, most debugging symbols are removed in the normal package. In order to debug Debian packages with make gdb, corresponding *-dbg packages need to be installed (e.g. libc6-dbg in the case of libc6).

If the package to be debugged does not provide *-dbg package, you need to install rebuild packages as follows:

$ mkdir /path/new ; cd /path/new
$ sudo aptitude update
$ sudo aptitude dist-upgrade
$ sudo aptitude install fakeroot devscripts build-essential
$ sudo apt-get build-dep source_package_name
$ apt-get source package_name
$ cd package_name*
  • fix bugs if needed.

$ dch -i
  • bump package version to one which does not collide with official Debian versions, e.g. one appended with "+debug1" when recompiling existing package version, or one appended with "~pre1" when compiling unreleased package version.

$ export DEB_BUILD_OPTIONS=nostrip,noopt
$ debuild
$ cd ..
$ sudo debi package_name*.changes 

You need to check the build scripts of the package and ensure to use CFLAGS=-g -Wall for compiling binaries.

13.3.2.3. Obtaining backtrace

When you encounter program crash, reporting bug report with cut-and-pasted backtrace information is a good idea.

The backtrace can be obtained by the following steps:

  • run the program under gdb,

  • reproduce crash to be dropped back to the gdb prompt, and

  • type "bt" at the gdb prompt.

In case of program freeze, you can crash the program by pressing Ctrl-C in the terminal running gdb to obtain gdb prompt.

[Tip] Tip

Often, you will see a backtrace where one or more of the top lines is in malloc() or g_malloc(). When this happens, chances are your backtrace isn't very useful. The easiest way to find some useful information is to set the environment variable MALLOC_CHECK_ to a value of 2. You can do this while running gdb by doing this:

 $ MALLOC_CHECK_=2 gdb hello

13.3.2.4. Advanced gdb commands

Table 13.11.  List of advanced gdb commands

objective

commands

To get a backtrace for all threads for multi-threaded program.

(gdb) thread apply all bt

To get parameters came on the stack of function calls.

(gdb) bt full

To get a backtrace and parameters as the combination of the preceding options.

(gdb) thread apply all bt full

To get them for top 10 calls to cut off irrelevant output.

(gdb) thread apply all bt full 10

To write log of gdb output to a file (the default is gdb.txt).

(gdb) set logging on


13.3.2.5. Debugging X Errors

If the Gnome program has received an X error; i.e. you see a message of the form:

The program 'preview1' received an X Window System error.

then you can try running the program with "--sync", and break on the "gdk_x_error" function in order to obtain a backtrace.

13.3.3. Check dependency on libraries

Use ldd(1) to find out a program's dependency on libraries:

$ ldd /bin/ls
        librt.so.1 => /lib/librt.so.1 (0x4001e000)
        libc.so.6 => /lib/libc.so.6 (0x40030000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x40153000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

For ls(1) to work in a chrooted environment, the above libraries must be available in your chrooted environment.

See also Section 10.5.6, “Trace program activities”.

13.3.4. Debugging with memory leak detection tools

There are several memory leak detection tools available in Debian.

Table 13.12.  List of memory leak detection tools

package

popcon

size

description

libc6-dev

V:28, I:65

11072

mtrace(1): malloc debugging functionality in glibc

valgrind

V:1.0, I:5

117335

memory debugger and profiler

kmtrace

V:0.2, I:1.6

344

KDE memory leak tracer using glibc's "mtrace(1)"

alleyoop

V:0.07, I:0.4

1516

Gnome front-end to the Valgrind memory checker

electric-fence

V:0.08, I:1.0

108

malloc(3) debugger

ccmalloc

V:0.05, I:0.4

232

memory profiler/debugger

leaktracer

V:0.01, I:0.15

116

memory-leak tracer for C++ programs

libdmalloc5

V:0.01, I:0.09

356

debug memory allocation library

mpatrolc2

V:0.01, I:0.05

3592

library for debugging memory allocations


13.3.5. Disassemble binary

You can disassemble binary code with objdump(1). For example:

$  objdump -m i386 -b binary -D /usr/lib/grub/x86_64-pc/stage1
[Note] Note

gdb(1) may be used to disassemble code interactively.

13.3.6. Flex -- a better Lex

Flex is a a Lex-compatible fast lexical analyzer generator.

Tutorial for flex(1) can be found in "info flex".

You need to provide your own main() and yywrap(), or your program.l should look like this to compile without a library (yywrap is a macro; %option main turns on %option noyywrap implicitly):

%option main
%%
.|\n    ECHO ;
%%

Alternatively, you may compile with the -lfl linker option at the end of your cc command line (like AT&T-Lex with -ll). No %option is needed in this case.

13.3.7. Bison -- a better Yacc

Several packages provide a Yacc-compatible lookahead LR parser or LALR parser generator in Debian:

Table 13.13.  List of Yacc-compatible LALR parser generators

package

popcon

size

description

bison

V:2, I:18

1392

GNU LALR parser generator

byacc

V:0.15, I:1.2

156

The Berkeley LALR parser generator

btyacc

V:0.00, I:0.07

248

Backtracking parser generator based on byacc


Tutorial for bison(1) can be found in "info bison".

You need to provide your own main() and yyerror(). main() calls yyparse() which calls yylex(), usually created with Flex.

%%

%%

13.4. Autoconf

Autoconf is a tool for producing shell scripts that automatically configure software source code packages to adapt to many kinds of Unix-like systems using the entire GNU build system.

autoconf(1) produces the configuration script configure. configure automatically creates a customized Makefile using the Makefile.in template.

13.4.1. Compile and install a program

[Warning] Warning

Do not overwrite system files with your compiled programs when installing them.

Debian does not touch files in /usr/local/ or /opt. So if you compile a program from source, install it into /usr/local/ so it will not interfere with Debian.

$ cd src
$ ./configure --prefix=/usr/local
$ make
$ make install # this puts the files in the system

13.4.2. Uninstall program

If you still have the source and if it uses autoconf/automake and if you can remember how you configured it:

$ ./configure ''all-of-the-options-you-gave-it''
# make uninstall

Alternatively, if you are absolutely sure that the install process puts files only under /usr/local/ and there is nothing important there, you can erase all its contents by:

# find /usr/local -type f -print0 | xargs -0 rm -f

If you are not sure where files are installed, you should consider using checkinstall, which provides a clean path for the uninstall.

13.5. Perl short script madness

Although any AWK scripts can be automatically rewritten in Perl using a2p(1), one-liner AWK scripts are best converted to one-liner perl scripts manually. For example

awk '($2=="1957") { print $3 }' |

is equivalent to any one of the following lines:

perl -ne '@f=split; if ($f[1] eq "1957") { print "$f[2]\n"}' |

perl -ne 'if ((@f=split)[1] eq "1957") { print "$f[2]\n"}' |

perl -ne '@f=split; print $f[2] if ( $f[1]==1957 )' |

perl -lane 'print $F[2] if $F[1] eq "1957"' |

perl -lane 'print$F[2]if$F[1]eq+1957' |

The last one is a riddle. It took advantage of the Perl features that

  • the whitespace is optional and

  • the automatic conversion from number to the string.

See perlrun(1) for the command-line options. For more crazy Perl scripts, Perl Golf may be interesting.

13.6. Web

Basic interactive dynamic web pages can be made as follows:

  • Queries are presented to the browser user using HTML forms.

  • Filling and clicking on the form entries will send an URL with encoded parameters from the browser to the web server. For example:

    • http://www.foo.dom/cgi-bin/program.pl?VAR1=VAL1&VAR2=VAL2&VAR3=VAL3

    • http://www.foo.dom/cgi-bin/program.py?VAR1=VAL1&VAR2=VAL2&VAR3=VAL3

    • http://www.foo.dom/program.php?VAR1=VAL1&VAR2=VAL2&VAR3=VAL3

  • %nn in URL is replaced with a character with hexadecimal nn value.

  • The environment variable is set as: QUERY_STRING="VAR1=VAL1 VAR2=VAL2 VAR3=VAL3"

  • CGI program (any one of program.*) on the web server executes itself with the environment variable "QUERY_STRING".

  • stdout of CGI program will be sent to the web browser and is presented as an interactive dynamic web page.

For security reasons it is better not to hand craft new hacks for parsing CGI parameters. There are established modules for them in Perl and Python. PHP comes with these functionalities. When client data storage is needed, cookies are used. When client side data processing is needed, javascript is frequently used.

For more, see The Common Gateway Interface, The Apache Software Foundation, and JavaScript.

Searching "CGI tutorial" on Google by typing encoded URL http://www.google.com/search?hl=en&ie=UTF-8&q=CGI+tutorial directly to the browser address is a good way to see the CGI script in action on the Google server.

13.7. Static code analysis tools

There are lint like tools for static code analysis:

Table 13.14.  List of tools for static code analysis

package

popcon

size

description

splint

V:0.08, I:0.5

1836

A tool for statically checking C programs for bugs

rats

V:0.06, I:0.2

768

Rough Auditing Tool for Security (C, C++, PHP, Perl, and Python code)

flawfinder

V:0.04, I:0.2

192

A tool to examine C/C++ source code and looks for security weaknesses

perl

V:84, I:99

18892

The perl package has internal code static checker: B::Lint(3perl)

pylint

V:0.07, I:0.4

676

A python code static checker

jlint

V:0.01, I:0.13

184

A Java program checker

weblint-perl

V:0.17, I:1.0

64

A syntax and minimal style checker for HTML

linklint

V:0.06, I:0.3

432

A fast link checker and web site maintenance tool

libxml2-utils

V:5, I:56

116

The libxml2-utils package provides xmllint(1) command to validate XML files


13.8. The source code translation

There are programs to convert source codes:

Table 13.15.  List of source code translation tools.

package

popcon

size

keyword

function

perl

V:84, I:99

18892

AWK->PERL

The a2p converter from AWK to PERL.

f2c

V:0.14, I:1.1

440

FORTRAN->C

The converter from A FORTRAN 77 to C/C++.

protoize

V:0.01, I:0.09

136

ANSI C

Create/remove ANSI prototypes from C code.

intel2gas

V:0.00, I:0.10

344

intel->gas

The converter from NASM (intel format) to the GNU Assembler (GAS).


13.9. Making Debian package

If you want to make a Debian package, read:

There are packages such as dh-make, dh-make-perl, etc., which help packaging.

Appendix A. Appendix

A.1. The Debian maze

The Linux system is a very powerful computing platform for a networked computer. However, learning how to use all its capabilities is not easy. Setting up the LPR printer with non-PostScript printer was a good example of stumble points. (There are no issues anymore since newer installations use new CUPS system.)

There is a complete, detailed map called the "SOURCE CODE". This is very accurate but very hard to understand. There are also references called HOWTO and mini-HOWTO. They are easier to understand but tend to give too much detail and lose the big picture. I sometimes have a problem finding the right section in a long HOWTO when I need a few commands to invoke.

I hope this "Debian Reference (version 2)" will provide good starting direction for people in the Debian maze.

A.2. Copyright history

Debian Reference was initiated by Osamu Aoki <osamu at debian dot org> as a personal system administration memo. Many contents came from the knowledge I gained from the debian-user mailing list and other Debian resources.

Following a suggestion from Josip Rodin, who was very active with the Debian Documentation Project (http://www.debian.org/doc/ddp) (DDP), "Debian Reference (version 1, 2001-2007)" was created as a part of DDP documents.

After 6 years, Osamu realized that the original "Debian Reference (version 1)" was outdated and started to rewrite many contents. New "Debian Reference (version 2)" is released in 2008.

The tutorial contents can trace its origin and its inspiration in:

  • "Linux User's Guide" by Larry Greenfield. (December 1996)

    • obsoleted by "Debian Tutorial".

  • "Debian Tutorial" by Havoc Pennington. (11 December, 1998)

    • Some contents was written by Oliver Elphick, Ole Tetlie, James Treacy, Craig Sawyer, and Ivan E. Moore II.

    • obsoleted by "Debian GNU/Linux: Guide to Installation and Usage".

  • "Debian GNU/Linux: Guide to Installation and Usage" by John Goerzen and Ossama Othman. (1999)

    • obsoleted by "Debian Reference (version 1)".

The package and archive description can trace some of their origin and their inspiration in:

  • "Debian FAQ" (March 2002 version, when this was maintained by Josip Rodin)

The other contents can trace some of their origin and their inspiration in:

  • "Debian Reference (version 1)" by Osamu Aoki. (2001–2007)

    • obsoleted by this new "Debian Reference (version 2)".

The previous "Debian Reference (version 1)" was created with,

  • the major contents contribution on network configuration topics by Thomas Hood,

  • significant contents contribution on X and VCS related topics by Brian Nelson,

  • the help on the build scripts and many content corrections by Jens Seidel,

  • extensive proofreading by David Sewell, and

  • many contributions by the translators, contributors, and bug reporters.

Many manual pages and info pages on the Debian system were used as the primary references to write this document. To the extent Osamu Aoki considered within the fair quotation stature, many parts of them, especially command definitions, were used as phrase pieces after careful editorial efforts to fit them into the style and the objective of this document.

The gdb debugger description was expanded using wiki contents on backtrace with consent by Ari Pollak, Loïc Minier, and Dafydd Harries.

Contents of "Debian Reference (version 2)" are mostly my own work except as mentioned above. These have been updated by the Wiki contributors too.

The author, Osamu Aoki, thanks all those who helped make this document possible.

A.3. Document format

This document is written using the MoinMoin Wiki since it is less typing than raw XML text and supports table in very intuitive format. The Wiki pages at http://wiki.debian.org/DebianReference are extracted as DocBookXML to be the XML source of this document. This document can be viewed as HTML, plain text, PostScript, and PDF. Currently, only HTML and plain text conversions are enabled now.