Table of Contents
I provide some pointers for people to learn programming on the Debian system enough to trace the packaged source code. Here are notable packages and corresponding documentation packages for programing.
Table 13.1. List of packages to help programing.
package |
popcon |
size |
documentation |
---|---|---|---|
|
V:4, I:26 |
1676 |
" |
|
V:2, I:14 |
1700 |
" |
|
V:86, I:99 |
1300 |
" |
|
V:3, I:20 |
1392 |
" |
|
V:46, I:87 |
72 |
" |
|
V:0.5, I:3 |
4104 |
" |
|
V:1.3, I:6 |
288 |
|
|
V:2, I:19 |
996 |
" |
|
V:22, I:25 |
1794 |
" |
|
V:17, I:65 |
64 |
" |
|
V:13, I:58 |
7128 |
" |
|
V:9, I:50 |
7980 |
" |
|
V:1.2, I:7 |
40 |
" |
|
V:0.4, I:2 |
1172 |
Help provided via menu |
|
V:0.2, I:1.6 |
488 |
Help provided via menu |
|
V:87, I:99 |
11452 |
" |
|
V:23, I:85 |
1220 |
" |
|
V:69, I:99 |
244 |
" |
|
V:84, I:99 |
18888 |
" |
|
V:59, I:95 |
612 |
" |
|
V:8, I:43 |
3308 |
" |
|
V:6, I:35 |
2800 |
" |
|
V:9, I:23 |
100 |
" |
|
V:13, I:28 |
1728 |
Help(F1) menu provided by |
|
I:0.03 |
48 |
|
|
I:0.09 |
48 |
Online references are available by typing man name
after installing manpages
and manpages-dev
packages. Online references for the GNU tools are available by typing info program_name
after installing the pertinent documentation packages. You may need to include the contrib
and non-free
archives in addition to the main
archive since GFDL is not DSFG compliant.
![]() |
Tip |
---|---|
Code examples of creating "Song 99 Bottles of Beer" should give you good idea of practically all the programming languages. |
![]() |
Caution |
---|---|
You should install software programs directly compiled from source into |
![]() |
Warning |
---|---|
Do not use " |
The shell script is a text file with the execution bit set and contains the commands in the following format.
#!/bin/sh ... command lines ...
The first line specifies the shell interpreter which read and execute this file contents.
Reading shell scripts is the best way to understand how a Unix-like system works. Here, I give some pointers and reminders for shell programming. See "Shell Mistakes" (http://www.greenend.org.uk/rjk/2001/04/shell.html) to learn from mistakes.
Unlike shell interactive mode (see: Section 2.5, “The simple shell command” and Section 2.6, “Unix-like text processing”), parameters, conditionals, and loops are used frequently.
Many system scripts may be interpreted by any one of POSIX shells (see Table 2.14, “ List of shell programs. ”). For this shell bash
may be used, but for the sake of speed, dash
is recommended. You can switch actual shell program to be used by changing symlink for /bin/sh
.
Avoid writing a shell script with "bashisms" or "zshisms" to make it portable among all POSIX shells:
Table 13.2. List of bashizms.
Good: POSIX |
Avoid: bashism |
---|---|
|
|
|
|
|
|
octal format: |
hexadecimal format: |
The "echo
" command must be used with care since its implementation differs among shell built-in commands and external command.
Avoid using any command options except "-n
". (Notably avoid "-e
" and "-E
")
Avoid using escape sequences in the string since their handling varies.
![]() |
Note |
---|---|
Although " |
![]() |
Tip |
---|---|
Use the " |
Special shell parameters are frequently used in the shell script:
Table 13.3. List of shell parameters.
shell parameter |
value |
---|---|
|
name of the shell or shell script |
|
first(1) shell argument |
|
ninth(9) shell argument |
|
number of positional parameters |
|
|
|
|
|
exit status of the most recent command |
|
PID of this shell script |
|
PID of most recently started background job |
Basic parameter expansions to remember:
Table 13.4. List of shell parameter expansions.
parameter expression form |
If |
If |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
(echo string to stderr and exit with error) |
Here, the colon ':' in all of these operators is actually optional.
With ':' = operator test for "exist" and "not null".
Without ':' = operator test for "exist" only.
Table 13.5. List of key shell parameter substitutions.
parameter substitution form |
Result |
---|---|
|
Remove smallest suffix pattern |
|
Remove largest suffix pattern |
|
Remove smallest prefix pattern |
|
Remove largest prefix pattern |
Each command returns an exit status which can be used for conditional expressions:
Success: 0 (True)
Error: 1--255 (False)
Note that the use here of a 0 value to mean "true" differs from the usual convention in some other areas of computing. Also, `[' is the equivalent of the test
command, which evaluates its arguments up to `]' as a conditional expression.
Basic conditional idioms to remember are:
<command> && <if_success_run_this_command_too> || true
<command> || <if_not_success_run_this_command_too> || true
if [ <conditional_expression> ]; then <if_success_run_this_command> else <if_not_success_run_this_command> fi
Here trailing "|| true
" was needed to ensure this shell script will not exit at this line accidentally when shell is invoked with -e
flag.
Table 13.6. List of file comparison operators in the conditional expression.
equation |
value |
---|---|
-e <file> |
True if <file> exists. |
-d <file> |
True if <file> exists and is a directory. |
-f <file> |
True if <file> exists and is a regular file. |
-w <file> |
True if <file> exists and is writable. |
-x <file> |
True if <file> exists and is executable. |
<file1> -nt <file2> |
True if <file1> is newer than <file2>. (modification). |
<file1> -ot <file2> |
True if <file1> is older than <file2>. (modification). |
<file1> -ef <file2> |
True if they are the same device and inode number. |
Table 13.7. List of string comparison operators in the conditional expression.
equation |
value |
---|---|
-z <str> |
True if the length of <str> is zero. |
-n <str> |
True if the length of <str> is non-zero. |
<str1> = <str2> |
True if <str1> and <str2> are equal. |
<str1> != <str2> |
True if <str1> and <str2> are not equal. |
<str1> < <str2> |
True if <str1> sorts before <str2>. (locale dependent) |
<str1> > <str2> |
True if <str1> sorts after <str2>. (locale dependent) |
Arithmetic integer comparison operators in the conditional expression are -eq
, -ne
, -lt
, -le
, -gt
, and -ge
.
There are several loop idioms to use in POSIX shell:
"for name [ in word ] ; do list ; done
": loops over list of words.
"while list; do list; done
": repeats while true.
"until list; do list; done
": repeats while not true.
"break
": enables to exit from the loop.
"continue
" enables to resume the next iteration of the loop.
![]() |
Note |
---|---|
The C-language like numeric iteration can be realized by using the |
The shell processes a script as following sequence:
split into tokens by the metacharacters: SPACE TAB NEWLINE ; ( ) < > | &
check keyword if not within "..." or '...' (loop)
expand alias if not within "..." or '...' (loop)
expand tilde, ~
<user> -> <user>'s home directory, if not within "..." or '...'
expand parameter, $
PARAMETER, if not within '...'
expand command substitution, $(
command)
, if not within '...'
split into words with $IFS if not within "..." or '...'
expand * ? [ ]
in pathname if not within "..." or '...'
look up command
function
built-in
file in $PATH
loop
Single quotes within double quotes have no effect.
Executing set -x
in the shell or invoking the shell with -x
option make the shell to print all of commands executed. This is quite handy for debugging.
When writing script to looping over each file matching *.ext
, there are several options. The following examples are written as an part of shell script file.
You must replace each line break with a semicolon (excluding ones escaped by preceding \
which is replaced with ) for the interactive shell prompt.
for x in *.ext; do if test -f "$x"; then command "$x" fi done
find
and xargs
combination:
find . -type f -maxdepth 1 -name '*.ext' -print0 | xargs -0 -n 1 command
find
with -exec
option with a command:
find . -type f -maxdepth 1 -name '*.ext' -exec command '{}' \;
find
with -exec
option with a short shell script:
find . -type f -maxdepth 1 -name '*.ext' \ -exec sh -c "command '{}' && echo 'successful'" \;
The above examples are written to ensures proper handling of funny file names such as ones containing spaces.
In order to make your shell program as portable as possible across Debian system, it is good idea to limit utility programs used within Essential programs listed by "aptitude search ~E
" as much as possible.
The coreutils
, bsdutils
, and debianutils
packages contain many useful small utilities.
The user interface of a simple shell program can be improved from dull interaction by the "echo
" and "read
" commands to more interactive one by using one of the so-called dialog program etc.
Table 13.8. List of user interface programs.
package |
popcon |
size |
function |
---|---|---|---|
|
V:33, I:67 |
52 |
The |
|
V:11, I:21 |
632 |
The |
|
V:51, I:99 |
96 |
displays user-friendly dialog boxes from shell scripts. (newt) |
|
V:5, I:22 |
1496 |
displays user-friendly dialog boxes from shell scripts. (ncurses) |
|
V:5, I:51 |
4336 |
display graphical dialog boxes from shell scripts. (gtk2.0) |
|
V:0.2, I:1.1 |
1176 |
X11 replacement for the text utility |
|
V:0.06, I:0.3 |
488 |
GUI-creation command-line utility based on GTK+ library. (gtk2.0+glade2) |
|
V:0.02, I:0.11 |
152 |
Shell Scripts Frontend Tool. (wrapper for zenity, kdialog, and dialog with gettext) |
|
V:9, I:50 |
7980 |
The |
Here is a simple script which creates ISO image with RS02 data supplemented by dvdisaster
(1):
#!/bin/sh -e # gmkrs02 : Copyright (C) 2007 Osamu Aoki <osamu@debian.org>, Public Domain #set -x error_exit() { echo "$1" >&2 exit 1 } # Initialize variables DATA_ISO="$HOME/Desktop/iso-$$.img" LABEL=$(date +%Y%m%d-%H%M%S-%Z) if [ $# != 0 ] && [ -d "$1" ]; then DATA_SRC="$1" else # Select directory for creating ISO image from folder on desktop DATA_SRC=$(zenity --file-selection --directory \ --title="Select the directory tree root to create ISO image") \ || error_exit "Exit on directory selection" fi # Check size of archive xterm -T "Check size $DATA_SRC" -e du -s $DATA_SRC/* SIZE=$(($(du -s $DATA_SRC | awk '{print $1}')/1024)) if [ $SIZE -le 520 ] ; then zenity --info --title="Dvdisaster RS02" --width 640 --height 400 \ --text="The data size is good for CD backup:\\n $SIZE MB" elif [ $SIZE -le 3500 ]; then zenity --info --title="Dvdisaster RS02" --width 640 --height 400 \ --text="The data size is good for DVD backup :\\n $SIZE MB" else zenity --info --title="Dvdisaster RS02" --width 640 --height 400 \ --text="The data size is too big to backup : $SIZE MB" error_exit "The data size is too big to backup :\\n $SIZE MB" fi # only xterm is sure to have working -e option # Create raw ISO image rm -f "$DATA_ISO" || true xterm -T "genisoimage $DATA_ISO" \ -e genisoimage -r -J -V "$LABEL" -o "$DATA_ISO" "$DATA_SRC" # Create RS02 supplemental redundancy xterm -T "dvdisaster $DATA_ISO" -e dvdisaster -i "$DATA_ISO" -mRS02 -c zenity --info --title="Dvdisaster RS02" --width 640 --height 400 \ --text="ISO/RS02 data ($SIZE MB) \\n created at: $DATA_ISO" # EOF
You may wish to create launcher on the desktop with command set something like "/usr/local/bin/gmkrs02 %d
".
Make is a utility to maintain groups of programs. Upon execution of make
(1), make
read the rule file, Makefile
, and updates a target if it depends on prerequisite files that have been modified since the target was last modified, or if the target does not exist. The execution of these updates may occur concurrently.
The rule file syntax is :
target: [ prerequisites ... ] [TAB] command1 [TAB] -command2 # ignore errors [TAB] @command3 # suppress echoing
Here " [TAB]
" is a TAB code. Each line is interpreted by the shell after make
variable substitution. Use \
at the end of a line to continue the script. Use "$$
" to enter "$
" for environment values for a shell script.
Implicit rules for the target and prerequisites can be written, for example, as:
%.o: %.c header.h
Here, the target contains the character "%
" (exactly one of them). The "%
" can match any nonempty substring in the actual target filenames. The prerequisites likewise use "%
" to show how their names relate to the actual target name.
Table 13.9. List of make automatic variables.
automatic variable |
value |
---|---|
|
target |
|
first prerequisite |
|
all newer prerequisites |
|
all prerequisites |
|
|
Table 13.10. List of make variable expansions.
variable expansion |
effects |
---|---|
|
One-time expansion |
|
Recursive expansion |
|
Append |
Run make -p -f/dev/null
to see automatic internal rules.
You can set up proper environment to compile programs written in the C programming language by:
# aptitude install glibc-doc manpages-dev libc6-dev gcc build-essential
The libc6-dev
package, i.e., GNU C Library, provides C standard library which is collection of header files and library routines used by the C programming language.
References for C:
info libc
(C library function reference)
gcc
(1) and info gcc
each_C_library_function_name
(3)
Kernighan & Ritchie, "The C Programming Language", 2nd edition (Prentice Hall).
A simple example to compile example.c
with a library libm
into an executable run_example
: Debian New Maintainers' Guide
$ cat > example.c << EOF #include <stdio.h> #include <math.h> #include <string.h> int main(int argc, char **argv, char **envp){ double x; char y[11]; x=sqrt(argc+7.5); strncpy(y, argv[0], 10); /* prevent buffer overflow */ y[10] = '\0'; /* fill to make sure string ends with '\0' */ printf("%5i, %5.3f, %10s, %10s\n", argc, x, y, argv[1]); return 0; } EOF $ gcc -Wall -g -o run_example example.c -lm $ ./run_example 1, 2.915, ./run_exam, (null) $ ./run_example 1234567890qwerty 2, 3.082, ./run_exam, 1234567890qwerty
Here, -lm
is needed to link library libm
for sqrt()
. The actual library is in /lib/
with filename libm.so.6
, which is a symlink to libm-2.1.3.so
.
Look at the last parameter in the output text. There are more than 10 characters even though %10s
is specified.
The use of pointer memory operation functions without boundary checks, such as sprintf
and strcpy
, is deprecated to prevent buffer overflow exploits that leverage the above overrun effects. Instead, use snprintf
and strncpy
.
In order to be a good Debian user, you must be able to produce meaningful bug report using debugger. The fist step is to install gdb
:
# aptitude install gdb gdb-doc build-essential devscripts
Good tutorial of gdb
is provided by "info gdb
" or found elsewhere on the web.
Here is a simple example of using gdb
(1) on a program
compiled with the -g
option to produce debugging information.
$ gdb program (gdb) b 1 # set break point at line 1 (gdb) run args # run program with args (gdb) next # next line ... (gdb) step # step forward ... (gdb) p parm # print parm ... (gdb) p parm=12 # set value to 12 ... (gdb) quit
![]() |
Tip |
---|---|
Many |
Since all installed binaries should be stripped on the Debian system by default, most debugging symbols are removed in the normal package. In order to debug Debian packages with make gdb
, corresponding *-dbg
packages need to be installed (e.g. libc6-dbg in the case of libc6).
If the package to be debugged does not provide *-dbg
package, you need to install rebuild packages as follows:
$ mkdir /path/new ; cd /path/new $ sudo aptitude update $ sudo aptitude dist-upgrade $ sudo aptitude install fakeroot devscripts build-essential $ sudo apt-get build-dep source_package_name $ apt-get source package_name $ cd package_name* ... fix bugs if needed $ dch -i ... bump version $ export DEB_BUILD_OPTIONS=nostrip,noopt $ debuild $ cd .. $ sudo debi package_name*.changes
![]() |
Tip |
---|---|
Use of non-official different package version such as one appended with |
You need to check the build scripts of the package and ensure to use CFLAGS=-g -Wall
for compiling binaries.
When you encounter program crash, reporting bug report with cut-and-pasted backtrace information is a good idea.
The backtrace can be obtained by the following steps:
run the program under gdb
,
reproduce crash to be dropped back to the gdb prompt, and
type "bt
" at the gdb prompt.
In case of program freeze, you can crash the program by pressing Ctrl-C
in the terminal running gdb to obtain gdb prompt.
![]() |
Tip |
---|---|
Often, you will see a backtrace where one or more of the top lines is in malloc() or g_malloc(). When this happens, chances are your backtrace isn't very useful. The easiest way to find some useful information is to set the environment variable MALLOC_CHECK_ to a value of 2. You can do this while running gdb by doing this: |
$ MALLOC_CHECK_=2 gdb hello
Table 13.11. List of advanced gdb commands
objective |
commands |
---|---|
To get a backtrace for all threads for multi-threaded program. |
|
To get parameters came on the stack of function calls. |
|
To get a backtrace and parameters as the combination of the preceding options. |
|
To get them for top 10 calls to cut off irrelevant output. |
|
To write log of gdb output to a file (the default is gdb.txt). |
|
Use ldd
to find out a program's dependency on libraries:
$ ldd /bin/ls librt.so.1 => /lib/librt.so.1 (0x4001e000) libc.so.6 => /lib/libc.so.6 (0x40030000) libpthread.so.0 => /lib/libpthread.so.0 (0x40153000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
For ls
to work in a chroot
ed environment, the above libraries must be available in your chroot
ed environment.
The following commands will also be useful:
strace
: trace system calls and signals
ltrace
: trace library calls
There are several memory leak detection tools available in Debian.
Table 13.12. List of memory leak detection tools
package |
popcon |
size |
description |
---|---|---|---|
|
V:25, I:65 |
11068 |
|
|
V:1.0, I:4 |
103468 |
memory debugger and profiler |
|
V:0.2, I:1.8 |
344 |
KDE memory leak tracer using glibc's " |
|
V:0.06, I:0.5 |
1516 |
Gnome front-end to the Valgrind memory checker |
|
V:0.08, I:0.9 |
108 |
malloc(3) debugger |
|
V:0.05, I:0.4 |
232 |
memory profiler/debugger |
|
V:0.02, I:0.16 |
116 |
memory-leak tracer for C++ programs |
|
V:0.01, I:0.07 |
356 |
debug memory allocation library |
|
V:0.00, I:0.04 |
3592 |
library for debugging memory allocations |
You can disassemble binary code with objdump
(1). For example:
$ objdump -m i386 -b binary -D /usr/lib/grub/x86_64-pc/stage1
![]() |
Note |
---|---|
|
Flex is a a Lex-compatible fast lexical analyzer generator.
Tutorial for flex
(1) can be found in "info flex
".
You need to provide your own main()
and yywrap()
, or your program.l
should look like this to compile without a library (yywrap
is a macro; %option main
turns on %option noyywrap
implicitly):
%option main %% .|\n ECHO ; %%
Alternatively, you may compile with the -lfl
linker option at the end of your cc
command line (like AT&T-Lex with -ll
). No %option
is needed in this case.
Several packages provide a Yacc-compatible lookahead LR parser or LALR parser generator in Debian:
Table 13.13. List of Yacc-compatible LALR parser generators
package |
popcon |
size |
description |
---|---|---|---|
|
V:3, I:20 |
1392 |
|
|
V:0.14, I:1.2 |
164 |
The Berkeley LALR parser generator |
|
V:0.01, I:0.07 |
248 |
Backtracking parser generator based on |
Tutorial for bison
(1) can be found in "info bison
".
You need to provide your own main()
and yyerror()
. main()
calls yyparse()
which calls yylex()
, usually created with Flex.
%% %%
Autoconf is a tool for producing shell scripts that automatically configure software source code packages to adapt to many kinds of Unix-like systems using the entire GNU build system.
autoconf
(1) produces the configuration script configure
. configure
automatically creates a customized Makefile
using the Makefile.in
template.
![]() |
Warning |
---|---|
Do not overwrite system files with your compiled programs when installing them. |
Debian does not touch files in /usr/local/
or /opt
. So if you compile a program from source, install it into /usr/local/
so it will not interfere with Debian.
$ cd src $ ./configure --prefix=/usr/local $ make $ make install # this puts the files in the system
If you still have the source and if it uses autoconf
/automake
and if you can remember how you configured it:
$ ./configure ''all-of-the-options-you-gave-it'' # make uninstall
Alternatively, if you are absolutely sure that the install process puts files only under /usr/local/
and there is nothing important there, you can erase all its contents by:
# find /usr/local -type f -print0 | xargs -0 rm -f
If you are not sure where files are installed, you should consider using checkinstall
, which provides a clean path for the uninstall.
Although any AWK scripts can be automatically rewritten in Perl using a2p
(1), one-liner AWK scripts are best converted to one-liner perl scripts manually. For example
awk '($2=="1957") { print $3 }' |
is equivalent to any one of the following lines:
perl -ne '@f=split; if ($f[1] eq "1957") { print "$f[2]\n"}' |
perl -ne 'if ((@f=split)[1] eq "1957") { print "$f[2]\n"}' |
perl -ne '@f=split; print $f[2] if ( $f[1]==1957 )' |
perl -lane 'print $F[2] if $F[1] eq "1957"' |
perl -lane 'print$F[2]if$F[1]eq+1957' |
The last one is a riddle. It took advantage of the Perl features that
the whitespace is optional and
the automatic conversion from number to the string.
See perlrun
(1) for the command-line options. For more crazy Perl scripts, Perl Golf may be interesting.
Basic interactive dynamic web pages can be made as follows:
Queries are presented to the browser user using HTML forms.
Filling and clicking on the form entries will send an URL with encoded parameters from the browser to the web server. For example:
http://www.foo.dom/cgi-bin/program.pl?VAR1=VAL1&VAR2=VAL2&VAR3=VAL3
http://www.foo.dom/cgi-bin/program.py?VAR1=VAL1&VAR2=VAL2&VAR3=VAL3
http://www.foo.dom/program.php?VAR1=VAL1&VAR2=VAL2&VAR3=VAL3
%nn
in URL is replaced with a character with hexadecimal nn
value.
The environment variable is set as: QUERY_STRING="VAR1=VAL1 VAR2=VAL2 VAR3=VAL3"
CGI program (any one of program.*
) on the web server executes itself with the environment variable "QUERY_STRING
".
stdout
of CGI program will be sent to the web browser and is presented as an interactive dynamic web page.
For security reasons it is better not to hand craft new hacks for parsing CGI parameters. There are established modules for them in Perl and Python. PHP comes with these functionalities. When client data storage is needed, cookies are used. When client side data processing is needed, javascript is frequently used.
For more, see The Common Gateway Interface, The Apache Software Foundation, and JavaScript.
Searching "CGI tutorial" on Google by typing encoded URL http://www.google.com/search?hl=en&ie=UTF-8&q=CGI+tutorial directly to the browser address is a good way to see the CGI script in action on the Google server.
There are lint like tools for static code analysis:
Table 13.14. List of tools for static code analysis
package |
popcon |
size |
description |
---|---|---|---|
|
V:0.07, I:0.6 |
1788 |
A tool for statically checking C programs for bugs |
|
V:0.05, I:0.2 |
192 |
A tool to examine C/C++ source code and looks for security weaknesses |
|
V:84, I:99 |
18888 |
The |
|
V:0.11, I:0.4 |
644 |
A python code static checker |
|
V:0.02, I:0.14 |
184 |
A Java program checker |
|
V:0.19, I:1.1 |
64 |
A syntax and minimal style checker for HTML |
|
V:0.06, I:0.3 |
432 |
A fast link checker and web site maintenance tool |
|
V:5, I:55 |
116 |
The |
There are program to convert source codes:
Table 13.15. List of source code translation tools.
package |
popcon |
size |
keyword |
function |
---|---|---|---|---|
|
V:84, I:99 |
18888 |
AWK->PERL |
The |
|
V:0.13, I:1.1 |
440 |
FORTRAN->C |
The converter from A FORTRAN 77 to C/C++. |
|
V:0.00, I:0.08 |
136 |
ANSI C |
Create/remove ANSI prototypes from C code. |
|
V:0.00, I:0.10 |
344 |
intel->gas |
The converter from NASM (intel format) to the GNU Assembler (GAS). |
If you want to make a Debian package, read:
Chapter 3, Debian package management to understand the basic package system,
Section 3.4.8, “Port a package to the stable system” to understand basic porting process,
Section 10.6, “The chroot” to understand basic chroot techniques,
manpages of pbuilder
(1) and pdebuild
(1) commands,
Debian New Maintainers' Guide as tutorial (maint-guide
package),
Debian Developer's Reference (developers-reference
package), and
Debian Policy Manual (debian-policy
package).
There are packages such as dh-make
, dh-make-perl
, etc., which help packaging.