Chapter 12. Data conversion

Table of Contents

12.1. Text data conversion tools
12.1.1. To convert a text file with iconv
12.1.2. To convert file names with iconv
12.1.3. EOL conversion
12.1.4. TAB conversion
12.1.5. Editors with auto-conversion
12.1.6. Plain text extraction
12.1.7. Highlighting and formatting plain text data
12.2. XML data
12.2.1. Basic hints for XML
12.2.2. XML processing
12.2.3. The XML data extraction
12.3. Printable data
12.3.1. The Ghostscript
12.3.2. Merge two PS or PDF files
12.3.3. Printable data utilities
12.3.4. Printing with CUPS
12.4. Type setting
12.4.1. roff typesetting
12.4.2. TeX/LaTeX
12.4.3. Pretty print a manual page
12.4.4. Creating a manual page
12.5. The mail data conversion
12.5.1. Mail data basics
12.6. Graphic data tools
12.7. Miscellaneous data conversion

Standard based tools are in very good shape but support for proprietary data formats are limited.

12.1. Text data conversion tools

Following packages for the text data conversion caught my eyes:

Table 12.1.  List of text data conversion tools.

package

popcon

size

keyword

function

libc6

V:87, I:99

11452

charset

The text encoding conversion between locales with iconv command. (fundamental)

recode

V:1.8, I:7

780

charset+eol

The text encoding conversion between locales. (versatile, more aliases and features)

konwert

V:0.3, I:3

188

charset

The text encoding conversion between locales. (fancy)

nkf

V:0.4, I:2

300

charset

The character set translator for Japanese.

tcs

V:0.03, I:0.2

544

charset

The character set translator.

unaccent

V:0.02, I:0.11

60

charset

Replace accented letters by their unaccented equivalent.

tofrodos

V:1.4, I:8

80

eol

The text format converter between DOS and Unix: fromdos and todos

macutils

V:0.10, I:0.9

356

eol

The text format converter between Macintosh and Unix: frommac and tomac


12.1.1. To convert a text file with iconv

The iconv command is provided as a part of the libc6 package and it is always available on all system to convert the encoding of characters:

$ iconv -f encoding1 -t encoding2 input.txt >output.txt

Encoding values are case insensitive and ignore "-" and "_" for matching. The supported encodings can be checked by the "iconv -l" command.

Table 12.2.  List of encoding values and their usage.

encoding value

usage

ASCII.

American Standard Code for Information Interchange. 7 bit code w/o accented characters.

UTF-8

Standard multilingual compatibility] for all modern OSs.

ISO-8859-1

Old standard for western European languages, ASCII + accented characters.

ISO-8859-2

Old standard for eastern European languages, ASCII + accented characters.

ISO-8859-15

Old standard for western European languages, ISO-8859-1 with euro sign.

CP850

Code page 850, Microsoft DOS characters with graphics for western European languages. ISO-8859-1 variant.

CP932

Code page 932, Microsoft Windows style Shift-JIS variant, for Japanese.

CP936

Code page 936, Microsoft Windows style GB2312, GBK, or GB18030 variant, for Simplified Chinese.

CP949

Code page 949, Microsoft Windows style EUC-KR or Unified Hangul Code variant, for Korean.

CP950

Code page 950, Microsoft Windows style Big5 variant, for Traditional Chinese.

CP1251

Code page 1251, Microsoft Windows style encoding for the Cyrillic alphabet.

CP1252

Code page 1252, Microsoft Windows style ISO-8859-15 variant for western European languages.

KOI8-R

Old Russian UNIX standard for the Cyrillic alphabet.

ISO-2022-JP

Standard encoding for Japanese e-mail which uses only 7 bit codes.

eucJP

Old Japanese UNIX standard 8 bit code and completely different from Shift-JIS.

Shift-JIS

JIS X 0208 Appendix 1 standard, for Japanese. See CP932 above.


[Note] Note

Some encodings are only supported for the data conversion and are not used as locale values (Section 9.3.1, “Basics of encoding”).

For character sets which fit in single byte such as ASCII and ISO-8859 character sets, the character encoding means almost the same thing as the character set.

For character sets with many characters such as JIS X 0213 for Japanese or Universal Character Set (UCS, Unicode, ISO-10646-1) for practically all languages, there are many encoding schemes to fit them into the sequence of the byte data:

For these, there are clear differentiation between the character set and the character encoding.

The code page is used as the synonym to the character encoding tables for some vender specific ones.

[Note] Note

Please note most encoding systems share the same code with ASCII for the 7 bit characters. But there are some exceptions. If you are converting old Japanese C programs and URLs data from the casually-called shift-JIS encoding format to UTF-8 format, use "CP932" as the encoding name instead of "shift-JIS" to get the expected results: 0x5C -> "\" and 0x7E -> "~" . Otherwise, these are converted to wrong characters.

[Tip] Tip

The recode command may be used too and offers more than the combined functionality of the iconv, fromdos, todos, frommac, and tomac commands. For more, see pertinent description in the "info recode".

12.1.2. To convert file names with iconv

Here is an example script to convert encoding of the file name from ones created under older OS to modern UTF-8 ones for the simple case.

#!/bin/sh
ENCDN=iso-8859-1
for x in *;
 do
 mv "$x" $(echo "$x" | iconv -f $ENCDN -t utf-8)
done

The "$ENCDN" variable should be set by the encoding values from Table 12.2, “ List of encoding values and their usage. ” .

For more complicated case, please mount disk drive containing such file names with proper encoding as the mount(8) option (see Section 9.3.6, “Filename encoding”) and copy entire disk to another disk drive mounted as UTF-8 with "cp -a" command.

12.1.3. EOL conversion

The text file format, specifically the end-of-line (EOL) code, is dependent on the platform:

Table 12.3.  List of EOL conversion tools.

platform

EOL code

EOL control sequence

EOL ASCII value

Debian (unix)

LF

^J

10

MSDOS and Windows

CR-LF

^M^J

13, 10

Apple's Macintosh

CR

^M

13


The EOL format conversion programs, fromdos(1), todos(1), frommac(1), and tomac(1), are quite handy. The recode command is also useful.

[Tip] Tip

The use of "sed -e '/\r$/!s/$/\r/'" instead of "todos" is better when you want to unify the EOL style to the MSDOS style from the mixed MSDOS and Unix style. (e.g., after merging 2 MSDOS style files with diff3.) This is because "todos" adds CR to all lines.

[Note] Note

Some data on the Debian system, such as the wiki page data for the python-moinmoin, use MSDOS style CR-LF as the EOL code. So the above rule is just general rule.

[Note] Note

Most editors (eg. vim, emacs, gedit, ...) can handle files in MSDOS style EOL transparently.

12.1.4. TAB conversion

You can expand the tab code in the text to the multiple spaces in vim using the ":retab" command.

There are few popular specialized programs to convert the tab codes:

Table 12.4.  List of TAB conversion commands from bsdmainutils and coreutils packages.

function

bsdmainutils

coreutils

expand tab to spaces

"col -x"

expand

unexpand tab from spaces

"col -h"

unexpand


The indent(1) from the indent package completely reformats whitespaces in the C program.

12.1.5. Editors with auto-conversion

Intelligent modern editors such as the vim program are quite smart and copes well with any encoding systems and any file formats. You should use these editors under the UTF-8 locale in the UTF-8 capable console for the best compatibility.

An old western European Unix text file, "u-file.txt", stored in the latin1 encoding can be edited simply with vim as:

$ vim u-file.txt

This is possible since the auto detection mechanism of the file encoding in vim assumes the UTF-8 encoding first and, if it fails, assumes it to be latin1.

An old Polish Unix text file, "pu-file.txt", stored in the latin2 encoding can be edited with vim as:

$ vim '+e ++enc=latin2 pu-file.txt'

An old Japanese unix text file, "ju-file.txt", stored in the eucJP encoding can be edited with vim as:

$ vim '+e ++enc=eucJP ju-file.txt'

An old Japanese MS-Windows text file, "jw-file.txt", stored in the so called shift-JIS encoding (more precisely: CP932) can be edited with vim as:

$ vim '+e ++enc=CP932 ++ff=dos jw-file.txt'

When a file is opened with "++enc" and "++ff" options, the "w" in the Vim command line stores it in the original format and overwrite the original file. You can also specify the saving format and the file name in the Vim command line, e.g., "w ++enc=utf8 new.txt".

Please refer to the mbyte.txt "multi-byte text support" in vim on-line help.

The emacs family of programs can perform the equivalent functions.

12.1.6. Plain text extraction

Following will read a web page into a text file. This is very useful when copying configurations off the Web or applying basic Unix text tools such as grep on the web page.

$ lynx -dump http://www.remote-site.com/help-info.html >textfile

Similarly, you can extract plain text data from other formats using followings:

Table 12.5.  List of tools to extract plain text data.

package

popcon

size

keyword

function

html2text

V:11, I:39

288

html->text

An advanced HTML to text converter. (Better than "lynx -dump")

w3m

V:18, I:85

1968

html->text

An HTML to text converter with the "w3m -dump" command.

lynx

V:7, I:26

44

html->text

An HTML to text converter with the "lynx -dump" command.

elinks

V:2, I:6

1296

html->text

An HTML to text converter with the "elinks -dump" command.

links

V:2, I:9

1324

html->text

An HTML to text converter with the "links -dump" command.

links2

V:0.8, I:4

3224

html->text

An HTML to text converter with the "links2 -dump" command.

antiword

V:1.1, I:2

780

MSWord->text,ps

This converts !MSWord files to plain text or ps.

catdoc

V:0.8, I:2

2664

MSWord->text,TeX

This converts !MSWord files to plain text or TeX.

pstotext

V:0.8, I:1.6

160

ps/pdf->text

Extract text from PostScript and PDF files.

unhtml

V:0.04, I:0.2

76

html->text

Remove the markup tags from an HTML file.

odt2txt

V:0.4, I:0.6

104

odt->text

The converter from OpenDocument Text to text.

wpd2sxw

V:0.03, I:0.2

156

WordPerfect->sxw

WordPerfect to OpenOffice.org/StarOffice writer document converter.


12.1.7. Highlighting and formatting plain text data

Table 12.6.  List of tools to highlight plain text data.

package

popcon

size

keyword

function

vim-runtime

V:3, I:32

24464

highlight

Vim can convert source code to HTML with :source $VIMRUNTIME/syntax/html.vim (vim MACRO)

cxref

V:0.11, I:0.8

1104

c->html

The converter for the C program to latex and HTML. (C)

src2tex

V:0.06, I:0.3

1968

highlight

This convert many source codes to TeX. (C)

source-highlight

V:0.09, I:0.6

1980

highlight

This convert many source codes to HTML, XHTML, LaTeX, Texinfo, ANSI color escape sequences and DocBook files with highlight. (C++)

highlight

V:0.07, I:0.4

1664

highlight

This convert many source codes to HTML, XHTML, RTF, LaTeX, TeX or XSL-FO files with highlight. (C++)

grc

V:0.04, I:0.12

164

text->color

The generic colouriser for everything. (Python)

txt2html

V:0.08, I:0.4

296

text->html

Text to HTML converter. (Perl)

markdown

V:0.10, I:0.5

96

text->html

The converter from text to (X)HTML. (Perl)

asciidoc

V:0.10, I:0.7

2756

text->any

A text document formatter to XML. (Python)

txt2tags

V:0.07, I:0.4

1556

text->any

The document conversion from text to HTML, SGML, LaTeX, man page, MoinMoin, Magic Point and PageMaker. (Python)

udo

V:0.01, I:0.08

556

text->any

universal document - text processing utility. (C)

stx2any

V:0.01, I:0.07

484

text->any

The document converter from structured plain text to other formats. (m4)

rest2web

V:0.01, I:0.09

644

text->html

The document converter from ReStructured Text to html. (Python)

aft

V:0.00, I:0.09

336

text->any

The "free form" document preparation system. (Perl)

yodl

V:0.00, I:0.06

1756

text->any

A pre-document language and tools to process it. (C)

sdf

V:0.01, I:0.11

1940

text->any

The simple document parser. (Perl)

sisu

V:0.01, I:0.07

5316

text->any

The document structuring, publishing and search framework. (Ruby)


12.2. XML data

The Extensible Markup Language (XML) is a markup language for documents containing structured information.

XML.COM has good introductory information:

12.2.1. Basic hints for XML

XML text looks somewhat like HTML. It enables us to manage multiple formats of output for a document. One easy XML system is docbook-xsl, which is used here.

Each XML file starts with standard XML declaration:

<?xml version="1.0" encoding="UTF-8"?>

The basic syntax for one XML element is marked up as:

<name attribute="value">content</name>

XML element with empty content is marked up in the short form as:

<name attribute="value"/>

The "attribute="value"" in the above examples are optional.

The comment section in XML is marked up as:

<!-- comment -->

Other than adding markups, XML requires minor conversion to the content using predefined entities for the following character:

Table 12.7.  List of predefined entities for XML.

predefined entity

character to be converted from

&quot;

" : quote

&apos;

' : apostrophe

&lt;

< : less-than

&gt;

> : greater-than

&amp;

& : ampersand


[Caution] Caution

"<" or "&" can not be used in attributes or elements.

[Note] Note

When SGML style user defined entities, e.g. "&some-tag:", are used, the first definition wins over others. The entity definition is expressed in "<!ENTITY some-tag "entity value">".

[Note] Note

As long as the XML markup are done consistently with certain set of the tag name (either some data as content or attribute value), conversion to another XML is trivial task using XSLT.

12.2.2. XML processing

There are many tools available to process XML files such as the Extensible Stylesheet Language (XSL).

Basically, once you create well formed XML file, you can convert it to any format using Extensible Stylesheet Language for Transformation (XSLT).

Although the Extensible Stylesheet Language for Formatting Object (XSL-FO) is supposed to be solution for formatting, FOP program is not in the Debian main (yet?). So the LaTeX code is usually generated from XML using XSLT and the LaTeX system is used to create printable file such as DVI, PostScript, and PDF.

Table 12.8.  List of XML tools.

package

popcon

size

keyword

function

docbook-xml

V:36, I:59

2488

xml

This package contains the XML document type definition (DTD) for DocBook.

xsltproc

V:4, I:57

180

xslt

XSLT command line processor. (XML-> XML, HTML, plain text, etc.)

docbook-xsl

V:0.7, I:6

11064

xml/xslt

This contains XSL stylesheets for processing DocBook XML to various output formats with XSLT.

xmlto

V:0.2, I:2

232

xml/xslt

XML-to-any converter with XSLT.

dblatex

V:0.03, I:0.3

5284

xml/xslt

This converts Docbook files to DVI, PostScript, PDF documents with XSLT.


Since XML is subset of Standard Generalized Markup Language (SGML), it can be processed by the extensive tools available for SGML, such as Document Style Semantics and Specification Language (DSSSL).

Table 12.9.  List of DSSL tools.

package

popcon

size

keyword

function

openjade

V:0.5, I:4

1212

dsssl

Implementation of the DSSSL language based on James Clark's Jade software.

jade

V:0.7, I:3

1056

dsssl

James lark's DSSSL language.

docbook-dsssl

V:1.0, I:6

3100

xml/dsssl

This contains DSSSL stylesheets for processing DocBook XML to various output formats with DSSSL.

docbook-utils

V:0.3, I:2

440

xml/dsssl

The utilities for Docbook files including conversion to other formats (HTML, RTF, PS, man, PDF) with docbook2* commands with DSSSL.

sgml2x

V:0.01, I:0.12

216

SGML/dsssl

The converter from SGML and XML using DSSSL stylesheets.


12.2.3. The XML data extraction

You can extract HTML or XML data from other formats using followings:

Table 12.10.  List of XML data extraction tools.

package

popcon

size

keyword

function

wv

V:1.5, I:4

2136

MSWord->any

The document converter from Microsoft Word to HTML, LaTeX, etc..

texi2html

V:0.5, I:3

1752

texi->html

The converter from Texinfo to HTML.

man2html

V:0.3, I:2

364

manpage->html

The converter from manpage to HTML. (CGI support)

tex4ht

V:0.2, I:1.8

936

tex<->html

The converter between (La)TeX and HTML.

xlhtml

V:0.6, I:1.6

184

MSExcel->html

The converter from !MSExcel .xls to HTML.

ppthtml

V:0.5, I:1.6

120

MSPowerPoint->html

The converter from !MSPowerPoint to HTML.

unrtf

V:0.4, I:1.1

276

rtf->html

The document converter from RTF to HTML, etc..

info2www

V:0.3, I:1.5

156

info->html

The converter from GNU info to HTML. (CGI support)

ooo2dbk

V:0.04, I:0.2

941

sxw->xml

The converter from OpenOffice.org SXW documents to DocBook XML.

wp2x

V:0.03, I:0.13

240

WordPerfect->any

WordPerfect 5.0 and 5.1 files to TeX, LaTeX, troff, GML and HTML.

doclifter

V:0.00, I:0.06

420

troff->xml

The converter from troff to DocBook XML.


For non-XML HTML files, you can convert them to XHTML which is an instance of well formed XML and can be processed by XML tools.

Table 12.11.  List of XML pretty print tools.

package

popcon

size

keyword

function

libxml2-utils

V:5, I:55

116

xml<->html<->xhtml

The command line XML tool with "xmllint" command. (syntax check, reformat, lint, ...)

tidy

V:2, I:19

100

xml<->html<->xhtml

HTML syntax checker and reformatter.


Once proper XML is generated, you can use XSLT technology to extract data based on the mark-up context etc.

12.3. Printable data

12.3.1. The Ghostscript

The core of printable data manipulation is the Ghostscript PostScript interpreter. CUPS uses the Ghostscript as its backend.

The latest upstream Ghostscript from Artifex was re-licensed from AFPL to GPL and merged all the latest ESP version changes such as CUPS related ones at 8.60 release as unified release.

Table 12.12.  List of Ghostscript PostScript interpreters.

package

popcon

size

keyword

description

ghostscript

V:13, I:20

3528

ps, pdf

GPL unified version - lenny: recommended

gs-esp

V:13, I:59

72

ps, pdf

GPL ESP version - etch: recommended for use with CUPS


12.3.2. Merge two PS or PDF files

You can merge two PS or PDF files using the gs(1) command of the Ghostscript.

$ gs -q -dNOPAUSE -dBATCH -sDEVICE=pswrite -sOutputFile=bla.ps -f foo1.ps foo2.ps
$ gs -q -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=bla.pdf -f foo1.pdf foo2.pdf
[Note] Note

The Portable Document Format (PDF), which is widely used cross-platform printable data format, is essentially the compressed PS format with few additional features and extensions.

[Tip] Tip

From command line, psmerge(1) and other commands from the psutils package are useful for manipulating PostScript documents. Commands in the pdfjam package work similarly for manipulating PDF documents. pdftk(1) from the pdftk package is useful for manipulating PDF documents, too.

12.3.3. Printable data utilities

The following packages for the printable data utilities caught my eyes:

Table 12.13.  List of printable data utilities.

package

popcon

size

keyword

function

poppler-utils

V:13, I:52

448

pdf->ps,text,...

PDF utilities. (pdftops, pdfinfo, pdfimages, pdftotext, and pdffonts)

psutils

V:4, I:27

408

ps->ps

PostScript document conversion tools

poster

V:3, I:20

80

ps->ps

Create large posters out of PostScript pages.

xpdf-utils

V:3, I:12

4612

pdf->ps,text,...

PDF utilities. (pdftops, pdfinfo, pdfimages, pdftotext, and pdffonts)

enscript

V:3, I:24

2424

text->ps, html, rtf

Converts ASCII text to Postscript, HTML, RTF or Pretty-Print.

a2ps

V:1.9, I:7

4288

text->ps

'Anything to PostScript' converter and pretty-printer.

pdftk

V:0.7, I:4

3292

pdf->pdf

PDF document conversion tool: (pdftk)

mpage

V:0.3, I:2

224

text,ps->ps

Print multiple pages per sheet.

html2ps

V:0.3, I:2

260

html->ps

The converter from HTML to PostScript.

pdfjam

V:0.3, I:1.8

112

pdf->pdf

PDF document conversion tools: pdf90, pdfjoin, and pdfnup

gnuhtml2latex

V:0.16, I:1.2

24

html->latex

The converter from html to latex.

latex2rtf

V:0.16, I:1.1

544

latex->rtf

This converts documents from LaTeX to RTF which can be read by MS Word.

ps2eps

V:0.6, I:4

116

ps->eps

The converter from PostScript to EPS (Encapsulated PostScript).

e2ps

V:0.02, I:0.2

188

text->ps

Text to PostScript converter with Japanese encoding support.

impose+

V:0.04, I:0.2

180

ps->ps

Postscript utilities.

trueprint

V:0.03, I:0.2

188

text->ps

This pretty print many source codes (C, C++, Java, Pascal, Perl, Pike, Sh, and Verilog) to PostScript. (C)

pdf2svg

V:0.02, I:0.19

60

ps->svg

Converter from PDF to Scalable vector graphics format.

pdftoipe

V:0.03, I:0.17

648

ps->ipe

Converter from PDF to IPE's XML format.


12.3.4. Printing with CUPS

Both lp and lpr commands offered by Common Unix Printing System (CUPS) provides options for customized printing the printable data.

For printing 3 copies of a file collated:

$ lp -n 3 -o Collate=True filename

, or

$ lpr -#3 -o Collate=True filename

You can further customize printer operation by using printer option such as "-o number-up=2", "-o page-set=even", "-o page-set=odd", "-o scaling=200", "-o natural-scaling=200", etc., documented at Command-Line Printing and Options.

12.4. Type setting

The Unix troff originally developed by AT&T can be used for simple type setting. It is usually used to create manpages.

TeX created by Donald Knuth is very powerful type setting tool and is the de facto standard . LaTeX originally written by Leslie Lamport enables a high-level access to the power of TeX.

Table 12.14.  List of type setting tools.

package

popcon

size

keyword

function

texlive-base

V:4, I:14

17372

(La)TeX

TeX system for typesetting, previewing and printing.

groff

V:0.8, I:5

5528

troff

GNU troff text-formatting system.


12.4.1. roff typesetting

Traditionally, roff is the main Unix text processing system.

See roff(7), groff(7), groff(1), grotty(1), troff(1), groff_mdoc(7), groff_man(7), groff_ms(7), groff_me(7), groff_mm(7), and info groff.

A good tutorial on -me macros exists. If you have groff (1.18 or newer), find /usr/share/doc/groff/meintro.me.gz and do the following:

$ zcat /usr/share/doc/groff/meintro.me.gz | \
     groff -Tascii -me - | less -R

The following will make a completely plain text file:

$ zcat /usr/share/doc/groff/meintro.me.gz | \
    GROFF_NO_SGR=1 groff -Tascii -me - | col -b -x > meintro.txt

For printing, use PostScript output.

$ groff -Tps meintro.txt | lpr
$ groff -Tps meintro.txt | mpage -2 | lpr

12.4.2. TeX/LaTeX

Preparation:

# aptitude install texlive

References for LaTeX:

  • The teTeX HOWTO: The Linux-teTeX Local Guide (http://www.tldp.org/HOWTO/TeTeX-HOWTO.html)

  • tex(1)

  • latex(1)

  • "The TeXbook", by Donald E. Knuth, (Addison-Wesley)

  • LaTeX - A Document Preparation System, by Leslie Lamport, (Addison-Wesley)

  • The LaTeX Companion, by Goossens, Mittelbach, Samarin, (Addison-Wesley)

This is the most powerful typesetting environment. Many SGML processors use this as their back end text processor. Lyx provided by lyx, lyx-xforms, or lyx-qt and GNU TeXmacs provided by texmacs package offers nice WYSIWYG editing environment for LaTeX while many use Emacs and Vim as the choice for the source editor.

There are many online resources available:

When documents become bigger, sometimes TeX may cause errors. You must increase pool size in /etc/texmf/texmf.cnf (or more appropriately edit /etc/texmf/texmf.d/95NonPath and run update-texmf) to fix this.

[Note] Note

The TeX source of "The TeXbook" is available at ftp://ftp.dante.de/pub/tex/systems/knuth/tex/texbook.tex . This file contains most of the required macros. I heard that you can process this document with tex after commenting lines 7 to 10 and adding "\input manmac \proofmodefalse". It's strongly recommended to buy this book (and all other books from Donald E. Knuth) instead of using the online version but the source is a great example of TeX input!

12.4.3. Pretty print a manual page

The following will print a manual page into a PostScript file/printer.

$ man -Tps some_manpage | lpr
$ man -Tps some_manpage | mpage -2 | lpr

12.4.4. Creating a manual page

Although writing manpage in plain troff is possible, there are few helper packages to create the manpage.

Table 12.15.  List of packages to help creating the manpage.

package

popcon

size

keyword

function

docbook-to-man

V:0.7, I:3

248

SGML->manpage

The converter from DocBook SGML into roff man macros.

help2man

V:0.15, I:1.0

232

text->manpage

Automatic manpage generator from --help.

info2man

V:0.03, I:0.18

204

info->manpage

The converter from GNU info to POD or man pages.

txt2man

V:0.03, I:0.18

88

text->manpage

Converts flat ASCII text to man page format.


12.5. The mail data conversion

The following packages for the mail data conversion caught my eyes:

Table 12.16.  List of packages to help mail data conversion.

package

popcon

size

keyword

function

sharutils

V:7, I:88

976

mail

shar, unshar, uuencode, uudecode

mpack

V:5, I:84

120

mail

The encoder and decoder MIME messages: mpack and munpack.

tnef

V:0.5, I:1.6

156

mail

unpacking MIME attachments of type "application/ms-tnef" which is a Microsoft only format.

uudeview

V:0.2, I:1.7

128

mail

The encoder and decoder for the following formats: uuencode, xxencode, BASE64, quoted printable, and BinHex

mimedecode

V:0.14, I:1.0

76

mail

This decodes transfer encoded text type mime messages.

readpst

V:0.05, I:0.3

228

windows/mail

This converts Outlook PST files to mbox format.


[Tip] Tip

The Internet Message Access Protocol version 4 (IMAP4) server (see: Section 7.1.5.5, “POP3/IMAP4 server”) may be used to move mails out from the proprietary mail system if the mail client software can be configured to use IMAP4 server too.

12.5.1. Mail data basics

Mail (SMTP) data should be limited to 7 bit. So binary data and 8 bit text data are encoded into 7 bit format with the Multipurpose Internet Mail Extensions (MIME) and the selection of the charset (see: Section 9.3.1, “Basics of encoding”).

The standard mail storage format is mbox formatted according to RFC822, RFC2822. See man 5 mbox (provided by the mutt package).

For European languages, "Content-Transfer-Encoding: quoted-printable" with the ISO-8859-1 charset is usually used since there are no much 8 bit characters. If the text is in UTF-8, "Content-Transfer-Encoding: quoted-printable" is also used since it is mostly 7 bit data.

For Japanese, traditionally "Content-Type: text/plain; charset=ISO-2022-JP" should be used to keep text in 7 bits. But mails from older Microsoft systems may use in Shift-JIS without proper declaration. For Japanese, if the text is in UTF-8, it contains many 8 bit data and is encoded into 7 bit data by Base64. The situation of other Asian languages is similar.

[Note] Note

If your non-Unix mail data is accessible by a non-Debian client software which can talk to the IMAP4 server, you may be able to move them out by running your own IMAP4 server (see: Section 7.1.5.5, “POP3/IMAP4 server”).

[Note] Note

If you use other mail storage formats, moving them to mbox format is the good first step. The versatile client program such as mutt may be handy for this.

You can split mailbox contents to each message using procmail(1) and formail(1).

Each mail message can be unpacked using the munpack(1) command from the mpack package (or other specialized tools) to obtain the MIME encoded contents.

12.6. Graphic data tools

The following packages for the graphic data conversion, editing, and organization tools caught my eyes:

Table 12.17.  List of graphic data tools.

package

popcon

size

keyword

function

gimp

V:13, I:56

11720

image(bitmap)

The GNU Image Manipulation Program.

imagemagick

V:14, I:30

4464

image(bitmap)

Image manipulation programs.

graphicsmagick

V:1.2, I:2

3432

image(bitmap)

Image manipulation programs. (folk of imagemagick)

xsane

V:5, I:46

744

image(bitmap)

GTK+-based X11 frontend for SANE (Scanner Access Now Easy).

netpbm

V:4, I:21

4408

image(bitmap)

Graphics conversion tools.

icoutils

V:0.06, I:0.5

196

png<->ico(bitmap)

Converts MS Windows icons and cursors to and from PNG formats (favicon.ico)

xpm2wico

V:0.04, I:0.18

80

xpm->ico(bitmap)

Converts XPM to MS Windows icon formats

openoffice.org-draw

V:21, I:53

8472

image(vector)

OpenOffice.org office suite - drawing

inkscape

V:5, I:13

61376

image(vector)

The SVG (Scalable Vector Graphics) editor.

dia-gnome

V:2, I:7

596

image(vector)

Diagram editor (Gnome)

dia

V:2, I:5

596

image(vector)

Diagram editor (Gtk)

xfig

V:2, I:5

1584

image(vector)

Facility for Interactive Generation of figures under X11

pstoedit

V:1.1, I:8

768

ps/pdf->image(vector)

PostScript and PDF files to editable vector graphics converter. (SVG)

libwmf-bin

V:1.0, I:7

84

Windows/image(vector)

Windows metafile (vector graphic data) conversion tools.

fig2sxd

V:0.07, I:0.4

208

fig->sxd(vector)

Convert XFig files to OpenOffice.org Draw format

unpaper

V:0.18, I:1.2

736

image->image

Post-processing tool for scanned pages for OCR.

tesseract-ocr

V:0.19, I:1.0

2072

image->text

Free OCR software based on the HP's commercial OCR engine.

tesseract-ocr-eng

I:0.3

1760

image->text

OCR engine data: tesseract-ocr language files for English text.

clara

V:0.08, I:0.4

1000

image->text

Free OCR software.

gocr

V:0.9, I:6

1068

image->text

Free OCR software.

ocrad

V:0.9, I:7

364

image->text

Free OCR software.

gtkam

V:0.4, I:2

884

image(Exif)

Manipulates digital camera photo files (GNOME) - GUI

gphoto2

V:0.6, I:3

944

image(Exif)

Manipulates digital camera photo files (GNOME) - command line

kamera

I:20

392

image(Exif)

Manipulates digital camera photo files (KDE)

jhead

V:0.8, I:4

128

image(Exif)

Manipulates the non-image part of Exif compliant JPEG (digital camera photo) files

exif

V:0.2, I:1.8

164

image(Exif)

Command-line utility to show EXIF information in JPEG files

exiftags

V:0.2, I:1.2

240

image(Exif)

Utility to read Exif tags from a digital camera JPEG file

exiftran

V:0.2, I:1.5

92

image(Exif)

Transforms digital camera jpeg images

exifprobe

V:0.04, I:0.2

484

image(Exif)

Reads metadata from digital pictures

dcraw

V:1.2, I:6

408

image(Raw)->ppm

Decodes raw digital camera images

findimagedupes

V:0.09, I:0.5

124

image->fingerprint

Finds visually similar or duplicate images

ale

V:0.03, I:0.2

812

image->image

Merges images to increase fidelity or create mosaics

imageindex

V:0.04, I:0.4

192

image(Exif)->html

Generates static HTML galleries from images

bins

V:0.04, I:0.2

2008

image(Exif)->html

Generates static HTML photo albums using XML and EXIF tags

galrey

V:0.02, I:0.18

116

image(Exif)->html

Generates browsable HTML photo albums with thumbnails

stegdetect

V:0.03, I:0.2

1672

jpeg

Detects and extracts steganography messages inside JPEG

outguess

V:0.02, I:0.18

248

jpeg,png

Universal Steganographic tool


[Tip] Tip

Search more image tools using regex "~Gworks-with::image" in aptitude (see Section 3.1.7.5, “Search method options with aptitude”).

Although GUI programs such as gimp are very powerful, command line tools such as imagemagik are quite useful for automating image manipulation with the script.

The de facto image file format of the digital camera is the Exchangeable Image File Format (EXIF) which is the JPEG image file format with additional metadata tags. It can hold information such as date, time, and camera settings.

The Lempel-Ziv-Welch (LZW) lossless data compression patent has been expired. Graphics Interchange Format (GIF) utilities which use the LZW compression method are now freely available on the Debian system.

[Tip] Tip

Any digital camera or scanner with removable recording media will work with Linux through USB Mass Storage readers.

12.7. Miscellaneous data conversion

There are many other programs for converting data. Following packages caught my eyes using regex "~Guse::converting" in aptitude (see Section 3.1.7.5, “Search method options with aptitude”):

Table 12.18.  List of miscellaneous data conversion tools.

package

popcon

size

keyword

function

alien

V:2, I:14

276

rpm/tgz->deb

The converter for the foreign package into the Debian package.

freepwing

V:0.00, I:0.05

568

EB->EPWING

The converter from "Electric Book" (popular in Japan) to a single JIS X 4081 format (a subset of the EPWING V1).


You can also extract data from RPM format with:

$ rpm2cpio file.src.rpm | cpio --extract