No.
If you want to pretend it stands for Directory Virtual Storage Host or anything else you can invent go right ahead. I won't issue a fatwah.
Because what makes this backup system distinct is that it writes to spinning media. That reminded me of the whirling dervishes.
At first I rejected this for several reasons but I finally decided that it was just too anti-PC and anti-(so-called)-multi-cultural to resist.
Think it of a fast rotating backup system and take it for a spin.
Not yet. I am not an artist. If you are and have an idea for one I'll be glad to consider it. While I won't pay you for your contribution I would give credit where due.
Dirvish uses cheap disk space to maintain the appearance of multiple copies of source file trees. Traditional backup systems write to tape.
Disk mirroring (RAID-1) and RAID-5 (not really mirroring) are good at protecting you from certain kinds of disk failure. Unfortunately they can't protect the data from human error, OS and hardware induced filesystem corruption, or anything that destroys the computer. For that you need a copy of the data that is isolated from the cause of failure.
Ideally the backup server should be in a different building.
Look at the price of tape drives, robots and blank media. If you are already doing backups to tape, When was the last time a tape or tape drive failed? Compare that with the price and longevity of disk space, case and controllers.
By putting your dirvish server in a separate location from your production servers it becomes an off-site backup.
That said, you may still want to make tapes. With dirvish you can relegate tapes to off-site storage and long-term archive so you will make far fewer of them. And you can make the tapes from your backup server during working hours with no down-time.
Dirvish uses rsync for network transfer. Where an incremental tape backup requires only requires transmitting the changed files, dirvish only requires transmitting the changed parts of the changed files. So while the result is full backups every time, the volume of data sent is even less than incremental tape backups.
In fact the volume of data transferred can be sufficiently low that backups over the internet are feasible.
In some cases the network will actually be faster than the disk subsystems. When that is the case the whole-file parameter will actually improve overall performance at the expense of network load.
In an ideal world you wouldn't need any backups. I don't always know that I need a given file restored on the day it gets trashed. Often it will take several days before I even notice it. Most users are much worse. Someone deletes or modifies a file or deletes a whole directory and it turns out a few weeks later that someone else still needed it. One or two versions just don't cut it.
Dirvish is free. The license is OSL.
I would like to know if dirvish is helping so if you are using it please let me know. Also let me know about any bugs you find or improvements you might come up with. I would be particularly interested in getting statistics on real-world capacity requirements.
The dirvish home is http://www.pegasys.ws/dirvish. Questions, patches, feedback etc can be sent to dirvish@pegasys.ws. At this time there is no mailing list but if you let me know you are using it i can add your email address to a confidential list and notify you of any bug fixes, security issues and upgrades.
A reasonable rotation program should need about one and half to three times the space of the original filesystem. This is very dependant on the rate of change for a given directory tree and the number and age-range of images in the rotation.
Consideration should be given to the nature and probable change rates of a given backup area and the value of older data. Project areas and home directories may have relatively low rates of change but are subject to sudden spikes and their data is important enough to retain for extended periods. Conversely /var has a high change rate but the data is really only valuable for one or two days. So a vault for /home might want thrice the space of the production area to hold images ranging from one to three months or longer while /var may only need fifty percent more space in it's vault and be expired after 2 or three days.
Dirvish can back up almost any filesystem. Only the vaults on the backup server have any specific requirements.
Only regular files will be shared between images. Device nodes, symlinks, directories and other file types will be recreated for each image. This means that there will be a lot more inodes used in a vault than the source filesystem.
The best filesystem type for dirvish would be one that doesn't set a fixed number of inodes at build time. The vaults will also need to be built with a filesystem type that supports hard links.
While a journaling filesystem is a recommended the journals can have an adverse affect on performance. Dirvish and dirvish-expire do a lot of filesystem meta-data changes. This will stress the journal which in some cases is not as well optimized as one would like. Experience has shown that some journaling filesystems perform extremely poorly under dirvish. While no benchmarks have been made the difference in speed has been as high as 10:1. I would not recommend data journaling.
Using a filesystem type that allows resizing makes it much easier to adapt to the real world requirements of each backup set.
If you are building filesystems with a fixed number of inodes such as UFS or ext2 you should create the filesystem with a bytes per inode value that is half or even just a quarter of what you would normally. Because directories will not be shared it may be good to use a smaller block or fragment size to reduce internal fragmentation.
I would also recommend using RAID-5 arrays for the vaults. You are going to have a great deal of important data here and disk-fault tolerance is a good idea. RAID-5 isn't nearly as expensive as mirroring and should be more than fast enough. Logical Volume management is also a good idea for the vaults as you will probably wish to resize some of them over time.
Because dirvish shares unchanged files between images the actual disk space used is considerably less than you might think. For most filesystems only a small percentage of files will change over a period of time.
The dirvish-expire utility will automatically delete old images based on their assigned expiration date. If you execute this regularly (see cron) each vault will soon reach a steady state where it grows very slowly in response to the growth of the clients.
Yes. But you are unlikely to ever come close to the limits.
|
I can remember a version of UFS that had a link count limit
of 1023 but I doubt current versions are so limited. I
haven't checked the commercial UNIXes but an examination of
the 2.4 Linux kernel source shows that link counts are stored in an
unsigned short so in theory would be limited to 65535.
The 2.6 kernel is expected to raise this limit and use
unsigned long (32 bit).
Each filesystem type has it's own limit as shown in the
table.
Performing a quick test I determined that indeed I could
create exactly 32000 hard links of a file on ext2.
What this means is that on most of the filesystems even if you had a file with 100 hard links (busybox perhaps) you could still support over 300 images sharing those links. | ||||||||||||||||||||||||||
In the event you are using a filesystem type that has a risk of hitting the limit you could change hard links on the client to symlinks. None of these filesystems have limits lower than 126.
First make sure you aren't backing up useless files. The exclude patterns can help there.
Look at the dirvish logs. They will show you what is changing. When I first started using dirvish I found web browser caches were a constant source of change. There was no reason to back them up so I added exclude patterns to block them. Spool areas are similar sources of waste.
Set reasonable expire-rules in the configuration files.
Some applications provide choices regarding file layout. A particularly good example is email. Because a changed file is not shared large mbox mail folders that change daily can be responsible for a considerable amount of backup space. Transitioning to maildir format means more small files but those files can be shared across images as long as they remain in the same status and folder. Such applications often have global configuration files in which system administrator can set a desired default that most users will not override. Similarly there can be an advantage to rotating log files more often.
Examine the logs. Some services will create log files in places you don't expect. Adding them to the exclude list is a workaround. Moving them to /var will correct the problem and make your system more robust.
Because dirvish saves space by sharing unchanged files across multiple images changing many files will cause dirvish's disk usage to spike.
Some of the things that can cause this are:
Usually it is the same files that change over and over. Because of this deleting intermediate images may save nearly as much space as deleting old images.
Really old images can be archived to removable media and then deleted.
Sometimes the pressure is transient. A spike may be the result of one image having captured a temporary file such as a web download. In such a case the spike will go away with that one image. If someone recently changed a large number of files that will cause a spike in the disk usage. Such a spike will form a new plateau until the older images are expired.
If your rotation just won't fit examine your exclude lists, expiration rules and consider adding backup space.
Compression is a wonderful thing but one of dirvish's primary goals is transparency. If dirvish compressed files you couldn't do a transparent restore.
Because of the file sharing between images compressing individual images into compressed archives is unlikely to save you space and will break the transparency. Experience has shown that the disk usage of dirvish is vastly less than that of compressed snapshots.
It may be worthwhile to use a filesystem that supports transparent compression such as e2compr for some vaults.
It is worth remembering that more and more applications are storing their data in compressed formats such as jpg, ogg, mpeg and gzip'ed XML (office suites). If the files are already compressed it won't save space storing them on a compressed filesystem or compressing them externally. These compressed format files will also defeat the hardware compression found on tape drives.
Dirvish will output a message to STDERR that it thinks it may have run out of space and will remove the incomplete destination tree. The meta-data including log files will be left to assist with debugging.
First, delete any failed image. If dirvish actually runs out of space and cannot complete a backup image that image should be deleted. It will be missing files and have other problems that would cause successive images to not share correctly.
You may wish to delete some of your images or enlarge your vault filesystem. That is of course your call.
See the section on running out of space
Very little.
Dirvish, dirvish-runall and dirvish-expire will report errors when detected. Running them under cron, even in quiet mode will still cause email notification on error.
Dirvish-expire will use the expire options to manage the rotation of images automatically. That mainly just leaves monitoring the disk space to ensure that you don't run out of room and making archives.
Each image is a complete copy of what existed at the time it was made so all that is needed to restore from an image is to copy the files. It is essential to preserve ownership, permissions and modification time of restored files.
Rsync is a very good way but scp or streaming tar or cpio archives from the backup server will work as well.
It is also possible to do a read-only export of a dirvish vault using a network file system such as NFS or CIFS/SMB. This or network mounting the source directories on the backup server will allow the use of a simple copy command to restore files. It should be remembered that NFS over UDP does have a measurable error rate so exercise caution doing large restores over NFS. The permissions of all the files in a vault are the same as the source location so there is little security risk to doing so. It might however be better not to give users access to this as it will encourage lazy habits.
You first need to identify the file you are looking for. Examine the source tree or one of the dirvish indexes (you instructed dirvish to create indexes, right?). Optimally you want a perl regex pattern that will only find the file you want.
Let us suppose i'm looking for a version of my .muttrc file. I would use the pattern /jw/.muttrc The slashes have no special meaning and the pattern is anchored the at the end so this won't match a .muttrc.orig file. The dirvish-locate command would look like this:
# dirvish-locate home '/jw/.muttrc'
2 matches in 29 images
/e/home/jw/.muttrc
Apr 9 18:38 030427, 030426, 030425, 030424, 030423, 030422, 030421
030420, 030419, 030418, 030417, 030416, 030415, 030414
030413
Mar 26 22:24 030406
Mar 26 22:24 030403, 030330
Mar 15 06:09 030323, 030316
Mar 9 17:26 030309
Jan 14 21:46 030223, 030216, 030209, 030202
Oct 5 18:20 030105, 021103
Oct 5 18:20 021006
Aug 17 20:15 020901
From this we can see a partial history of that file. Now i
don't have to look in every image to find the version i
want. I can either pick a version or look at one image per
version to decide which one i really want to restore.
You can use any utility that will make an archive from a directory. Feel free to use tar, cpio or dump. It is even reasonable to burn CDs or DVDs if your data will fit. The nice thing is that this won't interact with the production systems so you can do this during working hours.
Dirvish supports arbitrary pre and post processing commands on the client and server. This means that you can pause a database during backups or have dirvish create a database dump just prior to backing up the dump directory.
Dirvish can connect to an rsync daemon running on the clients just fine. Specifying the tree parameter with a colon prefix will direct the dirvish to connect to a rsync daemon.