This is the Texinfo-format usage manual for e2compr-0.4.x.

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided also that the sections entitled ``Copying'' and ``GNU General Public License'' are included exactly as in the original, and provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

e2compr/0.4.7
*************

Quickstart
**********

I'd encourage people to read the rest of the manual too, but this page should keep you out of the worst trouble.

Things not to compress
     Don't try to compress swap files, kernel images to be loaded with LILO, or files on which you mount a loop device (with losetup).  All of these subsystems (swap, loop driver, LILO) access the disk blocks straight from the device rather than going through the filesystem. Other bootloaders, such as grub, are almost certainly the same, though I haven't checked.

Programs to avoid
     Don't use a defragmenter other than the special e2compr version of e2defrag.  Don't use dump-like backup programs that read from the device rather than going through the filesystem.  See Other packages.

How do I install e2compr?
     See the INSTALL file that comes with the main e2compr distribution.  Main things to look out for are to check for any .rej files from patch, and that you have an appropriate version of e2fsck.

Remember...
     This manual contains lots more useful information.  If you have any trouble, or are wondering how to do something, then this manual is a good place to start.

Introduction
************

This chapter gives some explanations about the e2compr patch.  It describes what the patch is for and how you can compress your files.

What does e2compr do?
=====================

E2compr is a small patch against the ext2 filesystem that allows on-the-fly compression and decompression.  It compresses only regular files; the administrative data (superblock, inodes, directory files etc.)  are not compressed (mainly for safety reasons).  Access to compressed blocks is provided for read and write operations.  The compression algorithm and cluster size (see Clusters) is specified on a per-file basis.  Directories can also be marked for compression, in which case every newly created file in the directory will be automatically compressed with the same cluster size and the same algorithm that was specified for the directory.

E2compr is not a new filesystem.  It is only a patch to the ext2 filesystem made to support the EXT2_COMPR_FL flag.  It does not require you to make a new partition, and will continue to read or write existing ext2 filesystems.  You should consider it is only a way for the read and write routines to access files that could have been created by a simple utility similar to gzip or compress.  Compressed and uncompressed files will coexist nicely on your ext2 partitions.

Why was e2compr written?
========================

When I (Antoine) got my laptop and installed Linux on it, it was pretty obvious that the hard drive was too small (240MB) to get a complete installation of all the tools I needed, while still having a DOS partition (games for my children) and some space for users.  Buying a new drive would have been quite expensive and not very practical for a laptop.

That's why I immediately looked for tools that could do on-the-fly compression and decompression.  I installed zlibc and tcx, and used them for a while.  Though these are nice tools, there are some limitations.  For instance, you can't write to compressed files through zlibc; and tcx works only for executables.  Of course these are minor problems.  These tools are doing what they were written for. I used them for some months, and one day I reinstalled some packages on my system: I realised that each time I reinstalled something, I had to compress by hand the manuals, the binaries if I wanted to... I found this very painful, and I started to dream about a file system that would do it in a transparent way.  What I was looking for was a system that could provide write access to compressed files, and for which I wouldn't need to compress newly installed files myself.

Someone alerted me to the existence of DouBle by Jean-Marc Verbavatz (ftp://achaz.saclay.cea.fr/pub/double/).  I got it and looked at how it did its work.  There was however one thing that made me feel a little insecure: administrative data is compressed as well.  This means that if your system crashes you may lose much more than a few blocks.  Maybe I am a little over-anxious, but I couldn't bear the idea of my son pressing the power off button while I was working.  I felt this was a major problem for me.  I was only ready to use DouBle instead of zlibc for files I could easily recover.  [pjm: I can confirm that a filesystem compressed with DouBle can run into big problems if the machine is powered off in the middle of a write.  If you lose the first DouBle cluster of the partition then you might as well say goodbye to the whole filesystem.  Unlike e2compr, there wasn't even any error recovery means except to read the source code until you can work out the file format and then use a raw disk editor.]

I then started to think about what I would really like to have on my machine.  The result is the e2compr patch for which you are now reading the documentation.  The ideal tools would meet the following requirements:

 a. automatic compression

 b. automatic decompression

 c. file access in both read and write modes

 d. flexible configuration

 e. secure

From the 3 tools mentioned above, DouBle is probably the best one.  It meets requirements a, b, c and d.  Unfortunately I can't live with the idea that inodes are blindly compressed.  Both tcx and zlibc meet requirements b and e.  I don't consider they are very flexible because there is no way to tune them.  Note also that they will both fail to decompress data on a completely full file system.

E2compr obviously meets requirements a, b, c, d and e.  And that's why I wrote it.

How do I install e2compr?
=========================

Installations instructions are located in the INSTALL file that comes with this e2compr distribution.  Note in particular that you need the e2compr-enabled version of e2fsck.

How do I use e2compr?
=====================

Basically, a file is compressed with the command chattr +c filename, and decompressed with the command chattr -c filename.  You can also do chattr +c (etc.) on a directory.  The directory itself won't be compressed, but any new files created in that directory inherit that directory's compress flag and compression algorithm and cluster size. (See ext2_new_inode() in fs/ext2/ialloc.c.)

Use lsattr to see if a file should be compressed or not, and to see what compression algorithm and cluster size will be used.

Use e2ratio to see how much disk space is used by the compressed form compared to the uncompressed form.

You can also try the -u or -r options to lsattr. (The difference is that, for directories, e2ratio finds the disk usage of the whole directory tree, like du, whereas lsattr -u only shows the space taken by the directory file, like ls -s.)

Examples
--------

Here are a few examples you may want to type to experiment with e2compr. These examples assume that the e2compr versions of lsattr and chattr have been installed. (Actually, this output is from an old version of lsattr, that doesn't show as many flags as the current version.)  $  is the shell prompt.

First we create some temporary directories, and copy some files (from an old e2compr distribution, in this case) to them.  tmp1 will be the reference directory in this example.

     $ mkdir tmp1
     $ cp /sbin/lsattr HOWTO lsattr.c e2compr.patch tmp1
     $ ls -l tmp1
     total 144
     -rw-r--r--   1 root     root        17864 Mar 25 23:20 HOWTO
     -rw-r--r--   1 root     root        67149 Mar 25 23:20 e2compr.patch
     -rwxr-xr-x   1 root     root        51524 Mar 25 23:20 lsattr
     -rw-r--r--   1 root     root         5811 Mar 25 23:20 lsattr.c
     $ du tmp1
     145     tmp1
     $ cp -r tmp1 tmp2

Then we compress some files in directory tmp2:

     $ chattr +c tmp2/*
     $ lsattr tmp2
     --c----  8 lzv1    HOWTO
     --c----  8 lzv1    e2compr.patch
     --c----  8 lzv1    lsattr

     --c----  8 lzv1    lsattr.c
                ^^^^--------------- algorithm
              ^-------------------- cluster size (here 8 blocks per cluster)
       ^--------------------------- file should be compressed

     $ du tmp2
     94      tmp2

Ok, we gained 51 blocks.  We didn't specify a compression algorithm or cluster size to chattr, so it used the system default (which in the above example was <LZV1, 8>).  Now we decompress the file that we just compressed:

     $ chattr -c tmp2/*
     $ lsattr tmp2
     -------  - -       HOWTO
     -------  - -       e2compr.patch
     -------  - -       lsattr
     -------  - -       lsattr.c
     $ du tmp2
     145     tmp2

We can use different cluster sizes with the -b option:

     $ chattr +c -b 16 tmp2/*
     $ lsattr tmp2
     --c---- 16 lzv1    HOWTO
     --c---- 16 lzv1    e2compr.patch
     --c---- 16 lzv1    lsattr
     --c---- 16 lzv1    lsattr.c
     $ du tmp2
     88      tmp2

Of course, lsattr reports the new cluster size.  As you can see, a bigger cluster size seems to compress better (but it is slower, particularly for random access).

Supported cluster sizes are 4, 8, 16 or 32 blocks.  (However, if the filesystem has a blocksize different from the usual 1024 bytes, then there is the additional constraint that the cluster size is less than 32KB.  If you don't understand this parenthesis then it probably doesn't apply to you, but see Block size, if you're interested.)

Now we decompress the files again and compress them with another algorithm, using the -m option (Note: the following example will not work if your kernel was not configured to use LZRW3A.  Also note that if you have an older version of chattr that doesn't support -m (i.e. if chattr complains that it doesn't understand your -m option), then you should use -A in its place.):

     $ chattr -c tmp2/*
     $ chattr +c -b 16 -m lzrw3a tmp2/*
     $ lsattr tmp2
     --c---- 16 lzrw3a  HOWTO
     --c---- 16 lzrw3a  e2compr.patch
     --c---- 16 lzrw3a  lsattr
     --c---- 16 lzrw3a  lsattr.c
     $ du tmp2
     81      tmp2

Well, lzrw3a compresses better.  Now we create another directory and mark it as compressed:

     $ chattr +c -b 16 tmp3
     $ lsattr -d tmp3
     --c---- 16 lzv1    tmp3
     $ cp tmp1/* tmp3
     $ lsattr tmp3
     --c---- 16 lzv1    HOWTO
     --c---- 16 lzv1    e2compr.patch
     --c---- 16 lzv1    lsattr
     --c---- 16 lzv1    lsattr.c

Files in tmp3 have automatically been compressed with the same cluster size and algorithm as tmp3.  Good.

For skeptics:

     $ diff tmp1/HOWTO tmp3/HOWTO
     $ ./tmp3/lsattr tmp3
     --c---- 16 lzv1    HOWTO
     --c---- 16 lzv1    e2compr.patch
     --c---- 16 lzv1    lsattr
     --c---- 16 lzv1    lsattr.c

Compressed files and uncompressed ones hold the same data!  Compressed binaries behave as uncompressed ones.

And now, the ultimate test:

     # cd /usr/src/linux-1.1.76
     # make clean
     # time make depend
     81.66user 50.67system 2:24.37elapsed 91%CPU ...
     # du -s
     8753    .
     # time make
     1006.18user 138.49system 19:37.92elapsed 97%CPU ...
     # du -s
     11842   .

     # chattr +c -R -b 16 -m lzrw3a /usr/src/linux-1.1.76
     # cd /usr/src/linux-1.1.76
     # make clean
     # time make depend
     77.27user 70.95system 2:45.50elapsed 89%CPU ...
     # du -s
     4269    .
     # time make
     1002.23user 173.16system 20:01.90elapsed 97%CPU ...
     # du -s
     6449    .

(pjm: A clean linux 2.1.42 tree with compression method <gzip9, 32> compresses as follows:

     $ e2ratio -s linux-2.1.42
     32720   11491    35.1%  linux-2.1.42

Current limitations
===================

In the current version, you cannot change the cluster size once a file has been compressed.  You must decompress the file first, and then compress it with the new cluster size.

If you change the compression algorithm for a compressed file, already-compressed clusters will not be recompressed with the new algorithm.  If you want to do so, you should decompress the file before compressing it again with the new algorithm.  An exception is made for the `none' algorithm: changing from `none' to some other algorithm results in the kernel attempting to compress all currently-uncompressed clusters.  (Clusters already compressed with some other algorithm are left compressed in that algorithm.)

Inside e2compr
**************

E2compr implementation
======================

This section presents enough information that the reader can predict and understand e2compr behaviour and its implications.  You can happily read this section even if you have no intention of ,looking at or modifying` the code, and it helps your understanding of what's going on.

For more detailed/technical information, see fs/ext2/Readme.e2compr in the kernel source tree.

Disk allocation without e2compr
-------------------------------

For a given file, data is stored in blocks.  The inode holds (directly and possibly indirectly) a list of every block in the file.  When you want to retrieve a byte, the filesystem code determines in which block it is stored, gets the block, and reads the byte from it.  For instance, if we assume that the block size is 1KB, a file whose length is 3897 bytes holds 4 blocks, the first three holding 1024 bytes, and the last one holding 199 bytes (plus 825 trailing null bytes that aren't usually shown to the user).  The file looks like this:

         block#          bytes
         ------          -----

          [0]           0 to 1023
          [1]        1024 to 2047
          [2]        2048 to 3071
          [3]        3072 to 3896

A file can contain holes, i.e., some missing blocks.  In this case the apparent size of the file is much bigger than the product of the number of blocks it really holds and the block size.  A file with holes may look like this:

         block#
         ------

          [0]        data
          [1]        missing block
          [2]        data
          [3]        missing block
          [4]        missing block
          [5]        data
          [6]        data
          [7]        data

A missing block is considered as a block filled with zeros, but does not waste space on the disk.  In the above example, the size of the file is 8096 bytes, while the real space used on the disk is only 5120 bytes.

Clusters
--------

E2compr works by conceptually dividing a file into sections of N blocks (where N is constant for a given file).  We call these sections `clusters', and N the `cluster size'.  E2compr replaces non-hole data in a cluster with blocks of compressed data followed by one or more holes.  Suppose we have a file that looks like this (11 blocks of data), and that the cluster size is 4 blocks:

         block#
         ------

          [0]        data
          [1]        data
          [2]        data
          [3]        data
          [4]        data
          [5]        data
          [6]        data
          [7]        data
          [8]        data
          [9]        data
          [10]       data

If we want to compress the file, we cut the file into clusters and compress every cluster individually:

         block#
         ------

          [0]        data              <- cluster #0
          [1]        data
          [2]        data
          [3]        data

          [4]        data              <- cluster #1
          [5]        data
          [6]        data
          [7]        data

          [8]        data              <- cluster #2
          [9]        data
          [10]       data

Suppose the cluster #0 compressed into 1 block, the cluster #1 into 2 blocks and the cluster #3 into 2 blocks.  We just replace the uncompressed blocks with the compressed one, at the same place, and remove the unneeded blocks.  The file becomes like this:

         block#          bytes
         ------          -----

          [0]        compressed data   <- cluster #0
          [1]        missing block
          [2]        missing block
          [3]        missing block

          [4]        compressed data   <- cluster #1
          [5]        compressed data
          [6]        missing block
          [7]        missing block

          [8]        compressed data   <- cluster #2
          [9]        compressed data
          [10]       missing block

The file now uses only 5 blocks on the disk.

Uncompressed clusters
---------------------

In some cases, a cluster does not compress well.  For instance, a cluster with only one block of data will not compress into less than one block.  (At the time of writing, `fragments' are not supported in the ext2 filesystem, so the minimum allocation grain size is a block.) Sometimes compressed data occupies more bytes than if uncompressed. In such cases we just leave the cluster as it was.  For instance if cluster #1 does not compress well, the compressed file would look like this:

         block#          bytes
         ------          -----

          [0]        compressed data   <- cluster #0
          [1]        missing block
          [2]        missing block
          [3]        missing block

          [4]        data              <- cluster #1
          [5]        data                  (does not compress well)
          [6]        data
          [7]        data

          [8]        compressed data   <- cluster #2
          [9]        compressed data
          [10]       missing block

How does the kernel know which clusters are compressed?
-------------------------------------------------------

Recall that the ext2 filesystem keeps pointers to each of the data blocks in a file.  For holes, a zero is used for this pointer.  However, for compressed clusters, the missing blocks are indicated not with a zero but with the largest representable 32-bit number, hexadecimal FFFFFFFF.

Yes, this does mean that the largest possible ext2 filesystem is now 4KB less than the previous maximum of 16 terabytes (assuming 4KB-blocks).

That's it
---------

Well, that's all.  The big big advantage of this scheme is that there is almost nothing to change in the file system.  We just have to hack a little bit the read and the write routine, in order to compress or decompress the data if it turns out that the block we are working with belongs to a compressed cluster.  This is just a matter of a few lines.  [pjm: I think Antoine's understating things here a little.]

See linux/fs/ext2/Readme.e2compr of the kernel source tree, for a more detailed description and a map of the source code.

e2compr at work
===============

Implementation details, including an introductory text, is included in the file /usr/src/linux/fs/ext2/Readme.e2compr (once you have applied the patch; alternatively, read the file as it appears in the patch).

When does de/compression really occur?
--------------------------------------

In old versions of e2compr, compression would only happen when the file in question had been closed by all processes.

However, compression now also happens whenever a cluster boundary is reached during writing.

Decompression occurs when needed, i.e. every time the kernel wants to read a compressed cluster from disk into memory.  The cluster remains compressed on the disk.  The kernel does cache the file data that it reads from disk, just like any other file (see the cached heading of the output of free).

(At the time of writing, this caching is suboptimal in that all files are given equal weight, whether they're compressed or uncompressed, and whether they're on a fast hard drive or on a slow floppy or CD-ROM.  I understand that things like weighted caching and even caching to swapspace are planned for the 2.3 kernel series.)

What if I try to decompress a file on a full device?
----------------------------------------------------

You will get a `no space left on device' (ENOSPC) error, and the file will keep its compress flag.

(In previous releases, the behaviour was what I consider to be a bit nastier, in that the command would succeed and the compress flag would be cleared, but the file would still be stored in compressed form on disk.)

Compression errors
------------------

[Note: the information below pertains to e2compr-0.3.  Once I've added the appropriate code to e2fsck, I'll probably get rid of the EXT2_ECOMPR_FL flag and just call ext2_error() when corrupted compressed data is found.  Users can use tune2fs -e to tell ext2_error() what to do.]

If the read/write routine in the kernel finds an error when accessing compressed clusters, then the kernel raises the EXT2_ECOMPR_FL flag, which shows in lsattr output as an E where the c would usually be.

The kernel will still allow you to try to read the file (though you may get an I/O error for erroneous clusters), but the kernel won't allow you to open the file with write access while the E flag is set.

You can clear the E flag using chattr -E, but chances are that it will be set again the next time the file is accessed, unless you found the origin of the error and corrected it.

Of course, data are not lost.  e2decompress should be able to be used to retrieve most of the data, with any undecompressible clusters being copied verbatim (i.e. the raw data), with a warning on standard error.  [Newbie note: `on standard error' generally means `to the screen'.]

[In e2compr versions prior to 0.3.5, user programs didn't have read access to raw compressed data, so e2decompress wouldn't work in the way it was supposed to (so it was useless for recovering data) unless one were to reboot to a kernel without e2compr support.]

Compression methods
*******************

Choosing a compression method
=============================

The table below gives some idea of the efficiency of the various algorithm/cluster size pairs.  (The data were collected on a 66MHz 486 with the BIOS set with all the slow options, by the way.)

The information could certainly be improved.  (E.g. what effect does file size have?  What about writing?  (I (pjm) had hoped chattr +c would give this information, but I note that the times for tar and chattr -c do not correlate as well as I'd expected.  I wonder if tar is reading 4KB at a time while chattr -c a cluster at a time?  Is st_blksize set to cluster size for compressed files?  [pjm: No, it's not.  However, I wouldn't have thought it would make much difference, since decompression always happens a cluster at a time, while compression only happens when the file is closed.]))  If you have the curiosity to do some timing of your own, send me the results (together with the script for generating them).  Two hints on timing: (i) Don't use the machine for anything else while timing.  It does make a difference, and not just to elapsed wall-clock time.  (Kill off daemons if you're really keen.)  (ii) Make sure you haven't got any energy-saving things active in your BIOS setup.

     for m in bzip2 gzip9 gzip6 gzip3 lzrw3a lzv1 lzo; do
       for b in 32 16 8 4; do
         for t in 1 2; do
           chattr +X -R /usr/doc; chattr -X -R /usr/doc
           time chattr +c -R -m $m -b $b /usr/doc
           chattr +X -R /usr/doc; chattr -X -R /usr/doc
           time du -s /usr/doc
           time tar cf - /usr/doc | md5sum
           chattr +X -R /usr/doc; chattr -X -R /usr/doc
           time chattr -c -R /usr/doc
           echo done $b $m
         done
         echo
       done
     done

The table below has been formatted, and sorted by compression ratio. (The md5 sums were identical, you'll be happy to know.)

      ch+               du               tar (read)        ch-              cs alg
     ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ ~~ ~~~~~
     0.16u 203.9s 3:51 2257 0.16u 0.84s 0.59u 93.88s 1:49 0.17u 20.09s 0:36 32 gzip9
     0.20u 204.0s 3:50 2257 0.21u 0.81s 0.62u 93.75s 1:49 0.20u 19.43s 0:36 32 gzip9

     0.17u 127.3s 2:33 2260 0.15u 0.80s 0.56u 93.86s 1:48 0.29u 20.36s 0:38 32 gzip6
     0.11u 127.4s 2:35 2260 0.14u 0.81s 0.65u 93.68s 1:48 0.10u 20.23s 0:36 32 gzip6

     0.17u  66.7s 1:31 2400 0.13u 0.76s 0.46u 95.83s 1:50 0.19u 20.31s 0:36 32 gzip3
     0.23u  66.5s 1:32 2400 0.17u 0.69s 0.66u 95.67s 1:50 0.26u 20.06s 0:38 32 gzip3

     0.16u 145.3s 2:52 2406 0.18u 0.64s 0.61u 58.82s 1:13 0.25u 20.24s 0:38 16 gzip9
     0.22u 145.2s 2:53 2406 0.14u 0.80s 0.47u 58.00s 1:13 0.24u 20.26s 0:38 16 gzip9

     0.17u 107.5s 2:14 2409 0.17u 0.72s 0.69u 57.87s 1:13 0.24u 20.10s 0:38 16 gzip6
     0.28u 107.0s 2:12 2409 0.12u 0.82s 0.53u 58.11s 1:13 0.33u 20.07s 0:38 16 gzip6

     0.15u  64.2s 1:29 2525 0.18u 0.78s 0.65u 58.80s 1:14 0.16u 20.40s 0:39 16 gzip3
     0.16u  63.8s 1:29 2525 0.17u 0.85s 0.70u 58.85s 1:15 0.23u 20.49s 0:41 16 gzip3

     0.22u 109.3s 2:22 2681 0.15u 0.80s 0.73u 37.23s 0:54 0.15u 21.65s 0:46 08 gzip9

     0.30u  92.9s 2:06 2682 0.13u 0.78s 0.65u 38.03s 0:56 0.17u 21.38s 0:50 08 gzip6

     0.21u  63.4s 1:35 2757 0.15u 0.79s 0.71u 37.62s 0:55 0.18u 21.51s 0:46 08 gzip3

     0.15u  29.0s 0:53 2898 0.20u 0.80s 0.56u 41.78s 0:58 0.13u 13.61s 0:32 32 lzrw3a
     0.15u  29.0s 0:52 2898 0.17u 0.96s 0.69u 41.64s 0:58 0.19u 13.99s 0:32 32 lzrw3a

     0.22u  93.6s 2:11 2919 0.21u 0.64s 0.76u 25.62s 0:45 0.22u 24.51s 1:04 04 gzip9

     0.17u  87.7s 2:07 2919 0.17u 0.78s 0.72u 25.54s 0:46 0.25u 24.16s 1:03 04 gzip6

     0.17u  68.0s 1:45 2983 0.21u 0.68s 0.67u 25.99s 0:46 0.22u 25.17s 1:04 04 gzip3

     0.16u  29.0s 0:53 3080 0.14u 0.70s 0.56u 28.95s 0:47 0.20u 13.88s 0:34 16 lzrw3a
     0.22u  28.9s 0:53 3080 0.19u 0.74s 0.67u 28.96s 0:47 0.18u 13.75s 0:33 16 lzrw3a

     0.12u  17.6s 0:39 3179 0.17u 0.76s 0.56u 23.81s 0:41 0.18u  8.87s 0:28 32 lzv1
     0.21u  17.5s 0:41 3179 0.14u 0.83s 0.67u 23.82s 0:42 0.19u  8.59s 0:30 32 lzv1

     0.16u  17.9s 0:38 3310 0.11u 0.85s 0.63u 16.75s 0:35 0.15u  9.00s 0:30 16 lzv1
     0.18u  17.7s 0:41 3310 0.15u 0.95s 0.75u 16.75s 0:37 0.21u  8.98s 0:29 16 lzv1

     0.17u  29.9s 1:03 3385 0.20u 0.80s 0.58u 21.87s 0:42 0.18u 14.49s 0:40 08 lzrw3a

     0.19u  18.0s 0:58 3559 0.17u 0.74s 0.66u 12.92s 0:37 0.11u  9.14s 0:40 08 lzv1

     0.19u  32.3s 1:10 3965 0.16u 0.71s 0.64u 17.71s 0:40 0.21u 16.35s 0:58 04 lzrw3a

     0.17u  19.3s 1:00 4025 0.11u 0.74s 0.66u 10.36s 0:34 0.26u  9.33s 0:45 04 lzv1

`none' method
=============

The ``compression algorithm'' in fs/ext2/none.c just returns 0, meaning ``I couldn't compress this cluster to any smaller than it is, so just store it in uncompressed form.''

Changing the method from `none' to some other algorithm (using chattr) makes it apply the new compression algorithm to any uncompressed clusters (and only uncompressed clusters).  (This is a special case for the `none' algorithm.  Comments welcome on the e2compr list.)

This allows you to defer compression until late at night, so that you won't notice the slowdown.

Here's one example of how it could be used.

Do touch /scratch/file; chattr -m none FILES_AND_DIRS. Further writes to these files and within these directories will not use compression.  Then when you want the compression to take place:

     find FILES_AND_DIRS -type f -newer /scratch/file -print0 |
       xargs -0 chattr -m gzip9;
     touch /scratch/file2;
     find FILES_AND_DIRS -type f -newer /scratch/file -print0 |
       xargs -0 chattr -m none;
     mv /scratch/file2 /scratch/file

(Of course /scratch/file, /scratch/file2 and FILES_AND_DIRS should be replaced with whatever's suitable/applicable for you.)

Not a perfect solution, but it's a useful trick.

Future methods?
===============

An LZO algorithm is included in the current 0.4 patches, but is not enabled.  I couldn't get it to work.  The person who sent it me, Chris Old, has a different version working on 2.0 kernels, so I'm sure it won't be hard to get working if you want to have a look.

A bzip-derived algorithm is now included in e2compr, thanks primarily to Julian Seward, who has done a lot of work on making it more suitable to e2compr's needs.  The compression ratio is not as impressive as for the bzip2 program, because e2compr currently uses relatively small cluster sizes (32KB versus the bzip2 program's 900kB).  Nevertheless, for many text files, the bzip algorithm compresses to 10--15% smaller than gzip9.

E2compr interactions with other packages
****************************************

How does e2compr interact with e2fsck?
======================================

e2fsck needs special provision for e2compr so that it understands that 0xffffffff isn't a real block pointer (see zigamorph).

(If you were to try to run the normal e2fsck, it would refuse to run, because it can see in the superblock that the filesystem uses a special extension.)

Can I still use a defragmenter?
===============================

Have a look at http://debs.fuller.edu/e2compr/defrag.html.  From there you can get a copy of e2defrag specially modified to work with (version 0.4.x of) e2compr.

Can I still use e2dump?
=======================

I've made some modifications to e2dump and it may or may not work (I haven't tested it).

In any case, e2dump only works for x86 platforms, on filesystems with 1KB blocks, and was never intended to be high-quality code.

Can I still use zlibc?
======================

Yes, there is no problem.  Beware however about some oddnesses in zlibc.  For instance, when you stat a (zlibc) compressed file, you will probably see it as a regular file.  When you open it for reading, you can get a pipe instead of a truly regular file.  Programs that set inode flags (e.g. chattr) don't like that.

Can I still use DouBle?
=======================

Yes, there is no problem.  But if you mount DouBle over an ext2 file, you probably shouldn't e2compress that file, because bmap() isn't implemented for blocks that fall within e2compressed clusters.  I haven't tried using DouBle over an ext2 file and I don't know whether or not DouBle uses bmap(), but there isn't much point in trying.

e2compr with the loop driver
============================

You can't mount a loop-back device on an ext2-compressed file, i.e.:

     # chattr +c myfile
     # losetup /dev/loop0 myfile

If you want to do this (i.e. create a large block device from a smaller file or block device) then you should look at DouBle, which was built for this very purpose.  (see DouBle reference.)

For those interested: The reason the loop driver won't work on a compressed file is that the loop driver uses bmap, i.e. it asks the filesystem ``Which blocks hold this file?'' and tries to access that block directly.  Since the file data is stored in compressed format, there is no correct answer to that question.  You'll get a couple of syslog messages, like so:

     EXT2-fs warning (device 03:02): ext2_bmap: compressed cluster
     loop: block 1 not present
     end_request: I/O error, dev 07:00, sector 2

FAQ
***

Here are a few questions that were asked by early testers, and that can be of interest for everybody. I didn't know where to put them, and I decided an FAQ list would be a good place... If you have some question/s that you consider worth putting here, then send them to the e2compr mailing list, e2compr@debs.fuller.edu (if possible with answers).  You can be sure that the maintainer will see this, and posting to the e2compr list also allows other e2compr users to make alterations, or provide an answer if you haven't.

Is it possible to compress a swap file?
=======================================

No, don't do this.  At best you will get the error `cannot find swap signature'.  At worst, you will crash your computer.

The main reason that this doesn't work is that the swap routines currently use bmap, which isn't supported for compressed clusters.

I can see that it would be useful if we could get this to work, though we'd have to be careful to avoid deadlocks (e.g. e2compr routines trying to allocate memory, when there isn't any memory until something's been swapped out).  I don't think we can very well guarantee that e2compr won't try to make any memory requests, so it's more a matter of making sure that the memory management code doesn't wait until there's nothing left before trying to use e2compr services.  Thus I think that this feature will have to wait until e2compr is accepted into the standard kernel, since memory management code changes quite frequently.

We'd also have to handle the case where the kernel is trying to write to the swap file but there's not enough disk space to hold the data.

It seems to me that files will be fragmented a lot.
===================================================

For over 99% of files, fragmentation is no worse than in an uncompressed filesystem. (I haven't yet checked any of this.)  As the file is being written, we compress whenever we reach the end of a cluster.  When we start allocating blocks for the next cluster, we try to allocate them right next to the blocks of the previous, compressed, cluster.  (Don't worry, the ext2 block allocation strategy does the right thing concerning holes.)

It is only when you write over a cluster other than the last in the file that compression can cause extra fragmentation.  This is because the new data might compress to a different number of blocks than previously, so you either get a gap in allocation (if the new data takes up fewer blocks) or you get a block that has to be written out of sequence (if the new data takes up more blocks than previously allocated to the cluster).  The only example of this sort of file that I can think of is large (more than one cluster) database files.  (updatedb doesn't count, because (I believe) it gets truncated to zero length before being written over.)

As someone else (kragen@pobox.com) pointed out, compression can actually reduce fragmentation in some cases, simply because we don't fill up the disk as quickly.

Nevertheless, people who are short on disk space (as many e2compr users are) tend to have a high turnover of files (deleting files to make way for new files, which have to be written in the cracks occupied by the files just deleted), which causes high fragmentation. See defragmenter for comments on using a defragmenter.

Can I make a compressed boot/root floppy?
=========================================

No problem, and you can put a lot of things on it (providing you have sufficient inodes; see filesystem full).  The only precaution is, of course, to make sure that you don't compress files that are needed before the kernel is fully loaded and before the ext2 code can be used!!!  You thus shouldn't compress your kernel image (which is probably already compressed) or any file in the /boot directory. [pjm: I'd have thought that System.map and module-info would be safe, but I haven't tested that.]  But you can compress dynamic libraries and binaries without problem.

However, if you're reading that floppy into a RAM disk, and memory consumption isn't too much of a problem, then you will be better off using the compressed ramdisk feature of the kernel instead of e2compr, as you should be able to fit more files in.

What files should(n't) I compress?
==================================

Don't compress swap files (see swap file).

You shouldn't compress kernel images with e2compr, or anything else that you will need to access before a kernel is loaded.  (I guess you could patch lilo, but it really isn't worth it.)  Besides, kernel images are mostly compressed with gzip or bzip anyway.

Don't compress files you use with the loop driver (see loop).

Putting the compressed attribute on /tmp doesn't gain you much, because most files placed in that directory are very short-lived.

I anticipate that sometimes I'll have to start up with a kernel without e2compr (e.g. when upgrading or when I need to boot from an emergency boot floppy).  For this reason I try to keep essential files (mostly in /sbin, /bin, /etc, /lib, /usr/lib/*.so*, /usr/sbin, a few things in /usr/bin, a few other things) uncompressed.

How do I uninstall e2compr?
===========================

Decompress every file with chattr -c, and then restore your kernel and the original copies of e2fsck, lsattr and chattr.  (It shouldn't matter if you don't restore original copies of e2fsck, lsattr and chattr.)

E2compr doesn't seem to compress as well as gzip.
=================================================

First of all, check that the ``best'' (i.e. most compressing) compression algorithm and cluster size are being used.

     $ chattr -c myfile
     $ df myfile    # Just check that we didn't run out of space.
                    # Another, surer way of doing this if we have the e2compr
                    # version of lsattr is `lsattr myfile' and see if an
                    # algorithm name is still displayed.

     Filesystem         1024-blocks  Used Available Capacity Mounted on
     /dev/hda2             149806  124412    17658     88%   /

                    #                        ^^^^^
                    # No, we probably didn't run out of space.
                    # (see device full)

     $ chattr +c -m gzip9 -b 32 myfile

Now take a look at how it compares with straight gzip -9.

       $ e2ratio -l myfile
       1488    721      48.5%  myfile
       $ gzip -9 < myfile > myfile.gz
       $ du -k myfile.gz
       601     /tmp/delme.gz

There is still a difference (721 versus 601 1KB-blocks).  This difference is because e2compr divides a file into ``clusters'' ---sections of a fixed size (in this case 32KB)--- and compresses each of those clusters independently: any similarity between the contents of different clusters will be ignored.  The decision to do things this way was made in order to reduce the amount of time it takes for random access.  (With gzip, there's no way of accessing the last byte of the decompressed stream without decompressing the whole stream.  Other compression methods, more suitable to random access, are conceivable, but designing a good compression method is a serious undertaking.)

Ideas for future e2compr development
************************************

This chapter comprises the former TODO file, somewhat reorganised.

I've tried to split it up so that you can go straight to the bits that you're most suited to.  Within each section, the most interesting items are near the top.

User space work
===============

  * A user-level script run by cron that chooses an algorithm and compression cluster size based on last access time.

  * Patch tune2fs in order to tune dynamically determine the default cluster size or the default algorithm?  Of course, default compression algorithm and cluster size for chattr is currently set on a per-kernel basis rather than a per-filesystem basis

Algorithms
==========

  * I understand that a new release of zlib has been made, which corrects a bug that in rare circumstances results in incorrect compression.  The same bug probably exists in e2compr, so try to integrate this change into e2compr.

  * Make the algorithms into kernel modules.  This would reduce physical memory usage, and may have other advantages, e.g. adding a new algorithm without rebooting.

  * Get LZO working.

  * Build e2decompress with Checker, ElectricFence or something, and try to get it (and hence the kernel, which shares the compression algorithm source code) to handle erroneous data better (i.e. without crashing).

     This isn't all that important because we store a checksum on the compressed data, which should usually protect us from trying to decompress corrupted data.

Kernel
======

  * There are various `todo' and `fixme' comments in the code.  This is an obvious place to start for improving kernel code quality.

  * Allow two de/compression work areas: one for compression, the other for decompression.

  * Once the above works, make the compression work area allocated/deallocated according to demand.  One way of doing this is just to deallocate the work area as soon as we're finished (unless there are other processes lining up to use it) and reallocate when we next need it (which could be almost immediately, unfortunately, e.g. a process creating a large file).  There are other solutions...

     Perhaps the best (of the relatively easy) is as follows: have one work area that is always present, just as at present (0.4.1).  If we get a request for a read while another process is using the main area for writing (i.e. compressing), then allocate a new work area.  In order to avoid allocating/deallocating in quick succession, the area isn't deallocated until the write is finished.  (Alternatively, the new area could become the primary one...  This would be more efficient in some ways (e.g. our contents cache is more useful), but we'd have to use the one allocation method for both areas rather than vmalloc() for the stable one and kmalloc() for the transient one.  I don't know what the advantages of vmalloc() are over kmalloc(), so I don't know how costly this is.)  Note that this `deallocate when compression finishes' scheme provides some help in the `many decompressions in succession' case, but not the `many compressions in succession' case (e.g. process creating large file).  So maybe this idea isn't such a great improvement on that suggested in previous paragraph after all.  Its only advantage is that less memory is spent when the system is compressing but not decompressing.

     Of course, compression in user space would provide an excellent solution.

  * Free preallocated blocks when we fail to decompress a cluster.

  * Support the SYNC flag.  (SYNC is only partially supported in standard 2.0 kernels, so this isn't a high immediate priority; on the other hand, it is better-, or maybe even fully-, supported on 2.1.)

  * Support compress-on-idle.  Have the kernel maintain a queue of inodes to compress.  When a file's data is accessed (read), move it to the end of the queue if it is in the queue.  When we raise EXT2_DIRTY_FL (in ext2_write_file(), ext2_truncate(), ext2_ioctl()), insert inode into queue if it is not there already. Have ext2_cleanup_compressed_inode() remove the inode from the queue (wenn appropriate).

  * Try to reduce fragmentation.  (Fragmentation will always be a problem for files that are frequently written to, but for files that are created and then never changed, you can decompress then recompress using e2compress.

  * When I (Antoine) was thinking about the ideal compressed file system, I imagined we could wait a little more before we really compress file that have been accessed.  Since access to compressed cluster is slower, we could uncompress them and mark the file dirty, but instead of compressing it again when the inode is put, just link it into a special directory that would hold all dirty files.  Files in this directory could be compressed again after a certain amount of time, or when we start to lack free blocks.  This is a feature I liked in tcx.  This is no more than a cache where the uncompressed block would be stored on the disk, and that would persist even after the machine has been stopped.

  * Get rid of EXT2_NOCOMPR_FL (i.e. chattr +X, the attribute flag that provides access to raw compressed data and prevents more than one process to have the file open) and replace it with GNU's O_NOTRANS fcntl flag.  (The advantage of O_NOTRANS is that (i) it's more standard (but only on HURD, not Linux) and (ii) other processes can still have the file open for normal access (though I think that this should be implemented as a per-file-descriptor flag rather than a per-file-opening flag).

     Linux already has mandatory file locking support, so we can use that instead where we need it.

  * Better provision for logfiles, where we'd like to compress all but the last (incomplete) cluster.  (If the last cluster is compressed then we have to uncompress and recompress on every write -- and remember that logfiles are usually sync'ed after every line.)

     This has now been partially provided for with the `none' algorithm: set the algorithm to `none', then every once in a while change it to `gzip' and then immediately back to `none'.  Changing the algorithm from `none' to any other algorithm is a special case: the kernel looks at all clusters that aren't already compressed (which, for log files, should mean only the last cluster or two) and tries to compress them with the new algorithm.

  * Support the EXT2_SECRM_FL (i.e. secure deletion) flag.  (This won't be supported even in the standard 2.2 kernel, so this isn't a high priority.)

  * Add some mount options?  Anything useful come to mind?

  * Make an e2compr kernel module?  The aim would be that people can insmod e2compr into a kernel even if that kernel already has the ext2 fs (without e2compr) built in.  Useful if you don't have a choice of the base kernel, as may be the case when upgrading some Linux distributions.  However, I'm not sure how to implement this. The e2compr patch must modify some core ext2 routines, which, although I (pjm) don't know much about modules, I think is impossible if the ext2 filesystem is already compiled into the kernel not as a module. (Does anyone know?)

  * Support bmap.

     bmap returns the block number on the filesystem where a given <inode, block> pair is stored, information which is presumably going to be used to access the raw device at that point rather than go through the filesystem (and e2compr).

     I don't think we can implement bmap directly: it would require decompressing all clusters that were requested through bmap calls, which is undesirable at best, particularly on a read-only filesystem.

     More sensible might be to look at all the callers of bmap and see if they can be coerced to go through e2compr for files with EXT2_COMPRBLK_FL set.

     One might alternatively think about having a `virtual device' for e2compr clusters.  A virtual device would also help with caching uncompressed clusters, btw.  Antoine's objection to creating a virtual device was that there's no trivial mapping between <inode *, blockno> pairs and a 32-bit block number on the virtual device. We could probably grab the allocation code from any of the filesystems, though I suggest we could benefit by optimising it for sparse occupation of the 32-bit block address space.  I've done some preliminary work for this, but we won't see any real work done on it until the current incarnation of e2compr is working reasonably well.

  * Until the bmap problem is addressed, try to make it impossible to compress a file that's being used as a swapfile.

  * Allow modification of cluster size even for already compressed files.  (Note that this would involve decompressing then recompressing the whole file.)

     Not very important, as this can be done with e2compress.

  * Recompress the whole file when the algorithm is changed?  We certainly wouldn't want this to happen if we change the algorithm to `none'.  This functionality is being added to e2compress, but I don't think that the usual kernel behaviour will change.

e2compress
==========

e2compress will require a lot of work.  It's not available under e2compr-0.4 at all yet.

Hopefully I will do these changes myself.  e2compress is a big program.

  * Put the verification and correction code into a library, so that it can be shared by e2fsck.

  * Make e2compress refuse to work (or call the kernel code) for files that are open by other processes.  (Of particular importance is preventing writes from other processes from being lost.)

     One way of doing this ---the way that the code in progress has been using--- is to use the EXT2_NOCOMPR_FL bit, which can only be raised if no other processes have the file open, and which, while raised, prevents other processes from opening it.

     Another way is to use mandatory file locking (flock() etc.).

  * Have e2compress able to compress or decompress ``in place'', thus reducing problems of multiple links and processes wishing to write to the file.

  * Better safety checks.  Don't just decompress in memory, but read the file as written (thus catching some I/O errors) and warn if the uncompressed data could pass for data of a compressed cluster.

This work was in progress, but, for the 0.3.5 release, I decided just to fix one or two problems in an earlier version of e2compress rather than wait until the more featureful e2compress was ready.

Additions
=========

If you have ideas that are not listed in this chapter, or opinions about how things that are listed should be done, please write to the main e2compr mailing list (e2compr@debs.fuller.edu) or maintainer (pjm@debs.fuller.edu).

Reference
*********

This chapter is intended to be a reference-only chapter (stuff you don't read until you decide to look it up), though a few things (like parts of the program descriptions) are general text.

e2compress
==========

N.B. e2compress is not yet available for e2compr-0.4, and will be one of the last things to be done.  You should probably skip this section.

e2compress and e2decompress are alternatives to using chattr {+|-}c.  chattr lets the kernel do the compression/decompression; e2compress and e2decompress do the compression in user space.

Note that e2compress is not yet available for e2compr-0.4.  Do not try to use 0.3's e2compress under e2compr-0.4, as 0.4 has a different file format.

e2compress overview
-------------------

e2compress compresses files directly into the format readable by the kernel on a second extended file system.  Unlike chattr +c, e2compress does the compression jobs in the user space. This may have some advantages.  e2compress opens the file it is working on, and create a new temporary file (whose name is derived from the original file by adding SUFFIX (~z~ by default)) for which the automatic compression will be turned off.  It then reads from the input file, compresses each cluster in memory, verifies the compression by decompressing to a work area and comparing with the original, and then writes the compressed data to the temporary file, using the lseek() system call to create the necessary holes as it goes.  When it is done, automatic compression is turned on again for the temporary file, and the temporary file is renamed to replace the original file.

e2decompress is to e2compress as gunzip is to gzip: it is the same program invoked under a different name, whose default behaviour is to decompress rather than to compress.

When e2decompress or e2compress -d is called to decompress a file, we create a temporary file, remove the compression attribute, attempt to disable automatic decompression on the input file.

Using e2compress instead of chattr +c has some advantages. The main one is that the resulting file will not be fragmented as it would be with chattr: this comes from the fact that e2compress does not free some blocks as the kernel would do, but instead writes to a new file.  Also, the compression is done in the user space.  Some people may consider this better.  e2compress is also more safe, since the compressed data is immediately checked for decompression.

One disadvantage of using e2compress is that the original file and the temporarily must coexist.  This may be not possible if the file system is completely full.  chattr +c will work even on a completely full file system.

Invoking e2compress
-------------------

Options may be intermixed with file names.  An option applies to all filenames following that option on the command line (or up to an option that contradicts the first); for example, e2compress foo -d bar -c baz attempts to compress foo and baz and to decompress bar.

-S SUFFIX
--suffix=SUFFIX
     Uses the provided suffix to create the temporary file, instead of the default suffix (which is ~z~).

-f
--force
     Force:

       * Write over existing temporary (backup) file.

       * Continue even if the file has more than one link.  (The other links will remain unchanged.)

       * Attempt to decompress even if original isn't flagged as being compressed.

       * Disables verification of compression.

-b SIZE
--cluster-blocks=SIZE
     Set cluster size (specified as a number of filesystem blocks) for compression.

     If not specified, or if specified as 0, then the cluster size is determined by the kernel.  (The kernel policy up to the time of writing ---e2compr 0.3.6--- is that it comes from the cluster size of the containing directory if that has the compression attribute (EXT2_COMPR_FL) raised; otherwise it is the default set during make config or in linux/include/linux/ext2_fs.h at kernel compile time.)

--compress
     Compress the files.  This is the default behaviour (unless the program is invoked under the name e2decompress or e2uncompress).

-d
--decompress
--uncompress
     Decompress the following files.  (This is the default behaviour if the program is invoked as e2decompress or e2uncompress.)

-c
--to-stdout
     Do not change the file; write to standard output.  (At present this works only for decompression.)

-m METHOD
--method=METHOD
--algorithm=METHOD
     Set the compression algorithm for compression.

     If not specified, or if specified as kernel, then the compression method is determined by the kernel.  (The kernel policy up to the time of writing ---e2compr 0.3.6--- is that it comes from the compression method of the containing directory if that has the compression attribute (EXT2_COMPR_FL) raised; otherwise it is the default set during make config or in linux/include/linux/ext2_fs.h at kernel compile time.)

-s METHOD
--subsequent-method=METHOD
     Set the compression algorithm for subsequent writes.  e2compress -mgzip9 -snone myfile is the same as e2compress -mgzip9 myfile && chattr -m none myfile.

     If not specified, or if specified as same, then no chattr-like operations is attempted (so subsequent writes use the same compression algorithm as was specified with -m).

     This option only applies to files that are being compressed.

--believe-bitmap
     When decompressing, trust the cluster bitmap even if a cluster otherwise appears to be a valid compressed cluster.

     Usually when decompressing, if we find a valid compressed cluster but the cluster bitmap says that the cluster isn't compressed, a warning is given but the cluster is treated as compressed (and the decompressed data is copied to output).  At present this is reasonable default behaviour because e2compr isn't yet standard, and many tools don't know about cluster bitmaps.  Thus it's usually more likely that the cluster bitmap was lost by such a tool than that random data just happened to form a valid compressed cluster.

     Use this option if you are confident that the bitmap is in fact correct. (You will still get a warning that the cluster was otherwise valid.)

-h
--help
     Show a usage message and exit without processing any files.

-v
--verbose
     Verbose operation.  (Display cluster statistics.)

--version
     Display the version number and exit without processing any files.

e2compress bugs and limitations
-------------------------------

This program still isn't finished --- but it's now ready enough that the author uses it to compress directory trees (find DIRS -type f -links 1 -print0|xargs -0 nice e2compress -mgzip9 -b32).

The precise behaviour of this program in certain circumstances has changed. This will continue to occur until it is considered ``finished''.

Any writes to the file while e2compress is in action (that is to say after e2compress has read the file but before the file is replaced with the compressed version) are lost.  (For decompressing, no writes are possible because the EXT2_NOCOMPR_FL flag is set on the input file, which prevents any other processes from having the file open.)  In both cases (compressing and decompressing) the results of ,any chmod, chown or touch operation during this period` will be lost.

Make sure you're subscribed to the e2compr-announce mailing list (see mailing lists) to receive news of updates.

e2bitmap
========

e2bitmap is a quick hack I (pjm) wrote to satisfy my curiosity about what sort of cluster bitmaps most files have.

I include it in the main distribution because someone (Stefan Monnier) once asked for a program to do this.

The e2bitmap program tells you which clusters of a file are compressed, and which are uncompressed.  The bitmap is represented as a list of ones and zeros, where 1 means a compressed cluster and 0 an uncompressed cluster, and the first cluster of the file is represented by the left-most digit.

If there are no compressed clusters then the word uncompressed is displayed instead of a list of digits. (More accurately, the test is based on whether or not the EXT2_COMPRBLK_FL flag is set for the file.  In some, rare, circumstances, the EXT2_COMPRBLK_FL flag can still be set even when no clusters of that file are compressed.)

Another use for it is in cases where you want ext2fs to do only the online decompression while leaving the compression to a cron job (for instance).  The cron job might use e2bitmap to see which files deserve recompression in order to avoid recompressing unchanged files over and over again.  Use chattr -m none FILES to prevent the kernel from doing the compression right away, and, for example, chattr -m gzip9 -s none FILES in the cron job.

e2ratio
=======

e2ratio overview
----------------

e2ratio is similar to du but displays the compression ratio for the given files as well as (or instead of) the number of blocks used.

e2ratio recursively descends into directories.

e2ratio output
--------------

Normally, e2ratio displays four fields:

 1. the number of blocks the file would use if uncompressed.

 2. the number of blocks the file actually uses (same as the first field if the file weren't compressed).

 3. the compression ratio, i.e., the number of block actually used expressed as a percentage of the original size.

 4. the name of the file.

With the -b option, only the last two fields (ratio and file name) are displayed.

Invoking e2ratio
----------------

-a
     Generate an entry for each file. (The default behaviour is to generate an entry only for directories.)

-b
     Generate brief entries: only show the ratio and file name.

-s
     Only display the grand total for each of the specified directories.

-f
     Display the compression factor instead of the compression ratio, i.e., the original size of the file divided by the actual (allocated) size.  (This is the reciprocal of what is usually displayed, and is expressed as a decimal number rather than a percentage.)

-l
     Generate the usual, four-field entries.

     (This option is provided mainly for compatibility reasons; older versions of e2ratio enabled -b by default.)

e2ratio bugs and limitations
----------------------------

The ratio is computed rather than stored in the inode. This is expensive because it requires reading every cluster head in the file.

Several files may appear more than once if they are hard links.  The total ratio is probably false in this case, as well as the total number of blocks reported with the -l option.

Error messages and strange behaviour encountered with e2compr
=============================================================

I get the error Filesystem full.
--------------------------------

C. Niemi reported receiving an unexpected `filesystem full' error for a floppy disk even though df reported there being about 900KB free on the floppy. The explanation he found was that, because of the compression, it was possible to put many more files than usual on the disk.  As a consequence, there were no more inodes left, even if 900KB data was still available.  (Try df --inodes.)

In such a case, I suggest that you increase the amount of inodes when you create the file system on the floppy.  See the manual pages for mke2fs.

On a related note, device full for e2compr behaviour when the filesystem really is full.

I get the error I/O Error.
--------------------------

Then do a lsattr on the file.  It will probably display E at the place you would have expected c.  This means the file is in a special state where write access is disallowed, the reason being that an error (related to compression) occurred earlier.  The file is flagged so that the chances of more data loss are minimized.

Try checking whether the file appears to be OK.  The kernel marks files with EXT2_ECOMPR_FL if there is anything suspicious about the file, even if it hasn't checked whether or not the file seems to decompress OK.  If you think that the file is correct, then do chattr -E THEFILE.

The e2decompress program is usually able to recover data from files that have compression errors.

When reading, I see strange data in my file.
--------------------------------------------

Then do a lsattr on the file.  It will probably display either an X flag or a * at the place you would have expected c (depending on your version of lsattr).  One way to re-enable automatic decompression is to do chattr -X on the file. (The X flag will be removed from e2compr at some future point.)

If this is not the case then it is probably a bug in e2compr, and should be reported to the e2compr list (e2compr@debs.fuller.edu).  (I (pjm) have already found and corrected two or three such bugs, so there may be more.)

URLs usw. relevant to e2compr.
==============================

E2compr home page
-----------------

The e2compr home page is http://debs.fuller.edu/e2compr/.  There is an Australian mirror at http://netspace.net.au/~reiter/e2compr/.

Old work (up to the point where Peter took over e2compr's maintenance) can be found under Antoine's home page, http://www.etca.fr/Users/Antoine%20de%20Maricourt/.

Contact details for e2compr maintainer
--------------------------------------

The current e2compr maintainer can be contacted at pjm@debs.fuller.edu.

The FSF suggest also giving snail mail address (for people without network access; mostly third-world countries these days, I suppose).  Here it is:

     Peter Moulder
     1/36 Auburn Grove
     Hawthorn East
     Victoria, 3123
     Australia

e2fsprogs URLs
--------------

The primary ftp site for e2fsprogs (i.e. the sources for standard e2fsck, lsattr, ext2lib, etc.) is ftp://tsx-11.mit.edu/pub/linux/packages/ext2fs/.

A relevant web page is http://web.mit.edu/tytso/www/linux/e2fsprogs-release.html.

e2compr mailing lists
---------------------

Two mailing lists exist for e2compr.

It is recommended that all e2compr users subscribe to at least the announcement list, in order to be up-to-date with bug fixes.

At the time of writing, all e2compr-0.4-specific stuff is written only to the plain e2compr list, so in fact you should subscribe to both mailing lists if you're reading this document (which is specific to e2compr-0.4).

The e2compr list is for general discussion for both users and developers, concerning e2compr.  Anyone can post to this list, whether as a new user asking for help, or to alert users and developers of a possible bug, or to suggest a new feature, or to propose an implementation of a feature and ask for comments.

The e2compr-announce list is a lower-volume list that just contains announcements (e.g. `new version available, corrects bugs X and Y').  At present only Peter Moulder (the current maintainer) and the list maintainer can post to this list.

These are BeroList mailing lists. Subscribe to the lists by sending a message with the word "subscribe" in the subject to whatever-the-name-of-the-list@debs.fuller.edu.  Unsubscribe works in the same way. Just use the word "unsubscribe" in the subject line.  There is no -request stuff as there used to be.

Thanks to Christoph Lameter for setting up these lists.  Send any list problems to me (pjm@debs.fuller.edu), though.

Reference for DouBle
--------------------

DouBle, by Jean-Marc Verbavatz, provides a compressed block device. You tell it the name of a block device or (fixed-size) regular file, some compression parameters (compression algorithm, cluster size, storage granularity), and a (conservative) expected compression ratio. It then provides a block device (say /dev/dbl0) that has more blocks than the file or device where the data is actually being stored. If things don't compress as well as you expected, and it runs out of space, then you have a problem.

DouBle is a good choice if you need a block device for something other than putting an ext2 filesystem on.  I would not recommend DouBle for storing a filesystem.

Anyway, a URL for DouBle is ftp://achaz.saclay.cea.fr/pub/double/. There's also a (old, last time I looked) version on sunsite.

See Why, Loop device, Using DouBle with e2compr, for other comments on DouBle.

Glossary
********

This chapter contains explanations for some concepts and terms that some users may not have encountered.

If you are an experienced UNIX user then you probably don't need to read this chapter.

If you're looking for an explanation of an e2compr-specific term, then you should probably look at Implementation.

Block size
==========

In the ext2 filesystem (at least until fragments are implemented), disk space is allocated to a file in units called blocks.

When you create a filesystem (using mkfs or mke2fs), you have the option of specifying the block size that all files on that filesystem will use, which will be either 1024, 2048, or 4096 bytes. (If you don't specify then the blocksize will be 1024 bytes.)

There is very little reason to use a blocksize other than 1024 bytes, by the way.

(I try to use the term `filesystem blocks', because some people might otherwise think that I meant a different unit of storage when I write `block'.  This document always intends `filesystem block' when it uses the noun `block'.)

In e2compr, cluster sizes are specified in terms of number of filesystem blocks rather than number of kilobytes.  If you say chattr -b 4 FILE, and FILE is on a filesystem with 4096-byte blocks, then FILE will use clusters of 16KB (i.e. 4 * 4096 bytes).  When you do lsattr FILE, it will display 4 for the cluster size, which means 4 filesystem blocks (consistent with chattr), which means 16KB on a filesystem with 4096-byte blocks.

Fortunately, most people's filesystems have just 1024-byte blocks, in which case it doesn't matter whether they think in terms of kilobytes or filesystem blocks.

Links
=====

Some programs, including gzip and e2compress, refuse to operate on files with multiple links (unless you specify the --force option or some such).

This section describes what `multiple links' are, some of their properties, why such programs dislike operating on such files, and how to identify a file with multiple links.

See Identify, if you just want to know how to tell from ls -li output whether a file has one or multiple links.

What is a link?
---------------

Let's divide the concept of a (computer) file into its contents and its name in a directory listing.

A given content can have more than one file name that refers to it. When I do ls -li /bin/*zip /bin/*cat I see:

        2391 -rwxr-xr-x   1 root     root        16568 Aug 10  1996 /bin/cat
        2050 -rwxr-xr-x   3 root     root        45056 Aug 20  1996 /bin/gunzip
        2357 lrwxrwxrwx   1 root     root            4 Sep 18  1996 /bin/gzcat -> gzip
        2050 -rwxr-xr-x   3 root     root        45056 Aug 20  1996 /bin/gzip
        2050 -rwxr-xr-x   3 root     root        45056 Aug 20  1996 /bin/zcat

In this example, the file names /bin/gunzip, /bin/gzip and /bin/zcat all refer to the same file content: they all refer to inode 2050, which points to certain disk blocks where the file content is stored.  The inode also stores the permission bits, the file's owner's user id and owner-group id, timestamps, ext2 attributes (including whether or not the file is e2compressed), data length, and other things.

The link count of `3' indicates that there are three names currently in the filesystem that correspond to the same file (or, more accurately, to the same inode).

By coincidence, all three of these links (names) happen to be in the same directory in the above example, but in general they needn't be; they need only be in the same filesystem.

/bin/cat, on the other hand, is a different file: you can see that it has a different length, a different modification date, and you would guess that its data is stored in different blocks on the hard drive.

/bin/gzcat, on the third hand, is a symbolic link. Symbolic links are a different kettle of fish altogether.  In general, when we say `a file with multiple links', we're not including symbolic links, but only those multiple file names that share the same inode number.  The `link count' always excludes symbolic links from the count.

How to locate the other (hard) links to a file.
-----------------------------------------------

Let's say that I notice that the file referenced by the file name quux has more than one link:

     $ ls -li quux
       33059 -rw-r--r--   2 reiter   staff      172340 Jun 30 17:08 quux

To find the other links (names) of a file, you need to know the file's inode number, and in general also its filesystem (or rather the mount point of that filesystem).

You can see the inode number when you give the -i (or --inode) option to ls.  In the above example (ls -li), it's the left-most field, 33059.

To find what filesystem a file belongs to, do df FILE.  The mount-point of this filesystem is the last field, under the heading Mounted on.  So in our example:

     $ df quux
     Filesystem         1024-blocks  Used Available Capacity Mounted on
     /dev/hda2             149806  123118    18952     87%   /home

so the relevant mount point is /home.

Putting it all together, the general command is find MOUNTPOINT -xdev -inum INUM (optionally ---or necessarily, if you don't use GNU find--- followed by -ls, -print or some such).  To continue our above example:

     $ find /home -xdev -inum 33059 -ls
       33059  169 -rw-r--r--   2 reiter   staff      172340 Jun 30 17:08 /home/gonzo/grault
       33059  169 -rw-r--r--   2 reiter   staff      172340 Jun 30 17:08 /home/reiter/corge/quux

How to identify a file with multiple links
------------------------------------------

For a given file, named FILE, you can see by doing ls -l FILE; the number of links is shown in the second column, i.e. after the permission bits and before the file owner's username.

To find all regular files in a given set of directories TOPDIRS (and their subdirectories) that have more than one link, do find TOPDIRS -type f -links +1.  (To find the files that only have a single link, change the -links +1 to -links 1.)

Other useful options to find are -xdev (which stops find from trying to descend other filesystems (especially /proc, which would usually result in a lot of error messages being printed out)); and -fstype ext2 (which is useful in the context find / -fstype ext2 ..., which is a simpler way of writing find `sed -n 's/^[^ ]* \([^ ]*\) ext2 .*/\1/p' /proc/mounts` ..., both of which go through all ext2 filesystems currently mounted).