Home | History | Annotate | Download | only in squashfs-tools
      1 	SQUASHFS 4.3 - A squashed read-only filesystem for Linux
      2 
      3 	Copyright 2002-2014 Phillip Lougher <phillip (a] lougher.demon.co.uk>
      4 
      5 	Released under the GPL licence (version 2 or later).
      6 
      7 Welcome to Squashfs version 4.3.  Please read the README-4.3 and CHANGES files
      8 for details of changes.
      9 
     10 Squashfs is a highly compressed read-only filesystem for Linux.
     11 It uses either gzip/xz/lzo/lz4 compression to compress both files, inodes
     12 and directories.  Inodes in the system are very small and all blocks are
     13 packed to minimise data overhead. Block sizes greater than 4K are supported
     14 up to a maximum of 1Mbytes (default block size 128K).
     15 
     16 Squashfs is intended for general read-only filesystem use, for archival
     17 use (i.e. in cases where a .tar.gz file may be used), and in constrained
     18 block device/memory systems (e.g. embedded systems) where low overhead is
     19 needed.
     20 
     21 1. SQUASHFS OVERVIEW
     22 --------------------
     23 
     24 1. Data, inodes and directories are compressed.
     25 
     26 2. Squashfs stores full uid/gids (32 bits), and file creation time.
     27 
     28 3. In theory files up to 2^64 bytes are supported.  In theory filesystems can
     29    be up to 2^64 bytes.
     30 
     31 4. Inode and directory data are highly compacted, and packed on byte
     32    boundaries.  Each compressed inode is on average 8 bytes in length
     33    (the exact length varies on file type, i.e. regular file, directory,
     34    symbolic link, and block/char device inodes have different sizes).
     35 
     36 5. Squashfs can use block sizes up to 1Mbyte (the default size is 128K).
     37    Using 128K blocks achieves greater compression ratios than the normal
     38    4K block size.
     39 
     40 6. File duplicates are detected and removed.
     41 
     42 7. Filesystems can be compressed with gzip, xz (lzma2), lzo or lz4
     43    compression algorithms.
     44 
     45 1.1 Extended attributes (xattrs)
     46 --------------------------------
     47 
     48 Squashfs filesystems now have extended attribute support.  The
     49 extended attribute implementation has the following features:
     50 
     51 1. Layout can store up to 2^48 bytes of compressed xattr data.
     52 2. Number of xattrs per inode unlimited.
     53 3. Total size of xattr data per inode 2^48 bytes of compressed data.
     54 4. Up to 4 Gbytes of data per xattr value.
     55 5. Inline and out-of-line xattr values supported for higher performance
     56    in xattr scanning (listxattr & getxattr), and to allow xattr value
     57    de-duplication.
     58 6. Both whole inode xattr duplicate detection and individual xattr value
     59    duplicate detection supported.  These can obviously nest, file C's
     60    xattrs can be a complete duplicate of file B, and file B's xattrs
     61    can be a partial duplicate of file A.
     62 7. Xattr name prefix types stored, allowing the redundant "user.", "trusted."
     63    etc. characters to be eliminated and more concisely stored.
     64 8. Support for files, directories, symbolic links, device nodes, fifos
     65    and sockets.
     66 
     67 Extended attribute support is in 2.6.35 and later kernels.  Filesystems
     68 with extended attributes can be mounted on 2.6.29 and later kernels, the
     69 extended attributes will be ignored with a warning.
     70 
     71 2. USING SQUASHFS
     72 -----------------
     73 
     74 Squashfs filesystems should be mounted with 'mount' with the filesystem type
     75 'squashfs'.  If the filesystem is on a block device, the filesystem can be
     76 mounted directly, e.g.
     77 
     78 %mount -t squashfs /dev/sda1 /mnt
     79 
     80 Will mount the squashfs filesystem on "/dev/sda1" under the directory "/mnt".
     81 
     82 If the squashfs filesystem has been written to a file, the loopback device
     83 can be used to mount it (loopback support must be in the kernel), e.g.
     84 
     85 %mount -t squashfs image /mnt -o loop
     86 
     87 Will mount the squashfs filesystem in the file "image" under
     88 the directory "/mnt".
     89 
     90 3. MKSQUASHFS
     91 -------------
     92 
     93 3.1 Mksquashfs options and overview
     94 -----------------------------------
     95 
     96 As squashfs is a read-only filesystem, the mksquashfs program must be used to
     97 create populated squashfs filesystems.
     98 
     99 SYNTAX:./mksquashfs source1 source2 ...  dest [options] [-e list of exclude
    100 dirs/files]
    101 
    102 Filesystem build options:
    103 -comp <comp>		select <comp> compression
    104 			Compressors available:
    105 				gzip (default)
    106 				lzo
    107 				lz4
    108 				xz
    109 -b <block_size>		set data block to <block_size>.  Default 128 Kbytes
    110 			Optionally a suffix of K or M can be given to specify
    111 			Kbytes or Mbytes respectively
    112 -no-exports		don't make the filesystem exportable via NFS
    113 -no-sparse		don't detect sparse files
    114 -no-xattrs		don't store extended attributes
    115 -xattrs			store extended attributes (default)
    116 -noI			do not compress inode table
    117 -noD			do not compress data blocks
    118 -noF			do not compress fragment blocks
    119 -noX			do not compress extended attributes
    120 -no-fragments		do not use fragments
    121 -always-use-fragments	use fragment blocks for files larger than block size
    122 -no-duplicates		do not perform duplicate checking
    123 -all-root		make all files owned by root
    124 -force-uid uid		set all file uids to uid
    125 -force-gid gid		set all file gids to gid
    126 -nopad			do not pad filesystem to a multiple of 4K
    127 -keep-as-directory	if one source directory is specified, create a root
    128 			directory containing that directory, rather than the
    129 			contents of the directory
    130 
    131 Filesystem filter options:
    132 -p <pseudo-definition>	Add pseudo file definition
    133 -pf <pseudo-file>	Add list of pseudo file definitions
    134 -sort <sort_file>	sort files according to priorities in <sort_file>.  One
    135 			file or dir with priority per line.  Priority -32768 to
    136 			32767, default priority 0
    137 -ef <exclude_file>	list of exclude dirs/files.  One per line
    138 -wildcards		Allow extended shell wildcards (globbing) to be used in
    139 			exclude dirs/files
    140 -regex			Allow POSIX regular expressions to be used in exclude
    141 			dirs/files
    142 
    143 Filesystem append options:
    144 -noappend		do not append to existing filesystem
    145 -root-becomes <name>	when appending source files/directories, make the
    146 			original root become a subdirectory in the new root
    147 			called <name>, rather than adding the new source items
    148 			to the original root
    149 
    150 Mksquashfs runtime options:
    151 -version		print version, licence and copyright message
    152 -exit-on-error		treat normally ignored errors as fatal
    153 -recover <name>		recover filesystem data using recovery file <name>
    154 -no-recovery		don't generate a recovery file
    155 -info			print files written to filesystem
    156 -no-progress		don't display the progress bar
    157 -progress		display progress bar when using the -info option
    158 -processors <number>	Use <number> processors.  By default will use number of
    159 			processors available
    160 -mem <size>		Use <size> physical memory.  Currently set to 1922M
    161 			Optionally a suffix of K, M or G can be given to specify
    162 			Kbytes, Mbytes or Gbytes respectively
    163 
    164 Miscellaneous options:
    165 -root-owned		alternative name for -all-root
    166 -noInodeCompression	alternative name for -noI
    167 -noDataCompression	alternative name for -noD
    168 -noFragmentCompression	alternative name for -noF
    169 -noXattrCompression	alternative name for -noX
    170 
    171 -Xhelp			print compressor options for selected compressor
    172 
    173 Compressors available and compressor specific options:
    174 	gzip (default)
    175 	  -Xcompression-level <compression-level>
    176 		<compression-level> should be 1 .. 9 (default 9)
    177 	  -Xwindow-size <window-size>
    178 		<window-size> should be 8 .. 15 (default 15)
    179 	  -Xstrategy strategy1,strategy2,...,strategyN
    180 		Compress using strategy1,strategy2,...,strategyN in turn
    181 		and choose the best compression.
    182 		Available strategies: default, filtered, huffman_only,
    183 		run_length_encoded and fixed
    184 	lzo
    185 	  -Xalgorithm <algorithm>
    186 		Where <algorithm> is one of:
    187 			lzo1x_1
    188 			lzo1x_1_11
    189 			lzo1x_1_12
    190 			lzo1x_1_15
    191 			lzo1x_999 (default)
    192 	  -Xcompression-level <compression-level>
    193 		<compression-level> should be 1 .. 9 (default 8)
    194 		Only applies to lzo1x_999 algorithm
    195 	lz4
    196 	  -Xhc
    197 		Compress using LZ4 High Compression
    198 	xz
    199 	  -Xbcj filter1,filter2,...,filterN
    200 		Compress using filter1,filter2,...,filterN in turn
    201 		(in addition to no filter), and choose the best compression.
    202 		Available filters: x86, arm, armthumb, powerpc, sparc, ia64
    203 	  -Xdict-size <dict-size>
    204 		Use <dict-size> as the XZ dictionary size.  The dictionary size
    205 		can be specified as a percentage of the block size, or as an
    206 		absolute value.  The dictionary size must be less than or equal
    207 		to the block size and 8192 bytes or larger.  It must also be
    208 		storable in the xz header as either 2^n or as 2^n+2^(n+1).
    209 		Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K
    210 		etc.
    211 
    212 Source1 source2 ... are the source directories/files containing the
    213 files/directories that will form the squashfs filesystem.  If a single
    214 directory is specified (i.e. mksquashfs source output_fs) the squashfs
    215 filesystem will consist of that directory, with the top-level root
    216 directory corresponding to the source directory.
    217 
    218 If multiple source directories or files are specified, mksquashfs will merge
    219 the specified sources into a single filesystem, with the root directory
    220 containing each of the source files/directories.  The name of each directory
    221 entry will be the basename of the source path.   If more than one source
    222 entry maps to the same name, the conflicts are named xxx_1, xxx_2, etc. where
    223 xxx is the original name.
    224 
    225 To make this clear, take two example directories.  Source directory
    226 "/home/phillip/test" contains  "file1", "file2" and "dir1".
    227 Source directory "goodies" contains "goodies1", "goodies2" and "goodies3".
    228 
    229 usage example 1:
    230 
    231 %mksquashfs /home/phillip/test output_fs
    232 
    233 This will generate a squashfs filesystem with root entries
    234 "file1", "file2" and "dir1".
    235 
    236 example 2:
    237 
    238 %mksquashfs /home/phillip/test goodies output_fs
    239 
    240 This will create a squashfs filesystem with the root containing
    241 entries "test" and "goodies" corresponding to the source
    242 directories "/home/phillip/test" and "goodies".
    243 
    244 example 3:
    245 
    246 %mksquashfs /home/phillip/test goodies test output_fs
    247 
    248 This is the same as the previous example, except a third
    249 source directory "test" has been specified.  This conflicts
    250 with the first directory named "test" and will be renamed "test_1".
    251 
    252 Multiple sources allow filesystems to be generated without needing to
    253 copy all source files into a common directory.  This simplifies creating
    254 filesystems.
    255 
    256 The -keep-as-directory option can be used when only one source directory
    257 is specified, and you wish the root to contain that directory, rather than
    258 the contents of the directory.  For example:
    259 
    260 example 4:
    261 
    262 %mksquashfs /home/phillip/test output_fs -keep-as-directory
    263 
    264 This is the same as example 1, except for -keep-as-directory.
    265 This will generate a root directory containing directory "test",
    266 rather than the "test" directory contents "file1", "file2" and "dir1".
    267 
    268 The Dest argument is the destination where the squashfs filesystem will be
    269 written.  This can either be a conventional file or a block device.  If the file
    270 doesn't exist it will be created, if it does exist and a squashfs
    271 filesystem exists on it, mksquashfs will append.  The -noappend option will
    272 write a new filesystem irrespective of whether an existing filesystem is
    273 present.
    274 
    275 3.2 Changing compression algorithm and compression specific options
    276 -------------------------------------------------------------------
    277 
    278 By default Mksquashfs will compress using the gzip compression
    279 algorithm.  This algorithm offers a good trade-off between compression
    280 ratio, and memory and time taken to decompress.
    281 
    282 Squashfs also supports LZ4, LZO and XZ (LZMA2) compression.  LZO offers worse
    283 compression ratio than gzip, but is faster to decompress.  XZ offers better
    284 compression ratio than gzip, but at the expense of greater memory and time
    285 to decompress (and significantly more time to compress).  LZ4 is similar
    286 to LZO, but, support for it is not yet in the mainline kernel, and so
    287 its usefulness is currently limited to using Squashfs with Mksquashfs/Unsquashfs
    288 as an archival system like tar.
    289 
    290 If you're not building the squashfs-tools and kernel from source, then
    291 the tools and kernel may or may not have been built with support for LZ4, LZO or
    292 XZ compression.  The compression algorithms supported by the build of
    293 Mksquashfs can be found by typing mksquashfs without any arguments.  The
    294 compressors available are displayed at the end of the help message, e.g. 
    295 
    296 Compressors available and compressor specific options:
    297 	gzip (default)
    298 	  -Xcompression-level <compression-level>
    299 		<compression-level> should be 1 .. 9 (default 9)
    300 	  -Xwindow-size <window-size>
    301 		<window-size> should be 8 .. 15 (default 15)
    302 	  -Xstrategy strategy1,strategy2,...,strategyN
    303 		Compress using strategy1,strategy2,...,strategyN in turn
    304 		and choose the best compression.
    305 		Available strategies: default, filtered, huffman_only,
    306 		run_length_encoded and fixed
    307 	lzo
    308 	  -Xalgorithm <algorithm>
    309 		Where <algorithm> is one of:
    310 			lzo1x_1
    311 			lzo1x_1_11
    312 			lzo1x_1_12
    313 			lzo1x_1_15
    314 			lzo1x_999 (default)
    315 	  -Xcompression-level <compression-level>
    316 		<compression-level> should be 1 .. 9 (default 8)
    317 		Only applies to lzo1x_999 algorithm
    318 	lz4
    319 	  -Xhc
    320 		Compress using LZ4 High Compression
    321 	xz
    322 	  -Xbcj filter1,filter2,...,filterN
    323 		Compress using filter1,filter2,...,filterN in turn
    324 		(in addition to no filter), and choose the best compression.
    325 		Available filters: x86, arm, armthumb, powerpc, sparc, ia64
    326 	  -Xdict-size <dict-size>
    327 		Use <dict-size> as the XZ dictionary size.  The dictionary size
    328 		can be specified as a percentage of the block size, or as an
    329 		absolute value.  The dictionary size must be less than or equal
    330 		to the block size and 8192 bytes or larger.  It must also be
    331 		storable in the xz header as either 2^n or as 2^n+2^(n+1).
    332 		Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K
    333 		etc.
    334 
    335 If the compressor offers compression specific options (all the compressors now
    336 have compression specific options except the deprecated lzma1 compressor)
    337 then these options are also displayed (.i.e. in the above XZ is shown with two
    338 compression specific options).  The compression specific options are, obviously,
    339 specific to the compressor in question, and the compressor documentation and
    340 web sites should be consulted to understand their behaviour.  In general
    341 the Mksquashfs compression defaults for each compressor are optimised to
    342 give the best performance for each compressor, where what constitutes
    343 best depends on the compressor.  For gzip/xz best means highest compression,
    344 for LZO/LZ4 best means a tradeoff between compression and (de)-compression
    345 overhead (LZO/LZ4 by definition are intended for weaker processors).
    346 
    347 3.3 Changing global compression defaults used in mksquashfs
    348 -----------------------------------------------------------
    349 
    350 There are a large number of options that can be used to control the 
    351 compression in mksquashfs.  By and large the defaults are the most
    352 optimum settings and should only be changed in exceptional circumstances!
    353 Note, this does not apply to the block size, increasing the block size
    354 from the default of 128Kbytes will increase compression (especially
    355 for the xz compressor) and should increase I/O performance too.  However,
    356 a block size of greater than 128Kbytes may increase latency in certain
    357 cases (where the filesystem contains lots of fragments, and no locality
    358 of reference is observed).  For this reason the block size default is
    359 configured to the less optimal 128Kbytes.  Users should experiment
    360 with 256Kbyte sizes or above.
    361 
    362 The -noI, -noD and -noF options (also -noInodeCompression, -noDataCompression
    363 and -noFragmentCompression) can be used to force mksquashfs to not compress
    364 inodes/directories, data and fragments respectively.  Giving all options
    365 generates an uncompressed filesystem.
    366 
    367 The -no-fragments tells mksquashfs to not generate fragment blocks, and rather
    368 generate a filesystem similar to a Squashfs 1.x filesystem.  It will of course
    369 still be a Squashfs 4.0 filesystem but without fragments, and so it won't be
    370 mountable on a Squashfs 1.x system.
    371 
    372 The -always-use-fragments option tells mksquashfs to always generate
    373 fragments for files irrespective of the file length.  By default only small
    374 files less than the block size are packed into fragment blocks.  The ends of
    375 files which do not fit fully into a block, are NOT by default packed into
    376 fragments.  To illustrate this, a 100K file has an initial 64K block and a 36K
    377 remainder.  This 36K remainder is not packed into a fragment by default.  This
    378 is because to do so leads to a 10 - 20% drop in sequential I/O performance, as a
    379 disk head seek is needed to seek to the initial file data and another disk seek
    380 is need to seek to the fragment block.  Specify this option if you want file
    381 remainders to be packed into fragment blocks.  Doing so may increase the
    382 compression obtained BUT at the expense of I/O speed.
    383 
    384 The -no-duplicates option tells mksquashfs to not check the files being
    385 added to the filesystem for duplicates.  This can result in quicker filesystem
    386 generation and appending although obviously compression will suffer badly if
    387 there is a lot of duplicate files.
    388 
    389 The -b option allows the block size to be selected, both "K" and "M" postfixes
    390 are supported, this can be either 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K or
    391 1M bytes.
    392 
    393 3.4 Specifying the UIDs/GIDs used in the filesystem
    394 ---------------------------------------------------
    395 
    396 By default files in the generated filesystem inherit the UID and GID ownership
    397 of the original file.  However,  mksquashfs provides a number of options which
    398 can be used to override the ownership.
    399 
    400 The options -all-root and -root-owned (both do exactly the same thing) force all
    401 file uids/gids in the generated Squashfs filesystem to be root.  This allows
    402 root owned filesystems to be built without root access on the host machine.
    403 
    404 The "-force-uid uid"  option forces all files in the generated Squashfs
    405 filesystem to be owned by the specified uid.  The uid can be specified either by
    406 name (i.e. "root") or by number.
    407 
    408 The "-force-gid gid" option forces all files in the generated Squashfs
    409 filesystem to be group owned by the specified gid.  The gid can be specified
    410 either by name (i.e. "root") or by number.
    411 
    412 3.5 Excluding files from the filesystem
    413 ---------------------------------------
    414 
    415 The -e and -ef options allow files/directories to be specified which are
    416 excluded from the output filesystem.  The -e option takes the exclude
    417 files/directories from the command line, the -ef option takes the
    418 exlude files/directories from the specified exclude file, one file/directory
    419 per line.
    420 
    421 Two styles of exclude file matching are supported: basic exclude matching, and
    422 extended wildcard matching.  Basic exclude matching is a legacy feature
    423 retained for backwards compatibility with earlier versions of Mksquashfs.
    424 Extended wildcard matching should be used in preference.
    425 
    426 3.5.1 Basic exclude matching
    427 ----------------------------
    428 
    429 Each exclude file is treated as an exact match of a file/directory in
    430 the source directories.  If an exclude file/directory is absolute (i.e.
    431 prefixed with /, ../, or ./) the entry is treated as absolute, however, if an
    432 exclude file/directory is relative, it is treated as being relative to each of
    433 the sources in turn, i.e.
    434 
    435 %mksquashfs /tmp/source1 source2  output_fs -e ex1 /tmp/source1/ex2 out/ex3
    436 
    437 Will generate exclude files /tmp/source1/ex2, /tmp/source1/ex1, source2/ex1,
    438 /tmp/source1/out/ex3 and source2/out/ex3.
    439 
    440 3.5.2 Extended exclude file handling
    441 ------------------------------------
    442 
    443 Extended exclude file matching treats each exclude file as a wildcard or
    444 regex expression.  To enable wildcard matching specify the -wildcards
    445 option, and to enable regex matching specify the -regex option.  In most
    446 cases the -wildcards option should be used rather than -regex because wildcard
    447 matching behaviour is significantly easier to understand!
    448 
    449 In addition to wildcards/regex expressions, exclude files can be "anchored" or
    450 "non-anchored".  An anchored exclude is one which matches from the root of the
    451 directory and nowhere else, a non-anchored exclude matches anywhere.  For
    452 example given the directory hierarchy "a/b/c/a/b", the anchored exclude
    453 "a/b" will match "a/b" at the root of the directory hierarchy, but
    454 it will not match the "/a/b" sub-directory within directory "c", whereas a
    455 non-anchored exclude would.
    456 
    457 A couple of examples should make this clearer.
    458  
    459 Anchored excludes
    460 
    461   1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz'
    462 
    463      Exclude all files matching "*.gz" in the top level directory "test".
    464 
    465   2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*'
    466 
    467      Exclude all files beginning with "example" inside directories called
    468      "Test" or "test", that occur inside any top level directory.
    469 
    470   Using extended wildcards, negative matching is also possible.
    471 
    472   3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz'
    473 
    474      Exclude all files matching "*.gz" in top level directory "test",
    475      except those with "data" in the name.
    476 
    477 Non-anchored excludes
    478 
    479   By default excludes match from the top level directory, but it is
    480   often useful to exclude a file matching anywhere in the source directories.
    481   For this non-anchored excludes can be used, specified by pre-fixing the
    482   exclude with "...".
    483 
    484   Examples:
    485 
    486   1. mksquashfs example image.sqsh -wildcards -e '... *.gz'
    487 
    488      Exclude files matching "*.gz" anywhere in the source directories.
    489      For example this will match "example.gz", "test/example.gz", and
    490      "test/test/example.gz".
    491 
    492   2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz'
    493 
    494      Exclude files matching "*.gz" inside directories called "Test" or
    495      "test" that occur anywhere in the source directories.
    496 
    497   Again, using extended wildcards, negative matching is also possible.
    498 
    499   3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz'
    500 
    501      Exclude all files matching "*.gz" anywhere in the source directories,
    502      except those with "data" in the name.
    503 
    504 3.5.3 Exclude files summary
    505 ---------------------------
    506 
    507 The -e and -ef exclude options are usefully used in archiving the entire
    508 filesystem, where it is wished to avoid archiving /proc, and the filesystem
    509 being generated, i.e.
    510 
    511 %mksquashfs / /tmp/root.sqsh -e proc /tmp/root.sqsh
    512 
    513 Multiple -ef options can be specified on the command line, and the -ef
    514 option can be used in conjuction with the -e option.
    515 
    516 3.6 Appending to squashfs filesystems
    517 -------------------------------------
    518 
    519 Running squashfs with the destination directory containing an existing
    520 filesystem will add the source items to the existing filesystem.  By default,
    521 the source items are added to the existing root directory.
    522 
    523 To make this clear... An existing filesystem "image" contains root entries
    524 "old1", and "old2".  Source directory "/home/phillip/test" contains  "file1",
    525 "file2" and "dir1".
    526 
    527 example 1:
    528 
    529 %mksquashfs /home/phillip/test image
    530 
    531 Will create a new "image" with root entries "old1", "old2", "file1", "file2" and
    532 "dir1"
    533 
    534 example 2:
    535 
    536 %mksquashfs /home/phillip/test image -keep-as-directory
    537 
    538 Will create a new "image" with root entries "old1", "old2", and "test".
    539 As shown in the previous section, for single source directories
    540 '-keep-as-directory' adds the source directory rather than the
    541 contents of the directory.
    542 
    543 example 3:
    544 
    545 %mksquashfs /home/phillip/test image -keep-as-directory -root-becomes
    546 original-root
    547 
    548 Will create a new "image" with root entries "original-root", and "test".  The
    549 '-root-becomes' option specifies that the original root becomes a subdirectory
    550 in the new root, with the specified name.
    551 
    552 The append option with file duplicate detection, means squashfs can be
    553 used as a simple versioning archiving filesystem. A squashfs filesystem can
    554 be created with for example the linux-2.4.19 source.  Appending the linux-2.4.20
    555 source will create a filesystem with the two source trees, but only the
    556 changed files will take extra room, the unchanged files will be detected as
    557 duplicates.
    558 
    559 3.7 Appending recovery file feature
    560 -----------------------------------
    561 
    562 Recovery files are created when appending to existing Squashfs
    563 filesystems.  This allows the original filesystem to be recovered
    564 if Mksquashfs aborts unexpectedly (i.e. power failure).
    565 
    566 The recovery files are called squashfs_recovery_xxx_yyy, where
    567 "xxx" is the name of the filesystem being appended to, and "yyy" is a
    568 number to guarantee filename uniqueness (the PID of the parent Mksquashfs
    569 process).
    570 
    571 Normally if Mksquashfs exits correctly the recovery file is deleted to
    572 avoid cluttering the filesystem.  If Mksquashfs aborts, the "-recover"
    573 option can be used to recover the filesystem, giving the previously
    574 created recovery file as a parameter, i.e.
    575 
    576 mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234
    577 
    578 The writing of the recovery file can be disabled by specifying the
    579 "-no-recovery" option.
    580 
    581 3.8 Pseudo file support
    582 -----------------------
    583 
    584 Mksquashfs supports pseudo files, these allow fake files, directories, character
    585 and block devices to be specified and added to the Squashfs filesystem being
    586 built, rather than requiring them to be present in the source directories.
    587 This, for example, allows device nodes to be added to the filesystem without
    588 requiring root access.
    589 
    590 Mksquashfs 4.1 added support for "dynamic pseudo files" and a modify operation.
    591 Dynamic pseudo files allow files to be dynamically created when Mksquashfs
    592 is run, their contents being the result of running a command or piece of
    593 shell script.  The modifiy operation allows the mode/uid/gid of an existing
    594 file in the source filesystem to be modified.
    595 
    596 Two Mksquashfs options are supported, -p allows one pseudo file to be specified
    597 on the command line, and -pf allows a pseudo file to be specified containing a
    598 list of pseduo definitions, one per line.
    599 
    600 3.8.1. Creating a dynamic file
    601 ------------------------------
    602 
    603 Pseudo definition
    604 
    605 Filename f mode uid gid command
    606 
    607 mode is the octal mode specifier, similar to that expected by chmod.
    608 
    609 uid and gid can be either specified as a decimal number, or by name.
    610 
    611 command can be an executable or a piece of shell script, and it is executed
    612 by running "/bin/sh -c command".   The stdout becomes the contents of
    613 "Filename".
    614 
    615 Examples:
    616 
    617 Running a basic command
    618 -----------------------
    619 
    620 /somedir/dmesg f 444 root root dmesg
    621 
    622 creates a file "/somedir/dmesg" containing the output from dmesg.
    623 
    624 Executing shell script
    625 ----------------------
    626 
    627 RELEASE f 444 root root \
    628 		if [ ! -e /tmp/ver ]; then \
    629 			echo 0 > /tmp/ver; \
    630 		fi; \
    631                 ver=`cat /tmp/ver`; \
    632                 ver=$((ver +1)); \
    633                 echo $ver > /tmp/ver; \
    634                 echo -n `cat /tmp/release`; \
    635                 echo "-dev #"$ver `date` "Build host" `hostname`
    636 
    637 Creates a file RELEASE containing the release name, date, build host, and
    638 an incrementing version number.  The incrementing version is a side-effect
    639 of executing the shell script, and ensures every time Mksquashfs is run a
    640 new version number is used without requiring any other shell scripting.
    641 
    642 The above example also shows that commands can be split across multiple lines
    643 using "\".  Obviously as the script will be presented to the shell as a single
    644 line, a semicolon is need to separate individual shell commands within the
    645 shell script.
    646 
    647 Reading from a device (or fifo/named socket)
    648 --------------------------------------------
    649 
    650 input f 444 root root dd if=/dev/sda1 bs=1024 count=10
    651 
    652 Copies 10K from the device /dev/sda1 into the file input.  Ordinarily Mksquashfs
    653 given a device, fifo, or named socket will place that special file within the
    654 Squashfs filesystem, the above allows input from these special files to be
    655 captured and placed in the Squashfs filesystem.
    656 
    657 3.8.2. Creating a block or character device
    658 -------------------------------------------
    659 
    660 Pseudo definition
    661 
    662 Filename type mode uid gid major minor
    663 
    664 Where type is either
    665 	b - for block devices, and
    666 	c - for character devices
    667 
    668 mode is the octal mode specifier, similar to that expected by chmod.
    669 
    670 uid and gid can be either specified as a decimal number, or by name.
    671 
    672 For example:
    673 
    674 /dev/chr_dev c 666 root root 100 1
    675 /dev/blk_dev b 666 0 0 200 200
    676 
    677 creates a character device "/dev/chr_dev" with major:minor 100:1 and
    678 a block device "/dev/blk_dev" with major:minor 200:200, both with root
    679 uid/gid and a mode of rw-rw-rw.
    680 
    681 3.8.3. Creating a directory
    682 ---------------------------
    683 
    684 Pseudo definition
    685 
    686 Filename d mode uid gid
    687 
    688 mode is the octal mode specifier, similar to that expected by chmod.
    689 
    690 uid and gid can be either specified as a decimal number, or by name.
    691 
    692 For example:
    693 
    694 /pseudo_dir d 666 root root
    695 
    696 creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw.
    697 
    698 3.8.4. Modifying attributes of an existing file
    699 -----------------------------------------------
    700 
    701 Pseudo definition
    702 
    703 Filename m mode uid gid
    704 
    705 mode is the octal mode specifier, similar to that expected by chmod.
    706 
    707 uid and gid can be either specified as a decimal number, or by name.
    708 
    709 For example:
    710 
    711 dmesg m 666 root root
    712 
    713 Changes the attributes of the file "dmesg" in the filesystem to have
    714 root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained
    715 from the source filesystem.
    716 
    717 3.9 Miscellaneous options
    718 -------------------------
    719 
    720 The -info option displays the files/directories as they are compressed and
    721 added to the filesystem.  The original uncompressed size of each file
    722 is printed, along with DUPLICATE if the file is a duplicate of a
    723 file in the filesystem.
    724 
    725 The -nopad option informs mksquashfs to not pad the filesystem to a 4K multiple.
    726 This is performed by default to enable the output filesystem file to be mounted
    727 by loopback, which requires files to be a 4K multiple.  If the filesystem is
    728 being written to a block device, or is to be stored in a bootimage, the extra
    729 pad bytes are not needed.
    730 
    731 4. UNSQUASHFS
    732 -------------
    733 
    734 Unsquashfs allows you to decompress and extract a Squashfs filesystem without
    735 mounting it.  It can extract the entire filesystem, or a specific
    736 file or directory.
    737 
    738 The Unsquashfs usage info is:
    739 
    740 SYNTAX: ./unsquashfs [options] filesystem [directories or files to extract]
    741 	-v[ersion]		print version, licence and copyright information
    742 	-d[est] <pathname>	unsquash to <pathname>, default "squashfs-root"
    743 	-n[o-progress]		don't display the progress bar
    744 	-no[-xattrs]		don't extract xattrs in file system
    745 	-x[attrs]		extract xattrs in file system (default)
    746 	-u[ser-xattrs]		only extract user xattrs in file system.
    747 				Enables extracting xattrs
    748 	-p[rocessors] <number>	use <number> processors.  By default will use
    749 				number of processors available
    750 	-i[nfo]			print files as they are unsquashed
    751 	-li[nfo]		print files as they are unsquashed with file
    752 				attributes (like ls -l output)
    753 	-l[s]			list filesystem, but don't unsquash
    754 	-ll[s]			list filesystem with file attributes (like
    755 				ls -l output), but don't unsquash
    756 	-f[orce]		if file already exists then overwrite
    757 	-s[tat]			display filesystem superblock information
    758 	-e[f] <extract file>	list of directories or files to extract.
    759 				One per line
    760 	-da[ta-queue] <size>	Set data queue to <size> Mbytes.  Default 256
    761 				Mbytes
    762 	-fr[ag-queue] <size>	Set fragment queue to <size> Mbytes.  Default
    763 				256 Mbytes
    764 	-r[egex]		treat extract names as POSIX regular expressions
    765 				rather than use the default shell wildcard
    766 				expansion (globbing)
    767 
    768 Decompressors available:
    769 	gzip
    770 	lzo
    771 	lz4
    772 	xz
    773 
    774 To extract a subset of the filesystem, the filenames or directory
    775 trees that are to be extracted can be specified on the command line.  The
    776 files/directories should be specified using the full path to the
    777 files/directories as they appear within the Squashfs filesystem.  The
    778 files/directories will also be extracted to those positions within the specified
    779 destination directory.
    780 
    781 The extract files can also be given in a file using the "-e[f]" option.
    782 
    783 Similarly to Mksquashfs, wildcard matching is performed on the extract
    784 files.  Wildcard matching is enabled by default.
    785 
    786 Examples:
    787 
    788   1. unsquashfs image.sqsh 'test/*.gz'
    789 
    790      Extract all files matching "*.gz" in the top level directory "test".
    791 
    792   2. unsquashfs image.sqsh '[Tt]est/example*'
    793 
    794      Extract all files beginning with "example" inside top level directories
    795      called "Test" or "test".
    796 
    797   Using extended wildcards, negative matching is also possible.
    798 
    799   3. unsquashfs image.sqsh 'test/!(*data*).gz'
    800 
    801      Extract all files matching "*.gz" in top level directory "test",
    802      except those with "data" in the name.
    803 
    804 
    805 4.1 Unsquashfs options
    806 ----------------------
    807 
    808 The "-ls" option can be used to list the contents of a filesystem without
    809 decompressing the filesystem data itself.  The "-lls" option is similar
    810 but it also displays file attributes (ls -l style output).
    811 
    812 The "-info" option forces Unsquashfs to print each file as it is decompressed.
    813 The -"linfo" is similar but it also displays file attributes.
    814 
    815 The "-dest" option specifies the directory that is used to decompress
    816 the filesystem data.  If this option is not given then the filesystem is
    817 decompressed to the directory "squashfs-root" in the current working directory.
    818 
    819 The "-force" option forces Unsquashfs to output to the destination
    820 directory even if files or directories already exist.  This allows you
    821 to update an existing directory tree, or to Unsquashfs to a partially
    822 filled directory.  Without the "-force" option, Unsquashfs will
    823 refuse to overwrite any existing files, or to create any directories if they
    824 already exist.  This is done to protect data in case of mistakes, and
    825 so the "-force" option should be used with caution.
    826 
    827 The "-stat" option displays filesystem superblock information.  This is
    828 useful to discover the filesystem version, byte ordering, whether it has a NFS
    829 export table, and what options were used to compress the filesystem, etc.
    830 
    831 Unsquashfs can decompress all Squashfs filesystem versions, 1.x, 2.x, 3.x and
    832 4.0 filesystems.
    833 
    834 5. FILESYSTEM LAYOUT
    835 --------------------
    836 
    837 A squashfs filesystem consists of a maximum of nine parts, packed together on a
    838 byte alignment:
    839 
    840 	 ---------------
    841 	|  superblock 	|
    842 	|---------------|
    843 	|  compression  |
    844 	|    options    |
    845 	|---------------|
    846 	|  datablocks   |
    847 	|  & fragments  |
    848 	|---------------|
    849 	|  inode table	|
    850 	|---------------|
    851 	|   directory	|
    852 	|     table     |
    853 	|---------------|
    854 	|   fragment	|
    855 	|    table      |
    856 	|---------------|
    857 	|    export     |
    858 	|    table      |
    859 	|---------------|
    860 	|    uid/gid	|
    861 	|  lookup table	|
    862 	|---------------|
    863 	|     xattr     |
    864 	|     table	|
    865 	 ---------------
    866 
    867 Compressed data blocks are written to the filesystem as files are read from
    868 the source directory, and checked for duplicates.  Once all file data has been
    869 written the completed super-block, compression options, inode, directory,
    870 fragment, export, uid/gid lookup and xattr tables are written.
    871 
    872 5.1 Compression options
    873 -----------------------
    874 
    875 Compressors can optionally support compression specific options (e.g.
    876 dictionary size).  If non-default compression options have been used, then
    877 these are stored here.
    878 
    879 5.2 Inodes
    880 ----------
    881 
    882 Metadata (inodes and directories) are compressed in 8Kbyte blocks.  Each
    883 compressed block is prefixed by a two byte length, the top bit is set if the
    884 block is uncompressed.  A block will be uncompressed if the -noI option is set,
    885 or if the compressed block was larger than the uncompressed block.
    886 
    887 Inodes are packed into the metadata blocks, and are not aligned to block
    888 boundaries, therefore inodes overlap compressed blocks.  Inodes are identified
    889 by a 48-bit number which encodes the location of the compressed metadata block
    890 containing the inode, and the byte offset into that block where the inode is
    891 placed (<block, offset>).
    892 
    893 To maximise compression there are different inodes for each file type
    894 (regular file, directory, device, etc.), the inode contents and length
    895 varying with the type.
    896 
    897 To further maximise compression, two types of regular file inode and
    898 directory inode are defined: inodes optimised for frequently occurring
    899 regular files and directories, and extended types where extra
    900 information has to be stored.
    901 
    902 5.3 Directories
    903 ---------------
    904 
    905 Like inodes, directories are packed into compressed metadata blocks, stored
    906 in a directory table.  Directories are accessed using the start address of
    907 the metablock containing the directory and the offset into the
    908 decompressed block (<block, offset>).
    909 
    910 Directories are organised in a slightly complex way, and are not simply
    911 a list of file names.  The organisation takes advantage of the
    912 fact that (in most cases) the inodes of the files will be in the same
    913 compressed metadata block, and therefore, can share the start block.
    914 Directories are therefore organised in a two level list, a directory
    915 header containing the shared start block value, and a sequence of directory
    916 entries, each of which share the shared start block.  A new directory header
    917 is written once/if the inode start block changes.  The directory
    918 header/directory entry list is repeated as many times as necessary.
    919 
    920 Directories are sorted, and can contain a directory index to speed up
    921 file lookup.  Directory indexes store one entry per metablock, each entry
    922 storing the index/filename mapping to the first directory header
    923 in each metadata block.  Directories are sorted in alphabetical order,
    924 and at lookup the index is scanned linearly looking for the first filename
    925 alphabetically larger than the filename being looked up.  At this point the
    926 location of the metadata block the filename is in has been found.
    927 The general idea of the index is ensure only one metadata block needs to be
    928 decompressed to do a lookup irrespective of the length of the directory.
    929 This scheme has the advantage that it doesn't require extra memory overhead
    930 and doesn't require much extra storage on disk.
    931 
    932 5.4 File data
    933 -------------
    934 
    935 Regular files consist of a sequence of contiguous compressed blocks, and/or a
    936 compressed fragment block (tail-end packed block).   The compressed size
    937 of each datablock is stored in a block list contained within the
    938 file inode.
    939 
    940 To speed up access to datablocks when reading 'large' files (256 Mbytes or
    941 larger), the code implements an index cache that caches the mapping from
    942 block index to datablock location on disk.
    943 
    944 The index cache allows Squashfs to handle large files (up to 1.75 TiB) while
    945 retaining a simple and space-efficient block list on disk.  The cache
    946 is split into slots, caching up to eight 224 GiB files (128 KiB blocks).
    947 Larger files use multiple slots, with 1.75 TiB files using all 8 slots.
    948 The index cache is designed to be memory efficient, and by default uses
    949 16 KiB.
    950 
    951 5.5 Fragment lookup table
    952 -------------------------
    953 
    954 Regular files can contain a fragment index which is mapped to a fragment
    955 location on disk and compressed size using a fragment lookup table.  This
    956 fragment lookup table is itself stored compressed into metadata blocks.
    957 A second index table is used to locate these.  This second index table for
    958 speed of access (and because it is small) is read at mount time and cached
    959 in memory.
    960 
    961 5.6 Uid/gid lookup table
    962 ------------------------
    963 
    964 For space efficiency regular files store uid and gid indexes, which are
    965 converted to 32-bit uids/gids using an id look up table.  This table is
    966 stored compressed into metadata blocks.  A second index table is used to
    967 locate these.  This second index table for speed of access (and because it
    968 is small) is read at mount time and cached in memory.
    969 
    970 5.7 Export table
    971 ----------------
    972 
    973 To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems
    974 can optionally (disabled with the -no-exports Mksquashfs option) contain
    975 an inode number to inode disk location lookup table.  This is required to
    976 enable Squashfs to map inode numbers passed in filehandles to the inode
    977 location on disk, which is necessary when the export code reinstantiates
    978 expired/flushed inodes.
    979 
    980 This table is stored compressed into metadata blocks.  A second index table is
    981 used to locate these.  This second index table for speed of access (and because
    982 it is small) is read at mount time and cached in memory.
    983 
    984 5.8 Xattr table
    985 ---------------
    986 
    987 The xattr table contains extended attributes for each inode.  The xattrs
    988 for each inode are stored in a list, each list entry containing a type,
    989 name and value field.  The type field encodes the xattr prefix
    990 ("user.", "trusted." etc) and it also encodes how the name/value fields
    991 should be interpreted.  Currently the type indicates whether the value
    992 is stored inline (in which case the value field contains the xattr value),
    993 or if it is stored out of line (in which case the value field stores a
    994 reference to where the actual value is stored).  This allows large values
    995 to be stored out of line improving scanning and lookup performance and it
    996 also allows values to be de-duplicated, the value being stored once, and
    997 all other occurences holding an out of line reference to that value.
    998 
    999 The xattr lists are packed into compressed 8K metadata blocks.
   1000 To reduce overhead in inodes, rather than storing the on-disk
   1001 location of the xattr list inside each inode, a 32-bit xattr id
   1002 is stored.  This xattr id is mapped into the location of the xattr
   1003 list using a second xattr id lookup table.
   1004 
   1005 6. AUTHOR INFO
   1006 --------------
   1007 
   1008 Squashfs was written by Phillip Lougher, email phillip (a] lougher.demon.co.uk,
   1009 in Chepstow, Wales, UK.   If you like the program, or have any problems,
   1010 then please email me, as it's nice to get feedback!
   1011