Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 1 | SQUASHFS 4.3 - A squashed read-only filesystem for Linux |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 2 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 3 | Copyright 2002-2014 Phillip Lougher <phillip@lougher.demon.co.uk> |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 4 | |
| 5 | Released under the GPL licence (version 2 or later). |
| 6 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 7 | Welcome to Squashfs version 4.3. Please read the README-4.3 and CHANGES files |
Phillip Lougher | 06034ad | 2014-08-08 21:56:15 +0100 | [diff] [blame] | 8 | for details of changes. |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 9 | |
Phillip Lougher | e9332f5 | 2014-08-08 19:47:20 +0100 | [diff] [blame] | 10 | Squashfs is a highly compressed read-only filesystem for Linux. |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 11 | It uses either gzip/xz/lzo/lz4 compression to compress both files, inodes |
| 12 | and directories. Inodes in the system are very small and all blocks are |
| 13 | packed to minimise data overhead. Block sizes greater than 4K are supported |
| 14 | up to a maximum of 1Mbytes (default block size 128K). |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 15 | |
| 16 | Squashfs is intended for general read-only filesystem use, for archival |
| 17 | use (i.e. in cases where a .tar.gz file may be used), and in constrained |
| 18 | block device/memory systems (e.g. embedded systems) where low overhead is |
| 19 | needed. |
| 20 | |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 21 | 1. SQUASHFS OVERVIEW |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 22 | -------------------- |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 23 | |
| 24 | 1. Data, inodes and directories are compressed. |
| 25 | |
| 26 | 2. Squashfs stores full uid/gids (32 bits), and file creation time. |
| 27 | |
Phillip Lougher | 2bb7423 | 2014-08-08 21:24:28 +0100 | [diff] [blame] | 28 | 3. In theory files up to 2^64 bytes are supported. In theory filesystems can |
| 29 | be up to 2^64 bytes. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 30 | |
| 31 | 4. Inode and directory data are highly compacted, and packed on byte |
| 32 | boundaries. Each compressed inode is on average 8 bytes in length |
| 33 | (the exact length varies on file type, i.e. regular file, directory, |
| 34 | symbolic link, and block/char device inodes have different sizes). |
| 35 | |
Phillip Lougher | 13b3f39 | 2014-08-08 21:46:45 +0100 | [diff] [blame] | 36 | 5. Squashfs can use block sizes up to 1Mbyte (the default size is 128K). |
| 37 | Using 128K blocks achieves greater compression ratios than the normal |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 38 | 4K block size. |
| 39 | |
| 40 | 6. File duplicates are detected and removed. |
| 41 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 42 | 7. Filesystems can be compressed with gzip, xz (lzma2), lzo or lz4 |
| 43 | compression algorithms. |
| 44 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 45 | 1.1 Extended attributes (xattrs) |
| 46 | -------------------------------- |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 47 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 48 | Squashfs filesystems now have extended attribute support. The |
| 49 | extended attribute implementation has the following features: |
| 50 | |
| 51 | 1. Layout can store up to 2^48 bytes of compressed xattr data. |
| 52 | 2. Number of xattrs per inode unlimited. |
| 53 | 3. Total size of xattr data per inode 2^48 bytes of compressed data. |
| 54 | 4. Up to 4 Gbytes of data per xattr value. |
| 55 | 5. Inline and out-of-line xattr values supported for higher performance |
| 56 | in xattr scanning (listxattr & getxattr), and to allow xattr value |
| 57 | de-duplication. |
| 58 | 6. Both whole inode xattr duplicate detection and individual xattr value |
| 59 | duplicate detection supported. These can obviously nest, file C's |
| 60 | xattrs can be a complete duplicate of file B, and file B's xattrs |
| 61 | can be a partial duplicate of file A. |
| 62 | 7. Xattr name prefix types stored, allowing the redundant "user.", "trusted." |
| 63 | etc. characters to be eliminated and more concisely stored. |
| 64 | 8. Support for files, directories, symbolic links, device nodes, fifos |
| 65 | and sockets. |
| 66 | |
| 67 | Extended attribute support is in 2.6.35 and later kernels. Filesystems |
| 68 | with extended attributes can be mounted on 2.6.29 and later kernels, the |
| 69 | extended attributes will be ignored with a warning. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 70 | |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 71 | 2. USING SQUASHFS |
| 72 | ----------------- |
| 73 | |
| 74 | Squashfs filesystems should be mounted with 'mount' with the filesystem type |
| 75 | 'squashfs'. If the filesystem is on a block device, the filesystem can be |
| 76 | mounted directly, e.g. |
| 77 | |
| 78 | %mount -t squashfs /dev/sda1 /mnt |
| 79 | |
| 80 | Will mount the squashfs filesystem on "/dev/sda1" under the directory "/mnt". |
| 81 | |
| 82 | If the squashfs filesystem has been written to a file, the loopback device |
| 83 | can be used to mount it (loopback support must be in the kernel), e.g. |
| 84 | |
| 85 | %mount -t squashfs image /mnt -o loop |
| 86 | |
| 87 | Will mount the squashfs filesystem in the file "image" under |
| 88 | the directory "/mnt". |
| 89 | |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 90 | 3. MKSQUASHFS |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 91 | ------------- |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 92 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 93 | 3.1 Mksquashfs options and overview |
| 94 | ----------------------------------- |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 95 | |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 96 | As squashfs is a read-only filesystem, the mksquashfs program must be used to |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 97 | create populated squashfs filesystems. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 98 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 99 | SYNTAX:./mksquashfs source1 source2 ... dest [options] [-e list of exclude |
Phillip Lougher | c11f006 | 2014-08-08 20:39:26 +0100 | [diff] [blame] | 100 | dirs/files] |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 101 | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 102 | Filesystem build options: |
| 103 | -comp <comp> select <comp> compression |
| 104 | Compressors available: |
| 105 | gzip (default) |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 106 | lzo |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 107 | lz4 |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 108 | xz |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 109 | -b <block_size> set data block to <block_size>. Default 128 Kbytes |
| 110 | Optionally a suffix of K or M can be given to specify |
| 111 | Kbytes or Mbytes respectively |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 112 | -no-exports don't make the filesystem exportable via NFS |
| 113 | -no-sparse don't detect sparse files |
| 114 | -no-xattrs don't store extended attributes |
| 115 | -xattrs store extended attributes (default) |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 116 | -noI do not compress inode table |
| 117 | -noD do not compress data blocks |
| 118 | -noF do not compress fragment blocks |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 119 | -noX do not compress extended attributes |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 120 | -no-fragments do not use fragments |
| 121 | -always-use-fragments use fragment blocks for files larger than block size |
| 122 | -no-duplicates do not perform duplicate checking |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 123 | -all-root make all files owned by root |
| 124 | -force-uid uid set all file uids to uid |
| 125 | -force-gid gid set all file gids to gid |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 126 | -nopad do not pad filesystem to a multiple of 4K |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 127 | -keep-as-directory if one source directory is specified, create a root |
| 128 | directory containing that directory, rather than the |
| 129 | contents of the directory |
| 130 | |
| 131 | Filesystem filter options: |
| 132 | -p <pseudo-definition> Add pseudo file definition |
| 133 | -pf <pseudo-file> Add list of pseudo file definitions |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 134 | -sort <sort_file> sort files according to priorities in <sort_file>. One |
| 135 | file or dir with priority per line. Priority -32768 to |
| 136 | 32767, default priority 0 |
| 137 | -ef <exclude_file> list of exclude dirs/files. One per line |
| 138 | -wildcards Allow extended shell wildcards (globbing) to be used in |
| 139 | exclude dirs/files |
| 140 | -regex Allow POSIX regular expressions to be used in exclude |
| 141 | dirs/files |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 142 | |
| 143 | Filesystem append options: |
| 144 | -noappend do not append to existing filesystem |
| 145 | -root-becomes <name> when appending source files/directories, make the |
| 146 | original root become a subdirectory in the new root |
| 147 | called <name>, rather than adding the new source items |
| 148 | to the original root |
| 149 | |
| 150 | Mksquashfs runtime options: |
| 151 | -version print version, licence and copyright message |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 152 | -exit-on-error treat normally ignored errors as fatal |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 153 | -recover <name> recover filesystem data using recovery file <name> |
| 154 | -no-recovery don't generate a recovery file |
| 155 | -info print files written to filesystem |
| 156 | -no-progress don't display the progress bar |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 157 | -progress display progress bar when using the -info option |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 158 | -processors <number> Use <number> processors. By default will use number of |
| 159 | processors available |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 160 | -mem <size> Use <size> physical memory. Currently set to 1922M |
| 161 | Optionally a suffix of K, M or G can be given to specify |
| 162 | Kbytes, Mbytes or Gbytes respectively |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 163 | |
| 164 | Miscellaneous options: |
| 165 | -root-owned alternative name for -all-root |
| 166 | -noInodeCompression alternative name for -noI |
| 167 | -noDataCompression alternative name for -noD |
| 168 | -noFragmentCompression alternative name for -noF |
| 169 | -noXattrCompression alternative name for -noX |
| 170 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 171 | -Xhelp print compressor options for selected compressor |
| 172 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 173 | Compressors available and compressor specific options: |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 174 | gzip (default) |
| 175 | -Xcompression-level <compression-level> |
| 176 | <compression-level> should be 1 .. 9 (default 9) |
| 177 | -Xwindow-size <window-size> |
| 178 | <window-size> should be 8 .. 15 (default 15) |
| 179 | -Xstrategy strategy1,strategy2,...,strategyN |
| 180 | Compress using strategy1,strategy2,...,strategyN in turn |
| 181 | and choose the best compression. |
| 182 | Available strategies: default, filtered, huffman_only, |
| 183 | run_length_encoded and fixed |
| 184 | lzo |
| 185 | -Xalgorithm <algorithm> |
| 186 | Where <algorithm> is one of: |
| 187 | lzo1x_1 |
| 188 | lzo1x_1_11 |
| 189 | lzo1x_1_12 |
| 190 | lzo1x_1_15 |
| 191 | lzo1x_999 (default) |
| 192 | -Xcompression-level <compression-level> |
| 193 | <compression-level> should be 1 .. 9 (default 8) |
| 194 | Only applies to lzo1x_999 algorithm |
| 195 | lz4 |
| 196 | -Xhc |
| 197 | Compress using LZ4 High Compression |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 198 | xz |
| 199 | -Xbcj filter1,filter2,...,filterN |
| 200 | Compress using filter1,filter2,...,filterN in turn |
| 201 | (in addition to no filter), and choose the best compression. |
| 202 | Available filters: x86, arm, armthumb, powerpc, sparc, ia64 |
| 203 | -Xdict-size <dict-size> |
| 204 | Use <dict-size> as the XZ dictionary size. The dictionary size |
| 205 | can be specified as a percentage of the block size, or as an |
| 206 | absolute value. The dictionary size must be less than or equal |
| 207 | to the block size and 8192 bytes or larger. It must also be |
| 208 | storable in the xz header as either 2^n or as 2^n+2^(n+1). |
| 209 | Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K |
| 210 | etc. |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 211 | |
Phillip Lougher | e162193 | 2014-08-08 05:30:01 +0100 | [diff] [blame] | 212 | Source1 source2 ... are the source directories/files containing the |
| 213 | files/directories that will form the squashfs filesystem. If a single |
| 214 | directory is specified (i.e. mksquashfs source output_fs) the squashfs |
| 215 | filesystem will consist of that directory, with the top-level root |
| 216 | directory corresponding to the source directory. |
| 217 | |
| 218 | If multiple source directories or files are specified, mksquashfs will merge |
| 219 | the specified sources into a single filesystem, with the root directory |
| 220 | containing each of the source files/directories. The name of each directory |
| 221 | entry will be the basename of the source path. If more than one source |
| 222 | entry maps to the same name, the conflicts are named xxx_1, xxx_2, etc. where |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 223 | xxx is the original name. |
Phillip Lougher | e162193 | 2014-08-08 05:30:01 +0100 | [diff] [blame] | 224 | |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 225 | To make this clear, take two example directories. Source directory |
| 226 | "/home/phillip/test" contains "file1", "file2" and "dir1". |
| 227 | Source directory "goodies" contains "goodies1", "goodies2" and "goodies3". |
Phillip Lougher | e162193 | 2014-08-08 05:30:01 +0100 | [diff] [blame] | 228 | |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 229 | usage example 1: |
| 230 | |
| 231 | %mksquashfs /home/phillip/test output_fs |
| 232 | |
| 233 | This will generate a squashfs filesystem with root entries |
| 234 | "file1", "file2" and "dir1". |
| 235 | |
| 236 | example 2: |
| 237 | |
| 238 | %mksquashfs /home/phillip/test goodies output_fs |
| 239 | |
| 240 | This will create a squashfs filesystem with the root containing |
| 241 | entries "test" and "goodies" corresponding to the source |
| 242 | directories "/home/phillip/test" and "goodies". |
| 243 | |
| 244 | example 3: |
| 245 | |
| 246 | %mksquashfs /home/phillip/test goodies test output_fs |
| 247 | |
| 248 | This is the same as the previous example, except a third |
| 249 | source directory "test" has been specified. This conflicts |
| 250 | with the first directory named "test" and will be renamed "test_1". |
Phillip Lougher | e162193 | 2014-08-08 05:30:01 +0100 | [diff] [blame] | 251 | |
| 252 | Multiple sources allow filesystems to be generated without needing to |
| 253 | copy all source files into a common directory. This simplifies creating |
| 254 | filesystems. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 255 | |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 256 | The -keep-as-directory option can be used when only one source directory |
| 257 | is specified, and you wish the root to contain that directory, rather than |
| 258 | the contents of the directory. For example: |
| 259 | |
| 260 | example 4: |
| 261 | |
| 262 | %mksquashfs /home/phillip/test output_fs -keep-as-directory |
| 263 | |
| 264 | This is the same as example 1, except for -keep-as-directory. |
| 265 | This will generate a root directory containing directory "test", |
| 266 | rather than the "test" directory contents "file1", "file2" and "dir1". |
| 267 | |
| 268 | The Dest argument is the destination where the squashfs filesystem will be |
| 269 | written. This can either be a conventional file or a block device. If the file |
| 270 | doesn't exist it will be created, if it does exist and a squashfs |
| 271 | filesystem exists on it, mksquashfs will append. The -noappend option will |
Phillip Lougher | c11f006 | 2014-08-08 20:39:26 +0100 | [diff] [blame] | 272 | write a new filesystem irrespective of whether an existing filesystem is |
| 273 | present. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 274 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 275 | 3.2 Changing compression algorithm and compression specific options |
| 276 | ------------------------------------------------------------------- |
| 277 | |
| 278 | By default Mksquashfs will compress using the gzip compression |
| 279 | algorithm. This algorithm offers a good trade-off between compression |
| 280 | ratio, and memory and time taken to decompress. |
| 281 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 282 | Squashfs also supports LZ4, LZO and XZ (LZMA2) compression. LZO offers worse |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 283 | compression ratio than gzip, but is faster to decompress. XZ offers better |
| 284 | compression ratio than gzip, but at the expense of greater memory and time |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 285 | to decompress (and significantly more time to compress). LZ4 is similar |
| 286 | to LZO, but, support for it is not yet in the mainline kernel, and so |
| 287 | its usefulness is currently limited to using Squashfs with Mksquashfs/Unsquashfs |
| 288 | as an archival system like tar. |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 289 | |
| 290 | If you're not building the squashfs-tools and kernel from source, then |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 291 | the tools and kernel may or may not have been built with support for LZ4, LZO or |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 292 | XZ compression. The compression algorithms supported by the build of |
| 293 | Mksquashfs can be found by typing mksquashfs without any arguments. The |
| 294 | compressors available are displayed at the end of the help message, e.g. |
| 295 | |
| 296 | Compressors available and compressor specific options: |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 297 | gzip (default) |
| 298 | -Xcompression-level <compression-level> |
| 299 | <compression-level> should be 1 .. 9 (default 9) |
| 300 | -Xwindow-size <window-size> |
| 301 | <window-size> should be 8 .. 15 (default 15) |
| 302 | -Xstrategy strategy1,strategy2,...,strategyN |
| 303 | Compress using strategy1,strategy2,...,strategyN in turn |
| 304 | and choose the best compression. |
| 305 | Available strategies: default, filtered, huffman_only, |
| 306 | run_length_encoded and fixed |
| 307 | lzo |
| 308 | -Xalgorithm <algorithm> |
| 309 | Where <algorithm> is one of: |
| 310 | lzo1x_1 |
| 311 | lzo1x_1_11 |
| 312 | lzo1x_1_12 |
| 313 | lzo1x_1_15 |
| 314 | lzo1x_999 (default) |
| 315 | -Xcompression-level <compression-level> |
| 316 | <compression-level> should be 1 .. 9 (default 8) |
| 317 | Only applies to lzo1x_999 algorithm |
| 318 | lz4 |
| 319 | -Xhc |
| 320 | Compress using LZ4 High Compression |
| 321 | xz |
| 322 | -Xbcj filter1,filter2,...,filterN |
| 323 | Compress using filter1,filter2,...,filterN in turn |
| 324 | (in addition to no filter), and choose the best compression. |
| 325 | Available filters: x86, arm, armthumb, powerpc, sparc, ia64 |
| 326 | -Xdict-size <dict-size> |
| 327 | Use <dict-size> as the XZ dictionary size. The dictionary size |
| 328 | can be specified as a percentage of the block size, or as an |
| 329 | absolute value. The dictionary size must be less than or equal |
| 330 | to the block size and 8192 bytes or larger. It must also be |
| 331 | storable in the xz header as either 2^n or as 2^n+2^(n+1). |
| 332 | Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K |
| 333 | etc. |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 334 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 335 | If the compressor offers compression specific options (all the compressors now |
| 336 | have compression specific options except the deprecated lzma1 compressor) |
| 337 | then these options are also displayed (.i.e. in the above XZ is shown with two |
| 338 | compression specific options). The compression specific options are, obviously, |
| 339 | specific to the compressor in question, and the compressor documentation and |
| 340 | web sites should be consulted to understand their behaviour. In general |
| 341 | the Mksquashfs compression defaults for each compressor are optimised to |
| 342 | give the best performance for each compressor, where what constitutes |
| 343 | best depends on the compressor. For gzip/xz best means highest compression, |
| 344 | for LZO/LZ4 best means a tradeoff between compression and (de)-compression |
| 345 | overhead (LZO/LZ4 by definition are intended for weaker processors). |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 346 | |
| 347 | 3.3 Changing global compression defaults used in mksquashfs |
| 348 | ----------------------------------------------------------- |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 349 | |
| 350 | There are a large number of options that can be used to control the |
| 351 | compression in mksquashfs. By and large the defaults are the most |
| 352 | optimum settings and should only be changed in exceptional circumstances! |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 353 | Note, this does not apply to the block size, increasing the block size |
| 354 | from the default of 128Kbytes will increase compression (especially |
| 355 | for the xz compressor) and should increase I/O performance too. However, |
| 356 | a block size of greater than 128Kbytes may increase latency in certain |
| 357 | cases (where the filesystem contains lots of fragments, and no locality |
| 358 | of reference is observed). For this reason the block size default is |
| 359 | configured to the less optimal 128Kbytes. Users should experiment |
| 360 | with 256Kbyte sizes or above. |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 361 | |
| 362 | The -noI, -noD and -noF options (also -noInodeCompression, -noDataCompression |
| 363 | and -noFragmentCompression) can be used to force mksquashfs to not compress |
| 364 | inodes/directories, data and fragments respectively. Giving all options |
| 365 | generates an uncompressed filesystem. |
| 366 | |
| 367 | The -no-fragments tells mksquashfs to not generate fragment blocks, and rather |
| 368 | generate a filesystem similar to a Squashfs 1.x filesystem. It will of course |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 369 | still be a Squashfs 4.0 filesystem but without fragments, and so it won't be |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 370 | mountable on a Squashfs 1.x system. |
| 371 | |
| 372 | The -always-use-fragments option tells mksquashfs to always generate |
| 373 | fragments for files irrespective of the file length. By default only small |
| 374 | files less than the block size are packed into fragment blocks. The ends of |
| 375 | files which do not fit fully into a block, are NOT by default packed into |
| 376 | fragments. To illustrate this, a 100K file has an initial 64K block and a 36K |
| 377 | remainder. This 36K remainder is not packed into a fragment by default. This |
| 378 | is because to do so leads to a 10 - 20% drop in sequential I/O performance, as a |
| 379 | disk head seek is needed to seek to the initial file data and another disk seek |
| 380 | is need to seek to the fragment block. Specify this option if you want file |
| 381 | remainders to be packed into fragment blocks. Doing so may increase the |
| 382 | compression obtained BUT at the expense of I/O speed. |
| 383 | |
| 384 | The -no-duplicates option tells mksquashfs to not check the files being |
| 385 | added to the filesystem for duplicates. This can result in quicker filesystem |
| 386 | generation and appending although obviously compression will suffer badly if |
| 387 | there is a lot of duplicate files. |
| 388 | |
Phillip Lougher | 13b3f39 | 2014-08-08 21:46:45 +0100 | [diff] [blame] | 389 | The -b option allows the block size to be selected, both "K" and "M" postfixes |
| 390 | are supported, this can be either 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K or |
| 391 | 1M bytes. |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 392 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 393 | 3.4 Specifying the UIDs/GIDs used in the filesystem |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 394 | --------------------------------------------------- |
| 395 | |
| 396 | By default files in the generated filesystem inherit the UID and GID ownership |
| 397 | of the original file. However, mksquashfs provides a number of options which |
| 398 | can be used to override the ownership. |
| 399 | |
| 400 | The options -all-root and -root-owned (both do exactly the same thing) force all |
| 401 | file uids/gids in the generated Squashfs filesystem to be root. This allows |
| 402 | root owned filesystems to be built without root access on the host machine. |
| 403 | |
| 404 | The "-force-uid uid" option forces all files in the generated Squashfs |
| 405 | filesystem to be owned by the specified uid. The uid can be specified either by |
| 406 | name (i.e. "root") or by number. |
| 407 | |
| 408 | The "-force-gid gid" option forces all files in the generated Squashfs |
| 409 | filesystem to be group owned by the specified gid. The gid can be specified |
| 410 | either by name (i.e. "root") or by number. |
| 411 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 412 | 3.5 Excluding files from the filesystem |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 413 | --------------------------------------- |
| 414 | |
Phillip Lougher | 1c75624 | 2014-08-08 19:56:10 +0100 | [diff] [blame] | 415 | The -e and -ef options allow files/directories to be specified which are |
| 416 | excluded from the output filesystem. The -e option takes the exclude |
| 417 | files/directories from the command line, the -ef option takes the |
| 418 | exlude files/directories from the specified exclude file, one file/directory |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 419 | per line. |
| 420 | |
| 421 | Two styles of exclude file matching are supported: basic exclude matching, and |
| 422 | extended wildcard matching. Basic exclude matching is a legacy feature |
| 423 | retained for backwards compatibility with earlier versions of Mksquashfs. |
| 424 | Extended wildcard matching should be used in preference. |
| 425 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 426 | 3.5.1 Basic exclude matching |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 427 | ---------------------------- |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 428 | |
| 429 | Each exclude file is treated as an exact match of a file/directory in |
| 430 | the source directories. If an exclude file/directory is absolute (i.e. |
| 431 | prefixed with /, ../, or ./) the entry is treated as absolute, however, if an |
| 432 | exclude file/directory is relative, it is treated as being relative to each of |
| 433 | the sources in turn, i.e. |
Phillip Lougher | e162193 | 2014-08-08 05:30:01 +0100 | [diff] [blame] | 434 | |
| 435 | %mksquashfs /tmp/source1 source2 output_fs -e ex1 /tmp/source1/ex2 out/ex3 |
| 436 | |
| 437 | Will generate exclude files /tmp/source1/ex2, /tmp/source1/ex1, source2/ex1, |
| 438 | /tmp/source1/out/ex3 and source2/out/ex3. |
| 439 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 440 | 3.5.2 Extended exclude file handling |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 441 | ------------------------------------ |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 442 | |
| 443 | Extended exclude file matching treats each exclude file as a wildcard or |
| 444 | regex expression. To enable wildcard matching specify the -wildcards |
| 445 | option, and to enable regex matching specify the -regex option. In most |
| 446 | cases the -wildcards option should be used rather than -regex because wildcard |
| 447 | matching behaviour is significantly easier to understand! |
| 448 | |
| 449 | In addition to wildcards/regex expressions, exclude files can be "anchored" or |
| 450 | "non-anchored". An anchored exclude is one which matches from the root of the |
| 451 | directory and nowhere else, a non-anchored exclude matches anywhere. For |
| 452 | example given the directory hierarchy "a/b/c/a/b", the anchored exclude |
| 453 | "a/b" will match "a/b" at the root of the directory hierarchy, but |
| 454 | it will not match the "/a/b" sub-directory within directory "c", whereas a |
| 455 | non-anchored exclude would. |
| 456 | |
| 457 | A couple of examples should make this clearer. |
| 458 | |
| 459 | Anchored excludes |
| 460 | |
| 461 | 1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz' |
| 462 | |
| 463 | Exclude all files matching "*.gz" in the top level directory "test". |
| 464 | |
| 465 | 2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*' |
| 466 | |
| 467 | Exclude all files beginning with "example" inside directories called |
| 468 | "Test" or "test", that occur inside any top level directory. |
| 469 | |
| 470 | Using extended wildcards, negative matching is also possible. |
| 471 | |
| 472 | 3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz' |
| 473 | |
| 474 | Exclude all files matching "*.gz" in top level directory "test", |
| 475 | except those with "data" in the name. |
| 476 | |
| 477 | Non-anchored excludes |
| 478 | |
| 479 | By default excludes match from the top level directory, but it is |
| 480 | often useful to exclude a file matching anywhere in the source directories. |
| 481 | For this non-anchored excludes can be used, specified by pre-fixing the |
| 482 | exclude with "...". |
| 483 | |
| 484 | Examples: |
| 485 | |
| 486 | 1. mksquashfs example image.sqsh -wildcards -e '... *.gz' |
| 487 | |
| 488 | Exclude files matching "*.gz" anywhere in the source directories. |
| 489 | For example this will match "example.gz", "test/example.gz", and |
| 490 | "test/test/example.gz". |
| 491 | |
| 492 | 2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz' |
| 493 | |
| 494 | Exclude files matching "*.gz" inside directories called "Test" or |
| 495 | "test" that occur anywhere in the source directories. |
| 496 | |
| 497 | Again, using extended wildcards, negative matching is also possible. |
| 498 | |
| 499 | 3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz' |
| 500 | |
| 501 | Exclude all files matching "*.gz" anywhere in the source directories, |
| 502 | except those with "data" in the name. |
| 503 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 504 | 3.5.3 Exclude files summary |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 505 | --------------------------- |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 506 | |
Phillip Lougher | 1c75624 | 2014-08-08 19:56:10 +0100 | [diff] [blame] | 507 | The -e and -ef exclude options are usefully used in archiving the entire |
| 508 | filesystem, where it is wished to avoid archiving /proc, and the filesystem |
| 509 | being generated, i.e. |
Phillip Lougher | e162193 | 2014-08-08 05:30:01 +0100 | [diff] [blame] | 510 | |
| 511 | %mksquashfs / /tmp/root.sqsh -e proc /tmp/root.sqsh |
| 512 | |
Phillip Lougher | 1c75624 | 2014-08-08 19:56:10 +0100 | [diff] [blame] | 513 | Multiple -ef options can be specified on the command line, and the -ef |
| 514 | option can be used in conjuction with the -e option. |
| 515 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 516 | 3.6 Appending to squashfs filesystems |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 517 | ------------------------------------- |
| 518 | |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 519 | Running squashfs with the destination directory containing an existing |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 520 | filesystem will add the source items to the existing filesystem. By default, |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 521 | the source items are added to the existing root directory. |
| 522 | |
| 523 | To make this clear... An existing filesystem "image" contains root entries |
| 524 | "old1", and "old2". Source directory "/home/phillip/test" contains "file1", |
| 525 | "file2" and "dir1". |
| 526 | |
| 527 | example 1: |
| 528 | |
| 529 | %mksquashfs /home/phillip/test image |
| 530 | |
| 531 | Will create a new "image" with root entries "old1", "old2", "file1", "file2" and |
| 532 | "dir1" |
| 533 | |
| 534 | example 2: |
| 535 | |
| 536 | %mksquashfs /home/phillip/test image -keep-as-directory |
| 537 | |
| 538 | Will create a new "image" with root entries "old1", "old2", and "test". |
| 539 | As shown in the previous section, for single source directories |
| 540 | '-keep-as-directory' adds the source directory rather than the |
| 541 | contents of the directory. |
| 542 | |
| 543 | example 3: |
| 544 | |
Phillip Lougher | c11f006 | 2014-08-08 20:39:26 +0100 | [diff] [blame] | 545 | %mksquashfs /home/phillip/test image -keep-as-directory -root-becomes |
| 546 | original-root |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 547 | |
| 548 | Will create a new "image" with root entries "original-root", and "test". The |
| 549 | '-root-becomes' option specifies that the original root becomes a subdirectory |
| 550 | in the new root, with the specified name. |
| 551 | |
| 552 | The append option with file duplicate detection, means squashfs can be |
| 553 | used as a simple versioning archiving filesystem. A squashfs filesystem can |
| 554 | be created with for example the linux-2.4.19 source. Appending the linux-2.4.20 |
| 555 | source will create a filesystem with the two source trees, but only the |
| 556 | changed files will take extra room, the unchanged files will be detected as |
| 557 | duplicates. |
| 558 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 559 | 3.7 Appending recovery file feature |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 560 | ----------------------------------- |
| 561 | |
| 562 | Recovery files are created when appending to existing Squashfs |
| 563 | filesystems. This allows the original filesystem to be recovered |
| 564 | if Mksquashfs aborts unexpectedly (i.e. power failure). |
| 565 | |
| 566 | The recovery files are called squashfs_recovery_xxx_yyy, where |
| 567 | "xxx" is the name of the filesystem being appended to, and "yyy" is a |
| 568 | number to guarantee filename uniqueness (the PID of the parent Mksquashfs |
| 569 | process). |
| 570 | |
| 571 | Normally if Mksquashfs exits correctly the recovery file is deleted to |
| 572 | avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover" |
| 573 | option can be used to recover the filesystem, giving the previously |
| 574 | created recovery file as a parameter, i.e. |
| 575 | |
| 576 | mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234 |
| 577 | |
| 578 | The writing of the recovery file can be disabled by specifying the |
| 579 | "-no-recovery" option. |
| 580 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 581 | 3.8 Pseudo file support |
| 582 | ----------------------- |
| 583 | |
| 584 | Mksquashfs supports pseudo files, these allow fake files, directories, character |
| 585 | and block devices to be specified and added to the Squashfs filesystem being |
| 586 | built, rather than requiring them to be present in the source directories. |
| 587 | This, for example, allows device nodes to be added to the filesystem without |
| 588 | requiring root access. |
| 589 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 590 | Mksquashfs 4.1 added support for "dynamic pseudo files" and a modify operation. |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 591 | Dynamic pseudo files allow files to be dynamically created when Mksquashfs |
| 592 | is run, their contents being the result of running a command or piece of |
| 593 | shell script. The modifiy operation allows the mode/uid/gid of an existing |
| 594 | file in the source filesystem to be modified. |
| 595 | |
| 596 | Two Mksquashfs options are supported, -p allows one pseudo file to be specified |
| 597 | on the command line, and -pf allows a pseudo file to be specified containing a |
| 598 | list of pseduo definitions, one per line. |
| 599 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 600 | 3.8.1. Creating a dynamic file |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 601 | ------------------------------ |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 602 | |
| 603 | Pseudo definition |
| 604 | |
| 605 | Filename f mode uid gid command |
| 606 | |
| 607 | mode is the octal mode specifier, similar to that expected by chmod. |
| 608 | |
| 609 | uid and gid can be either specified as a decimal number, or by name. |
| 610 | |
| 611 | command can be an executable or a piece of shell script, and it is executed |
| 612 | by running "/bin/sh -c command". The stdout becomes the contents of |
| 613 | "Filename". |
| 614 | |
| 615 | Examples: |
| 616 | |
| 617 | Running a basic command |
| 618 | ----------------------- |
| 619 | |
| 620 | /somedir/dmesg f 444 root root dmesg |
| 621 | |
| 622 | creates a file "/somedir/dmesg" containing the output from dmesg. |
| 623 | |
| 624 | Executing shell script |
| 625 | ---------------------- |
| 626 | |
| 627 | RELEASE f 444 root root \ |
| 628 | if [ ! -e /tmp/ver ]; then \ |
| 629 | echo 0 > /tmp/ver; \ |
| 630 | fi; \ |
| 631 | ver=`cat /tmp/ver`; \ |
| 632 | ver=$((ver +1)); \ |
| 633 | echo $ver > /tmp/ver; \ |
| 634 | echo -n `cat /tmp/release`; \ |
| 635 | echo "-dev #"$ver `date` "Build host" `hostname` |
| 636 | |
| 637 | Creates a file RELEASE containing the release name, date, build host, and |
| 638 | an incrementing version number. The incrementing version is a side-effect |
| 639 | of executing the shell script, and ensures every time Mksquashfs is run a |
| 640 | new version number is used without requiring any other shell scripting. |
| 641 | |
| 642 | The above example also shows that commands can be split across multiple lines |
| 643 | using "\". Obviously as the script will be presented to the shell as a single |
| 644 | line, a semicolon is need to separate individual shell commands within the |
| 645 | shell script. |
| 646 | |
| 647 | Reading from a device (or fifo/named socket) |
| 648 | -------------------------------------------- |
| 649 | |
| 650 | input f 444 root root dd if=/dev/sda1 bs=1024 count=10 |
| 651 | |
| 652 | Copies 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs |
| 653 | given a device, fifo, or named socket will place that special file within the |
| 654 | Squashfs filesystem, the above allows input from these special files to be |
| 655 | captured and placed in the Squashfs filesystem. |
| 656 | |
| 657 | 3.8.2. Creating a block or character device |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 658 | ------------------------------------------- |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 659 | |
| 660 | Pseudo definition |
| 661 | |
| 662 | Filename type mode uid gid major minor |
| 663 | |
| 664 | Where type is either |
| 665 | b - for block devices, and |
| 666 | c - for character devices |
| 667 | |
| 668 | mode is the octal mode specifier, similar to that expected by chmod. |
| 669 | |
| 670 | uid and gid can be either specified as a decimal number, or by name. |
| 671 | |
| 672 | For example: |
| 673 | |
| 674 | /dev/chr_dev c 666 root root 100 1 |
| 675 | /dev/blk_dev b 666 0 0 200 200 |
| 676 | |
| 677 | creates a character device "/dev/chr_dev" with major:minor 100:1 and |
| 678 | a block device "/dev/blk_dev" with major:minor 200:200, both with root |
| 679 | uid/gid and a mode of rw-rw-rw. |
| 680 | |
| 681 | 3.8.3. Creating a directory |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 682 | --------------------------- |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 683 | |
| 684 | Pseudo definition |
| 685 | |
| 686 | Filename d mode uid gid |
| 687 | |
| 688 | mode is the octal mode specifier, similar to that expected by chmod. |
| 689 | |
| 690 | uid and gid can be either specified as a decimal number, or by name. |
| 691 | |
| 692 | For example: |
| 693 | |
| 694 | /pseudo_dir d 666 root root |
| 695 | |
| 696 | creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw. |
| 697 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 698 | 3.8.4. Modifying attributes of an existing file |
| 699 | ----------------------------------------------- |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 700 | |
| 701 | Pseudo definition |
| 702 | |
| 703 | Filename m mode uid gid |
| 704 | |
| 705 | mode is the octal mode specifier, similar to that expected by chmod. |
| 706 | |
| 707 | uid and gid can be either specified as a decimal number, or by name. |
| 708 | |
| 709 | For example: |
| 710 | |
| 711 | dmesg m 666 root root |
| 712 | |
| 713 | Changes the attributes of the file "dmesg" in the filesystem to have |
| 714 | root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained |
| 715 | from the source filesystem. |
| 716 | |
| 717 | 3.9 Miscellaneous options |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 718 | ------------------------- |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 719 | |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 720 | The -info option displays the files/directories as they are compressed and |
| 721 | added to the filesystem. The original uncompressed size of each file |
| 722 | is printed, along with DUPLICATE if the file is a duplicate of a |
| 723 | file in the filesystem. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 724 | |
Phillip Lougher | fd0e08a | 2014-08-08 20:55:59 +0100 | [diff] [blame] | 725 | The -nopad option informs mksquashfs to not pad the filesystem to a 4K multiple. |
| 726 | This is performed by default to enable the output filesystem file to be mounted |
| 727 | by loopback, which requires files to be a 4K multiple. If the filesystem is |
| 728 | being written to a block device, or is to be stored in a bootimage, the extra |
| 729 | pad bytes are not needed. |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 730 | |
Phillip Lougher | 676fcce | 2014-08-08 21:27:38 +0100 | [diff] [blame] | 731 | 4. UNSQUASHFS |
| 732 | ------------- |
| 733 | |
| 734 | Unsquashfs allows you to decompress and extract a Squashfs filesystem without |
| 735 | mounting it. It can extract the entire filesystem, or a specific |
| 736 | file or directory. |
| 737 | |
| 738 | The Unsquashfs usage info is: |
| 739 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 740 | SYNTAX: ./unsquashfs [options] filesystem [directories or files to extract] |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 741 | -v[ersion] print version, licence and copyright information |
| 742 | -d[est] <pathname> unsquash to <pathname>, default "squashfs-root" |
| 743 | -n[o-progress] don't display the progress bar |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 744 | -no[-xattrs] don't extract xattrs in file system |
| 745 | -x[attrs] extract xattrs in file system (default) |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 746 | -u[ser-xattrs] only extract user xattrs in file system. |
| 747 | Enables extracting xattrs |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 748 | -p[rocessors] <number> use <number> processors. By default will use |
| 749 | number of processors available |
| 750 | -i[nfo] print files as they are unsquashed |
| 751 | -li[nfo] print files as they are unsquashed with file |
| 752 | attributes (like ls -l output) |
| 753 | -l[s] list filesystem, but don't unsquash |
| 754 | -ll[s] list filesystem with file attributes (like |
| 755 | ls -l output), but don't unsquash |
| 756 | -f[orce] if file already exists then overwrite |
| 757 | -s[tat] display filesystem superblock information |
| 758 | -e[f] <extract file> list of directories or files to extract. |
| 759 | One per line |
| 760 | -da[ta-queue] <size> Set data queue to <size> Mbytes. Default 256 |
| 761 | Mbytes |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 762 | -fr[ag-queue] <size> Set fragment queue to <size> Mbytes. Default |
| 763 | 256 Mbytes |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 764 | -r[egex] treat extract names as POSIX regular expressions |
| 765 | rather than use the default shell wildcard |
| 766 | expansion (globbing) |
Phillip Lougher | 676fcce | 2014-08-08 21:27:38 +0100 | [diff] [blame] | 767 | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 768 | Decompressors available: |
| 769 | gzip |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 770 | lzo |
| 771 | lz4 |
| 772 | xz |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 773 | |
Phillip Lougher | 13b3f39 | 2014-08-08 21:46:45 +0100 | [diff] [blame] | 774 | To extract a subset of the filesystem, the filenames or directory |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 775 | trees that are to be extracted can be specified on the command line. The |
Phillip Lougher | 13b3f39 | 2014-08-08 21:46:45 +0100 | [diff] [blame] | 776 | files/directories should be specified using the full path to the |
| 777 | files/directories as they appear within the Squashfs filesystem. The |
| 778 | files/directories will also be extracted to those positions within the specified |
| 779 | destination directory. |
Phillip Lougher | 676fcce | 2014-08-08 21:27:38 +0100 | [diff] [blame] | 780 | |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 781 | The extract files can also be given in a file using the "-e[f]" option. |
| 782 | |
| 783 | Similarly to Mksquashfs, wildcard matching is performed on the extract |
| 784 | files. Wildcard matching is enabled by default. |
| 785 | |
| 786 | Examples: |
| 787 | |
| 788 | 1. unsquashfs image.sqsh 'test/*.gz' |
| 789 | |
| 790 | Extract all files matching "*.gz" in the top level directory "test". |
| 791 | |
| 792 | 2. unsquashfs image.sqsh '[Tt]est/example*' |
| 793 | |
| 794 | Extract all files beginning with "example" inside top level directories |
| 795 | called "Test" or "test". |
| 796 | |
| 797 | Using extended wildcards, negative matching is also possible. |
| 798 | |
| 799 | 3. unsquashfs image.sqsh 'test/!(*data*).gz' |
| 800 | |
| 801 | Extract all files matching "*.gz" in top level directory "test", |
| 802 | except those with "data" in the name. |
| 803 | |
| 804 | |
| 805 | 4.1 Unsquashfs options |
| 806 | ---------------------- |
Phillip Lougher | 676fcce | 2014-08-08 21:27:38 +0100 | [diff] [blame] | 807 | |
| 808 | The "-ls" option can be used to list the contents of a filesystem without |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 809 | decompressing the filesystem data itself. The "-lls" option is similar |
| 810 | but it also displays file attributes (ls -l style output). |
Phillip Lougher | 676fcce | 2014-08-08 21:27:38 +0100 | [diff] [blame] | 811 | |
| 812 | The "-info" option forces Unsquashfs to print each file as it is decompressed. |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 813 | The -"linfo" is similar but it also displays file attributes. |
Phillip Lougher | 676fcce | 2014-08-08 21:27:38 +0100 | [diff] [blame] | 814 | |
| 815 | The "-dest" option specifies the directory that is used to decompress |
| 816 | the filesystem data. If this option is not given then the filesystem is |
| 817 | decompressed to the directory "squashfs-root" in the current working directory. |
| 818 | |
| 819 | The "-force" option forces Unsquashfs to output to the destination |
| 820 | directory even if files or directories already exist. This allows you |
| 821 | to update an existing directory tree, or to Unsquashfs to a partially |
| 822 | filled directory. Without the "-force" option, Unsquashfs will |
| 823 | refuse to overwrite any existing files, or to create any directories if they |
| 824 | already exist. This is done to protect data in case of mistakes, and |
| 825 | so the "-force" option should be used with caution. |
| 826 | |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 827 | The "-stat" option displays filesystem superblock information. This is |
| 828 | useful to discover the filesystem version, byte ordering, whether it has a NFS |
| 829 | export table, and what options were used to compress the filesystem, etc. |
| 830 | |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 831 | Unsquashfs can decompress all Squashfs filesystem versions, 1.x, 2.x, 3.x and |
| 832 | 4.0 filesystems. |
Phillip Lougher | 676fcce | 2014-08-08 21:27:38 +0100 | [diff] [blame] | 833 | |
| 834 | 5. FILESYSTEM LAYOUT |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 835 | -------------------- |
| 836 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 837 | A squashfs filesystem consists of a maximum of nine parts, packed together on a |
| 838 | byte alignment: |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 839 | |
| 840 | --------------- |
| 841 | | superblock | |
| 842 | |---------------| |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 843 | | compression | |
| 844 | | options | |
| 845 | |---------------| |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 846 | | datablocks | |
| 847 | | & fragments | |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 848 | |---------------| |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 849 | | inode table | |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 850 | |---------------| |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 851 | | directory | |
| 852 | | table | |
| 853 | |---------------| |
| 854 | | fragment | |
| 855 | | table | |
| 856 | |---------------| |
| 857 | | export | |
| 858 | | table | |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 859 | |---------------| |
| 860 | | uid/gid | |
| 861 | | lookup table | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 862 | |---------------| |
| 863 | | xattr | |
| 864 | | table | |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 865 | --------------- |
| 866 | |
| 867 | Compressed data blocks are written to the filesystem as files are read from |
| 868 | the source directory, and checked for duplicates. Once all file data has been |
Phillip Lougher | 0254342 | 2014-08-08 22:10:59 +0100 | [diff] [blame] | 869 | written the completed super-block, compression options, inode, directory, |
| 870 | fragment, export, uid/gid lookup and xattr tables are written. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 871 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 872 | 5.1 Compression options |
| 873 | ----------------------- |
| 874 | |
| 875 | Compressors can optionally support compression specific options (e.g. |
| 876 | dictionary size). If non-default compression options have been used, then |
| 877 | these are stored here. |
| 878 | |
| 879 | 5.2 Inodes |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 880 | ---------- |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 881 | |
| 882 | Metadata (inodes and directories) are compressed in 8Kbyte blocks. Each |
| 883 | compressed block is prefixed by a two byte length, the top bit is set if the |
| 884 | block is uncompressed. A block will be uncompressed if the -noI option is set, |
| 885 | or if the compressed block was larger than the uncompressed block. |
| 886 | |
| 887 | Inodes are packed into the metadata blocks, and are not aligned to block |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 888 | boundaries, therefore inodes overlap compressed blocks. Inodes are identified |
| 889 | by a 48-bit number which encodes the location of the compressed metadata block |
| 890 | containing the inode, and the byte offset into that block where the inode is |
| 891 | placed (<block, offset>). |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 892 | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 893 | To maximise compression there are different inodes for each file type |
| 894 | (regular file, directory, device, etc.), the inode contents and length |
| 895 | varying with the type. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 896 | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 897 | To further maximise compression, two types of regular file inode and |
| 898 | directory inode are defined: inodes optimised for frequently occurring |
| 899 | regular files and directories, and extended types where extra |
| 900 | information has to be stored. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 901 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 902 | 5.3 Directories |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 903 | --------------- |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 904 | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 905 | Like inodes, directories are packed into compressed metadata blocks, stored |
| 906 | in a directory table. Directories are accessed using the start address of |
| 907 | the metablock containing the directory and the offset into the |
| 908 | decompressed block (<block, offset>). |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 909 | |
| 910 | Directories are organised in a slightly complex way, and are not simply |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 911 | a list of file names. The organisation takes advantage of the |
| 912 | fact that (in most cases) the inodes of the files will be in the same |
| 913 | compressed metadata block, and therefore, can share the start block. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 914 | Directories are therefore organised in a two level list, a directory |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 915 | header containing the shared start block value, and a sequence of directory |
| 916 | entries, each of which share the shared start block. A new directory header |
| 917 | is written once/if the inode start block changes. The directory |
| 918 | header/directory entry list is repeated as many times as necessary. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 919 | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 920 | Directories are sorted, and can contain a directory index to speed up |
| 921 | file lookup. Directory indexes store one entry per metablock, each entry |
| 922 | storing the index/filename mapping to the first directory header |
| 923 | in each metadata block. Directories are sorted in alphabetical order, |
| 924 | and at lookup the index is scanned linearly looking for the first filename |
| 925 | alphabetically larger than the filename being looked up. At this point the |
| 926 | location of the metadata block the filename is in has been found. |
| 927 | The general idea of the index is ensure only one metadata block needs to be |
| 928 | decompressed to do a lookup irrespective of the length of the directory. |
| 929 | This scheme has the advantage that it doesn't require extra memory overhead |
| 930 | and doesn't require much extra storage on disk. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 931 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 932 | 5.4 File data |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 933 | ------------- |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 934 | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 935 | Regular files consist of a sequence of contiguous compressed blocks, and/or a |
| 936 | compressed fragment block (tail-end packed block). The compressed size |
| 937 | of each datablock is stored in a block list contained within the |
| 938 | file inode. |
Phillip Lougher | 9e37ac0 | 2014-08-08 05:15:38 +0100 | [diff] [blame] | 939 | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 940 | To speed up access to datablocks when reading 'large' files (256 Mbytes or |
| 941 | larger), the code implements an index cache that caches the mapping from |
| 942 | block index to datablock location on disk. |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 943 | |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 944 | The index cache allows Squashfs to handle large files (up to 1.75 TiB) while |
| 945 | retaining a simple and space-efficient block list on disk. The cache |
| 946 | is split into slots, caching up to eight 224 GiB files (128 KiB blocks). |
| 947 | Larger files use multiple slots, with 1.75 TiB files using all 8 slots. |
| 948 | The index cache is designed to be memory efficient, and by default uses |
| 949 | 16 KiB. |
| 950 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 951 | 5.5 Fragment lookup table |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 952 | ------------------------- |
| 953 | |
| 954 | Regular files can contain a fragment index which is mapped to a fragment |
| 955 | location on disk and compressed size using a fragment lookup table. This |
| 956 | fragment lookup table is itself stored compressed into metadata blocks. |
| 957 | A second index table is used to locate these. This second index table for |
| 958 | speed of access (and because it is small) is read at mount time and cached |
| 959 | in memory. |
| 960 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 961 | 5.6 Uid/gid lookup table |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 962 | ------------------------ |
| 963 | |
| 964 | For space efficiency regular files store uid and gid indexes, which are |
| 965 | converted to 32-bit uids/gids using an id look up table. This table is |
| 966 | stored compressed into metadata blocks. A second index table is used to |
| 967 | locate these. This second index table for speed of access (and because it |
| 968 | is small) is read at mount time and cached in memory. |
| 969 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 970 | 5.7 Export table |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 971 | ---------------- |
| 972 | |
| 973 | To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems |
| 974 | can optionally (disabled with the -no-exports Mksquashfs option) contain |
| 975 | an inode number to inode disk location lookup table. This is required to |
| 976 | enable Squashfs to map inode numbers passed in filehandles to the inode |
| 977 | location on disk, which is necessary when the export code reinstantiates |
| 978 | expired/flushed inodes. |
| 979 | |
| 980 | This table is stored compressed into metadata blocks. A second index table is |
| 981 | used to locate these. This second index table for speed of access (and because |
| 982 | it is small) is read at mount time and cached in memory. |
| 983 | |
Phillip Lougher | a650bcc | 2014-08-08 22:02:51 +0100 | [diff] [blame] | 984 | 5.8 Xattr table |
Phillip Lougher | bef677b | 2014-08-08 21:59:19 +0100 | [diff] [blame] | 985 | --------------- |
| 986 | |
| 987 | The xattr table contains extended attributes for each inode. The xattrs |
| 988 | for each inode are stored in a list, each list entry containing a type, |
| 989 | name and value field. The type field encodes the xattr prefix |
| 990 | ("user.", "trusted." etc) and it also encodes how the name/value fields |
| 991 | should be interpreted. Currently the type indicates whether the value |
| 992 | is stored inline (in which case the value field contains the xattr value), |
| 993 | or if it is stored out of line (in which case the value field stores a |
| 994 | reference to where the actual value is stored). This allows large values |
| 995 | to be stored out of line improving scanning and lookup performance and it |
| 996 | also allows values to be de-duplicated, the value being stored once, and |
| 997 | all other occurences holding an out of line reference to that value. |
| 998 | |
| 999 | The xattr lists are packed into compressed 8K metadata blocks. |
| 1000 | To reduce overhead in inodes, rather than storing the on-disk |
| 1001 | location of the xattr list inside each inode, a 32-bit xattr id |
| 1002 | is stored. This xattr id is mapped into the location of the xattr |
| 1003 | list using a second xattr id lookup table. |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 1004 | |
Phillip Lougher | 676fcce | 2014-08-08 21:27:38 +0100 | [diff] [blame] | 1005 | 6. AUTHOR INFO |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 1006 | -------------- |
| 1007 | |
Phillip Lougher | 1918283 | 2014-08-08 21:49:25 +0100 | [diff] [blame] | 1008 | Squashfs was written by Phillip Lougher, email phillip@lougher.demon.co.uk, |
Phillip Lougher | 2ce29a9 | 2014-08-08 05:44:57 +0100 | [diff] [blame] | 1009 | in Chepstow, Wales, UK. If you like the program, or have any problems, |
| 1010 | then please email me, as it's nice to get feedback! |