Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 1 | erofs-utils |
| 2 | =========== |
| 3 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 4 | erofs-utils includes user-space tools for EROFS filesystem. |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 5 | Currently mkfs.erofs, (experimental) erofsfuse, dump.erofs, fsck.erofs |
| 6 | are available. |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 7 | |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 8 | Dependencies & build |
| 9 | -------------------- |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 10 | |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 11 | lz4 1.8.0+ for lz4 enabled [2], lz4 1.9.3+ highly recommended [4][5]. |
Gao Xiang | aa7d722 | 2021-10-30 10:01:16 +0800 | [diff] [blame] | 12 | XZ Utils 5.3.2alpha [6] or later versions for MicroLZMA enabled. |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 13 | |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 14 | libfuse 2.6+ for erofsfuse enabled as a plus. |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 15 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 16 | How to build with lz4-1.9.0 or above |
| 17 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 18 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 19 | To build, you can run the following commands in order: |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 20 | |
| 21 | :: |
| 22 | |
| 23 | $ ./autogen.sh |
| 24 | $ ./configure |
| 25 | $ make |
| 26 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 27 | mkfs.erofs binary will be generated under mkfs folder. |
| 28 | |
| 29 | * For lz4 < 1.9.2, there are some stability issues about |
| 30 | LZ4_compress_destSize(). (lz4hc isn't impacted) [3]. |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 31 | |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 32 | ** For lz4 = 1.9.2, there is a noticeable regression about |
| 33 | LZ4_decompress_safe_partial() [5], which impacts erofsfuse |
| 34 | functionality for legacy images (without 0PADDING). |
| 35 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 36 | How to build with lz4-1.8.0~1.8.3 |
| 37 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 38 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 39 | For these old lz4 versions, lz4hc algorithm cannot be supported |
| 40 | without lz4-static installed due to LZ4_compress_HC_destSize() |
| 41 | unstable api usage, which means lz4 will only be available if |
| 42 | lz4-static isn't found. |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 43 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 44 | On Fedora, lz4-static can be installed by using: |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 45 | |
| 46 | yum install lz4-static.x86_64 |
| 47 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 48 | However, it's still not recommended using those versions directly |
| 49 | since there are serious bugs in these compressors, see [2] [3] [4] |
| 50 | as well. |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 51 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 52 | How to build with liblzma |
| 53 | ~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 54 | |
| 55 | In order to enable LZMA support, build with the following commands: |
| 56 | $ ./configure --enable-lzma |
| 57 | $ make |
| 58 | |
| 59 | Additionally, you could specify liblzma build paths with: |
| 60 | --with-liblzma-incdir and --with-liblzma-libdir |
| 61 | |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 62 | mkfs.erofs |
| 63 | ---------- |
| 64 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 65 | two main kinds of EROFS images can be generated: (un)compressed. |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 66 | |
| 67 | - For uncompressed images, there will be none of compression |
| 68 | files in these images. However, it can decide whether the tail |
| 69 | block of a file should be inlined or not properly [1]. |
| 70 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 71 | - For compressed images, it'll try to use specific algorithms |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 72 | first for each regular file and see if storage space can be |
| 73 | saved with compression. If not, fallback to an uncompressed |
| 74 | file. |
| 75 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 76 | How to generate EROFS images (lz4 for Linux 5.3+, lzma for Linux 5.16+) |
| 77 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 78 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 79 | Currently lz4(hc) and lzma are available for compression, e.g. |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 80 | $ mkfs.erofs -zlz4hc foo.erofs.img foo/ |
| 81 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 82 | Or leave all files uncompressed as an option: |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 83 | $ mkfs.erofs foo.erofs.img foo/ |
| 84 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 85 | In addition, you could specify a higher compression level to get a |
| 86 | (slightly) better compression ratio than the default level, e.g. |
| 87 | $ mkfs.erofs -zlz4hc,12 foo.erofs.img foo/ |
| 88 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 89 | Note that all compressors are still single-threaded for now, thus it |
| 90 | could take more time on the multiprocessor platform. Multi-threaded |
| 91 | approach is already in our TODO list. |
| 92 | |
Gao Xiang | 645734f | 2021-05-31 22:31:16 +0800 | [diff] [blame] | 93 | How to generate EROFS big pcluster images (Linux 5.13+) |
| 94 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 95 | |
| 96 | In order to get much better compression ratios (thus better sequential |
| 97 | read performance for common storage devices), big pluster feature has |
| 98 | been introduced since linux-5.13, which is not forward-compatible with |
| 99 | old kernels. |
| 100 | |
| 101 | In details, -C is used to specify the maximum size of each big pcluster |
| 102 | in bytes, e.g. |
| 103 | $ mkfs.erofs -zlz4hc -C65536 foo.erofs.img foo/ |
| 104 | |
| 105 | So in that case, pcluster size can be 64KiB at most. |
| 106 | |
| 107 | Note that large pcluster size can cause bad random performance, so |
| 108 | please evaluate carefully in advance. Or make your own per-(sub)file |
| 109 | compression strategies according to file access patterns if needed. |
| 110 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 111 | How to generate legacy EROFS images (Linux 4.19+) |
| 112 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Gao Xiang | 1f11cf0 | 2019-07-16 11:15:27 +0800 | [diff] [blame] | 113 | |
| 114 | Decompression inplace and compacted indexes have been introduced in |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 115 | Linux upstream v5.3, which are not forward-compatible with older |
| 116 | kernels. |
Gao Xiang | 1f11cf0 | 2019-07-16 11:15:27 +0800 | [diff] [blame] | 117 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 118 | In order to generate _legacy_ EROFS images for old kernels, |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 119 | consider adding "-E legacy-compress" to the command line, e.g. |
Gao Xiang | 1f11cf0 | 2019-07-16 11:15:27 +0800 | [diff] [blame] | 120 | |
| 121 | $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/ |
| 122 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 123 | For Linux kernel >= 5.3, legacy EROFS images are _NOT recommended_ |
| 124 | due to runtime performance loss compared with non-legacy images. |
| 125 | |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 126 | Obsoleted erofs.mkfs |
| 127 | ~~~~~~~~~~~~~~~~~~~~ |
| 128 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 129 | There is an original erofs.mkfs version developed by Li Guifu, |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 130 | which was replaced by the new erofs-utils implementation. |
| 131 | |
| 132 | git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs |
| 133 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 134 | PLEASE NOTE: This version is highly _NOT recommended_ now. |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 135 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 136 | erofsfuse (experimental) |
| 137 | ------------------------ |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 138 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 139 | erofsfuse is introduced to support EROFS format for various platforms |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 140 | (including older linux kernels) and new on-disk features iteration. |
| 141 | It can also be used as an unpacking tool for unprivileged users. |
| 142 | |
| 143 | It supports fixed-sized output decompression *without* any in-place |
| 144 | I/O or in-place decompression optimization. Also like the other FUSE |
| 145 | implementations, it suffers from most common performance issues (e.g. |
| 146 | significant I/O overhead, double caching, etc.) |
| 147 | |
| 148 | Therefore, NEVER use it if performance is the top concern. |
| 149 | |
| 150 | Note that xattr & ACL aren't implemented yet due to the current Android |
| 151 | use-case vs limited time. If you have some interest, contribution is, |
| 152 | as always, welcome. |
| 153 | |
| 154 | How to build erofsfuse |
| 155 | ~~~~~~~~~~~~~~~~~~~~~~ |
| 156 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 157 | It's disabled by default as an experimental feature for now due to |
| 158 | the extra libfuse dependency, to enable and build it manually: |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 159 | |
| 160 | $ ./configure --enable-fuse |
| 161 | $ make |
| 162 | |
| 163 | erofsfuse binary will be generated under fuse folder. |
| 164 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 165 | How to mount an EROFS image with erofsfuse |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 166 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 167 | |
| 168 | As the other FUSE implementations, it's quite simple to mount with |
| 169 | erofsfuse, e.g.: |
| 170 | $ erofsfuse foo.erofs.img foo/ |
| 171 | |
| 172 | Alternatively, to make it run in foreground (with debugging level 3): |
| 173 | $ erofsfuse -f --dbglevel=3 foo.erofs.img foo/ |
| 174 | |
| 175 | To debug erofsfuse (also automatically run in foreground): |
| 176 | $ erofsfuse -d foo.erofs.img foo/ |
| 177 | |
| 178 | To unmount an erofsfuse mountpoint as a non-root user: |
| 179 | $ fusermount -u foo/ |
| 180 | |
Gao Xiang | dbe72d1 | 2021-11-21 19:25:27 +0800 | [diff] [blame] | 181 | dump.erofs and fsck.erofs (experimental) |
| 182 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 183 | |
| 184 | dump.erofs and fsck.erofs are two new experimental tools to analyse |
| 185 | and check EROFS file systems. |
| 186 | |
| 187 | They are still incomplete and actively under development by the |
| 188 | community. But you could check them out if needed in advance. |
| 189 | |
| 190 | Report, feedback and/or contribution are welcomed. |
| 191 | |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 192 | Contribution |
| 193 | ------------ |
| 194 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 195 | erofs-utils is under GPLv2+ as a part of EROFS filesystem project, |
Gao Xiang | 447df4c | 2021-10-26 11:39:49 +0800 | [diff] [blame] | 196 | feel free to send patches or feedback to: |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 197 | linux-erofs mailing list <linux-erofs@lists.ozlabs.org> |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 198 | |
| 199 | Comments |
| 200 | -------- |
| 201 | |
Gao Xiang | 89b9b82 | 2021-05-31 22:31:15 +0800 | [diff] [blame] | 202 | [1] According to the EROFS on-disk format, the tail block of files |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 203 | could be inlined aggressively with its metadata in order to reduce |
| 204 | the I/O overhead and save the storage space (called tail-packing). |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 205 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 206 | [2] There was a bug until lz4-1.8.3, which can crash erofs-utils |
| 207 | randomly. Fortunately bugfix by our colleague Qiuyang Sun was |
| 208 | merged in lz4-1.9.0. |
Gao Xiang | 587bd19 | 2019-04-11 08:46:44 +0800 | [diff] [blame] | 209 | |
| 210 | For more details, please refer to |
| 211 | https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82 |
| 212 | |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 213 | [3] There were many bugfixes merged into lz4-1.9.2 for |
| 214 | LZ4_compress_destSize(), and I once ran into some crashs due to |
| 215 | those issues. * Again lz4hc is not affected. * |
Gao Xiang | 68d9b69 | 2019-10-23 17:20:31 +0800 | [diff] [blame] | 216 | |
| 217 | [LZ4_compress_destSize] Allow 2 more bytes of match length |
| 218 | https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda |
| 219 | |
| 220 | [LZ4_compress_destSize] Fix rare data corruption bug |
| 221 | https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1 |
| 222 | |
| 223 | [LZ4_compress_destSize] Fix overflow condition |
| 224 | https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a |
| 225 | |
| 226 | [LZ4_compress_destSize] Fix off-by-one error in fix |
| 227 | https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333 |
| 228 | |
| 229 | [LZ4_compress_destSize] Fix off-by-one error |
| 230 | https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba |
| 231 | |
| 232 | since upstream lz4 doesn't have stable branch for old versions, it's |
| 233 | preferred to use latest upstream lz4 library (although some regressions |
| 234 | could happen since new features are also introduced to latest upstream |
| 235 | version as well) or backport all stable bugfixes to old stable versions, |
Gao Xiang | ac8f789 | 2020-11-21 10:26:23 +0800 | [diff] [blame] | 236 | e.g. our unofficial lz4 fork: https://github.com/erofs/lz4 |
Gao Xiang | 68d9b69 | 2019-10-23 17:20:31 +0800 | [diff] [blame] | 237 | |
Gao Xiang | 8d6777c | 2020-11-21 10:26:22 +0800 | [diff] [blame] | 238 | [4] LZ4HC didn't compress long zeroed buffer properly with |
| 239 | LZ4_compress_HC_destSize() |
| 240 | https://github.com/lz4/lz4/issues/784 |
| 241 | |
| 242 | which has been resolved in |
| 243 | https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1 |
| 244 | |
| 245 | and already included in lz4-1.9.3, see: |
| 246 | https://github.com/lz4/lz4/releases/tag/v1.9.3 |
| 247 | |
Gao Xiang | 9555d9e | 2020-12-05 23:41:41 +0800 | [diff] [blame] | 248 | [5] LZ4_decompress_safe_partial is broken in 1.9.2 |
| 249 | https://github.com/lz4/lz4/issues/783 |
| 250 | |
| 251 | which is also resolved in lz4-1.9.3. |
Gao Xiang | aa7d722 | 2021-10-30 10:01:16 +0800 | [diff] [blame] | 252 | |
| 253 | [6] https://tukaani.org/xz/xz-5.3.2alpha.tar.xz |