| commit | 68142324290f2932df0e271747cdccc371d6dded | [log] [tgz] |
|---|---|---|
| author | Nick Terrell <terrelln@fb.com> | Tue Oct 29 15:46:22 2019 -0700 |
| committer | Fangrui Song <maskray@google.com> | Tue Oct 29 15:49:08 2019 -0700 |
| tree | da4fba934c980ecbe120819fc6ae67e9906036e2 | |
| parent | dbcb690fb78193e99452748c4af72eccb262e4e2 [diff] |
[LLD][ELF] Support --[no-]mmap-output-file with F_no_mmap Summary: Add a flag `F_no_mmap` to `FileOutputBuffer` to support `--[no-]mmap-output-file` in ELF LLD. LLD currently explicitly ignores this flag for compatibility with GNU ld and gold. We need this flag to speed up link time for large binaries in certain scenarios. When we link some of our larger binaries we find that LLD takes 50+ GB of memory, which causes memory pressure. The memory pressure causes the VM to flush dirty pages of the output file to disk. This is normally okay, since we should be flushing cold pages. However, when using BtrFS with compression we need to write 128KB at a time when we flush a page. If any page in that 128KB block is written again, then it must be flushed a second time, and so on. Since LLD doesn't write sequentially this causes write amplification. The same 128KB block will end up being flushed multiple times, causing the linker to many times more IO than necessary. We've observed 3-5x faster builds with -no-mmap-output-file when we hit this scenario. The bad scenario only applies to compressed filesystems, which group together multiple pages into a single compressed block. I've tested BtrFS, but the problem will be present for any compressed filesystem on Linux, since it is caused by the VM. Silently ignoring --no-mmap-output-file caused a silent regression when we switched from gold to lld. We pass --no-mmap-output-file to fix this edge case, but since lld silently ignored the flag we didn't realize it wasn't being respected. Benchmark building a 9 GB binary that exposes this edge case. I linked 3 times with --mmap-output-file and 3 times with --no-mmap-output-file and took the average. The machine has 24 cores @ 2.4 GHz, 112 GB of RAM, BtrFS mounted with -compress-force=zstd, and an 80% full disk. | Mode | Time | |---------|-------| | mmap | 894 s | | no mmap | 126 s | When compression is disabled, BtrFS performs just as well with and without mmap on this benchmark. I was unable to reproduce the regression with any binaries in lld-speed-test. Reviewed By: ruiu, MaskRay Differential Revision: https://reviews.llvm.org/D69294
This directory and its subdirectories contain source code for LLVM, a toolkit for the construction of highly optimized compilers, optimizers, and runtime environments.
Taken from https://llvm.org/docs/GettingStarted.html.
Welcome to the LLVM project!
The LLVM project has multiple components. The core of the project is itself called "LLVM". This contains all of the tools, libraries, and header files needed to process intermediate representations and converts it into object files. Tools include an assembler, disassembler, bitcode analyzer, and bitcode optimizer. It also contains basic regression tests.
C-like languages use the Clang front end. This component compiles C, C++, Objective C, and Objective C++ code into LLVM bitcode -- and from there into object files, using LLVM.
Other components include: the libc++ C++ standard library, the LLD linker, and more.
The LLVM Getting Started documentation may be out of date. The Clang Getting Started page might have more accurate information.
This is an example workflow and configuration to get and build the LLVM source:
Checkout LLVM (including related subprojects like Clang):
git clone https://github.com/llvm/llvm-project.git
Or, on windows, git clone --config core.autocrlf=false https://github.com/llvm/llvm-project.git
Configure and build LLVM and Clang:
cd llvm-project
mkdir build
cd build
cmake -G <generator> [options] ../llvm
Some common generators are:
Ninja --- for generating Ninja build files. Most llvm developers use Ninja.Unix Makefiles --- for generating make-compatible parallel makefiles.Visual Studio --- for generating Visual Studio projects and solutions.Xcode --- for generating Xcode projects.Some Common options:
-DLLVM_ENABLE_PROJECTS='...' --- semicolon-separated list of the LLVM subprojects you'd like to additionally build. Can include any of: clang, clang-tools-extra, libcxx, libcxxabi, libunwind, lldb, compiler-rt, lld, polly, or debuginfo-tests.
For example, to build LLVM, Clang, libcxx, and libcxxabi, use -DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi".
-DCMAKE_INSTALL_PREFIX=directory --- Specify for directory the full pathname of where you want the LLVM tools and libraries to be installed (default /usr/local).
-DCMAKE_BUILD_TYPE=type --- Valid options for type are Debug, Release, RelWithDebInfo, and MinSizeRel. Default is Debug.
-DLLVM_ENABLE_ASSERTIONS=On --- Compile with assertion checks enabled (default is Yes for Debug builds, No for all other build types).
Run your build tool of choice!
The default target (i.e. ninja or make) will build all of LLVM.
The check-all target (i.e. ninja check-all) will run the regression tests to ensure everything is in working order.
CMake will generate build targets for each tool and library, and most LLVM sub-projects generate their own check-<project> target.
Running a serial build will be slow. To improve speed, try running a parallel build. That's done by default in Ninja; for make, use make -j NNN (NNN is the number of parallel jobs, use e.g. number of CPUs you have.)
For more information see CMake
Consult the Getting Started with LLVM page for detailed information on configuring and compiling LLVM. You can visit Directory Layout to learn about the layout of the source code tree.