Teach Clang how to use response files when calling other tools
Patch by Rafael Auler!
This patch addresses PR15171 and teaches Clang how to call other tools
with response files, when the command line exceeds system limits. This
is a problem for Windows systems, whose maximum command-line length is
32kb.
I introduce the concept of "response file support" for each Tool object.
A given Tool may have full support for response files (e.g. MSVC's
link.exe) or only support file names inside response files, but no flags
(e.g. Apple's ld64, as commented in PR15171), or no support at all (the
default case). Therefore, if you implement a toolchain in the clang
driver and you want clang to be able to use response files in your
tools, you must override a method (getReponseFileSupport()) to tell so.
I designed it to support different kinds of tools and
internationalisation needs:
- VS response files ( UTF-16 )
- GNU tools ( uses system's current code page, windows' legacy intl.
support, with escaped backslashes. On unix, fallback to UTF-8 )
- Clang itself ( UTF-16 on windows, UTF-8 on unix )
- ld64 response files ( only a limited file list, UTF-8 on unix )
With this design, I was able to test input file names with spaces and
international characters for Windows. When the linker input is large
enough, it creates a response file with the correct encoding. On a Mac,
to test ld64, I temporarily changed Clang's behavior to always use
response files regardless of the command size limit (avoiding using huge
command line inputs). I tested clang with the LLVM test suite (compiling
benchmarks) and it did fine.
Test Plan: A LIT test that tests proper response files support. This is
tricky, since, for Unix systems, we need a 2MB response file, otherwise
Clang will simply use regular arguments instead of a response file. To
do this, my LIT test generate the file on the fly by cloning many -DTEST
parameters until we have a 2MB file. I found out that processing 2MB of
arguments is pretty slow, it takes 1 minute using my notebook in a debug
build, or 10s in a Release build. Therefore, I also added "REQUIRES:
long_tests", so it will only run when the user wants to run long tests.
In the full discussion in
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20130408/171463.html,
Rafael Espindola discusses a proper way to test
llvm::sys::argumentsFitWithinSystemLimits(), and, there, Chandler
suggests to use 10 times the current system limit (20MB resp file), so
we guarantee that the system will always use response file, even if a
new linux comes up that can handle a few more bytes of arguments.
However, by testing with a 20MB resp file, the test takes long 8 minutes
just to perform a silly check to see if the driver will use a response
file. I found it to be unreasonable. Thus, I discarded this approach and
uses a 2MB response file, which should be enough.
Reviewers: asl, rafael, silvas
Reviewed By: silvas
Subscribers: silvas, rnk, thakis, cfe-commits
Differential Revision: http://reviews.llvm.org/D4897
llvm-svn: 217792
diff --git a/clang/lib/Driver/Tools.h b/clang/lib/Driver/Tools.h
index 9579341..8ab1456 100644
--- a/clang/lib/Driver/Tools.h
+++ b/clang/lib/Driver/Tools.h
@@ -90,7 +90,7 @@
mutable std::unique_ptr<visualstudio::Compile> CLFallback;
public:
- Clang(const ToolChain &TC) : Tool("clang", "clang frontend", TC) {}
+ Clang(const ToolChain &TC) : Tool("clang", "clang frontend", TC, RF_Full) {}
bool hasGoodDiagnostics() const override { return true; }
bool hasIntegratedAssembler() const override { return true; }
@@ -106,7 +106,8 @@
class LLVM_LIBRARY_VISIBILITY ClangAs : public Tool {
public:
ClangAs(const ToolChain &TC) : Tool("clang::as",
- "clang integrated assembler", TC) {}
+ "clang integrated assembler", TC,
+ RF_Full) {}
bool hasGoodDiagnostics() const override { return true; }
bool hasIntegratedAssembler() const override { return false; }
@@ -118,12 +119,22 @@
const char *LinkingOutput) const override;
};
+ /// \brief Base class for all GNU tools that provide the same behavior when
+ /// it comes to response files support
+ class GnuTool : public Tool {
+ virtual void anchor();
+
+ public:
+ GnuTool(const char *Name, const char *ShortName, const ToolChain &TC)
+ : Tool(Name, ShortName, TC, RF_Full, llvm::sys::WEM_CurrentCodePage) {}
+ };
+
/// gcc - Generic GCC tool implementations.
namespace gcc {
- class LLVM_LIBRARY_VISIBILITY Common : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Common : public GnuTool {
public:
Common(const char *Name, const char *ShortName,
- const ToolChain &TC) : Tool(Name, ShortName, TC) {}
+ const ToolChain &TC) : GnuTool(Name, ShortName, TC) {}
void ConstructJob(Compilation &C, const JobAction &JA,
const InputInfo &Output,
@@ -178,9 +189,9 @@
namespace hexagon {
// For Hexagon, we do not need to instantiate tools for PreProcess, PreCompile and Compile.
// We simply use "clang -cc1" for those actions.
- class LLVM_LIBRARY_VISIBILITY Assemble : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool {
public:
- Assemble(const ToolChain &TC) : Tool("hexagon::Assemble",
+ Assemble(const ToolChain &TC) : GnuTool("hexagon::Assemble",
"hexagon-as", TC) {}
bool hasIntegratedCPP() const override { return false; }
@@ -193,9 +204,9 @@
const char *LinkingOutput) const override;
};
- class LLVM_LIBRARY_VISIBILITY Link : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Link : public GnuTool {
public:
- Link(const ToolChain &TC) : Tool("hexagon::Link",
+ Link(const ToolChain &TC) : GnuTool("hexagon::Link",
"hexagon-ld", TC) {}
bool hasIntegratedCPP() const override { return false; }
@@ -248,8 +259,13 @@
}
public:
- MachOTool(const char *Name, const char *ShortName,
- const ToolChain &TC) : Tool(Name, ShortName, TC) {}
+ MachOTool(
+ const char *Name, const char *ShortName, const ToolChain &TC,
+ ResponseFileSupport ResponseSupport = RF_None,
+ llvm::sys::WindowsEncodingMethod ResponseEncoding = llvm::sys::WEM_UTF8,
+ const char *ResponseFlag = "@")
+ : Tool(Name, ShortName, TC, ResponseSupport, ResponseEncoding,
+ ResponseFlag) {}
};
class LLVM_LIBRARY_VISIBILITY Assemble : public MachOTool {
@@ -272,7 +288,9 @@
const InputInfoList &Inputs) const;
public:
- Link(const ToolChain &TC) : MachOTool("darwin::Link", "linker", TC) {}
+ Link(const ToolChain &TC) : MachOTool("darwin::Link", "linker", TC,
+ RF_FileList, llvm::sys::WEM_UTF8,
+ "-filelist") {}
bool hasIntegratedCPP() const override { return false; }
bool isLinkJob() const override { return true; }
@@ -327,9 +345,9 @@
/// openbsd -- Directly call GNU Binutils assembler and linker
namespace openbsd {
- class LLVM_LIBRARY_VISIBILITY Assemble : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool {
public:
- Assemble(const ToolChain &TC) : Tool("openbsd::Assemble", "assembler",
+ Assemble(const ToolChain &TC) : GnuTool("openbsd::Assemble", "assembler",
TC) {}
bool hasIntegratedCPP() const override { return false; }
@@ -340,9 +358,9 @@
const llvm::opt::ArgList &TCArgs,
const char *LinkingOutput) const override;
};
- class LLVM_LIBRARY_VISIBILITY Link : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Link : public GnuTool {
public:
- Link(const ToolChain &TC) : Tool("openbsd::Link", "linker", TC) {}
+ Link(const ToolChain &TC) : GnuTool("openbsd::Link", "linker", TC) {}
bool hasIntegratedCPP() const override { return false; }
bool isLinkJob() const override { return true; }
@@ -356,9 +374,9 @@
/// bitrig -- Directly call GNU Binutils assembler and linker
namespace bitrig {
- class LLVM_LIBRARY_VISIBILITY Assemble : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool {
public:
- Assemble(const ToolChain &TC) : Tool("bitrig::Assemble", "assembler",
+ Assemble(const ToolChain &TC) : GnuTool("bitrig::Assemble", "assembler",
TC) {}
bool hasIntegratedCPP() const override { return false; }
@@ -368,9 +386,9 @@
const llvm::opt::ArgList &TCArgs,
const char *LinkingOutput) const override;
};
- class LLVM_LIBRARY_VISIBILITY Link : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Link : public GnuTool {
public:
- Link(const ToolChain &TC) : Tool("bitrig::Link", "linker", TC) {}
+ Link(const ToolChain &TC) : GnuTool("bitrig::Link", "linker", TC) {}
bool hasIntegratedCPP() const override { return false; }
bool isLinkJob() const override { return true; }
@@ -384,9 +402,9 @@
/// freebsd -- Directly call GNU Binutils assembler and linker
namespace freebsd {
- class LLVM_LIBRARY_VISIBILITY Assemble : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool {
public:
- Assemble(const ToolChain &TC) : Tool("freebsd::Assemble", "assembler",
+ Assemble(const ToolChain &TC) : GnuTool("freebsd::Assemble", "assembler",
TC) {}
bool hasIntegratedCPP() const override { return false; }
@@ -396,9 +414,9 @@
const llvm::opt::ArgList &TCArgs,
const char *LinkingOutput) const override;
};
- class LLVM_LIBRARY_VISIBILITY Link : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Link : public GnuTool {
public:
- Link(const ToolChain &TC) : Tool("freebsd::Link", "linker", TC) {}
+ Link(const ToolChain &TC) : GnuTool("freebsd::Link", "linker", TC) {}
bool hasIntegratedCPP() const override { return false; }
bool isLinkJob() const override { return true; }
@@ -412,11 +430,11 @@
/// netbsd -- Directly call GNU Binutils assembler and linker
namespace netbsd {
- class LLVM_LIBRARY_VISIBILITY Assemble : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool {
public:
Assemble(const ToolChain &TC)
- : Tool("netbsd::Assemble", "assembler", TC) {}
+ : GnuTool("netbsd::Assemble", "assembler", TC) {}
bool hasIntegratedCPP() const override { return false; }
@@ -425,11 +443,11 @@
const llvm::opt::ArgList &TCArgs,
const char *LinkingOutput) const override;
};
- class LLVM_LIBRARY_VISIBILITY Link : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Link : public GnuTool {
public:
Link(const ToolChain &TC)
- : Tool("netbsd::Link", "linker", TC) {}
+ : GnuTool("netbsd::Link", "linker", TC) {}
bool hasIntegratedCPP() const override { return false; }
bool isLinkJob() const override { return true; }
@@ -443,9 +461,9 @@
/// Directly call GNU Binutils' assembler and linker.
namespace gnutools {
- class LLVM_LIBRARY_VISIBILITY Assemble : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool {
public:
- Assemble(const ToolChain &TC) : Tool("GNU::Assemble", "assembler", TC) {}
+ Assemble(const ToolChain &TC) : GnuTool("GNU::Assemble", "assembler", TC) {}
bool hasIntegratedCPP() const override { return false; }
@@ -455,9 +473,9 @@
const llvm::opt::ArgList &TCArgs,
const char *LinkingOutput) const override;
};
- class LLVM_LIBRARY_VISIBILITY Link : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Link : public GnuTool {
public:
- Link(const ToolChain &TC) : Tool("GNU::Link", "linker", TC) {}
+ Link(const ToolChain &TC) : GnuTool("GNU::Link", "linker", TC) {}
bool hasIntegratedCPP() const override { return false; }
bool isLinkJob() const override { return true; }
@@ -471,9 +489,9 @@
}
/// minix -- Directly call GNU Binutils assembler and linker
namespace minix {
- class LLVM_LIBRARY_VISIBILITY Assemble : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool {
public:
- Assemble(const ToolChain &TC) : Tool("minix::Assemble", "assembler",
+ Assemble(const ToolChain &TC) : GnuTool("minix::Assemble", "assembler",
TC) {}
bool hasIntegratedCPP() const override { return false; }
@@ -484,9 +502,9 @@
const llvm::opt::ArgList &TCArgs,
const char *LinkingOutput) const override;
};
- class LLVM_LIBRARY_VISIBILITY Link : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Link : public GnuTool {
public:
- Link(const ToolChain &TC) : Tool("minix::Link", "linker", TC) {}
+ Link(const ToolChain &TC) : GnuTool("minix::Link", "linker", TC) {}
bool hasIntegratedCPP() const override { return false; }
bool isLinkJob() const override { return true; }
@@ -529,9 +547,9 @@
/// dragonfly -- Directly call GNU Binutils assembler and linker
namespace dragonfly {
- class LLVM_LIBRARY_VISIBILITY Assemble : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Assemble : public GnuTool {
public:
- Assemble(const ToolChain &TC) : Tool("dragonfly::Assemble", "assembler",
+ Assemble(const ToolChain &TC) : GnuTool("dragonfly::Assemble", "assembler",
TC) {}
bool hasIntegratedCPP() const override { return false; }
@@ -541,9 +559,9 @@
const llvm::opt::ArgList &TCArgs,
const char *LinkingOutput) const override;
};
- class LLVM_LIBRARY_VISIBILITY Link : public Tool {
+ class LLVM_LIBRARY_VISIBILITY Link : public GnuTool {
public:
- Link(const ToolChain &TC) : Tool("dragonfly::Link", "linker", TC) {}
+ Link(const ToolChain &TC) : GnuTool("dragonfly::Link", "linker", TC) {}
bool hasIntegratedCPP() const override { return false; }
bool isLinkJob() const override { return true; }
@@ -560,7 +578,8 @@
namespace visualstudio {
class LLVM_LIBRARY_VISIBILITY Link : public Tool {
public:
- Link(const ToolChain &TC) : Tool("visualstudio::Link", "linker", TC) {}
+ Link(const ToolChain &TC) : Tool("visualstudio::Link", "linker", TC,
+ RF_Full, llvm::sys::WEM_UTF16) {}
bool hasIntegratedCPP() const override { return false; }
bool isLinkJob() const override { return true; }
@@ -573,7 +592,8 @@
class LLVM_LIBRARY_VISIBILITY Compile : public Tool {
public:
- Compile(const ToolChain &TC) : Tool("visualstudio::Compile", "compiler", TC) {}
+ Compile(const ToolChain &TC) : Tool("visualstudio::Compile", "compiler", TC,
+ RF_Full, llvm::sys::WEM_UTF16) {}
bool hasIntegratedAssembler() const override { return true; }
bool hasIntegratedCPP() const override { return true; }