[ORC] Add generic initializer/deinitializer support.
Initializers and deinitializers are used to implement C++ static constructors
and destructors, runtime registration for some languages (e.g. with the
Objective-C runtime for Objective-C/C++ code) and other tasks that would
typically be performed when a shared-object/dylib is loaded or unloaded by a
statically compiled program.
MCJIT and ORC have historically provided limited support for discovering and
running initializers/deinitializers by scanning the llvm.global_ctors and
llvm.global_dtors variables and recording the functions to be run. This approach
suffers from several drawbacks: (1) It only works for IR inputs, not for object
files (including cached JIT'd objects). (2) It only works for initializers
described by llvm.global_ctors and llvm.global_dtors, however not all
initializers are described in this way (Objective-C, for example, describes
initializers via specially named metadata sections). (3) To make the
initializer/deinitializer functions described by llvm.global_ctors and
llvm.global_dtors searchable they must be promoted to extern linkage, polluting
the JIT symbol table (extra care must be taken to ensure this promotion does
not result in symbol name clashes).
This patch introduces several interdependent changes to ORCv2 to support the
construction of new initialization schemes, and includes an implementation of a
backwards-compatible llvm.global_ctor/llvm.global_dtor scanning scheme, and a
MachO specific scheme that handles Objective-C runtime registration (if the
Objective-C runtime is available) enabling execution of LLVM IR compiled from
Objective-C and Swift.
The major changes included in this patch are:
(1) The MaterializationUnit and MaterializationResponsibility classes are
extended to describe an optional "initializer" symbol for the module (see the
getInitializerSymbol method on each class). The presence or absence of this
symbol indicates whether the module contains any initializers or
deinitializers. The initializer symbol otherwise behaves like any other:
searching for it triggers materialization.
(2) A new Platform interface is introduced in llvm/ExecutionEngine/Orc/Core.h
which provides the following callback interface:
- Error setupJITDylib(JITDylib &JD): Can be used to install standard symbols
in JITDylibs upon creation. E.g. __dso_handle.
- Error notifyAdding(JITDylib &JD, const MaterializationUnit &MU): Generally
used to record initializer symbols.
- Error notifyRemoving(JITDylib &JD, VModuleKey K): Used to notify a platform
that a module is being removed.
Platform implementations can use these callbacks to track outstanding
initializers and implement a platform-specific approach for executing them. For
example, the MachOPlatform installs a plugin in the JIT linker to scan for both
__mod_inits sections (for C++ static constructors) and ObjC metadata sections.
If discovered, these are processed in the usual platform order: Objective-C
registration is carried out first, then static initializers are executed,
ensuring that calls to Objective-C from static initializers will be safe.
This patch updates LLJIT to use the new scheme for initialization. Two
LLJIT::PlatformSupport classes are implemented: A GenericIR platform and a MachO
platform. The GenericIR platform implements a modified version of the previous
llvm.global-ctor scraping scheme to provide support for Windows and
Linux. LLJIT's MachO platform uses the MachOPlatform class to provide MachO
specific initialization as described above.
Reviewers: sgraenitz, dblaikie
Subscribers: mgorny, hiraditya, mgrang, ributzka, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D74300
diff --git a/llvm/lib/ExecutionEngine/Orc/CMakeLists.txt b/llvm/lib/ExecutionEngine/Orc/CMakeLists.txt
index e3a7a67..7eb2742 100644
--- a/llvm/lib/ExecutionEngine/Orc/CMakeLists.txt
+++ b/llvm/lib/ExecutionEngine/Orc/CMakeLists.txt
@@ -12,6 +12,8 @@
Legacy.cpp
Layer.cpp
LLJIT.cpp
+ MachOPlatform.cpp
+ Mangling.cpp
NullResolver.cpp
ObjectLinkingLayer.cpp
ObjectTransformLayer.cpp
diff --git a/llvm/lib/ExecutionEngine/Orc/CompileOnDemandLayer.cpp b/llvm/lib/ExecutionEngine/Orc/CompileOnDemandLayer.cpp
index 29d18b6..f22ae01 100644
--- a/llvm/lib/ExecutionEngine/Orc/CompileOnDemandLayer.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/CompileOnDemandLayer.cpp
@@ -7,6 +7,7 @@
//===----------------------------------------------------------------------===//
#include "llvm/ExecutionEngine/Orc/CompileOnDemandLayer.h"
+#include "llvm/ExecutionEngine/Orc/ExecutionUtils.h"
#include "llvm/IR/Mangler.h"
#include "llvm/IR/Module.h"
@@ -68,18 +69,18 @@
class PartitioningIRMaterializationUnit : public IRMaterializationUnit {
public:
PartitioningIRMaterializationUnit(ExecutionSession &ES,
- const ManglingOptions &MO,
+ const IRSymbolMapper::ManglingOptions &MO,
ThreadSafeModule TSM, VModuleKey K,
CompileOnDemandLayer &Parent)
: IRMaterializationUnit(ES, MO, std::move(TSM), std::move(K)),
Parent(Parent) {}
PartitioningIRMaterializationUnit(
- ThreadSafeModule TSM, SymbolFlagsMap SymbolFlags,
- SymbolNameToDefinitionMap SymbolToDefinition,
+ ThreadSafeModule TSM, VModuleKey K, SymbolFlagsMap SymbolFlags,
+ SymbolStringPtr InitSymbol, SymbolNameToDefinitionMap SymbolToDefinition,
CompileOnDemandLayer &Parent)
: IRMaterializationUnit(std::move(TSM), std::move(K),
- std::move(SymbolFlags),
+ std::move(SymbolFlags), std::move(InitSymbol),
std::move(SymbolToDefinition)),
Parent(Parent) {}
@@ -172,21 +173,23 @@
auto I = DylibResources.find(&TargetD);
if (I == DylibResources.end()) {
auto &ImplD =
- getExecutionSession().createJITDylib(TargetD.getName() + ".impl");
+ getExecutionSession().createBareJITDylib(TargetD.getName() + ".impl");
+ JITDylibSearchOrder NewSearchOrder;
TargetD.withSearchOrderDo(
[&](const JITDylibSearchOrder &TargetSearchOrder) {
- auto NewSearchOrder = TargetSearchOrder;
- assert(
- !NewSearchOrder.empty() &&
- NewSearchOrder.front().first == &TargetD &&
- NewSearchOrder.front().second ==
- JITDylibLookupFlags::MatchAllSymbols &&
- "TargetD must be at the front of its own search order and match "
- "non-exported symbol");
- NewSearchOrder.insert(std::next(NewSearchOrder.begin()),
- {&ImplD, JITDylibLookupFlags::MatchAllSymbols});
- ImplD.setSearchOrder(std::move(NewSearchOrder), false);
+ NewSearchOrder = TargetSearchOrder;
});
+
+ assert(
+ !NewSearchOrder.empty() && NewSearchOrder.front().first == &TargetD &&
+ NewSearchOrder.front().second == JITDylibLookupFlags::MatchAllSymbols &&
+ "TargetD must be at the front of its own search order and match "
+ "non-exported symbol");
+ NewSearchOrder.insert(std::next(NewSearchOrder.begin()),
+ {&ImplD, JITDylibLookupFlags::MatchAllSymbols});
+ ImplD.setSearchOrder(NewSearchOrder, false);
+ TargetD.setSearchOrder(std::move(NewSearchOrder), false);
+
PerDylibResources PDR(ImplD, BuildIndirectStubsManager());
I = DylibResources.insert(std::make_pair(&TargetD, std::move(PDR))).first;
}
@@ -251,8 +254,15 @@
auto &ES = getExecutionSession();
GlobalValueSet RequestedGVs;
for (auto &Name : R.getRequestedSymbols()) {
- assert(Defs.count(Name) && "No definition for symbol");
- RequestedGVs.insert(Defs[Name]);
+ if (Name == R.getInitializerSymbol())
+ TSM.withModuleDo([&](Module &M) {
+ for (auto &GV : getStaticInitGVs(M))
+ RequestedGVs.insert(&GV);
+ });
+ else {
+ assert(Defs.count(Name) && "No definition for symbol");
+ RequestedGVs.insert(Defs[Name]);
+ }
}
/// Perform partitioning with the context lock held, since the partition
@@ -272,7 +282,8 @@
// If the partition is empty, return the whole module to the symbol table.
if (GVsToExtract->empty()) {
R.replace(std::make_unique<PartitioningIRMaterializationUnit>(
- std::move(TSM), R.getSymbols(), std::move(Defs), *this));
+ std::move(TSM), R.getVModuleKey(), R.getSymbols(),
+ R.getInitializerSymbol(), std::move(Defs), *this));
return;
}
diff --git a/llvm/lib/ExecutionEngine/Orc/CompileUtils.cpp b/llvm/lib/ExecutionEngine/Orc/CompileUtils.cpp
index 160e5ba..f8efed1 100644
--- a/llvm/lib/ExecutionEngine/Orc/CompileUtils.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/CompileUtils.cpp
@@ -24,9 +24,9 @@
namespace llvm {
namespace orc {
-IRMaterializationUnit::ManglingOptions
+IRSymbolMapper::ManglingOptions
irManglingOptionsFromTargetOptions(const TargetOptions &Opts) {
- IRMaterializationUnit::ManglingOptions MO;
+ IRSymbolMapper::ManglingOptions MO;
MO.EmulatedTLS = Opts.EmulatedTLS;
diff --git a/llvm/lib/ExecutionEngine/Orc/Core.cpp b/llvm/lib/ExecutionEngine/Orc/Core.cpp
index a7af34a..6a0df10 100644
--- a/llvm/lib/ExecutionEngine/Orc/Core.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/Core.cpp
@@ -11,7 +11,6 @@
#include "llvm/ADT/STLExtras.h"
#include "llvm/Config/llvm-config.h"
#include "llvm/ExecutionEngine/Orc/OrcError.h"
-#include "llvm/IR/Mangler.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/Format.h"
@@ -421,12 +420,6 @@
QueryRegistrations.clear();
}
-MaterializationResponsibility::MaterializationResponsibility(
- JITDylib &JD, SymbolFlagsMap SymbolFlags, VModuleKey K)
- : JD(JD), SymbolFlags(std::move(SymbolFlags)), K(std::move(K)) {
- assert(!this->SymbolFlags.empty() && "Materializing nothing?");
-}
-
MaterializationResponsibility::~MaterializationResponsibility() {
assert(SymbolFlags.empty() &&
"All symbols should have been explicitly materialized or failed");
@@ -501,9 +494,13 @@
void MaterializationResponsibility::replace(
std::unique_ptr<MaterializationUnit> MU) {
+
for (auto &KV : MU->getSymbols())
SymbolFlags.erase(KV.first);
+ if (MU->getInitializerSymbol() == InitSymbol)
+ InitSymbol = nullptr;
+
LLVM_DEBUG(JD.getExecutionSession().runSessionLocked([&]() {
dbgs() << "In " << JD.getName() << " replacing symbols with " << *MU
<< "\n";
@@ -519,6 +516,7 @@
if (NewKey == VModuleKey())
NewKey = K;
+ SymbolStringPtr DelegatedInitSymbol;
SymbolFlagsMap DelegatedFlags;
for (auto &Name : Symbols) {
@@ -528,10 +526,14 @@
"instance");
DelegatedFlags[Name] = std::move(I->second);
+ if (Name == InitSymbol)
+ std::swap(InitSymbol, DelegatedInitSymbol);
+
SymbolFlags.erase(I);
}
return MaterializationResponsibility(JD, std::move(DelegatedFlags),
+ std::move(DelegatedInitSymbol),
std::move(NewKey));
}
@@ -550,7 +552,7 @@
AbsoluteSymbolsMaterializationUnit::AbsoluteSymbolsMaterializationUnit(
SymbolMap Symbols, VModuleKey K)
- : MaterializationUnit(extractFlags(Symbols), std::move(K)),
+ : MaterializationUnit(extractFlags(Symbols), nullptr, std::move(K)),
Symbols(std::move(Symbols)) {}
StringRef AbsoluteSymbolsMaterializationUnit::getName() const {
@@ -581,7 +583,7 @@
ReExportsMaterializationUnit::ReExportsMaterializationUnit(
JITDylib *SourceJD, JITDylibLookupFlags SourceJDLookupFlags,
SymbolAliasMap Aliases, VModuleKey K)
- : MaterializationUnit(extractFlags(Aliases), std::move(K)),
+ : MaterializationUnit(extractFlags(Aliases), nullptr, std::move(K)),
SourceJD(SourceJD), SourceJDLookupFlags(SourceJDLookupFlags),
Aliases(std::move(Aliases)) {}
@@ -972,7 +974,7 @@
// Assert that this symbol exists and has not reached the ready state
// already.
assert(OtherSymI != OtherJITDylib.Symbols.end() &&
- (OtherSymI->second.getState() != SymbolState::Ready &&
+ (OtherSymI->second.getState() < SymbolState::Ready &&
"Dependency on emitted/ready symbol"));
#endif
@@ -1101,6 +1103,7 @@
Error JITDylib::emit(const SymbolFlagsMap &Emitted) {
AsynchronousSymbolQuerySet CompletedQueries;
SymbolNameSet SymbolsInErrorState;
+ DenseMap<JITDylib *, SymbolNameVector> ReadySymbols;
ES.runSessionLocked([&, this]() {
std::vector<SymbolTable::iterator> Worklist;
@@ -1145,6 +1148,7 @@
// dependencies) then notify any pending queries.
for (auto &KV : MI.Dependants) {
auto &DependantJD = *KV.first;
+ auto &DependantJDReadySymbols = ReadySymbols[&DependantJD];
for (auto &DependantName : KV.second) {
auto DependantMII =
DependantJD.MaterializingInfos.find(DependantName);
@@ -1184,6 +1188,7 @@
// Since this dependant is now ready, we erase its MaterializingInfo
// and update its materializing state.
DependantSymEntry.setState(SymbolState::Ready);
+ DependantJDReadySymbols.push_back(DependantName);
for (auto &Q : DependantMI.takeQueriesMeeting(SymbolState::Ready)) {
Q->notifySymbolMetRequiredState(
@@ -1192,22 +1197,21 @@
CompletedQueries.insert(Q);
Q->removeQueryDependence(DependantJD, DependantName);
}
-
- DependantJD.MaterializingInfos.erase(DependantMII);
}
}
}
+ auto &ThisJDReadySymbols = ReadySymbols[this];
MI.Dependants.clear();
if (MI.UnemittedDependencies.empty()) {
SymI->second.setState(SymbolState::Ready);
+ ThisJDReadySymbols.push_back(Name);
for (auto &Q : MI.takeQueriesMeeting(SymbolState::Ready)) {
Q->notifySymbolMetRequiredState(Name, SymI->second.getSymbol());
if (Q->isComplete())
CompletedQueries.insert(Q);
Q->removeQueryDependence(*this, Name);
}
- MaterializingInfos.erase(MII);
}
}
});
@@ -1882,6 +1886,57 @@
}
}
+Platform::~Platform() {}
+
+Expected<DenseMap<JITDylib *, SymbolMap>> Platform::lookupInitSymbols(
+ ExecutionSession &ES,
+ const DenseMap<JITDylib *, SymbolLookupSet> &InitSyms) {
+
+ DenseMap<JITDylib *, SymbolMap> CompoundResult;
+ Error CompoundErr = Error::success();
+ std::mutex LookupMutex;
+ std::condition_variable CV;
+ uint64_t Count = InitSyms.size();
+
+ LLVM_DEBUG({
+ dbgs() << "Issuing init-symbol lookup:\n";
+ for (auto &KV : InitSyms)
+ dbgs() << " " << KV.first->getName() << ": " << KV.second << "\n";
+ });
+
+ for (auto &KV : InitSyms) {
+ auto *JD = KV.first;
+ auto Names = std::move(KV.second);
+ ES.lookup(
+ LookupKind::Static,
+ JITDylibSearchOrder({{JD, JITDylibLookupFlags::MatchAllSymbols}}),
+ std::move(Names), SymbolState::Ready,
+ [&, JD](Expected<SymbolMap> Result) {
+ {
+ std::lock_guard<std::mutex> Lock(LookupMutex);
+ --Count;
+ if (Result) {
+ assert(!CompoundResult.count(JD) &&
+ "Duplicate JITDylib in lookup?");
+ CompoundResult[JD] = std::move(*Result);
+ } else
+ CompoundErr =
+ joinErrors(std::move(CompoundErr), Result.takeError());
+ }
+ CV.notify_one();
+ },
+ NoDependenciesToRegister);
+ }
+
+ std::unique_lock<std::mutex> Lock(LookupMutex);
+ CV.wait(Lock, [&] { return Count == 0 || CompoundErr; });
+
+ if (CompoundErr)
+ return std::move(CompoundErr);
+
+ return std::move(CompoundResult);
+}
+
ExecutionSession::ExecutionSession(std::shared_ptr<SymbolStringPool> SSP)
: SSP(SSP ? std::move(SSP) : std::make_shared<SymbolStringPool>()) {
}
@@ -1895,7 +1950,7 @@
});
}
-JITDylib &ExecutionSession::createJITDylib(std::string Name) {
+JITDylib &ExecutionSession::createBareJITDylib(std::string Name) {
assert(!getJITDylibByName(Name) && "JITDylib with that name already exists");
return runSessionLocked([&, this]() -> JITDylib & {
JDs.push_back(
@@ -1904,6 +1959,14 @@
});
}
+Expected<JITDylib &> ExecutionSession::createJITDylib(std::string Name) {
+ auto &JD = createBareJITDylib(Name);
+ if (P)
+ if (auto Err = P->setupJITDylib(JD))
+ return std::move(Err);
+ return JD;
+}
+
void ExecutionSession::legacyFailQuery(AsynchronousSymbolQuery &Q, Error Err) {
assert(!!Err && "Error should be in failure state");
@@ -2144,11 +2207,11 @@
Expected<JITEvaluatedSymbol>
ExecutionSession::lookup(const JITDylibSearchOrder &SearchOrder,
- SymbolStringPtr Name) {
+ SymbolStringPtr Name, SymbolState RequiredState) {
SymbolLookupSet Names({Name});
if (auto ResultMap = lookup(SearchOrder, std::move(Names), LookupKind::Static,
- SymbolState::Ready, NoDependenciesToRegister)) {
+ RequiredState, NoDependenciesToRegister)) {
assert(ResultMap->size() == 1 && "Unexpected number of results");
assert(ResultMap->count(Name) && "Missing result for symbol");
return std::move(ResultMap->begin()->second);
@@ -2157,14 +2220,15 @@
}
Expected<JITEvaluatedSymbol>
-ExecutionSession::lookup(ArrayRef<JITDylib *> SearchOrder,
- SymbolStringPtr Name) {
- return lookup(makeJITDylibSearchOrder(SearchOrder), Name);
+ExecutionSession::lookup(ArrayRef<JITDylib *> SearchOrder, SymbolStringPtr Name,
+ SymbolState RequiredState) {
+ return lookup(makeJITDylibSearchOrder(SearchOrder), Name, RequiredState);
}
Expected<JITEvaluatedSymbol>
-ExecutionSession::lookup(ArrayRef<JITDylib *> SearchOrder, StringRef Name) {
- return lookup(SearchOrder, intern(Name));
+ExecutionSession::lookup(ArrayRef<JITDylib *> SearchOrder, StringRef Name,
+ SymbolState RequiredState) {
+ return lookup(SearchOrder, intern(Name), RequiredState);
}
void ExecutionSession::dump(raw_ostream &OS) {
@@ -2195,17 +2259,5 @@
}
}
-MangleAndInterner::MangleAndInterner(ExecutionSession &ES, const DataLayout &DL)
- : ES(ES), DL(DL) {}
-
-SymbolStringPtr MangleAndInterner::operator()(StringRef Name) {
- std::string MangledName;
- {
- raw_string_ostream MangledNameStream(MangledName);
- Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
- }
- return ES.intern(MangledName);
-}
-
} // End namespace orc.
} // End namespace llvm.
diff --git a/llvm/lib/ExecutionEngine/Orc/ExecutionUtils.cpp b/llvm/lib/ExecutionEngine/Orc/ExecutionUtils.cpp
index 3d97fe9..a98445a 100644
--- a/llvm/lib/ExecutionEngine/Orc/ExecutionUtils.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/ExecutionUtils.cpp
@@ -113,6 +113,26 @@
CtorDtorIterator(DtorsList, true));
}
+bool StaticInitGVIterator::isStaticInitGlobal(GlobalValue &GV) {
+ if (GV.isDeclaration())
+ return false;
+
+ if (GV.hasName() && (GV.getName() == "llvm.global_ctors" ||
+ GV.getName() == "llvm.global_dtors"))
+ return true;
+
+ if (ObjFmt == Triple::MachO) {
+ // FIXME: These section checks are too strict: We should match first and
+ // second word split by comma.
+ if (GV.hasSection() &&
+ (GV.getSection().startswith("__DATA,__objc_classlist") ||
+ GV.getSection().startswith("__DATA,__objc_selrefs")))
+ return true;
+ }
+
+ return false;
+}
+
void CtorDtorRunner::add(iterator_range<CtorDtorIterator> CtorDtors) {
if (CtorDtors.empty())
return;
@@ -198,6 +218,30 @@
return JD.define(absoluteSymbols(std::move(RuntimeInterposes)));
}
+void ItaniumCXAAtExitSupport::registerAtExit(void (*F)(void *), void *Ctx,
+ void *DSOHandle) {
+ std::lock_guard<std::mutex> Lock(AtExitsMutex);
+ AtExitRecords[DSOHandle].push_back({F, Ctx});
+}
+
+void ItaniumCXAAtExitSupport::runAtExits(void *DSOHandle) {
+ std::vector<AtExitRecord> AtExitsToRun;
+
+ {
+ std::lock_guard<std::mutex> Lock(AtExitsMutex);
+ auto I = AtExitRecords.find(DSOHandle);
+ if (I != AtExitRecords.end()) {
+ AtExitsToRun = std::move(I->second);
+ AtExitRecords.erase(I);
+ }
+ }
+
+ while (!AtExitsToRun.empty()) {
+ AtExitsToRun.back().F(AtExitsToRun.back().Ctx);
+ AtExitsToRun.pop_back();
+ }
+}
+
DynamicLibrarySearchGenerator::DynamicLibrarySearchGenerator(
sys::DynamicLibrary Dylib, char GlobalPrefix, SymbolPredicate Allow)
: Dylib(std::move(Dylib)), Allow(std::move(Allow)),
diff --git a/llvm/lib/ExecutionEngine/Orc/IndirectionUtils.cpp b/llvm/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
index 1ac9a58..1e991d8 100644
--- a/llvm/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
@@ -28,7 +28,7 @@
CompileCallbackMaterializationUnit(SymbolStringPtr Name,
CompileFunction Compile, VModuleKey K)
: MaterializationUnit(SymbolFlagsMap({{Name, JITSymbolFlags::Exported}}),
- std::move(K)),
+ nullptr, std::move(K)),
Name(std::move(Name)), Compile(std::move(Compile)) {}
StringRef getName() const override { return "<Compile Callbacks>"; }
diff --git a/llvm/lib/ExecutionEngine/Orc/LLJIT.cpp b/llvm/lib/ExecutionEngine/Orc/LLJIT.cpp
index 4218ca4..5b2f868 100644
--- a/llvm/lib/ExecutionEngine/Orc/LLJIT.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/LLJIT.cpp
@@ -8,15 +8,827 @@
#include "llvm/ExecutionEngine/Orc/LLJIT.h"
#include "llvm/ExecutionEngine/JITLink/JITLinkMemoryManager.h"
+#include "llvm/ExecutionEngine/Orc/MachOPlatform.h"
#include "llvm/ExecutionEngine/Orc/ObjectLinkingLayer.h"
#include "llvm/ExecutionEngine/Orc/OrcError.h"
#include "llvm/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.h"
#include "llvm/ExecutionEngine/SectionMemoryManager.h"
+#include "llvm/IR/GlobalVariable.h"
+#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/Mangler.h"
+#include "llvm/IR/Module.h"
+#include "llvm/Support/DynamicLibrary.h"
+
+#include <map>
+
+using namespace llvm;
+using namespace llvm::orc;
+
+namespace {
+
+/// Add a reference to the __dso_handle global to the given module.
+/// Returns a reference to the __dso_handle IR decl.
+GlobalVariable *addDSOHandleDecl(Module &M) {
+ auto DSOHandleTy = StructType::create(M.getContext(), "lljit.dso_handle");
+ return new GlobalVariable(M, DSOHandleTy, true, GlobalValue::ExternalLinkage,
+ nullptr, "__dso_handle");
+}
+
+/// Adds helper function decls and wrapper functions that call the helper with
+/// some additional prefix arguments.
+///
+/// E.g. For wrapper "foo" with type i8(i8, i64), helper "bar", and prefix
+/// args i32 4 and i16 12345, this function will add:
+///
+/// declare i8 @bar(i32, i16, i8, i64)
+///
+/// define i8 @foo(i8, i64) {
+/// entry:
+/// %2 = call i8 @bar(i32 4, i16 12345, i8 %0, i64 %1)
+/// ret i8 %2
+/// }
+///
+Function *addHelperAndWrapper(Module &M, StringRef WrapperName,
+ FunctionType *WrapperFnType,
+ GlobalValue::VisibilityTypes WrapperVisibility,
+ StringRef HelperName,
+ ArrayRef<Value *> HelperPrefixArgs) {
+ std::vector<Type *> HelperArgTypes;
+ for (auto *Arg : HelperPrefixArgs)
+ HelperArgTypes.push_back(Arg->getType());
+ for (auto *T : WrapperFnType->params())
+ HelperArgTypes.push_back(T);
+ auto *HelperFnType =
+ FunctionType::get(WrapperFnType->getReturnType(), HelperArgTypes, false);
+ auto *HelperFn = Function::Create(HelperFnType, GlobalValue::ExternalLinkage,
+ HelperName, M);
+
+ auto *WrapperFn = Function::Create(
+ WrapperFnType, GlobalValue::ExternalLinkage, WrapperName, M);
+ WrapperFn->setVisibility(WrapperVisibility);
+
+ auto *EntryBlock = BasicBlock::Create(M.getContext(), "entry", WrapperFn);
+ IRBuilder<> IB(EntryBlock);
+
+ std::vector<Value *> HelperArgs;
+ for (auto *Arg : HelperPrefixArgs)
+ HelperArgs.push_back(Arg);
+ for (auto &Arg : WrapperFn->args())
+ HelperArgs.push_back(&Arg);
+ auto *HelperResult = IB.CreateCall(HelperFn, HelperArgs);
+ if (HelperFn->getReturnType()->isVoidTy())
+ IB.CreateRetVoid();
+ else
+ IB.CreateRet(HelperResult);
+
+ return WrapperFn;
+}
+
+class GenericLLVMIRPlatformSupport;
+
+/// orc::Platform component of Generic LLVM IR Platform support.
+/// Just forwards calls to the GenericLLVMIRPlatformSupport class below.
+class GenericLLVMIRPlatform : public Platform {
+public:
+ GenericLLVMIRPlatform(GenericLLVMIRPlatformSupport &S) : S(S) {}
+ Error setupJITDylib(JITDylib &JD) override;
+ Error notifyAdding(JITDylib &JD, const MaterializationUnit &MU) override;
+ Error notifyRemoving(JITDylib &JD, VModuleKey K) override {
+ // Noop -- Nothing to do (yet).
+ return Error::success();
+ }
+
+private:
+ GenericLLVMIRPlatformSupport &S;
+};
+
+/// This transform parses llvm.global_ctors to produce a single initialization
+/// function for the module, records the function, then deletes
+/// llvm.global_ctors.
+class GlobalCtorDtorScraper {
+public:
+ GlobalCtorDtorScraper(GenericLLVMIRPlatformSupport &PS) : PS(PS) {}
+ Expected<ThreadSafeModule> operator()(ThreadSafeModule TSM,
+ MaterializationResponsibility &R);
+
+private:
+ GenericLLVMIRPlatformSupport &PS;
+};
+
+/// Generic IR Platform Support
+///
+/// Scrapes llvm.global_ctors and llvm.global_dtors and replaces them with
+/// specially named 'init' and 'deinit'. Injects definitions / interposes for
+/// some runtime API, including __cxa_atexit, dlopen, and dlclose.
+class GenericLLVMIRPlatformSupport : public LLJIT::PlatformSupport {
+public:
+ // GenericLLVMIRPlatform &P) : P(P) {
+ GenericLLVMIRPlatformSupport(LLJIT &J) : J(J) {
+
+ getExecutionSession().setPlatform(
+ std::make_unique<GenericLLVMIRPlatform>(*this));
+
+ setInitTransform(J, GlobalCtorDtorScraper(*this));
+
+ MangleAndInterner Mangle(getExecutionSession(), J.getDataLayout());
+ SymbolMap StdInterposes;
+
+ StdInterposes[Mangle("__lljit.platform_support_instance")] =
+ JITEvaluatedSymbol(pointerToJITTargetAddress(this), JITSymbolFlags());
+ StdInterposes[Mangle("__lljit.cxa_atexit_helper")] = JITEvaluatedSymbol(
+ pointerToJITTargetAddress(registerAtExitHelper), JITSymbolFlags());
+ StdInterposes[Mangle("__lljit.run_atexits_helper")] = JITEvaluatedSymbol(
+ pointerToJITTargetAddress(runAtExitsHelper), JITSymbolFlags());
+
+ cantFail(
+ J.getMainJITDylib().define(absoluteSymbols(std::move(StdInterposes))));
+ cantFail(setupJITDylib(J.getMainJITDylib()));
+ cantFail(J.addIRModule(J.getMainJITDylib(), createPlatformRuntimeModule()));
+ }
+
+ ExecutionSession &getExecutionSession() { return J.getExecutionSession(); }
+
+ /// Adds a module that defines the __dso_handle global.
+ Error setupJITDylib(JITDylib &JD) {
+ auto Ctx = std::make_unique<LLVMContext>();
+ auto M = std::make_unique<Module>("__standard_lib", *Ctx);
+ M->setDataLayout(J.getDataLayout());
+
+ auto *Int64Ty = Type::getInt64Ty(*Ctx);
+ auto *DSOHandle = new GlobalVariable(
+ *M, Int64Ty, true, GlobalValue::ExternalLinkage,
+ ConstantInt::get(Int64Ty, reinterpret_cast<uintptr_t>(&JD)),
+ "__dso_handle");
+ DSOHandle->setVisibility(GlobalValue::HiddenVisibility);
+ DSOHandle->setInitializer(
+ ConstantInt::get(Int64Ty, pointerToJITTargetAddress(&JD)));
+ return J.addIRModule(JD, ThreadSafeModule(std::move(M), std::move(Ctx)));
+ }
+
+ Error notifyAdding(JITDylib &JD, const MaterializationUnit &MU) {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+ if (auto &InitSym = MU.getInitializerSymbol())
+ InitSymbols[&JD].add(InitSym);
+ return Error::success();
+ }
+
+ Error initialize(JITDylib &JD) override {
+ if (auto Initializers = getInitializers(JD)) {
+ for (auto InitFnAddr : *Initializers) {
+ auto *InitFn = jitTargetAddressToFunction<void (*)()>(InitFnAddr);
+ InitFn();
+ }
+ } else
+ return Initializers.takeError();
+ return Error::success();
+ }
+
+ Error deinitialize(JITDylib &JD) override {
+ if (auto Deinitializers = getDeinitializers(JD)) {
+ for (auto DeinitFnAddr : *Deinitializers) {
+ auto *DeinitFn = jitTargetAddressToFunction<void (*)()>(DeinitFnAddr);
+ DeinitFn();
+ }
+ } else
+ return Deinitializers.takeError();
+
+ return Error::success();
+ }
+
+ void registerInitFunc(JITDylib &JD, SymbolStringPtr InitName) {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+ InitFunctions[&JD].add(InitName);
+ }
+
+private:
+ Expected<std::vector<JITTargetAddress>> getInitializers(JITDylib &JD) {
+ if (auto Err = issueInitLookups(JD))
+ return std::move(Err);
+
+ DenseMap<JITDylib *, SymbolLookupSet> LookupSymbols;
+ std::vector<JITDylib *> DFSLinkOrder;
+
+ {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+ DFSLinkOrder = getDFSLinkOrder(JD);
+
+ for (auto *NextJD : DFSLinkOrder) {
+ auto IFItr = InitFunctions.find(NextJD);
+ if (IFItr != InitFunctions.end()) {
+ LookupSymbols[NextJD] = std::move(IFItr->second);
+ InitFunctions.erase(IFItr);
+ }
+ }
+ }
+
+ auto &ES = getExecutionSession();
+ auto LookupResult = Platform::lookupInitSymbols(ES, LookupSymbols);
+
+ if (!LookupResult)
+ return LookupResult.takeError();
+
+ std::vector<JITTargetAddress> Initializers;
+ while (!DFSLinkOrder.empty()) {
+ auto &NextJD = *DFSLinkOrder.back();
+ DFSLinkOrder.pop_back();
+ auto InitsItr = LookupResult->find(&NextJD);
+ if (InitsItr == LookupResult->end())
+ continue;
+ for (auto &KV : InitsItr->second)
+ Initializers.push_back(KV.second.getAddress());
+ }
+
+ return Initializers;
+ }
+
+ Expected<std::vector<JITTargetAddress>> getDeinitializers(JITDylib &JD) {
+ auto &ES = getExecutionSession();
+
+ MangleAndInterner Mangle(getExecutionSession(), J.getDataLayout());
+ auto LLJITRunAtExits = Mangle("__lljit_run_atexits");
+
+ DenseMap<JITDylib *, SymbolLookupSet> LookupSymbols;
+ std::vector<JITDylib *> DFSLinkOrder;
+
+ {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+ DFSLinkOrder = getDFSLinkOrder(JD);
+
+ for (auto *NextJD : DFSLinkOrder) {
+ auto &JDLookupSymbols = LookupSymbols[NextJD];
+ auto DIFItr = DeInitFunctions.find(NextJD);
+ if (DIFItr != DeInitFunctions.end()) {
+ LookupSymbols[NextJD] = std::move(DIFItr->second);
+ DeInitFunctions.erase(DIFItr);
+ }
+ JDLookupSymbols.add(LLJITRunAtExits,
+ SymbolLookupFlags::WeaklyReferencedSymbol);
+ }
+ }
+
+ auto LookupResult = Platform::lookupInitSymbols(ES, LookupSymbols);
+
+ if (!LookupResult)
+ return LookupResult.takeError();
+
+ std::vector<JITTargetAddress> DeInitializers;
+ for (auto *NextJD : DFSLinkOrder) {
+ auto DeInitsItr = LookupResult->find(NextJD);
+ assert(DeInitsItr != LookupResult->end() &&
+ "Every JD should have at least __lljit_run_atexits");
+
+ auto RunAtExitsItr = DeInitsItr->second.find(LLJITRunAtExits);
+ if (RunAtExitsItr != DeInitsItr->second.end())
+ DeInitializers.push_back(RunAtExitsItr->second.getAddress());
+
+ for (auto &KV : DeInitsItr->second)
+ if (KV.first != LLJITRunAtExits)
+ DeInitializers.push_back(KV.second.getAddress());
+ }
+
+ return DeInitializers;
+ }
+
+ // Returns a DFS traversal order of the JITDylibs reachable (via
+ // links-against edges) from JD, starting with JD itself.
+ static std::vector<JITDylib *> getDFSLinkOrder(JITDylib &JD) {
+ std::vector<JITDylib *> DFSLinkOrder;
+ std::vector<JITDylib *> WorkStack({&JD});
+ DenseSet<JITDylib *> Visited;
+
+ while (!WorkStack.empty()) {
+ auto &NextJD = *WorkStack.back();
+ WorkStack.pop_back();
+ if (Visited.count(&NextJD))
+ continue;
+ Visited.insert(&NextJD);
+ DFSLinkOrder.push_back(&NextJD);
+ NextJD.withSearchOrderDo([&](const JITDylibSearchOrder &SearchOrder) {
+ for (auto &KV : SearchOrder)
+ WorkStack.push_back(KV.first);
+ });
+ }
+
+ return DFSLinkOrder;
+ }
+
+ /// Issue lookups for all init symbols required to initialize JD (and any
+ /// JITDylibs that it depends on).
+ Error issueInitLookups(JITDylib &JD) {
+ DenseMap<JITDylib *, SymbolLookupSet> RequiredInitSymbols;
+
+ {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+
+ auto DFSLinkOrder = getDFSLinkOrder(JD);
+
+ for (auto *NextJD : DFSLinkOrder) {
+ auto ISItr = InitSymbols.find(NextJD);
+ if (ISItr != InitSymbols.end()) {
+ RequiredInitSymbols[NextJD] = std::move(ISItr->second);
+ InitSymbols.erase(ISItr);
+ }
+ }
+ }
+
+ return Platform::lookupInitSymbols(getExecutionSession(),
+ RequiredInitSymbols)
+ .takeError();
+ }
+
+ static void registerAtExitHelper(void *Self, void (*F)(void *), void *Ctx,
+ void *DSOHandle) {
+ static_cast<GenericLLVMIRPlatformSupport *>(Self)->AtExitMgr.registerAtExit(
+ F, Ctx, DSOHandle);
+ }
+
+ static void runAtExitsHelper(void *Self, void *DSOHandle) {
+ static_cast<GenericLLVMIRPlatformSupport *>(Self)->AtExitMgr.runAtExits(
+ DSOHandle);
+ }
+
+ // Constructs an LLVM IR module containing platform runtime globals,
+ // functions, and interposes.
+ ThreadSafeModule createPlatformRuntimeModule() {
+ auto Ctx = std::make_unique<LLVMContext>();
+ auto M = std::make_unique<Module>("__standard_lib", *Ctx);
+ M->setDataLayout(J.getDataLayout());
+
+ auto *GenericIRPlatformSupportTy =
+ StructType::create(*Ctx, "lljit.GenericLLJITIRPlatformSupport");
+
+ auto *PlatformInstanceDecl = new GlobalVariable(
+ *M, GenericIRPlatformSupportTy, true, GlobalValue::ExternalLinkage,
+ nullptr, "__lljit.platform_support_instance");
+
+ auto *DSOHandleDecl = addDSOHandleDecl(*M);
+
+ auto *Int8Ty = Type::getInt8Ty(*Ctx);
+ auto *IntTy = Type::getIntNTy(*Ctx, sizeof(int) * CHAR_BIT);
+ auto *VoidTy = Type::getVoidTy(*Ctx);
+ auto *BytePtrTy = PointerType::getUnqual(Int8Ty);
+ auto *AtExitCallbackTy = FunctionType::get(VoidTy, {BytePtrTy}, false);
+ auto *AtExitCallbackPtrTy = PointerType::getUnqual(AtExitCallbackTy);
+
+ addHelperAndWrapper(
+ *M, "__cxa_atexit",
+ FunctionType::get(IntTy, {AtExitCallbackPtrTy, BytePtrTy, BytePtrTy},
+ false),
+ GlobalValue::HiddenVisibility, "__lljit.cxa_atexit_helper",
+ {PlatformInstanceDecl});
+
+ addHelperAndWrapper(
+ *M, "__lljit_run_atexits", FunctionType::get(VoidTy, {}, false),
+ GlobalValue::HiddenVisibility, "__lljit.run_atexits_helper",
+ {PlatformInstanceDecl, DSOHandleDecl});
+
+ return ThreadSafeModule(std::move(M), std::move(Ctx));
+ }
+
+ std::mutex PlatformSupportMutex;
+ LLJIT &J;
+ DenseMap<JITDylib *, SymbolLookupSet> InitSymbols;
+ DenseMap<JITDylib *, SymbolLookupSet> InitFunctions;
+ DenseMap<JITDylib *, SymbolLookupSet> DeInitFunctions;
+ ItaniumCXAAtExitSupport AtExitMgr;
+};
+
+Error GenericLLVMIRPlatform::setupJITDylib(JITDylib &JD) {
+ return S.setupJITDylib(JD);
+}
+
+Error GenericLLVMIRPlatform::notifyAdding(JITDylib &JD,
+ const MaterializationUnit &MU) {
+ return S.notifyAdding(JD, MU);
+}
+
+Expected<ThreadSafeModule>
+GlobalCtorDtorScraper::operator()(ThreadSafeModule TSM,
+ MaterializationResponsibility &R) {
+ auto Err = TSM.withModuleDo([&](Module &M) -> Error {
+ auto &Ctx = M.getContext();
+ auto *GlobalCtors = M.getNamedGlobal("llvm.global_ctors");
+
+ // If there's no llvm.global_ctors or it's just a decl then skip.
+ if (!GlobalCtors || GlobalCtors->isDeclaration())
+ return Error::success();
+
+ std::string InitFunctionName;
+ raw_string_ostream(InitFunctionName)
+ << "__orc_init." << M.getModuleIdentifier();
+
+ MangleAndInterner Mangle(PS.getExecutionSession(), M.getDataLayout());
+ auto InternedName = Mangle(InitFunctionName);
+ if (auto Err =
+ R.defineMaterializing({{InternedName, JITSymbolFlags::Callable}}))
+ return Err;
+
+ auto *InitFunc =
+ Function::Create(FunctionType::get(Type::getVoidTy(Ctx), {}, false),
+ GlobalValue::ExternalLinkage, InitFunctionName, &M);
+ InitFunc->setVisibility(GlobalValue::HiddenVisibility);
+ std::vector<std::pair<Function *, unsigned>> Inits;
+ for (auto E : getConstructors(M))
+ Inits.push_back(std::make_pair(E.Func, E.Priority));
+ llvm::sort(Inits, [](const std::pair<Function *, unsigned> &LHS,
+ const std::pair<Function *, unsigned> &RHS) {
+ return LHS.first < RHS.first;
+ });
+ auto *EntryBlock = BasicBlock::Create(Ctx, "entry", InitFunc);
+ IRBuilder<> IB(EntryBlock);
+ for (auto &KV : Inits)
+ IB.CreateCall(KV.first);
+ IB.CreateRetVoid();
+
+ PS.registerInitFunc(R.getTargetJITDylib(), InternedName);
+ GlobalCtors->eraseFromParent();
+ return Error::success();
+ });
+
+ if (Err)
+ return std::move(Err);
+
+ return TSM;
+}
+
+class MachOPlatformSupport : public LLJIT::PlatformSupport {
+public:
+ using DLOpenType = void *(*)(const char *Name, int Mode);
+ using DLCloseType = int (*)(void *Handle);
+ using DLSymType = void *(*)(void *Handle, const char *Name);
+ using DLErrorType = const char *(*)();
+
+ struct DlFcnValues {
+ Optional<void *> RTLDDefault;
+ DLOpenType dlopen = nullptr;
+ DLCloseType dlclose = nullptr;
+ DLSymType dlsym = nullptr;
+ DLErrorType dlerror = nullptr;
+ };
+
+ static Expected<std::unique_ptr<MachOPlatformSupport>>
+ Create(LLJIT &J, JITDylib &PlatformJITDylib) {
+
+ // Make process symbols visible.
+ {
+ std::string ErrMsg;
+ auto Lib = sys::DynamicLibrary::getPermanentLibrary(nullptr, &ErrMsg);
+ if (!Lib.isValid())
+ return make_error<StringError>(std::move(ErrMsg),
+ inconvertibleErrorCode());
+ }
+
+ DlFcnValues DlFcn;
+
+ // Add support for RTLDDefault on known platforms.
+#ifdef __APPLE__
+ DlFcn.RTLDDefault = reinterpret_cast<void *>(-2);
+#endif // __APPLE__
+
+ if (auto Err = hookUpFunction(DlFcn.dlopen, "dlopen"))
+ return std::move(Err);
+ if (auto Err = hookUpFunction(DlFcn.dlclose, "dlclose"))
+ return std::move(Err);
+ if (auto Err = hookUpFunction(DlFcn.dlsym, "dlsym"))
+ return std::move(Err);
+ if (auto Err = hookUpFunction(DlFcn.dlerror, "dlerror"))
+ return std::move(Err);
+
+ std::unique_ptr<MachOPlatformSupport> MP(
+ new MachOPlatformSupport(J, PlatformJITDylib, DlFcn));
+ return std::move(MP);
+ }
+
+ Error initialize(JITDylib &JD) override {
+ if (auto InitSeq = MP.getInitializerSequence(JD)) {
+ for (auto &KV : *InitSeq) {
+ KV.second.registerObjCSelectors();
+ if (auto Err = KV.second.registerObjCClasses()) {
+ // FIXME: Roll back registrations on error?
+ return Err;
+ }
+ }
+ for (auto &KV : *InitSeq)
+ KV.second.runModInits();
+ } else
+ return InitSeq.takeError();
+ return Error::success();
+ }
+
+ Error deinitialize(JITDylib &JD) override {
+ auto &ES = J.getExecutionSession();
+ if (auto DeinitSeq = MP.getDeinitializerSequence(JD)) {
+ for (auto &KV : *DeinitSeq) {
+ auto DSOHandleName = ES.intern("___dso_handle");
+
+ // FIXME: Run DeInits here.
+ auto Result = ES.lookup(
+ {{KV.first, JITDylibLookupFlags::MatchAllSymbols}},
+ SymbolLookupSet(DSOHandleName,
+ SymbolLookupFlags::WeaklyReferencedSymbol));
+ if (!Result)
+ return Result.takeError();
+ if (Result->empty())
+ continue;
+ assert(Result->count(DSOHandleName) &&
+ "Result does not contain __dso_handle");
+ auto *DSOHandle = jitTargetAddressToPointer<void *>(
+ Result->begin()->second.getAddress());
+ AtExitMgr.runAtExits(DSOHandle);
+ }
+ } else
+ return DeinitSeq.takeError();
+ return Error::success();
+ }
+
+private:
+ template <typename FunctionPtrTy>
+ static Error hookUpFunction(FunctionPtrTy &Fn, const char *Name) {
+ if (auto *FnAddr = sys::DynamicLibrary::SearchForAddressOfSymbol(Name)) {
+ Fn = reinterpret_cast<FunctionPtrTy>(Fn);
+ return Error::success();
+ }
+
+ return make_error<StringError>((Twine("Can not enable MachO JIT Platform: "
+ "missing function: ") +
+ Name)
+ .str(),
+ inconvertibleErrorCode());
+ }
+
+ MachOPlatformSupport(LLJIT &J, JITDylib &PlatformJITDylib, DlFcnValues DlFcn)
+ : J(J), MP(setupPlatform(J)), DlFcn(std::move(DlFcn)) {
+
+ MangleAndInterner Mangle(J.getExecutionSession(), J.getDataLayout());
+ SymbolMap HelperSymbols;
+
+ // platform and atexit helpers.
+ HelperSymbols[Mangle("__lljit.platform_support_instance")] =
+ JITEvaluatedSymbol(pointerToJITTargetAddress(this), JITSymbolFlags());
+ HelperSymbols[Mangle("__lljit.cxa_atexit_helper")] = JITEvaluatedSymbol(
+ pointerToJITTargetAddress(registerAtExitHelper), JITSymbolFlags());
+ HelperSymbols[Mangle("__lljit.run_atexits_helper")] = JITEvaluatedSymbol(
+ pointerToJITTargetAddress(runAtExitsHelper), JITSymbolFlags());
+
+ // dlfcn helpers.
+ HelperSymbols[Mangle("__lljit.dlopen_helper")] = JITEvaluatedSymbol(
+ pointerToJITTargetAddress(dlopenHelper), JITSymbolFlags());
+ HelperSymbols[Mangle("__lljit.dlclose_helper")] = JITEvaluatedSymbol(
+ pointerToJITTargetAddress(dlcloseHelper), JITSymbolFlags());
+ HelperSymbols[Mangle("__lljit.dlsym_helper")] = JITEvaluatedSymbol(
+ pointerToJITTargetAddress(dlsymHelper), JITSymbolFlags());
+ HelperSymbols[Mangle("__lljit.dlerror_helper")] = JITEvaluatedSymbol(
+ pointerToJITTargetAddress(dlerrorHelper), JITSymbolFlags());
+
+ cantFail(
+ PlatformJITDylib.define(absoluteSymbols(std::move(HelperSymbols))));
+ cantFail(MP.setupJITDylib(J.getMainJITDylib()));
+ cantFail(J.addIRModule(PlatformJITDylib, createPlatformRuntimeModule()));
+ }
+
+ static MachOPlatform &setupPlatform(LLJIT &J) {
+ auto Tmp = std::make_unique<MachOPlatform>(
+ J.getExecutionSession(),
+ static_cast<ObjectLinkingLayer &>(J.getObjLinkingLayer()),
+ createStandardSymbolsObject(J));
+ auto &MP = *Tmp;
+ J.getExecutionSession().setPlatform(std::move(Tmp));
+ return MP;
+ }
+
+ static std::unique_ptr<MemoryBuffer> createStandardSymbolsObject(LLJIT &J) {
+ LLVMContext Ctx;
+ Module M("__standard_symbols", Ctx);
+ M.setDataLayout(J.getDataLayout());
+
+ auto *Int64Ty = Type::getInt64Ty(Ctx);
+
+ auto *DSOHandle =
+ new GlobalVariable(M, Int64Ty, true, GlobalValue::ExternalLinkage,
+ ConstantInt::get(Int64Ty, 0), "__dso_handle");
+ DSOHandle->setVisibility(GlobalValue::HiddenVisibility);
+
+ return cantFail(J.getIRCompileLayer().getCompiler()(M));
+ }
+
+ ThreadSafeModule createPlatformRuntimeModule() {
+ auto Ctx = std::make_unique<LLVMContext>();
+ auto M = std::make_unique<Module>("__standard_lib", *Ctx);
+ M->setDataLayout(J.getDataLayout());
+
+ auto *MachOPlatformSupportTy =
+ StructType::create(*Ctx, "lljit.MachOPlatformSupport");
+
+ auto *PlatformInstanceDecl = new GlobalVariable(
+ *M, MachOPlatformSupportTy, true, GlobalValue::ExternalLinkage, nullptr,
+ "__lljit.platform_support_instance");
+
+ auto *Int8Ty = Type::getInt8Ty(*Ctx);
+ auto *IntTy = Type::getIntNTy(*Ctx, sizeof(int) * CHAR_BIT);
+ auto *VoidTy = Type::getVoidTy(*Ctx);
+ auto *BytePtrTy = PointerType::getUnqual(Int8Ty);
+ auto *AtExitCallbackTy = FunctionType::get(VoidTy, {BytePtrTy}, false);
+ auto *AtExitCallbackPtrTy = PointerType::getUnqual(AtExitCallbackTy);
+
+ addHelperAndWrapper(
+ *M, "__cxa_atexit",
+ FunctionType::get(IntTy, {AtExitCallbackPtrTy, BytePtrTy, BytePtrTy},
+ false),
+ GlobalValue::DefaultVisibility, "__lljit.cxa_atexit_helper",
+ {PlatformInstanceDecl});
+
+ addHelperAndWrapper(*M, "dlopen",
+ FunctionType::get(BytePtrTy, {BytePtrTy, IntTy}, false),
+ GlobalValue::DefaultVisibility, "__lljit.dlopen_helper",
+ {PlatformInstanceDecl});
+
+ addHelperAndWrapper(*M, "dlclose",
+ FunctionType::get(IntTy, {BytePtrTy}, false),
+ GlobalValue::DefaultVisibility,
+ "__lljit.dlclose_helper", {PlatformInstanceDecl});
+
+ addHelperAndWrapper(
+ *M, "dlsym",
+ FunctionType::get(BytePtrTy, {BytePtrTy, BytePtrTy}, false),
+ GlobalValue::DefaultVisibility, "__lljit.dlsym_helper",
+ {PlatformInstanceDecl});
+
+ addHelperAndWrapper(*M, "dlerror", FunctionType::get(BytePtrTy, {}, false),
+ GlobalValue::DefaultVisibility,
+ "__lljit.dlerror_helper", {PlatformInstanceDecl});
+
+ return ThreadSafeModule(std::move(M), std::move(Ctx));
+ }
+
+ static void registerAtExitHelper(void *Self, void (*F)(void *), void *Ctx,
+ void *DSOHandle) {
+ static_cast<MachOPlatformSupport *>(Self)->AtExitMgr.registerAtExit(
+ F, Ctx, DSOHandle);
+ }
+
+ static void runAtExitsHelper(void *Self, void *DSOHandle) {
+ static_cast<MachOPlatformSupport *>(Self)->AtExitMgr.runAtExits(DSOHandle);
+ }
+
+ void *jit_dlopen(const char *Path, int Mode) {
+ JITDylib *JDToOpen = nullptr;
+ // FIXME: Do the right thing with Mode flags.
+ {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+
+ // Clear any existing error messages.
+ dlErrorMsgs.erase(std::this_thread::get_id());
+
+ if (auto *JD = J.getExecutionSession().getJITDylibByName(Path)) {
+ auto I = JDRefCounts.find(JD);
+ if (I != JDRefCounts.end()) {
+ ++I->second;
+ return JD;
+ }
+
+ JDRefCounts[JD] = 1;
+ JDToOpen = JD;
+ }
+ }
+
+ if (JDToOpen) {
+ if (auto Err = initialize(*JDToOpen)) {
+ recordError(std::move(Err));
+ return 0;
+ }
+ }
+
+ // Fall through to dlopen if no JITDylib found for Path.
+ return DlFcn.dlopen(Path, Mode);
+ }
+
+ static void *dlopenHelper(void *Self, const char *Path, int Mode) {
+ return static_cast<MachOPlatformSupport *>(Self)->jit_dlopen(Path, Mode);
+ }
+
+ int jit_dlclose(void *Handle) {
+ JITDylib *JDToClose = nullptr;
+
+ {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+
+ // Clear any existing error messages.
+ dlErrorMsgs.erase(std::this_thread::get_id());
+
+ auto I = JDRefCounts.find(Handle);
+ if (I != JDRefCounts.end()) {
+ --I->second;
+ if (I->second == 0) {
+ JDRefCounts.erase(I);
+ JDToClose = static_cast<JITDylib *>(Handle);
+ } else
+ return 0;
+ }
+ }
+
+ if (JDToClose) {
+ if (auto Err = deinitialize(*JDToClose)) {
+ recordError(std::move(Err));
+ return -1;
+ }
+ return 0;
+ }
+
+ // Fall through to dlclose if no JITDylib found for Path.
+ return DlFcn.dlclose(Handle);
+ }
+
+ static int dlcloseHelper(void *Self, void *Handle) {
+ return static_cast<MachOPlatformSupport *>(Self)->jit_dlclose(Handle);
+ }
+
+ void *jit_dlsym(void *Handle, const char *Name) {
+ JITDylibSearchOrder JITSymSearchOrder;
+
+ // FIXME: RTLD_NEXT, RTLD_SELF not supported.
+ {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+
+ // Clear any existing error messages.
+ dlErrorMsgs.erase(std::this_thread::get_id());
+
+ if (JDRefCounts.count(Handle)) {
+ JITSymSearchOrder.push_back(
+ {static_cast<JITDylib *>(Handle),
+ JITDylibLookupFlags::MatchExportedSymbolsOnly});
+ } else if (Handle == DlFcn.RTLDDefault) {
+ for (auto &KV : JDRefCounts)
+ JITSymSearchOrder.push_back(
+ {static_cast<JITDylib *>(KV.first),
+ JITDylibLookupFlags::MatchExportedSymbolsOnly});
+ }
+ }
+
+ if (!JITSymSearchOrder.empty()) {
+ MangleAndInterner Mangle(J.getExecutionSession(), J.getDataLayout());
+ auto MangledName = Mangle(Name);
+ SymbolLookupSet Syms(MangledName,
+ SymbolLookupFlags::WeaklyReferencedSymbol);
+ if (auto Result = J.getExecutionSession().lookup(JITSymSearchOrder, Syms,
+ LookupKind::DLSym)) {
+ auto I = Result->find(MangledName);
+ if (I != Result->end())
+ return jitTargetAddressToPointer<void *>(I->second.getAddress());
+ } else {
+ recordError(Result.takeError());
+ return 0;
+ }
+ }
+
+ // Fall through to dlsym.
+ return DlFcn.dlsym(Handle, Name);
+ }
+
+ static void *dlsymHelper(void *Self, void *Handle, const char *Name) {
+ return static_cast<MachOPlatformSupport *>(Self)->jit_dlsym(Handle, Name);
+ }
+
+ const char *jit_dlerror() {
+ {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+ auto I = dlErrorMsgs.find(std::this_thread::get_id());
+ if (I != dlErrorMsgs.end())
+ return I->second->c_str();
+ }
+ return DlFcn.dlerror();
+ }
+
+ static const char *dlerrorHelper(void *Self) {
+ return static_cast<MachOPlatformSupport *>(Self)->jit_dlerror();
+ }
+
+ void recordError(Error Err) {
+ std::lock_guard<std::mutex> Lock(PlatformSupportMutex);
+ dlErrorMsgs[std::this_thread::get_id()] =
+ std::make_unique<std::string>(toString(std::move(Err)));
+ }
+
+ std::mutex PlatformSupportMutex;
+ LLJIT &J;
+ MachOPlatform &MP;
+ DlFcnValues DlFcn;
+ ItaniumCXAAtExitSupport AtExitMgr;
+ DenseMap<void *, unsigned> JDRefCounts;
+ std::map<std::thread::id, std::unique_ptr<std::string>> dlErrorMsgs;
+};
+
+} // end anonymous namespace
namespace llvm {
namespace orc {
+void LLJIT::PlatformSupport::setInitTransform(
+ LLJIT &J, IRTransformLayer::TransformFunction T) {
+ J.InitHelperTransformLayer->setTransform(std::move(T));
+}
+
+LLJIT::PlatformSupport::~PlatformSupport() {}
+
Error LLJITBuilderState::prepareForConstruction() {
if (!JTMB) {
@@ -57,7 +869,7 @@
Error LLJIT::defineAbsolute(StringRef Name, JITEvaluatedSymbol Sym) {
auto InternedName = ES->intern(Name);
SymbolMap Symbols({{InternedName, Sym}});
- return Main.define(absoluteSymbols(std::move(Symbols)));
+ return Main->define(absoluteSymbols(std::move(Symbols)));
}
Error LLJIT::addIRModule(JITDylib &JD, ThreadSafeModule TSM) {
@@ -67,7 +879,8 @@
TSM.withModuleDo([&](Module &M) { return applyDataLayout(M); }))
return Err;
- return TransformLayer->add(JD, std::move(TSM), ES->allocateVModule());
+ return InitHelperTransformLayer->add(JD, std::move(TSM),
+ ES->allocateVModule());
}
Error LLJIT::addObjectFile(JITDylib &JD, std::unique_ptr<MemoryBuffer> Obj) {
@@ -128,16 +941,23 @@
}
LLJIT::LLJIT(LLJITBuilderState &S, Error &Err)
- : ES(S.ES ? std::move(S.ES) : std::make_unique<ExecutionSession>()),
- Main(this->ES->createJITDylib("<main>")), DL(""),
- TT(S.JTMB->getTargetTriple()),
+ : ES(S.ES ? std::move(S.ES) : std::make_unique<ExecutionSession>()), Main(),
+ DL(""), TT(S.JTMB->getTargetTriple()),
ObjLinkingLayer(createObjectLinkingLayer(S, *ES)),
- ObjTransformLayer(*this->ES, *ObjLinkingLayer), CtorRunner(Main),
- DtorRunner(Main) {
+ ObjTransformLayer(*this->ES, *ObjLinkingLayer) {
ErrorAsOutParameter _(&Err);
- if (auto DLOrErr = S.JTMB->getDefaultDataLayoutForTarget())
+ if (auto MainOrErr = this->ES->createJITDylib("main"))
+ Main = &*MainOrErr;
+ else {
+ Err = MainOrErr.takeError();
+ return;
+ }
+
+ if (S.DL)
+ DL = std::move(*S.DL);
+ else if (auto DLOrErr = S.JTMB->getDefaultDataLayoutForTarget())
DL = std::move(*DLOrErr);
else {
Err = DLOrErr.takeError();
@@ -153,10 +973,12 @@
CompileLayer = std::make_unique<IRCompileLayer>(
*ES, ObjTransformLayer, std::move(*CompileFunction));
TransformLayer = std::make_unique<IRTransformLayer>(*ES, *CompileLayer);
+ InitHelperTransformLayer =
+ std::make_unique<IRTransformLayer>(*ES, *TransformLayer);
}
if (S.NumCompileThreads > 0) {
- TransformLayer->setCloneToNewContextOnEmit(true);
+ InitHelperTransformLayer->setCloneToNewContextOnEmit(true);
CompileThreads =
std::make_unique<ThreadPool>(hardware_concurrency(S.NumCompileThreads));
ES->setDispatchMaterialization(
@@ -167,6 +989,11 @@
CompileThreads->async(std::move(Work));
});
}
+
+ if (S.SetUpPlatform)
+ Err = S.SetUpPlatform(*this);
+ else
+ setUpGenericLLVMIRPlatform(*this);
}
std::string LLJIT::mangle(StringRef UnmangledName) {
@@ -184,15 +1011,24 @@
if (M.getDataLayout() != DL)
return make_error<StringError>(
- "Added modules have incompatible data layouts",
+ "Added modules have incompatible data layouts: " +
+ M.getDataLayout().getStringRepresentation() + " (module) vs " +
+ DL.getStringRepresentation() + " (jit)",
inconvertibleErrorCode());
return Error::success();
}
-void LLJIT::recordCtorDtors(Module &M) {
- CtorRunner.add(getConstructors(M));
- DtorRunner.add(getDestructors(M));
+void setUpGenericLLVMIRPlatform(LLJIT &J) {
+ J.setPlatformSupport(std::make_unique<GenericLLVMIRPlatformSupport>(J));
+}
+
+Error setUpMachOPlatform(LLJIT &J) {
+ auto MP = MachOPlatformSupport::Create(J, J.getMainJITDylib());
+ if (!MP)
+ return MP.takeError();
+ J.setPlatformSupport(std::move(*MP));
+ return Error::success();
}
Error LLLazyJITBuilderState::prepareForConstruction() {
@@ -205,13 +1041,8 @@
Error LLLazyJIT::addLazyIRModule(JITDylib &JD, ThreadSafeModule TSM) {
assert(TSM && "Can not add null module");
- if (auto Err = TSM.withModuleDo([&](Module &M) -> Error {
- if (auto Err = applyDataLayout(M))
- return Err;
-
- recordCtorDtors(M);
- return Error::success();
- }))
+ if (auto Err = TSM.withModuleDo(
+ [&](Module &M) -> Error { return applyDataLayout(M); }))
return Err;
return CODLayer->add(JD, std::move(TSM), ES->allocateVModule());
@@ -256,7 +1087,7 @@
// Create the COD layer.
CODLayer = std::make_unique<CompileOnDemandLayer>(
- *ES, *TransformLayer, *LCTMgr, std::move(ISMBuilder));
+ *ES, *InitHelperTransformLayer, *LCTMgr, std::move(ISMBuilder));
if (S.NumCompileThreads > 0)
CODLayer->setCloneToNewContextOnEmit(true);
diff --git a/llvm/lib/ExecutionEngine/Orc/Layer.cpp b/llvm/lib/ExecutionEngine/Orc/Layer.cpp
index ebc7801..63b04a0 100644
--- a/llvm/lib/ExecutionEngine/Orc/Layer.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/Layer.cpp
@@ -7,7 +7,10 @@
//===----------------------------------------------------------------------===//
#include "llvm/ExecutionEngine/Orc/Layer.h"
+
+#include "llvm/ExecutionEngine/Orc/ExecutionUtils.h"
#include "llvm/IR/Constants.h"
+#include "llvm/Object/MachO.h"
#include "llvm/Object/ObjectFile.h"
#include "llvm/Support/Debug.h"
@@ -23,10 +26,11 @@
*this, *getManglingOptions(), std::move(TSM), std::move(K)));
}
-IRMaterializationUnit::IRMaterializationUnit(ExecutionSession &ES,
- const ManglingOptions &MO,
- ThreadSafeModule TSM, VModuleKey K)
- : MaterializationUnit(SymbolFlagsMap(), std::move(K)), TSM(std::move(TSM)) {
+IRMaterializationUnit::IRMaterializationUnit(
+ ExecutionSession &ES, const IRSymbolMapper::ManglingOptions &MO,
+ ThreadSafeModule TSM, VModuleKey K)
+ : MaterializationUnit(SymbolFlagsMap(), nullptr, std::move(K)),
+ TSM(std::move(TSM)) {
assert(this->TSM && "Module must not be null");
@@ -34,6 +38,7 @@
this->TSM.withModuleDo([&](Module &M) {
for (auto &G : M.global_values()) {
// Skip globals that don't generate symbols.
+
if (!G.hasName() || G.isDeclaration() || G.hasLocalLinkage() ||
G.hasAvailableExternallyLinkage() || G.hasAppendingLinkage())
continue;
@@ -72,13 +77,23 @@
SymbolFlags[MangledName] = JITSymbolFlags::fromGlobalValue(G);
SymbolToDefinition[MangledName] = &G;
}
+
+ // If we need an init symbol for this module then create one.
+ if (!llvm::empty(getStaticInitGVs(M))) {
+ std::string InitSymbolName;
+ raw_string_ostream(InitSymbolName)
+ << "$." << M.getModuleIdentifier() << ".__inits";
+ InitSymbol = ES.intern(InitSymbolName);
+ SymbolFlags[InitSymbol] = JITSymbolFlags();
+ }
});
}
IRMaterializationUnit::IRMaterializationUnit(
ThreadSafeModule TSM, VModuleKey K, SymbolFlagsMap SymbolFlags,
- SymbolNameToDefinitionMap SymbolToDefinition)
- : MaterializationUnit(std::move(SymbolFlags), std::move(K)),
+ SymbolStringPtr InitSymbol, SymbolNameToDefinitionMap SymbolToDefinition)
+ : MaterializationUnit(std::move(SymbolFlags), std::move(InitSymbol),
+ std::move(K)),
TSM(std::move(TSM)), SymbolToDefinition(std::move(SymbolToDefinition)) {}
StringRef IRMaterializationUnit::getName() const {
@@ -105,7 +120,8 @@
}
BasicIRLayerMaterializationUnit::BasicIRLayerMaterializationUnit(
- IRLayer &L, const ManglingOptions &MO, ThreadSafeModule TSM, VModuleKey K)
+ IRLayer &L, const IRSymbolMapper::ManglingOptions &MO, ThreadSafeModule TSM,
+ VModuleKey K)
: IRMaterializationUnit(L.getExecutionSession(), MO, std::move(TSM),
std::move(K)),
L(L), K(std::move(K)) {}
@@ -150,22 +166,26 @@
Expected<std::unique_ptr<BasicObjectLayerMaterializationUnit>>
BasicObjectLayerMaterializationUnit::Create(ObjectLayer &L, VModuleKey K,
std::unique_ptr<MemoryBuffer> O) {
- auto SymbolFlags =
- getObjectSymbolFlags(L.getExecutionSession(), O->getMemBufferRef());
+ auto ObjSymInfo =
+ getObjectSymbolInfo(L.getExecutionSession(), O->getMemBufferRef());
- if (!SymbolFlags)
- return SymbolFlags.takeError();
+ if (!ObjSymInfo)
+ return ObjSymInfo.takeError();
+
+ auto &SymbolFlags = ObjSymInfo->first;
+ auto &InitSymbol = ObjSymInfo->second;
return std::unique_ptr<BasicObjectLayerMaterializationUnit>(
- new BasicObjectLayerMaterializationUnit(L, K, std::move(O),
- std::move(*SymbolFlags)));
+ new BasicObjectLayerMaterializationUnit(
+ L, K, std::move(O), std::move(SymbolFlags), std::move(InitSymbol)));
}
BasicObjectLayerMaterializationUnit::BasicObjectLayerMaterializationUnit(
ObjectLayer &L, VModuleKey K, std::unique_ptr<MemoryBuffer> O,
- SymbolFlagsMap SymbolFlags)
- : MaterializationUnit(std::move(SymbolFlags), std::move(K)), L(L),
- O(std::move(O)) {}
+ SymbolFlagsMap SymbolFlags, SymbolStringPtr InitSymbol)
+ : MaterializationUnit(std::move(SymbolFlags), std::move(InitSymbol),
+ std::move(K)),
+ L(L), O(std::move(O)) {}
StringRef BasicObjectLayerMaterializationUnit::getName() const {
if (O)
@@ -184,35 +204,5 @@
// filter to pass to the object layer along with the object itself.
}
-Expected<SymbolFlagsMap> getObjectSymbolFlags(ExecutionSession &ES,
- MemoryBufferRef ObjBuffer) {
- auto Obj = object::ObjectFile::createObjectFile(ObjBuffer);
-
- if (!Obj)
- return Obj.takeError();
-
- SymbolFlagsMap SymbolFlags;
- for (auto &Sym : (*Obj)->symbols()) {
- // Skip symbols not defined in this object file.
- if (Sym.getFlags() & object::BasicSymbolRef::SF_Undefined)
- continue;
-
- // Skip symbols that are not global.
- if (!(Sym.getFlags() & object::BasicSymbolRef::SF_Global))
- continue;
-
- auto Name = Sym.getName();
- if (!Name)
- return Name.takeError();
- auto InternedName = ES.intern(*Name);
- auto SymFlags = JITSymbolFlags::fromObjectSymbol(Sym);
- if (!SymFlags)
- return SymFlags.takeError();
- SymbolFlags[InternedName] = std::move(*SymFlags);
- }
-
- return SymbolFlags;
-}
-
} // End namespace orc.
} // End namespace llvm.
diff --git a/llvm/lib/ExecutionEngine/Orc/LazyReexports.cpp b/llvm/lib/ExecutionEngine/Orc/LazyReexports.cpp
index 97f36ed..27d08e1 100644
--- a/llvm/lib/ExecutionEngine/Orc/LazyReexports.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/LazyReexports.cpp
@@ -51,7 +51,7 @@
auto LookupResult = ES.lookup(
makeJITDylibSearchOrder(SourceJD, JITDylibLookupFlags::MatchAllSymbols),
- SymbolName);
+ SymbolName, SymbolState::Ready);
if (!LookupResult) {
ES.reportError(LookupResult.takeError());
@@ -123,10 +123,9 @@
LazyCallThroughManager &LCTManager, IndirectStubsManager &ISManager,
JITDylib &SourceJD, SymbolAliasMap CallableAliases, ImplSymbolMap *SrcJDLoc,
VModuleKey K)
- : MaterializationUnit(extractFlags(CallableAliases), std::move(K)),
+ : MaterializationUnit(extractFlags(CallableAliases), nullptr, std::move(K)),
LCTManager(LCTManager), ISManager(ISManager), SourceJD(SourceJD),
- CallableAliases(std::move(CallableAliases)),
- AliaseeTable(SrcJDLoc) {}
+ CallableAliases(std::move(CallableAliases)), AliaseeTable(SrcJDLoc) {}
StringRef LazyReexportsMaterializationUnit::getName() const {
return "<Lazy Reexports>";
diff --git a/llvm/lib/ExecutionEngine/Orc/MachOPlatform.cpp b/llvm/lib/ExecutionEngine/Orc/MachOPlatform.cpp
new file mode 100644
index 0000000..4618d53
--- /dev/null
+++ b/llvm/lib/ExecutionEngine/Orc/MachOPlatform.cpp
@@ -0,0 +1,459 @@
+//===------ MachOPlatform.cpp - Utilities for executing MachO in Orc ------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ExecutionEngine/Orc/MachOPlatform.h"
+
+#include "llvm/BinaryFormat/MachO.h"
+#include "llvm/Support/BinaryByteStream.h"
+
+namespace {
+
+struct objc_class;
+struct objc_image_info;
+struct objc_object;
+struct objc_selector;
+
+using Class = objc_class *;
+using id = objc_object *;
+using SEL = objc_selector *;
+
+using ObjCMsgSendTy = id (*)(id, SEL, ...);
+using ObjCReadClassPairTy = Class (*)(Class, const objc_image_info *);
+using SelRegisterNameTy = SEL (*)(const char *);
+
+enum class ObjCRegistrationAPI { Uninitialized, Unavailable, Initialized };
+
+ObjCRegistrationAPI ObjCRegistrationAPIState =
+ ObjCRegistrationAPI::Uninitialized;
+ObjCMsgSendTy objc_msgSend = nullptr;
+ObjCReadClassPairTy objc_readClassPair = nullptr;
+SelRegisterNameTy sel_registerName = nullptr;
+
+} // end anonymous namespace
+
+namespace llvm {
+namespace orc {
+
+template <typename FnTy>
+static Error setUpObjCRegAPIFunc(FnTy &Target, sys::DynamicLibrary &LibObjC,
+ const char *Name) {
+ if (void *Addr = LibObjC.getAddressOfSymbol(Name))
+ Target = reinterpret_cast<FnTy>(Addr);
+ else
+ return make_error<StringError>(
+ (Twine("Could not find address for ") + Name).str(),
+ inconvertibleErrorCode());
+ return Error::success();
+}
+
+Error enableObjCRegistration(const char *PathToLibObjC) {
+ // If we've already tried to initialize then just bail out.
+ if (ObjCRegistrationAPIState != ObjCRegistrationAPI::Uninitialized)
+ return Error::success();
+
+ ObjCRegistrationAPIState = ObjCRegistrationAPI::Unavailable;
+
+ std::string ErrMsg;
+ auto LibObjC =
+ sys::DynamicLibrary::getPermanentLibrary(PathToLibObjC, &ErrMsg);
+
+ if (!LibObjC.isValid())
+ return make_error<StringError>(std::move(ErrMsg), inconvertibleErrorCode());
+
+ if (auto Err = setUpObjCRegAPIFunc(objc_msgSend, LibObjC, "objc_msgSend"))
+ return Err;
+ if (auto Err = setUpObjCRegAPIFunc(objc_readClassPair, LibObjC,
+ "objc_readClassPair"))
+ return Err;
+ if (auto Err =
+ setUpObjCRegAPIFunc(sel_registerName, LibObjC, "sel_registerName"))
+ return Err;
+
+ ObjCRegistrationAPIState = ObjCRegistrationAPI::Initialized;
+ return Error::success();
+}
+
+bool objcRegistrationEnabled() {
+ return ObjCRegistrationAPIState == ObjCRegistrationAPI::Initialized;
+}
+
+void MachOJITDylibInitializers::runModInits() const {
+ for (const auto &ModInit : ModInitSections) {
+ for (uint64_t I = 0; I != ModInit.NumPtrs; ++I) {
+ auto *InitializerAddr = jitTargetAddressToPointer<uintptr_t *>(
+ ModInit.Address + (I * sizeof(uintptr_t)));
+ auto *Initializer =
+ jitTargetAddressToFunction<void (*)()>(*InitializerAddr);
+ Initializer();
+ }
+ }
+}
+
+void MachOJITDylibInitializers::registerObjCSelectors() const {
+ assert(objcRegistrationEnabled() && "ObjC registration not enabled.");
+
+ for (const auto &ObjCSelRefs : ObjCSelRefsSections) {
+ for (uint64_t I = 0; I != ObjCSelRefs.NumPtrs; ++I) {
+ auto SelEntryAddr = ObjCSelRefs.Address + (I * sizeof(uintptr_t));
+ const auto *SelName =
+ *jitTargetAddressToPointer<const char **>(SelEntryAddr);
+ auto Sel = sel_registerName(SelName);
+ *jitTargetAddressToPointer<SEL *>(SelEntryAddr) = Sel;
+ }
+ }
+}
+
+Error MachOJITDylibInitializers::registerObjCClasses() const {
+ assert(objcRegistrationEnabled() && "ObjC registration not enabled.");
+
+ struct ObjCClassCompiled {
+ void *Metaclass;
+ void *Parent;
+ void *Cache1;
+ void *Cache2;
+ void *Data;
+ };
+
+ auto *ImageInfo =
+ jitTargetAddressToPointer<const objc_image_info *>(ObjCImageInfoAddr);
+ auto ClassSelector = sel_registerName("class");
+
+ for (const auto &ObjCClassList : ObjCClassListSections) {
+ for (uint64_t I = 0; I != ObjCClassList.NumPtrs; ++I) {
+ auto ClassPtrAddr = ObjCClassList.Address + (I * sizeof(uintptr_t));
+ auto Cls = *jitTargetAddressToPointer<Class *>(ClassPtrAddr);
+ auto *ClassCompiled =
+ *jitTargetAddressToPointer<ObjCClassCompiled **>(ClassPtrAddr);
+ objc_msgSend(reinterpret_cast<id>(ClassCompiled->Parent), ClassSelector);
+ auto Registered = objc_readClassPair(Cls, ImageInfo);
+
+ // FIXME: Improve diagnostic by reporting the failed class's name.
+ if (Registered != Cls)
+ return make_error<StringError>("Unable to register Objective-C class",
+ inconvertibleErrorCode());
+ }
+ }
+ return Error::success();
+}
+
+void MachOJITDylibInitializers::dump() const {
+ for (auto &Extent : ModInitSections)
+ dbgs() << formatv("{0:x16}", Extent.Address) << " -- "
+ << formatv("{0:x16}", Extent.Address + 8 * Extent.NumPtrs) << "\n";
+}
+
+MachOPlatform::MachOPlatform(
+ ExecutionSession &ES, ObjectLinkingLayer &ObjLinkingLayer,
+ std::unique_ptr<MemoryBuffer> StandardSymbolsObject)
+ : ES(ES), ObjLinkingLayer(ObjLinkingLayer),
+ StandardSymbolsObject(std::move(StandardSymbolsObject)) {
+ ObjLinkingLayer.addPlugin(std::make_unique<InitScraperPlugin>(*this));
+}
+
+Error MachOPlatform::setupJITDylib(JITDylib &JD) {
+ auto ObjBuffer = MemoryBuffer::getMemBuffer(
+ StandardSymbolsObject->getMemBufferRef(), false);
+ return ObjLinkingLayer.add(JD, std::move(ObjBuffer));
+}
+
+Error MachOPlatform::notifyAdding(JITDylib &JD, const MaterializationUnit &MU) {
+ const auto &InitSym = MU.getInitializerSymbol();
+ if (!InitSym)
+ return Error::success();
+
+ std::lock_guard<std::mutex> Lock(PlatformMutex);
+ RegisteredInitSymbols[&JD].add(InitSym);
+ return Error::success();
+}
+
+Error MachOPlatform::notifyRemoving(JITDylib &JD, VModuleKey K) {
+ llvm_unreachable("Not supported yet");
+}
+
+Expected<MachOPlatform::InitializerSequence>
+MachOPlatform::getInitializerSequence(JITDylib &JD) {
+
+ std::vector<JITDylib *> DFSLinkOrder;
+
+ while (true) {
+ // Lock the platform while we search for any initializer symbols to
+ // look up.
+ DenseMap<JITDylib *, SymbolLookupSet> NewInitSymbols;
+ {
+ std::lock_guard<std::mutex> Lock(PlatformMutex);
+ DFSLinkOrder = getDFSLinkOrder(JD);
+
+ for (auto *InitJD : DFSLinkOrder) {
+ auto RISItr = RegisteredInitSymbols.find(InitJD);
+ if (RISItr != RegisteredInitSymbols.end()) {
+ NewInitSymbols[InitJD] = std::move(RISItr->second);
+ RegisteredInitSymbols.erase(RISItr);
+ }
+ }
+ }
+
+ if (NewInitSymbols.empty())
+ break;
+
+ // Outside the lock, issue the lookup.
+ if (auto R = lookupInitSymbols(JD.getExecutionSession(), NewInitSymbols))
+ ; // Nothing to do in the success case.
+ else
+ return R.takeError();
+ }
+
+ // Lock again to collect the initializers.
+ InitializerSequence FullInitSeq;
+ {
+ std::lock_guard<std::mutex> Lock(PlatformMutex);
+ for (auto *InitJD : reverse(DFSLinkOrder)) {
+ auto ISItr = InitSeqs.find(InitJD);
+ if (ISItr != InitSeqs.end()) {
+ FullInitSeq.emplace_back(InitJD, std::move(ISItr->second));
+ InitSeqs.erase(ISItr);
+ }
+ }
+ }
+
+ return FullInitSeq;
+}
+
+Expected<MachOPlatform::DeinitializerSequence>
+MachOPlatform::getDeinitializerSequence(JITDylib &JD) {
+ std::vector<JITDylib *> DFSLinkOrder = getDFSLinkOrder(JD);
+
+ DeinitializerSequence FullDeinitSeq;
+ {
+ std::lock_guard<std::mutex> Lock(PlatformMutex);
+ for (auto *DeinitJD : DFSLinkOrder) {
+ FullDeinitSeq.emplace_back(DeinitJD, MachOJITDylibDeinitializers());
+ }
+ }
+
+ return FullDeinitSeq;
+}
+
+std::vector<JITDylib *> MachOPlatform::getDFSLinkOrder(JITDylib &JD) {
+ std::vector<JITDylib *> Result, WorkStack({&JD});
+ DenseSet<JITDylib *> Visited;
+
+ while (!WorkStack.empty()) {
+ auto *NextJD = WorkStack.back();
+ WorkStack.pop_back();
+ if (Visited.count(NextJD))
+ continue;
+ Visited.insert(NextJD);
+ Result.push_back(NextJD);
+ NextJD->withSearchOrderDo([&](const JITDylibSearchOrder &SO) {
+ for (auto &KV : SO)
+ WorkStack.push_back(KV.first);
+ });
+ }
+
+ return Result;
+}
+
+void MachOPlatform::registerInitInfo(
+ JITDylib &JD, JITTargetAddress ObjCImageInfoAddr,
+ MachOJITDylibInitializers::SectionExtent ModInits,
+ MachOJITDylibInitializers::SectionExtent ObjCSelRefs,
+ MachOJITDylibInitializers::SectionExtent ObjCClassList) {
+ std::lock_guard<std::mutex> Lock(PlatformMutex);
+
+ auto &InitSeq = InitSeqs[&JD];
+
+ InitSeq.setObjCImageInfoAddr(ObjCImageInfoAddr);
+
+ if (ModInits.Address)
+ InitSeq.addModInitsSection(std::move(ModInits));
+
+ if (ObjCSelRefs.Address)
+ InitSeq.addObjCSelRefsSection(std::move(ObjCSelRefs));
+
+ if (ObjCClassList.Address)
+ InitSeq.addObjCClassListSection(std::move(ObjCClassList));
+}
+
+static Expected<MachOJITDylibInitializers::SectionExtent>
+getSectionExtent(jitlink::LinkGraph &G, StringRef SectionName) {
+ auto *Sec = G.findSectionByName(SectionName);
+ if (!Sec)
+ return MachOJITDylibInitializers::SectionExtent();
+ jitlink::SectionRange R(*Sec);
+ if (R.getSize() % G.getPointerSize() != 0)
+ return make_error<StringError>(SectionName + " section size is not a "
+ "multiple of the pointer size",
+ inconvertibleErrorCode());
+ return MachOJITDylibInitializers::SectionExtent(
+ R.getStart(), R.getSize() / G.getPointerSize());
+}
+
+void MachOPlatform::InitScraperPlugin::modifyPassConfig(
+ MaterializationResponsibility &MR, const Triple &TT,
+ jitlink::PassConfiguration &Config) {
+
+ Config.PrePrunePasses.push_back([this, &MR](jitlink::LinkGraph &G) -> Error {
+ JITLinkSymbolVector InitSectionSymbols;
+ preserveInitSectionIfPresent(InitSectionSymbols, G, "__mod_init_func");
+ preserveInitSectionIfPresent(InitSectionSymbols, G, "__objc_selrefs");
+ preserveInitSectionIfPresent(InitSectionSymbols, G, "__objc_classlist");
+
+ if (!InitSymbolDeps.empty()) {
+ std::lock_guard<std::mutex> Lock(InitScraperMutex);
+ InitSymbolDeps[&MR] = std::move(InitSectionSymbols);
+ }
+
+ if (auto Err = processObjCImageInfo(G, MR))
+ return Err;
+
+ return Error::success();
+ });
+
+ Config.PostFixupPasses.push_back([this, &JD = MR.getTargetJITDylib()](
+ jitlink::LinkGraph &G) -> Error {
+ MachOJITDylibInitializers::SectionExtent ModInits, ObjCSelRefs,
+ ObjCClassList;
+
+ JITTargetAddress ObjCImageInfoAddr = 0;
+ if (auto *ObjCImageInfoSec = G.findSectionByName("__objc_image_info")) {
+ if (auto Addr = jitlink::SectionRange(*ObjCImageInfoSec).getStart()) {
+ ObjCImageInfoAddr = Addr;
+ dbgs() << "Recorded __objc_imageinfo @ " << formatv("{0:x16}", Addr);
+ }
+ }
+
+ // Record __mod_init_func.
+ if (auto ModInitsOrErr = getSectionExtent(G, "__mod_init_func"))
+ ModInits = std::move(*ModInitsOrErr);
+ else
+ return ModInitsOrErr.takeError();
+
+ // Record __objc_selrefs.
+ if (auto ObjCSelRefsOrErr = getSectionExtent(G, "__objc_selrefs"))
+ ObjCSelRefs = std::move(*ObjCSelRefsOrErr);
+ else
+ return ObjCSelRefsOrErr.takeError();
+
+ // Record __objc_classlist.
+ if (auto ObjCClassListOrErr = getSectionExtent(G, "__objc_classlist"))
+ ObjCClassList = std::move(*ObjCClassListOrErr);
+ else
+ return ObjCClassListOrErr.takeError();
+
+ MP.registerInitInfo(JD, ObjCImageInfoAddr, std::move(ModInits),
+ std::move(ObjCSelRefs), std::move(ObjCClassList));
+
+ return Error::success();
+ });
+}
+
+ObjectLinkingLayer::Plugin::LocalDependenciesMap
+MachOPlatform::InitScraperPlugin::getSyntheticSymbolLocalDependencies(
+ MaterializationResponsibility &MR) {
+ std::lock_guard<std::mutex> Lock(InitScraperMutex);
+ auto I = InitSymbolDeps.find(&MR);
+ if (I != InitSymbolDeps.end()) {
+ LocalDependenciesMap Result;
+ Result[MR.getInitializerSymbol()] = std::move(I->second);
+ InitSymbolDeps.erase(&MR);
+ return Result;
+ }
+ return LocalDependenciesMap();
+}
+
+void MachOPlatform::InitScraperPlugin::preserveInitSectionIfPresent(
+ JITLinkSymbolVector &Symbols, jitlink::LinkGraph &G,
+ StringRef SectionName) {
+ if (auto *Sec = G.findSectionByName(SectionName)) {
+ auto SecBlocks = Sec->blocks();
+ if (!llvm::empty(SecBlocks))
+ Symbols.push_back(
+ &G.addAnonymousSymbol(**SecBlocks.begin(), 0, 0, false, true));
+ }
+}
+
+Error MachOPlatform::InitScraperPlugin::processObjCImageInfo(
+ jitlink::LinkGraph &G, MaterializationResponsibility &MR) {
+
+ // If there's an ObjC imagine info then either
+ // (1) It's the first __objc_imageinfo we've seen in this JITDylib. In
+ // this case we name and record it.
+ // OR
+ // (2) We already have a recorded __objc_imageinfo for this JITDylib,
+ // in which case we just verify it.
+ auto *ObjCImageInfo = G.findSectionByName("__objc_imageinfo");
+ if (!ObjCImageInfo)
+ return Error::success();
+
+ auto ObjCImageInfoBlocks = ObjCImageInfo->blocks();
+
+ // Check that the section is not empty if present.
+ if (llvm::empty(ObjCImageInfoBlocks))
+ return make_error<StringError>("Empty __objc_imageinfo section in " +
+ G.getName(),
+ inconvertibleErrorCode());
+
+ // Check that there's only one block in the section.
+ if (std::next(ObjCImageInfoBlocks.begin()) != ObjCImageInfoBlocks.end())
+ return make_error<StringError>("Multiple blocks in __objc_imageinfo "
+ "section in " +
+ G.getName(),
+ inconvertibleErrorCode());
+
+ // Check that the __objc_imageinfo section is unreferenced.
+ // FIXME: We could optimize this check if Symbols had a ref-count.
+ for (auto &Sec : G.sections()) {
+ if (&Sec != ObjCImageInfo)
+ for (auto *B : Sec.blocks())
+ for (auto &E : B->edges())
+ if (E.getTarget().isDefined() &&
+ &E.getTarget().getBlock().getSection() == ObjCImageInfo)
+ return make_error<StringError>("__objc_imageinfo is referenced "
+ "within file " +
+ G.getName(),
+ inconvertibleErrorCode());
+ }
+
+ auto &ObjCImageInfoBlock = **ObjCImageInfoBlocks.begin();
+ auto *ObjCImageInfoData = ObjCImageInfoBlock.getContent().data();
+ auto Version = support::endian::read32(ObjCImageInfoData, G.getEndianness());
+ auto Flags =
+ support::endian::read32(ObjCImageInfoData + 4, G.getEndianness());
+
+ // Lock the mutex while we verify / update the ObjCImageInfos map.
+ std::lock_guard<std::mutex> Lock(InitScraperMutex);
+
+ auto ObjCImageInfoItr = ObjCImageInfos.find(&MR.getTargetJITDylib());
+ if (ObjCImageInfoItr != ObjCImageInfos.end()) {
+ // We've already registered an __objc_imageinfo section. Verify the
+ // content of this new section matches, then delete it.
+ if (ObjCImageInfoItr->second.first != Version)
+ return make_error<StringError>(
+ "ObjC version in " + G.getName() +
+ " does not match first registered version",
+ inconvertibleErrorCode());
+ if (ObjCImageInfoItr->second.second != Flags)
+ return make_error<StringError>("ObjC flags in " + G.getName() +
+ " do not match first registered flags",
+ inconvertibleErrorCode());
+
+ // __objc_imageinfo is valid. Delete the block.
+ for (auto *S : ObjCImageInfo->symbols())
+ G.removeDefinedSymbol(*S);
+ G.removeBlock(ObjCImageInfoBlock);
+ } else {
+ // We haven't registered an __objc_imageinfo section yet. Register and
+ // move on. The section should already be marked no-dead-strip.
+ ObjCImageInfos[&MR.getTargetJITDylib()] = std::make_pair(Version, Flags);
+ }
+
+ return Error::success();
+}
+
+} // End namespace orc.
+} // End namespace llvm.
diff --git a/llvm/lib/ExecutionEngine/Orc/Mangling.cpp b/llvm/lib/ExecutionEngine/Orc/Mangling.cpp
new file mode 100644
index 0000000..7ec7869
--- /dev/null
+++ b/llvm/lib/ExecutionEngine/Orc/Mangling.cpp
@@ -0,0 +1,131 @@
+//===----------- Mangling.cpp -- Name Mangling Utilities for ORC ----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ExecutionEngine/Orc/Mangling.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/Mangler.h"
+#include "llvm/Object/MachO.h"
+#include "llvm/Object/ObjectFile.h"
+#include "llvm/Support/Debug.h"
+
+#define DEBUG_TYPE "orc"
+
+namespace llvm {
+namespace orc {
+
+MangleAndInterner::MangleAndInterner(ExecutionSession &ES, const DataLayout &DL)
+ : ES(ES), DL(DL) {}
+
+SymbolStringPtr MangleAndInterner::operator()(StringRef Name) {
+ std::string MangledName;
+ {
+ raw_string_ostream MangledNameStream(MangledName);
+ Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
+ }
+ return ES.intern(MangledName);
+}
+
+void IRSymbolMapper::add(ExecutionSession &ES, const ManglingOptions &MO,
+ ArrayRef<GlobalValue *> GVs,
+ SymbolFlagsMap &SymbolFlags,
+ SymbolNameToDefinitionMap *SymbolToDefinition) {
+ if (GVs.empty())
+ return;
+
+ MangleAndInterner Mangle(ES, GVs[0]->getParent()->getDataLayout());
+ for (auto *G : GVs) {
+ assert(G && "GVs cannot contain null elements");
+ if (!G->hasName() || G->isDeclaration() || G->hasLocalLinkage() ||
+ G->hasAvailableExternallyLinkage() || G->hasAppendingLinkage())
+ continue;
+
+ if (G->isThreadLocal() && MO.EmulatedTLS) {
+ auto *GV = cast<GlobalVariable>(G);
+
+ auto Flags = JITSymbolFlags::fromGlobalValue(*GV);
+
+ auto EmuTLSV = Mangle(("__emutls_v." + GV->getName()).str());
+ SymbolFlags[EmuTLSV] = Flags;
+ if (SymbolToDefinition)
+ (*SymbolToDefinition)[EmuTLSV] = GV;
+
+ // If this GV has a non-zero initializer we'll need to emit an
+ // __emutls.t symbol too.
+ if (GV->hasInitializer()) {
+ const auto *InitVal = GV->getInitializer();
+
+ // Skip zero-initializers.
+ if (isa<ConstantAggregateZero>(InitVal))
+ continue;
+ const auto *InitIntValue = dyn_cast<ConstantInt>(InitVal);
+ if (InitIntValue && InitIntValue->isZero())
+ continue;
+
+ auto EmuTLST = Mangle(("__emutls_t." + GV->getName()).str());
+ SymbolFlags[EmuTLST] = Flags;
+ if (SymbolToDefinition)
+ (*SymbolToDefinition)[EmuTLST] = GV;
+ }
+ continue;
+ }
+
+ // Otherwise we just need a normal linker mangling.
+ auto MangledName = Mangle(G->getName());
+ SymbolFlags[MangledName] = JITSymbolFlags::fromGlobalValue(*G);
+ if (SymbolToDefinition)
+ (*SymbolToDefinition)[MangledName] = G;
+ }
+}
+
+Expected<std::pair<SymbolFlagsMap, SymbolStringPtr>>
+getObjectSymbolInfo(ExecutionSession &ES, MemoryBufferRef ObjBuffer) {
+ auto Obj = object::ObjectFile::createObjectFile(ObjBuffer);
+
+ if (!Obj)
+ return Obj.takeError();
+
+ SymbolFlagsMap SymbolFlags;
+ for (auto &Sym : (*Obj)->symbols()) {
+ // Skip symbols not defined in this object file.
+ if (Sym.getFlags() & object::BasicSymbolRef::SF_Undefined)
+ continue;
+
+ // Skip symbols that are not global.
+ if (!(Sym.getFlags() & object::BasicSymbolRef::SF_Global))
+ continue;
+
+ auto Name = Sym.getName();
+ if (!Name)
+ return Name.takeError();
+ auto InternedName = ES.intern(*Name);
+ auto SymFlags = JITSymbolFlags::fromObjectSymbol(Sym);
+ if (!SymFlags)
+ return SymFlags.takeError();
+ SymbolFlags[InternedName] = std::move(*SymFlags);
+ }
+
+ SymbolStringPtr InitSymbol;
+
+ if (auto *MachOObj = dyn_cast<object::MachOObjectFile>(Obj->get())) {
+ for (auto &Sec : MachOObj->sections()) {
+ auto SecType = MachOObj->getSectionType(Sec);
+ if ((SecType & MachO::SECTION_TYPE) == MachO::S_MOD_INIT_FUNC_POINTERS) {
+ std::string InitSymString;
+ raw_string_ostream(InitSymString)
+ << "$." << ObjBuffer.getBufferIdentifier() << ".__inits";
+ InitSymbol = ES.intern(InitSymString);
+ break;
+ }
+ }
+ }
+
+ return std::make_pair(std::move(SymbolFlags), std::move(InitSymbol));
+}
+
+} // End namespace orc.
+} // End namespace llvm.
diff --git a/llvm/lib/ExecutionEngine/Orc/ObjectLinkingLayer.cpp b/llvm/lib/ExecutionEngine/Orc/ObjectLinkingLayer.cpp
index 6575b0a..db88ec8 100644
--- a/llvm/lib/ExecutionEngine/Orc/ObjectLinkingLayer.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/ObjectLinkingLayer.cpp
@@ -144,6 +144,10 @@
if (!ExtraSymbolsToClaim.empty())
if (auto Err = MR.defineMaterializing(ExtraSymbolsToClaim))
return notifyFailed(std::move(Err));
+
+ if (const auto &InitSym = MR.getInitializerSymbol())
+ InternedResult[InitSym] = JITEvaluatedSymbol();
+
if (auto Err = MR.notifyResolved(InternedResult)) {
Layer.getExecutionSession().reportError(std::move(Err));
MR.failMaterialization();
@@ -184,8 +188,12 @@
}
private:
- using JITLinkSymbolSet = DenseSet<const Symbol *>;
- using LocalToNamedDependenciesMap = DenseMap<const Symbol *, JITLinkSymbolSet>;
+ struct LocalSymbolNamedDependencies {
+ SymbolNameSet Internal, External;
+ };
+
+ using LocalSymbolNamedDependenciesMap =
+ DenseMap<const Symbol *, LocalSymbolNamedDependencies>;
Error externalizeWeakAndCommonSymbols(LinkGraph &G) {
auto &ES = Layer.getExecutionSession();
@@ -216,6 +224,7 @@
auto &ES = MR.getTargetJITDylib().getExecutionSession();
auto LocalDeps = computeLocalDeps(G);
+ // Compute dependencies for symbols defined in the JITLink graph.
for (auto *Sym : G.defined_symbols()) {
// Skip local symbols: we do not track dependencies for these.
@@ -239,15 +248,12 @@
assert(TargetSym.isDefined() &&
"local symbols must be defined");
auto I = LocalDeps.find(&TargetSym);
- if (I != LocalDeps.end())
- for (auto &S : I->second) {
- assert(S->hasName() &&
- "LocalDeps should only contain named values");
- if (S->isExternal())
- ExternalSymDeps.insert(ES.intern(S->getName()));
- else if (S != Sym)
- InternalSymDeps.insert(ES.intern(S->getName()));
- }
+ if (I != LocalDeps.end()) {
+ for (auto &S : I->second.External)
+ ExternalSymDeps.insert(S);
+ for (auto &S : I->second.Internal)
+ InternalSymDeps.insert(S);
+ }
}
}
@@ -261,11 +267,33 @@
InternalNamedSymbolDeps[SymName] = std::move(InternalSymDeps);
}
+ for (auto &P : Layer.Plugins) {
+ auto SyntheticLocalDeps = P->getSyntheticSymbolLocalDependencies(MR);
+ if (SyntheticLocalDeps.empty())
+ continue;
+
+ for (auto &KV : SyntheticLocalDeps) {
+ auto &Name = KV.first;
+ auto &LocalDepsForName = KV.second;
+ for (auto *Local : LocalDepsForName) {
+ assert(Local->getScope() == Scope::Local &&
+ "Dependence on non-local symbol");
+ auto LocalNamedDepsItr = LocalDeps.find(Local);
+ if (LocalNamedDepsItr == LocalDeps.end())
+ continue;
+ for (auto &S : LocalNamedDepsItr->second.Internal)
+ InternalNamedSymbolDeps[Name].insert(S);
+ for (auto &S : LocalNamedDepsItr->second.External)
+ ExternalNamedSymbolDeps[Name].insert(S);
+ }
+ }
+ }
+
return Error::success();
}
- LocalToNamedDependenciesMap computeLocalDeps(LinkGraph &G) {
- LocalToNamedDependenciesMap DepMap;
+ LocalSymbolNamedDependenciesMap computeLocalDeps(LinkGraph &G) {
+ DenseMap<jitlink::Symbol *, DenseSet<jitlink::Symbol *>> DepMap;
// For all local symbols:
// (1) Add their named dependencies.
@@ -319,7 +347,26 @@
}
} while (Changed);
- return DepMap;
+ // Intern the results to produce a mapping of jitlink::Symbol* to internal
+ // and external symbol names.
+ auto &ES = Layer.getExecutionSession();
+ LocalSymbolNamedDependenciesMap Result;
+ for (auto &KV : DepMap) {
+ auto *Local = KV.first;
+ assert(Local->getScope() == Scope::Local &&
+ "DepMap keys should all be local symbols");
+ auto &LocalNamedDeps = Result[Local];
+ for (auto *Named : KV.second) {
+ assert(Named->getScope() != Scope::Local &&
+ "DepMap values should all be non-local symbol sets");
+ if (Named->isExternal())
+ LocalNamedDeps.External.insert(ES.intern(Named->getName()));
+ else
+ LocalNamedDeps.Internal.insert(ES.intern(Named->getName()));
+ }
+ }
+
+ return Result;
}
void registerDependencies(const SymbolDependenceMap &QueryDeps) {
diff --git a/llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp b/llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp
index ff8289a..0eaf13c 100644
--- a/llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp
+++ b/llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp
@@ -88,7 +88,8 @@
void RTDyldObjectLinkingLayer::emit(MaterializationResponsibility R,
std::unique_ptr<MemoryBuffer> O) {
assert(O && "Object must not be null");
-
+ dbgs() << "Emitting via RTDyldObjectLinkingLayer:\n"
+ << R.getSymbols() << "\n";
// This method launches an asynchronous link step that will fulfill our
// materialization responsibility. We need to switch R to be heap
// allocated before that happens so it can live as long as the asynchronous
@@ -229,6 +230,9 @@
Symbols.erase(KV.first);
}
+ if (const auto &InitSym = R.getInitializerSymbol())
+ Symbols[InitSym] = JITEvaluatedSymbol();
+
if (auto Err = R.notifyResolved(Symbols)) {
R.failMaterialization();
return Err;