Fix a integer overflow in SimplifyCFG's look up table formation logic.
If the width is very large it gets truncated from uint64_t to uint32_t when
passed to TD->fitsInLegalInteger. The truncated value can fit in a register.
This manifested in massive memory usage or crashes (PR13946).
git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@164784 91177308-0d34-0410-b5e6-96231b3b80d8
diff --git a/lib/Transforms/Utils/SimplifyCFG.cpp b/lib/Transforms/Utils/SimplifyCFG.cpp
index 299c059..065325b 100644
--- a/lib/Transforms/Utils/SimplifyCFG.cpp
+++ b/lib/Transforms/Utils/SimplifyCFG.cpp
@@ -3414,6 +3414,10 @@
return false;
// FIXME: If the type is wider than it needs to be, e.g. i8 but all values
// are <= 15, we could try to narrow the type.
+
+ // Avoid overflow, fitsInLegalInteger uses unsigned int for the width.
+ if (TableSize >= UINT_MAX/IT->getBitWidth())
+ return false;
return TD->fitsInLegalInteger(TableSize * IT->getBitWidth());
}