Fix integer division constant folding corner cases
ESSL has undefined behavior when the integer modulus operator is used
on negative operands. Generate a warning and fold the result to zero
in this case.
In case the minimum representable signed integer is divided by -1, the
result is defined to be either the maximum or minimum representable
value. We choose to fold the calculation to the maximum representable
value in this case.
BUG=angleproject:1537
TEST=angle_unittests
Change-Id: I57fac6b54a3553b7a0f0e36cc6ba0ed59a88eea9
Reviewed-on: https://chromium-review.googlesource.com/390251
Reviewed-by: Corentin Wallez <cwallez@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Olli Etuaho <oetuaho@nvidia.com>
diff --git a/src/compiler/translator/IntermNode.cpp b/src/compiler/translator/IntermNode.cpp
index 256b72d..5b96a33 100644
--- a/src/compiler/translator/IntermNode.cpp
+++ b/src/compiler/translator/IntermNode.cpp
@@ -1204,14 +1204,42 @@
}
else
{
+ int lhs = leftArray[i].getIConst();
+ int divisor = rightArray[i].getIConst();
if (op == EOpDiv)
{
- resultArray[i].setIConst(leftArray[i].getIConst() / rightArray[i].getIConst());
+ // Check for the special case where the minimum representable number is
+ // divided by -1. If left alone this leads to integer overflow in C++.
+ // ESSL 3.00.6 section 4.1.3 Integers:
+ // "However, for the case where the minimum representable value is
+ // divided by -1, it is allowed to return either the minimum
+ // representable value or the maximum representable value."
+ if (lhs == -0x7fffffff - 1 && divisor == -1)
+ {
+ resultArray[i].setIConst(0x7fffffff);
+ }
+ else
+ {
+ resultArray[i].setIConst(lhs / divisor);
+ }
}
else
{
ASSERT(op == EOpIMod);
- resultArray[i].setIConst(leftArray[i].getIConst() % rightArray[i].getIConst());
+ if (lhs < 0 || divisor < 0)
+ {
+ // ESSL 3.00.6 section 5.9: Results of modulus are undefined when
+ // either one of the operands is negative.
+ diagnostics->warning(getLine(),
+ "Negative modulus operator operand "
+ "encountered during constant folding",
+ "%", "");
+ resultArray[i].setIConst(0);
+ }
+ else
+ {
+ resultArray[i].setIConst(lhs % divisor);
+ }
}
}
break;