Clamp stencil reference value to D3D11 8-bit limitations.
D3D11 can only mask 8 bits of a stencil reference value, so the
upper bits are ignored. GL, however, expects the reference value
to be able to scale to 32-bits.
We will expect to fail certain edge cases where the reference value,
and mask, both use particular 9+ bit values.
Change-Id: I8c7451270d9e40310f4154955d38e51bbf7de4df
Reviewed-on: https://chromium-review.googlesource.com/181555
Reviewed-by: Geoff Lang <geofflang@chromium.org>
Tested-by: Jamie Madill <jmadill@chromium.org>
diff --git a/src/libGLESv2/renderer/d3d11/Renderer11.cpp b/src/libGLESv2/renderer/d3d11/Renderer11.cpp
index 72fbe0e..32c55fc 100644
--- a/src/libGLESv2/renderer/d3d11/Renderer11.cpp
+++ b/src/libGLESv2/renderer/d3d11/Renderer11.cpp
@@ -786,7 +786,13 @@
"setting the default depth stencil state.");
}
- mDeviceContext->OMSetDepthStencilState(dxDepthStencilState, static_cast<UINT>(stencilRef));
+ // Max D3D11 stencil reference value is 0xFF, corresponding to the max 8 bits in a stencil buffer
+ // GL specifies we should clamp the ref value to the nearest bit depth when doing stencil ops
+ META_ASSERT(D3D11_DEFAULT_STENCIL_READ_MASK == 0xFF);
+ META_ASSERT(D3D11_DEFAULT_STENCIL_WRITE_MASK == 0xFF);
+ UINT stencilRef = std::min(stencilRef, 0xFFu);
+
+ mDeviceContext->OMSetDepthStencilState(dxDepthStencilState, stencilRef);
mCurDepthStencilState = depthStencilState;
mCurStencilRef = stencilRef;