TurboJPEG: Properly handle gigapixel images
Prevent several integer overflow issues and subsequent segfaults that
occurred when attempting to compress or decompress gigapixel images with
the TurboJPEG API:
- Modify tjBufSize(), tjBufSizeYUV2(), and tjPlaneSizeYUV() to avoid
integer overflow when computing the return values and to return an
error if such an overflow is unavoidable.
- Modify tjunittest to validate the above.
- Modify tjCompress2(), tjEncodeYUVPlanes(), tjDecompress2(), and
tjDecodeYUVPlanes() to avoid integer overflow when computing the row
pointers in the 64-bit TurboJPEG C API.
- Modify TJBench (both C and Java versions) to avoid overflowing the
size argument to malloc()/new and to fail gracefully if such an
overflow is unavoidable.
In general, this allows gigapixel images to be accommodated by the
64-bit TurboJPEG C API when using automatic JPEG buffer (re)allocation.
Such images cannot currently be accommodated without automatic JPEG
buffer (re)allocation, due to the fact that tjAlloc() accepts a 32-bit
integer argument (oops.) Such images cannot be accommodated in the
TurboJPEG Java API due to the fact that Java always uses a signed 32-bit
integer as an array index.
Fixes #361
diff --git a/tjunittest.c b/tjunittest.c
index a5baf5b..da7c6ff 100644
--- a/tjunittest.c
+++ b/tjunittest.c
@@ -554,6 +554,42 @@
}
+#if SIZEOF_SIZE_T == 8
+#define CHECKSIZE(function) { \
+ if ((unsigned long long)size < (unsigned long long)0xFFFFFFFF) \
+ THROW(#function " overflow"); \
+}
+#else
+#define CHECKSIZE(function) { \
+ if (size != (unsigned long)(-1) || \
+ !strcmp(tjGetErrorStr2(NULL), "No error")) \
+ THROW(#function " overflow"); \
+}
+#endif
+
+static void overflowTest(void)
+{
+ /* Ensure that the various buffer size functions don't overflow */
+ unsigned long size;
+
+ size = tjBufSize(26755, 26755, TJSAMP_444);
+ CHECKSIZE(tjBufSize());
+ size = TJBUFSIZE(26755, 26755);
+ CHECKSIZE(TJBUFSIZE());
+ size = tjBufSizeYUV2(37838, 1, 37838, TJSAMP_444);
+ CHECKSIZE(tjBufSizeYUV2());
+ size = TJBUFSIZEYUV(37838, 37838, TJSAMP_444);
+ CHECKSIZE(TJBUFSIZEYUV());
+ size = tjBufSizeYUV(37838, 37838, TJSAMP_444);
+ CHECKSIZE(tjBufSizeYUV());
+ size = tjPlaneSizeYUV(0, 65536, 0, 65536, TJSAMP_444);
+ CHECKSIZE(tjPlaneSizeYUV());
+
+bailout:
+ return;
+}
+
+
static void bufSizeTest(void)
{
int w, h, i, subsamp;
@@ -865,6 +901,7 @@
}
if (alloc) printf("Testing automatic buffer allocation\n");
if (doYUV) num4bf = 4;
+ overflowTest();
doTest(35, 39, _3byteFormats, 2, TJSAMP_444, "test");
doTest(39, 41, _4byteFormats, num4bf, TJSAMP_444, "test");
doTest(41, 35, _3byteFormats, 2, TJSAMP_422, "test");