tree d25bf4c2b0e4fe5e357ec51d5621988a1f675dfc
parent 249b621f9efeb8c47f34b698875c54c9c3108df3
author Rich Felker <dalias@aerifal.cx> 1518400094 -0500
committer Rich Felker <dalias@aerifal.cx> 1518400094 -0500

fix incorrect overflow check for allocation in fmemopen

when a null buffer pointer is passed to fmemopen, requesting it
allocate its own memory buffer, extremely large size arguments near
SIZE_MAX could overflow and result in underallocation. this results
from omission of the size of the cookie structure in the overflow
check but inclusion of it in the calloc call.

instead of accounting for individual small contributions to the total
allocation size needed, simply reject sizes larger than PTRDIFF_MAX,
which will necessarily fail anyway. then adding arbitrary fixed-size
structures is safe without matching up the expressions in the
comparison and the allocation.