hugetlb: add new testcase hugeshmat05.c

shmget()/shmat() fails to allocate huge pages shared memory segment
with EINVAL if its size is not in the range [ N*HUGE_PAGE_SIZE - 4095,
N*HUGE_PAGE_SIZE ]. This is a problem in the memory segment size round
up algorithm. The requested size is rounded up to PAGE_SIZE (4096), but
if this roundup does not match HUGE_PAGE_SIZE (2Mb) boundary - the
allocation fails.

This bug is present in all RHEL6 versions, but not in RHEL7. It looks
like this was fixed in mainline kernel > v3.3 by the following patches:

091d0d5 shm: fix null pointer deref when userspace specifies
        invalid hugepage size
af73e4d hugetlbfs: fix mmap failure in unaligned size request
42d7395 mm: support more pagesizes for MAP_HUGETLB/SHM_HUGETLB
40716e2 hugetlbfs: fix alignment of huge page requests

Signed-off-by: Li Wang <liwang@redhat.com>
3 files changed