verify: fix problem with hole punching on newer Linux kernels
Jeff Moyer writes:
Our QE team noticed fio failures on recent kernels. I simplified the
job file to the following:
[global]
name=fio-mmap
rw=write
bs=4K
direct=1
end_fsync=1
verify=crc32
[file3]
size=100M
ioengine=mmap
mem=malloc
direct=1
After fio completes (and returns verify errors), the file is completely
full of zeroes.
Here is what the verify logic is doing:
static void do_verify(struct thread_data *td, uint64_t verify_bytes)
{
...
/*
* sync io first and invalidate cache, to make sure we really
* read from disk.
*/
for_each_file(td, f, i) {
if (!fio_file_open(f))
continue;
if (fio_io_sync(td, f))
break;
if (file_invalidate_cache(td, f)) <--------
break;
}
That invalidate cache call looks like so:
static int __file_invalidate_cache(struct thread_data *td, struct fio_file *f,
unsigned long long off,
unsigned long long len)
{
...
/*
* FIXME: add blockdev flushing too
*/
if (f->mmap_ptr) {
ret = posix_madvise(f->mmap_ptr, f->mmap_sz, POSIX_MADV_DONTNEED);
(void) posix_madvise(f->mmap_ptr, f->mmap_sz, FIO_MADV_FREE); <-------
FIO_MADV_FREE can be defined as MADV_REMOVE, which will actually punch a
hole in the file (a hole the size of the entire file, btw). Of course,
unallocated blocks are returned as zeroes by the file system, so that
explains that!
---
For now, only do the MADV_FREE on non-files to avoid punching holes.
Further investigation is needed on the blockdev side, but at least
this should cure the immediate problem.
Reported-by: Jeff Moyer <jmoyer@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
diff --git a/filesetup.c b/filesetup.c
index 6427f3e..7d3a061 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -388,7 +388,8 @@
if (f->mmap_ptr) {
ret = posix_madvise(f->mmap_ptr, f->mmap_sz, POSIX_MADV_DONTNEED);
#ifdef FIO_MADV_FREE
- (void) posix_madvise(f->mmap_ptr, f->mmap_sz, FIO_MADV_FREE);
+ if (f->filetype == FIO_TYPE_BD)
+ (void) posix_madvise(f->mmap_ptr, f->mmap_sz, FIO_MADV_FREE);
#endif
} else if (f->filetype == FIO_TYPE_FILE) {
ret = posix_fadvise(f->fd, off, len, POSIX_FADV_DONTNEED);