fio fails to benchmark [passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0xe75a000: Invalid argument (os error 22)]

[Animeshz]

The fio (used in most popular disk benchmarker like KDiskMark) fails with the following errors:

  fuse-mt/example on  master [?] via 🦀 v1.64.0-nightly >>> fio --loops=2 --size=512m --filename=./mountpoint/fiotest.tmp --stonewall --ioengine=libaio --
direct=1 \
                                                                    --name=Seqread --bs=1m --rw=read \
                                                                    --name=4kQD32read --bs=4k --iodepth=32 --rw=randread
Seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
4kQD32read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-3.31
Starting 2 processes
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=0, buflen=1048576
fio: pid=24904, err=22/file:io_u.c:1846, func=io_u error, error=Invalid argument
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=136785920, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=97517568, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=322498560, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=21110784, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=243191808, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=331292672, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=373399552, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=152776704, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=154963968, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=45359104, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=207089664, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=33755136, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=36495360, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=197152768, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=524095488, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=105418752, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=216604672, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=123449344, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=353476608, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=156409856, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=100343808, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=429608960, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=298127360, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=95707136, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=233742336, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=287895552, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=5505024, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=82231296, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=453476352, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=242589696, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=406925312, buflen=4096
fio: io_u error on file ./mountpoint/fiotest.tmp: Invalid argument: read offset=306835456, buflen=4096
fio: pid=24905, err=22/file:io_u.c:1846, func=io_u error, error=Invalid argument

Seqread: (groupid=0, jobs=1): err=22 (file:io_u.c:1846, func=io_u error, error=Invalid argument): pid=24904: Sat Sep  3 16:12:56 2022
  cpu          : usr=0.00%, sys=0.00%, ctx=2, majf=0, minf=47
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
4kQD32read: (groupid=1, jobs=1): err=22 (file:io_u.c:1846, func=io_u error, error=Invalid argument): pid=24905: Sat Sep  3 16:12:56 2022
  cpu          : usr=0.00%, sys=0.00%, ctx=33, majf=0, minf=46
  IO depths    : 1=3.1%, 2=6.2%, 4=12.5%, 8=25.0%, 16=50.0%, 32=3.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=66.7%, 8=0.0%, 16=0.0%, 32=33.3%, 64=0.0%, >=64=0.0%
     issued rwts: total=32,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):

Run status group 1 (all jobs):

Logs generated from the example passthrough fs:

  fuse-mt/example on  master [?] via 🦀 v1.64.0-nightly took 3m57s >>> cargo run -- mounted mountpoint
# ...
fuser::request: DEBUG: FUSE( 48) ino 0x0000000000000001 STATFS
fuse_mt::fusemt: DEBUG: statfs: "/"
passthrufs::passthrough: DEBUG: statfs: "/"
fuser::request: DEBUG: FUSE( 50) ino 0x0000000000000001 LOOKUP name "fiotest.tmp"
fuse_mt::fusemt: DEBUG: lookup: "/", "fiotest.tmp"
passthrufs::passthrough: DEBUG: getattr: "/fiotest.tmp"
passthrufs::passthrough: DEBUG: stat_real: "mounted/fiotest.tmp"
fuse_mt::inode_table: DEBUG: adding 2 -> "/fiotest.tmp" with 0 lookups
fuse_mt::inode_table: DEBUG: lookups on 2 -> Some("/fiotest.tmp") now 1
fuser::request: DEBUG: FUSE( 52) ino 0x0000000000000001 GETATTR
fuse_mt::fusemt: DEBUG: getattr: "/"
passthrufs::passthrough: DEBUG: getattr: "/"
passthrufs::passthrough: DEBUG: stat_real: "mounted/"
fuser::request: DEBUG: FUSE( 54) ino 0x0000000000000002 OPEN flags 0xc000
fuse_mt::fusemt: DEBUG: open: "/fiotest.tmp"
passthrufs::passthrough: DEBUG: open: "/fiotest.tmp" flags=0xc000
fuser::request: DEBUG: FUSE( 56) ino 0x0000000000000002 READ fh FileHandle(5), offset 0, size 131072
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x20000 @ 0x0
fuse_mt::fusemt: DEBUG: initializing threadpool with 1 threads
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x20000 @ 0x0
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x20000 @ 0x0: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 58) ino 0x0000000000000002 FLUSH fh FileHandle(5), lock owner LockOwner(15463802717853242365)
fuse_mt::fusemt: DEBUG: flush: "/fiotest.tmp"
passthrufs::passthrough: DEBUG: flush: "/fiotest.tmp"
fuser::request: DEBUG: FUSE( 60) ino 0x0000000000000002 RELEASE fh FileHandle(5), flags 0xc000, flush false, lock owner Some(LockOwner(0))
fuse_mt::fusemt: DEBUG: release: "/fiotest.tmp"
passthrufs::passthrough: DEBUG: release: "/fiotest.tmp"
fuser::request: DEBUG: FUSE( 62) ino 0x0000000000000002 GETATTR
fuse_mt::fusemt: DEBUG: getattr: "/fiotest.tmp"
passthrufs::passthrough: DEBUG: getattr: "/fiotest.tmp"
passthrufs::passthrough: DEBUG: stat_real: "mounted/fiotest.tmp"
fuser::request: DEBUG: FUSE( 64) ino 0x0000000000000002 OPEN flags 0xc000
fuse_mt::fusemt: DEBUG: open: "/fiotest.tmp"
passthrufs::passthrough: DEBUG: open: "/fiotest.tmp" flags=0xc000
fuser::request: DEBUG: FUSE( 66) ino 0x0000000000000002 READ fh FileHandle(5), offset 136785920, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x8273000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x8273000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x8273000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 68) ino 0x0000000000000002 READ fh FileHandle(5), offset 97517568, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x5d00000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x5d00000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x5d00000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 70) ino 0x0000000000000002 READ fh FileHandle(5), offset 322498560, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1338f000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1338f000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x1338f000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 72) ino 0x0000000000000002 READ fh FileHandle(5), offset 21110784, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1422000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1422000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x1422000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 74) ino 0x0000000000000002 READ fh FileHandle(5), offset 243191808, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xe7ed000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xe7ed000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0xe7ed000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 76) ino 0x0000000000000002 READ fh FileHandle(5), offset 331292672, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x13bf2000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x13bf2000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x13bf2000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 78) ino 0x0000000000000002 READ fh FileHandle(5), offset 373399552, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1641a000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1641a000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x1641a000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 80) ino 0x0000000000000002 READ fh FileHandle(5), offset 152776704, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x91b3000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x91b3000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x91b3000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 82) ino 0x0000000000000002 READ fh FileHandle(5), offset 154963968, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x93c9000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x93c9000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x93c9000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 84) ino 0x0000000000000002 READ fh FileHandle(5), offset 45359104, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x2b42000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x2b42000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x2b42000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 86) ino 0x0000000000000002 READ fh FileHandle(5), offset 207089664, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xc57f000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xc57f000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0xc57f000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 88) ino 0x0000000000000002 READ fh FileHandle(5), offset 33755136, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x2031000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x2031000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x2031000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 90) ino 0x0000000000000002 READ fh FileHandle(5), offset 36495360, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x22ce000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x22ce000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x22ce000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 92) ino 0x0000000000000002 READ fh FileHandle(5), offset 197152768, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xbc05000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xbc05000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0xbc05000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 94) ino 0x0000000000000002 READ fh FileHandle(5), offset 524095488, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1f3d1000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1f3d1000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x1f3d1000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 96) ino 0x0000000000000002 READ fh FileHandle(5), offset 105418752, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x6489000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x6489000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x6489000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE( 98) ino 0x0000000000000002 READ fh FileHandle(5), offset 216604672, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xce92000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xce92000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0xce92000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(100) ino 0x0000000000000002 READ fh FileHandle(5), offset 123449344, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x75bb000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x75bb000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x75bb000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(102) ino 0x0000000000000002 READ fh FileHandle(5), offset 353476608, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1511a000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1511a000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x1511a000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(104) ino 0x0000000000000002 READ fh FileHandle(5), offset 156409856, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x952a000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x952a000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x952a000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(106) ino 0x0000000000000002 READ fh FileHandle(5), offset 100343808, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x5fb2000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x5fb2000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x5fb2000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(108) ino 0x0000000000000002 READ fh FileHandle(5), offset 429608960, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x199b5000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x199b5000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x199b5000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(110) ino 0x0000000000000002 READ fh FileHandle(5), offset 298127360, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x11c51000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x11c51000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x11c51000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(112) ino 0x0000000000000002 READ fh FileHandle(5), offset 95707136, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x5b46000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x5b46000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x5b46000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(114) ino 0x0000000000000002 READ fh FileHandle(5), offset 233742336, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xdeea000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xdeea000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0xdeea000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(116) ino 0x0000000000000002 READ fh FileHandle(5), offset 287895552, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1128f000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1128f000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x1128f000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(118) ino 0x0000000000000002 READ fh FileHandle(5), offset 5505024, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x540000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x540000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x540000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(120) ino 0x0000000000000002 READ fh FileHandle(5), offset 82231296, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x4e6c000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x4e6c000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x4e6c000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(122) ino 0x0000000000000002 READ fh FileHandle(5), offset 453476352, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1b078000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1b078000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x1b078000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(124) ino 0x0000000000000002 READ fh FileHandle(5), offset 242589696, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xe75a000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0xe75a000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0xe75a000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(126) ino 0x0000000000000002 READ fh FileHandle(5), offset 406925312, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x18413000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x18413000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x18413000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(128) ino 0x0000000000000002 READ fh FileHandle(5), offset 306835456, size 4096
fuse_mt::fusemt: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1249f000
passthrufs::passthrough: DEBUG: read: "/fiotest.tmp" 0x1000 @ 0x1249f000
passthrufs::passthrough: ERROR: read "/fiotest.tmp", 0x1000 @ 0x1249f000: Invalid argument (os error 22)
fuser::request: DEBUG: FUSE(130) ino 0x0000000000000002 FLUSH fh FileHandle(5), lock owner LockOwner(6080363487507376540)
fuse_mt::fusemt: DEBUG: flush: "/fiotest.tmp"
passthrufs::passthrough: DEBUG: flush: "/fiotest.tmp"
fuser::request: DEBUG: FUSE(132) ino 0x0000000000000002 RELEASE fh FileHandle(5), flags 0xc000, flush false, lock owner Some(LockOwner(0))
fuse_mt::fusemt: DEBUG: release: "/fiotest.tmp"
passthrufs::passthrough: DEBUG: release: "/fiotest.tmp"
fuser::request: DEBUG: FUSE(134) ino 0x0000000000000001 LOOKUP name ".git"
fuse_mt::fusemt: DEBUG: lookup: "/", ".git"
passthrufs::passthrough: DEBUG: getattr: "/.git"
passthrufs::passthrough: DEBUG: stat_real: "mounted/.git"
passthrufs::passthrough: ERROR: lstat("/.git"): No such file or directory (os error 2)
fuser::request: DEBUG: FUSE(136) ino 0x0000000000000001 LOOKUP name ".git"
fuse_mt::fusemt: DEBUG: lookup: "/", ".git"
passthrufs::passthrough: DEBUG: getattr: "/.git"
passthrufs::passthrough: DEBUG: stat_real: "mounted/.git"
passthrufs::passthrough: ERROR: lstat("/.git"): No such file or directory (os error 2)
fuser::request: DEBUG: FUSE(138) ino 0x0000000000000001 OPENDIR flags 0x18800
fuse_mt::fusemt: DEBUG: opendir: "/"
passthrufs::passthrough: DEBUG: opendir: "mounted/" (flags = 0o304000)
fuser::request: DEBUG: FUSE(140) ino 0x0000000000000001 READDIR fh FileHandle(2), offset 0, size 4096
fuse_mt::fusemt: DEBUG: readdir: "/" @ 0
fuse_mt::fusemt: DEBUG: entries not yet fetched; requesting with fh 94456831506736
passthrufs::passthrough: DEBUG: readdir: "/"
fuse_mt::fusemt: DEBUG: directory has 4 entries
fuse_mt::fusemt: DEBUG: readdir: adding entry #0, "hi"
fuse_mt::fusemt: DEBUG: readdir: adding entry #1, "."
fuse_mt::fusemt: DEBUG: readdir: adding entry #2, ".."
fuse_mt::fusemt: DEBUG: readdir: adding entry #3, "fiotest.tmp"
fuser::request: DEBUG: FUSE(142) ino 0x0000000000000001 LOOKUP name ".gitignore"
fuse_mt::fusemt: DEBUG: lookup: "/", ".gitignore"
passthrufs::passthrough: DEBUG: getattr: "/.gitignore"
passthrufs::passthrough: DEBUG: stat_real: "mounted/.gitignore"
passthrufs::passthrough: ERROR: lstat("/.gitignore"): No such file or directory (os error 2)
fuser::request: DEBUG: FUSE(144) ino 0x0000000000000001 RELEASEDIR fh FileHandle(2), flags 0x18800, flush false, lock owner Some(LockOwner(0))
fuse_mt::fusemt: DEBUG: releasedir: "/"
passthrufs::passthrough: DEBUG: releasedir: "/"
fuser::request: DEBUG: FUSE(146) ino 0x0000000000000001 STATFS
fuse_mt::fusemt: DEBUG: statfs: "/"
passthrufs::passthrough: DEBUG: statfs: "/"
fuser::request: DEBUG: FUSE(148) ino 0x0000000000000001 STATFS
fuse_mt::fusemt: DEBUG: statfs: "/"
passthrufs::passthrough: DEBUG: statfs: "/"
fuser::request: DEBUG: FUSE(150) ino 0x0000000000000001 STATFS
fuse_mt::fusemt: DEBUG: statfs: "/"
passthrufs::passthrough: DEBUG: statfs: "/"
fuser::request: DEBUG: FUSE(152) ino 0x0000000000000001 STATFS
fuse_mt::fusemt: DEBUG: statfs: "/"
passthrufs::passthrough: DEBUG: statfs: "/"

Not too sure what is missing, haven't take a good look at the codebase. Although stuffs like bindfs based on C High-Level API works as expected.

[wfraser]

Hi, sorry for super late reply. I was working on a maybe-related bug in the passthru filesystem (fixed in 7ed8e6b), but getting an EINVAL on a read call strongly suggests it's coming from the underlying backing filesystem here. I noticed fio is opening the file with O_DIRECT, but this requires the buffers used for reads to be properly aligned (4096 bytes I think?), which the example passthrufs implementation definitely doesn't do, so you get that error.

Note that even with correctly-aligned buffers, you're probably still not going to get good benchmark data this way, since Fuse has its own internal cache that would need to be separately disabled. In general, O_DIRECT is tricky to do with Fuse unfortunately.