munmap tracer programs cause panic on RHEL 9.x

[gmarler]

lightswitch v0.3.1, when started on RHEL 9.6, immediately panics with the following (with RUST_BACKTRACE=full):

...
2025-10-10T11:50:53.422374Z DEBUG lightswitch::profiler: adding kernel module [vmlinux] [0xffffffffa0600000 - 0xffffffffa1600000)

thread 'main' panicked at src/profiler.rs:721:31:
attach tracers: Error: Permission denied (os error 13)
stack backtrace:
   0:     0x555555c59992 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h4af010e4b1354d4d
   1:     0x555555c7f7b3 - core::fmt::write::h17347e42c9203d6d
   2:     0x555555c55b63 - std::io::Write::write_fmt::h05610498df74ae59
   3:     0x555555c597e2 - std::sys::backtrace::BacktraceLock::print::h0286e2cc1356b990
   4:     0x555555c5b2a9 - std::panicking::default_hook::{{closure}}::hb566dc96699e42f6
   5:     0x555555c5b08b - std::panicking::default_hook::h1b340523e54d7d17
   6:     0x5555557d4421 - lightswitch::panic_thread_hook::{{closure}}::hca7a44bbcd05f0af
   7:     0x555555c5bd7e - std::panicking::rust_panic_with_hook::hceef4321c6f4ad8a
   8:     0x555555c5ba4a - std::panicking::begin_panic_handler::{{closure}}::hf9fd67a226c3bb3d
   9:     0x555555c59e89 - std::sys::backtrace::__rust_end_short_backtrace::h52410ec1fdc70787
  10:     0x555555c5b6dd - __rustc[507298ddafb0ef2f]::rust_begin_unwind
  11:     0x555555c7d1d0 - core::panicking::panic_fmt::hf04b323265684a46
  12:     0x555555c7d596 - core::result::unwrap_failed::hdf92484becbba54e
  13:     0x555555806bd4 - lightswitch::profiler::Profiler::run::h8ede74104368aa90
  14:     0x5555557d62ab - lightswitch::main::he67577b2f10368df
  15:     0x5555557db546 - std::sys::backtrace::__rust_begin_short_backtrace::h852a5d5477f6819b
  16:     0x5555557de57c - std::rt::lang_start::{{closure}}::h99f025f2b5cd6b24
  17:     0x555555c4dc40 - std::rt::lang_start_internal::hefa4f6595ff6acba
  18:     0x5555557d8c7d - main
  19:     0x7ffff7c295d0 - __libc_start_call_main
  20:     0x7ffff7c29680 - __libc_start_main@GLIBC_2.2.5
  21:     0x55555579256e - _start
  22:                0x0 - <unknown>

Traced this down to the munmap() tracepoints in src/bpf/tracers.bpf.c. The first problem relates to the fact that the struct munmap_entry_args passed to entry tracepoint for munmap() isn't quite CO-RE compliant, and has changed for the RHEL 9.x kernel.

That's easy to change and make CO-RE compliant, as the offset of the addr field remains constant, so just need generic padding at the start of the struct.

But even after fixing this, the exit tracepoint for munmap() continues to panic on startup, and libbpf-rs isn't giving a clear indication of why, nor does the kernel bpf log.

[javierhonduco]

As discussed, let's not disable the tracers but fix them forwards instead. When there's loading errors of any kind, adding --libbpf-debug should help. Can you also share the output of /sys/kernel/debug/tracing/events/$RELEVANT_FOLDER_AND_NAME/format for a kernel that fails loading these programs?

[gmarler]

I'll create a publicly visible PR - as I noted, the first issue was the explicit arguments you used in the struct munmap_entry_args before the addr parameter - none of which we need to use.

The CO-RE recommendation seems to be to just using padding for the arguments before the ones you care about - rather than explicitly enumerate them, as the offset of the addr parameter doesn't change from kernel to kernel.

After correcting that issue, another cropped up, but I failed to use --libbpf-debug, so didn't get info on why that failure happens - I'll give you that here once I get that PR working in a day or so.

[gmarler]

As an aside, this was just a distraction from the most critical issue at hand, as we need to get the agent cleaning up the procs and mappings to the exclusion of all other concerns for the moment - that's the primary reason the profiler is failing to produce valid data over time within hours/days after lightswitch startup.

Once that's cleared up, then we can come back to this. Certainly not disabling it for more than a week or so, but have to do one or two releases without it to make progress.

[javierhonduco]

I'll create a publicly visible PR - as I noted, the first issue was the explicit arguments you used in the struct munmap_entry_args before the addr parameter - none of which we need to use.

Got it, thanks. Perhaps worth exploring btf tracepoints, introduced in https://lore.kernel.org/netdev/20201203204634.1325171-1-andrii@kernel.org/t/

[gmarler]

Created #327 .

The first commit fixes the issue with the sys_entry_munmap panic, whose signature changed slightly, so using padding to just ignore everything before the addr arg - here's the contents of /sys/kernel/debug/tracing/events/syscalls/sys_enter_munmap/format on RHEL 9.6 - note that even though the initial args change a bit, the addr arg remains at offset 16:

name: sys_enter_munmap
ID: 655
format:
        field:unsigned short common_type;       offset:0;       size:2; signed:0;
        field:unsigned char common_flags;       offset:2;       size:1; signed:0;
        field:unsigned char common_preempt_count;       offset:3;       size:1; signed:0;
        field:int common_pid;   offset:4;       size:4; signed:1;
        field:unsigned char common_preempt_lazy_count;  offset:8;       size:1; signed:0;

        field:int __syscall_nr; offset:12;      size:4; signed:1;
        field:unsigned long addr;       offset:16;      size:8; signed:0;
        field:size_t len;       offset:24;      size:8; signed:0;

print fmt: "addr: 0x%08lx, len: 0x%08lx", ((unsigned long)(REC->addr)), ((unsigned long)(REC->len))

After that, the panic moves to the sys_exit_unmap tracepoint, and the end of the output looks like so with --libbpf-debug:

...
libbpf: map 'tracer_events': created successfully, fd=30
libbpf: map 'tracked_munmap': created successfully, fd=31
libbpf: map 'exec_mappings': skipping creation (preset fd=32)
libbpf: map 'percpu_stats': created successfully, fd=33
libbpf: map 'tracers_.rodata': created successfully, fd=34
2025-10-14T14:40:43.874927Z  INFO lightswitch::profiler: munmap and process exit tracing BPF programs loaded
libbpf: prog 'tracer_exit_munmap': failed to create BPF link for perf_event FD 186: -EACCES
libbpf: prog 'tracer_exit_munmap': failed to attach to tracepoint 'syscalls/sys_exit_munmap': -EACCES
libbpf: prog 'tracer_exit_munmap': failed to auto-attach: -EACCES

Including the entirety of that output as an attachment below.

lightswitch-munmap-exit-panic.txt

[gmarler]

Drilling into this further with retsnoop, the EACCES seems to bubble up from here inside perf_event_set_bpf_prog().

Opening a case with Redhat specific to RHEL 9.6 to see if this is either a configuration issue, or a bug they need to fix in that range of kernels.

[gmarler]

Redhat has determined that:

• The RHEL 9.x kernel range afflicted with this issue is RHEL 9.2 (5.14.0-284.166.1.el9_2) - 5.14.0-285.el9.
• RHEL 10.x is not affected

They should have the offending commit shortly.