Remove old data (exited processes, unmapped mappings, object files, etc)

[javierhonduco]

The profiler currently keeps track of all processes we are profiling. For each of them, we store its mappings, and it also stores all the object files it knows about. We know when we don't need any of the aforementioned objects anymore, but we are never removing them.

[javierhonduco]

I have a commit to start fixing this, will continue testing it, make some other code improvements and will create a PR!

[javierhonduco]

Targeting v0.4.0

[gmarler]

This exhibits within hours of starting lightswitch as the eBPF map exec_mappings filling up and producing ENOSPC errors, thus no further profiling of new processes occurring - only the very longest lived processes continue to be reflected in the profile at all after that point - a very noticeable problem.

I think there's a pretty simple workflow which I'll work on in my clean-mappings branch while you continue work on the https://github.com/javierhonduco/lightswitch/tree/scheduled-delete branch:

• Process exits are tracked via eBPF events
• Memory unmapping events for PIDs are also tracked, but for this scenario aren't strictly necessary to do the needed cleanup
• Certain items are deleted immediately after a process exits, while other mappings are retained for a time to ensure that stacks already collected before the process exit can still be unwound, etc, then published off-host
• We can mark process exits with timestamps, then place the PIDs in a queue that can be scanned after every session_duration expires.
• Once a process has been in the queue for a duration of at least 3 - 4 x session_duration, that should have been ample time to use any of the mappings, and all mappings for the given PID can be purged at once, thus eliminating the most pressing current problem. If I'm missing some critical piece of info, do let me know.
• Emitting debug info on how many mappings are still present at each stage after a process exits will be useful in determining if we've missed anything and that all per-process related info has been eliminated.

[gmarler]

As of https://github.com/gmarler/lightswitch/tree/handle-all-exits, and the addition of a function that reports the currently consumed amounts for processes, exec_mappings, and object_files:

• The number of processes achieves steady state very quickly, so all good there.
• The number exec_mappings continues to climb without bound - exhausts all available (1_000_000) slots within 48 to 72 hours on all machines. Working to add debug info on what information is being left in place even after process cleanup on a per PID basis by destructuring the PIDs out of the keys for this HashMap
• object_files stays generally low overall, not seeing any issues there even after weeks of running - but that could just be because we're not cleaning up the exec_mappings properly yet - we'll see

[gmarler]

Started work on eliminating the exec_mappings that get left behind when processes exit (https://github.com/gmarler/lightswitch/tree/cleanup-mappings):

• Added the necessary from_bytes for exec_mappings_key to allow us to extract the portions of each exec_mappings key to determine the PID, etc it came from
• After a process exit results in a cleanup, iterating over exec_mappings for each PID to see what mappings have not been deleted - producing output like the following at present - may need to modify:
  • The mappings stored in proc_info.mappings.0 seem to be a subset of what are actually stored in exec_mappings for each PID
  • There are definitely mappings being left behind!

PID 193607 has 8 known mappings
PID:  193607 mapping addr: 000056033BA34000 prefix_len: 00000054
PID:  193607 mapping addr: 00007FE86EC06000 prefix_len: 00000053
PID:  193607 mapping addr: 00007FE86EC08000 prefix_len: 00000054
PID:  193607 mapping addr: 00007FE86EE0A000 prefix_len: 00000053
PID:  193607 mapping addr: 00007FE86EE0C000 prefix_len: 00000053
PID:  193607 mapping addr: 00007FE86EE0E000 prefix_len: 00000054
PID:  193607 mapping addr: 00007FE86F010000 prefix_len: 00000050
PID:  193607 mapping addr: 00007FE86F020000 prefix_len: 0000004F
PID:  193607 mapping addr: 00007FE86F040000 prefix_len: 0000004E
PID:  193607 mapping addr: 00007FE86F080000 prefix_len: 0000004D
PID:  193607 mapping addr: 00007FE86F100000 prefix_len: 0000004D
PID:  193607 mapping addr: 00007FE86F180000 prefix_len: 0000004E
PID:  193607 mapping addr: 00007FE86F1C0000 prefix_len: 00000050
PID:  193607 mapping addr: 00007FE86F1D0000 prefix_len: 00000051
PID:  193607 mapping addr: 00007FE86F1D8000 prefix_len: 00000052
PID:  193607 mapping addr: 00007FE86F1DC000 prefix_len: 00000054
PID:  193607 mapping addr: 00007FE86F3E7000 prefix_len: 00000054
PID:  193607 mapping addr: 00007FE86F3E8000 prefix_len: 00000051
PID:  193607 mapping addr: 00007FE86F3F0000 prefix_len: 00000050
PID:  193607 mapping addr: 00007FE86F400000 prefix_len: 00000051
PID:  193607 mapping addr: 00007FE86F615000 prefix_len: 00000054
PID:  193607 mapping addr: 00007FE86F616000 prefix_len: 00000053
PID:  193607 mapping addr: 00007FE86F618000 prefix_len: 00000051
PID:  193607 mapping addr: 00007FE86F620000 prefix_len: 0000004F
PID:  193607 mapping addr: 00007FE86F640000 prefix_len: 00000052
PID 193607 still has 25 mappings!

[gmarler]

In addition to the messaging above so that we can start to see patterns in mappings that we are losing track of during the lifetime of a process, I've added code to the cleanup-mappings branch to just go ahead and delete the mappings after they're supposed to be gone anyway.

Will run this on a few tests hosts for several days to make sure we're not "leaking" mappings anymore, then begin to refactor this change into a PR in earnest.