Early WIP: rr/sampling profiler hybrid

[Keno]

This is an idea I've been kicking around. When doing sampling profiling, you really want to minimize the amount of work you do while sampling in order for it to a) be fast and b) not disrupt the program too much. Unfortunately this is of course in direct conflict with actually collecting anything useful. What I'm proposing here is to use rr to do the actual data collection as a post-processing step. The way this is done is to sample the current ip and the value of the retired branch counter, thus hopefully allowing us to find this position again during replay and do whatever we want to do (backtrace, collect values, more fancy things, etc...).

This is nowhere near done, but I figured people may have early feedback.

[bgirard]

Looks very interesting! Thanks for exploring this. You forgot to mention an important benefit from this approach which is having the option to 'Jump to the debugger' when looking at a profile to analyze what causes a slow path.

It looks like in the patch you're 'sampling' every 4k CPU cycles (for the current process?). AIUI this would effectively be a 'user/process' CPU time trigger rather than a wall clock trigger?

[rocallahan]

Seems to me you could send a signal to the tracee, like the perf-event signal now, that interrupts the tracee and is treated like any other async signal by rr so you can easily replay to delivery of that signal using the existing logic.

[Keno]

Looks very interesting! Thanks for exploring this. You forgot to mention an important benefit from this approach which is having the option to 'Jump to the debugger' when looking at a profile to analyze what causes a slow path.

Yes, that's a little tricky of course with sampling. However, I do consider this essentially the same problem. In theory your sampling profiler could just walk the stack, record all variables, etc, but in practice nobody does because it would make sampling impractical.

It looks like in the patch you're 'sampling' every 4k CPU cycles (for the current process?). AIUI this would effectively be a 'user/process' CPU time trigger rather than a wall clock trigger?

Not quite, by setting the freq field. I'm asking for a sample at 4kHz, i.e. wall clock time.

[Keno]

Seems to me you could send a signal to the tracee, like the perf-event signal now, that interrupts the tracee and is treated like any other async signal by rr so you can easily replay to delivery of that signal using the existing logic.

I was hoping to avoid the overhead of the extra context switch.

[GitMensch]

@Keno Would you mind to rebase the changes on current master?
Are the changes to PerfCounters.cc "intrusive" and/or the new command not usable? If yes then it seems reasonable to convert this PR to a draft, otherwise it may could go in as experimental feature instead of laying around another 5 years...