Refactor scalene_profiler.py into modular components (#967)

* Refactor scalene_profiler.py into modular components

Extract separation of concerns from the main profiler:
- ScaleneCPUProfiler: CPU sample processing logic
- ScaleneTracing: File/function filtering with lru_cache
- ScaleneLifecycle: Profiler lifecycle management

Updates:
- Reduce scalene_profiler.py from ~1885 to ~1584 lines
- Use Filename type consistently across tracing APIs
- Update scalene_tracer.py type signatures
- Document new modules in scalene/README.md

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>

* Fix mypy error for Python 3.9 compatibility

Add unused-ignore to type comment for IPython.get_ipython() call
to handle both Python versions that do and don't flag this.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>

---------

Co-authored-by: Claude Opus 4.5 <[email protected]>

Author: emeryberger

refactoring_todo.md

@@ -0,0 +1,66 @@
+# Scalene Profiler Refactoring Plan
+
+## Goal
+Refactor `scalene/scalene_profiler.py` into multiple files with clear separation of concerns.
+
+## Status: ✅ COMPLETE
+
+All verification checks pass:
+- `pytest tests/` - 147 tests passed
+- `mypy scalene` - No issues found
+- `ruff check scalene` - All checks passed
+
+## File Sizes
+
+| File | Lines |
+|------|-------|
+| `scalene_profiler.py` | 1,584 (was 1,885) |
+| `scalene_cpu_profiler.py` | 228 (new) |
+| `scalene_tracing.py` | 225 (new) |
+| `scalene_lifecycle.py` | 198 (new) |
+
+**Net reduction**: ~300 lines from main profiler, with reusable logic extracted
+
+## New Modules Created
+
+### 1. `scalene_cpu_profiler.py` ✅
+- **Class**: `ScaleneCPUProfiler`
+- **Purpose**: CPU profiling sample processing
+- **Key methods**:
+  - `process_cpu_sample` - Main CPU sample handler
+  - `_update_main_thread_stats` - Main thread statistics
+  - `_update_thread_stats` - Other thread statistics
+
+### 2. `scalene_tracing.py` ✅
+- **Class**: `ScaleneTracing`
+- **Purpose**: Tracing decisions and file filtering with `lru_cache`
+- **Key methods**:
+  - `should_trace` - Main entry point (cached)
+  - `_passes_exclusion_rules` - Library exclusions
+  - `_should_trace_by_location` - Path-based filtering
+  - `_is_system_library` - System library detection
+
+### 3. `scalene_lifecycle.py` ✅
+- **Class**: `ScaleneLifecycle`
+- **Purpose**: Profiler lifecycle management (prepared for future use)
+
+## Architecture
+
+```
+scalene_profiler.py (Scalene class)
+    ├── ScaleneCPUProfiler (CPU sample processing)
+    ├── ScaleneTracing (file/function filtering)
+    ├── ScaleneMemoryProfiler (already existed)
+    └── ScaleneSignalManager (already existed)
+```
+
+## Completed Tasks
+
+- [x] Extracted CPU profiling logic (~150 lines)
+- [x] Extracted tracing/filtering logic (~150 lines)
+- [x] Created lifecycle module for future use
+- [x] Updated type signatures to use `Filename` consistently
+- [x] Applied proper `lru_cache` usage
+- [x] All tests passing (147/147)
+- [x] Type checking passing (mypy)
+- [x] Linting passing (ruff)

scalene/README.md

@@ -24,24 +24,36 @@
 * `scalene_arguments.py`:
   `ScaleneArguments` holds command-line arguments and their default values.
 
+* `scalene_cpu_profiler.py`:
+  `ScaleneCPUProfiler` handles CPU profiling sample processing, including main thread and background thread statistics collection. Extracted from `scalene_profiler.py` for separation of concerns.
+
 * `scalene_gpu.py`:
   `ScaleneGPU` wraps the NVIDIA library to conveniently provide access to GPU statistics required by Scalene.
 
+* `scalene_lifecycle.py`:
+  `ScaleneLifecycle` manages profiler lifecycle operations including start, stop, and signal management.
+
 * `scalene_magics.py`:
   Sets up the "magics" for using Scalene within Jupyter notebooks (`%scrun` and `%%scalene`).
 
+* `scalene_memory_profiler.py`:
+  `ScaleneMemoryProfiler` handles memory profiling sample processing for malloc, free, and memcpy operations.
+
 * `scalene_output.py`:
   `ScaleneOutput` encapsulates functions used for generating Scalene's profiles either as text or HTML.
 
 * `scalene_profiler.py`:
-  The core of the Scalene profiler.
-  
+  The core of the Scalene profiler. Coordinates CPU, memory, and GPU profiling by delegating to specialized modules (`ScaleneCPUProfiler`, `ScaleneTracing`, `ScaleneMemoryProfiler`).
+
 * `scalene_signals.py`:
   Defines the Unix signals that Scalene uses (some of which must be kept in sync with `include/sampleheap.hpp`).
 
 * `scalene_statistics.py`:
   Operations for managing the statistics generated by Scalene.
 
+* `scalene_tracing.py`:
+  `ScaleneTracing` handles tracing decisions and file filtering. Determines which files and functions should be profiled based on exclusion rules, profile-only patterns, and system library detection. Uses `lru_cache` for performance.
+
 * `scalene_version.py`:
   The version number of Scalene which ultimately is reflected on `pypi` (for `pip` installs, used by `setup.py` in the top level directory).
 

scalene/scalene_cpu_profiler.py

@@ -0,0 +1,228 @@
+"""CPU profiling logic for Scalene."""
+
+from __future__ import annotations
+
+import math
+import threading
+from typing import TYPE_CHECKING, Callable, cast
+
+from scalene.runningstats import RunningStats
+from scalene.scalene_funcutils import ScaleneFuncUtils
+from scalene.scalene_statistics import (
+    ByteCodeIndex,
+    Filename,
+    LineNumber,
+    ScaleneStatistics,
+)
+from scalene.scalene_utility import add_stack, enter_function_meta
+from scalene.time_info import TimeInfo
+
+if TYPE_CHECKING:
+    from types import FrameType
+
+
+class ScaleneCPUProfiler:
+    """Handles CPU profiling sample processing."""
+
+    def __init__(self, stats: ScaleneStatistics, available_cpus: int) -> None:
+        """Initialize the CPU profiler.
+
+        Args:
+            stats: The statistics object to update with CPU samples.
+            available_cpus: Number of available CPUs for utilization calculations.
+        """
+        self._stats = stats
+        self._available_cpus = available_cpus
+
+    def process_cpu_sample(
+        self,
+        new_frames: list[tuple[FrameType, int, FrameType]],
+        now: TimeInfo,
+        gpu_load: float,
+        gpu_mem_used: float,
+        prev: TimeInfo,
+        is_thread_sleeping: dict[int, bool],
+        should_trace: Callable[[Filename, str], bool],
+        last_cpu_interval: float,
+        stacks_enabled: bool,
+    ) -> None:
+        """Handle interrupts for CPU profiling.
+
+        Args:
+            new_frames: List of (frame, thread_id, original_frame) tuples.
+            now: Current time information.
+            gpu_load: Current GPU load (0.0-1.0).
+            gpu_mem_used: Current GPU memory usage.
+            prev: Previous time information.
+            is_thread_sleeping: Dict mapping thread IDs to sleep status.
+            should_trace: Function to check if a file/function should be traced.
+            last_cpu_interval: The last CPU sampling interval.
+            stacks_enabled: Whether stack collection is enabled.
+        """
+        if not new_frames:
+            return
+
+        elapsed = now - prev
+
+        # Skip samples with negative values (can occur in multi-process settings)
+        if any([elapsed.virtual < 0, elapsed.wallclock < 0, elapsed.user < 0]):
+            return
+
+        # Calculate CPU utilization
+        cpu_utilization = 0.0
+        if elapsed.wallclock != 0:
+            cpu_utilization = elapsed.user / elapsed.wallclock
+
+        core_utilization = cpu_utilization / self._available_cpus
+        if cpu_utilization > 1.0:
+            cpu_utilization = 1.0
+            elapsed.wallclock = elapsed.user
+
+        # Handle NaN GPU load
+        if math.isnan(gpu_load):
+            gpu_load = 0.0
+        assert 0.0 <= gpu_load <= 1.0
+
+        gpu_time = gpu_load * elapsed.wallclock
+        self._stats.gpu_stats.total_gpu_samples += gpu_time
+
+        python_time = last_cpu_interval
+        c_time = max(elapsed.virtual - python_time, 0)
+        total_time = python_time + c_time
+
+        # Count non-sleeping frames
+        total_frames = sum(
+            not is_thread_sleeping[tident] for frame, tident, orig_frame in new_frames
+        )
+        if total_frames == 0:
+            total_frames = 1
+
+        normalized_time = total_time / total_frames
+        average_python_time = python_time / total_frames
+        average_c_time = c_time / total_frames
+        average_cpu_time = (python_time + c_time) / total_frames
+
+        # Process main thread
+        main_thread_frame = new_frames[0][0]
+
+        if stacks_enabled:
+            add_stack(
+                main_thread_frame,
+                should_trace,
+                self._stats.stacks,
+                average_python_time,
+                average_c_time,
+                average_cpu_time,
+            )
+
+        enter_function_meta(main_thread_frame, should_trace, self._stats)
+        fname = Filename(main_thread_frame.f_code.co_filename)
+        lineno = LineNumber(main_thread_frame.f_lineno)
+
+        main_tid = cast(int, threading.main_thread().ident)
+        if not is_thread_sleeping[main_tid]:
+            self._update_main_thread_stats(
+                fname,
+                lineno,
+                now,
+                average_python_time,
+                average_c_time,
+                average_cpu_time,
+                cpu_utilization,
+                core_utilization,
+                gpu_load,
+                gpu_mem_used,
+                elapsed,
+            )
+
+        # Process other threads
+        for frame, tident, orig_frame in new_frames:
+            if frame == main_thread_frame:
+                continue
+
+            add_stack(
+                frame,
+                should_trace,
+                self._stats.stacks,
+                average_python_time,
+                average_c_time,
+                average_cpu_time,
+            )
+
+            fname = Filename(frame.f_code.co_filename)
+            lineno = LineNumber(frame.f_lineno)
+            enter_function_meta(frame, should_trace, self._stats)
+
+            if is_thread_sleeping[tident]:
+                continue
+
+            self._update_thread_stats(
+                fname,
+                lineno,
+                orig_frame,
+                normalized_time,
+                cpu_utilization,
+                core_utilization,
+            )
+
+        # Cleanup
+        del new_frames[:]
+        del new_frames
+        del is_thread_sleeping
+        self._stats.cpu_stats.total_cpu_samples += total_time
+
+    def _update_main_thread_stats(
+        self,
+        fname: Filename,
+        lineno: LineNumber,
+        now: TimeInfo,
+        average_python_time: float,
+        average_c_time: float,
+        average_cpu_time: float,
+        cpu_utilization: float,
+        core_utilization: float,
+        gpu_load: float,
+        gpu_mem_used: float,
+        elapsed: TimeInfo,
+    ) -> None:
+        """Update statistics for the main thread."""
+        cpu_stats = self._stats.cpu_stats
+        gpu_stats = self._stats.gpu_stats
+
+        cpu_stats.cpu_samples_list[fname][lineno].append(now.wallclock)
+        cpu_stats.cpu_samples_python[fname][lineno] += average_python_time
+        cpu_stats.cpu_samples_c[fname][lineno] += average_c_time
+        cpu_stats.cpu_samples[fname] += average_cpu_time
+        cpu_stats.cpu_utilization[fname][lineno].push(cpu_utilization)
+        cpu_stats.core_utilization[fname][lineno].push(core_utilization)
+
+        gpu_stats.gpu_samples[fname][lineno] += gpu_load * elapsed.wallclock
+        gpu_stats.n_gpu_samples[fname][lineno] += elapsed.wallclock
+        gpu_stats.gpu_mem_samples[fname][lineno].push(gpu_mem_used)
+
+    def _update_thread_stats(
+        self,
+        fname: Filename,
+        lineno: LineNumber,
+        orig_frame: FrameType,
+        normalized_time: float,
+        cpu_utilization: float,
+        core_utilization: float,
+    ) -> None:
+        """Update statistics for non-main threads."""
+        cpu_stats = self._stats.cpu_stats
+
+        # Check if the original caller is stuck inside a call
+        if ScaleneFuncUtils.is_call_function(
+            orig_frame.f_code,
+            ByteCodeIndex(orig_frame.f_lasti),
+        ):
+            # Attribute time to native
+            cpu_stats.cpu_samples_c[fname][lineno] += normalized_time
+        else:
+            # Attribute time to Python
+            cpu_stats.cpu_samples_python[fname][lineno] += normalized_time
+
+        cpu_stats.cpu_samples[fname] += normalized_time
+        cpu_stats.cpu_utilization[fname][lineno].push(cpu_utilization)
+        cpu_stats.core_utilization[fname][lineno].push(core_utilization)