Instrument plan creation

[geoffreyclaude]

Which issue does this PR close?

Closes #32 and #35.

Rationale for this change

See linked issues. Users have no visibility into DataFusion's planning pipeline, making it hard to identify optimization bottlenecks or understand which rules actually affect the plan.

What changes are included in this PR?

This PR instruments the entire query planning pipeline: analyzer, logical optimizer, physical plan creation, and physical optimizer.

Each phase gets a parent span (analyze_logical_plan, optimize_logical_plan, optimize_physical_plan) that groups individual rule executions. Rules that actually modify the plan are marked with a (modified) suffix in their span name, and the phase span records which rules were effective. For the logical optimizer, pass numbers are tracked since it can run multiple iterations.

The RuleInstrumentationOptions type controls verbosity: Disabled for no instrumentation, PhaseOnly for just the phase spans, or Full for phase spans plus individual rule spans. There's also an option to include unified diffs showing exactly what changed.

Expensive operations like plan formatting and diffing are skipped when spans are disabled (lazy instrumentation).

New macros (instrument_rules_with_info_spans!, etc.) make it easy to instrument a SessionState. When physical optimizer instrumentation is enabled, the query planner is automatically wrapped to trace physical plan creation.

Example Trace

Usage

use datafusion::execution::SessionStateBuilder;
use datafusion_tracing::{instrument_rules_with_info_spans, RuleInstrumentationOptions};

// Build a session state with default features
let session_state = SessionStateBuilder::new()
    .with_default_features()
    .build();

// Configure rule instrumentation (full instrumentation with plan diffs)
let options = RuleInstrumentationOptions::full().with_plan_diff();

// Instrument all planning phases at INFO level
let session_state = instrument_rules_with_info_spans!(
    options: options,
    state: session_state
);

// Create the session context
let ctx = SessionContext::new_with_state(session_state);

For finer control, use the builder:

let options = RuleInstrumentationOptions::builder()
    .analyzer()              // Full instrumentation for analyzer
    .optimizer()             // Full instrumentation for logical optimizer
    .physical_optimizer()    // Full instrumentation for physical optimizer
    .plan_diff()             // Include unified diffs when rules modify plans
    .build();

Or for less verbose traces, use phase-only instrumentation:

let options = RuleInstrumentationOptions::phase_only(); // No individual rule spans

Are these changes tested?

Yes, the integration test suite now enables rule instrumentation and all snapshot tests have been updated to capture the new spans.

Are there any user-facing changes?

New public APIs: RuleInstrumentationOptions, RuleInstrumentationOptionsBuilder, and the instrument_rules_with_*_spans! macros. No breaking changes.