feat(spans): Flush oversized segments in chunks

[lvthanh03]

Refs STREAM-826

• Adds an option spans.buffer.flush-oversized-segments to allow for flushing entire segments that exceed spans.buffer.max-segments-bytes bytes.
• Adds the _chunk_segment() function in the flusher that splits span payloads into chunks, each chunk under max-segment-bytes

When the option is enabled, the flusher produces one Kafka message per chunk instead of one per segment.

[linear-code]

STREAM-826 Don't drop spans when segment is too large

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[evanh]

Can we also track a distribution on the number of chunks we have? metrics.timing(..., len(chunks))?

[untitaker]

We also drop spans during ingestion into the redis lua script, those spans are still dropped. Instead we should stop merging sets and flush them out individually. This will make it unnecessary to do the same in the flusher, and preserves the invariant that there is only one root span per flushed chunk. Right now it's possible that multiple unrelated spans (i.e. distinct trees) are flushed in a single chunk and it's not clear to me how the segments consumer handles that (or that it's supposed to)