[feat] PIP-468: Scalable Topic Controller

[merlimat]

Summary

Adds PIP-468, a sub-PIP of PIP-460: Scalable Topics, describing the broker-side Scalable Topic Controller — the component responsible for segment lifecycle management, split/merge coordination, consumer assignment, leader election, and DAG watch sessions for scalable topics.

The document covers:

• Architecture — the four-layer design (Admin API → ScalableTopicService → ScalableTopicController → ScalableTopicResources) and the three broker roles (any broker for DAG watch sessions, controller leader for layout mutations and consumer assignment, segment-owning brokers for produce/consume).
• Split and merge protocols — the step-by-step create-first / update-metadata-last ordering that guarantees no messages are lost and all subscription cursors exist before producers are redirected, plus cross-broker coordination via the Segments admin API.
• Consumer assignment — StreamConsumer / CheckpointConsumer receive coordinated assignments from the controller leader; QueueConsumer attaches to all segments with no controller-side assignment. Includes a detailed Consumer Session Lifecycle section clarifying what is persisted (the ConsumerRegistration) vs. what is in-memory on the leader (the keep-alive state and grace-period timer), and how controller leader failover restarts grace-period timers from zero.
• Leader election, failover, and metadata store schema — including the new SubscriptionMetadata and ConsumerRegistration paths.
• Public-facing changes — new binary protocol commands (CommandScalableTopicLookup / Update / Close), new Admin REST endpoints for scalable topics and segments, new ScalableTopics admin client interface, and the scalableTopicEnabled broker configuration.

This PIP only changes documentation under pip/. Implementation will land in subsequent PRs against the parent PIP-460 tracking issue.

Test plan

• Review the document on the PIP mailing list thread (TBD link to be added once thread is open).