[Feature] Introduce Leader-Follower Architecture for Discovery and Engine Components in Multi-Replica Deployment

[thunguo]

What would you like to be added:

Introduce a Leader-Follower (Master-Slave) architecture for the Discovery and Engine components to ensure data consistency when Admin is deployed with multiple replicas.

Two approaches are proposed (see full design in #1380):

1. Kubernetes Lease
   Leverage Kubernetes' native Lease mechanism for leader election. This approach is widely adopted by Kubernetes-native applications, but requires the Engine component to be of Kubernetes type.

2. GORM + Database
   Implement leader election using GORM and database atomicity — e.g., combining UPDATE ... WHERE statements with unique constraints or version numbers to simulate a lease mechanism and elect a single Leader. See the detailed solution in #1380 (comment).

Implementation notes:

• Leader election should be triggered during the start lifecycle phase (not init).
• Only components that require leader election should participate in the election process.
• Follower nodes must not execute business logic (i.e., list-watch and DB writes).
• When the Leader fails, the remaining Follower nodes must be able to re-elect a new Leader and resume business logic.

Why is this needed:

Currently, when Admin is deployed with multiple replicas, each replica independently performs list-watch operations and writes data to the DB store. Due to factors such as network latency, this can lead to data inconsistencies across replicas.

By evolving Discovery and Engine into a Leader-Follower architecture, only one replica actively performs list-watch and writes to the store at any given time, eliminating race conditions and ensuring data consistency in multi-replica deployments.

Proposal: #1380