Skip to content

✨ Add pre-start hook support to manager#3429

Open
leontappe wants to merge 6 commits intokubernetes-sigs:mainfrom
mittwald:mw
Open

✨ Add pre-start hook support to manager#3429
leontappe wants to merge 6 commits intokubernetes-sigs:mainfrom
mittwald:mw

Conversation

@leontappe
Copy link

I have a use case similar to the one described in a previous PR and this issue.

My controller needs to consume job status files on start, that would normally be worked on by a custom runnable during runtime. With vanilla controller-runtime this is currently not possible as any code that is run before reconciliation, has no access to a running manager.

This PR provides a way to register runnables as hooks that you can supply to your manager.

This PR is based on a fork of https://github.com/terinjokes/controller-runtime/tree/prestart-hook which is in turn a fork of this repo. I rebased the current master onto the latest development state of the original PR and implemented some of the proposals made by @JoelSpeed

terinjokes and others added 4 commits August 25, 2024 15:47
When implementing a controller that uses leader election, there maybe be
work that needs to be done after winning the election but before
processing enqueued requests. For example, a controller may need to
build up an internal mapping of the current state of the cluster before
it can begin reconciling.

This changeset adds support for adding prestart hooks to
controller-runtime's manager implementation. This hook runs after the
manager has been elected leader, immediately before the leader election
controllers are started.

Related kubernetes-sigs#607
feat(manager): add prestart hook support

See merge request asylum/upstream/controller-runtime!1
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: leontappe
Once this PR has been reviewed and has the lgtm label, please assign fillzpp for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested a review from troy0820 January 23, 2026 16:47
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Jan 23, 2026

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot
Copy link
Contributor

Welcome @leontappe!

It looks like this is your first PR to kubernetes-sigs/controller-runtime 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/controller-runtime has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 23, 2026
@k8s-ci-robot
Copy link
Contributor

Hi @leontappe. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 23, 2026
@troy0820
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 23, 2026
@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 24, 2026
@k8s-ci-robot
Copy link
Contributor

@leontappe: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-controller-runtime-apidiff 60a2b1e link false /test pull-controller-runtime-apidiff

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jan 24, 2026

// HookTimeout is the duration given to each hook to return successfully.
// To use hooks without timeout, set to a negative duration, e.g. time.Duration(-1)
HookTimeout *time.Duration

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When @leontappe and I evaluated the proposed change in the old PR we decided to keep the -1 -> no-timeout in order to conform with previously decided behavior for GracefulShutdownTimeout.
Other projects also use -1 as a value to the denote absence of any timeout/ratelimiting (rest.Config.QPS)

Given that a 0 value is not logically valid, we could reset to the default here, this is also done in GracefulShotdownTimeout

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the comparisons are not great, GracefulShutdownTimeout has an additional state of disable this which is indicated by 0 and restConfig.QPS is not a pointer.

I would probably build this as a value <= 0 means no timeout

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer an explicit value meaning no timeout that we document personally, so I'd say if this is 0, it's no timeout

  • nil: no opinion, use something default
  • explicitly 0: no timeout
  • explicitly >0: use this as the timeout
  • explicitly <0: error

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What other timeout fields do we have in CR as prior art? Or do we only have GracefulShutdownTimeout? (just to end up with something somewhat consistent, it makes sense to make different decisions if the cases are different but we should minimize variations if possible)

Copy link
Member

@alvaroaleman alvaroaleman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AddReadyzCheck(name string, check healthz.Checker) error

// Hook allows to add Runnables as hooks to modify the behavior.
Hook(hook HookType, runnable Runnable) error
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know @vincepri had suggested this name in the other PR, but I am not a huge fan because IMHO:

  • It is different from Add in that its name is what is being added rather than whats being done/the operation
  • Having a HookType argument when there is only one HookType seems needlessly complicated to me and requires us to spread out godocs explaining what this does to this method and the HookType rather than having one place where we can explain it

I would suggest AddPrestartHook since that follows the existing naming pattern and is easy to discover. If we ever add another hook, we can then name that Add${OtherType}Hook.

But lets wait for a second opinion from @sbueringer before you end up changing this back and forth

Copy link
Member

@sbueringer sbueringer Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, AddPrestartHook sounds good.

Is there some prior art from other components (kube-controller-manager, other controller frameworks) that might give us an idea how likely it is that we'll have other types of hooks in the future and what these might be?

(but if we don't have more data let's go with AddPrestartHook)

Hm. Is "Prestart" precise enough to describe "before leader election runnables"? No strong opinion, but it's pretty generic.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+ "to modify the behavior" sounds strange. Let's try to explain this a bit better

// +optional
Controller config.Controller

// HookTimeout is the duration given to each hook to return successfully.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about the idea of having a per-hook timeout rather than a global one for all hooks? Different hook may need different timeouts


func (cm *controllerManager) startLeaderElectionRunnables() error {
cm.logger.Info("Running prestart hooks")
for _, hook := range cm.prestartHooks {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than that being what you found in the original PR, what is the reason for starting the hooks sequentially and not in parallel? Couldn't the sequential starting be achieved if the user instead submitted one hook that does everything sequentlally?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably need to think about how we would collect errors from pre-start hooks on startup and use that to bail out if there are any errors, but that sounds like it would be fairly straightforward with a waitgroup

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. I think we should start them in parallel

}

// All the prestart hooks have ben run, clear the slice to free the underlying resources.
cm.prestartHooks = nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a log here finished running hooks or such so that if this hangs, there is at least some chance to figure out it is because of the hooks?


// HookTimeout is the duration given to each hook to return successfully.
// To use hooks without timeout, set to a negative duration, e.g. time.Duration(-1)
HookTimeout *time.Duration
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the comparisons are not great, GracefulShutdownTimeout has an additional state of disable this which is indicated by 0 and restConfig.QPS is not a pointer.

I would probably build this as a value <= 0 means no timeout

@alvaroaleman alvaroaleman added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Jan 24, 2026
defaultRenewDeadline = 10 * time.Second
defaultRetryPeriod = 2 * time.Second
defaultGracefulShutdownPeriod = 30 * time.Second
defaultHookPeriod = 15 * time.Second
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
defaultHookPeriod = 15 * time.Second
defaultHookTimeoutPeriod = 15 * time.Second

nit

}

switch hook {
case HookPrestartType:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be HookTypePrestart?


func (cm *controllerManager) startLeaderElectionRunnables() error {
cm.logger.Info("Running prestart hooks")
for _, hook := range cm.prestartHooks {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. I think we should start them in parallel

@@ -648,6 +674,27 @@ func (cm *controllerManager) initLeaderElector() (*leaderelection.LeaderElector,
}

func (cm *controllerManager) startLeaderElectionRunnables() error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's rename the func to startPreStartHooksAndLeaderElectionRunnables

AddReadyzCheck(name string, check healthz.Checker) error

// Hook allows to add Runnables as hooks to modify the behavior.
Hook(hook HookType, runnable Runnable) error
Copy link
Member

@sbueringer sbueringer Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, AddPrestartHook sounds good.

Is there some prior art from other components (kube-controller-manager, other controller frameworks) that might give us an idea how likely it is that we'll have other types of hooks in the future and what these might be?

(but if we don't have more data let's go with AddPrestartHook)

Hm. Is "Prestart" precise enough to describe "before leader election runnables"? No strong opinion, but it's pretty generic.


// HookTimeout is the duration given to each hook to return successfully.
// To use hooks without timeout, set to a negative duration, e.g. time.Duration(-1)
HookTimeout *time.Duration
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What other timeout fields do we have in CR as prior art? Or do we only have GracefulShutdownTimeout? (just to end up with something somewhat consistent, it makes sense to make different decisions if the cases are different but we should minimize variations if possible)

AddReadyzCheck(name string, check healthz.Checker) error

// Hook allows to add Runnables as hooks to modify the behavior.
Hook(hook HookType, runnable Runnable) error
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+ "to modify the behavior" sounds strange. Let's try to explain this a bit better


if options.HookTimeout == nil {
hookTimeout := defaultHookPeriod
options.HookTimeout = &hookTimeout
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: if we keep this let's use ptr.To()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants