✨ Add pre-start hook support to manager#3429
✨ Add pre-start hook support to manager#3429leontappe wants to merge 6 commits intokubernetes-sigs:mainfrom
Conversation
When implementing a controller that uses leader election, there maybe be work that needs to be done after winning the election but before processing enqueued requests. For example, a controller may need to build up an internal mapping of the current state of the cluster before it can begin reconciling. This changeset adds support for adding prestart hooks to controller-runtime's manager implementation. This hook runs after the manager has been elected leader, immediately before the leader election controllers are started. Related kubernetes-sigs#607
…into prestart-hook
feat(manager): add prestart hook support See merge request asylum/upstream/controller-runtime!1
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: leontappe The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @leontappe! |
|
Hi @leontappe. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
📖 fix typo in hook type validation error
|
@leontappe: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
|
||
| // HookTimeout is the duration given to each hook to return successfully. | ||
| // To use hooks without timeout, set to a negative duration, e.g. time.Duration(-1) | ||
| HookTimeout *time.Duration |
There was a problem hiding this comment.
When @leontappe and I evaluated the proposed change in the old PR we decided to keep the -1 -> no-timeout in order to conform with previously decided behavior for GracefulShutdownTimeout.
Other projects also use -1 as a value to the denote absence of any timeout/ratelimiting (rest.Config.QPS)
Given that a 0 value is not logically valid, we could reset to the default here, this is also done in GracefulShotdownTimeout
There was a problem hiding this comment.
I think the comparisons are not great, GracefulShutdownTimeout has an additional state of disable this which is indicated by 0 and restConfig.QPS is not a pointer.
I would probably build this as a value <= 0 means no timeout
There was a problem hiding this comment.
I'd prefer an explicit value meaning no timeout that we document personally, so I'd say if this is 0, it's no timeout
- nil: no opinion, use something default
- explicitly 0: no timeout
- explicitly >0: use this as the timeout
- explicitly <0: error
There was a problem hiding this comment.
What other timeout fields do we have in CR as prior art? Or do we only have GracefulShutdownTimeout? (just to end up with something somewhat consistent, it makes sense to make different decisions if the cases are different but we should minimize variations if possible)
| AddReadyzCheck(name string, check healthz.Checker) error | ||
|
|
||
| // Hook allows to add Runnables as hooks to modify the behavior. | ||
| Hook(hook HookType, runnable Runnable) error |
There was a problem hiding this comment.
I know @vincepri had suggested this name in the other PR, but I am not a huge fan because IMHO:
- It is different from
Addin that its name is what is being added rather than whats being done/the operation - Having a
HookTypeargument when there is only oneHookTypeseems needlessly complicated to me and requires us to spread out godocs explaining what this does to this method and theHookTyperather than having one place where we can explain it
I would suggest AddPrestartHook since that follows the existing naming pattern and is easy to discover. If we ever add another hook, we can then name that Add${OtherType}Hook.
But lets wait for a second opinion from @sbueringer before you end up changing this back and forth
There was a problem hiding this comment.
Agree, AddPrestartHook sounds good.
Is there some prior art from other components (kube-controller-manager, other controller frameworks) that might give us an idea how likely it is that we'll have other types of hooks in the future and what these might be?
(but if we don't have more data let's go with AddPrestartHook)
Hm. Is "Prestart" precise enough to describe "before leader election runnables"? No strong opinion, but it's pretty generic.
There was a problem hiding this comment.
+ "to modify the behavior" sounds strange. Let's try to explain this a bit better
| // +optional | ||
| Controller config.Controller | ||
|
|
||
| // HookTimeout is the duration given to each hook to return successfully. |
There was a problem hiding this comment.
WDYT about the idea of having a per-hook timeout rather than a global one for all hooks? Different hook may need different timeouts
|
|
||
| func (cm *controllerManager) startLeaderElectionRunnables() error { | ||
| cm.logger.Info("Running prestart hooks") | ||
| for _, hook := range cm.prestartHooks { |
There was a problem hiding this comment.
Other than that being what you found in the original PR, what is the reason for starting the hooks sequentially and not in parallel? Couldn't the sequential starting be achieved if the user instead submitted one hook that does everything sequentlally?
There was a problem hiding this comment.
Probably need to think about how we would collect errors from pre-start hooks on startup and use that to bail out if there are any errors, but that sounds like it would be fairly straightforward with a waitgroup
There was a problem hiding this comment.
Agree. I think we should start them in parallel
| } | ||
|
|
||
| // All the prestart hooks have ben run, clear the slice to free the underlying resources. | ||
| cm.prestartHooks = nil |
There was a problem hiding this comment.
Maybe add a log here finished running hooks or such so that if this hangs, there is at least some chance to figure out it is because of the hooks?
|
|
||
| // HookTimeout is the duration given to each hook to return successfully. | ||
| // To use hooks without timeout, set to a negative duration, e.g. time.Duration(-1) | ||
| HookTimeout *time.Duration |
There was a problem hiding this comment.
I think the comparisons are not great, GracefulShutdownTimeout has an additional state of disable this which is indicated by 0 and restConfig.QPS is not a pointer.
I would probably build this as a value <= 0 means no timeout
| defaultRenewDeadline = 10 * time.Second | ||
| defaultRetryPeriod = 2 * time.Second | ||
| defaultGracefulShutdownPeriod = 30 * time.Second | ||
| defaultHookPeriod = 15 * time.Second |
There was a problem hiding this comment.
| defaultHookPeriod = 15 * time.Second | |
| defaultHookTimeoutPeriod = 15 * time.Second |
nit
| } | ||
|
|
||
| switch hook { | ||
| case HookPrestartType: |
There was a problem hiding this comment.
Should this be HookTypePrestart?
|
|
||
| func (cm *controllerManager) startLeaderElectionRunnables() error { | ||
| cm.logger.Info("Running prestart hooks") | ||
| for _, hook := range cm.prestartHooks { |
There was a problem hiding this comment.
Agree. I think we should start them in parallel
| @@ -648,6 +674,27 @@ func (cm *controllerManager) initLeaderElector() (*leaderelection.LeaderElector, | |||
| } | |||
|
|
|||
| func (cm *controllerManager) startLeaderElectionRunnables() error { | |||
There was a problem hiding this comment.
Let's rename the func to startPreStartHooksAndLeaderElectionRunnables
| AddReadyzCheck(name string, check healthz.Checker) error | ||
|
|
||
| // Hook allows to add Runnables as hooks to modify the behavior. | ||
| Hook(hook HookType, runnable Runnable) error |
There was a problem hiding this comment.
Agree, AddPrestartHook sounds good.
Is there some prior art from other components (kube-controller-manager, other controller frameworks) that might give us an idea how likely it is that we'll have other types of hooks in the future and what these might be?
(but if we don't have more data let's go with AddPrestartHook)
Hm. Is "Prestart" precise enough to describe "before leader election runnables"? No strong opinion, but it's pretty generic.
|
|
||
| // HookTimeout is the duration given to each hook to return successfully. | ||
| // To use hooks without timeout, set to a negative duration, e.g. time.Duration(-1) | ||
| HookTimeout *time.Duration |
There was a problem hiding this comment.
What other timeout fields do we have in CR as prior art? Or do we only have GracefulShutdownTimeout? (just to end up with something somewhat consistent, it makes sense to make different decisions if the cases are different but we should minimize variations if possible)
| AddReadyzCheck(name string, check healthz.Checker) error | ||
|
|
||
| // Hook allows to add Runnables as hooks to modify the behavior. | ||
| Hook(hook HookType, runnable Runnable) error |
There was a problem hiding this comment.
+ "to modify the behavior" sounds strange. Let's try to explain this a bit better
|
|
||
| if options.HookTimeout == nil { | ||
| hookTimeout := defaultHookPeriod | ||
| options.HookTimeout = &hookTimeout |
There was a problem hiding this comment.
nit: if we keep this let's use ptr.To()
I have a use case similar to the one described in a previous PR and this issue.
My controller needs to consume job status files on start, that would normally be worked on by a custom runnable during runtime. With vanilla controller-runtime this is currently not possible as any code that is run before reconciliation, has no access to a running manager.
This PR provides a way to register runnables as hooks that you can supply to your manager.
This PR is based on a fork of https://github.com/terinjokes/controller-runtime/tree/prestart-hook which is in turn a fork of this repo. I rebased the current master onto the latest development state of the original PR and implemented some of the proposals made by @JoelSpeed