Ephemeral Runners seem to get stuck when job is canceled or interrupted

[surgiie]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.13.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Unsure of reproduction, seems to be intermittent, happens at random.

Describe the bug

We've been noticing issues with runners randomly getting stuck and prevents workflows from running. Im unsure on a root but it seems that the longest running runners have this error:

[RUNNER 2025-11-07 14:43:16Z INFO Runner] Skipping message Job. Job message not found 'e420db7b-c780-5ae4-8145-4eea09b87aea'. job was canceled

Best i can tell, this seems to be hanging and doesnt get cleaned up. I assume the job was interrupted or manually canceled by the controller or a user. We can get runners rolling again when we delete any ephemeralrunners that have this in their logs message.

However it is unclear to us if this causes another issue we are seeing at the same but it seems some pods have the following error but may just be a result of the above action:

[RUNNER 2025-11-07 14:43:07Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2025-11-07 14:43:22Z WARN GitHubActionsService] Attempt 1 of POST request to https://run-actions-1-azure-eastus.actions.githubusercontent.com/119/acquirejob failed (HTTP Status: GatewayTimeout). The operation will be retried in 11.082 seconds.
[RUNNER 2025-11-07 14:43:38Z ERR  GitHubActionsService] POST request to https://run-actions-1-azure-eastus.actions.githubusercontent.com/119/acquirejob failed. HTTP Status: Conflict
[RUNNER 2025-11-07 14:43:38Z INFO Runner] Skipping message Job. Job message already acquired '269c5bb1-9d88-5672-9f70-5522ca917969'. job assignment is invalid: MissingKey
[RUNNER 2025-11-07 15:33:40Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2025-11-07 15:33:40Z ERR  GitHubActionsService] POST request to https://pipelinesghubeus25.actions.githubusercontent.com/<token>/_apis/oauth2/token failed. HTTP Status: BadRequest
[RUNNER 2025-11-07 15:33:40Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer] Catch exception during request
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer] GitHub.Services.OAuth.VssOAuthTokenRequestException: Registration 470d6c87-f8c4-411f-87d0-6b069233ccc7 was not found.
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at GitHub.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at GitHub.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at GitHub.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at GitHub.Services.Common.RawHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpRequestMessage message, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Uri requestUri, HttpContent content, IEnumerable`1 queryParameters, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at GitHub.Actions.RunService.WebApi.BrokerHttpClient.GetRunnerMessageAsync(Nullable`1 sessionId, String runnerVersion, Nullable`1 status, String os, String architecture, Nullable`1 disableUpdate, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at GitHub.Runner.Common.BrokerServer.<>c__DisplayClass7_0.<<GetRunnerMessageAsync>b__0>d.MoveNext()
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer] --- End of stack trace from previous location ---
[RUNNER 2025-11-07 15:33:40Z ERR  BrokerServer]    at GitHub.Runner.Common.RunnerService.RetryRequest[T](Func`1 func, CancellationToken cancellationToken, Int32 maxAttempts, Func`2 shouldRetry)
[RUNNER 2025-11-07 15:33:40Z WARN BrokerServer] Back off 6.292 seconds before next retry. 4 attempt left.
[RUNNER 2025-11-07 15:33:46Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2025-11-07 15:33:46Z ERR  GitHubActionsService] POST request to https://pipelinesghubeus25.actions.githubusercontent.com/<token>/_apis/oauth2/token failed. HTTP Status: BadRequest
[RUNNER 2025-11-07 15:33:46Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer] Catch exception during request
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer] GitHub.Services.OAuth.VssOAuthTokenRequestException: Registration 470d6c87-f8c4-411f-87d0-6b069233ccc7 was not found.
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at GitHub.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at GitHub.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at GitHub.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at GitHub.Services.Common.RawHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpRequestMessage message, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Uri requestUri, HttpContent content, IEnumerable`1 queryParameters, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at GitHub.Actions.RunService.WebApi.BrokerHttpClient.GetRunnerMessageAsync(Nullable`1 sessionId, String runnerVersion, Nullable`1 status, String os, String architecture, Nullable`1 disableUpdate, CancellationToken cancellationToken)
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at GitHub.Runner.Common.BrokerServer.<>c__DisplayClass7_0.<<GetRunnerMessageAsync>b__0>d.MoveNext()
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer] --- End of stack trace from previous location ---
[RUNNER 2025-11-07 15:33:46Z ERR  BrokerServer]    at GitHub.Runner.Common.RunnerService.RetryRequest[T](Func`1 func, CancellationToken cancellationToken, Int32 maxAttempts, Func`2 shouldRetry)
[RUNNER 2025-11-07 15:33:46Z WARN BrokerServer] Back off 11.706 seconds before next retry. 3 attempt left.

Right now we are getting around this issue by using a cron that cleans up the following:

resources="$(kubectl get ephemeralrunner --namespace "$NS" -l app.kubernetes.io/component=runner -o yaml | yq -r '.items[] | .metadata.name')"

echo "$resources" | while IFS= read -r resource; do
    if [ -z "$resource" ]; then continue; fi
    echo "Checking resource: $resource"
    logs=$(kubectl -n "$NS" logs "$resource" 2>/dev/null)

    if echo "$logs" | grep -qE "MissingKey|job was canceled|token request: Unknown"; then
        echo "Deleting resource: $resource"
        kubectl -n "$NS" delete ephemeralrunner $resource
    fi
done

Describe the expected behavior

Runners should handle this and clean themselves up.

Additional Context

No values that i believe are relevant

Controller Logs

https://gist.github.com/surgiie/ef34874c8942ac965f2d514ff8277b53

Runner Pod Logs

Waiting for docker to be ready.
Docker is ready.
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] No proxy settings were found based on environmental variables (http_proxy/https_proxy/HTTP_PROXY/HTTPS_PROXY)
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'Credentials': '/home/runner/.credentials'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'Runner': '/home/runner/.runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Adding extra user agent 'actions-runner-controller/0.13.0' to all HTTP requests.
[RUNNER 2025-11-07 14:43:14Z INFO Listener] Runner is built for Linux (X64) - linux-x64.
[RUNNER 2025-11-07 14:43:14Z INFO Listener] RuntimeInformation: Ubuntu 22.04.5 LTS.
[RUNNER 2025-11-07 14:43:14Z INFO Listener] Version: 2.329.0
[RUNNER 2025-11-07 14:43:14Z INFO Listener] Commit: c76c59996c8fa32e989164a5a695efb12608ae5a
[RUNNER 2025-11-07 14:43:14Z INFO Listener] Culture: 
[RUNNER 2025-11-07 14:43:14Z INFO Listener] UI Culture: 
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO Listener] Validating directory permissions for: '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO CommandLineParser] Parse
[RUNNER 2025-11-07 14:43:14Z INFO CommandLineParser] Parsing 1 args
[RUNNER 2025-11-07 14:43:14Z INFO CommandLineParser] parsing argument
[RUNNER 2025-11-07 14:43:14Z INFO CommandLineParser] HasArgs: False
[RUNNER 2025-11-07 14:43:14Z INFO CommandLineParser] Adding Command: run
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Removing env var: 'ACTIONS_RUNNER_INPUT_JITCONFIG'
[RUNNER 2025-11-07 14:43:14Z INFO Listener] Arguments parsed
[RUNNER 2025-11-07 14:43:14Z INFO Runner] ExecuteCommand
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] currentAssemblyLocation: /home/runner/bin/Runner.Listener.dll
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] binPath: /home/runner/bin
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] RootFolder: /home/runner
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'Runner': '/home/runner/.runner'
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] ConfigFilePath: /home/runner/.runner
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'MigratedRunner': '/home/runner/.runner_migrated'
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] MigratedConfigFilePath: /home/runner/.runner_migrated
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'Credentials': '/home/runner/.credentials'
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] CredFilePath: /home/runner/.credentials
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'MigratedCredentials': '/home/runner/.credentials_migrated'
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] MigratedCredFilePath: /home/runner/.credentials_migrated
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'Service': '/home/runner/.service'
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] ServiceConfigFilePath: /home/runner/.service
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Flag 'help': 'False'
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Flag 'version': 'False'
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Flag 'commit': 'False'
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Flag 'check': 'False'
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Command 'configure': 'False'
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Command 'remove': 'False'
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Command 'warmup': 'False'
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Env arg 'jitconfig': '***'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO Runner] Saved 426 bytes to '/home/runner/.runner'.
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO Runner] Saved 225 bytes to '/home/runner/.credentials'.
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO Runner] Saved 1631 bytes to '/home/runner/.credentials_rsaparams'.
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] No proxy settings were found based on environmental variables (http_proxy/https_proxy/HTTP_PROXY/HTTPS_PROXY)
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'Credentials': '/home/runner/.credentials'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'Runner': '/home/runner/.runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Adding extra user agent 'actions-runner-controller/0.13.0' to all HTTP requests.
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationManager] LoadSettings
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] IsConfigured()
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] IsConfigured: True
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationManager] Is configured: True
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] Read setting file: 426 chars
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationManager] Settings Loaded
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] IsServiceConfigured()
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] IsServiceConfigured: False
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Command 'run': 'True'
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] IsConfigured()
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] IsConfigured: True
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationManager] Is configured: True
[RUNNER 2025-11-07 14:43:14Z INFO Runner] Could not parse the argument value '' for StartupType. Defaulting to Manual
[RUNNER 2025-11-07 14:43:14Z INFO Runner] Set runner startup type - Manual
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Flag 'once': 'False'
[RUNNER 2025-11-07 14:43:14Z INFO CommandSettings] Flag 'once': 'False'
[RUNNER 2025-11-07 14:43:14Z INFO Runner] RunAsync
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationManager] LoadMigratedSettings
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] IsMigratedConfigured()
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] IsMigratedConfigured: False
[RUNNER 2025-11-07 14:43:14Z INFO Runner] Failed to load migrated settings: No migrated configuration found.
[RUNNER 2025-11-07 14:43:14Z INFO Runner] Falling back to original .runner settings
[RUNNER 2025-11-07 14:43:14Z INFO Runner] Using BrokerMessageListener
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener] Using provided settings
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener] {
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener]   "AgentId": 731233,
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener]   "AgentName": "rr-runner-prod-s8pj5-runner-4sgcl",
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener]   "PoolId": 1,
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener]   "DisableUpdate": true,
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener]   "Ephemeral": true,
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener]   "ServerUrl": "https://pipelinesghubeus25.actions.githubusercontent.com/ZIz4WCgNC1Nnx7QQ74CWOxb0bFEkMdgYcZAOAAJCIPID0UzwsS/",
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener]   "WorkFolder": "_work",
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener]   "UseV2Flow": true,
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener]   "ServerUrlV2": "https://broker.actions.githubusercontent.com/"
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener] }
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener] Loading Credentials
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] HasCredentials()
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] stored True
[RUNNER 2025-11-07 14:43:14Z INFO CredentialManager] GetCredentialProvider
[RUNNER 2025-11-07 14:43:14Z INFO CredentialManager] Creating type OAuth
[RUNNER 2025-11-07 14:43:14Z INFO CredentialManager] Creating credential type: OAuth
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Bin': '/home/runner/bin'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known directory 'Root': '/home/runner'
[RUNNER 2025-11-07 14:43:14Z INFO HostContext] Well known config file 'RSACredentials': '/home/runner/.credentials_rsaparams'
[RUNNER 2025-11-07 14:43:14Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener] Attempt to create session.
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener] Connecting to the Broker Server...
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] HasCredentials()
[RUNNER 2025-11-07 14:43:14Z INFO ConfigurationStore] stored True
[RUNNER 2025-11-07 14:43:14Z INFO CredentialManager] GetCredentialProvider
[RUNNER 2025-11-07 14:43:14Z INFO CredentialManager] Creating type OAuth
[RUNNER 2025-11-07 14:43:14Z INFO CredentialManager] Creating credential type: OAuth
[RUNNER 2025-11-07 14:43:14Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener] VssConnection created
[RUNNER 2025-11-07 14:43:14Z INFO BrokerMessageListener] Connecting to the Runner server...
[RUNNER 2025-11-07 14:43:14Z INFO RunnerServer] EstablishVssConnection
[RUNNER 2025-11-07 14:43:14Z INFO RunnerServer] Establish connection with 100 seconds timeout.
[RUNNER 2025-11-07 14:43:14Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[RUNNER 2025-11-07 14:43:14Z INFO RunnerServer] EstablishVssConnection
[RUNNER 2025-11-07 14:43:14Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[RUNNER 2025-11-07 14:43:14Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[RUNNER 2025-11-07 14:43:14Z INFO RunnerServer] EstablishVssConnection
[RUNNER 2025-11-07 14:43:14Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[RUNNER 2025-11-07 14:43:14Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[RUNNER 2025-11-07 14:43:15Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[RUNNER 2025-11-07 14:43:15Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[RUNNER 2025-11-07 14:43:15Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[RUNNER 2025-11-07 14:43:16Z INFO BrokerMessageListener] VssConnection created
[RUNNER 2025-11-07 14:43:16Z INFO Terminal] WRITE LINE: 

√ Connected to GitHub
[RUNNER 2025-11-07 14:43:16Z INFO Terminal] WRITE LINE: 

[RUNNER 2025-11-07 14:43:16Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2025-11-07 14:43:16Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2025-11-07 14:43:16Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2025-11-07 14:43:16Z INFO BrokerMessageListener] Session created.
[RUNNER 2025-11-07 14:43:16Z INFO Terminal] WRITE LINE: Current runner version: '2.329.0'
Current runner version: '2.329.0'
[RUNNER 2025-11-07 14:43:16Z INFO Terminal] WRITE LINE: 2025-11-07 14:43:16Z: Listening for Jobs
2025-11-07 14:43:16Z: Listening for Jobs
[RUNNER 2025-11-07 14:43:16Z INFO JobDispatcher] Set runner/worker IPC timeout to 30 seconds.
[RUNNER 2025-11-07 14:43:16Z INFO ConfigurationStore] HasCredentials()
[RUNNER 2025-11-07 14:43:16Z INFO ConfigurationStore] stored True
[RUNNER 2025-11-07 14:43:16Z INFO CredentialManager] GetCredentialProvider
[RUNNER 2025-11-07 14:43:16Z INFO CredentialManager] Creating type OAuth
[RUNNER 2025-11-07 14:43:16Z INFO CredentialManager] Creating credential type: OAuth
[RUNNER 2025-11-07 14:43:16Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2025-11-07 14:43:16Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2025-11-07 14:43:16Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/runner/.credentials_rsaparams
[RUNNER 2025-11-07 14:43:16Z INFO GitHubActionsService] AAD Correlation ID for this token request: Unknown
[RUNNER 2025-11-07 14:43:16Z ERR  GitHubActionsService] POST request to https://run-actions-3-azure-eastus.actions.githubusercontent.com/64/acquirejob failed. HTTP Status: NotFound
[RUNNER 2025-11-07 14:43:16Z INFO Runner] Skipping message Job. Job message not found 'e420db7b-c780-5ae4-8145-4eea09b87aea'. job was canceled

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[badstreff]

If you do a kubectl get ephemeralrunners do you see a lot of them sitting in a failed state? I have PR - #4272 that may also resolve the issue you see here but has not been getting much traction.