Skip to content

Conversation

@lhotari
Copy link
Member

@lhotari lhotari commented Dec 4, 2025

@lhotari
Copy link
Member Author

lhotari commented Dec 15, 2025

Running into some issues: https://gist.github.com/lhotari/8fc6117098dfcba10bf5758b2953c93d

[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] panic: runtime error: invalid memory address or nil pointer dereference
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xcc9938]
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] goroutine 72 [running]:
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/pingcap/errors.IsNotFound({0x0?, 0x0?})
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:112 +0x18
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/metadata.(*clientImpl[...]).Upsert(0x2dce600?, {0x7ffcc39dc60d, 0x41fd25?}, {0x7ffcc39dc629?, 0x21}, 0xc00028e7e0)
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/metadata/k8s_client.go:109 +0x1c5
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/metadata.(*metadataProviderConfigMap).Store(0xc0002cd220, 0xc00051ba88, {0xc00019c9d8, 0x4})
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/metadata/metadata_configmap.go:143 +0x2d8
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/resource.(*status).UpdateShardMetadata.func1()
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/resource/status_resource.go:152 +0x3a
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1()
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:18 +0x13
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/cenkalti/backoff/v4.doRetryNotify[...](0xc0004bcf78?, {0x2da3960, 0xc00003fda0}, 0xc0004bd0e8, {0x0, 0x0?})
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:88 +0x11c
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/cenkalti/backoff/v4.RetryNotifyWithTimer(0x0?, {0x2da3960?, 0xc00003fda0?}, 0x526fd8?, {0x0?, 0x0?})
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:61 +0x56
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/cenkalti/backoff/v4.RetryNotify(...)
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:49
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/resource.(*status).UpdateShardMetadata(0xc0003b7040, {0xc00019ca50, 0xa}, 0x8, {0x2, 0x1, 0x0, {0xc00020d100, 0x3, 0x3}, ...})
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/resource/status_resource.go:151 +0x3e9
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/controller.(*ShardElection).start(0xc00026f400)
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/controller/shard_controller_election.go:411 +0x179
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/controller.(*ShardElection).Start.func1()
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/controller/shard_controller_election.go:504 +0x29
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/cenkalti/backoff/v4.doRetryNotify[...](0xc0004bd858?, {0x7f2829499e00, 0xc00009bac0}, 0xc0004bd848, {0x0, 0x0?})
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:88 +0x155
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/cenkalti/backoff/v4.RetryNotifyWithData[...](...)
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:54
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/controller.(*ShardElection).Start(0x2d99ba0?)
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/controller/shard_controller_election.go:503 +0x1fe
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/controller.(*shardController).onElectLeader(0xc000126b60, 0x0)
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/controller/shard_controller.go:326 +0x378
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/controller.(*shardController).run(0xc000126b60, 0xc0001f2f50)
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/controller/shard_controller.go:196 +0x15f
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/controller.NewShardController.func2.1()
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/controller/shard_controller.go:174 +0x1b
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/common/process.DoWithLabels.func1({0x2dc2468?, 0xc00002ecc0?})
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/common/process/pprof.go:46 +0x13
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] runtime/pprof.Do({0x2dc24a0?, 0xc0002e6960?}, {{0xc00003e4e0?, 0x2e06e40?, 0x64?}}, 0xc000531dd8)
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/usr/local/go/src/runtime/pprof/runtime.go:51 +0x8c
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/common/process.DoWithLabels({0x2dc24a0, 0xc0002e6960}, 0xc000531f38?, 0xc000531f68)
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/common/process/pprof.go:42 +0x295
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] github.com/oxia-db/oxia/oxiad/coordinator/controller.NewShardController.func2()
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/src/oxia/oxiad/coordinator/controller/shard_controller.go:167 +0x257
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] sync.(*WaitGroup).Go.func1()
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/usr/local/go/src/sync/waitgroup.go:239 +0x4a
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] created by sync.(*WaitGroup).Go in goroutine 1
[pod/pulsar-ci-oxia-coordinator-b548d97b6-ggqqm/coordinator] 	/usr/local/go/src/sync/waitgroup.go:237 +0x73

@lhotari lhotari changed the title Upgrade Oxia to 0.15.0 Upgrade Oxia to 0.15.1 Dec 15, 2025
@lhotari lhotari merged commit 804192a into apache:master Dec 15, 2025
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant