SDS usage crashes unit tests

[igor-sirotin]

Description

For some reason, unit test runtime decreased from 30 to 8 minutes. Something is broken.

So far #7149 seem to be the culprit:

Coverage decreased by 8 percent:

Why PR checks didn't detect this?

1. Unfortunately, tests-develop job was disabled. Perhaps codecov project limits would have caught this. Though maybe not, it's configured for 10% drop.
2. I'm not sure why the CI still passed, I'm investigating.

Are tests broken?

They pass on CI.

Tried to run locally, got this:

2025-12-16T14:52:42.102Z	DEBUG	9b0adc.reliability.sds	sds/logging.go:34	Creating new Reliability Manager
SIGSEGV: segmentation violation
PC=0x1084d51cc m=18 sigcode=2 addr=0x0
signal arrived during cgo execution

goroutine 440731 gp=0x1400104cc40 m=18 mp=0x1400312e008 [syscall]:
runtime.cgocall(0x104f0274c, 0x1400d584508)
	/Users/sirotin/go/go1.24.7/src/runtime/cgocall.go:167 +0x44 fp=0x1400d5844d0 sp=0x1400d584490 pc=0x1029c1444
github.com/waku-org/sds-go-bindings/sds._Cfunc_cGoSdsNewReliabilityManager(0xc82a77340)
	_cgo_gotypes.go:166 +0x34 fp=0x1400d584500 sp=0x1400d5844d0 pc=0x103a37ff4
github.com/waku-org/sds-go-bindings/sds.NewReliabilityManager.func6(...)
	/Users/sirotin/go/pkg/mod/github.com/waku-org/sds-go-bindings@v0.0.0-20251119184907-e78f76307965/sds/sds.go:169
github.com/waku-org/sds-go-bindings/sds.NewReliabilityManager()
	/Users/sirotin/go/pkg/mod/github.com/waku-org/sds-go-bindings@v0.0.0-20251119184907-e78f76307965/sds/sds.go:169 +0x240 fp=0x1400d5845c0 sp=0x1400d584500 pc=0x103a38870
github.com/status-im/status-go/messaging/layers/reliability.newSdsReliabilityManager(0x140012e3080)
	/Users/sirotin/Repositories/status/status-go/messaging/layers/reliability/sds.go:15 +0x98 fp=0x1400d5846a0 sp=0x1400d5845c0 pc=0x103a49c38
github.com/status-im/status-go/messaging/layers/reliability.NewReliability({0x1065fa528, 0x14020873e68}, 0x14013c9fb60, 0x24?)
	/Users/sirotin/Repositories/status/status-go/messaging/layers/reliability/reliability.go:43 +0xd4 fp=0x1400d5846e0 sp=0x1400d5846a0 pc=0x103a48274
github.com/status-im/status-go/messaging.newCore({0x1066382c0, 0x14000f4c780}, {0x14013c9fb60, {0x140284e9d40, 0x24}, 0x0, {0x0, {0x0, 0x0}, 0x0, ...}, ...}, ...)
	/Users/sirotin/Repositories/status/status-go/messaging/core.go:90 +0x3a4 fp=0x1400d5847a0 sp=0x1400d5846e0 pc=0x103b86ad4
github.com/status-im/status-go/messaging.(*TestMessagingEnvironment).NewTestCore(0x14026974c40, {0x14013c9fb60, {0x140284e9d40, 0x24}, 0x0, {0x0, {0x0, 0x0}, 0x0, 0x0, ...}, ...}, ...)
	/Users/sirotin/Repositories/status/status-go/messaging/core_test_utils.go:51 +0xe8 fp=0x1400d5848b0 sp=0x1400d5847a0 pc=0x103b89498
github.com/status-im/status-go/protocol.newTestMessenger(0x14026974c40, {{0x1400a3b99e0, 0x6}, 0x14013c9fb60, 0x14002150680, 0x0, 0x0, 0x1401c901808, 0x14001779b08, {0x1400dc32780, ...}})
	/Users/sirotin/Repositories/status/status-go/protocol/messenger_builder_test.go:129 +0xa84 fp=0x1400d5858c0 sp=0x1400d5848b0 pc=0x104940284
github.com/status-im/status-go/protocol.newTestCommunitiesMessenger(0x14002f16f00, 0x14026974c40, {{{0x0, 0x0}, 0x14013c9fb60, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	/Users/sirotin/Repositories/status/status-go/protocol/communities_messenger_helpers_test.go:331 +0x1010 fp=0x1400d585ab0 sp=0x1400d5858c0 pc=0x10487d7a0
github.com/status-im/status-go/protocol.(*CommunitiesMessengerTestSuiteBase).newMessengerWithConfig(0x14002f16f00, {{0x0, 0x0}, 0x14013c9fb60, 0x0, 0x0, 0x0, 0x0, 0x0, {0x1402af02718, ...}}, ...)
	/Users/sirotin/Repositories/status/status-go/protocol/communities_messenger_test_suite_base_test.go:58 +0x118 fp=0x1400d585c40 sp=0x1400d585ab0 pc=0x104909868
github.com/status-im/status-go/protocol.(*EventSenderCommunityEventsSuiteBase).newMessenger(0x14002f16f00, {0x0, 0x0}, {0x1082a1020, 0x0, 0x0})
	/Users/sirotin/Repositories/status/status-go/protocol/communities_messenger_admin_test.go:84 +0x2c0 fp=0x1400d585d60 sp=0x1400d585c40 pc=0x1048794f0
github.com/status-im/status-go/protocol.(*EventSenderCommunityEventsSuiteBase).SetupTest(0x14002f16f00)
	/Users/sirotin/Repositories/status/status-go/protocol/communities_messenger_admin_test.go:65 +0x78 fp=0x1400d585db0 sp=0x1400d585d60 pc=0x104879068
github.com/stretchr/testify/suite.Run.func1(0x140273d21c0)
	/Users/sirotin/go/pkg/mod/github.com/stretchr/testify@v1.11.1/suite/suite.go:188 +0x170 fp=0x1400d585f60 sp=0x1400d585db0 pc=0x103823e70
testing.tRunner(0x140273d21c0, 0x14005e0b830)
	/Users/sirotin/go/go1.24.7/src/testing/testing.go:1792 +0xe4 fp=0x1400d585fb0 sp=0x1400d585f60 pc=0x102aef164
testing.(*T).Run.gowrap1()
	/Users/sirotin/go/go1.24.7/src/testing/testing.go:1851 +0x2c fp=0x1400d585fd0 sp=0x1400d585fb0 pc=0x102aeffec
runtime.goexit({})
	/Users/sirotin/go/go1.24.7/src/runtime/asm_arm64.s:1223 +0x4 fp=0x1400d585fd0 sp=0x1400d585fd0 pc=0x1029cdb04
created by testing.(*T).Run in goroutine 241937
	/Users/sirotin/go/go1.24.7/src/testing/testing.go:1851 +0x374

[igor-sirotin]

cc @kaichaosun

[kaichaosun]

@igor-sirotin did you enable the flag in your local sdsForCommunitiesEnabled? This should be disabled when running tests since the existing tests may be affected. It should be fixed once we enable this flag by default.

[igor-sirotin]

@igor-sirotin did you enable the flag in your local sdsForCommunitiesEnabled? This should be disabled when running tests since the existing tests may be affected. It should be fixed once we enable this flag by default.

No, I didn't. Running simply on latest develop.

But I don't see this panic on CI. And I wonder what happened with the tests runtime.

The only thing I noticed so far is tests count. This is on SDS PR:

This is previous PR:

[igor-sirotin]

@kaichaosun I've temporarily reverted the integration here: #7218

Tests actually fail, which can be see here: https://ci.status.im/job/status-go/job/prs/job/tests/job/PR-7149/34/testReport/github/com_status-im_status-go_protocol/

*** bit out of range 0 - FD_SETSIZE on fd_set ***: terminated
SIGABRT: abort
PC=0x7f4771b4b84c m=16 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 661772 gp=0xc0004d9a40 m=16 mp=0xc000481008 [syscall]:
runtime.cgocall(0x2bd3e60, 0xc00fd6df08)
	/nix/store/5gkj2kc7drgf6pclxcl4fyg8zlcaqqmd-go-1.24.6/share/go/src/runtime/cgocall.go:167 +0x4b fp=0xc00fd6dee0 sp=0xc00fd6dea8 pc=0x47bc4b
github.com/waku-org/sds-go-bindings/sds._Cfunc_cGoSdsNewReliabilityManager(0x7f46e8d5dae0)
	_cgo_gotypes.go:167 +0x45 fp=0xc00fd6df08 sp=0xc00fd6dee0 pc=0x16ba245
github.com/waku-org/sds-go-bindings/sds.NewReliabilityManager.func6(...)
	/home/jenkins/workspace/status-go_prs_tests_PR-7149@tmp/go/pkg/mod/github.com/waku-org/sds-go-bindings@v0.0.0-20251119184907-e78f76307965/sds/sds.go:169
github.com/waku-org/sds-go-bindings/sds.NewReliabilityManager()
	/home/jenkins/workspace/status-go_prs_tests_PR-7149@tmp/go/pkg/mod/github.com/waku-org/sds-go-bindings@v0.0.0-20251119184907-e78f76307965/sds/sds.go:169 +0x2aa fp=0xc00fd6dfc0 sp=0xc00fd6df08 pc=0x16bacaa

We're checking why the failing test didn't fail the CI job.

[igor-sirotin]

This PR confirms that exit code is kept as 0, although test failed:

• ci: add exit code files to artifacts #7219

[kaichaosun]

xai has some clue which seems related to cgo usage in sds-go-bindings, maybe you have more context about it @Ivansete-status ?

[igor-sirotin]

Ovservations

1. It's always different tests.
2. I reproduced locally, but it crashes with SIGSEGV (not SIGABRT)
3. When running a single test or a test suite, it passes fine. You need to run the whole protocol package with run_unit_tests.sh

Explanations

Why it crashed?

The crashes come from sds.NewReliabilityManager(). Every test that builds a Core/Messenger allocates another sds.ReliabilityManager, but nothing ever calls Cleanup() on it. Over the protocol suite this leaks hundreds of SDS instances.

SDS uses select internally; once the leaked instances accumulate enough open fds, it ends up calling FD_SET with an fd outside FD_SETSIZE.

Why different stop signals?

On Linux that trips the runtime check and aborts with *** bit out of range 0 - FD_SETSIZE on fd_set *** → SIGABRT.
On macOS the same bad fd just produces a null/invalid access, so it shows up as SIGSEGV.

Why is CI green?

The protocol run hit the same SDS leak and the process aborted (SIGABRT). Because UNIT_TEST_RERUN_FAILS=true in CI, gotestsum immediately reran only the tests it recorded as failed before the abort. That rerun happens in a fresh process (no pile of leaked SDS managers yet) so it completes and exits 0. run_unit_tests.sh records that 0 into
exit_code_1.txt and keeps going.

Net effect: the first process died mid-run, so only ~134 protocol tests ran; the rerun only covered the handful marked as failed, so the remaining ~800 protocol tests were never executed. With the rerun passing, gotestsum’s exit status is 0 and the pipeline stays green even though most of the suite was skipped.

How to catch such issues on CI?

SImplest option: don't re-run tests.

Or add even more shenanigans into run_unit_tests.sh to treat an abort/short run as failure. E.g. compare the reported test count from gotestsum/JUnit vs an expected baseline, or grep the log for SIGABRT/SIGSEGV and exit non-zero. Either forces the job red when the process dies before running the full suite.

[igor-sirotin]

It got better with the cleanup, but not exactly as it should be.

There's still SIGABRT / SIGSEGV in logs.

[igor-sirotin]

This is suspicious.

> cat .run_local_3/test_1.log | grep "Creating new Reliability Manager" | wc -l
      69
> cat .run_local_3/test_1.log | grep "Cleaning up reliability manager" | wc -l 
      37