validation: Remove addrman.cpp suppression per reviewer feedback #4574

Job	Run time
Build slim container / Build container	30s
Build container / Build container	24s
arm-linux-gnueabihf / Build depends	16m 5s
x86_64-pc-linux-gnu / Build depends	17m 13s
x86_64-pc-linux-gnu_nowallet / Build depends	15m 14s
x86_64-apple-darwin / Build depends	15m 19s
x86_64-w64-mingw32 / Build depends	18m 27s
x86_64-pc-linux-gnu_multiprocess	-1s
linux64_multiprocess-build	0s
linux64_tsan-build	0s
linux64_multiprocess-test	0s
linux64_tsan-test	0s
linux64_nowallet-build / Build source	21m 42s
mac-build / Build source	28m 58s
arm-linux-build / Build source	26m 3s
linux64_sqlite-build	-1s
linux64_ubsan-build	-1s
linux64_fuzz-build	-1s
linux64-build / Build source	20m 42s
linux64_sqlite-test	0s
linux64_ubsan-test	0s
win64-build / Build source	34m 32s
linux64_nowallet-test / Test source	7m 36s
linux64-test / Test source	9m 15s
	3h 51m 56s

Provide feedback