Giant hex dump in logs

[paullinator]

We occasionally see a giant blob of hex in the logs of the format below. There is no prefix to the dump so we can't find where it is being printed. Is there any concern and if this is benign, is there a way to turn this off?

Dec 01 16:15:33 monerolws-s1-wusa1 monero-lws-daemon[106097]: bd0fd236359f38ea0573492b3b3aaf2f479cf04787abb7831e1dd0cfd1b320e72787f96e7f5d965d91f23ff00d33b8dfa12e401f112e882e827918adbf8c7a166cdb77896c4bc968f533748dd3d19cd84e45cb286c39f11c661e0b8926dcbef8f47cb17bfe8f719966138a31c6b7d0e59ec66ba91a09df165d7e2fd5c3b806f36facf4fc327880f65835fac3c37d7d04d600fa3989c4c202f08603dd960c216d08dc1f6f4fecb3bb30000f0f8cb3c9bbf83e3d5d9711facf6f1a4c743eae1a2ec13fab4dc18399e8eee7b67d5c071fdc77253e6e8059e81095736ef31e419d58fab92d5cce7c5285ab07c45aa796adebc38efa65bd28f4f239c8dc2118c7ade5d964dc00b9f3ea71af69fc0903deb2679ad3fcf72247fa27d3e4b1ddcb6a3f361e40c666c70b4c42702156eab09006464c8b6071b4fa7f0cf2dc96c3cc49ebc3fe44f6fc10ef167d3ecb98fe59237f72e377c43d23e66ddfe65ee6dc3d2d6767880b45810d679c5b1a82e32321f1724ab71f569f51f5bed69eec7b25773e111f0394b0e0103cf20b293b7ca5b870707dcd203af564eb8b97cbedb68a2c6f0200168e7321e756612725cd807ecc8872f4c3c43a16beece77e1129c834fcdb6efc03c9ea34116c05f4248cd8273cb3b726d526b07990af2418638fd4275043a807ab4181349ba458895f564f24b08f0ef5e5986c90210a611bf519f566108f99e23f642ec3b8c0f03dc795df04af620d2e5d4e39089751ef840cc090c48ea6c9e764a31c620fe2212e7bf7b8bb96837c242fc067c37a9d7c3f5bf306ffea0478218937b9c2c472c100c30193e309aeb1f589f081f528e4877a9890b81234d9c5711ec3c18b91a10d64faf0e62b1231686fbcbab23fa9b07791cd8119807b8a5e5b821f565d5c332cc8df5e16cd98242e076e76a63a1fc0f39b233af336ef48cfa8f738716c3b389aada3616412c96cf71d4971e461b19237d7883691b8df7046ecb944b896b6129c8312121c49e11c6176ac8e36243c5352658ac9997d918226c6747f72ebf16866c7a8ace3983e1e0df2da37f6c27c8db10c9a451c0b02013c97330cf3b72960801be53481917af0d9c4bc466e1884df36fe1b6d9cbec8bd60d5a4a5b590b378a5b5c369cc70858b7d45c2348ab7361a438b198970e9b1d3598e8daab10a20db7186e0c9e7034cbabc1033e0b618eed12794743e0f35dfbf07b9f88ea27462f0b94e23c41c73673b6784c6fd24b2c4ea36ded8ac9d8b921763b2024877c2dc5e88c239b0d231d83395c5d87b9703dc345f9659c3ee107b946e176d8e6464985be9d4096ddc48747885816e2cf276394f993df7227f2671a4ade678ec68839973e018c6dee6ea3d8e123f13f0b93b8b2e5e06e1cf3b7be3c1e62b9b4ee39e43c4f87d3c699c43138a7b683ab17234f077e17041bbc56cc606ca752749690f1e3d4f32a434e21d6d8ac2d0c7fb9820da04b99ae4ce79b0e33fc4a9e4e50f16439c7831e1176999e039c9bc7c766a678cf029ce1c6e345f4f17533d1658deacbb20b6d8b42536e53e1d8349aa4cb22d396efb6de7602fd37b1d65a8e4fb937c9c888c9efc8e788e2f702a65930f7c1890697782684df03c9ee50edd329ee1c23c45bdc7cd98b24e639db5f20404e7d44cef57e96931cc712475629cf9b679b241f078b570f10f0ec7820b9b0e11558758023878bb1c4718c5b2f576c274325396db1f522dc778b8972edc27b7ed98f3bc4396c4c70dbadb7065be039b979f10c823c66c1e1e105912cc7ea75e1cb71edc7a8ddea23988eecf0455858fa659cdde56ae700b2cdd947e9c43be11d488117ea1425be1c263bbcc7ad9d7649d27c8438f760df50f921e2eb89739658783c782a5e6663991419e059a4407e2082ea3c7bf1c478213b9e3c4a1d4ec1ee20f01b3e15b1e121b53ebc171cc730d771f67481d3d4bc39c3e57b13f52c2d44dd3f0a78b837296f5bdf883b7d63e54d91c0c3a6399e078298231942f890410e60f31f0b70c4191d49ee3bc60751221c7186d2320c72ceedc796e5e0ea4e38b85bca1e377cb38ec9c9534f07cdc2e36f27c36dcaa4ed9fc1770573e0ba8cc326e7f707e277c3043c5c5c2e0dce75309c0f370f9bcbca1cce01881c30e36ebe09cc1021ebc3adcfc08847082082cbf5677a5992f32ebb2b3dc26ae1e1c6d2f74318be0ae6e466f87c87323c359e373c7b0fcf8b1fb96c38dd8e081ecbfcd89848e163618f2cf70b891699dc9077b2dee52ef372573e91cfcc1b823c0bdc9884bb3dc9bc90724f5e02488e78bf11e4220820c7320790339f038765cf850d9c9b99a5b707c40b4b8784e7c5d8f8b57c99ced202ea13989b0832ee305ea789c992eb0644c300b635db6ddb23dadcb564b818ece0c3f0af76e3e04f71b24c658b8781c4baf8f7f01222c7904164c7d1cbe8cf1c44db7277b97d4c5e63887b8e797dc4fe63e5d8c73973ee4e8ea0e72e682d4c82c0ea09c0b9dc2e7e0111df8efbbd4cf87319c33e4b1d9227834bd20df1e25f7e0e6d55b716d989b0bc4a1b64375846221e25b10c2efe0661c442fd26b57b2e6b8420a73cc1d41c6d9043798c37e6e5d79130ee403f73cc82858af8087a41f77099bcc591164f56108cf88be26f4f8ba78980f0064eff0077832c81e07d33fb8cd1b788f97a5bcddee7b9e478e485d43e07fb3b9451c884f89dfa9b990ce2dc945c05c2cb20ff0010f01927c190d8a3e23c6c32082f6be23b219288f8df012ee6134f2fa8e7c58c2c6f86648e7fdfe1215c9ef9b9f765a147c2d112cdc672108ebc0dc81974c61e21041b63c7964f39f183398f866dc664e23bd9197065bea5e19b0e8ca3f73cea7e018df19c1768bbf8c6c7567e0e91757d5cc5babfdf072dcfe013c20bfdb7bf051221ec3748f1c305d2692bc950821c4108460838ba8239b232d9318210e277cb3ddbcb0e445218092f4dbb76960328789f5f1217894730ee4783623bf3c65c4c38b74993ec5fed8fe2e8e27e005ace4b36d508e48343cddae99e1a737fadf1fa5c230717043cdc223c08f096430e610f90df09fa85f89b9b52de3e0808830964478ef9e37c7a8ea249e1b278e2d066cfc3f886df0f29e1d72e5323011c0cb27c5664f2903a9c273e61c43863968fe0fc441e023107b222ed8c38859b06c7d1b38e2e73c5df92

[vtnerd]

Does lws shutdown after this? Or does it keep running?

[vtnerd]

This looks like a raw call to std::cout as there's no timestamp or log level from lws.

[paullinator]

It keeps running, but it makes it hard to review logs. Can you tell what part of the code is outputting it?

[vtnerd]

I am uncertain on the origins of this dump. No one else has reported this, but it's also possible people are ignoring it.

As stated earlier, there is no logging prefix to indicate the log-level or timestamp, so it looks like a raw call to printf or std::cout. I have searched the codebase and found nothing useful. It might be in upstream monero, but it seems like this would've been caught by now.

Can you at least confirm that everything before [106097]: is from docker or some 3rd party instrumentation? In other words, the hex dump is what monero-lws-daemon output, and everything before that is from some other utility?

[paullinator]

Yes, this entire section Dec 01 16:15:33 monerolws-s1-wusa1 monero-lws-daemon[106097]: is from the syslog. I'm using journalctl -fu monero-lws to get the logs.

[vtnerd]

This might be the core monero code dumping the hex for tx_extra when parsing it fails. The fix would be patching the monero code that monero-lws uses to build.