in local dev setup, webpa can't always find a connected device #96
Description
The device is successfully redirected to a talaria and connects. However, when you query it from webpa (tr1d1um), it returns a 404. If you connect directly to talaria instead of redirecting with petasos, webpa can find the connected device.
This situation occurs when you run parodus in a separate docker container but within the same docker-compose network (should be no different that running it in docker-compose). Here is how to start parodus using a locally built parodus docker image:
docker run --network docker-compose_xmidt parodusfull --hw-model=TGXXX --hw-serial-number=E8GBUEXXXXXXXXX --hw-manufacturer=ARRIS --hw-mac=112233445577 --hw-last-reboot-reason=unknown --fw-name=TG1682_DEV_master_20170512115046sdy --boot-time=1494590301 --webpa-ping-timeout=180 --webpa-interface-used=eth0 --webpa-url=http://petasos:6400 --webpa-backoff-max=9 --parodus-local-url=tcp://127.0.0.1:6666 --partner-id=comcast --token-server-url=http://themis:6501/issue
parodus logs:
[1742487833][PARODUS][Info]: Received temporary redirection response message Redirect:http://f991b1598295:6200/api/v2/device
[1742487833][PARODUS][Info]: full url: http://f991b1598295:6200/api/v2/device
[1742487833][PARODUS][Info]: server address copied from url
[1742487833][PARODUS][Info]: server f991b1598295, port 6200, http_match 1
[1742487833][PARODUS][Info]: nopoll_ctx.c:338 Unregistered connection id 2
[1742487833][PARODUS][Info]: cloud_status set as offline after connection close
[1742487833][PARODUS][Error]: Error removing the file /tmp/webpanotifyready
[1742487833][PARODUS][Info]: nopoll_conn.c:331 IPv4 address of f991b1598295 is 172.19.0.9
[1742487833][PARODUS][Info]: nopoll_conn.c:377 Create socket with non blocking-mode
[1742487833][PARODUS][Info]: nopoll_conn.c:263 Result of wait after connect EINPROGRESS = 0
[1742487833][PARODUS][Info]: nopoll_conn.c:5249 *****End nopoll_conn_wait_for_status_until_connection_ready ****
[1742487833][PARODUS][Info]: Connected to server
[1742487833][PARODUS][Info]: connect_time-diff-boot_time=88737
[1742487833][PARODUS][Info]: metadata encoding is successful with size 365
[1742487833][PARODUS][Info]: cloud_status
Talaria device is connected to is in DC0:
Talaria shows that device as connected when you query it:
mpicci200@RESML-1722218 anemoi.prod-na % curl -i -H "Authorization: Basic dXNlcjpwYXNz" "http://localhost:6210/api/v2/devices"
HTTP/1.1 200 OK
Content-Type: application/json
X-Talaria-Build: unknown
X-Talaria-Flavor: mint
X-Talaria-Region: east
X-Talaria-Server: f991b1598295
X-Talaria-Start-Time: 19 Mar 25 16:54 UTC
Date: Thu, 20 Mar 2025 17:15:35 GMT
Content-Length: 480
{"devices":[{"id": "mac:112233445566", "pending": 0, "statistics": {"bytesSent": 0, "messagesSent": 0, "bytesReceived": 0, "messagesReceived": 0, "duplications": 0, "connectedAt": "2025-03-20T14:43:50.251420354Z", "upTime": "2h31m45.544740118s"}},{"id": "mac:112233445577", "pending": 0, "statistics": {"bytesSent": 0, "messagesSent": 0, "bytesReceived": 0, "messagesReceived": 0, "duplications": 0, "connectedAt": "2025-03-20T16:23:53.31567913Z", "upTime": "51m42.480490493s"}}]}%
Scytale Logs (all fanouts fail):
{"key":"debug","ts":"2025-03-20T16:24:32Z","message":"authentication accepted by enforcer","requestHeaders":{"Accept-Encoding":["gzip"],"Authorization-Type":["Basic"],"Traceparent":["00-eb3eacb7d54a390c01856673ae9e6092-b8c9818738fefe1f-01"],"User-Agent":["Go-http-client/1.1"]},"requestURL":"/api/v3/device/mac:112233445577/stat","method":"GET","span-id":"b8c9818738fefe1f","trace-id":"eb3eacb7d54a390c01856673ae9e6092","ts":"2025-03-20T16:24:32Z"}
{"key":"debug","ts":"2025-03-20T16:24:32Z","message":"fanout request complete","requestHeaders":{"Accept-Encoding":["gzip"],"Authorization-Type":["Basic"],"Traceparent":["00-eb3eacb7d54a390c01856673ae9e6092-b8c9818738fefe1f-01"],"User-Agent":["Go-http-client/1.1"]},"requestURL":"/api/v3/device/mac:112233445577/stat","method":"GET","span-id":"b8c9818738fefe1f","trace-id":"eb3eacb7d54a390c01856673ae9e6092","statusCode":404,"url":"http://2163403a20ac:6200/api/v3/device/mac:112233445577/stat"}
{"key":"debug","ts":"2025-03-20T16:24:32Z","message":"fanout request complete","requestHeaders":{"Accept-Encoding":["gzip"],"Authorization-Type":["Basic"],"Traceparent":["00-eb3eacb7d54a390c01856673ae9e6092-b8c9818738fefe1f-01"],"User-Agent":["Go-http-client/1.1"]},"requestURL":"/api/v3/device/mac:112233445577/stat","method":"GET","span-id":"b8c9818738fefe1f","trace-id":"eb3eacb7d54a390c01856673ae9e6092","statusCode":404,"url":"http://314973c937d3:6200/api/v3/device/mac:112233445577/stat"}
{"key":"error","ts":"2025-03-20T16:24:32Z","message":"all fanout requests failed","requestHeaders":{"Accept-Encoding":["gzip"],"Authorization-Type":["Basic"],"Traceparent":["00-eb3eacb7d54a390c01856673ae9e6092-b8c9818738fefe1f-01"],"User-Agent":["Go-http-client/1.1"]},"requestURL":"/api/v3/device/mac:112233445577/stat","method":"GET","span-id":"b8c9818738fefe1f","trace-id":"eb3eacb7d54a390c01856673ae9e6092","statusCode":404,"url":"/api/v3/device/mac:112233445577/stat"}
{"key":"debug","ts":"2025-03-20T16:24:32Z","message":"wrote fanout response","requestHeaders":{"Accept-Encoding":["gzip"],"Authorization-Type":["Basic"],"Traceparent":["00-eb3eacb7d54a390c01856673ae9e6092-b8c9818738fefe1f-01"],"User-Agent":["Go-http-client/1.1"]},"requestURL":"/api/v3/device/mac:112233445577/stat","method":"GET","span-id":"b8c9818738fefe1f","trace-id":"eb3eacb7d54a390c01856673ae9e6092","statusCode":404}
404 from tr1d1um:
mpicci200@RESML-1722218 anemoi.prod-na % curl -i -H "Authorization: Basic dXNlcjpwYXNz" "http://localhost:6100/api/v2/device/mac:112233445577/stat"
HTTP/1.1 404 Not Found
X-Midt-Server: tr1d1um
X-Midt-Version: development
X-Scytale-Build: unkown
X-Scytale-Flavor: mint
X-Scytale-Region: east
X-Scytale-Server: 3de95bb419fe
X-Scytale-Start-Time: 19 Mar 25 16:54 UTC
X-Talaria-Build: unknown
X-Talaria-Flavor: mint
X-Talaria-Region: east
X-Talaria-Server: 314973c937d3
X-Talaria-Start-Time: 19 Mar 25 16:54 UTC
X-Webpa-Transaction-Id: FIxC2i6cV-5ZkNAZ4pnjdg
X-Xmidt-Span: "http://314973c937d3:6200/api/v3/device/mac:112233445577/stat","2025-03-20T17:18:31Z","1.446206ms"
X-Xmidt-Span: "http://2163403a20ac:6200/api/v3/device/mac:112233445577/stat","2025-03-20T17:18:31Z","2.086113ms"
Date: Thu, 20 Mar 2025 17:18:31 GMT
Content-Length: 0