retry/stamina not working properly on timeout?

[smnorris]

It doesn't look like the request features function is correctly retrying on timeout/504?

Try

• adding a 504 specific test
• only retry on >500 errors as per stamina tutorial https://stamina.hynek.me/en/stable/tutorial.html
• perhaps upgrade stamina?

2025-06-20 15:23:35,852:INFO:bcdata.bc2pg: Writing WHSE_FOREST_VEGETATION.VEG_CONSOLIDATED_CUT_BLOCKS_SP to database as bcdata.veg_consolidated_cut_blocks_sp
2025-06-20 15:24:40,794:INFO:bcdata.wfs: https://openmaps.gov.bc.ca/geo/pub/wfs?service=WFS&version=2.0.0&request=GetFeature&typeName=WHSE_FOREST_VEGETATION.VEG_CONSOLIDATED_CUT_BLOCKS_SP&outputFormat=json&SRSNAME=EPSG%3A3005&sortby=VEG_CONSOLIDATED_CUT_BLOCK_ID&startIndex=450000&count=10000
2025-06-20 15:24:40,794:WARNING:bcdata.wfs: HTTP error: 504
2025-06-20 15:24:40,794:WARNING:bcdata.wfs: Response headers: {'date': 'Fri, 20 Jun 2025 15:24:40 GMT', 'content-type': 'application/json; charset=utf-8', 'transfer-encoding': 'chunked', 'ratelimit-limit': '60000', 'ratelimit-remaining': '59996', 'ratelimit-reset': '1', 'x-ratelimit-remaining-second': '59996', 'x-ratelimit-limit-second': '60000', 'x-kong-upstream-latency': '60000', 'x-kong-proxy-latency': '0'}
2025-06-20 15:24:40,794:WARNING:bcdata.wfs: Response text: {"message":"Upstream service error - The upstream server is timing out","upstream":{"message":"The upstream server is timing out"}}
Traceback (most recent call last):
 File "/opt/venv/bin/bcdata", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/bcdata/cli.py", line 431, in bc2pg
    out_table = bcdata.bc2pg(
                ^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/bcdata/bc2pg.py", line 160, in bc2pg
    df = WFS.request_features(url=url, as_gdf=True, lowercase=True)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/bcdata/wfs.py", line 392, in request_features
    featurecollection = self._request_featurecollection(url)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/stamina/_core.py", line 726, in sync_inner
    for attempt in retry_ctx.with_name(  # noqa: RET503
  File "/opt/venv/lib/python3.12/site-packages/stamina/_core.py", line 508, in __iter__
    for r in _t.Retrying(
  File "/opt/venv/lib/python3.12/site-packages/tenacity/__init__.py", line 443, in __iter__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
   File "/opt/venv/lib/python3.12/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/venv/lib/python3.12/site-packages/stamina/_core.py", line 730, in sync_inner
    return wrapped(*args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/bcdata/wfs.py", line 193, in _request_featurecollection
    r.raise_for_status()
  File "/opt/venv/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://openmaps.gov.bc.ca/geo/pub/wfs?service=WFS&version=2.0.0&request=GetFeature&typeName=WHSE_FOREST_VEGETATION.VEG_CONSOLIDATED_CUT_BLOCKS_SP&outputFormat=json&SRSNAME=EPSG%3A3005&sortby=VEG_CONSOLIDATED_CUT_BLOCK_ID&startIndex=450000&count=10000
Error: Process completed with exit code 1.