feat: add retry utility with exponential backoff

[bnusunny]

Adds a reusable retry decorator with exponential backoff and jitter for handling transient failures.

[cortex-assistant]

PR Review: feat: add retry utility with exponential backoff

Thanks for this refactor! The new retry_with_backoff implementation is a clear improvement over the original retry — adding jitter, max delay capping, proper logging, and a cleaner API. However, there are several issues that need to be addressed before this can be merged.

---

🔴 Breaking Change — Existing callers will break

This PR removes the retry function entirely and replaces it with retry_with_backoff, which has a completely different signature. The existing codebase has callers that import and use the old API, e.g.:

# samcli/local/docker/container.py
from samcli.lib.utils.retry import retry

The old retry() decorator takes exc as a positional arg, plus exc_raise and exc_raise_msg parameters — none of which exist in the new API. This PR will break all existing callers at runtime.

Recommendation: Either:

1. Keep the old retry() function (deprecated) alongside the new retry_with_backoff() for backward compatibility, or
2. Update all call sites in the same PR to use the new API.

---

🔴 No tests included

This PR modifies core retry logic but includes no new or updated tests. The existing tests for the old retry function will also be broken. Please add unit tests covering:

• Successful call on first attempt (no retry)
• Successful call after N transient failures
• All retries exhausted → exception re-raised
• max_delay cap is respected
• Only specified exceptions are caught (other exceptions propagate immediately)
• Jitter produces non-deterministic delays (mock random.uniform)

---

🟡 Behavioral difference in exception re-raising

The old function raised a custom exception (exc_raise(exc_raise_msg)) after retries were exhausted. The new function re-raises the last caught exception. This is arguably better behavior (preserves the original traceback), but it's a semantic change that callers may depend on. Please document this intentional change in the PR description.

---

🟡 Minor code-level comments below.

Overall the implementation logic is sound — the exponential backoff + jitter + max_delay pattern is well-established. Just needs the surrounding issues addressed.

[cortex-assistant]

🟡 exceptions parameter type mismatch with except clause

The default exceptions=(Exception,) is a tuple, which works correctly with except. However, if a caller passes a single exception class (not wrapped in a tuple), e.g. retry_with_backoff(exceptions=ValueError), this will also work because except accepts both. That said, it would be good to add a type hint to make the expected input clear:

from typing import Type, Tuple, Union

def retry_with_backoff(
    max_retries: int = 3,
    base_delay: float = 1.0,
    max_delay: float = 30.0,
    exceptions: Union[Type[BaseException], Tuple[Type[BaseException], ...]] = (Exception,),
):

[cortex-assistant]

🟡 Jitter is additive, not multiplicative — The current formula:

delay = min(base_delay * (2 ** attempt) + random.uniform(0, 1), max_delay)

Adds a fixed [0, 1) second jitter regardless of the backoff magnitude. For very small base_delay values (e.g., 0.01s), the jitter dominates. For very large delays (e.g., 16s), the jitter is negligible.

Consider using proportional jitter instead:

delay = min(base_delay * (2 ** attempt), max_delay)
delay = delay * random.uniform(0.5, 1.0)  # "decorrelated" or "equal" jitter

Or at minimum, document the current jitter strategy so callers understand the behavior.

[cortex-assistant]

🟢 Nice addition of logging. This is a great improvement over the old silent retry. The log format is clear and actionable.

Minor nit: consider using LOG.info instead of LOG.warning for expected transient retries — warning level may be too noisy in production if retries are a normal/expected path. Alternatively, make the log level configurable.

[cortex-assistant]

🔴 raise last_exception loses the original traceback in Python 3.

When you do raise last_exception, Python creates a new traceback from this point. To preserve the original traceback, use:

raise last_exception from last_exception

Or even better, use bare raise inside the except block:

except exceptions as e:
    last_exception = e
    if attempt == max_retries:
        raise  # preserves original traceback

This restructuring would preserve the full stack trace for debugging.