Support option to skip the optimizer for training step

[RissyRan]

Description

This PR introduces a mechanism to skip training steps during severe loss or gradient anomalies (b/489540436). Reference implementation at OLMo-core.

• Add configs in base.yml & types.py
• Implemented skip_step_on_spikes as an optax.GradientTransformationExtraArgs wrapper
• Training loop integration for optimizer updates

Tests

• Add a unit test in tests/unit/optimizers_test.py
• End-to-end training functional comparison: link

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[codecov]

Codecov Report

❌ Patch coverage is 86.79245% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/maxtext/trainers/pre_train/train.py	16.66%	4 Missing and 1 partial ⚠️
src/maxtext/optimizers/optimizers.py	95.74%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

🤖 Hi @RissyRan, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[github-actions]

## 📋 Review Summary

This PR successfully implements an optimizer wrapper to skip training steps during severe loss or gradient anomalies, effectively porting the OLMo-core logic to MaxText using JAX. The core logic elegantly computes rolling statistics and appropriately bypasses the inner optimizer during a spike to prevent momentum poisoning.

🔍 General Feedback

• JAX Idioms: The usage of jax.lax.cond to defer and conditionalize the inner optimizer step is very cleanly implemented.
• Resilience: Added a few critical suggestions to explicitly handle NaN or Inf loss cases. Preventing buffer poisoning and explicitly skipping on non-finite metrics will make this logic foolproof against catastrophic anomalies.
• Kwargs Forwarding: Recommended using .pop() on **extra_args to ensure consumed arguments like loss aren't passed downstream, guaranteeing better compatibility with inner optimizers.

[gagika]

thanks, one minor comment

[aireenmei]

Thanks for the work!

[aireenmei]

Will user see is_skipped in logging? It would be good if the logging shows "skipped" when the step is skipped, but no need to show is_skipped for every step since the majority are non-skipped. Alternatively we can show a separate warning message when a step is skipped.