[BUG] Compatibility: Support execution on FIPS-enabled Linux kernels (RHEL 9 / Rocky 9)

[oberstet]

Porting OSS to the "Enterprise/Government" RHEL world, a potential issue is that Python's hashlib binds to the system OpenSSL. On RHEL 9 with FIPS mode enabled, OpenSSL will ruthlessly abort any call to a non-validated algorithm (like MD5) unless explicitly told it is not being used for security purposes.

Context

Deploying this project in regulated environments (US Defense / FedRAMP) requires the underlying OS to run in FIPS 140-2/3 mode.
Specifically, when running on RHEL 9 or Rocky Linux 9 (and potentially Amazon Linux 2023) where the kernel is booted with fips=1, the system OpenSSL library disables non-approved cryptographic algorithms.

The Problem

Currently, parts of the codebase might use hashlib.md5() (and possibly other non-FIPS algorithms). In a strict FIPS environment, invoking hashlib.md5() without specific flags causes the Python runtime to crash with a ValueError.

Example Traceback:

ValueError: [digital envelope routines] unsupported

Or in older OpenSSL versions:

ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips

While MD5 is broken for cryptographic security (signatures, collision resistance), it is frequently used for non-security purposes, such as:

• Generating deterministic UUIDs
• Cache keys
• File fingerprinting / ETags
• Internal ID generation

Proposed Solution

Python 3.9+ introduced the usedforsecurity flag in hashlib. We need to audit the codebase and update all instances of non-cryptographic hashing to explicitly flag them.

Before:

m = hashlib.md5(data)
# OR
m = hashlib.new('md5', data)

After:

# explicitly signal to OpenSSL that this is not for security
m = hashlib.md5(data, usedforsecurity=False) 
# OR
m = hashlib.new('md5', data, usedforsecurity=False)

Note that in the context of US Government compliance, the flag usedforsecurity=False does not mean "This is insecure." It effectively translates to:

    bypass_fips_allowlist=True

or

    i_accept_liability_for_using_math_nist_does_not_understand=True

and thus is (in my eyes) badly named (in Python).

Certain cryptographically secure algorithms are simply unknown or too new for NIST (again, professional opinion):

• BLAKE2 / RIPEMD-160 are not broken. They are just "foreign" or "too new" for NIST. The fact that RHEL crashes on them is a policy enforcement implementation detail, not a security vulnerability in our code.

• NIST FIPS standards lag behind modern cryptography by 5-10 years. (e.g., They are only just now finalizing PQC, while the industry has moved on). The fact that they force a crash on BLAKE2 (which is arguably safer than the approved SHA-1) is absurd, but it is the reality of the RHEL kernel.

Affected Environments

• Python: CPython 3.11+, PyPy 3.11+
• OS: RHEL 9, Rocky Linux 9, AlmaLinux 9 (UBI 9 images)
• Architecture: x86_64, aarch64

Action Items

• Audit hashlib.md5 usage: Grep the codebase for all instances of md5.
• Audit hashlib.sha1 usage: SHA-1 is restricted in some newer FIPS profiles for signing, though usually allowed for HMAC. Check if usedforsecurity=False is applicable here as well.
• Audit other algorithms: Ensure no usage of ARC4, Blowfish, or RIPEMD, which may be completely removed in FIPS builds regardless of flags.
• Apply Fix: Update calls to use usedforsecurity=False where appropriate (targeting Python 3.9+ syntax).
• Verify PyPy Compatibility: Ensure PyPy 3.11+ respects this flag or fails gracefully.
• CI/Test: (Optional) Add a test case that mocks a FIPS environment or verifies the flag is passed.

Reference

• Python 3.9 hashlib documentation
• Red Hat RHEL 9 Security Hardening

A quick note on PyPy

PyPy 7.3.11+ (implementing Python 3.9) does support the usedforsecurity argument in the signature to maintain API compatibility, but historically, PyPy's _hashlib implementation was sometimes inconsistent in how it passed that flag down to the C-level OpenSSL EVP_MD_CTX_set_flags.

If running PyPy on RHEL 9, we definitely want to verify this manually. If PyPy ignores the flag and passes the call to a FIPS-locked OpenSSL, it will still crash.