unsafe refers to memory safety, not generally "dangerous" APIs. I don't think it should be use for this kind of use cases. For example, neither the secrecy or untrusted use unsafe this way.
To warn against exporting the secret key, I'd recommend removing unsafe and renaming it to subtle_to_bytes or something similar.