How should we handle raw function pointers?

[tgross35]

There are a handful of cases where we have something that is canonically a function pointer, but also accepts integer values (sigaction-related items are one example). Just using usize/size_t works but it loses that type information. We have Option<extern "C" fn(...) -> ...> in some cases, but that makes stumbling into UB too easy if the API expects integer sentinels.

What we really need is raw function pointers, but that would be a long way away. See #t-lang > Function pointers that work like references.

I'm thinking maybe we should add a wrapper type to help out here:

use core::mem::MaybeUninit;

/// Also acts as `Sealed`. Mostly serves as a lint within libc that we don't forget
/// the `unsafe extern "C"`.
/// 
/// # Safety
/// Only implement on `unsafe extern "C"` function pointers. `Option<Self>` must
/// be the same size as `Self`.
unsafe trait FfiSafeFn: Copy {}

unsafe impl<T1, U> FfiSafeFn for unsafe extern "C" fn(T1) -> U {}
unsafe impl<T1, T2, U> FfiSafeFn for unsafe extern "C" fn(T1, T2) -> U {}
unsafe impl<T1, T2, T3, U> FfiSafeFn for unsafe extern "C" fn(T1, T2, T3) -> U {}

/// Has the same repr as `F`
#[allow(private_bounds)]
#[repr(transparent)]
#[derive(Clone, Copy, Debug)]
pub struct RawFnPointer<F: FfiSafeFn>(MaybeUninit<Option<F>>);

#[allow(private_bounds)]
impl<F: FfiSafeFn> RawFnPointer<F> {
    // Internal helper.
    const fn from_int(val: usize) -> Self { /* ... */ }
}

/* Maybe? To make constructing easier.
 *
 * Free functions to avoid breakage if `RawFnPointer` can become an alias
 * in the future. */

/// # Safety
///
/// Must point to a valid function.
#[allow(private_bounds)]
pub unsafe fn raw_to_fn_pointer<F: FfiSafeFn>(f: RawFnPointer<F>) -> F {
    unsafe { f.0.assume_init().unwrap() }
}

#[allow(private_bounds)]
pub fn raw_from_fn_pointer<F: FfiSafeFn>(f: F) -> RawFnPointer<F> {
    RawFnPointer(MaybeUninit::new(Some(f)))
}

With the intent being that users would have to transmute to a function pointer if they know it is valid and want to call it. This isn't any worse than what needs to happen in places where we use usize, though constructing is less convenient.

If we don't provide any public API then we would hopefully have a path to change this to a typedef in a future where raw function pointers exist.

For now this is probably only likely to be used in places where we know we might have sentinel values, though in the ideal world we'd use it everywhere.

Related background at #1637