Review all panics and decide if they need to be errors

[tigrannajaryan]

See for example #370

All panics that can happen due to invalid input received externally (e.g. in decoders or in converters from other formats) must be errors instead.

Fixes must be done carefully to avoid performance regressions since these are often on hot paths. Always fully benchmark affected code.