Consistent debug logging levels and messages

[mbutrovich]

What is the problem the feature request solves?

To date Comet has been inconsistent with its debug logging. There are a handful of logWarnings for exceptional cases and minimal logInfo and logDebug. I know personally I try to use printing to help with development, and often strip it all back out before submitting a PR. Maybe a subset of that should be kept.

As users deploy Comet in production, we should make it easier to debug Comet behavior in Spark clusters.

Describe the potential solution

This is mostly for discussion right now, but I think we should try to make Comet's logging consistent with Spark and Iceberg. We don't want it to be too verbose, but some info about plan conversation, plan distribution/serialization, and execution would be helpful.

I'm not sure if we should try to break this into smaller tasks like checking individual classes or modules to make it more manageable, or if the first task is to try to codify guidance into the developer guide.

Additional context

No response

[andygrove]

+1 for writing guidance in the contributor guide as a starting point