Store parquet volume metadata on the parquet tokencounts file

[bmschmidt]

I've been happily using feather the last year as a backend with volume metadata on the files.

But parquet is smaller than feather, and also supports volume and chunk-level metadata.

So rather than storing metadata in a separate .json file, I propose storing the data in the footer of the parquet file, and caching it when needed with pyarrow.parquet.read_metadata.

Reason:

1. Fewer files. The big one for me--taking up 2x as many inodes is a problem in an HPC environment.
2. More portable

Downsides:

1. Requires a pyarrow dependency for parquet, rather than flexibility about using fastparquet instead.
2. Requires either breaking exist pyarrow installations, or some term-unlimited multiple format support.

[bmschmidt]

Goes without saying I'm volunteering to do these tasks. Can push to the Denver repo first. I've decided to move my personal workflow away from feather from parquet because it's more space-efficient, but would need these cghanges to do it.

[organisciak]

I think that's fine. In the short term, I wouldn't want to make the json metadata unsupported - perhaps just discouraged - because it's too soon after the last changes.

[bmschmidt]

Hmm. Does raise an issue though. The current schema makes it sensible to store only, say, character counts as parquet, in which case you'd need to bind the metadata to all three... Rather than deprecate or discourage json metadata, I'll just allow it to be written, but also include the metadata in tokenlist and read from there first.