Is it possible to use liquid-cache as a transparency cache layer?

[hengfeiyang]

My understand liquid-cache is trying to implement a middle layer to accelerate parquet search.

In most case it will be that:

1. i have a lot of parquet files need to query.
2. i have no cache, let's query parquets and cache the file to liquid format.
3. i have a new query will use 90% cached file & 10% new parquet file.
4. Register liquid-cache as a parquetReader, it can decides to use cache or parquet.

For the end user, just replace the parquetReader(parquetExec).

Currently, i think we need group the files to cache files & no-cache files, then:

• cached files register liquid-cache to read it
• no-cache files register a parquet table to read it

Am i right? i am talking everything based on single node, single datafusion with cache.

[XiangpengHao]

Hi @hengfeiyang, thank you for your questions! Yes, that's a good summary of how LiquidCache works.

I want to add a few details:

1. LiquidCache caches every Parquet file on disk and caches hot portions of the Parquet file in memory.
2. LiquidCache automatically decides which parts should be in memory/disk (or no caching at all, coming soon). In other words, it is fully automatic, so end users don't have to worry about manual partitioning or tuning cache timing, etc.
3. LiquidCache is highly optimized for network cache; typically, multiple DataFusion instances will share the same cache node. LiquidCache minimizes network traffic by sending only filtered/aggregated data over the network; this significantly reduces network cost.

That said, Foyer might be a good choice if you are looking for single-node transparent local caching.

[hengfeiyang]

okay, so you want to build liquid-cache as a cache server, not a cache module embed to others.

we don't need Foyer, we look at liquid-cache because of it has some special format (instead of parquet) for accelerate local query.

[XiangpengHao]

not a cache module embed to others

We planned to do this but haven't got enough bandwidth to support it properly.

we look at liquid-cache because of it has some special format (instead of parquet) for accelerate local query

Glad to hear that! My end goal is to modularize the LiquidCache format for use in the embedded use cases. The implementation is here: https://github.com/XiangpengHao/liquid-cache/tree/main/src/liquid_parquet
And our paper has some more details: https://github.com/XiangpengHao/liquid-cache/blob/main/dev/doc/liquid-cache-vldb.pdf

I'll let you know when it is ready for embedded use. Meanwhile, we welcome contributions!

[hengfeiyang]

Yep. i watched the video also read your paper. i will try to embed in sometime.