prompt_cache support in AWS Bedrock LLM

[iRajul]

AWS Bedrock support prompt_cache as seen in https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html and https://github.com/aws-samples/amazon-bedrock-samples/tree/main/introduction-to-bedrock/prompt-caching

I tried to support it locally but I found that livekit uses aioboto3 which does not support prompt_cache
but this support is there in boto3 since 1.37.24 , so unable to use it.
What are the options available to support this?

[DK013]

any news on this?

[davidzhao]

please consider opening a PR if this is important to you

[Thanush101]

any update?

[iRajul]

last time I check i think it is supported.

[Thanush101]

Thanks for checking. Could you please share a link to the documentation or any examples? I haven’t been able to find any prompt caching documentation for LiveKit with AWS Bedrock. I’d really appreciate it.Also, since LiveKit is currently using aioboto3, and aioboto3 does not support AWS Bedrock prompt caching, I don’t see how to implement caching through LiveKit yet. I’d really appreciate any examples or links that show how to do it.

[Thanush101]

Update: I figured it out. The current plugin version actually does support caching. You can enable it by simply adding cache_system=True to the LLM initialization:

llm = aws.LLM(
model="......",
region="...",
cache_system=True
)
I verified the logs, and cacheReadInputTokens is showing hits.