LMCache: OCI Cache integration with OCI Data Science for KV Cache Offloading

Oracle Cloud Infrastructure (OCI) Cache is a managed service that enables you to build and manage Cache clusters, which are memory-based storage solutions for applications. You can achieve submillisecond latency with OCI Cache, a fully managed Valkey and Redis service engineered for exceptional scalability and consistent high availability. For complete information, you can refer to the public documentation. For multiple data science use-cases like token retrieval, KV caching, and session management, a cloud native solution is needed for smooth integration, especially with LLM deployment.

In this blog, we will see a sample Data Science LLM deployment on OCI Data Science with OCI Cache integration. We will see how KV cache can be offloaded in a vLLM-hosted LLM deployment. KV cache offloading enables the movement of large caches from GPU memory to CPU, disk, or a centralised cache to improve the KV cache hits. vLLM stack relies on LMCache for the KV cache offloading function. The centralised cache approach enables the delivery of KV caches across LLMs, which is crucial in a distributed deployment environment.

Introduction

Let’s get familiar with the tech stack first.

OCI Data Science — OCI Data Science is a fully managed platform for teams of data scientists to build, train, deploy, and manage machine learning (ML) models using Python and open source tools.

OCI Cache — OCI Cache is a managed service that enables you to build and manage Redis Cache clusters on the cloud.

OCI Registry — It is an Oracle-managed container registry that makes it easy for you as a developer to store, share, and manage container images (such as Docker images).

vLLM — vLLM is a fast and easy-to-use library for LLM inference and serving. It has become a go-to framework for deploying all new LLMs being released, due to its efficient management of attention key and value memory with PagedAttention.

KV Cache — A KV cache (Key-Value cache) is a technique used in transformer models, especially LLMs, to speed up inference by storing and reusing intermediate results. Specifically, it stores the “key” and “value” matrices from the attention mechanism in previous inference steps, allowing the model to avoid redundant computations when processing subsequent tokens.

LMCache — It is a fast and scalable caching of long-context KV cache. LMCache helps reduce inference costs and ensures SLOs for both latency and throughput at scale.

High-level design

While vLLM alone offers KV cache offloading on GPU or disk, it does not provide a distributed cache management feature, across nodes or cards, easily. For distributed scaling of vLLM deployments, we need a mechanism that allows tokens calculated on one vLLM instance to be available on another vLLM instance, in order to avoid recomputation and save GPU cycles. LMCache Redis implementation serves as the backend to store the KV cache and reuse tokens on all further prompts, on all other vLLM instances. We will be utilising the OCI Cache Redis cluster as the LMCache endpoint for the KV cache. LLM deployed on OCI Data Science will be connecting via private connectivity to OCI Cache cluster, both utilising the same subnet configuration.

LMCache integration with OCI Cache

LMCache is already fully supported with vLLM 0.8.5 V1 engine. Let’s look at the steps for a successful integration on OCI Data Science.

1. Bring the LMCache+vLLM image to OCI Registry (OCIR). Refer to public docs for details. docker pull lmcache/vllm-openai
2. Now we need to bring this image to OCIR. You can use our samples repository for details on how to push to OCIR.
3. Create an OCI Cache cluster with the required networking. For private networking, set up a VCN and subnet with service gateway access. This subnet will later be required for creating a vLLM deployment. Create the cluster using documentation and fetch the Redis cluster endpoint.
4. Bring the LLM of your choice on OCIR using any of the documented methods.
5. We are now ready with the setup requirements. We have the model, network, image, and caching platform. Next comes creating the deployment on the OCI Data Science Model deployment using the BYOC option. Using Python SDK, we will use the below payload to create a vLLM deployment.

data_science_client.create_model_deployment(create_model_deployment_details = { “displayName” : “vLLM-lmcache”, “description” : “Testing LMCache”, “projected” : “<PROJECT_OCID>”, “compartmentId” : “<COMPARTMENT_OCID>”, “modelDeploymentConfigurationDetails” : { “deploymentType” : “SINGLE_MODEL”, “modelConfigurationDetails” : { “modelId” : “<MODEL_OCID>”, “instanceConfiguration”: { “instanceShapeName”: “<SHAPE_NAME>”, “subnetId” : <OCI_CACHE_SUBNET_OCID> }, “scalingPolicy” : { “policyType” : “FIXED_SIZE”, “instanceCount” : 2 }, “bandwidthMbps” : 10, “maximumBandwidthMbps” : 10 }, “environmentConfigurationDetails” : { “environmentConfigurationType” : “OCIR_CONTAINER”, “image” : “<IMAGE_NAME_FROM_STEP_1>”, “cmd” : None, “entrypoint” : None, “serverPort” : 8080, “healthCheckPort” : 8080, “environmentVariables”: { “MODEL_DEPLOY_PREDICT_ENDPOINT”: “/v1/chat/completions”, “PORT”: “8080”, “MODEL”: “/opt/ds/model/deployed_model”, “SHM_SIZE”: “10g”, “LMCACHE_LOCAL_CPU”:”True”, “LMCACHE_USE_EXPERIMENTAL”:”True”, “LMCACHE_CHUNK_SIZE”: “256”, “LMCACHE_REMOTE_URL”: “rediss://<REDIS_ENDPOINT>”, “LMCACHE_REMOTE_SERDE”: “naive”, “PARAMS”: “ — enforce-eager — tensor-parallel-size=<NUMBER_OF_GPU_CARDS> — kv-transfer-config=\’{\”kv_connector\”:\”LMCacheConnectorV1\”,\”kv_role\”:\”kv_both\”}\’” } } }, “categoryLogDetails” : {<LOGS_RECOMMENDED>}, “deploymentMode” : “HTTPS_ONLY”, “freeformTags” : { }, “definedTags” : { } }) 6. Once model deployment is ACTIVE, we can run the inference request again using the Python SDK.

`predict_url = ‘<MODEL_DEPLOYMENT_URL>’

predict_body = json.dumps({“model”: “/opt/ds/model/deployed_model”, “messages”: [ { “role”:”user”, “content”:”Hello! How are you?” }, { “role”:”assistant”, “content”:”Hi! I am quite well, how can I help you today?” }, { “role”:”user”, “content”:”Can you write me a song?” } ], “top_p”: 1, “n”: 1, “max_tokens”: 200, “stream”: false, “frequency_penalty”: 1.0, “stop”: [“hello”] }) predict_headers = { ‘content-type’: ‘application/json’ } response = requests.request(“POST”, predict_url, headers=predict_headers, data=predict_body, auth=get_auth(), verify=False) print(response.content) 7. Make 2–3 requests and observe the predict logs. The first request would land on 1 of the hosts and create tokens that will be preserved in OCI Cache. You can connect to the cluster using [redis-cli](https://redis.io/docs/latest/develop/tools/cli/) and input the query **KEYS ***. It will show a couple of entries in this format. “vllm@/models@1@0@084ddc3f06864e64c587b3f36210ab8f7f62069abd3acd1519c68872ae413271kv_bytes” “vllm@/models@1@0@c9729f836ee3dda68adcadefdd0b0f2134c3b9c3786429a4680006691d65c216metadata” “vllm@/models@1@0@2894628fae9ea8b3bf55cb6be121bc55d591e21ddd60b2c55d25fb09c0010b64metadata” “vllm@/models@1@0@28ca7d623982ea0e087a7edfe530ab6808792f991ed45909ff556fb110f6d184kv_bytes” 8. Once the request lands on the second host, you will observe the LMcache hit rate to be higher than 0%(based on prompt), and many tokens will be loaded from cache, rather than re-calculation. vLLM logs show in the below format - LMCache INFO: Reqid: chatcmpl-17522b7acfa845dba0e3cc12fdc247c1, Total tokens 70, LMCache hit tokens: 70, need to load: 21 (vllm_v1_adapter.py:836:lmcache.integration.vllm.vllm_v1_adapter). “POST /v1/chat/completions HTTP/1.1” 200 OK`

Conclusion

In this blog, we saw a successful integration of vLLM, LMCache, OCI Cache, & OCI Data Science. We can observe how GPU cycles can be preserved by hosting multiple vLLM deployments in a centralised, distributed manner. This optimises the need for repetitive token generation for similar queries in a distributed environment. Beyond pure performance gains, LMCache plays a significant role in cost optimization on OCI. By reducing the computational load on GPUs during repeated prompts, teams can lower their cloud costs substantially. This aligns with OCI’s value proposition of providing competitive pricing while delivering enterprise-grade performance, offering businesses an economical path to scale LLM-powered applications.

References

Model deployment samples

LMCache integration with vLLM

LMCache Redis Integration

More blogs from the authors