CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving-Reference-Cited by-同舟云学术

CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

Published:2024-08-04 Issue: Volume:4 Page:38-56
ISSN:
Container-title:Proceedings of the ACM SIGCOMM 2024 Conference
language:
Short-container-title:

Author:

Liu Yuhan¹^ORCID,Li Hanchen¹^ORCID,Cheng Yihua¹^ORCID,Ray Siddhant²^ORCID,Huang Yuyang²^ORCID,Zhang Qizheng³^ORCID,Du Kuntai¹^ORCID,Yao Jiayi⁴^ORCID,Lu Shan⁵^ORCID,Ananthanarayanan Ganesh⁶^ORCID,Maire Michael⁷^ORCID,Hoffmann Henry²^ORCID,Holtzman Ari⁸^ORCID,Jiang Junchen²^ORCID

Affiliation:

1. University of Chicago, Chicago, USA

2. University of Chicago, Chicago, United States of America

3. Stanford University, Palo Alto, United States of America

4. The Chinese University of Hong Kong, Shenzhen, Shenzhen, China

5. Microsoft Research, Seattle, United States of America

6. Microsoft, Seattle, United States of America

7. The University of Chicago, Chicago, United States of America

8. Meta, University of Chicago, Chicago, United States of America

Funder

NSF

UChicago CERES Center

MARIN and Stuart Rice Research Award

Chameleon Projects

Publisher

ACM

Link

https://dl.acm.org/doi/pdf/10.1145/3651890.3672274

Reference157 articles.

1. 2021. How latency affects user engagement. https://pusher.com/blog/how-latency-affects-user-engagement/. (2021). (Accessed on 09/21/2023).

2. 2023. Best Practices for Deploying Large Language Models (LLMs) in Production. https://medium.com/@_aigeek/best-practices-for-deploying-large-language-models-llms-in-production-fdc5bf240d6a. (2023). (Accessed on 09/21/2023).

3. 2023. Building RAG-based LLM Applications for Production. https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1. (2023). Accessed: 2024-01-25.

4. 2024. Amazon Bedrock Pricing. https://aws.amazon.com/bedrock/pricing/. (2024). Accessed: 2024-01-25.

5. 2024. Anyscale Pricing. https://docs.endpoints.anyscale.com/pricing. (2024). Accessed: 2024-01-25.