SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification-Reference-Cited by-同舟云学术

SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification

Published:2024-04-27 Issue: Volume: Page:
ISSN:
Container-title:Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3
language:
Short-container-title:

Author:

Miao Xupeng¹^ORCID,Oliaro Gabriele²^ORCID,Zhang Zhihao²^ORCID,Cheng Xinhao²^ORCID,Wang Zeyu²^ORCID,Zhang Zhengxin³^ORCID,Wong Rae Ying Yee⁴^ORCID,Zhu Alan¹^ORCID,Yang Lijie²^ORCID,Shi Xiaoxiang⁵^ORCID,Shi Chunan⁶^ORCID,Chen Zhuoming¹^ORCID,Arfeen Daiyaan²^ORCID,Abhyankar Reyna⁷^ORCID,Jia Zhihao²^ORCID

Affiliation:

1. Carnegie Mellon University, Pittsburgh, USA

2. Carnegie Mellon University, Pittsburgh, United States of America

3. Tsinghua University, Beijing, China

4. Stanford University, Stanford, USA

5. Shanghai Jiao Tong University, Shanghai, China

6. Peking University, Beijing, China

7. University of California San Diego, San Diego, USA

Funder

NSF

Amazon Research Award

Cisco Research Award

Google Faculty Research Award

Meta Research Award

Oracle Research Award

Qualcomm Innovation Fellowship

Samsung GRO Research Award

Publisher

ACM

Reference58 articles.

1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533--1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.

2. PIQA: Reasoning about Physical Commonsense in Natural Language

3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.

4. Speculative computation, parallelism, and functional programming

5. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.

Cited by 3 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. InSS: An Intelligent Scheduling Orchestrator for Multi-GPU Inference With Spatio-Temporal Sharing;IEEE Transactions on Parallel and Distributed Systems;2024-10

2. Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy;Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining;2024-08-24

3. Arlo: Serving Transformer-based Language Models with Dynamic Input Lengths;Proceedings of the 53rd International Conference on Parallel Processing;2024-08-12