inference optimization

_{scroll ↓ to Resources}

Note

• caching ^ef1842
  • prefix caching: Paged Attention, vAttention
  • prompt caching (also partial, in the middle)
    • [2311.04934] Prompt Cache: Modular Attention Reuse for Low-Latency Inference
  • query caching
    • GitHub - zilliztech/GPTCache: Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
• quantization
  • By quantizing the model to float16, int8 or int4 is important to check how much the model degrades across different tasks and languages or modalities.
• speculative decoding
• batching
  • continuous batching
• using different models depending on prompt complexity. Simple question sent to a local model, and complex to ChatGPT
  • RouteLLM is a library developing a special router-model to estimate prompt complexity.
• Mixture-of-Experts
  • Mixture of Experts Explained

Resources

• [2407.12391] LLM Inference Serving: Survey of Recent Advances and Opportunities

---

Links to this File

table file.inlinks, file.outlinks from [[]] and !outgoing([[]])  AND -"Changelog"