Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

104 points

by NicoConstant

5 hours ago

53 comments