Cutting LLM Batch Inference Time in Half: Dynamic Prefix Bucketing at Scale

1 comments

• sammysidhu 3 hours ago
  Part of the Daft team here! Happy to answer any questions