Spaces:

yusufs
/

vllm-inference

Paused

yusufs commited on Jan 28

Commit

8c5a84b

1 Parent(s): 5bd7bc7

feat(runner.sh): --enable-chunked-prefill and --enable-prefix-caching for faster generate

Files changed (1) hide show

runner.sh CHANGED Viewed

@@ -52,4 +52,6 @@ python -u /app/openai_compatible_api_server.py \
     --max-model-len 32768 \
     --dtype float16 \
     --enforce-eager \
-    --gpu-memory-utilization 0.9

     --max-model-len 32768 \
     --dtype float16 \
     --enforce-eager \
+    --gpu-memory-utilization 0.9 \
+    --enable-chunked-prefill \
+    --enable-prefix-caching