Spaces:
Running
on
CPU Upgrade
Running
on
CPU Upgrade
Energy Calculation for more models
#6
by
Sarvesh19
- opened
It would be great if we can see the calculation for more models, particularly for those available in the LMSYS dataset
Sure thinking of a specific model deployed on a specific hardware ? We can only have measurements for open weight models, but we can make estimates for the close ones.
For the Qwen 2.5 7B instruct, the calculation is done by measuring the energy used on the gpu live for each request (using nvml for an Nvidia L4). For the other models it is an estimation - based on inference time and mean power usage for the task.