Setting GPU Stateful Set

Stateful Set will allow you to place the LLM model over multiple GPUs in order for prompts to utilize multiple GPUs.

🚧

Caution

When editing the file, make sure only to add what is needed. Deleting or adding irrelevant fields can disrupt Layars ability to operate.

1. SSH into your environment.
2. Run sudo -i
3. Run vim /data/layar/layar.config
4. Press i to start editing.
5. Look for the field VLLM_GPU_COUNT
6. Add the number for desired number of GPUs to be used. IE VLLM_GPU_COUNT = 2
7. Press ESC to exit input mode.
8. Run :wq to commit the changes.