GPU Configuration Considerations
Layar supports NVIDIA MiG (Multi-Instance GPU User). This allows you to use a single GPU to simultaneously run both the vectorstore and the LLM. This guide will go over how to configure MiG and which helm chart variables need to updated.
Prerequisites
In order to MiG the GPU you need to ensure no GPU process are running. You can run nvidia-smi
to see if anything is running. If you need to kill the PID of the process, you can run kill -9 PID
.
Double-Check PID
Before running the kill script make sure to ensure the PID that needs to kill.
Enable MiG
You to run nvidia-smi
to acquire the ID of the GPU. If you have multiple GPUs you ensure you are choosing the correct ID. Once you have the ID you can insert it into the following script.
nvidia-smi -i GPUID -mig 1
Determining MiG Profile
In order to separate the instances on the GPU, you will need to provide GPU profiles. You can view these profiles by running the following.
nvidia-smi mig -lgip
Here's an example of what you will see.
Considerations
It's important to pay attention to the Instances Free/Total column. This indicates how many instances of the desired profile can be active on the GPU.
Creating GPU Instances
For the sake of this guide, we will be taking an A100 and splitting it into a 20gb memory + 3 cores and a 20gb memory + 4 cores GPU. In order to do this, we will need to use 2 profiles from the above list, 9 and 5 respectively.
Run the following to create the instances.
nvidia-smi mig -i GPU ID -cgi 9,5 -C
After running the above, restart the system. Once back up, you can use nvidia-smi mig -lgi
to see the MiG instances.
Updating the Layar Config File
In order to make use of these instances, the Layar Config file needs to be updated. Lets give the 4 core 20gb to the LLM and the 3 core 20gb to the Vectorstore.
VLLM_GPU_COUNT: 1
TGI_NUM_GPU: 1
VECTORSTORE_TRANSFORMERS_NUM_GPU: 1
VECTORSTORE_RERANKER_NUM_GPU: 0
TGI_GPU_INSTANCE: "nvidia.com/mig-4g.20gb"
VECTORSTORE_TRANSFORMERS_GPU_INSTANCE: "nvidia.com/mig-3g.20gb"
VECTORSTORE_RERANKER_GPU_INSTANCE: "nvidia.com/gpu"
You can edit the file by doing the following:
- Run
vim /data/layar/layar.config
- Press
i
to start editing. - Add the values to the bottom of the config file OR update existing configs.
Removing the MiG
To remove the instances, you will need to ensure no process are running on the GPU and run the following.
nvidia-smi mig -i 0 -dci
nvidia-smi mig -i 0 -dgi
nvidia-smi -i 0 -mig 0
Updated 3 months ago
The next article will go over further considerations if using lean GPUs.