GPU Configuration Considerations

Layar supports NVIDIA MiG (Multi-Instance GPU User). This allows you to use a single GPU to simultaneously run both the vectorstore and the LLM. This guide will go over how to configure MiG and which helm chart variables need to updated.

Prerequisites

In order to MiG the GPU you need to ensure no GPU process are running. You can run nvidia-smi to see if anything is running. If you need to kill the PID of the process, you can run kill -9 PID.

🚧

Double-Check PID

Before running the kill script make sure to ensure the PID that needs to kill.

Enable MiG

You to run nvidia-smi to acquire the ID of the GPU. If you have multiple GPUs you ensure you are choosing the correct ID. Once you have the ID you can insert it into the following script.

nvidia-smi -i GPUID -mig 1

Determining MiG Profile

In order to separate the instances on the GPU, you will need to provide GPU profiles. You can view these profiles by running the following.

nvidia-smi mig -lgip

Here's an example of what you will see.

📘

Considerations

It's important to pay attention to the Instances Free/Total column. This indicates how many instances of the desired profile can be active on the GPU.

Creating GPU Instances

For the sake of this guide, we will be taking an A100 and splitting it into a 20gb memory + 3 cores and a 20gb memory + 4 cores GPU. In order to do this, we will need to use 2 profiles from the above list, 9 and 5 respectively.

Run the following to create the instances.

nvidia-smi mig -i GPU ID -cgi 9,5 -C

After running the above, restart the system. Once back up, you can use nvidia-smi mig -lgi to see the MiG instances.

Updating the Layar Config File

In order to make use of these instances, the Layar Config file needs to be updated. Lets give the 4 core 20gb to the LLM and the 3 core 20gb to the Vectorstore.

VLLM_GPU_COUNT: 1
TGI_NUM_GPU: 1
VECTORSTORE_TRANSFORMERS_NUM_GPU: 1
VECTORSTORE_RERANKER_NUM_GPU: 0
TGI_GPU_INSTANCE: "nvidia.com/mig-4g.20gb"
VECTORSTORE_TRANSFORMERS_GPU_INSTANCE: "nvidia.com/mig-3g.20gb"
VECTORSTORE_RERANKER_GPU_INSTANCE: "nvidia.com/gpu"

You can edit the file by doing the following:

1. Run vim /data/layar/layar.config
2. Press i to start editing.
3. Add the values to the bottom of the config file OR update existing configs.

Removing the MiG

To remove the instances, you will need to ensure no process are running on the GPU and run the following.

nvidia-smi mig -i 0 -dci
nvidia-smi mig -i 0 -dgi
nvidia-smi -i 0 -mig 0