Setting Generative System Model

Models & Requirements

Default Model: Mistral 7B V0.1

🚧

Llama 3.1 VRAM Limitations

If you are on Layar 1.9, you will need to have GPUs with 80gb of VRAM. If there are further questions about this, please e-mail [email protected]

📘

GPU Configuration Considerations

If you are interested in putting the models on smaller GPU partitions, please review GPU Considerations

Setting Your System to a New Model

To change the model from the default, you will need to ssh into the instance in question.
Once you are logged in, follow these steps:

1. Become root with sudo su -
2. Once you have root access, you will be able to edit the Layar configuration file.
   Do this by running vi /data/layar/layar.config
3. Enter edit mode by typing i
4. Add the following line to the file: TGI_MODEL: model name where 'model name' is contents of the Name column above of the model you'd like to use.
5. Once you've made the edit, save and exit by first pressing ctrl+c to exit edit mode, then type :wqfollowed by enter to write the changes and exit the file.
   ⚠️Be careful not to change anything other than the new line containing TGI_MODEL: model name
6. Now that you've edited the Layar configuration file, you'll need to restart the pods that consume TGI MODEL so that they are started with the correct model in place.
7. Run k delete deployment certara-llm certara-tgi
8. Run kps every 10 seconds until you don't see the llm and tgi pod in the stack
9. Run /deployLayar.sh llm_model_swap to redeploy the pods with the new model.
10. You should be able to use kps and k <pod name> logsto watch as the pods start up. The model server(the TGI pod) should log that the model you specified has been loaded successfully.

Once the pods have finished loading, you should now be able to now use Composer and other apps that use the /generate endpoint

If you have issues, please contact Certara Support at [email protected]