HomeGuidesRecipesAPI EndpointsRelease NotesCommunity
Log In
Guides

Setting Generative System Model

Learn how to change the system model for Composer to any supported LLM

Models & Requirements

Default Model: Mistral 7B V0.1

ModelNameQuantizationMinimum RequirementsMinimum Layar Version
Mistral 7B V0.1mistralai/Mistral-7B-Instruct-v0.1NoneA10 24gb x11.7
Mixtral 8x7Bcasperhansen/mixtral-instruct-awqAWQA100 40gb x11.7
Llama 3 70Bcasperhansen/llama-3-70b-instruct-awqAWQA10 10gb x41.8
Llama 3 70Bmeta-llama/llama-3-70b-instructNoneA100 40gb x21.8
Llama 3.1 70Bmeta-llama/Meta-Llama-3.1-70B-InstructNoneA100 80gb x21.9
Llama 3.1 70Bhugging-quants/Meta-Llama-3.1-70B-Instruct-GPTQ-INT4GPTQA100 80gb x11.9
Llama 3.1 70Bhugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4AWQA100 80gb x11.9

🚧

Llama 3.1 VRAM Limitations

If you are on Layar 1.9, you will need to have GPUs with 80gb of VRAM. If there are further questions about this, please e-mail [email protected]

📘

GPU Configuration Considerations

If you are interested in putting the models on smaller GPU partitions, please review GPU Considerations

Setting Your System to a New Model

To change the model from the default, you will need to ssh into the instance in question.
Once you are logged in, follow these steps:

  1. Become root with sudo su -
  2. Once you have root access, you will be able to edit the Layar configuration file.
    Do this by running vi /data/layar/layar.config
  3. Enter edit mode by typing i
  4. Add the following line to the file: TGI_MODEL: model name where 'model name' is contents of the Name column above of the model you'd like to use.
  5. Once you've made the edit, save and exit by first pressing ctrl+c to exit edit mode, then type :wqfollowed by enter to write the changes and exit the file.
    ⚠️Be careful not to change anything other than the new line containing TGI_MODEL: model name
  6. Now that you've edited the Layar configuration file, you'll need to restart the pods that consume TGI MODEL so that they are started with the correct model in place.
  7. Run k delete deployment certara-llm certara-tgi
  8. Run kps every 10 seconds until you don't see the llm and tgi pod in the stack
  9. Run /deployLayar.sh llm_model_swap to redeploy the pods with the new model.
  10. You should be able to use kps and k <pod name> logsto watch as the pods start up. The model server(the TGI pod) should log that the model you specified has been loaded successfully.

Once the pods have finished loading, you should now be able to now use Composer and other apps that use the /generate endpoint

If you have issues, please contact Certara Support at [email protected]