Utilizing Custom Embedding Models

Introduction

With the release of Certara.AI version 1.18, embedding models obtained via HF or other sources can be utilized in the stack, allowing utilization of embedding models that work best for your use case.

Configuration

1. The model must be in a folder that just contains the model name. IE for Google embeddinggemma-300m, the folder name would be embeddinggemma-300m.
2. Run kubectl get pvc, this will output a list of the PVC names. Copy the name connected to the certara-tgi-pvc.
3. Copy the model folder to /data/vyasa-volumes/<tgi-pvc>/cutom_embedders/
   1. Example: kubectl cp embeddinggemma-300m /data/vyasa-volumes/<tgi-pvc>/cutom_embedders/
4. vim /data/layar/layar.configYou will need to edit the ALL_EMBEDDER_INFOportion of the config.
   1. Example:
      YAML

      ALL_EMBEDDER_INFO: '[{"embedder_name":"multi-qa-MiniLM-L6-cos-v1","gpu_ids":[0],"gpu_memory_percentage":"0.03"},{"embedder_name":"embeddinggemma-300m","gpu_ids":[0],"gpu_memory_percentage":"0.03"}]'

5. Save the changes to the config file and then restart the Layar stack.

📘

How To Restart Layar

If you need more information on how to restart Layar, review Upgrading Layar