Prompt Considerations
Introduction
When using Certara Generative AI, your prompt is critical to the accuracy of the response. This guide will go over ways to create prompts in order to provide the AI the information it needs to produce the response you are looking for.
Understanding the Data Set
The more you understand about your data set, the more the prompt can be fine tuned. Knowing small details like the variety of statistics or if the document will contain a table of information can be helpful.
Lets look at simple text, prompt.
Text: 10% of patients given the study drug had adverse effects while 2% given the placebo had adverse effects.
5% of patients who had adverse effects were positive for anti-drug antibodies.
What percent of patients had adverse effects? Only provide the percent. Do not elaborate.
We can see there are multiple percentages presented in the text, lets see how it responds.
10% of patients given the study drug had adverse effects,
and including the placebo group, 2% of all patients had adverse effects.
The additional information about anti-drug antibodies
does not affect the percentage of patients who had adverse effects.
The while the AI comes close to responding correctly. A curator may look at the number and realize the placebo percentage does not contribute to the overall percentage they are looking for. Let's tweak the prompt a bit and look at the output.
Text: 10% of patients given the study drug
had adverse effects while 2% given the placebo had adverse effects.
5% of patients who had adverse effects were positive for anti-drug antibodies.
What percent of patients had adverse effects? Do not include placebo patients.
Only provide the percent. Do not elaborate.
10%
(This calculation is based on the information provided that 10% of patients
given the study drug had adverse effects. The additional information about placebo patients
and anti-drug antibodies is not relevant to this specific question.)
The AI performs a lot better here by simply telling it to ignore placebo numbers.
When possible, tell the AI what you do and don't want.
Statement Repetition
While the above prompts get the number we wanted, it still included unnecessary elaboration. When talking to a human, "Only provide the percent" and "Do not elaborate" would indicate you do not want further explanation on where the percent came from. When prompting Certara Generative AI repeating what you want will help guide the response. Let's rewrite the prompt.
Text: 10% of patients given the study drug had adverse effects while 2% given the placebo
had adverse effects. 5% of patients who had adverse effects were positive for anti-drug antibodies.
What percent of patients had adverse effects? Do not include placebo patients.
Only provide the percent. Do not elaborate on where you found the answer.
Be concise, never supply supporting text.
10% (of patients given the study drug had adverse effects)
We are getting closer by adding "Be concise, never supply supporting text", essentially repeating "Do not elaborate". Lets see if we can just get the number,
Text: 10% of patients given the study drug had adverse effects while 2% given the placebo
had adverse effects. 5% of patients who had adverse effects were positive for anti-drug antibodies.
What percent of patients had adverse effects? Do not include placebo patients.
Only provide the percent. Do not elaborate on where you found the answer.
Be concise, never supply supporting text. Only answer with the number.
10%
Again repetition helps drive the response by only adding "Only answer with the number".
Longer or Shorter Responses
When using the API programmatically responses will need to be considered not just for the accuracy of the content but also the way the response is presented. More elaborate responses may need to be post processed. It's important to strike a balance between processing responses via prompt engineering or external scripting.
Typescript Definition Prompt
An alternative approach to formatting responses is prompting the AI to respond with a specific typescript definition. This is very useful when using the generate API programmatically. Lets take a look at how we can use this method with out previous prompts.
Text: 10% of patients given the study drug had adverse effects while 2% given the placebo
had adverse effects. 5% of patients who had adverse effects were positive for anti-drug antibodies.
What percent of patients had adverse effects. Do not include placebo patients?
Only provide the percent. Do not elaborate.
Always format your answer with with typescript definition {'answer' : integer}
{'answer' : 10}
With the initial prompt and adding our desired typescript definition, the response provides the number nested in the desired format. Even decimals can be given using this method and prompting the AI to return the number as a decimal.
Text: 10% of patients given the study drug had adverse effects while 2% given the placebo
had adverse effects. 5% of patients who had adverse effects were positive for anti-drug antibodies.
What percent of patients had adverse effects?
Do not include placebo patients. Only provide the percent.
Do not elaborate. Provide the answer as a decimal.
Always format your answer with with typescript definition {'answer' : integer}
{'answer' : 0.1}
If curation is necessary and we want an explanation of how this answer was found. We can add to the definition and remove the need for no elaboration.
Text: 10% of patients given the study drug had adverse effects while 2% given the placebo
had adverse effects. 5% of patients who had adverse effects were positive for anti-drug antibodies.
What percent of patients had adverse effects. Do not include placebo patients.
Only provide the percent. Provide the answer as a decimal.
Always format your answer with with typescript definition {'answer' : integer, 'explanation' : string}
{'answer' : 0.1, 'explanation' : '10% of patients given the study drug had adverse effects.'}
Converting to JSON
It's convenient to have a response as a typescript definition since this allows you to quickly convert it to dictionary format using
.json(), allowing you to iterate through the keys of the dictionary.
Chain Prompting
Lets look at a more detailed bit of text.
Text: Study 1 had a study drug group of 300 patients while the placebo group had 100 patients.
Study 2 had a study group of 400 patients while placebo group had 200 patients.
Assays were taken every 10, 20, and 30 weeks for both studies.
In study 1, 45/300 patients where positive for anti-drug antibodies while 100/300 had adverse effects.
While 100/200 given the placebo had adverse effects.
In study 2, 145/400 patients where positive for anti-drug antibodies while 200/400 had adverse effects.
We have multiple studies, some of the values are shared between each study while others aren't. Lets prompt the AI with a simple prompt and see what response is given.
Text: Study 1 had a study drug group of 300 patients while the placebo group had 100 patients.
Study 2 had a study group of 400 patients while placebo group had 200 patients.
Assays were taken every 10, 20, and 30 weeks for both studies.
In study 1, 45/300 patients where positive for anti-drug antibodies while 100/300 had adverse effects.
While 100/200 given the placebo had adverse effects.
In study 2, 145/400 patients where positive for anti-drug antibodies while 200/400 had adverse effects.
What percent of patients had adverse reactions?
To calculate the percentage of patients who had adverse reactions in each study,
we can use the following formula:
Percentage = (number of patients with adverse reactions / total number of patients) 100%
For Study 1, the calculation would be:
(100 (study drug group adverse reactions) + 100 (placebo group adverse reactions))
/ (300 (study drug group) + 200 (placebo group)) 100% = 50%
For Study 2, the calculation would be:
(200 (study drug group adverse reactions) + 200 (placebo group adverse reactions))
/ (400 (study drug group) + 200 (placebo group)) 100% = 50%
Therefore, in both Study 1 and Study 2, approximately 50% of the patients had adverse reactions.
The response is close to being correct, however it's assuming the study 1 placebo group is also linked to study 2. We can make this easier my chaining 2 prompts together. First we need to prompt for the study labels.
Text: Study 1 had a study drug group of 300 patients while the placebo group had 100 patients.
Study 2 had a study group of 400 patients while placebo group had 200 patients.
Assays were taken every 10, 20, and 30 weeks for both studies.
In study 1, 45/300 patients where positive for anti-drug antibodies while 100/300 had adverse effects.
While 100/200 given the placebo had adverse effects.
In study 2, 145/400 patients where positive for anti-drug antibodies while 200/400 had adverse effects.
What were the labels of the studies performed in the text? Do not elaborate.
Always format your answer with with typescript definition {'answer' : [list]}.
Be concise, never supply supporting text.
{'answer' : ['Study 1', 'Study 2']}
Combining Repetition and Typescript Definitions
In the prompt above, we are using both typescript definitions and repetition to ensure it only gives the typescript definition without explaining how it came to that conclusion.
Now that we have the study labels, we can use one of them to inform the AI in our second prompt.
Text: Study 1 had a study drug group of 300 patients while the placebo group had 100 patients.
Study 2 had a study group of 400 patients while placebo group had 200 patients.
Assays were taken every 10, 20, and 30 weeks for both studies.
In study 1, 45/300 patients where positive for anti-drug antibodies while 100/300 had adverse effects.
While 100/200 given the placebo had adverse effects.
In study 2, 145/400 patients where positive for anti-drug antibodies while 200/400 had adverse effects.
What percent of patients in study 2 had adverse reactions?
To find the percentage of patients in Study 2 who had adverse reactions, we can use the formula:
Percentage = (number of patients with adverse reactions / total number of patients) * 100%
In Study 2, there were 200 patients who had adverse reactions out of a total of 400 patients. So,
Percentage = (200/400) * 100%
Percentage = 50%
Therefore, 50% of the patients in Study 2 had adverse reactions.
We can see that going through one extra prompt and using the first answer to guide the second prompt allows us to target the proper information. Chain prompting is a great way to lower the scope of subsequent prompts.
System Prompts
Certara Generative AI allows you to utilize system prompts help drive responses. Think of a system prompt as an additional way to provide context for the AI. Lets look at table in text form, that denotes a table of dose groups, number of patients in the group, and number of patients who experience adverse effects in that group.
Text:
45 mg 1 Dose
SC 80 mg Q2W
240 mg Q2W
360 mg Q3W
460 mg Q2W
480 mg Q4W
600 mg Q3W SC
900 mg LD 600 mg Q3W SC
800 mg Q2W
Total
Analysis set: Safety
8
7
165
28
4
31
11
19
6
279
Subjects with 1 or more TEAEs
5
(62.5%)
7
(100.0%)
161
(97.6%)
22
(78.6%)
4
(100.0%)
29
(93.5%)
11
(100.0%)
16
(84.2%)
6
(100.0%)
261
(93.5%)
Now lets try prompting against the data.
In the above text, how many subjects were in the 900 mg LD 600 mg Q3W SC dose group?
According to the text, there were 11 subjects in the 900 mg LD 600 mg Q3W SC dose group.
The response given is incorrect, the AI is getting confused at how to scroll the text. Lets provide context with a system prompt.
Utilizing System Prompts With The Layar API
If you are interested in how to utilize system prompts when using the generate endpoint. Please reference the Generate Parameters Guide
The above text is listing percentages of subjects who have experienced TEAEs, a number of subjects
being part of specific dosing groups. A dose group can consist of multiple "MGs"
for example, "900 mg LD 600 mg Q3W SC" is one dose group.
The "Analysis set: Safety" row indicates how many patients
were in each dosing group. The "Subjects with 1 or more TEAEs" row indicates how many patients
experienced multiple TEAEs. An example would be, the number of subjects in the X mg
Dose is correlates to the "Analysis set: Safety" row.
The 45mg 1 dose group is the first group, If you are trying to correlate values in the
following sections back to the 45 mg 1 dose group, the correct value is the first value listed.
You will need to find the number relevant to the dose group.
The system prompt does two things.
- Describes what the AI is about to be looking at.
- Explains how to walk through the information and correlated headers to data.
Lets do the same prompt and see what the response is.
The number of subjects in the 900 mg LD 600 mg Q3W SC dose group,
as indicated in the "Analysis set: Safety" row, is 19.
Now the AI is responding correctly. We can take this a step further and change our prompt to request a summary of the information.
Summarize the attached text.
Based on the provided text, I will summarize the information as follows:
The text lists the number of subjects in each dosing group and the number of subjects
who experienced one or more TEAEs (Treatment Emergent Adverse Events) in each group.
Here is a summary of the data:
* 45 mg 1 Dose: 8 subjects, 5 (62.5%) experienced TEAEs
* SC 80 mg Q2W: 7 subjects, 7 (100.0%) experienced TEAEs
* 240 mg Q2W: 165 subjects, 161 (97.6%) experienced TEAEs
* 360 mg Q3W: 28 subjects, 22 (78.6%) experienced TEAEs
* 460 mg Q2W: 4 subjects, 4 (100.0%) experienced TEAEs
* 480 mg Q4W: 31 subjects, 29 (93.5%) experienced TEAEs
* 600 mg Q3W SC: 11 subjects, 11 (100.0%) experienced TEAEs
* 900 mg LD 600 mg Q3W SC: 19 subjects, 16 (84.2%) experienced TEAEs
* 800 mg Q2W: 6 subjects, 6 (100.0%) experienced TEAEs
Overall, there were a total of 279 subjects across all dosing groups, with 261 (93.5%)
experiencing one or more TEAEs.
The AI is able to respond correctly for all groups using context provided in the system prompt. System prompting is very useful when the sources being given to the AI are consistent in formatting, however you can use non-descriptive system prompts to cover a multitude of scenarios.
Updated 4 months ago