Prompting and Prompt Engineering

Templates

Prompt templates allow users to leverage structured prompts for generating text. With prompt templates, users can choose from pre-defined prompts that include placeholders for certain information and the model can generate text based on that information.

On the prompt page, users can select from a variety of templates in the Template dropdown.

Example of selecting the "Generic Classification" prompt template.

Variables

You can add variables into your prompt from your dataset using the format {{ variableName }}. Typing {{ variableName }} in the prompt editor will create a variable.

For instance, let's say we want to generate a text about a certain topic. We can define a prompt template like this: "Write me an event description for the event titled {{ event }} with a size of {{ size }} targeting the demographic {{ demographic }}."

Here, {{ event }}, {{ size }}, and {{ demographic }} are placeholders for specific variables that will be replaced with values during text generation.

To view generations on multiple rows of data at once, or to run evaluations, you can upload a dataset.

In this case, you can replace the text input variables with data from a column of the dataset you upload. For example, naming a variable the columnName will replace the placeholders with specific data in your dataset from the column named columnName.

Variables you use in your prompt will show up in the right sidebar under "Variables".

If you add variables in your prompt, they will show up in the sidebar.

If you select a dataset, the column names for the dataset will show up as variables alongside a dataset icon. Variables that are used in your prompt will be shown as filled in with a solid border, while any variables not referenced in your dataset will be shown as not filled in with a dotted line border.

Adding and referencing a dataset will also allow you to run inference using your prompt on multiple rows of your dataset at once.

In this example, "event", "demographic", and "size" are referenced as variables from the dataset in the prompt, while "description" is not.

Editing

You can directly edit the Prompt Template. Try adding more context about the problem or other prompt engineering techniques!

Edit the prompt template to add your own variables.

At inference time, all variables like {{ variableName }} will be replaced with the actual text on which to run LLM inference (i.e. the text for which to generate a new output).

Few-Shot Examples

Spellbook makes it easy to add few shot examples to your prompt, directly from your dataset!

This is performed with the help of the reserved keyword examples[k] where k is an integer >= 0. You can specify specific columns to pull examples from using the syntax examples[k].columnName.

For example, you can insert a 3-shot example using multiple inputs like:

Input 1: {{ examples[0].column1 }}
Input 2: {{ examples[0].column2 }}
Output: {{ examples[0].column5 }}
Input 1: {{ examples[1].column1 }}
Input 2: {{ examples[1].column2 }}
Output: {{ examples[1].column5 }}
Input 1: {{ examples[2].column1 }}
Input 2: {{ examples[2].column2 }}
Output: {{ examples[2].column5 }}

Then for the query:

Input 1: {{ column1 }}
Input 2: {{ column2 }}
Output:

At inference time, Spellbook will replace these keywords with actual values, as shown in the Few-Shot examples tab.

Example of a 3-shot classification prompt.

Corresponding 3-shot examples that will be filled into the prompt at inference time.

Model Parameters

Also included on the right the Prompt page are additional settings such as:

Base Model — the base LLM that will run inference using your new prompt
Temperature — a value from 0 to 1 (2 for OpenAI models), where higher numbers correspond with more "creative" responses from the LLM
Maximum Tokens — the upper limit on the number of tokens generated by the model at inference time
Stop Sequence — a text sequence that denotes the end of a sentence, line, answer, etc.
Output taxonomy (only certain models)— if you're building a classification app, restrict the model output to only values that appear in a certain column of your dataset

System Messages

Chat optimized models, like GPT-Turbo 3.5 and GPT-4, allows user to specify "system" messages. The system message helps set the behavior of the assistant. In the example above, the assistant was instructed with "You are a helpful assistant."

When selecting a chat-optimized model, you will be able to specify a system message.

Testing

You can quickly test your prompt by simply clicking the Generate Outputs button. This will run inference on the displayed inputs from the dataset using the prompt and settings you have set on the page.

As with Prompt Variables, you can sample new input values from the dataset by clicking the 🔀 button.

You can also enter your own custom input! Just type into the input field at the top of the Inputs table — when you click Generate Outputs, the model will take in the custom input and any dataset values, and produce outputs for them.

View Settings

Under view settings, you can configure both the number of dataset rows you see, and the text display.

You can also choose to alter your text display. You can show token probabilities, output whitespace, and/or select an expected output column

Choose "Show Expected Output Column" and choose a column from the dropdown to compare text side by side.

Choose "Show Output Whitespace" to see visual representation of where whitespace, like newlines, tabs, and spaces, appear.

Choose "Show Token Probabilities" to see token probabilities and alternative choices the model sampled from for each token in the output.

Saving Your Work

Once you're happy with your prompt and any other settings, you can save your current configuration as an App Variant. You can always make a clone of the new variant to make additional changes to the prompt and settings.