Chatbots and prompt engineering

Last updated: 20 November 2024

To see updates, click on the ‘+’ sign below:

DateUpdate
20 Nov 2024Added new section Retrieval Augmented Generation

Understanding how to use AI based chatbots or search engines to search for information is important in order to improve your chances of getting a better response. ‘Better’ in this case could mean one or more of clearer, more accurate or more complete response.

Throughout this post, where ‘chatbots’ are referred to, it should be taken to mean AI based chatbots’ and/or search engines’.

Please note that observations and comments in this post only apply to the free versions of the available chatbots as this is the extent of my experience.

On this blog, you will find a number of posts on using AI based chatbots to search for regulatory information. That information is by and large still valid although prompt engineering (defined below) was not employed for the searches. Some of the important points from those posts are reiterated below:

  • When attempting to find answers to regulatory questions (or prompts) it is always important to prompt more than one AI based chatbot/search engine.
  • This is because the answer may vary from being slightly different to being considerably different between chatbots. It would seem prudent to try at least three different chatbots.
  • Even using the same prompt with the same chatbot on the same day at different times or on different days is likely to give different answers.
All about prompts

Whilst there is no guarantee that you will get a clear, correct or complete response when prompting chatbots, what you ask and how you ask it will definitely impact the quality of the response that you will get.

What is prompt engineering?

You will probably have heard of the term prompt engineering.

Prompt engineering is the process of designing and refining prompts or instructions to elicit specific responses from artificial intelligence models, particularly large language models (LLMs)

Essentially, the goal is to structure prompts in a way that can be accurately interpreted and understood by generative AI models to produce the desired outputs (or response).

Don’t expect to become an overnight expert at prompt engineering. It requires practice and the more you do it, the better you will become at it. Perseverance is the key to improvement.

How should you write prompts?

On the website of at least some of the chatbots, you should find a prompt engineering guide. In this blog post, under the column ‘prompt guide’ links are provided to the prompt engineering guide for each chatbot, wherever one is available e.g. you can view the guide guide for claude.ai here. Prompt engineering guides are a reasonable starting place to find out more about how to write prompts.

This article2 from Forbes is well worth reading on how to write prompts. To learn more about important prompt types and techniques, read this article3 from VentureBeat.

You can also find many clips on prompt engineering on YouTube or TikTok.

Some general things to think about when writing prompts

Be clear, direct, and detailed. The more detail that you can provide when you write a prompt, the better the chances that you will get a response that you will find useful.

Avoid rambling when you write prompts! Keep your sentences short and precise.

Prompt engineering frameworks

There is a wide range of prompt engineering frameworks and you can try out several of them in order to determine which of them works best for you.

Prompt engineering frameworks provide structured approaches for crafting effective prompts when interacting with AI models like ChatGPT. Some frameworks are listed below.

CRISPE (Context, Request, Input, Specify, Example)

  • Context: Set the background/scenario
  • Request: State what you want done
  • Input: Provide necessary information
  • Specify: Detail format, style, length
  • Example: Show a sample of desired output

ICE (Instruction, Context, Examples)

  • Give clear instructions first
  • Provide relevant context
  • Include examples of desired output Much simpler than CRISPE but still effective for basic tasks

Role-Task-Format

  • Define the AI’s role (e.g., “Act as a marketing expert”)
  • Specify the task clearly
  • State desired output format Particularly good for specialized tasks

Chain-of-Thought

  • Break complex problems into steps
  • Ask the AI to explain its reasoning
  • Use “Let’s solve this step by step” Excellent for problem-solving and analytical tasks

Few-Shot Prompting

  • Show 2-3 examples of input-output pairs
  • Then provide your actual input
  • Works well for pattern-based tasks

Some other frameworks include:

CREATE framework

CREATE stands for Character, Request, Examples, Adjustment, Type of Output, and Extras. This framework helps organize prompts into six key components:

  • Character: Define the role or persona for the AI
  • Request: Specify the task or action needed
  • Examples: Provide sample outputs if applicable
  • Adjustment: Fine-tune the response as needed
  • Type of Output: Indicate the desired format
  • Extras: Include any additional context or constraints

RACE framework

RACE focuses on four key elements:

  • Role: Specify the AI’s role or persona
  • Action: Detail the action or task required
  • Context: Provide relevant background information
  • Expectations: Describe the desired outcome

APE framework

APE stands for Action, Purpose, Expectation:

Expectation: State the desired outcome

Action: Define the specific task or activity

Purpose: Explain the intention or goal

COAST Framework

COAST incorporates five components:

  • Context: Set the stage for the interaction
  • Objective: Describe the main goal
  • Actions: Explain the required steps
  • Scenario: Provide situational details
  • Task: Define the specific task

You can prompt any of the chatbots to provide you examples of any of the above frameworks. Note that the above list of frameworks is by no means exhaustible.

Some helpful sentences that you might wish to consider adding to your prompts
  • Please provide reasoning for your strategy.
  • Please provide reasons for your choice(s).
  • For all factual information in your response, please provide the url links from which you have obtained the information.
  • Ask me questions until you are 95% sure that you can accomplish the task.
  • Confirm the confidence level of your response and explain why you have assigned that level of confidence.

The above list is by no means exhaustive.

Hallucinations

AI hallucinations can and do occur. This is when an AI chatbot generates incorrect, misleading or entirely nonsensical information.

Be ever vigilant about hallucinations when you use chatbots to search for regulatory information.

You can find out more about hallucinations in this article from Google.

Currency of the information provided by chatbots
  • If you prompt a chatbot about very recent events or information e.g. a new regulation, a new guideline or a new scientific paper, be aware that you will more than likely not get a correct answer if you get an answer at all.
  • The reason is that chatbots have a date cut off date and this can be anything from three months or more from the date of your prompt. The currency of information for a chatbot is only as good as its most recent update.
  • Before you prompt a chatbot with whatever it is that you are trying to find out about, you can always prompt it to ask about the currency of the response that it will provide.
Context window

The following information (bullet points only) was obtained from a search on Google Gemini.

  • What it is: The context window is the number of tokens that a language model can process at once. 
  • How it’s measured: The context window is measured in tokens, which are machine-readable representations of words, parts of words, or punctuation. 
  • Why it’s important:
    • A context window helps AI chatbots understand the semantics and syntax of language, and produce relevant responses. 
    • It also helps chatbots maintain conversation flow, avoid redundancy and remember details from previous interactions. 
  • How it works: If a conversation exceeds the context window, the model may “forget” earlier parts of the conversation to stay within the limit. 
  • How it varies: The size of the context window varies across different language models (LLMs)

To better understand context windows, you can read this article from ZDNET. As the article states, “Hence, if a conversation stays within the token limit, the AI chatbot can maintain the full context. But if it exceeds the token limit, the earliest tokens will likely be ignored or lost to stay within the window, so the bot will potentially lose some context”.1

Interestingly, you don’t always get a useful response when you ask a chatbot about is context window. See responses below to the same prompt put to to Claude.ai and Chat GPT

Q. What is the maximum limit of your context window?

A. Claude.ai

I don’t actually have a specific maximum context window size. I’m an AI assistant named Claude, created by Anthropic to be helpful, harmless, and honest. I don’t have detailed information about my own architecture or training.

A. Chat GPT response:

I can handle a context window of about 8,192 tokens. This includes both your inputs and my responses. If our conversation gets close to that limit, I might lose some earlier context to keep things flowing!

A. Perplexity.ai

The maximum context window size for me (GPT-4) is 128,000 tokens1. This represents a significant increase from earlier versions of GPT models. For comparison, GPT-3.5-turbo has a context window of 16,385 tokens.

The context window refers to the total number of tokens that can be processed in a single interaction, including both the input (user messages, system instructions, etc.) and the model’s output. This large context window allows for processing of much longer texts and more complex conversations compared to earlier models.

It’s important to note that while the context window is 128,000 tokens, there are still limitations on the output. The maximum number of tokens that can be generated in a single response is capped at 4,096 tokens. This means that even though the model can process a very large input, its individual responses are still constrained to be relatively concise.

Retrieval Augmented Generation
What is it?

RAG (Retrieval Augmented Generation) is like giving an AI chatbot a personalised reference library. Instead of just relying on its built-in knowledge, the AI can access and use specific documents, data, or information that you provide to give more accurate and relevant answers.

How might you use RAG in your day to day interaction with a chatbot?

You could use RAG for your Personal Knowledge Management e.g. you could connect an AI chatbot to your personal notes or documents (by uploading them) to get more personalised assistance.

References

1. What does a long context window mean for an AI model, like Gemini? by Maria Diaz, 15 May 2024, ZDNET

2. How To Write Amazing Generative AI Prompts by Bernard Marr, 26 September 2024 – Forbes

3. Why prompt engineering is one of the most valuable skills today by Deven Pnachal of AT&T Labs, 22 September 2024 – VentureBeat.