プロンプトを隠すな、見せろ

Background

There are many libraries that aim to make the output of your LLMs better by re-writing or constructing the prompt for you. These libraries purport to make the output of your LLMs:

safer (ex: guardrails)

deterministic (ex: guidance)

structured (ex: instructor)

resilient (ex: langchain)

… or even optimized for an arbitrary metric (ex: DSPy).

A common theme among some of these tools is they encourage users to disintermediate themselves from prompting.

DSPy: “This is a new paradigm in which LMs and their prompts fade into the background …. you can compile your program again DSPy will create new effective prompts”

guidance “guidance is a programming paradigm that offers superior control and efficiency compared to conventional prompting …”

Even when tools don’t discourage prompting, I’ve often found it difficult to retrieve the final prompt(s) these tools send to the language model. The prompts sent by these tools to the LLM is a natural language description of what these tools are doing, and is the fastest way to understand how they work. Furthermore, some tools have dense terminology to describe internal constructs which can further obfuscate what they are doing.

For reasons I’ll explain below, I think most people would benefit from the following mindset:

In this blog post, I’ll show you how you can intercept API calls w/prompts for any tool, without having to fumble through docs or read source code. I’ll show you how to setup and operate mitmproxy with examples from the LLM the tools I previously mentioned.

Motivation: Minimize accidental complexity

Before adopting an abstraction, its important to consider the dangers of taking on accidental complexity. This danger is acute for LLM abstractions relative to programming abstractions. With LLM abstractions, we often force the user to regress towards writing code instead of conversing with the AI in natural language, which can run counter to the purpose of LLMs:

Programming abstraction -> a human-like language you can use to translate your task into machine code

LLM abstraction -> an unintelligible framework you can use to translate your task into human language

— Hamel Husain (@HamelHusain) February 5, 2024

While this is a cheeky comment, it’s worth keeping this in mind while evaluating tools. There are two primary types of automation that tools provide:

Interleaving code and LLMs: Expressing this automation is often best done through code, since code must be run to carry out the task. Examples include routing, executing functions, retries, chaining, etc.

Re-Writing and constructing prompts: Expressing your intent is often best done through natural language. However, there are exceptions! For example, it is convenient to express a function definition or schema from code instead of natural language.

Many frameworks offer both types of automation. However, going too far with the second type can have negative consequences. Seeing the prompt allows you decide:

Is this framework really necessary?

Should I just steal the final prompt (a string) and jettison the framework?

Can we write a better prompt than this (shorter, aligned with your intent, etc)?

Is this the best approach (do the # of API calls seem appropriate)?

In my experience, seeing the prompts and API calls are essential to making informed decisions.

Intercepting LLM API calls

There are many possible ways to intercept LLM API calls, such as monkey patching source code or finding a user-facing option. I’ve found that those approaches take far too much time since the quality of source code and documentation can vary greatly. After all, I just want to see API calls without worrying about how the code works!

A framework agnostic way to see API calls is to setup a proxy that logs your outgoing API requests. This is easy to do with mitmproxy, an free, open-source HTTPS proxy.

Setting Up mitmproxy

This is an opinionated way to setup mitmproxythat’s beginner-friendly for our intended purposes:

Follow the installation instructions on the website

Start the interactive UI by running mitmweb in the terminal. Pay attention to the url of the interactive UI in the logs which will look something like this: Web server listening at http://127.0.0.1:8081/

Next, you need to configure your device (i.e. your laptop) to route all traffic through mitproxy, which listens on http://localhost:8080. Per the documentation:

We recommend to simply search the web on how to configure an HTTP proxy for your system. Some operating system have a global settings, some browser have their own, other applications use environment variables, etc.

In my case, A google search for “set proxy for macos” returned these results:

choose Apple menu > System Settings, click Network in the sidebar, click a network service on the right, click Details, then click Proxies.

I then insert localhost and 8080 in the following places in the UI:

Next, navigate to http://mitm.it and it will give you instructions on how to install the mitmproxy Certificate Authority (CA), which you will need for intercepting HTTPS requests. (You can also do this manually here.) Also, take note of the location of the CA file as we will reference it later.

You can test that everything works by browsing to a website like https://mitmproxy.org/, and seeing the corresponding output in the mtimweb UI which for me is located at http://127.0.0.1:8081/ (look at the logs in your terminal to get the URL).

Now that you set everything up, you can disable the proxy that you previously enabled on your network. I do this on my mac by toggling the proxy buttons in the screenshot I showed above. This is because we want to scope the proxy to only the python program to eliminate unnecessary noise.

Tip

Networking related software commonly allows you to proxy outgoing requests by setting environment variables. This is the approach we will use to scope our proxy to specific Python programs. However, I encourage you to play with other types of programs to see what you find after you are comfortable!

Environment variables for Python

We need to set the following environment variables so that the requests and httpx libraries will direct traffic to the proxy and reference the CA file for HTTPS traffic:

Important

Make sure you set these environment variables before running any of the code snippets in this blog post.

import os

The location of my CA File

cert_file = '/Users/hamel/Downloads/mitmproxy-ca-cert.pem'

os.environ['REQUESTS_CA_BUNDLE'] = cert_file

os.environ['SSL_CERT_FILE'] = cert_file

os.environ['HTTPS_PROXY'] = 'http://127.0.0.1:8080'

You can do a minimal test by running the following code:

import requests

requests.post('https://httpbin.org/post',

data={'key': 'value'})

<Response [200]>

This will appear in the UI like so:

Examples

Now for the fun part, let’s run through some examples of LLM libraries and intercept their API calls!

Guardrails

Guardrails allows you specify structure and types, which it uses to validate and correct the outputs of large language models. This is a hello world example from the guardrails-ai/guardrails README:

from pydantic import BaseModel, Field

from guardrails import Guard

import openai

class Pet(BaseModel):

pet_type: str = Field(description="Species of pet")

name: str = Field(description="a unique pet name")

prompt = """

What kind of pet should I get and what should I name it?

${gr.complete_json_suffix_v2}

"""

guard = Guard.from_pydantic(output_class=Pet, prompt=prompt)

validated_output, *rest = guard(

llm_api=openai.completions.create,

engine="gpt-3.5-turbo-instruct"

)

print(f"{validated_output}")

{

"pet_type": "dog",

"name": "Buddy

What is happening here? How is this structured output and validation working? Looking at the mitmproxy UI, I can see that the above code resulted in two LLM API calls, the first one with this prompt:

What kind of pet should I get and what should I name it?

Given below is XML that describes the information to extract from this document and the tags to extract it into.

<output>

<string name="pet_type" description="Species of pet"/>

<string name="name" description="a unique pet name"/>

</output>

ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the name attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise.

Here are examples of simple (XML, JSON) pairs that show the expected behavior:

• <string name='foo' format='two-words lower-case' /> => {'foo': 'example one'}
• <list name='bar'><string format='upper-case' /></list> => {"bar": ['STRING ONE', 'STRING TWO', etc.]}
• <object name='baz'><string name="foo" format="capitalize two-words" /><integer name="index" format="1-indexed" /></object> => {'baz': {'foo': 'Some String', 'index': 1}}

Followed by another call with this prompt:

I was given the following response, which was not parseable as JSON.

"{\n \"pet_type\": \"dog\",\n \"name\": \"Buddy"

Help me correct this by making it valid JSON.

Given below is XML that describes the information to extract from this document and the tags to extract it into.

<output>

<string name="pet_type" description="Species of pet"/>

<string name="name" description="a unique pet name"/>

</output>

ONLY return a valid JSON object (no other text is necessary), where the key of the field in JSON is the name attribute of the corresponding XML, and the value is of the type specified by the corresponding XML's tag. The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise. If you are unsure anywhere, enter null.

Woof. That’s a whole lot of ceremony to get structured output! We learned that this library’s approach to structured output uses XML schemas (while others use function calling). It’s worth considering if you can fashion a better or simpler approach now that the magic has been lifted. Either way, we now have insight into how it works without dragging you into unnecessary complexity, which is a win.

Guidance

Guidance offers constrained generation and programming constructs for writing prompts. Let’s dive into a chat example from their tutorials:

import guidance

gpt35 = guidance.models.OpenAI("gpt-3.5-turbo")

import re

from guidance import gen, select, system, user, assistant

@guidance

def plan_for_goal(lm, goal: str):

# This is a helper function which we will use below

def parse_best(prosandcons, options):

best = re.search(r'Best=(\d+)', prosandcons)

if not best:

best = re.search(r'Best.*?(\d+)', 'Best= option is 3')

if best:

best = int(best.group(1))

else:

best = 0

return options[best]

# Some general instruction to the model

with system():

lm += "You are a helpful assistant."

# Simulate a simple request from the user

# Note that we switch to using 'lm2' here, because these are intermediate steps (so we don't want to overwrite the current lm object)

with user():

lm2 = lm + f"""\

I want to {goal}

Can you please generate one option for how to accomplish this?

Please make the option very short, at most one line."""

# Generate several options. Note that this means several sequential generation requests

n_options = 5

with assistant():

options = []

for i in range(n_options):

options.append((lm2 + gen(name='option', temperature=1.0, max_tokens=50))["option"])

# Have the user request pros and cons

with user():

lm2 += f"""\

I want to {goal}

Can you please comment on the pros and cons of each of the following options, and then pick the best option?

"""

for i, opt in enumerate(options):

lm2 += f"Option {i}: {opt}\n"

lm2 += f"""\

Please discuss each option very briefly (one line for pros, one for cons), and end by saying Best=X, where X is the number of the best option."""

# Get the pros and cons from the model

with assistant():

lm2 += gen(name='prosandcons', temperature=0.0, max_tokens=600, stop="Best=") + "Best=" + gen("best", regex="[0-9]+")

# The user now extracts the one selected as the best, and asks for a full plan

# We switch back to 'lm' because this is the final result we want

with user():

lm += f"""\

I want to {goal}

Here is my plan: {options[int(lm2["best"])]}

Please elaborate on this plan, and tell me how to best accomplish it."""

# The plan is generated

with assistant():

lm += gen(name='plan', max_tokens=500)

return lm

results = gpt35 + plan_for_goal(goal="read more books")

system

You are a helpful assistant.

user

I want to read more books

Here is my plan: Set aside 30 minutes of dedicated reading time each day.

Please elaborate on this plan, and tell me how to best accomplish it.

assistant

Setting aside 30 minutes of dedicated reading time each day is a great plan to read more books. Here are some tips to help you accomplish this goal:

1. Establish a routine: Choose a specific time of day that works best for you, whether it's in the morning, during lunch break, or before bed. Consistency is key to forming a habit.

1. Create a reading-friendly environment: Find a quiet and comfortable spot where you can focus on your reading without distractions. It could be a cozy corner in your home, a park bench, or a local library.

1. Minimize distractions: Put away your phone, turn off the TV, and avoid any other potential interruptions during your dedicated reading time. This will help you stay focused and fully immerse yourself in the book.

1. Choose books that interest you: Select books that align with your personal interests, hobbies, or goals. When you're genuinely interested in the subject matter, you'll be more motivated to read regularly.

1. Start with manageable goals: If you're new to reading or have a busy schedule, start with a smaller time commitment, such as 15 minutes, and gradually increase it to 30 minutes or more as you become more comfortable.

1. Set a timer: Use a timer or a reading app that allows you to track your reading time. This will help you stay accountable and ensure that you dedicate the full 30 minutes to reading.

1. Make reading enjoyable: Create a cozy reading atmosphere by lighting a candle, sipping a cup of tea, or playing soft background music. Engaging all your senses can enhance your reading experience.

1. Join a book club or reading group: Consider joining a book club or participating in a reading group to connect with fellow book lovers. This can provide additional motivation, discussion opportunities, and book recommendations.

1. Keep a reading log: Maintain a record of the books you've read, along with your thoughts and reflections. This can help you track your progress, discover patterns in your reading preferences, and serve as a source of inspiration for future reading.

1. Be flexible: While it's important to have a dedicated reading time, be flexible and adaptable. Life can sometimes get busy, so if you miss a day, don't be discouraged. Simply pick up where you left off and continue with your reading routine.

Remember, the goal is to enjoy the process of reading and make it a regular part of your life. Happy reading!

This looks pretty neat! But what is it doing exactly? This makes a total of 7 calls to OpenAI, which I have put in this gist. 5 of 7 of these API calls are “internal” thoughts asking the LLM to generate ideas. Even though the temperature is set to 1.0, these “ideas” are mostly redundant. The penultimate call to OpenAI enumerates these “ideas” which I’ve included below:

I want to read more books

Can you please comment on the pros and cons of each of the following options, and then pick the best option?

Option 0: Set aside dedicated time each day for reading.

Option 1: Set aside 30 minutes of dedicated reading time each day.

Option 2: Set aside dedicated time each day for reading.

Option 3: Set aside dedicated time each day for reading.

Option 4: Join a book club.

Please discuss each option very briefly (one line for pros, one for cons), and end by saying Best=X, where X is the number of the best option.

I know from experience that you are likely to get better results if you tell the language model to generate ideas in one shot. That way, the LLM can reference previous ideas and achieve more diversity. This is a good example of accidental complexity: its very tempting to take this design pattern and apply it blindly. This is less of a critique of this particular framework, since the code makes it clear that 5 independent calls will happen. Either way, its good idea to check your work by inspecting API calls!.

Langchain

Langchain is a multi-tool for all things LLM. Lots of people rely on Langchain when get started with LLMs. The core LangChain library doesn’t generally hide prompts from you, however there are experimental features that do. Let’s take a look at one of these features called SmartLLMChain:

from langchain.prompts import PromptTemplate

from langchain_experimental.smart_llm import SmartLLMChain

from langchain_openai import ChatOpenAI

hard_question = "I have a 12 liter jug and a 6 liter jug.\

I want to measure 6 liters. How do I do it?"

prompt = PromptTemplate.from_template(hard_question)

llm = ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo")

chain = SmartLLMChain(llm=llm, prompt=prompt,

n_ideas=2,

verbose=True)

result = chain.run({})

print(result)

Idea 1: 1. Fill the 12 liter jug completely.

1. Pour the contents of the 12 liter jug into the 6 liter jug. This will leave you with 6 liters in the 12 liter jug.
2. Empty the 6 liter jug.
3. Pour the remaining 6 liters from the 12 liter jug into the now empty 6 liter jug.
4. You now have 6 liters in the 6 liter jug.

Idea 2: 1. Fill the 12 liter jug completely.

1. Pour the contents of the 12 liter jug into the 6 liter jug. This will leave you with 6 liters in the 12 liter jug.
2. Empty the 6 liter jug.
3. Pour the remaining 6 liters from the 12 liter jug into the now empty 6 liter jug.
4. You now have 6 liters in the 6 liter jug.

Improved Answer:

1. Fill the 12 liter jug completely.
2. Pour the contents of the 12 liter jug into the 6 liter jug until the 6 liter jug is full. This will leave you with 6 liters in the 12 liter jug and the 6 liter jug completely filled.
3. Empty the 6 liter jug.
4. Pour the remaining 6 liters from the 12 liter jug into the now empty 6 liter jug.
5. You now have 6 liters in the 6 liter jug.

Full Answer:

To measure 6 liters using a 12 liter jug and a 6 liter jug, follow these steps:

1. Fill the 12 liter jug completely.
2. Pour the contents of the 12 liter jug into the 6 liter jug until the 6 liter jug is full. This will leave you with 6 liters in the 12 liter jug and the 6 liter jug completely filled.
3. Empty the 6 liter jug.
4. Pour the remaining 6 liters from the 12 liter jug into the now empty 6 liter jug.
5. You now have 6 liters in the 6 liter jug.

Neat! So what happened exactly? While this API emits logs that show you a lot of information (available on this gist), the API request pattern is interesting:

Two seperate api calls for each “idea”.

Another API call that incorporates the two ideas as context, with the prompt:

You are a researcher tasked with investigating the 2 resp