できることを蓄積せよ

*Agentic Engineering Patterns >*

Many of my tips for working productively with coding agents are extensions of advice I've found useful in my career without them. Here's a great example of that: hoard things you know how to do.

A big part of the skill in building software is understanding what's possible and what isn't, and having at least a rough idea of how those things can be accomplished.

These questions can be broad or quite obscure. Can a web page run OCR operations in JavaScript alone? Can an iPhone app pair with a Bluetooth device even when the app isn't running? Can we process a 100GB JSON file in Python without loading the entire thing into memory first?

The more answers to questions like this you have under your belt, the more likely you'll be able to spot opportunities to deploy technology to solve problems in ways other people may not have thought of yet.

Knowing that something is theoretically possible is not the same as having seen it done for yourself. A key asset to develop as a software professional is a deep collection of answers to questions like this, ideally illustrated by running code.

I hoard solutions like this in a number of different ways. My blog and TIL blog are crammed with notes on things I've figured out how to do. I have over a thousand GitHub repos collecting code I've written for different projects, many of them small proof-of-concepts that demonstrate a key idea.

More recently I've used LLMs to help expand my collection of code solutions to interesting problems.

tools.simonwillison.net is my largest collection of LLM-assisted tools and prototypes. I use this to collect what I call HTML tools - single HTML pages that embed JavaScript and CSS and solve a specific problem.

My simonw/research repository has larger, more complex examples where I’ve challenged a coding agent to research a problem and come back with working code and a written report detailing what it found out.

Recombining things from your hoard

Why collect all of this stuff? Aside from helping you build and extend your own abilities, the assets you generate along the way become incredibly powerful inputs for your coding agents.

One of my favorite prompting patterns is to tell an agent to build something new by combining two or more existing working examples.

A project that helped crystallize how effective this can be was the first thing I added to my tools collection - a browser-based OCR tool, described in more detail here.

I wanted an easy, browser-based tool for OCRing pages from PDF files - in particular PDFs that consist entirely of scanned images with no text version provided at all.

I had previously experimented with running the Tesseract.js OCR library in my browser, and found it to be very capable. That library provides a WebAssembly build of the mature Tesseract OCR engine and lets you call it from JavaScript to extract text from an image.

I didn’t want to work with images though, I wanted to work with PDFs. Then I remembered that I had also worked with Mozilla’s PDF.js library, which among other things can turn individual pages of a PDF into rendered images.

I had snippets of JavaScript for both of those libraries in my notes.

Here’s the full prompt I fed into a model (at the time it was Claude 3 Opus), combining my two examples and describing the solution I was looking for:

This code shows how to open a PDF and turn it into an image per page:

html>

head>

title>PDF to Imagestitle>

script src="https://cdnjs.cloudflare.com/ajax/libs/pdf.js/2.9.359/pdf.min.js">script>

style>

.image-container img {

margin-bottom: 10px;

}

.image-container p {

margin: 0;

font-size: 14px;

color: #888;

}

style>

head>

body>

input type="file" id="fileInput" accept=".pdf" />

div class="image-container">div>

script>

const desiredWidth = 800;

const fileInput = document.getElementById('fileInput');

const imageContainer = document.querySelector('.image-container');

fileInput.addEventListener('change', handleFileUpload);

pdfjsLib.GlobalWorkerOptions.workerSrc = 'https://cdnjs.cloudflare.com/ajax/libs/pdf.js/2.9.359/pdf.worker.min.js';

async function handleFileUpload(event) {

const file = event.target.files[0];

const imageIterator = convertPDFToImages(file);

for await (const { imageURL, size } of imageIterator) {

const imgElement = document.createElement('img');

imgElement.src = imageURL;

imageContainer.appendChild(imgElement);

const sizeElement = document.createElement('p');

sizeElement.textContent = Size: ${formatSize(size)};

imageContainer.appendChild(sizeElement);

}

}

async function* convertPDFToImages(file) {

try {

const pdf = await pdfjsLib.getDocument(URL.createObjectURL(file)).promise;

const numPages = pdf.numPages;

for (let i = 1; i numPages; i++) {

const page = await pdf.getPage(i);

const viewport = page.getViewport({ scale: 1 });

const canvas = document.createElement('canvas');

const context = canvas.getContext('2d');

canvas.width = desiredWidth;

canvas.height = (desiredWidth / viewport.width) * viewport.height;

const renderContext = {

canvasContext: context,

viewport: page.getViewport({ scale: desiredWidth / viewport.width }),

};

await page.render(renderContext).promise;

const imageURL = canvas.toDataURL('image/jpeg', 0.8);

const size = calculateSize(imageURL);

yield { imageURL, size };

}

} catch (error) {

console.error('Error:', error);

}

}

function calculateSize(imageURL) {

<