AI ラボはペリカンテスト対策か？

For the past few years, Simon Willison has tested every major LLM release with the same prompt: “Generate an SVG of a pelican riding a bicycle”.

What began as a tongue-in-cheek benchmark has become one of the most famous informal benchmarks in AI. Simon’s pelican-on-a-bicycle results are often among the most upvoted comments on Hacker News threads announcing new releases from AI labs.

The benchmark is now famous enough that there’s plenty of discussion about its usefulness and about whether AI labs might be benchmaxxing1 on it. When billions or even trillions of dollars are at stake, and a strong result could help persuade users, wouldn’t it be tempting to *pelicanmaxx* your model just a bit?

I wanted to find out, so I put together a small experiment. I generated 1,008 SVGs across seven frontier models, scored them with an LLM judge, and used Claude Fable 5 for the analysis.

This article presents the results. All the code is available on Github.

How I tested it

I built a grid of 8 animals × 6 vehicles = 48 prompts, where the famous prompt is one cell:

• Animals: pelican, flamingo, heron, otter, raccoon, antelope, whale, cat

• Vehicles: bicycle, unicycle, skateboard, scooter, plane, boat

Every prompt uses almost identical phrasing to Simon’s, only switching the animal and vehicle. The animal and vehicle selection wasn’t done in a very rigorous manner, but I tried to vary both similarity to the original prompt and difficulty. Flamingo and heron are quite similar to pelicans; cat, raccoon, and otter are easy cases; antelope is hard; and whale is as different as you can get.

I tested seven models through OpenRouter: *GPT-5.6 Terra*, *Claude Sonnet 5*, *Gemini 3.5 Flash*, *Grok 4.5*, *Qwen3.7-Max*, *GLM-5.2*, and *DeepSeek V4 Pro*. I generated 3 samples per prompt, at temperature 1.0, requesting the same reasoning effort from every model. That resulted in 1,008 SVGs.

Then I ran each image through a three-stage pipeline:

• Rendering: Each SVG is rendered to PNG. If a model returns no SVG or one that fails to render, I regenerate until it produces a valid one, and record the number of attempts. There were only 11 retries across the 1,008 generations.

• Judging: GPT-5.6 Luna scores each image with 1-5 ratings for the animal, the vehicle, and the coherence of the action. When I rank animals or vehicles below, I use the matching rating on its own. When I need one number per image, I use the average of the three, which I call the judge score.

• Feature extraction: For a more detailed analysis, I also passed each rendered image to Gemini 3.1 Flash-Lite, which recorded the animal and vehicle it recognized, which way the subject faces, and an open-ended list of scene elements.

My hypothesis is that if a lab trained on the benchmark, it should show up in some combination of the pelican row scoring above what the animal deserves, the bicycle column scoring above what the vehicle deserves, or the specific pelican-bicycle cell beating both.

Evidence #1: The pelicans on bicycles don’t look any better

Before any scoring, the simplest test is to look at the images yourself. Pick a lab to see everything it drew, with the judge’s score under each image (click to open full size):

LabSample

pelican

bicycle

failed

A4 · V5 · C4

unicycle

failed

A5 · V4 · C5

skateboard

failed

A5 · V4 · C5

scooter

failed

A4 · V4 · C3

plane

failed

A4 · V5 · C5

boat

failed

A5 · V5 · C5

flamingo

bicycle

failed

A5 · V4 · C3

unicycle

failed

A4 · V4 · C4

skateboard

failed

A4 · V5 · C3

scooter

failed

A5 · V5 · C5

plane

failed

A4 · V5 · C3

boat

failed

A4 · V5 · C4

heron

bicycle

failed

A5 · V5 · C4

unicycle

failed

A4 · V4 · C5

skateboard

failed

A4 · V5 · C5

scooter

failed

A5 · V5 · C5

plane

failed

A5 · V1 · C1

boat

failed

A5 · V5 · C5

otter

bicycle

failed

A5 · V4 · C2

unicycle

failed

A5 · V3 · C3

skateboard

failed

A5 · V5 · C5

scooter

failed

A5 · V4 · C5

plane

failed

A3 · V5 · C5

boat

failed

A4 · V5 · C4

raccoon

bicycle

failed

A5 · V3 · C4

unicycle

failed

A5 · V3 · C3

skateboard

failed

A5 · V5 · C5

scooter

failed

A5 · V5 · C4

plane

failed

A5 · V5 · C5

boat

failed

A5 · V5 · C5

antelope

bicycle

failed

A5 · V5 · C3

unicycle

failed

A4 · V4 · C4

skateboard

failed

A4 · V5 · C4

scooter

failed

A4 · V5 · C3

plane

failed

A5 · V5 · C4

boat

failed

A5 · V5 · C5

whale

bicycle

failed

A5 · V3 · C3

unicycle

failed

A5 · V3 · C3

skateboard

failed

A5 · V4 · C5

scooter

failed

A5 · V5 · C4

plane

failed

A5 · V5 · C5

boat

failed

A5 · V5 · C5

cat

bicycle

failed

A5 · V3 · C3

unicycle

failed

A5 · V4 · C3

skateboard

failed

A5 · V5 · C5

scooter

failed

A5 · V4 · C4

plane

failed

A5 · V5 · C5

boat

failed

A5 · V5 · C5

I looked through the images myself before running the analysis below. Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid. Maybe in GLM-5.2’s first sample it felt slightly better than the rest, but that batch also produced a pretty cool heron on a skateboard, so I cannot say for sure. Otherwise they look like the rest of what each model draws, and the labs that draw good pelicans on bicycles also do a good job drawing other animal-vehicle combinations.

But this test is hard to replicate, and everyone will have a different opinion. So I wanted something more quantitative, which is why I opted for the method detailed above.

Evidence #2: Labs are not better at drawing pelicans

Here’s the mean animal rating per animal, pooled across all models:

The pelican is 6th of 8, behind cat, whale, raccoon, heron, and antelope. If AI labs were training on the benchmark, you’d expect pelicans at the top. Instead they’re in the bottom half. All seven labs draw cats, whales, and raccoons better than pelicans.

Of course, a pelican may simply be harder to draw than a cat. A lab could train on pelicans and still not push them past the easy animals, so this ranking alone can’t rule that out. I’ll adjust for difficulty in Evidence #4.

Evidence #3: Labs are not better at drawing bicycles

Bicycles fare even worse. They sit second from last, in a near-tie with planes, which come in last:

If labs were training on the benchmark, you’d expect bicycles near the top of this ranking. They’re not. However, the same caveat applies here. A bicycle is harder to draw than a skateboard: it needs two matching wheels, a frame that reaches both axles, handlebars, a seat, and pedals. The judge flags a missing or disconnected one of those on 2/3 of the bicycle images. You can train on bicycle images and still not do a great job relative to simpler vehicles.

One note on the plane, though: I should’ve picked “airplane” instead of “plane” because models often read it geometrically. They drew the animal standing on a flat surface instead of flying an aircraft. The plane is the only vehicle where the feature extractor sometimes found no vehicle at all (25 of 168 images, against zero for the other five), and 20% of plane images scored a 1 or 2 on the vehicle rating, against 5% for bicycles and none at all for boats, scooters, or skateboards.

Evidence #4: Labs are not better at drawing pelicans on bicycles, even adjusting for difficulty

Put the two together and the “pelican on a bicycle” ends up near the bottom of the ranking, at #42 of 48:

But again, some combinations might be just harder to draw than others.

To account for that, I fit a fixed-effects regression on all 1,008 images: score ~ lab + animal × vehicle, plus per-lab interaction terms for pelican, bicycle, and the pelican-bicycle cell, with robust standard errors. The animal × vehicle terms absorb the inherent difficulty of all 48 combinations. The interactions measure each lab’s benchmark-specific boost relative to the average lab, with confidence intervals.

The results:

• Every per-lab pelican effect (the lab’s boost on pelicans across all six vehicles) lands between -0.11 and +0.14 judge points, and none comes close to significance (smallest p = 0.25).

• The per-lab bicycle effects (the lab’s boost on bicycles across all eight animals) run from Grok 4.5 at -0.18 (p=0.11) to Gemini 3.5 Flash at +0.27 (p=0.022). Only Gemini clears p < 0.05, and the seven point in both directions.

• No pelican-bicycle cell effect (the extra boost on the specific combination, on top of the lab’s pelican and bicycle effects) clears p < 0.05. The largest positive is GLM-5.2 at +0.35 (p=0.12), which is the one I mentioned earlier. It’s the closest thing to a signal in this experiment, but still within chance.

Here are the full per-lab estimates. A pelicanmaxxing lab would show dots to the right of the zero line across its whole row:

Every pelican interval and every cell interval contains zero. Exactly one doesn’t: *Gemini 3.5 Flash* in the bicycle column. But with 21 tests at p < 0.05, chance alone predicts about one false positive (21 × 0.05 ≈ 1.05), and one is exactly what came up. It also doesn’t survive a multiple-comparisons correction: the Bonferroni threshold across the 21 tests is 0.05/21 ≈ 0.002, and its p-value is 0.022. The full table of estimates and p-values is in the repo.

But these intervals are wide, about ±0.6 judge points on average. Any boost smaller than that won’t be captured by this test.

Evidence #5: The pelican-bicycle scenes don’t look memorized

Some have suggested that the pelican on a bicycle looks like a memorized composition, pointing to recurring patterns such as the pelican always facing right, or recurring elements like a sun or a scarf. So I wanted to know if this was true.

Direction: All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that.

However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest:

vehicle

left

right

ambiguous

scooter

9%

83%

8%

bicycle

11%

81%

8%

skateboard

19%

60%

21%

plane

21%

58%

21%

unicycle

22%

45%

33%

boat

33%

35%

32%

Pelicans are also among the animals that tend to face right:

animal

left

right

ambiguous

antelope

21%

78%

1%

pelican

22%

78%

0%

heron

22%

77%

1%

whale

34%

65%

1%

flamingo

36%

64%

0%

otter

8%

45%

47%

cat

8%

40%

52%

raccoon

3%

36%

61%

It’s hard to draw a pelican or a bicycle facing the viewer, so models almost always draw them from the side, facing left or right. That’s why so few of their images are ambiguous. Other combinations also come close to unanimous: antelope on a scooter and pelican on a scooter land at 20 of 21, and heron on a bicycle at 19 of 21. So 21 out of 21 doesn’t seem like an outlier.

Scene elements: I let the extractor name any element it saw in the image. These are the counts:

A memorized scene would show up as the same set of elements recurring picture after picture. I went looking for that, and found some combinations do tend to produce the same elements every time. Every single flamingo on a boat has a sun in it. Otters on planes wear scarves 38% of the time. Cats on bicycles get a basket 38% of the time.

The pelican on a bicycle doesn’t seem to have anything particularly different about it. It just has some elements that appear more frequently, like every other animal-vehicle combination.

Limitations

• Using a single LLM judge for scoring. Every score here comes from one model, GPT-5.6 Luna, looking at one image at a time. I didn’t do much alignment and didn’t check how often it agrees with itself on a re-run. If a model just can’t judge a drawing reliably, none of the numbers above mean much. The judge is also from the same family as one of the contestants, GPT-5.6 Terra. However, every lab draws all 48 combinations, so a judge that happens to like one lab’s style lifts that lab’s whole grid at once. But that doesn’t change the results because this analysis only cares about the within-lab differences.

• SVGmaxxing. A lab that optimized SVG generation as a whole (or a subset such as animals on vehicles) rises on every cell at once and looks identical to a lab that’s just good. Some labs, such as Google/DeepMind, openly do this. This experiment can’t detect that.

• Limited budget. The whole experiment ran on roughly $80 of API credits. That capped it at 3 samples per cell, a single judge, and 7 models. This also prevented me from iterating too much on the prompts and pipeline, as with the “plane” vs. “airplane” case.

Conclusion

Sorry, HN haters, but there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner.

Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict. GLM-5.2 comes closest: it has the largest boost on the exact pelican-bicycle cell, and its first pelican-on-bicycle sample caught my eye. But the effect is small and not significant, so I wouldn’t put too much weight on it.

The other thing that stands out is direction in the scene composition. All 21 pelican-bicycle images face right, the only combination in the grid where every image agrees. But it doesn’t seem that strange. Facing right is the norm across the experiment. Three other combinations land at 90% or above, and with 48 of them, I’m not surprised one reached 21 out of 21.

The more plausible story is SVGmaxxing like Google/DeepMind does. Other labs might be doing it more quietly. Sadly, this experiment can’t say who’s doing it. But at least you can sleep tonight knowing that AI labs are not producing terabytes of pelicans on bicycles just to trick Simon Willison.

If you want to look at the data yourself, the full pipeline is in the repo.

Footnotes

• the practice of optimizing AI models to achieve high scores on popular benchmarks.↩︎

Citation

BibTeX citation:

@online{castillo2026,
  author = {Castillo, Dylan},
  title = {Are {AI} Labs Pelicanmaxxing?},
  date = {2026-07-18},
  url = {https://dylancastillo.co/posts/pelicanmaxxing.html},
  langid = {en}
}

For attribution, please cite this work as:

Castillo, Dylan. 2026. “Are AI Labs Pelicanmaxxing?” July

1. https://dylancastillo.co/posts/pelicanmaxxing.html.