Surge AIがOpenAIの8,500問の数学問題データセットGSM8Kを構築した方法

BlogLeaderboardsWorkforceProductsResearchCareersContactLoginMenuCloseBack to BlogHow Surge AI Built OpenAI's GSM8K Dataset of 8,500 Math Problems

Imagine a world where AI helps us solve the Riemann hypothesis one day – producing not just the final answer, but also explanations of each mind-opening step. Could we teach it to do so? Just witness the magic already...

We recently created a dataset of 8,500 Grade School Math problems for OpenAI’s Reinforcement Learning team. The goal: to train language models like GPT-3 to solve natural language math problems and measure their reasoning ability.

Read the post by Karl Cobbe, Vineet Kosaraju, and John Schulman on OpenAI's blog! It’s also been adopted by many other research labs, including Google in their PaLM and Chain of Thought papers.

Dataset creation is a critical piece of AI, but it’s surprisingly underappreciated – ask most researchers, and they’ll have never inspected their datasets themselves! But how can you trust what you’re building when your inputs are junk? This is a real problem: for example, over 30% of Google’s GoEmotions dataset of Reddit comments is mislabeled…

At Surge AI, we care deeply about data, and we want everyone to share that same love. So we’re starting a “how we built it” series on dataset creation, starting with a deep dive into GSM8K.

How We Built It: Dataset Guidelines

First, what were the specifications behind the dataset? Here are the instructions that OpenAI created. A couple things to point out:

The instructions are fairly short! While long, detailed instructions are sometimes necessary – particularly when there’s very nuanced, unintuitive criteria – one of the benefits of using high-quality labelers is that you don’t need to write 20 pages of guidelines outlining obvious points.

Real-world examples are worth their weight in gold. We often see guidelines written by project managers who don’t understand the problems their product team is trying to solve; as a result, examples are nonexistent or armchair examples divorced from reality.

Your goal in this project is to write simple grade school level math word problems and their corresponding solutions.

All problems should contain 2-8 steps in their solution.

All calculations should be simple enough to do in your head without a calculator (e.g. 7*8, 36+110).

The answer should be a single integer value. Any units should be specified as part of the question (e.g. "How much money, in dollars, does Robert have?"). Simple decimal numbers (e.g. 3.25) can be part of the intermediate steps in the problem, but final answers should always be integers.

Writing "Jason has 8/2=4 apples" is preferable to writing "Jason has 4 apples", if you used the calculation 8/2 to determine how many apples Jason has.

Only use elementary arithmetic operations: +,-,*,/.

Don't reuse a problem setting. If you just wrote a problem about Samantha's trip to the zoo, don't write a second problem using that same premise.

Kendra has 3 more than twice as many berries as Sam. Sam has half as many berries as Martha. If Martha has 40 berries, how many berries does Kendra have?

Since Sam has half as many berries as Martha, he has 40/2 = 20 berries.

Since Kendra has 3 more than twice as many berries as Sam, she has 3+2*20 = 43 berries.

David has $300. He spent half of it on a new bike, and then he spent a third of what was left on a pair of sneakers. How much money did he have left?

David spent 300/2=150 on a new bike.

After buying the bike, David had 300-150=150 dollars left.

He spent 150/3=50 dollars on sneakers.

He had 150-50=100 dollars left.

Isaac has to buy some notebooks for the new school year. He buys 3 red notebooks and 2 yellow notebooks, for a total cost of $26. If the red notebooks cost $4 each, how much did the yellow notebooks cost each?

Let Y be the cost of the yellow notebooks.

Here’s what the labeling task looked like in our UI.

Building a Mathematical Labeling Team

Our first step in creating the dataset was building a team of mathematically proficient Surgers. While this project doesn’t require advanced mathematical knowledge, we found it helpful to leverage Surgers with math or STEM degrees:

They were less likely to make calculation errors. Even MIT Math PhDs inevitably make mistakes when solving middle school math problems, but having more experience makes it less likely.

They were naturally speedier. Some people can quickly calculate 81 / 3 in their heads; others can’t.

We wanted a diverse set of problems in order to capture the richness of mathematical reasoning, and experience helps.

Luckily, we do a lot of work creating advanced AI datasets – for example, creating datasets of Python programs to train coding assistants, or summarizing physics and chemistry papers – and so we already had hundreds of Surgers with technical degrees.

In order to further ensure that they could come up with well-designed math problems, everyone’s first 5 submissions