Google検索は劣化しているのか？2022年の検索品質を測定

BlogLeaderboardsWorkforceProductsResearchCareersContactLoginMenuCloseBack to BlogIs Google Search Deteriorating? Measuring Google's Search Quality in 2022

There's been a lot of discussion on Hacker News recently about the quality of Google Search, and whether it’s deteriorated.

Anecdotally, I’ve noticed this myself. For example, a friend and I were chatting about Databricks last week, and we searched "databricks series b valuation" in order to figure out what their Series B valuation was. Unfortunately, Google doesn't understand what "series b" means (it seems to confuse "b" and "billions"), so the first search result is irrelevant. I don't even get any information about their Series B until below the fold!

In contrast, Bing's search results page is much better. Information about the Series B is right in the expanded first search result (it doesn’t contain valuation information, but that’s expected because the Series B valuation isn’t public), and the right-hand sidebar is also quite helpful.

So why might Google Search be deteriorating? A couple plausible reasons:

Google has been prioritizing short-term ad revenue over search quality. Interestingly, Google has a well-known paper explaining why focusing on the long-term is better for users and their business!

Information is moving beyond traditional webpages. These days, content often lives on Twitter, Facebook, YouTube, Medium, Reddit, etc. The Internet today is very different from the Internet that Google Search was born in!

Historically, Google Search contained little ML. From what I've heard, this has changed in recent years, due to changes in leadership and improvements in AI. Is it possible that ML is inadvertently making quality worse?

Crucially, measuring search quality is a very difficult problem. Naively, for example, you might think that a better search algorithm leads to more clicks: when I search for "databricks series b valuation", you might think that I want to click on a website containing the information. But ideally I might never click on a website at all! The ideal SERP may be one that displays the valuation at the top of SERP itself. What’s more, clicking is often a bad sign: I might click on Google's first search result about the Series H, because I mistakenly think it contains information about the Series B too.

So is Google Search actually deteriorating? How good is it these days, and how does it compare to its competitors?

I used to work on Search Measurement at YouTube, Twitter, and Microsoft, and it's one of the major customer use cases Surge AI provides. So let’s play around and analyze just how good Google Search is in 2022!

First of all, how do you even measure the quality of a search engine in a rigorous way? As mentioned above, it's very difficult to measure search quality using traditional metrics.

Clicks aren't necessarily something you want to optimize for, for the reasons above.

Neither is time spent searching: is a short session a good thing (perhaps you found your answer immediately) or a bad thing (the search results were so bad you quickly gave up)?

Perhaps you can measure reformulations: if your initial search query failed, you may rewrite your query and try again, so an increase in reformulations could be viewed as a bad thing. But many people will give up instead of reformulating, and how do you tell whether a query is a reformulation anyways?

Maybe long-term metrics are the solution. Happy Google Searchers will continue searching on Google. But running long A/B tests is painful if you want to quickly iterate, and even if you're unhappy with Google, is it likely that you'll switch to a competitor?

What's a search engine to do? One alternative that Google pioneered is the idea of human evaluation: in order to measure search quality, why don't you simply ask human raters how good your search results are? In other words, you give human raters a set of search queries and search results, and ask them to rate how well each search result satisfies the intent behind the query.

There are many nuances to this approach. For example: how do raters know the intent behind the query?, do you rate search results individually or the SERP as a whole?, where do you get these raters? But overall, it's my favored approach as well.

So in order to measure the quality of Google Search, here was my process:

I leveraged a set of human raters from Surge AI. (We’re a new kind of data labeling platform with high-skill human raters, built with quality as our top focus — whether you need savvy social media users tuned into US politics to help clean up the Internet, computer science graduates to train AI to answer how neural networks work (many even from Ivy League schools!), high school teachers to train educational question-answering AI systems for students, or Fortnite players versed in platform jargon to build gaming NLP.)

In order to get search queri