SYLVIE DOUGLIS, BYLINE: NPR.
(SOUNDBITE OF DROP ELECTRIC SONG, "WAKING UP TO THE FIRE")
DARIAN WOODS, HOST:
A few years ago, Matti Wiegmann got a new laptop. Matti is working on his Ph.D. at Bauhaus University, Weimar, in Germany, and he wanted a new messenger bag so he could walk around with it. Matti searched for bags on Google, but he wasn't happy with the results.
MATTI WIEGMANN: All of the pages I stumbled on, they listed lots of backpacks, and they were describing them, but I felt very dissatisfied and disinformed after visiting them.
WAILIN WONG, HOST:
These were the kinds of spammy websites that had a ton of ads and links, but not really any insights.
WIEGMANN: A somewhat new and frustrating experience because we were used to just going on Google, clicking the first link, and it would work.
WOODS: And that got Matti asking, is Google getting worse?
WONG: Oh, a provocative question.
WOODS: Yes.
WONG: Of course, there's certainly a lot of clutter now, a ton of sponsored links, that new AI-generated box for some searches, which sometimes works, sometimes doesn't.
WOODS: Sometimes helpful, sometimes wrong, always confident.
WONG: (Laughter) That's how I approach life, Darian.
(LAUGHTER)
WONG: I often find myself just scrolling past that stuff to get to the - what I consider, like, the real search results at the bottom.
WOODS: And we, too, want to know about the real stuff, the Google search results.
(SOUNDBITE OF MUSIC)
WOODS: This is THE INDICATOR FROM PLANET MONEY. I'm Darian Woods.
WONG: And I'm Wailin Wong. Today on the show, testing Google search. There are many anecdotal complaints about Google not being what it used to be. Today, we test that claim and we bring it to Google itself.
(SOUNDBITE OF MUSIC)
WOODS: One way of figuring out the usefulness of Google over time is to look at product reviews. When you search for reviews for a new messenger bag, you could categorize the types of websites you might get. And on one side, there are websites dedicated to careful reviews. Maybe they even have a bag specialist going through the pros and cons. On the other side, is a category that has less useful types of websites.
WIEGMANN: A content farm, you could call it. Not very useful content, and also not very trustworthy if you looked at this into detail.
WOODS: You know the types, Wailin.
WONG: Oh, yes, I do. And on that content farm, we had some spam, E-I-E-I-O.
WOODS: SEO.
WONG: (Laughter) Our messenger bag shopper, Matti Wiegmann, is a Ph.D. candidate in computer science. And for his research, he decided to investigate. Matti and his colleagues collected the top 20 search results for more than 7,000 product review searches and tracked these every two weeks for over a year between 2022 and 2023. They did this for Google, but also for other search engines like Bing and DuckDuckGo.
WIEGMANN: So we ended up with hundreds of thousands of search results, and then measured properties of these pages, how many words are on the page, how many images are on this page, how many links, how long are the links? And so on. How many headlines are there?
WOODS: Now, in Google's favor, Matti found that out of the commercial search engines they looked at, Google performed the best at filtering out spammy websites without much content. That said, Matti found that certain legacy magazine websites have sections that churn out a lot of product review pages every day just to get advertising and link commission money. And these websites tend to get ranked highly on Google.
WIEGMANN: They put out so much content over such a broad area. And all of them is what we would consider reasonably low quality, right? They are very strongly designed so that Google ranks them highly. They're also very shallow, and they're very dissatisfying if you really want to have - get an informed decision.
WONG: Matti and his colleagues found that Google would issue periodic updates to their search algorithms, and that would push low-quality review websites off the results page. But then, as these websites learned the new ways to game the search engine, they would creep back up again.
WIEGMANN: This pattern were repeating somewhere or another.
WOODS: It sounds like a constant game of cat and mouse.
WIEGMANN: Yes, and this is exactly what we call it in our paper, too. It seems there is an ongoing struggle between the content publishers that try to get ranked very highly and Google themselves, who try to update their search engine.
WOODS: So what does Google itself say about this cat-and-mouse game? Pandu Nayak is the chief scientist for search at Google.
PANDU NAYAK: There's certainly an adversarial component to it. So, yes.
WONG: Pandu confirms there is this constant updating needed to keep search results relevant.
NAYAK: This has been sort of challenging since the beginning of search. And when Google started, they introduced a specific algorithm called page rank. It's about using links as sort of votes of confidence in a particular site and then aggregating it in a particular way that gave a very nice signal of reliability.
WOODS: Is Google getting worse over time?
NAYAK: Actually, all our measurements say that it's not getting worse over time. We launched, you know, thousands of changes every year to ensure that the changes we are making are actually to the benefit of users.
WOODS: Since Matti's paper, Pandu says Google has added all kinds of additional measures of whether a page is higher or low quality.
NAYAK: We complement this kind of algorithmic work with a set of spam policies that allows us to take action against sites that are trying to manipulate us. So there's sort of this comprehensive effort. It's a fairly significant effort that allows us to do these things.
WONG: Google had a big update in March aimed at improving search quality.
WOODS: By the way, if you're super interested in the inner workings of search, internal documents about Google actually leaked recently. And this has been one piece in all the scrutiny that Google has been under. And among all this, they've actually read Matti's paper, and Pandu points out that Matti's paper is just focused on product reviews, not all types of search.
NAYAK: There's just this tremendous variety of queries we get, and product reviews is just one of them.
WONG: Pandu says every day, 15% of search queries are something Google's never seen before. And so Google is always updating its search tools, even using artificial intelligence.
WOODS: Most noticeable to users these days is Google's AI overview. It's this box at the top of some searches which generates answers to users' queries using a large language model, kind of like ChatGPT.
WONG: Now, like all large language models, it's not always accurate, and this has occasionally spawned some unintentionally hilarious answers.
WOODS: Yeah, my favorite was an AI overview suggesting users add glue to a pizza to stop the cheese from falling off.
WONG: How do you know until you've tried it, Darian?
WOODS: It probably works. It may have some side effects, though.
WONG: Now, to be fair to Google, this AI overview box is marked as experimental, but it hasn't been great PR. Pandu says the algorithm of Google's core search engine has been using AI-like deep learning since at least 2015.
WOODS: The downside is, though, generative AI makes it easier to create a kind of junk website.
NAYAK: Well, I mean, I think that's certainly true for any technology that it can have good and bad uses, right? Now, it's possible that with generative AI, the scale of the problem might go up in the future. But once again, I think we are sort of ready and willing to engage with that problem.
WOODS: It's like AI has heightened this cat-and-mouse problem.
WONG: Yeah, now it's a jaguar versus capybara problem.
WOODS: (Laughter) Big cats and the world's biggest rodent.
WONG: (Laughter) With that said, Matti Wiegmann, the computer scientist, is feeling optimistic about the future of search, if only because of all this extra attention from academics and governments into how big tech operates.
WOODS: Plus, for all his scrutiny of Google, the search engine did eventually help him find his messenger bag.
WIEGMANN: I found a very interesting indie site about two pages down on Google that actually had, like, lots of test videos.
WONG: For all that fuss, he ended up settling on a plain, black messenger bag.
WOODS: It was the world's greatest deep dive just to buy a backpack.
WONG: You know what? I respect that research.
WOODS: Yeah.
WONG: That's what I would have done (laughter).
(SOUNDBITE OF MUSIC)
WOODS: Google, we should mention, is a sponsor of NPR. This episode was produced by Julia Ritchey with engineering by Neal Rauch. It was fact-checked by Sierra Juarez. Kate Concannon edits the show, and THE INDICATOR is a production of NPR.
(SOUNDBITE OF MUSIC)