The great search divide: How AI and traditional web searches differ

As anyone who uses the internet will know, the way we find information has fundamentally changed. For the last three decades, search engines have delivered ranked lists of links in response to our queries, and it was our job to search through them to find what we wanted. Now, major search engines use generative AI tools to deliver a single coherent answer, often embedded with a few links. But how does this approach compare with the traditional method? In a comprehensive new study, scientists compared these two approaches to see what we are gaining and losing.

Comparing AI with traditional searches

To figure this out, researchers from Ruhr University Bochum and the Max Planck Institute for Software Systems compared traditional Google Search with four generative search engines: Google AI Overview (AIO), Gemini, GPT-4o-Search and GPT-4o with Search Tool. The team ran thousands of queries covering six main areas, including general knowledge, politics, science and shopping.

Then they made a detailed comparison of the two search styles based on three key metrics. First, they analyzed source diversity by checking the websites AI used against traditional search’s top links. Second, they measured knowledge reliance to see how much AI relied on its own internal memory rather than searching the web for fresh information.

Then they examined conceptual coverage to determine whether each AI’s final answers covered a broader range of ideas than traditional top search results. Finally, they reran the experiment two months later to see how quickly AI sources and answers changed over time.

Clear differences emerge

As the team outlined in their paper published on the arXiv preprint server, there are clear differences between the two. AI casts a wider net, pulling information from a more diverse set of websites than traditional searches, but the links it used were often not among the top search results. However, just because AI uses more sources doesn’t mean its answers are more comprehensive. In fact, that often wasn’t the case.

“Differences in source selection and internal knowledge use can subtly shift which perspectives and facts users are exposed to, even when overall topic coverage appears similar,” wrote the study authors.

AI answers also tended to be less stable over time, changing significantly after two months.

The researchers also discovered differences between the AI models. For example, GPT-4o with Search Tool relied on its internal memory, while Google AI Overview and Gemini constantly pulled information from a larger, fresh set of external sites.

Overall, the team did not conclude whether one type of search was better than the other, although there is a clear trade-off when using AI. Users receive a diverse, summarized answer, but lose the reliable sources and consistent results they get from traditional searches. Most notably, the authors emphasized the need for new benchmarks and standards to properly evaluate answers returned by AI.

© 2025 Science X Network