How Web Search Engines Retrieve Results

One of the keys to understanding how AI recommends products is its behavior of using web search to retrieve results, which then AI will corroborate the narrative truth in the retrieved web pages to arrive to certain conclusions.

This article explores the basic fundamentals of web search engines work – taking details from ChatGPT’s own explanation of its own integrated web search provider, and from the likes of Google’s publication of how ranking works in their search engine.

Crawling

One thing that’s important to distinguish is that crawling is purely about discovery and retrieval, not about deciding whether a page is “good.” Ranking comes much later.

Think of a crawler as answering one question:

“What URLs exist that I should fetch next?”

The aspects you listed are some of the biggest sources of discovery, but there are many others. Here’s a broader picture.

Crawling – Components

1. Following hyperlinks

This is the classic method.

Every page the crawler downloads contains links.

Page A
 ├── Page B
 ├── Page C
 └── Page D

Each discovered URL gets added to the crawl queue.

2. Reading XML sitemaps

Many websites publish XML sitemaps listing URLs they want search engines to know about.

These often include:

• product pages
• blog posts
• category pages
• images
• videos

A sitemap doesn’t guarantee crawling—it simply advertises pages.

3. Revisiting previously known pages

Google constantly asks:

“Has this page changed since last time?”

Some pages are revisited:

• every few minutes
• every few hours
• every few days
• every few months

depending on how often they tend to change.

4. Discovering newly published pages

This happens through many signals:

• internal links
• sitemaps
• feeds
• links from other websites
• pages already being crawled

The crawler notices URLs it has never seen before.

5. Following internal site structure

A crawler learns a site’s architecture.

For example:

Home
├── Products
│    ├── Product A
│    ├── Product B
│    └── Product C
├── Blog
│    ├── Article 1
│    └── Article 2

Internal navigation helps expose deeper pages.

6. Discovering URLs from redirects

Suppose:

example.com/pageA

returns

301 →
example.com/pageB

The crawler learns that page B exists and updates its understanding.

7. Canonical URL discovery

Pages sometimes declare:

This page's preferred version is:
example.com/product

instead of

example.com/product?color=red

The crawler records these relationships, although canonicalization also affects later indexing decisions.

8. Respecting robots directives

Before crawling extensively, crawlers check rules such as those in robots.txt to understand which areas of a site should or shouldn’t be crawled.

For example:

Allow:
/blog/
Disallow:
/admin/

These directives influence crawl behavior.

9. Managing crawl budget

Large websites can contain millions of URLs.

A crawler cannot fetch everything continuously, so it prioritizes which URLs to visit based on factors such as:

• server responsiveness
• historical update frequency
• importance inferred from the site’s structure
• available crawling resources

10. Detecting duplicate URLs

Many URLs can lead to nearly identical content:

product?id=15
product?id=15&utm_source=email
product?id=15&sort=popular

The crawler discovers all of them, even though later stages may consolidate them.

11. Understanding URL patterns

As crawlers explore, they recognize repeating structures like:

/blog/post-1
/blog/post-2
/blog/post-3

or

/products/123
/products/124
/products/125

This helps them efficiently discover additional URLs within a site.

12. Handling JavaScript-generated links

Modern websites often create navigation dynamically.

Crawlers may render JavaScript (subject to resources and limitations) to discover links that aren’t present in the initial HTML.

13. Learning site update patterns

Over time, a crawler builds expectations.

For example:

• News homepage → changes every few minutes.
• Daily blog → changes once a day.
• Documentation → changes infrequently.

These observations influence how often pages are revisited.

14. Monitoring HTTP responses

Each crawl attempt provides signals such as:

• 200 OK (page available)
• 301/302 (redirect)
• 404 Not Found
• 410 Gone
• 500 (server error)

These responses help determine future crawl behavior.

Crawling – Conceptualization

You can think of crawling as a continuous loop:

Known URLs
      │
      ▼
Fetch page
      │
      ▼
Extract everything discoverable
      │
      ├── New hyperlinks
      ├── Sitemap references
      ├── Redirect targets
      ├── Canonical hints
      ├── HTTP responses
      ├── JavaScript-generated links
      └── Metadata about the page
      │
      ▼
Add eligible URLs to crawl queue
      │
      ▼
Repeat forever

This is why it’s helpful to think of crawling as building and maintaining the web graph—a constantly updated map of pages and the connections between them. Crawling’s primary job is to discover and refresh that map. Once pages have been fetched, the indexing system decides what information to store, and the ranking system later decides which indexed pages are most relevant for a particular search query.

Indexing

This is actually where things get really interesting. If crawling is “finding pages,” then indexing is “understanding and organizing pages.”

A useful analogy is a library:

• Crawling = the librarian receives a new book.
• Indexing = the librarian reads it, categorizes it, extracts key information, and decides whether to put it on the shelves.
• Ranking = when someone asks for a book, the librarian decides which books to recommend first.

Indexing isn’t just storing a webpage—it transforms it into a structured representation that can be searched efficiently.

Here are many of the major components of indexing.

Indexing – Components

1. Parse the HTML

Once a page is downloaded, the search engine begins extracting its contents.

It identifies things like:

• headings
• paragraphs
• links
• images
• videos
• metadata
• structured data
• titles
• descriptions

At this point, the raw HTML becomes meaningful pieces of information.

2. Extract the main content

Not everything on a webpage is equally important.

The indexer tries to separate:

• the main article or product description
• navigation menus
• headers and footers
• sidebars
• advertisements
• repeated template content

This helps focus on what the page is actually about.

3. Understand the language

The system identifies:

• English
• French
• Korean
• Japanese
• Spanish
• etc.

This helps match users with pages in the appropriate language.

4. Tokenization

The text is broken into searchable units, often called tokens.

For example:

“The red running shoes are waterproof.”

becomes something like:

• the
• red
• running
• shoes
• waterproof

These tokens become building blocks for later retrieval.

5. Normalize words

Different forms of the same idea may be connected.

For example:

• run
• running
• runs

or

• organize
• organizing
• organization

The search engine often relates these variations so they can be matched more effectively.

6. Identify entities

Rather than seeing only words, modern search engines recognize real-world things.

For example:

• people
• companies
• cities
• products
• organizations
• diseases
• technologies

This helps distinguish, for instance, “Apple” the company from the fruit based on context.

7. Understand topics

The indexer builds an understanding of what the page is about.

For example, a page may relate to:

• hiking
• camping
• backpacks
• outdoor gear

Even if those exact words aren’t all used repeatedly, the overall topic can still be inferred.

8. Analyze images

Images aren’t ignored.

The indexer may use signals such as:

• alt text
• surrounding text
• filenames
• captions
• image analysis

to understand what an image depicts.

9. Analyze videos

Similarly, videos can contribute information through:

• titles
• descriptions
• transcripts
• captions
• surrounding page content

10. Read structured data

Many websites provide explicit machine-readable information.

For example:

• products
• recipes
• events
• reviews
• organizations
• FAQs

This helps the search engine understand specific facts without having to infer everything from prose.

11. Detect duplicates

The web contains many identical or nearly identical pages.

For example:

• printer-friendly versions
• syndicated articles
• tracking-parameter URLs
• mirrored content

The indexer tries to identify these relationships and avoid storing unnecessary duplicates.

12. Canonicalization

If multiple URLs represent essentially the same page, the indexer attempts to choose a preferred version (often called the canonical version) for the index.

13. Evaluate quality signals

The search engine may assess characteristics such as:

• originality
• completeness
• clarity
• trustworthiness
• spam indicators

These assessments don’t necessarily determine rankings directly, but they can influence whether and how a page is indexed.

14. Build the inverted index

This is one of the most important steps.

Instead of storing:

Page → Words

the search engine builds a structure like:

Word → Pages

For example:

"backpack"
→ Page A
→ Page C
→ Page D
→ Page Z

This makes searching incredibly fast because the engine can immediately retrieve pages associated with a query term.

15. Store semantic representations

Modern search engines don’t rely only on keywords.

They also create richer representations of meaning, allowing them to understand relationships between concepts and retrieve relevant pages even when the exact wording differs.

Indexing – Conceptualization

You can think of indexing as a pipeline:

Downloaded webpage
        │
        ▼
Parse HTML
        │
        ▼
Extract main content
        │
        ▼
Understand language
        │
        ▼
Tokenize text
        │
        ▼
Recognize entities and topics
        │
        ▼
Analyze images, videos, and structured data
        │
        ▼
Detect duplicates and choose canonical pages
        │
        ▼
Build searchable indexes and semantic representations
        │
        ▼
Ready for retrieval during search

Connecting this to your Narr Theory work

This is where your framework starts to intersect with search in an interesting way.

A search engine isn’t indexing just words—it’s trying to index meaning. It transforms an unstructured webpage into a structured set of concepts, entities, relationships, and signals that can later be retrieved.

That’s why the transition from crawling to indexing is so significant:

• Crawling asks: “What pages exist?”
• Indexing asks: “What does each page actually mean, and how should I represent that meaning so I can find it later?”
• Ranking asks: “Given a user’s query, which indexed pages are the most relevant and useful to show first?”

This distinction becomes especially important when thinking about modern AI-assisted search, where systems increasingly retrieve information based on semantic understanding rather than exact keyword matches.

Ranking

Now we get to the stage that most people think is search.

Ironically, it’s actually the last major stage.

The pipeline looks like this:

The Web
    │
    ▼
Crawling
    │
    ▼
Indexing
    │
    ▼
User types a query
    │
    ▼
Ranking
    │
    ▼
Search Results

Notice something important:

Ranking doesn’t happen until a user asks a question.

Everything before ranking happens before anyone searches.

The Goal of Ranking

When someone searches:

“best running shoes for flat feet”

Google isn’t asking:

“What pages exist?”

It already knows.

It’s asking:

“Out of billions of indexed pages, which 10 are most likely to satisfy this user?”

That’s a completely different problem.

Stage 1 — Understand the query

Before looking at webpages, Google first analyzes the query itself.

For example:

best running shoes for flat feet

It tries to determine things like:

• What is the user trying to accomplish?
• Is this informational?
• Is this commercial?
• Is the user trying to buy something?
• Is there ambiguity?

This is essentially building a representation of the user’s intent.

Stage 2 — Retrieve candidate pages

Google does not rank every page on the internet.

That would be impossible.

Instead, it first retrieves a manageable candidate set.

Think of it like:

30 billion indexed pages
↓
40,000 candidates
↓
500 candidates
↓
100 candidates

This first filtering step is extremely fast.

The inverted index plays a major role here.

Stage 3 — Compute hundreds (or thousands) of signals

Now the real ranking begins.

Every candidate page receives scores based on many different signals.

Not one score.

Many.

Some examples:

Lexical relevance

Does the page actually discuss:

running shoes

or only mention it once?

Semantic relevance

Suppose the page says:

footwear for overpronation

without saying

flat feet

Modern search engines can understand that these concepts are closely related.

This is one reason search has become much more semantic over the years.

Topical coverage

Does the page discuss:

• cushioning
• stability
• arch support
• pronation
• durability

Or is it just a thin paragraph?

More comprehensive coverage can indicate that the page is a better match.

Freshness

Sometimes newer information matters.

For example:

Olympics
Election
Stock prices
AI models

Freshness can be very important.

For:

History of Ancient Rome

It usually matters much less.

Authority

Google asks questions like:

• Is this website generally reliable?
• Is it widely referenced?
• Does it have a strong reputation in this topic?

Historically, links have been an important signal, but authority is assessed through many signals rather than a single metric.

User experience

Examples include:

• mobile friendliness
• page speed
• security (HTTPS)
• intrusive pop-ups
• general usability

These factors can influence rankings, though they are typically not the strongest relevance signals.

Originality

If 100 websites copied the exact same article…

Which version should rank?

Search engines try to identify original or canonical sources where possible.

Entity relevance

Suppose someone searches:

Apple earnings

The search engine needs to determine whether the page is about:

• the company

or

• the fruit.

Entity recognition helps make that distinction.

Stage 4 — Combine signals

This is where many people imagine there is one magic formula.

In reality, it’s better to think of many signals contributing to an overall assessment.

Conceptually:

Relevance
Authority
Freshness
Originality
User Experience
Topic Coverage
Semantic Similarity
...

are all evaluated together.

The exact methods are far more sophisticated than a simple weighted sum and evolve continuously.

Stage 5 — Diversity

Google usually doesn’t want:

10 pages
from the exact same website

It often tries to provide diversity.

Likewise, for broad queries like:

python

you might see a mix of:

• programming language resources
• tutorials
• official documentation
• videos
• news
• community discussions

This helps satisfy different possible intents.

Stage 6 — Generate the Search Engine Results Page (SERP)

Finally Google decides:

#1
#2
#3
...
#10

and builds the search results page.

Ranking – Conceptualization

One misconception is that ranking is simply:

Query
↓
Sort pages

It’s much richer than that.

Think of it like:

User Query
      │
      ▼
Understand intent
      │
      ▼
Retrieve candidate pages
      │
      ▼
Score each page across many signals
      │
      ▼
Compare candidates
      │
      ▼
Diversify results
      │
      ▼
Return final rankings