AIPR – Phase 1 Hypothesis

Phase 1 Overview

Currently, we are entering into Phase 1 of AIPR Hypothesis testing after completing Phase 0 including observational case studies, validating and refining hypotheses against first-party SoA documents from OpenAI, and creating a cornerstone AIPR article including visual atlases to help articulate the idea into digestible terms.

Now, we are moving into reducing uncertainty in the strategic “so what” applications of AIPR, a territory in which observational data is significant in informing what practical, tangible action plan should be formed, as well as validating real world limitations to LLMs with respect to AI’s ability to search, reason, corroborate, and recommend products. We can also find further evidence on consumers’ underlying behavior that can lead to the final decision narrative point.

• Phase 0: Initial observational studies against first-party SoA validation
  • All findings are pointed to How AI Recommends Products article
    • ChatGPT specifics are pointed to How ChatGPT Recommends Products article.
• Phase 1: Refinement of key concepts informing “so what” applications

Here are the hypotheses Narr Theory will be testing for in Phase 1

---

Hypothesis Testing in Phase 1

We know the general ideas and mechanisms that LLMs tend to exhibit when finding evidence to corroborate its product recommendation.

The next step is to further examine the details in real-life case studies that will inform what is entirely necessary vs. what is clients can do with minimal input for maximum impact.

This step includes the following.

Recommendation Corroboration Scope

This is the question of “how much evidence does AI need to recommend a product?” What is the quantity of web pages?

What is the overall behavior?

We can arrive to one of these hypothesis, or some version of them:

• AI only uses the web pages that are visibly cited with the recommendation
• AI cites more web pages than the maximum of ~20 visible in chat, but does not is not visible either due to interface space limitation or simply because it’s not necessary to show more than the sources cited
• AI cites only visible web pages, but they have scanned through their domain/source more than just the single web page that is cited.
  • ChatGPT Case: I tried to find some information published by OpenAI on ChatGPT crawlers. ChatGPT cited sources that were illegitimate mirror copies of the original OpenAI documents, where my browser warned me not to enter. Later, ChatGPT confirmed that these were not original copies that should be trusted. This proof corroborates that ChatGPT does not always “check” everything when citing”
  • If AI systems are doing a “web search” – this would be inefficient to do every time. The crawlers of web search systems do the “scanning” and organizing – AI systems would simply scan through the web page results to find the relevant answers. AI would spend further energy in refining their web search to find better results, not necessarily scan through every web page that the domains publish.
• AI cites visible web pages, but their corroboration comes from a more deeply rooted method to their internal knowledge model, rather than just information from the web search. I imagine some version of this can be true because is constantly reasoning and pattern recognizing, so if it found information from web search that wildly contradicted some kind of base truth, it might be able to flag so. Simultaneously, ChatGPT has not always caught its contradictions in some cases.

Purpose & Implication

The purpose of this hypothesis is to understand:

• What variables need to change in order for a client product to be recommended?
• What is the exact process that AI uses to corroborate evidence?
  • This will inform the type of narrative intelligence tools needed to not only help client product become the top recommendation, but to defend this territory for long-term.

---

Source of Authority (SoA) Quality

In Phase 0, we observed that for some LLMs like ChatGPT, the free model cited different sources than paid versions.

This can be due to the model’s advanced overall reasoning protocol, which can be more sophisticated on certain paid plans.

Nonetheless, it is important to repeat observational case studies at a higher scale to particularly see how the free version of LLM responds first and foremost, and compare against higher paid models.

Generally speaking, the mass-scale adoption of AI at the consumer level is in the free version. Hence, this part is very important.

Purpose & Implication

• Having reduced uncertainty in this area can better inform the action plan for which SoAs to consider strategically at a mass-consumer level.
• Certain paid model hypothesis can be useful in determining the scope of reasoning that LLMs are capable of covering, and inform the AIPR strategy catered to certain user base that may be subscribed to such models. This could particularly be useful for emerging LLMs like Claude, however, also important to keep in mind that consumers will become more adapted to using AI, and may be open to adapting multiple AI systems for asking questions.

---

External Knowledge Search Behavior & Boundary

Certain LLMs will convert a decision narrative into a web search.

This converts the user’s initial narrative into keywords that are more web-search friendly.

This data can inform SoAs on how to create content that is AIPR-friendly for long-term.

As for the boundary, this will be an ongoing investigation into how far AI’ internal knowledge model expands, and exactly what type of information and when AI might be triggered to perform external search for what it “does not know”.

---

Narrative & Semantic Association Pattern

In order to inform a client’s action plan, it is important to observe the real-life narrative & semantic association patterns from the decision narrative and what the cited web pages are showing.

If the associations are more direct, it is possible that AI’ corroboration scope is rather narrow, though speculative still.

If the associations are not as direct (not as many clear associations), we can lean more towards the hypothesis that the product recommendation is closer to the internal knowledge model than what it shows at face value.

Purpose & Implication

This will also contribute significantly in advising the AIPR strategy for clients in a way that requires minimal effort for maximized output.

---