Synthetic vs Real Prompts in GEO Monitoring: How Real Chat Data Changes the Sources LLMs Cite

Last updated: 2026-10-14 · WaniWani Research note

Synthetic prompts are the questions a GEO monitoring setup generates with an LLM, usually from keywords and forum threads. Real prompts are what customers type into an AI assistant. When we compared the two on the same LLMs, using conversations from a live AI app, the LLMs cited a different mix of sources: the main official source rose from 29% to 47% of runs, policy research fell from 17% to 8%, and one publisher dropped out entirely.

That result matters for any brand that measures its AI visibility, whatever it sells. If the tracked prompts don’t match how customers talk, the citation report describes a market that doesn’t exist. Earlier studies tested this with survey answers, such as Spyglasses’ re-run of 143 human-written prompts for a single headphones query. Our data comes from production conversations.

About the study

The data comes from the AI app of a US health insurance brand. Health insurance was the test case, and the findings below are the ones that don’t depend on it. These findings describe how people talk to AI assistants and how LLMs pick sources, so they apply to any product that people research in a conversation, from insurance and banking to travel, telecoms or software. All data is anonymized. We don’t name the brand, and every website is grouped by type. This is an observational research note. It was not pre-registered, unlike WaniWani Research Vol. 1.

Key findings

• The source mix shifts. The main official source goes from 29% to 47% of runs on real prompts, and policy research from 17% to 8%.
• 2% of real openers end with a question mark, against 77% of synthetic prompts.
• 5% of real users use industry jargon, against 32% of synthetic prompts.
• Two life events start 65% of conversations. The synthetic set under-tracked the biggest one and over-tracked the second.
• Wording alone moves sources. Jargon questions cite official sources in 63% of runs, and messy questions in 33%. Forums go from 0% to 17%.
• The brand’s own content site drops from 13% to 8% of citations as wording moves from jargon to the way people type.

How we ran the comparison

We compared two prompt sets on the same LLMs and recorded which sites each answer cited. The first set was the AI-generated prompts the brand had tracked so far. The second came from real conversations in the brand’s AI app. We grouped openers by the situation behind them and cross-checked each group against the situation the app’s agent captured during the session. The figures in this post are the share of LLM runs that cited each source type. Earlier GEO research, such as the Princeton GEO study (Aggarwal et al., KDD 2024), measured how changes to content move citations. This comparison measures how changes to the query move them.

Do real users phrase questions the way synthetic prompts do?

No. Real users write in almost the opposite register to synthetic prompts. Only 2% of real openers are a clean question ending in “?”, and 19% are all lowercase with typos or run-on sentences. Real users rarely use the industry’s own terms (5% vs 32%). They describe what happened to them and leave the question to the assistant.

Here is the same intent written both ways, taken from the study. Swap the product and the pattern is the same: the synthetic prompt names a category, and the real user describes a situation.

Real messages also carry specifics that synthetic sets leave out: exact ages, the price they pay today, the provider they had, a hard deadline. Those details tell the LLM who is asking, and they change the answer.

Do synthetic prompts match what customers ask about?

No. Synthetic sets give each topic roughly equal weight, but real demand is concentrated. In the study, two life events started 65% of all conversations: 43% for the biggest and 22% for the second. Every other situation was 8% or less, and most were around 1 to 2%.

The synthetic set did not follow this curve. It under-tracked the most common event, over-tracked the second, and gave its wording experiments equal weight per intent however often each came up. A visibility score built that way over-reports the topics customers rarely raise.

Do real prompts change which sources LLMs cite?

Yes. Real prompts sent LLMs to a different set of sources. The main official source rose from 29% to 47% of runs. Sources that help someone in the middle of a problem gained too, such as the government body for that situation and advocacy groups. Policy research fell from 17% to 8%, and one personal finance publisher fell from 9% to zero.

The pattern is general. Synthetic prompts read like research questions, so LLMs answer them with explainers and policy analysis. Real prompts read like problems, so LLMs answer them with official help and practical sources. A brand that monitors only the synthetic set would chase citations that real customers never trigger.

Does the wording of a prompt change the sources an LLM cites?

Yes, even when the intent stays the same. We wrote four intents in three styles each. With jargon, 63% of runs cited the main official source and 21% cited the regulator’s own rules. With messy, real-world wording, official citations fell to 33% and forums rose from 0% to 17%.

The three styles:

• Jargon: industry terms in a tidy question.
• Plain: a simple everyday question.
• Messy: how people type in a chat, with lowercase, typos and their situation up front.

The cleaner and more technical the question, the more the LLM relies on official and regulatory sources. The messier and more personal it is, the more it relies on community sources. The brand’s own content does best on jargon (13%) and fades to 8% on the way people type, because most brand content answers clean, informational questions. To show up for real queries, a brand needs pages written in situation language and a presence in the communities LLMs trust for messy questions.

How to build GEO monitoring on real chat data

To build GEO monitoring on real chat data, take the openers from your own AI app or chat logs, group them by intent, and track them in your customers’ words. Weight each intent by how often it comes up, and keep a small branded set as a control. In the study, the rebuilt set has 94 prompts, and 82 of them come from real chat language.

1. Rewrite the core intents in real language. Turn each category question into the situation a customer describes. In the study this was 24 prompts.
2. Run phrasing experiments. Write a few intents in jargon, plain and messy form, so the wording effect stays measured over time. The study used four intents, so 12 prompts.
3. Track verbatim variants. Take real questions and track each in several actual user wordings, to see how far sources move inside one intent. The study used nine questions in three wordings each.
4. Weight by demand. Add real openers in proportion to how often each situation happens. After this step the biggest event made up 43% of the new prompts, the same share as in real conversations.
5. Keep a control. Freeze a small branded and reputation set, so you can compare the real set with the old baseline. The study kept 12 prompts.
6. Tag every prompt. Tag by topic, set, experiment, priority and branded vs unbranded. Filter by topic to find gaps against demand, and combine top priority with a topic to decide where new content goes first.

Limitations

• One brand, one vertical. The data comes from a single AI app. The direction of each finding is general, but the exact percentages will differ by industry and by brand.
• Observational, not pre-registered. We compared prompt sets that already existed. We did not fix hypotheses or sample sizes before the data came in.
• Small samples in the phrasing experiment. Each style cell rests on a small number of runs, so read the direction of the change, not the exact points.
• Share of runs, not share of sources. Each figure is the share of LLM runs that cited a source type. One run can cite several types, so the columns don’t add up to 100%.

What this study does not cover

• Conversion. We measured which sources LLMs cite. We did not measure whether a citation leads to a visit or a sale.
• Per-model differences. This note reports pooled figures and does not break them out by model.
• Content interventions. We did not change the brand’s content to test whether situation-language pages win more citations. That is the next test.

What this means for brands in any industry

Prompts written by a model show what a model thinks your customers ask. They miss the panic, the typos and the specifics, and LLMs answer that difference with different sources. If you run an AI app or a site chat, your conversations already hold the real phrasing and the real frequency of each need. That data is the right input for GEO monitoring.

WaniWani builds GEO tracking from your own real conversations, weights it by demand, and shows which sources win each kind of question. For more on what our research shows about LLM citations, read WaniWani Research Vol. 1, the 33-day experiment that took an app from 0 to 42% citation rate. For the conversion side, see why AI leads convert 3-5x better than search.

Frequently asked questions

What is a synthetic prompt in GEO monitoring?

A synthetic prompt is a question that an LLM or a GEO platform generates to stand in for a customer query, usually from keywords, a website or forum threads. Teams track how often their brand appears in the answers. In our data, synthetic prompts were tidier than real messages: 77% ended in a question mark, against 2% of real openers.

Are synthetic prompts accurate for measuring AI visibility?

Not on their own. When we compared synthetic prompts with real chat language from a live AI app, the main official source went from 29% to 47% of runs and policy research went from 17% to 8%. Synthetic prompts measure how LLMs answer tidy, jargon-heavy questions. Most customers don’t ask that kind of question, so the visibility score can point you at the wrong sources.

Where can I get real prompts for GEO tracking?

The best source is your own conversation data: the first messages in your AI app, site chat or support chat. Group them by intent, keep the original wording, and weight each intent by frequency. Search queries and survey answers are a weaker fallback, because they are shorter and tidier than what people type into an assistant.

Why do messy prompts get more forum citations?

In our test, messy wording sent LLMs to community sources. Forums went from 0% of citations on jargon questions to 17% on messy ones, and advocacy organizations went from 0% to 21%. A question about a personal situation looks like a forum post, and LLMs seem to match it to sources written in the same register.

How many prompts should a GEO monitoring set have?

There is no fixed number. Weighting matters more than volume. The rebuilt set in this study has 94 prompts: 82 from real chat language and a 12-prompt branded control. Each situation gets prompts in proportion to how often customers raise it, so the most common one is never under-tracked.

Do these findings apply outside health insurance?

Yes. The study used a health insurance AI app, but each finding here describes behavior that doesn’t depend on the product: people write in their own words, lead with their situation, and cluster around a few triggers, and LLMs cite different sources for messy questions than for tidy ones. The exact percentages will differ by industry. Measure them on your own conversations.