Your product exists. Does the AI know?
AI systems are deciding which products to recommend. Here's what changed, why most sites are invisible to them, and what to do about it — from someone who just implemented it.
I added a single file to my portfolio site recently. Forty minutes of work. It describes who I am, what I've built, and what problems I solve — written not for human visitors, but for the AI systems increasingly being asked those exact questions.
That file is called llms.txt. This is what it is, why it matters, and why most products are effectively invisible to the systems now deciding what to recommend.
What changed in how people find products online?
When someone used to search for a tool like yours, Google returned ten links. They scanned the list, clicked something, read it, maybe came back and clicked something else. Your job was to be in that list, ideally near the top.
Now a growing number of people type that same question into ChatGPT, Perplexity, or Google's AI Overview — and get a single synthesised answer. No list. No ten options. One response, sometimes with two or three sources cited, sometimes with none. The searching still happens, but it happens inside the model, invisibly, before the answer surfaces.
The numbers reflect this. When an AI summary appears at the top of a search page, users click traditional results about half as often. Referral traffic from ChatGPT alone grew over a hundred percent in the six months to early 2025. Gartner's 2025 forecast puts a quarter of traditional search volume migrating to generative platforms within the year.
This isn't a trend to prepare for. It's already the infrastructure.
Why is your site probably invisible to AI crawlers right now?
Here's what most builders don't catch: your SEO could be solid and you'd still be invisible to AI systems. They're different machines with different failure modes.
AI crawlers don't browse the way humans do. They issue HTTP requests and read the raw HTML response. They don't execute JavaScript, they don't click tabs, they don't open accordions. If your product features are described inside a component that renders client-side, the model never sees them. If your pricing is behind an interactive toggle, it doesn't exist as far as the crawler is concerned. Your site looks complete to a visitor and is effectively empty to the system deciding whether to mention you.
There's a second problem that catches more sites than people realise. Cloudflare updated its default bot management settings to block AI crawlers automatically. If your site runs behind Cloudflare and you haven't explicitly changed this, there's a real chance you've been invisible to most AI systems without knowing it. Check your Cloudflare dashboard under Security → Bots → Bot Fight Mode before anything else.
What is Answer Engine Optimization (AEO)?
AEO — Answer Engine Optimization — is the practice of making your content legible, credible, and citable to AI systems. The goal is different from SEO. SEO gets you into the index. AEO gets you cited from it.
The difference matters because AI models don't rank results — they construct responses. They're not ordering ten options with yours near the top. They're building an answer, and your content either gets pulled into that construction or it doesn't. Keyword density has almost no effect on whether it does. What does: expert perspective, specific numbers, inline references, and clean answers to clearly framed questions. A Princeton GEO study testing this across ten thousand queries found that adding expert quotes increased citation likelihood by around forty percent. The model is looking for content that behaves like a primary source — not content optimised to rank.
One finding that catches most people off guard: there's a ninety-two percent correlation between domains that rank in Google's top ten and domains cited in AI Overviews. The foundation hasn't changed. What you build on top of it has.
What is llms.txt and does it actually work?
The llms.txt file is a plain markdown document at the root of your site — think robots.txt, except instead of telling crawlers what to avoid, it tells AI systems what matters most.
You list your important pages with short descriptions in plain markdown. The model uses that as a map before crawling your site. The description you write next to each URL shapes how the model understands that page before it even fetches the content.
Here's a simplified version of the format, based on what I run at gabana.dev/llms.txt:
# Gabana — Product Engineer, System Thinker and AI Augmented Engineer
gabana.dev | [email protected] | Nairobi, Kenya
> Full-stack product engineer who has owned, built, and shipped
multi-tenant SaaS platforms and AI implementations.
## Products
- [PsTally](https://pstally.com): Gaming lounge management software —
session tracking, shift reconciliation, WhatsApp reporting for owners
- [Stoka](https://stoka.co.ke): Shop management and POS for Kenyan
retailers — inventory, staff shifts, M-Pesa integration
## Work
- [PsTally case study](https://gabana.dev/work/pstally): How I built
lounge management software because I owned a lounge and needed it
- [Stoka case study](https://gabana.dev/work/stoka): Building a POS
system around remote shop management, not just sales tracking
It's worth being honest about what this doesn't do. It doesn't guarantee citation. It doesn't replace good content. It doesn't shortcut the harder work of building genuine authority. What it does is remove friction for AI systems trying to understand your site. As of mid-2025, ChatGPT and Perplexity honour it. Google Gemini doesn't yet.
Implementing it takes an hour. Given that the alternative is hoping the AI correctly infers your most important content through crawling alone, that hour is worth spending.
What should you actually change if you're building a product?
Write for questions, not keywords. Your ideal user is now more likely to ask ChatGPT "what's the best way to manage a gaming lounge in Kenya" than to Google "gaming lounge software Kenya." That's a different kind of content — conversational, direct, answer-first. The page that wins here isn't the one stuffed with keywords; it's the one that gives the clearest answer to that specific question.
Make your content server-rendered. Anything that loads via JavaScript after the initial page response is invisible to AI crawlers. If your important pages — anything describing what your product does and who it's for — only exist after JavaScript runs, they don't exist to the crawler. A quick test: run curl yourdomain.com and look for your product name in the response. If it's not there, neither are you.
Build presence where the models look. Research consistently shows the same pattern in what AI systems cite: Wikipedia, Reddit, YouTube, Quora, LinkedIn — the platforms that appear most frequently in AI citation analysis. These aren't just SEO plays — they're inputs into how the model understands your brand. If the only place your product is described is your own website, the model has one source. If it's discussed across several credible platforms, it has more to work with and more reason to trust what it finds.
Be specific, not comprehensive. The instinct for search has been to cover everything — long guides that signal authority through volume. AI systems don't reward volume; they reward clarity. A concise, well-structured answer to a single question outperforms a sprawling guide that buries the answer on page three.
What most people building for AI search are getting wrong
There's a subtler shift underneath all of this that most AEO guides miss.
The search box trained a generation of users to think in keywords — short, stripped-down fragments that hint at intent. AI interfaces are doing the opposite. The average ChatGPT prompt runs around seventy words. People are describing their problem in full sentences, with context and constraints. They're adapting to a system that can handle nuance, and that changes everything about how products get discovered.
Your product isn't being found through a keyword anymore — it's being found because someone described a problem that your product solves, and the AI connected those two things.
For PsTally, that means being findable not when someone searches "lounge management software" but when they describe "how do I stop staff arguments about session times at my gaming cafe." For Stoka, it's not "POS Kenya" — it's "how do I know what's happening in my shop when I'm not there."
The gap between those two framings is the gap between being invisible and being cited. Your job is to describe your product in the language of the problem, not the language of the feature. The feature is the answer. The problem is the search.
What to do next
None of this requires abandoning what's already working. Strong SEO remains the strongest foundation for AI visibility — the correlation is high enough that the two reinforce each other. What changes is what you layer on top.
Start by checking whether AI crawlers can actually reach your site. Verify your robots.txt and your Cloudflare settings if relevant. Fetch your homepage with curl and confirm your content exists in the initial HTML response.
Add an llms.txt file. List your most important URLs with short, specific descriptions — not page titles, but actual summaries of what each page answers. It takes an hour.
Then look at your content through a different lens. Not "does this rank?" but "if someone asked an AI the question this page answers, would this be the clearest, most credible response available?" Where the answer is no, that's where to focus.
The window for doing this before it becomes standard practice is narrowing — not because AEO is new and trendy, but because the underlying behaviour shift is compounding quietly, month by month. The products legible to these systems now will have a material advantage over the ones that catch up later.
Discovery is being rebuilt. The question is whether your product is being built to be found in the new version of it.
Frequently asked questions
What is llms.txt?
A plain markdown file placed at the root of your website (yourdomain.com/llms.txt) that gives AI systems a structured map of your site — which pages exist, what they contain, and which matter most. Think of it as a site map written specifically for large language models.
Does llms.txt guarantee AI visibility? No. It removes friction for AI systems trying to understand your site structure and prioritise your content. It doesn't replace good content or established domain authority. Systems that currently support it: ChatGPT and Perplexity. Google Gemini does not yet honour it as of mid-2025.
What is the difference between SEO and AEO? SEO (Search Engine Optimization) gets your content into the search index and ranked in results. AEO (Answer Engine Optimization) gets your content cited when an AI constructs a response to a user's question. SEO is the foundation; AEO is built on top of it — the two are complementary, not competing.
Will Cloudflare block AI crawlers from reaching my site? Possibly. Cloudflare updated its default bot management settings to block AI crawlers. Check Security → Bots → Bot Fight Mode in your Cloudflare dashboard. Configure exceptions for the AI bot user agents you want to allow through.
How long does implementing llms.txt actually take? The file itself takes about an hour. The heavier work is auditing your content structure and rendering — making sure your important pages are server-rendered and written to answer specific questions directly. That's where most sites have the most to fix.
For the technical implementation — robots.txt configuration for AI crawlers, server-side rendering, schema markup, and a full audit checklist — read the companion piece: How AI systems actually read your site.
In this series
Leave a response