10 / WEB PLATFORM / SEO & AI VISIBILITY
Search is splitting in two, and the usual toolkit says nothing about whether ChatGPT can read or cite a site.
The challenge
Search is splitting in two. Classic on-page signals still matter, but a growing share of traffic now arrives through answers from ChatGPT, Claude, Perplexity and Google AI Overviews, and the usual toolkit says nothing about whether those systems can read a site, are allowed to crawl it, or ever mention it. I designed the platform to cover both halves as a paid product, with a tier ladder and Stripe billing planned from the start. Mid-build I changed the decision to free forever, and every paid dependency had to come out.
What I built
I built four services in one monorepo: a FastAPI backend, a separate Playwright crawler process, a Next.js 15 app, and a Cloudflare Worker fronting the public audit endpoint. The backend holds 55 classic SEO detectors and 35 AI-era ones, covering llms.txt validity, per-bot crawl policy for GPTBot, ClaudeBot and Google-Extended, LLM-oriented schema and E-E-A-T signals, each detector its own module behind a registry. Because I had kept every external provider behind an interface/mock/real/factory seam since the first commit, pulling the paid ones out was a swap rather than a rewrite: Claude and OpenAI became a tiered Ollama, Gemini, Groq and heuristic router, Voyage embeddings became local fastembed, SerpAPI became DuckDuckGo, DataForSEO became Common Crawl and OpenPageRank, and I neutered Stripe so the subscription lookup returns a synthetic free plan. One command brings the stack up with no keys at all, serving mock data inside the containers; run the backend directly, where `backend/.env` flips the `USE_MOCK_*` flags off, and the free providers take over.
How it works
What it looks like
The detail
`llm_router.py` tries a locally running Ollama, then Gemini, then Groq, then a deterministic heuristic tier that never touches the network and never raises. A cached two-second probe decides whether Ollama is up, and every network tier catches its own exceptions so it falls through rather than propagating. I made the same rule mandatory for every free client: embeddings degrade to a hash vector, PageSpeed to local heuristics, search to empty.
I enforce tenant isolation with a SQLAlchemy `do_orm_execute` listener that injects `WHERE tenant_id = :tenant` for any model tagged `@tenant_scoped`. I had built the criteria with `lambda model_cls, t=tenant: ...`, and SQLAlchemy caches lambda statements without tracking default arguments, so the first request's tenant id was baked in permanently and every later tenant's queries filtered on it. I fixed it by capturing the tenant as a tracked closure variable; until then I had been papering over the symptom with `_bypass_tenancy=True` escape hatches.
I front the public `/grade` endpoint with a TypeScript Cloudflare Worker whose URL validator rejects non-http(s) schemes, private and loopback IP literals, reserved TLDs, userinfo credentials and oversized URLs. The file states plainly that edge validation is necessary but not sufficient, and that the backend must re-resolve the hostname at connect time to defend against DNS rebinding. I wrote the limit down rather than assume it away.
Without paid engine APIs you cannot capture literal ChatGPT output, so I route all five engines through one free pipeline: live DuckDuckGo results fed to the free LLM with per-engine personas, every result labelled free-web-synthesis rather than dressed up as real engine output. Brand detection runs three layers, exact substring, then fuzzy match against canonical brand variants, then an LLM implicit-reference check only when the first two miss. I skip layer three deliberately for competitors, so cost stays flat however many a project tracks.
I declare `free_mode` once in `config.py` and read it nowhere, and my `docker-compose.yml` sets `FREE_MODE: "true"` on two services where it does nothing. It reads like the switch that turns the free stack on, and it is not one: inside the containers `backend/.env` is dockerignored and never copied, so the mock-on defaults win and the compose path serves fixtures. Real crawls need `CRAWLER_MODE` flipped off its mock default as well. A loose end I own.
The outcome
A working, self-hostable application rather than a plan. The frontend boots from the existing install, and every route I checked returned 200 with no backend, no database and no API keys: the marketing homepage, the free-audit flow, pricing, the five free tools, the project dashboard, AI visibility, issues, the Astra chat pane and sign-in. The repository holds 1,427 tracked files, with 806 test functions across 85 backend test modules, 78 frontend test files and 8 Playwright E2E specs, run against 565 recorded fixtures. I checked the detector counts in the docs against the source and they hold: 55 classic, 35 AI-era, 3 chatbot. I have never deployed it, so there are no usage or business figures behind it.