Custom and headless
GEO for custom websites, shipped as pull requests
If your site runs on Next.js, React, a headless CMS or your own code, we work the way your developers do: in your repo, on a branch, through review.
Your team merges. Nothing reaches production without your approval.
| Prompt | Engine | Visibility | Status |
|---|---|---|---|
| does google render javascript pages |  Google AI Overviews | 65% | Cited |
| nextjs server rendering for seo |  Gemini | 59% | Cited |
| why is my react site not cited by chatgpt |  ChatGPT | 33% | Not shown |
| best headless cms for marketing sites |  Perplexity | 46% | Mentioned |
| how to add json ld in nextjs | Gemini | 61% | Cited |
| static site vs server rendering seo | Google AI Mode | 38% | Mentioned |
Custom sites can be excellent for GEO because everything is controllable. They can also hide serious problems, because a page that looks perfect in a browser may return an almost empty shell to a bot. Whether a given AI crawler runs JavaScript is rarely documented, so we do not assume it does.
We cannot promise citations. We can make sure the site is readable without guesswork, make the answers on key pages easy to quote, and test real prompts before and after.
Platforms we implement on
WordPress
Shopify
WooCommerce
Webflow
Wix
Squarespace
Next.js
PHP
Logos belong to their owners. We are an independent agency with no partnership or endorsement implied.
The core issue
Rendering: what the crawler actually receives
Google documents how it processes JavaScript sites: it crawls, renders and then indexes, and it checks robots.txt before fetching a file. Google also says server side rendering or prerendering helps users and crawlers, and notes that not every bot can run JavaScript. That last point is the one that matters for GEO. Google tells us about Googlebot. Other AI crawlers publish far less about how they handle scripts.
So we apply a safe rule. Anything an engine should be able to quote must be in the initial HTML response: the main text, headings, links, the FAQ, and the structured data. We check this by fetching pages without running scripts and reading the raw response, the same way a simple crawler would.
Client side only apps often fail this test. A product page that loads its description after an API call can look empty to a bot. A route that shows an error screen but returns a success status can be treated as a real page. Google lists soft 404 handling, crawlable links and the History API as common issues for single page apps, and we review all three.
Test we always run: fetch the page with scripts off and read what is there.
Frameworks
Next.js, React and other stacks
The fix depends on how the page is built.
In Next.js, pages that can be prerendered or rendered on the server ship their content and metadata in the HTML. The Next.js documentation also describes streaming metadata: for most browsers and for Googlebot, metadata may be appended after the first part of the page, while bots that cannot run JavaScript receive it in the head. The framework detects such bots by user agent and lets you override the list. We check what each crawler class gets and adjust the rendering mode for pages that matter.
In a plain React single page app, we usually recommend moving the key public pages to server rendering or prerendering, rather than adding workarounds. If that is too large a change for the Pilot, we flag it as a roadmap item and fix what can be fixed in the meantime, such as meta tags, canonicals and a complete sitemap.
For static site generators and server frameworks the picture is simpler, but we still check build output, trailing slash rules, canonical URLs and redirects. Our post on GEO for JavaScript and headless sites goes deeper on the reasoning.
- Raw HTML contains the answer content
- Canonical set in HTML and not changed by script
- Real 404 and 301 status codes on the server
- Internal links are plain anchor elements with an href
Mentions and sources beyond your site.
Answer ready pages and FAQs.
Schema, entities and clean markup.
Crawlers can fetch and read.
Headless CMS
Headless CMS: content in one place, output in another
The CMS holds the content. The front end decides what bots see.
With a headless CMS, editors work in one system and a separate front end renders the pages. GEO problems tend to appear in the gap. Fields that editors fill in, such as author, date, summary or FAQ, may never reach the HTML. A preview mode may work while the production build omits the same data.
We map each content model to the page output. Then we add what is missing: a visible summary field, FAQ entries rendered as real page text, an author or organization reference, and structured data generated from fields rather than hard coded. When the data comes from the CMS, schema stays in step with the visible page. Google states that markup should describe content that readers can see, and generating it from the same source helps keep that true.
We also look at webhooks and build triggers. If an editor publishes a correction and the site rebuilds hours later, engines may keep reading the old page. Freshness matters for AI search, and our post on content freshness explains why.
- Content model mapped to page output
- Schema generated from CMS fields
- FAQ and summary rendered as visible text
- Publish to live delay measured
{ "@type": "LocalBusiness", "name": "Your brand", "areaServed": "Dubai", "sameAs": [ "linkedin.com/..." ] }
- ValidMatches visible page text
- ValidSame name on profiles
- CheckOpening hours missing
How we work in your repo
Git, staging and pull requests
- 1
Access and branch
You add us to your repository with the narrowest rights that work, usually write access to branches and no right to push to the main branch. We create a branch per task, for example crawler rules, structured data or sitemap.
- 2
Staging preview
Each branch goes to a staging or preview URL, such as a preview deployment from your host. We test the output there with scripts off, validate markup and send you the link.
- 3
Pull request
We open a pull request with a plain description: what changed, why, how we tested it and how to roll it back. Your developers review it like any other contribution and can ask for changes.
- 4
You merge and we verify
Your team merges and deploys on its own schedule. After release we fetch the live pages, confirm the fix is in place, and log it in the weekly summary.
What we change in code
Typical pull requests on the GEO Pilot
Small, reviewable changes that your team can understand.
Crawler rules
A robots.txt file or route with deliberate rules for AI crawlers, plus a sitemap reference. OpenAI documents separate crawlers for search, training and user requests, and we set each on purpose.
Sitemap and canonicals
A generated sitemap with only indexable URLs, correct canonical tags and consistent trailing slash handling.
Structured data
JSON LD components for Organization, WebSite, Article, Service or Product, and FAQPage, built from the same data as the visible page.
Rendering fixes
Moving key routes to server rendering or prerendering, and fixing soft 404s and script only links.
Answer ready content
Rewritten copy for up to 10 priority pages, delivered as content or component changes, with direct answers and real FAQs.
llms.txt
A short file served from the public folder or a route, if it is useful for your site. We do not promise any engine reads it.
The problem
Problems we often expect on custom builds
Empty shell responses
The server returns a loading container and the content arrives after scripts run. Some crawlers never see it.
Blocked by accident
A staging robots rule or bot protection rule reaches production and blocks crawlers.
Soft 404s
Missing pages return a success status with an error message, so engines read them as real pages.
Schema out of step
Hard coded markup keeps showing an old price, author or date.
Next.js specifics
robots.txt and JSON LD in the Next.js App Router
From the Next.js documentation, as read in October 2026.
The App Router supports a static robots.txt file in the root of the app directory, or a robots.ts file that returns a typed object. The documented shape has rules, each with a userAgent, allow and disallow values, and an optional sitemap field. You can pass an array of rules to give different crawlers different instructions, so an explicit group for OAI-SearchBot or for any crawler you want to restrict is a few lines in one file. The documentation also notes that robots.js is a route handler cached by default unless it uses a request time API.
For structured data, the Next.js guide recommends rendering JSON LD as a script tag in a layout or page component. It warns that JSON.stringify does not sanitize malicious strings, and suggests replacing the less than character with its unicode escape so injected markup cannot close the script tag. That matters when titles or descriptions come from a CMS or from users. The guide also points to the Rich Results Test and the Schema Markup Validator for testing, and to community typed packages such as schema dts.
We put both in version control as small, reviewable files. The robots rules live in one place, the JSON LD builders take the same data object that renders the visible page, and a test or a build check fails if a production build ships a disallow all rule that was meant for staging.
- robots.ts reviewed and rules explicit
- Staging disallow cannot reach production
- JSON LD built from the same data as the page
- Output sanitized before it is printed
Mentions and sources beyond your site.
Answer ready pages and FAQs.
Schema, entities and clean markup.
Crawlers can fetch and read.
Verification
How to check what a crawler receives on your site
Four tests any developer can repeat.
First, fetch the raw HTML with a tool such as curl and read it. Search for your main heading, the first answer paragraph, the FAQ text and the JSON LD. If any is missing, a crawler that does not run scripts will not see it. Second, fetch with a few different user agent strings and compare responses, since bot protection or middleware sometimes treats unknown agents differently. Third, check status codes on a missing URL, a redirected URL and a trailing slash variant. Fourth, compare the HTML canonical with the final URL.
Google's JavaScript SEO documentation supports the same approach. It says Googlebot queues pages that return a 200 status for rendering, that server side rendering or prerendering is still recommended, that only anchor elements with an href are followed as links, that canonical tags are best placed in the HTML and must match if script changes them, and that a noindex in the original HTML may stop rendering from happening at all. We log what we find as a table: URL, test, result, owner. Developers can then work through it like any other ticket list.
Search Console's URL inspection shows what Google rendered for a page. It speaks only about Google. For other engines we rely on the raw HTML test, because that is the lowest common denominator.
- Raw HTML contains heading, answer, FAQ and schema
- Responses compared across user agents
- 404, 301 and trailing slash behaviour tested
- Canonical in HTML matches the final URL
- CriticalAI crawlers blocked in robots.txt
- WarningMissing canonical on 14 pages
- WarningSchema does not match visible text
- WarningSlow LCP on mobile templates
- NoticeSitemap lists redirected URLs
What must be in the first response
- ✓Main heading and answer paragraph
- ✓FAQ text as real page text
- ✓JSON LD in the HTML
- ✓Links as anchors with an href
- Description loaded after an API call
- Error screen returning a success status
- Canonical changed by script
- Staging robots rule live on production
Compare
Rendering approaches compared for GEO
| Approach | What a non script crawler sees | When we recommend it |
|---|---|---|
| Static generation | Complete HTML for every page built in advance | Content that changes rarely, such as service and guide pages |
| Server side rendering | Complete HTML generated per request | Pages that change often or depend on live data |
| Incremental regeneration or caching | Complete HTML refreshed on a schedule or on demand | Large catalogs and publishing sites where freshness matters |
| Client side rendering only | An empty shell until scripts run | Logged in app areas that should not be public anyway |
| Prerendering service | A pre built copy served to bots | A stopgap when migration is not possible, checked carefully |
Choose per route, not per site. Public pages that should be cited belong in the first three rows.
What a crawler without scripts receives
Run it today
A developer checklist for AI crawler readiness
Related reading: GEO for WordPress and GEO for Shopify cover the hosted platforms. For crawler policy see what to allow in robots.txt, and test live access with the crawler access checker.
- 1
Disable scripts and load the page
Does the main heading, the first answer and the FAQ appear? If not, list the routes that fail.
- 2
Read robots.txt on production
Make sure no staging rule is live and AI crawlers you want are not disallowed.
- 3
Fetch the sitemap
Confirm it lists only canonical, indexable, 200 status URLs and is named in robots.txt.
- 4
List soft 404s
Request five invalid URLs. Each should return a real 404 status, not a 200 with an error message.
- 5
Check JSON LD
Parse every block on key templates. Fail the build on invalid JSON or empty required fields.
- 6
Time the publish path
Publish a change in the CMS and measure how long until the live HTML changes.
Worked example
Example, hypothetical: a UK B2B software company on Next.js with a headless CMS
Imagine a UK software company with a Next.js front end and a headless CMS. Its pricing and feature pages are client rendered, the FAQ is loaded after an API call, and a preview deployment left a disallow rule in the production robots file. This is an invented scenario to show the order of work, not a result.
In week 1 we fetch raw HTML for ten key routes and find the FAQ and feature text missing from six. Week 2 delivers three pull requests: one fixes the robots file and adds a build check, one moves the feature and pricing routes to server rendering, and one generates Organization, SoftwareApplication and FAQPage JSON LD from CMS fields. The team reviews and merges. In week 3 we rewrite up to 10 priority pages as content changes in the CMS, each opening with a direct answer. In week 4 we rerun the same buyer prompts and hand over a report and a 90 day roadmap that lists larger rendering work we did not attempt inside the Pilot.
Scope and limits
What the Pilot covers on a custom site
The GEO Pilot costs $999 fixed, one time, and runs inside 30 days. Week 1 is baseline and audit across five AI engines on up to 20 buyer prompts. Week 2 is technical implementation. Week 3 is up to 10 priority pages made answer ready. Week 4 is the retest, the before and after report and a 90 day roadmap. It covers one domain and one language.
It is not a site build, a framework migration or a performance project. If rendering needs a larger change than the Pilot can hold, we describe it clearly in the roadmap, and ongoing work is quoted. See technical GEO for the full list of checks and pricing for ranges.
We do not promise rankings, citations or traffic. One public example is Emtech, a Dubai IT company cited for IT AMC queries across several engines, described in the Emtech case. One client, one market, not a promise.
FAQ
Questions people ask
Do AI crawlers run JavaScript?
Google documents that Googlebot renders JavaScript. Most other AI crawlers do not publish clear details, and behavior can change. We avoid relying on it by making sure key content, links and structured data are in the initial HTML.
Do we have to move to server side rendering?
Not always. For pages that matter, server rendering or prerendering is the safest route. If a full change is too large for the Pilot, we fix metadata, canonicals, sitemap and status codes first and put rendering on the 90 day roadmap.
What access do you need to our repository?
Write access to branches is enough. We do not need rights to push to the main branch, and we do not need production credentials. Your team reviews and merges each pull request.
Can your changes be rolled back?
Yes. Each pull request is small and focused, and it states how to revert. Because your team merges, you decide when each change ships.
Do you work with a headless CMS?
Yes. We map content fields to page output, make sure summaries and FAQs reach the HTML, and generate structured data from CMS fields. We do not need to change your CMS vendor.
Will you add tracking scripts or third party code?
No. Our changes are content, markup and configuration. We do not add third party scripts to your site as part of the Pilot.
Is staging required?
It is strongly preferred. A preview deployment on your host is usually enough. If you have none, we can work with a local build you can run and a screenshot and output log for each change.
How do I add robots.txt rules for AI crawlers in Next.js?
In the App Router you can add a robots.ts file that returns rules with a user agent, allow and disallow values and a sitemap. The Next.js documentation shows this format. We write the rules with you, because which crawlers to allow is a business decision.
Is it safe to print JSON LD with JSON.stringify?
Not without care if any value comes from users or a CMS. The Next.js documentation advises replacing the less than character with its unicode escape, so injected text cannot break out of the script tag.
What is a soft 404 and why does it matter?
A soft 404 is a missing page that returns a success status with an error message. Google lists it as a common single page app problem. Engines can treat such pages as real, thin content. We make missing routes return a real 404.
Keep reading
Guides on this topic
Why your business is not showing in ChatGPT, and how to diagnose it
If ChatGPT never names your business, one of seven causes is usually behind it. Use this diagnostic to find which one, in the order that saves the most time.
Read moreAI Crawlers and robots.txt: What to Allow and What to Block
OpenAI, Perplexity and Anthropic each run separate bots for search, training and user requests. Here is which to allow, which to block and how to test it.
Read morellms.txt: Do You Actually Need One?
llms.txt is a proposal for a summary file that helps AI tools read your site. Here is what it is, what the evidence says, and when it is worth the ten minutes.
Read more- GEO by platform
- GEO for ShopifyShopify GEO done for you: robots.txt.liquid, theme code, product schema and answer ready pages, handled by us.
- GEO for WooCommerce
- GEO for WordPressWe fix your WordPress site for AI search ourselves: robots rules, schema, sitemap, llms.txt and answer ready pages.
- GEO Pilot, $999We implement the work in 30 days.
Start your GEO Pilot
Tell us your website and the five questions your buyers ask. We reply within one working day with a clear next step.
- Fixed $999, one time
- We implement the work for you
- Any platform, delivered in 30 days