Indexing, sitemap and AI discovery

Enable indexing at launch

The starter ships with SEO_INDEXABLE=false. Leave it false while replacing sample content and completing the launch checklist. When your public product pages are ready, set SEO_INDEXABLE=true in the production build environment and rebuild/redeploy. Deployment kickstart's explicit runtime allowlist includes this non-secret setting.

The SEO policy also checks the environment and origin. Development, localhost and Vercel preview builds remain noindex even if the flag is true. For another staging host, keep the flag false. The canonical origin comes from APP_URL; saved DEV/LIVE values do not override it. Review lib/seo/policy.mjs for the release's exact guard.

The supported deployment workflow disables automatic Vercel previews because the application shares one hosted Auth/email/MCP origin. The indexing guard still provides defense for a preview build someone creates separately; it does not make that preview an independently supported authentication environment.

Robots is a discovery preference

robots.txt tells cooperative crawlers where they may crawl. noindex asks them not to list a page. Neither prevents a person or an uncooperative crawler from requesting a public URL. Password protection, server authorization and database policies remain the security boundary.

After launch, public HTML needs to be crawlable for search engines to read its metadata. Private authentication, dashboard, OAuth, setup, preview and API routes retain exclusion headers. Sample privacy/terms pages keep their explicit noindex until you replace and review them. Do not remove server permissions after adding a robots rule.

What enters the sitemap

config/seo.ts lists real public website routes. Public product docs are loaded from content/docs; published blog stories come from the anonymous reader and publication policy. Blog URLs include stories beyond the first nine-item listing page. The implementation intentionally fails on a required provider-read error rather than publishing an incomplete success.

Only canonical HTML URLs belong in the sitemap. Exclude dashboard records, private API responses, redirects, query/filter variants and Markdown alternatives. The source emits no invented modification timestamps, priorities or update frequencies. If the product grows toward 45,000 blog posts, split the sitemap before exceeding the current single-file design.

Every MDX page in the public collection can enter discovery, including a page omitted from sidebar navigation. Removing its name from meta.json does not turn it into a private draft. Put unpublished material outside content/docs and outside every public loader.

Trace one public route

Use the /features/notes page from metadata and sharing:

1. Its Next.js route renders useful HTML and calls publicMetadata with its own path.
2. Its path is added to seo.pages, so the launched sitemap can include the canonical URL.
3. Its metadata can advertise a Markdown alternative only after you implement that alternative. Registration does not generate a new Markdown endpoint for arbitrary pages.
4. If you want it in the machine-readable home summary, add truthful notes copy to the marketing configuration consumed by the summary renderer, or implement a dedicated public Markdown route and link it deliberately.
5. Rebuild, inspect HTML/canonical and fetch the sitemap. A local noindex build should still refuse indexing; test inclusion on the intended launched build.

The built-in homepage already demonstrates the complete HTML/Markdown pair through / and /index.md. Docs and blog articles have their own corresponding export routes described below.

llms.txt and Markdown entry points

The spelling llms.txt follows an emerging convention. It is a convenient text entry point, not a ranking promise, an authentication protocol or evidence that a particular model will ingest the site. Google's AI search guidance does not require special AI files for its search features. See the llms.txt proposal for the format's purpose.

HTML advertises a describedby link to the discovery file. Home, docs and blog metadata expose Markdown alternatives. Text responses use canonical/discovery Link headers and noindex so search engines can prefer HTML. These exports do not grant agent access to customer data; REST/MCP authorization remains separate.

Custom-component text

When a guide includes a custom component, HTML alone is insufficient. Register its text representation in components/docs/markdown.tsx. The copied guide should preserve labels, commands, table data and a diagram's explanation. Avoid exports that contain only an unknown component tag or silently omit the important instructions.

The loader must never enumerate the whole repository. Keep internal docs/, environment files, receipts, test reports and workspace data outside its allowlist. The seller manual you are reading has a separate public collection and does not include your product's authenticated APIs.

Verification commands and expected results

Use these public requests against a running app. Replace the origin only with a target you own. In PowerShell use curl.exe.

curl --fail-with-body http://localhost:3000/robots.txt
curl --fail-with-body http://localhost:3000/sitemap.xml
curl --fail-with-body http://localhost:3000/llms.txt
curl --fail-with-body http://localhost:3000/llms-full.txt
curl --fail-with-body http://localhost:3000/index.md
curl --fail-with-body http://localhost:3000/api/docs/markdown/guides/documentation

On development, expect noindex/disallow behavior and no launched sitemap entries. Expect readable text for the allowed public exports. Search those outputs for an internal-only marker placed in an excluded fixture; it must be absent. Fetch a nonexistent/private Markdown path and confirm 404 instead of filesystem access.

Repeat against the actual production build with the launch flag off and on as appropriate. Confirm the launched sitemap contains every intended public route and no drafts/private paths; check malformed request Host headers cannot rewrite the canonical origin. Use the source SEO tests from testing and verification. Actual indexing and ranking are external outcomes, not results a successful build can establish.