sitemap.ts and robots.ts in Next.js 16: How We Control Google and AI Search Crawlers

Key takeaways

app/robots.ts and app/sitemap.ts in the Next.js App Router are special Route Handlers that Next.js caches by default, so a sitemap built from CMS data stays stale until the next build or an explicit revalidation.

Under RFC 9309, a crawler obeys only the most specific user-agent group that matches it, so a group for OAI-SearchBot replaces the * group instead of extending it and must repeat every Disallow path.

AI vendors run separate crawlers for search and for training: OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training, and Anthropic uses Claude-SearchBot and ClaudeBot. Blocking a training crawler does not block the search crawler.

Google ignores changefreq and priority in sitemaps and uses lastmod only when it is consistently accurate, so setting lastModified to the build time on every URL wastes the one field that matters.

A robots.txt Disallow rule stops crawling, not indexing. To keep a page out of search results, we serve noindex and leave the URL crawlable.

Why do we generate robots.txt and sitemap.xml from code?

At Devya we generate robots.txt and sitemap.xml from TypeScript on every Next.js project instead of copying static files between repositories. Static files fail quietly: staging gets indexed, a crawler matches a rule nobody wrote for it, and search engines learn to ignore dates that change on every build. Generating both files from the same route constants and data functions the pages use keeps them honest, and it lets us handle the AI crawlers that decide whether ChatGPT, Claude or Perplexity can cite a page.

What do robots.ts and sitemap.ts generate in Next.js 16?

app/robots.ts returns a MetadataRoute.Robots object that Next.js serves as /robots.txt, and app/sitemap.ts returns a MetadataRoute.Sitemap array that Next.js serves as /sitemap.xml. robots.ts must sit at the root of the app directory, while sitemap.ts can also live inside a route segment such as app/blog/sitemap.ts, which serves /blog/sitemap.xml.

The Next.js documentation describes both files as special Route Handlers that are cached by default unless they call a request-time API such as cookies() or headers(), or use a dynamic route config. In practice the output is computed at build time. When posts live in the repository, every deploy rebuilds the sitemap. When posts come from a CMS or an API, our publish handler saves the post and then calls revalidatePath('/sitemap.xml') next to the revalidation of the post page, so a new article appears in the sitemap without waiting for a rebuild.

Which AI crawlers do we allow, and which ones are only about training?

AI crawlers fall into three groups: search crawlers that fetch pages so an assistant can cite them with a link, training crawlers that collect data for future models, and user-triggered fetchers that load one page because a person asked for it in a chat. Blocking one group does not block the others.

OpenAI: OAI-SearchBot for ChatGPT search, GPTBot for training, and ChatGPT-User for user-triggered fetches.

Anthropic: Claude-SearchBot for search, ClaudeBot for training, and Claude-User for user-triggered fetches.

Perplexity: PerplexityBot for search and Perplexity-User for user-triggered fetches.

Google: Googlebot for Google Search, including AI Overviews, and the Google-Extended product token for use of content in Gemini models.

Apple: Applebot for search and the Applebot-Extended product token for model training. Common Crawl: CCBot, whose public dataset is widely used for training.

Google-Extended and Applebot-Extended are robots.txt product tokens, not separate crawlers. Google states that Google-Extended does not affect inclusion or ranking in Google Search, so AI Overviews follow Googlebot rules.

Our default is that a public site which exists to be read allows all of them. For licensed or paywalled content we block training crawlers and keep search crawlers, because OpenAI states that sites which block OAI-SearchBot are not shown in ChatGPT search answers except as navigational links. Perplexity documents that Perplexity-User generally ignores robots.txt because a person requested the page, so we treat user-triggered fetchers like browsers. Anything that must never be fetched goes behind authentication, because robots.txt is a request, not access control.

Why does a named crawler ignore our * rules?

A crawler obeys only the single most specific user-agent group that matches it, and the * group applies only when no named group matches. RFC 9309, the Robots Exclusion Protocol standard, defines this behaviour, and rules from * and from a named group are never merged.

The trap looks harmless: someone adds User-agent: OAI-SearchBot with Allow: / to make sure ChatGPT can see the site, and that crawler now ignores the Disallow: /api/ and Disallow: /admin/ lines in the * group. In our robots.ts the private paths live in one constant that is spread into every group that is not fully blocked, and a comment above each named group says that it replaces *.

Non-production builds return Disallow: / for every user agent. Vercel adds an X-Robots-Tag: noindex header to preview deployments by default, but self-hosted staging gets no such protection. Checking VERCEL_ENV alone is a self-hosting bug, because the variable does not exist there and production would block every crawler, so our check also accepts our own DEPLOY_ENV variable. Disallow only keeps well-behaved crawlers out of staging; authentication is the real protection, because a disallowed URL can still be indexed from external links.

How do we set lastModified, changeFrequency and priority?

We set lastModified from the date the content actually changed and leave out changeFrequency and priority. Google documents that it ignores changefreq and priority, and that it uses lastmod only when the value is consistently and verifiably accurate. Bing has also published guidance that accurate lastmod values help its crawler. Stamping every URL with new Date() on each build tells crawlers that everything changed, which teaches them to ignore the field.

Changed means the body, title or structured data changed, not a footer link that touches every page, so our post model stores updatedAt separately from publishedAt and bumps it only on edits.

For bilingual sites, the alternates.languages field in a sitemap entry renders hreflang links, and Google expects every language version to be listed with the full set of alternates, so our sitemap.ts emits one entry per language with the same alternates object. Hreflang only works when every language has its own crawlable URL. A site that switches language with a cookie on a single URL shows crawlers one language only, and no sitemap can fix that.

How do we split a sitemap that grows past 50,000 URLs?

The sitemaps.org protocol limits one sitemap file to 50,000 URLs or 50 MB uncompressed. Next.js handles larger sites with generateSitemaps, which returns a list of objects with an id, and Next.js serves each chunk at a path such as /blog/sitemap/0.xml. Since Next.js 16 the id prop passed to the sitemap function is a Promise that resolves to a string, so we await it and convert it to a number before calculating the page offset.

We list every chunk URL in the sitemap array of robots.ts, which accepts a list of strings, so crawlers discover each file directly instead of relying on a sitemap index.

How do we check that crawlers see what we intended?

We test the generated objects in CI, then check the deployed output and the search engines' own reports. Because robots() is a plain function, a Vitest test stubs the environment with vi.stubEnv, calls robots(), and asserts that every group that is not fully blocked still disallows /api/ and /admin/. A second test asserts that a preview environment returns Disallow: / for every user agent.

After deploy, we fetch /robots.txt and /sitemap.xml from the production domain and read them the way a crawler would.

Google Search Console shows the robots.txt Google last fetched, with parse errors, and how many sitemap URLs it discovered.

Bing Webmaster Tools offers the same checks, and Bing supports IndexNow for pinging URL changes on publish; Google does not use IndexNow.

Firewall and CDN bot rules can block a crawler before robots.txt is ever read, so we check firewall logs for the bot's user agent when a crawler seems to ignore an Allow rule.

FAQ

Q: Does blocking GPTBot remove a site from ChatGPT answers?

A: Not by itself. GPTBot collects training data, while ChatGPT search fetches pages with OAI-SearchBot. Blocking GPTBot and allowing OAI-SearchBot opts out of training while keeping pages eligible for cited ChatGPT search answers.

Q: Does a robots.txt Disallow rule keep a page out of Google?

A: No. Disallow stops crawling, but Google can still index a disallowed URL it finds through links, without its content. To keep a page out of results, return a noindex robots meta tag or X-Robots-Tag header and leave the URL crawlable so Googlebot can see it.

Q: Why is a new post missing from sitemap.xml in Next.js?

A: sitemap.ts is cached by default, so data read at build time stays until the next build or a revalidation. Call revalidatePath('/sitemap.xml') when you publish, or rebuild the site.

Q: Can we trust the crawler user agent in server logs?

A: No. Any client can send User-Agent: GPTBot. Google documents reverse DNS verification for Googlebot, and OpenAI publishes IP address ranges for its crawlers, so verify the source before treating traffic as a real crawler.