Skip to content

Setting up IndexNow on a Next.js site, including scheduled posts

18 Nov 2026

·

8 minute read

IndexNow lets a site tell search engines a URL has changed, so they can recrawl it within hours instead of finding it on their next visit. On a Next.js site it takes three pieces: a key file in the folder the site serves from its root, a request that sends the changed URLs, and something that sends it at the right moment. The third piece is the one guides skip, and it matters when you publish on a schedule. Our site is built ahead of time and leaves out any post dated in the future, so a post written today for Thursday does not exist on the live site until a rebuild on Thursday. Below is the setup that runs this site, and the mistake we found in it after moving hosts.

What IndexNow does, and what it does not

The protocol is a single HTTP request: you POST a list of URLs, your host name and a key to an IndexNow endpoint, and the engine queues them for crawling. You can send up to 10,000 URLs in one request, and engines that adopt the protocol agree to share submitted URLs with all the others (IndexNow documentation), so one submission reaches every participant. Bing is the best-known participant, and its index also feeds other products that search the web, which is part of why we bothered.

Google does not take part. Its own notification API is narrower than people assume: according to its Indexing API documentation, it can only be used for pages with job postings or livestream videos. For ordinary pages, Google still relies on crawling and sitemaps, plus a manual "request indexing" in Search Console for the occasional page you want seen sooner. So IndexNow is one of three channels, alongside a sitemap that is always current and an RSS feed.

The key file goes in public/

IndexNow proves you control a site by asking for a key you choose, 8 to 128 characters of letters, digits and dashes, served as a text file at the root of your site. Next.js serves anything in its public/ folder from the site root, so the site side of the setup is one file, <key>.txt, containing the key and nothing else. Ours is a 32-character hexadecimal string.

The key file has to live with the site that serves the domain, which sounds obvious and caught us anyway. When elrootlabs.com moved to a new Next.js app on Cloudflare on 16 November 2026, the new app came with its own key file. The old key file stayed in the old app's folder in the same repository. Our manual submission script still looked for the key in the old folder, and on 17 November its key file answered 404 on the live site while the new one answered 200. Every manual submission would have been refused, because IndexNow fetches the key file to check it. We pointed the script at the new app's folder and ran it once against the blog index: IndexNow 200 OK.

The script and the workflow send the same request

The protocol is one POST, so the code is short. Our manual script, indexnow.mjs, is 30 lines including comments. It finds the key by its file name, takes the URLs from its command line or from the live sitemap, refuses any URL on another host, and sends the four fields IndexNow expects: host, key, keyLocation (the full address of the key file) and urlList. It exits with an error unless the response is a success.

The first time we ran it, on 14 November 2026, it submitted the whole sitemap of 42 URLs and got back 202. The protocol defines 202 as "URL received", with key validation still pending, which is what a first submission from a new key should return. A 403 means the engine could not find or match the key file, which is exactly what our script would have hit after the move, and a 422 means a URL that does not belong to the host.

The same request also runs automatically as a GitHub Actions workflow. Both ran successfully on 16 November, once started by hand and once on its own after a rebuild of the live site.

Scheduled posts only exist after the rebuild

On this site a post is a markdown file with a publish date in its file name. When the site is built, a one-line filter drops any post dated after today, counting days in Indian time. That keeps future posts off the site, and it means a scheduled post appears only when the site is built again on or after its date.

A daily GitHub Actions workflow handles that. Every morning at 06:07 Indian time it checks whether any post on the live branch is dated that day. If one is, it makes an empty commit on the live branch, and the hosting platform rebuilds the site from it. If none is due, it does nothing.

That changes when IndexNow should run. A hook at merge time would announce URLs that return 404 for days. So the IndexNow workflow is triggered by the completion of two others, the owner's "publish to live" workflow and the daily one, and only when they succeed. It then has to wait until the new build is actually serving, and our first attempt at that wait was wrong. It polled the live site until the key file answered with the right contents, on the theory that this proved the rebuild had finished. It proved nothing of the sort, because the key file was already there on the previous build. On 17 November, the first publish after we added two posts, the ping went out about eight minutes before the new build was live and sent the old sitemap of 44 addresses, without the two new posts. Every daily rebuild would have done the same, announcing each day's posts a day late.

The wait now checks two things, every 30 seconds for up to 20 minutes. First, it reads the status GitHub records for the hosting platform's build of the live branch's latest commit, and carries on only when that build has finished successfully; a failed build stops the ping. Second, it fetches the live sitemap and carries on only when the sitemap lists every post whose date has arrived. Only then does it send the sitemap. A run started by hand with the fix in place reported the new build live and every due post in the sitemap, and IndexNow accepted all 46 addresses.

One detail in GitHub's behaviour forced that design: a push made by a workflow's own token does not start other workflows. The daily job's empty commit therefore cannot trigger anything by itself, so IndexNow listens for the daily workflow finishing instead of for a push. A small scheduling script queues finished posts at two a day by giving each the earliest date with a free slot, so a week of posts can be approved at once and released one morning at a time with nobody touching anything.

Mistakes to avoid

  • Submitting at merge or deploy time when your site schedules content. Submit after the build that makes the URL return 200.
  • Waiting on a signal the old build already gives. A file or page that existed before the deploy cannot tell you the deploy has finished; wait on the build status, or on the new content itself.
  • Leaving the key file behind when the site moves. Check the key URL on the live domain after any change of host or framework.
  • Resubmitting a large sitemap on every change. Our site has 42 URLs, so we send all of them after each rebuild; a site with thousands should send only what changed, which is what the protocol's 429 "too many requests" response guards against.
  • Serving the key file through a redirect or with a different content type than plain text. Check it with curl from outside your network.
  • Expecting Google to react. It will not, and nothing in this setup replaces a correct sitemap.

The rest of the publishing pipeline, from topic to review, is in our post on the blog pipeline. If you run a Next.js site with scheduled content and want the same setup, these are all the pieces it takes.

AI tools

Ask ChatGPTAsk Claude

Start with the problem that matters