Google Only Saw a Spinner on a Static Next.js Export: Multilingual Prerendering and robots.txt Fixes

My website is a Next.js static export, and I had always assumed that meant it was prerendered. When I finally audited it, every page's HTML contained a single spinner.

All the copy lived in JS, and when Google got around to running that JS, it saw the English version. I changed the public pages so the URL decides the language and the copy is written into the HTML at build time.

A week later Search Console reported 36 pages not indexed. After going through every one, only two were actual bugs, and both came from robots.txt.

EZLO is my own Bitfinex auto-lending SaaS. The frontend runs on Cloudflare Pages, and its users are in Taiwan.

Static export isn't prerendering

The whole site was wrapped in an I18nProvider that waited for the browser to mount and detect the language before rendering its children, so the built HTML had no copy at all.

Google renders in an en-US environment, so <html lang="en">, the title and the share image were all English too. My users search in Traditional Chinese, but to Google, EZLO was an English website.

A Next.js static export doesn't guarantee prerendered content. Open the HTML after the build and look.

The content had a problem as well. The brand name "EZLO" is completely taken by a smart-home brand with the same name, so the only terms that can bring in traffic are category terms like "Bitfinex lending bot" (searched in Chinese). The homepage H1 contained none of them.

Let the URL decide the language

Traditional Chinese lives at / and English at /en/.

I added src/i18n/site.ts, which determines the language from the path and creates a fixed-language, synchronously initialized i18next instance for each locale.

Public pages no longer wait for mount, and the copy is written into the HTML at build time. The logged-in app pages still render after mount as before. Those pages shouldn't be indexed anyway.

After the change, opening out/index.html shows the H1, the three steps and the FAQ, and out/en/index.html is in English. While I was at it, I also put "Bitfinex lending bot" into the homepage H1.

I considered using multiple root layouts, one per locale, so each could have its own <html lang>. But the 404 page would then need the experimental global-not-found, which hasn't been proven under static export.

Google mostly judges language from page content, so a single root layout is enough.

Metadata follows two rules. Structured data only includes things that actually exist on the site: no real reviews, no rating.

Prices don't go into the title or JSON-LD. The DB is the single source of truth for prices, and anything hardcoded into meta tags will go stale sooner or later.

STG and PR previews run the same code as PROD and must not be indexed.

Instead of adding another environment variable, I check whether NEXT_PUBLIC_API_URL points at the PROD API. Every extra env var is one more chance to forget setting it. If PROD missed it, the whole site would go noindex without a single error to warn me.

Pages in non-indexable environments carry noindex, nofollow, and robots.txt was set to Disallow: / at the time. That's where the problem was.

Two robots.txt bugs

The first: STG got indexed. Its homepage, help pages and campaign pages were all in the index, duplicating PROD's content.

The cause was two settings fighting each other. With Disallow: / in robots.txt, Googlebot doesn't crawl the pages, and without crawling, it can't read the noindex on them.

For URLs already in the index, Google never gets the chance to learn they should be removed. Google's recommended removal path is the opposite: let the crawler read the noindex.

For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.

That's from Google Search Central's Block Search indexing with noindex. Don't use Disallow and noindex on the same URL.

So STG's robots.txt became Allow: /, with only the page meta tag blocking indexing, and I submitted a temporary removal request for the STG domain.

The second was the API domain. The backend's /robots.txt was blocked by Spring Security for anonymous requests and returned 403. The PROD and STG API domains sat under "Blocked due to access forbidden (403)".

The fix was a backend robots.txt that returns Disallow: /, with the path added to permitAll and to the maintenance-mode allowlist.

robots.txt itself must return 200 to anonymous requests. Under Google's robots.txt spec, a 403 counts as having no robots.txt, so its Disallow rules are ignored. A 5xx during maintenance would make Google pause crawling the whole domain.

The API root stays 403. It has nothing to index, and being blocked by robots is the correct end state.

Reading Search Console

The reports lag by several days, so don't read them as the current state.

A few articles the report listed as "Crawled, currently not indexed" turned out to be indexed already when I checked each one with URL Inspection.

Use URL Inspection for a single page's status, not the aggregate numbers.

Not every item needs a validation request either. App pages blocked by robots are supposed to be blocked, 404s from external scanners probing the site should be 404s, and adding trailing slashes or redirecting http to https are normal redirects.

The current state is correct for these, and validating them only gets you a failure.

There's also a Cloudflare trap. For a while, the English homepage showed "Indexed, though blocked by robots.txt", even though the live robots.txt clearly allowed it.

Digging back, for a period Cloudflare's Managed robots.txt had prepended Content Signals comments and a block of AI bot rules to the file.

That feature is still on, and any change to the AI bot preferences will make Cloudflare rewrite the file again.