Discourage search engines from indexing this site: what the WordPress box does, and what it leaves alone

What does “discourage search engines from indexing this site” mean?

It means WordPress asks search engines to leave every page of the site out of their results. The checkbox that says discourage search engines from indexing this site lives under Settings, Reading, in the last row, Search engine visibility.

Under the box, WordPress prints one more line: “It is up to search engines to honor this request.” The box is a polite note to crawlers, not a wall.

Builders often tick it while a site is half done. The trouble starts when nobody clears it at launch. It is one of the first settings we open when a site on our WordPress maintenance plan is missing from search.

What changes when you tick the box?

The main change is in the page source. The WordPress documentation says that since version 5.3 the box writes a robots meta tag with noindex, nofollow into the head of the site. It also stops WordPress from pinging update services when you publish.

On 11 Oct 2026 we ran our WooCommerce test store on WordPress 7.1.2 and fetched pages the way a crawler would, without logging in. Then we ticked the box, clicked Save Changes, and fetched them again.

The home page head held this tag before:

<meta name='robots' content='max-image-preview:large' />

And this one after:

<meta name='robots' content='noindex, nofollow' />

Our test product page showed the same tag.

Does it change robots.txt?

Barely. Before WordPress 5.3, the box made robots.txt answer with Disallow: / for every crawler. A note from the WordPress core team in September 2019 explains why that rule was dropped. Blocking crawlers did not stop a site being listed, because “a site doesn’t have to be [crawled] to be listed.”

Our lab matched that. robots.txt had the same seven rules before and after: five WooCommerce paths, wp-admin, and admin-ajax.php allowed back in.

One line did vanish: Sitemap, which had pointed at wp-sitemap.xml. The sitemap itself also went from answering 200 to answering 404.

That is the right way round, by Google’s rules. Google says a page blocked in robots.txt never shows its noindex tag to the crawler, so it could still turn up in results. WordPress leaves crawling open so the tag can be read.

What does the box not do?

It does not keep people out. After we saved, the home page and the shop still answered 200 to our logged-out requests, with the full store on screen. The WordPress documentation says the same: the setting “allows normal visitors” and does not block access to the site.

With the box ticked, the WooCommerce badge in our admin toolbar still said Live. WooCommerce’s Coming soon mode documentation says the badge “reads Coming soon, Store coming soon, or Live based on your visibility settings”, so it follows a separate switch, covered in our post on WooCommerce coming soon mode. People often mix the two up.

How does a launched site stay invisible?

The box was ticked during the build and stayed ticked. Every page goes live carrying noindex, the sitemap is gone, and the owner sees a working site and a toolbar that says Live.

Google crawls the new pages, reads the tag, and leaves them out. The site does not show up for its own name, and nothing on screen explains why. Squarespace sites show the same symptom with other causes, covered in why a Squarespace site is not showing on Google.

A site moved over from a staging server can bring the tick with it. WordPress saves the setting to the database when you click Save Changes, so a copy of that database carries it along.

How do you get the site back into Google?

When a site comes to us with this problem, we untick the box, check robots.txt and the robots meta tag on the live site, and request indexing in Search Console. To do it yourself, clear the box under Settings, Reading and click Save Changes. View your home page source and confirm the noindex tag is gone. Open yoursite.com/robots.txt and look for the Sitemap line again.

Next, tell Google. Google’s documentation says the URL Inspection tool in Search Console can request a crawl of single pages, and that a sitemap is the way to hand over many addresses at once. Start with the home page, then submit wp-sitemap.xml.

Then wait, and do not keep pressing the button. Google’s recrawl page puts it at “a few days to a few weeks”, and repeat requests for one page will not make it faster.

What should you use to hide a staging copy?

A password. Google’s robots.txt guide names two ways to keep a page out of its results: noindex, or a password on the page. The password also stops people, which the box never does.

Tick the box on that protected copy too if you like, and put clearing it on the launch list. Another day-one setting worth a look on that list is the privacy policy page that may still be a draft.

If your WordPress site went live and still does not appear on Google, send us the address and we will check the box, the tag and the sitemap for you.