Update, 18 September 2026: This article has been revised to correct the incident year and separate the July 2026 Search Console observation from Google’s current documented behaviour. The recrawl failure was reported on some properties, not as a confirmed Google-wide outage. Google’s current documentation still lists Request a recrawl as an available function in the robots.txt report for critical changes and fetch errors.
A failed robots.txt recrawl request does not automatically mean Google cannot fetch your robots.txt file, and it does not prove that a site has a crawling or indexing problem. That distinction became useful in July 2026, when several Search Console properties returned an “Unknown error” after users tried to request a robots.txt recrawl. The incident is best treated as a diagnostic case: first establish whether the file itself is correct and reachable, then decide whether the problem sits with the site, Google’s cached copy, or the Search Console request interface.
- On 27 July 2026, Search Engine Roundtable documented reports of the robots.txt recrawl request failing on several properties while continuing to work on others.
- Google’s John Mueller said he could reproduce the error on some of his own sites, but this was not a formal Google incident announcement or confirmation of a universal bug.
- Google’s current robots.txt documentation says the recrawl request is mainly for critical changes or fetch errors. Normal robots.txt updates are discovered automatically.
- If the request button fails, the first priority is to verify the live robots.txt file, the latest fetched version in Search Console, and any host or server errors before assuming the tool is the root cause.
What happened in July 2026
Search Engine Roundtable reported the issue on 27 July 2026 after Jay Cooper described repeated failures from the Request a recrawl function in Google Search Console’s robots.txt report. The message shown was “Re-crawl request failed” followed by “Unknown error. Please wait a moment and try again.”
The important detail was inconsistency. Cooper said the error appeared across several properties. John Mueller initially found the feature working on one site, then reported seeing the same failure on some of his other sites. That is evidence of an intermittent Search Console problem being observed at the time, but it is not the same as Google publishing a status incident or identifying a root cause.
The original MOCOBIN article described the event as a 2025 issue and treated it as an unresolved current outage. Both points needed correction. The documented report is from July 2026, and Google’s current help documentation continues to describe the recrawl function as available.
The recrawl button and the robots.txt file are different things
For the wider rules, syntax, crawl-control use cases and common mistakes, MOCOBIN’s robots.txt guide remains the main evergreen resource. This page deals with a narrower operational problem: what to check when the Search Console request itself fails.
Google’s robots.txt report documentation says the report shows the files Google found, when each file was last checked, the fetch status, parsing issues and previously fetched versions. It also provides the manual recrawl request. Google says this request is generally unnecessary because robots.txt files are recrawled frequently, but it can be useful after a critical rule change or after fixing a fetch error.
Google’s robots.txt update documentation also says its crawlers normally refresh the cached file within about 24 hours. The manual request is therefore an acceleration mechanism, not the only path by which Google discovers an updated file. A failed request can be inconvenient, especially after an accidental site-wide block, but it does not by itself establish that Googlebot has stopped checking the file.
Start by identifying which layer is failing
Several different problems can produce similar symptoms. Treating all of them as a Search Console bug is usually the wrong place to start.
| What you observe | More likely issue | First check |
|---|---|---|
| The live robots.txt file is correct, Search Console shows a recent successful fetch, but Request a recrawl returns an error | Intermittent request-interface or reporting issue | Retry later and watch the normal robots.txt fetch cycle |
| The robots.txt report shows Not Fetched or a recent fetch failure | File availability, server, DNS, firewall or response problem | Check the exact robots.txt URL and server response |
| Your browser shows the corrected file but Search Console shows different or older rules | Cached copy, CDN, host, protocol or origin mismatch | Compare the exact host and protocol Google is reporting |
| Search Console shows a valid robots.txt file but an important URL is still reported as blocked | URL-level state may not have caught up, or another rule or host is involved | Inspect the affected URL and verify which robots.txt file applies to it |
1. Verify the live robots.txt file before retrying anything
Open the exact robots.txt URL in a browser, normally at the root of the relevant protocol and host, such as https://example.com/robots.txt. Check that the file you see contains the rules you intended to publish.
This sounds basic, but the host matters. A robots.txt file applies to a specific protocol and host combination. The rules at https://example.com/robots.txt do not automatically govern https://www.example.com/robots.txt or another subdomain. During migrations or CDN changes, it is possible for different hosts or edge locations to serve different content.
If the live file is wrong, fixing the Search Console request button is not the priority. Fix the file, origin, cache or deployment first.
2. Check what Search Console last fetched
The robots.txt report shows the latest fetch status and the version Google last retrieved. Compare that version with the file currently served by the site. If Search Console shows a successful recent fetch of the corrected rules, a failed manual request is less urgent because Google already has evidence of the new file.
If the report shows Not Fetched, investigate the fetch failure rather than repeatedly pressing the recrawl button. Google’s Crawl Stats documentation explains that robots.txt availability problems can affect crawling. HTTP 429 and 5xx responses are treated differently from a normal 200 response or a missing robots.txt file, and repeated availability failures can cause Google to slow or temporarily stop crawling while it keeps trying to retrieve the file.
This is where the wider crawling and indexing process matters. A Search Console interface error and an actual crawler-access failure can look similar from a distance, but the operational consequences are not the same.
3. Check an affected URL separately
If the robots.txt report looks healthy but Search Console still says an important page is blocked, inspect the page itself. The URL Inspection tool can show whether Google reports crawling as allowed for that specific URL and which indexed state it currently knows about.
Do not use the recrawl request as proof that an indexing problem has been fixed. Google’s robots.txt documentation explicitly notes that requesting a robots.txt recrawl does not guarantee an immediate recrawl of pages that were previously blocked. Updating the crawler’s cached rules and revisiting individual pages are separate steps.
For teams managing a launch or accidental Disallow: / rule, this distinction is especially important. Correcting robots.txt restores permission to crawl. It does not mean every affected URL will be revisited or indexed immediately.
4. Investigate cache and infrastructure mismatches when the evidence conflicts
The harder cases are those where a browser shows the correct file but Google’s reports continue to reflect older or different rules. Before assuming Search Console is wrong, compare the exact hostname, protocol and response that Google is checking.
CDN caches, reverse proxies, staging rules, host-level fallbacks and security layers can all create a situation where the site owner sees one robots.txt response while a crawler reaches another. A 2026 Search Central Community case, for example, was eventually traced to a cached fallback robots.txt file served through infrastructure rather than the file the site owner expected. Community cases are not proof of a general Google problem, but they illustrate why conflicting robots.txt evidence should trigger an infrastructure check.
Useful evidence includes the live response body, HTTP status, response headers, CDN cache behaviour, server logs and the fetched version displayed in Search Console. The goal is not to collect every possible signal. It is to identify where the first disagreement appears.
What to do if Request a recrawl still fails
If the live file is correct, Search Console does not show a file-level fetch error, and only the manual request is failing, retrying later is reasonable. Google’s documentation says the cached robots.txt version is refreshed through normal crawling, generally within about 24 hours, so the manual button should not be treated as the site’s only recovery path.
For urgent cases, keep a short record of when the robots.txt file changed, what rules were removed or added, when the live file became correct, and what Search Console showed at each stage. That timeline is much more useful than repeatedly submitting the same request without knowing whether Google’s fetched version has changed.
It is also worth avoiding an unrelated detour into crawl budget. A failed robots.txt recrawl request is not evidence that the site has exhausted crawl budget. First resolve file accessibility, cached rules and URL-level crawl permission. Crawl-demand questions belong later, if the site is accessible but important URLs are still not being revisited at a useful rate.
Current status and the role of this page
As of 18 September 2026, Google’s current robots.txt report documentation still lists Request a recrawl as a supported feature. The available evidence does not justify describing the July 2026 observation as either a confirmed Google-wide outage or a formally resolved incident.
The useful part of the episode is the diagnostic sequence it exposes. The main robots.txt guide explains how to create and use crawl rules safely. This supporting news analysis covers the narrower failure mode where Search Console cannot immediately refresh the cached file through its manual request. If the button fails again, check the file first, the fetched version second, the affected URL third, and infrastructure differences before changing unrelated SEO settings.
