site migrations and status codes

Site Migrations, HTTP Status Codes, and Crawl Infrastructure #

A comprehensive, authoritative engineering reference on HTTP status codes, domain and site migrations, crawl budget optimization, and crawlable site architecture based on official Google Search Central guidelines.

1. HTTP Status Codes Decision Tree & Indexing Behavior #

HTTP status codes are the primary signal Googlebot uses to determine whether a URL should be crawled, indexed, canonicalized, or purged from the search index.

graph TD
    Request[Googlebot HTTP Request] --> Code{HTTP Response Code}

    Code -->|200 OK| Serving[Standard Serving & Indexing Pipeline]

    Code -->|301 Moved Permanently| MovePerm[Permanent Move: Forward Canonical Signals, PageRank & Index Target]
    Code -->|302 Found / 307 Temp| MoveTemp[Temporary Move: Keep Source in Index & Follow Target for Crawl Only]

    Code -->|404 Not Found| NotFound[URL Missing: Drop from Index after Verification]
    Code -->|410 Gone| PermGone[Explicit Deletion: Fast-track Removal from Index]
    Code -->|200 with Empty / Error UI| Soft404[Soft 404 Penalty: De-indexed & Crawl Budget Wasted]

    Code -->|429 / 503| ServerStress[Throttle Crawl Rate & Exponential Backoff via Retry-After]

Complete Status Code Matrix for Googlebot #

2. Preventing and Fixing Soft 404 Errors #

A Soft 404 occurs when a server returns an HTTP 200 OK status for a page that is effectively an error page, empty document, or missing resource.

Common Causes of Soft 404s #

1. Missing Content with 200 OK: Rendering a "Product not found" or "Page does not exist" message while returning HTTP/1.1 200 OK.
2. Blank or Thin Templates: Dynamic single-page applications (SPAs) rendering an empty <div id="app"></div> when an API entity returns 404.
3. Blanket Homepage Redirects: Redirecting hundreds of deleted sub-pages via 301 to / instead of a 1-to-1 equivalent or returning 404/410.
4. Infinite Category / Search Pages: Faceted search or zero-result search result pages returning 200 OK with zero listings.

graph LR
    A[Missing / Deleted Page] --> B{HTTP Header Sent}
    B -->|200 OK with 'Not Found' Text| C[Google Flags as Soft 404]
    B -->|301 Redirect to Homepage| C
    B -->|404 Not Found / 410 Gone| D[Valid Crawl Termination]
    C --> E[Wasted Crawl Budget & Indexing Warnings in GSC]
    D --> F[Clean Index Hygiene]

Server Configuration Fixes #

NGINX Custom 404 / 410 Handling #

# Explicit 404 error page returning true 404 status
error_page 404 /errors/404.html;
location = /errors/404.html {
    root /var/www/site;
    internal;
}

# Decommissioned endpoint returning 410 Gone immediately
location /legacy-discontinued-api/ {
    return 410 "The requested endpoint has been permanently decommissioned.";
}

Apache .htaccess Configuration #

# Set custom 404 document
ErrorDocument 404 /errors/404.html

# Return 410 Gone for legacy directory
RewriteEngine On
RewriteRule ^old-discontinued-products/.*$ - [G,L]

3. Full Site & Domain Migrations Playbook #

A successful domain or site structure migration preserves search equity, historical traffic, and user trust by methodically managing URL transitions.

graph TD
    subgraph Phase 1: Pre-Migration Planning
        M1[Inventory All Live URLs] --> M2[Build 1-to-1 Redirect Mapping Matrix]
        M2 --> M3[Verify Staging Site & Canonical Parity]
    end

    subgraph Phase 2: Launch & Cutover
        L1[Deploy 301 Redirect Rules] --> L2[Keep Old XML Sitemap Active on Old Host]
        L2 --> L3[Submit New XML Sitemap to GSC on New Host]
        L3 --> L4[Execute Google Search Console Change of Address Tool]
    end

    subgraph Phase 3: Post-Migration Monitoring
        P1[Monitor GSC Page Indexing & Crawl Stats] --> P2[Verify 301 Logs & 404 Spikes]
        P2 --> P3[Maintain 301 Redirects for Minimum 12 Months]
    end

    Phase 1: Pre-Migration Planning --> Phase 2: Launch & Cutover
    Phase 2: Launch & Cutover --> Phase 3: Post-Migration Monitoring

Step 1: Pre-Migration Redirect Mapping Matrix #

Never rely on automated wildcard redirects for heterogeneous content. Build an exhaustive 1-to-1 CSV mapping matrix:

Step 2: XML Sitemaps Strategy During Migrations #

During a site move, do not delete the old sitemap immediately:

1. Maintain the Old Sitemap: Keep the old sitemap (listing all legacy old.example.com URLs) live and submitted in Google Search Console on the old property. This signals Googlebot to crawl the legacy URLs and discover the 301 redirects immediately.
2. Submit the New Sitemap: Deploy and submit the new sitemap (listing only 200 OK destination URLs on new.example.com) on the new Search Console property.
3. Run Concurrently: Keep both sitemaps submitted until the GSC Page Indexing report shows complete transition of indexation to the new domain (typically 3–6 months minimum).

Step 3: Google Search Console Change of Address Tool #

For domain-level migrations (e.g., example.com to example.io or oldbrand.com to newbrand.com), use the Change of Address tool:

1. Verify Both Properties: Ensure both the source domain and destination domain are verified in Google Search Console under the same Google account (Domain property verification via DNS TXT is recommended).
2. Implement 301 Redirects: Ensure sitewide 301 redirects are fully live and passing HTTP status code verification.
3. Open Change of Address:
   • In GSC, select the Old Property.
   • Navigate to Settings (⚙) -> Change of Address.
   • Select the New Property from the validated domain dropdown.
   • Run the automated pre-flight checks (redirect validation).
   • Click Submit.
4. Duration: The Change of Address signal remains active in Google's systems for 180 days while search signals and indexing are migrated.

4. Crawl Budget & Crawling Infrastructure for Large Sites #

Crawl budget is the number of URLs Googlebot can and wants to crawl on your site within a given timeframe. It is governed by Crawl Capacity Limit (server health and throughput) and Crawl Demand (site popularity and freshness).

The Control Triad: robots.txt vs. noindex vs. Sitemaps #

Blocked Resources: Preserving Full Browser Rendering #

Googlebot operates a full, modern rendering pipeline (Evergreen Chromium). To accurately evaluate page layouts, mobile-friendliness, Core Web Vitals, and hidden content, it requires access to all layout resources.

graph TD
    HTML[Fetch HTML] --> RenderEngine[Chromium Web Rendering Service]
    CSS[CSS Stylesheets] --> RenderEngine
    JS[JavaScript Bundles] --> RenderEngine
    Fonts[WebFonts] --> RenderEngine
    Images[Visual Assets] --> RenderEngine
    RenderEngine --> DOM[Rendered DOM & Computed Layout]
    DOM --> Indexer[Search & GEO Indexing Pipeline]

• Never Disallow Assets: Do not block /css/, /js/, /static/, /fonts/, or /images/ in robots.txt.
• Validation: Use the GSC URL Inspection Tool -> Test Live URL -> View Tested Page -> Screenshot / More Info to confirm no critical resources are blocked.

Preventing Infinite Crawl Traps & State-Changing Requests #

Crawlers can be trapped in infinite loops or overload dynamic servers by executing state-altering actions.

1. Blocking State-Changing Actions #

Search crawlers should never trigger side-effects:

```robots.txt

User-agent: *

Disallow: /cart/add

Disallow: /cart/checkout

Disallow: /api/auth/

Disallow: /account/logout

Disallow: /user/preferences

%%CODE_BLOCK_6%%robots.txt

User-agent: *

Disallow internal search results #

Disallow: /search

Disallow: /find?*

Disallow infinite filter permutations #

Disallow: /?sort=

Disallow: /?filter=

Disallow: /?sessionid=

%%CODE_BLOCK_7%%mermaid

graph LR

subgraph Crawlable

L1["<a href='/posts/guide/'>Valid Anchor</a>"]

L2["<a href='https://example.com/posts/'>Absolute URL</a>"]

end

subgraph Non-Crawlable

N1["<span onclick='navigate()'>JS Click</span>"]

N2["<a href='javascript:void(0)'>Void Anchor</a>"]

N3["<button data-url='/page'>Button Element</button>"]

N4["<a href='#sub-content'>Hash-only Fragment</a>"]

end

%%CODE_BLOCK_8%%html

<nav class="pagination" aria-label="Pagination Navigation">

<a href="/posts/page/1/" rel="prev" class="pagination-prev">← Previous Page</a>

<a href="/posts/page/1/">1</a>

<span class="pagination-current" aria-current="page">2</span>

<a href="/posts/page/3/">3</a>

<a href="/posts/page/3/" rel="next" class="pagination-next">Next Page →</a>

</nav>

%%CODE_BLOCK_9%%xml

<?xml version="1.0" encoding="UTF-8"?>

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<sitemap>

<loc>https://example.com/sitemaps/posts-sitemap.xml</loc>

<lastmod>2026-08-15T12:00:00+00:00</lastmod>

</sitemap>

<sitemap>

<loc>https://example.com/sitemaps/docs-sitemap.xml</loc>

<lastmod>2026-08-18T08:30:00+00:00</lastmod>

</sitemap>

</sitemapindex>

```

<lastmod> Freshness Integrity Standards #

The <lastmod> tag tells Googlebot when a page was last meaningfully modified so it can prioritize re-crawling.