Google Search Console Indexing Repair Status

Date: January 25-26, 2026 Status: ALL FIXES COMPLETE - Validation In Progress

Summary

Three indexing issues were identified and fully resolved for sempers.com:

Google Search Console validation was initiated on January 25, 2026. Full resolution expected in 2-4 weeks.

Issues Identified

1. Soft 404 Errors

• Cause: The .htaccess file was using RewriteRule . /404error.html [L,R=301] which returned a 301 redirect followed by 200, instead of a proper 404 status code
• Affected URLs: 2 pages on ftp.sempers.com (discovered later - see Issue 3)

2. 401 Unauthorized Error

• Cause: The /feedback/ directory requires a token parameter; crawlers got error messages
• Affected URLs: 1 page

3. ftp.sempers.com Serving Website Content (Discovered During Investigation)

• Cause: DNS record for ftp.sempers.com pointed to same IP as main site, serving duplicate content
• Affected URLs: The 2 soft 404 URLs were actually on ftp.sempers.com, not the main domain:
  • https://ftp.sempers.com/rest-breaks/disclaimers/terms-of-service.html
  • https://ftp.sempers.com/intermittent-fmla-leave/privacy/privacy-statement.html

All Changes Made

1. Production .htaccess Fix (Manual via SiteGround)

Location: SiteGround Site Tools → File Manager → public_html/.htaccess

Before:

# 404 handling
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule . /404error.html [L,R=301]

After:

# 404 handling
ErrorDocument 404 /404error.html

Result: Non-existent URLs now return HTTP/1.1 404 Not Found (verified via curl)

2. GitHub Workflow Files Fixed

The deployment workflows were hardcoding the broken .htaccess and overwriting it on every deploy. All three were fixed:

Change: Same as above - replaced RewriteRule with ErrorDocument 404

3. robots.txt Updated

File: /Users/zjs/engineering/sempers.com/public_html/robots.txt

Added:

Disallow: /feedback/
Disallow: /assets/includes/
Disallow: /assets/guides/

Commit: 14d0cc3

4. ftp.sempers.com Redirect Added

Location: SiteGround Site Tools → Domain → Redirects

Configuration:

• Source: ftp.sempers.com/*
• Destination: https://sempers.com
• Type: 301 Permanent
• Path forwarding: Enabled

Result: All requests to ftp.sempers.com now redirect to sempers.com (verified via curl)

Verification Results

All fixes verified working on January 26, 2026:

# 404 now returns proper status
$ curl -I https://sempers.com/fake-page-12345
HTTP/1.1 404 Not Found

# ftp subdomain redirects
$ curl -I https://ftp.sempers.com/
HTTP/1.1 301 Moved Permanently
Location: https://sempers.com

Google Search Console Actions Taken

1. Validated “Soft 404” fix - January 25, 2026

   • Status: Validation started
   • Affected pages: 2 URLs

2. robots.txt will be re-crawled automatically within 24-48 hours

Root Cause Analysis

Why the initial fix didn’t work via deployment:

1. Local .htaccess had the correct ErrorDocument 404 directive
2. CI/CD pipeline deployed it to production via rsync
3. BUT the workflow had a subsequent step that overwrote .htaccess with a hardcoded version containing the broken RewriteRule
4. This hardcoded version was in all three workflow files since their creation
5. Every deployment re-broke the 404 handling

Why we had to fix it manually on SiteGround:

• The production server’s .htaccess was different from our repo
• SiteGround support confirmed no ErrorDocument directive existed on the server
• Direct edit via SiteGround File Manager was required to fix immediately
• Workflow files were then fixed to prevent future deployments from reverting

Timeline

Expected Resolution

• robots.txt changes: 24-48 hours for Googlebot to re-fetch
• Soft 404 fixes: 1-4 weeks for re-crawl of affected URLs
• Full resolution in Search Console: 2-4 weeks
• Email notification: Google will send confirmation when validation completes

Files Modified (Summary)

Lessons Learned

1. Check deployment workflows for hardcoded overrides - The .htaccess was being overwritten by CI/CD, not just deployed from repo
2. Verify production matches local - Always check what’s actually on the server, not just what’s in git
3. Check all subdomains - ftp.sempers.com was serving duplicate content and causing indexing issues
4. SiteGround nginx sits in front of Apache - Some issues require server-level config, not just .htaccess

Last updated: January 26, 2026

Round 3: February 9, 2026

Branch: fix/seo-indexing-audit → merged via PR #115 Status: ALL FIXES COMPLETE — Monitor GSC in 2-4 weeks

Situation

Google Search Console showed 134 not-indexed pages vs 53 indexed. Investigation revealed multiple compounding issues.

Root Causes Identified

1. Production .htaccess overwritten on every deploy (AGAIN)

The deploy-production.yml workflow was overwriting .htaccess with a stripped-down version missing:

• HTTPS redirect
• www → non-www redirect
• Trailing slash removal
• Security headers
• Cache rules

Impact: 48 “Duplicate, Google chose different canonical” pages

2. Wildcard subdomain (*.sempers.com) serves full site

Any subdomain — www, ftp, aaa, ww, xyz123 — returns 200 OK with full content. This is an nginx/SiteGround-level issue.

Impact: ~65 URLs across ftp, www, aaa, ww subdomains in GSC

3. QA/ENG environments indexable by Google

qa.sempers.com and eng.sempers.com had no X-Robots-Tag noindex header.

4. Sitemap included Google verification file

googlede6683829bc41ed6.html was listed in sitemap.xml.

5. robots.txt had Crawl-delay: 10

Throttling crawler speed unnecessarily.

Fixes Applied (PR #115, deployed Feb 9, 2026)

SiteGround Changes (Manual, Feb 9, 2026)

GSC URL Breakdown (Feb 9, 2026)

Excluded by noindex (5) — All expected

• sempers.com terms-of-service and privacy-statement (intentionally noindexed)
• aaa, ftp, www versions of privacy-statement (subdomain issue)

Soft 404 (2) — Subdomain issue

• ftp.sempers.com bogus paths (will resolve with wildcard fix)

Crawled not indexed (3) — Subdomain issue

• ftp.sempers.com and aaa.sempers.com pages

Not found 404 (1) — No action needed

• sempers.com/working-off-the-clock/privacy/privacy-statement.html (malformed path, doesn’t exist)

Page with redirect (52) — Mix of issues

• www.sempers.com pages (needs SiteGround nginx fix)
• ftp.sempers.com pages (redirect already set up, clearing over time)
• HTTP versions (HTTPS redirect working)
• Old renamed pages (redirects working correctly)

Duplicate canonical (48) — Wildcard subdomain

• Mostly ftp, aaa, ww subdomain duplicates

Key Discovery: NGINX Direct Delivery was ON

Despite SiteGround support claiming it was disabled in January, NGINX Direct Delivery was enabled for all 4 domains. This means nginx was serving static files directly, bypassing Apache/.htaccess entirely. This is why the www→non-www .htaccess rule didn’t work on inner pages.

Cloudflare Solution (Feb 9, 2026)

Instead of waiting for SiteGround Expert Care, Cloudflare (free tier) was set up in front of SiteGround:

SiteGround NGINX Direct Delivery was then disabled for all 4 domains since Cloudflare handles edge caching. Apache now processes all requests via .htaccess.

SiteGround Expert Care request was cancelled (no longer needed).

Verification (Feb 9, 2026)

# www redirect — 301 to sempers.com (WORKING)
$ curl -I https://www.sempers.com/blog/blog.html
HTTP/1.1 301 Moved Permanently
Location: https://sempers.com/blog/blog.html

# Random subdomain — 301 to sempers.com (WORKING)
$ curl -I https://xyz123.sempers.com/
HTTP/1.1 301 Moved Permanently
Location: https://sempers.com/

# QA excluded from redirect (WORKING)
$ curl -I https://qa.sempers.com/
HTTP/1.1 401 Unauthorized

# Main site serves normally (WORKING)
$ curl -I https://sempers.com/
HTTP/1.1 200 OK

See business-files/CLOUDFLARE_SETUP_INSTRUCTIONS.md for full setup details.

Next Steps

1. Monitor GSC in 2-4 weeks for improvement
2. Sitemap resubmitted Feb 9 (121 pages discovered)

Last updated: February 9, 2026