Archived document. This file has been superseded or completed and is kept for historical reference.

Non-Indexed Pages Investigation - sempers.com

Date: January 25-26, 2026 Investigator: Claude Code Analysis Scope: All indexing issues EXCLUDING Soft 404 and Duplicate Canonical (already addressed)

Executive Summary

This report analyzes the remaining indexing issues from Google Search Console. Most issues are intentional and correctly configured. A few items require minor attention.

1. Page with Redirect (40 pages)

Analysis

The 40 redirect pages are caused by the www to non-www redirect configured in .htaccess:

RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.sempers.com [NC]
RewriteRule ^(.*)$ https://sempers.com/$1 [L,R=301]

Verification

$ curl -I https://www.sempers.com/
HTTP/1.1 301 Moved Permanently
Location: https://sempers.com/
HTTP/1.1 200 OK

Why 40 Pages?

The sitemap contains ~49 URLs. Google likely crawled some or all pages via both:

• https://www.sempers.com/page.html (redirects)
• https://sempers.com/page.html (canonical)

The 40 redirecting pages are the www versions.

Assessment

Recommendation

No action required. This is correct behavior. Google is simply reporting that it encountered 40 URLs that redirected. These are not errors - they are properly configured 301 redirects from www to non-www, which is the recommended approach for domain canonicalization.

2. Alternate Page with Proper Canonical Tag (8 pages)

Analysis

These are pages where the canonical tag points to a different URL. Google found this and correctly chose to index the canonical target instead.

Pages Identified

Based on source code analysis:

Deep Dive: common-forms-of-workplace-discrimination.html

File: /Users/zjs/engineering/sempers.com/public_html/discrimination-lawsuits/common-forms-of-workplace-discrimination.html

Current canonical tag:

<link href="../discrimination-lawsuits/filing-discrimination-claim-in-california.html" rel="canonical"/>

This page canonicalizes TO filing-discrimination-claim-in-california.html

Assessment:

• The page title is “Common Forms Of | Workplace Discrimination”
• The canonical target is the main discrimination filing page
• This appears to be intentional content consolidation - directing search engines to treat the main page as the authoritative version

Likely Additional Alternate Pages

Without Google Search Console access, the other 7 pages with proper canonical tags could include:

1. Pagination pages (if any exist with rel=“canonical” to page 1)
2. Parameter variations (URLs with tracking parameters)
3. Multiple internal URLs pointing to same content
4. Relative canonical resolution - some pages with relative canonicals may resolve differently than intended (this is covered in DUPLICATE_CANONICAL_INVESTIGATION.md)

Assessment

Recommendation

1. Review in Google Search Console - Check which 8 specific pages are reported
2. Verify intentionality - Confirm each alternate page should canonicalize to a different URL
3. For common-forms-of-workplace-discrimination.html:
   • If intentional consolidation: No action needed
   • If it should be indexed separately: Change canonical to self-referencing absolute URL:

     <link href="https://sempers.com/discrimination-lawsuits/common-forms-of-workplace-discrimination.html" rel="canonical"/>
     

3. Excluded by ‘noindex’ Tag (5 pages)

Analysis

Searched entire public_html directory for noindex:

public_html/feedback/feedback.html
public_html/assets/guides/10-things-after-being-fired.html

Only 2 pages have noindex tags in the source code. The other 3 reported by Google may be:

• Cached versions that have since been removed
• Pages that no longer exist (Google’s cache is stale)
• Dynamically generated pages

Pages with noindex Tags

1. /feedback/feedback.html

<meta name="robots" content="noindex, nofollow">

Purpose: Client feedback collection page requiring a token Should it be noindex? YES - This is a private page for client feedback, not for public search

Additional protection: Now also blocked by robots.txt:

Disallow: /feedback/

2. /assets/guides/10-things-after-being-fired.html

<meta name="robots" content="noindex, nofollow">

Purpose: Downloadable guide/resource page Should it be noindex? YES - This appears to be a lead magnet or downloadable resource, not meant for organic search

Additional protection: Now also blocked by robots.txt:

Disallow: /assets/guides/

Other Possible noindex Pages

The 5 total could include pages that:

• Google cached before noindex was added
• Were deleted but still in Google’s memory
• Have noindex set via HTTP headers (not visible in source)

Assessment

Recommendation

1. No action needed for existing noindex pages - They are correctly configured
2. Add noindex to /assets/includes/google-map.html (optional - already blocked by robots.txt)
3. Check GSC for the specific 5 pages to verify the remaining 3

4. Blocked Due to Unauthorized Request (401) - 1 page

Analysis

This is the /feedback/feedback.html page which:

• Requires a valid token parameter (?t=...)
• Shows an error state when accessed without a valid token
• Returns 200 OK status but displays “Invalid Link” message

Status

ALREADY FIXED - As documented in STATUS_OF_INDEXING_REPAIR.md:

Disallow: /feedback/

This was added to robots.txt on January 25, 2026.

Verification

The page is now:

1. Blocked by robots.txt (Googlebot won’t crawl)
2. Has noindex, nofollow meta tag (backup protection)
3. Returns 200 OK but shows error content (this is now irrelevant since blocked)

Assessment

Recommendation

No additional action required. Google will remove this from the 401 list after re-crawling robots.txt (24-48 hours) and confirming the Disallow directive.

5. Crawled - Currently Not Indexed (4 pages)

Analysis

These are pages Google crawled but chose not to include in its index. Common causes:

• Thin content
• Low perceived value
• Duplicate/similar content elsewhere
• Low internal/external links

Potential Candidates

Without GSC data, likely candidates based on site structure:

Glossary Page Analysis

The site has 44 glossary pages. Example content from /glossary/wrongful-termination.html:

• Page length: ~350 lines (mostly boilerplate)
• Unique content: ~200 words of definition
• Has contact form
• Has navigation

Assessment: These could be considered thin content by Google. Each page has:

• A short definition (1-2 paragraphs)
• Standard boilerplate (header, footer, contact form)
• No unique images or rich media
• Minimal internal linking to/from

Other Thin Content Candidates

Assessment

Recommendations

Short-term (Optional)

1. Check GSC for the specific 4 pages
2. Add noindex to utility pages that shouldn’t be indexed:
   • /404error.html (should not be indexed)
   • Google verification files (should not be indexed)

Long-term (Content Improvement)

For glossary pages to be indexed:

1. Expand content to 500+ words each
2. Add unique value - California-specific information, examples, case studies
3. Internal linking - Link from practice area pages to relevant glossary terms
4. FAQs - Add frequently asked questions to each glossary page

Summary of All Issues

Action Items

Immediate (No Action Required)

• [x] Redirects (40) - Correct configuration
• [x] 401 page (1) - Fixed via robots.txt
• [x] noindex pages (2 found) - Intentionally excluded

Review Recommended

• [ ] Check GSC for 8 alternate canonical pages - Verify each is intentional
• [ ] Check GSC for 4 not-indexed pages - Identify which specific pages

Optional Improvements

• [ ] Add noindex to /404error.html
• [ ] Add noindex to Google verification HTML files
• [ ] Consider expanding glossary page content (long-term)

Pages NOT in Sitemap (But Exist)

These pages exist on the site but are not in sitemap.xml:

Sitemap Recommendations

Consider adding to sitemap:

1. /case-evaluation/how-long-do-i-have-to-file.html
2. /case-evaluation/how-much-is-my-case-worth.html
3. /free-consultation/what-happens-when-i-call.html
4. Any newer blog posts not yet included

Appendix: File Inventory

Total HTML Files: 128

robots.txt Current Configuration

Sitemap: https://sempers.com/sitemap.xml
User-agent: *
Crawl-delay: 10
Disallow: /feedback/
Disallow: /assets/includes/
Disallow: /assets/guides/

Related Documents

• STATUS_OF_INDEXING_REPAIR.md - Soft 404 and 401 fixes
• DUPLICATE_CANONICAL_INVESTIGATION.md - Relative canonical URL issues
• NGINX_404_INVESTIGATION.md - 404 handling investigation

Last updated: January 26, 2026