Google’s Gary Illyes warns about the impact of soft 404 errors on web crawling and recommends proper error handling to improve SEO and site efficiency.
Soft 404 errors mislead web crawlers, wasting valuable resources. Proper HTTP status codes are crucial for efficient crawling. Addressing soft errors can significantly improve SEO and site performance. In a recent LinkedIn post, Google Analyst Gary Illyes highlighted two key issues affecting web crawlers: soft 404 errors and other “crypto” errors.
These seemingly minor mistakes can hurt SEO efforts.
Understanding Soft 404s
Soft 404 errors occur when a web server returns a standard “200 OK” HTTP status code for pages that don’t exist or contain error messages. This misleads web crawlers, causing them to waste resources on non-existent or unhelpful content. Google Analyst Gary Illyes likened the experience to visiting a coffee shop where every item is unavailable despite being listed on the menu. While this scenario might be frustrating for human customers, it poses a more serious problem for web crawlers.
As Illyes explains:
“Crawlers use the status codes to interpret whether a fetch was successful, even if the page’s contents are basically just an error message. They might happily go back to the same page repeatedly, wasting your resources, and if there are many such pages, exponentially more resources.”
The Hidden Costs Of Soft Errors
Soft 404 errors carry consequences beyond the inefficient use of crawler resources. According to Illyes, these pages often fail to appear in search results because they are filtered out during indexing.
Illyes recommends serving the correct HTTP status code whenever the server or client encounters an error to address this issue. This approach enables crawlers to assess the situation more accurately and allocate their resources more effectively.
Illyes also warned against using messages like “TOO MANY REQUESTS SLOW DOWN” to rate-limit crawlers, as such text-based instructions are not interpretable by crawlers.
Why We Care
Soft 404 errors can significantly impact a website’s crawlability and indexing. Addressing these issues allows crawlers to prioritize fetching and indexing pages that offer valuable content, potentially enhancing the site’s visibility in search results.
Moreover, eliminating soft 404 errors promotes the efficient use of server resources. Crawlers avoid wasting bandwidth by repeatedly visiting error pages, leading to better resource management overall.
How This Can Help You
To effectively manage and resolve soft 404 errors on your website, consider the following steps:
- Monitor Crawl Reports and Logs: Regularly check your website’s crawl reports and server logs to identify pages that return HTTP 200 status codes despite containing error messages.
- Implement Proper Error Handling: Ensure your server is configured to serve error pages with the correct HTTP status codes (e.g., 404 for not found, 410 for permanently removed). Use tools like Google Search Console to monitor your site’s coverage and identify pages flagged as soft 404 errors.
- Proactive Resolution: Taking proactive steps to address soft 404 errors can significantly enhance your website’s crawlability, indexing efficiency, and overall SEO performance.
Following these practices can optimize your site’s performance and ensure that crawlers prioritize valuable content over erroneous pages.
If you’re finding it challenging to navigate and resolve these issues on your own, consider exploring our monthly SEO packages. Our experts can help you in managing and optimizing your website’s SEO effectively.
