You’re so devoted to creating extra edgy and updated content for your website but do you still feel there is a huge gap that comes in when Googlebot is trying to crawl on your website or do you often feel clueless about google crawling behavior?

Don’t you worry, We have got you armored.

This issue was recently addressed by the one and the best John Mueller – Google’s Search Advocate, during the Google Search Central SEO office-hours hangout recorded on January 7.

Swyamdipta Chakraborty, an SEO professional, joins the broadcast to ask Mueller a series of questions, including how frequently Googlebot crawls his site.

Googlebot crawled his site every day when he published more frequently, but not as much when he published less frequently. He inquires about this with Mueller, maybe worried that a decrease in crawl frequency is a bad sign.

Mueller assures him that everything is fine and then goes on to explain the two types of crawling that Googlebot performs.

So, there are 2 types of Googlebot crawling, which Mueller talked about, let’s dive in to see what exactly they are.

Two types of Googlebot crawling

Google uses two sorts of web crawling: one for discovering new content and the other for refreshing previously published content. 

Google is a big fan of new and relevant content. The crawler will be interested if you update existing content or create new ones. The more often you’re crawled, the more opportunities you’ll have to improve your performance. However, this is only true if you provide high-quality updates. Always ensure that your content is well-written and free of unwanted keywords. Content that is poorly written will only have a negative impact.

When asked about the issue, Mueller says the changes are natural and goes into detail related to the two kinds of crawling

“That can happen. It’s not so much that we crawl a website, but we crawl individual pages of a website. And when it comes to crawling, we have two types of crawling.

One is a discovery crawl where we try to discover new pages on your website. And the other is a refresh crawl where we update existing pages that we know about.”

Crawl frequency might vary not only for the entire site but also for specific web pages.

If your site’s about us page, for example, is updated more frequently than other pages, you’ll notice more Googlebot activity on that page

Mueller continues:

“So for the most part, for example, we would refresh crawl the homepage, I don’t know, once a day, or every couple of hours, or something like that.

And if we find new links on their home page then we’ll go off and crawl those with the discovery crawl as well. And because of that you will always see a mix of discovery and refresh happening concerning crawling. And you’ll see some baseline of crawling happening every day.

But if we recognize that individual pages change very rarely, then we realize we don’t have to crawl them all the time.”

So, it means that crawling frequency is directly proportional to the refreshing or updating frequency of your website, the more you keep your website updated and fresh the more Googlebot, hang around, and crawl.

Consider your website a restaurant which changes and refreshes its menu now and then to give the best possible taste to their customers, obviously as a customer they’ll like to hang more in your restaurant, So, in this case, Googlebot is the customer who can give you huge ratings and opportunity to get good traffic, and your restaurant is a website which is fresh and updated according to the customers need. So, it’s very natural for Googlebot to crawl your website or in this case (eat in your restaurant).

Similarly, Googlebot learns and observes the content, recognizes the pattern, and adjusts its crawl frequency accordingly.

“For example, if you have a news website and you update it hourly, then we should learn that we need to crawl it hourly. Whereas if it’s a news website that updates once a month, then we should learn that we don’t need to crawl every hour.

And that’s not a sign of quality, or a sign of ranking, or anything like that. It’s just purely from a technical point of view we’ve learned we can crawl this once a day, or once a week, and that’s ok.”

So don’t be frightened if Googlebot appears to be visiting your site more or less frequently.

Furthermore, if Googlebot recently crawled your site and improvements to current content aren’t showing up in search results, don’t be afraid.

It’s possible that Google inspected your site to find new content rather than to update existing content.

If your site’s published content is barely updated, Googlebot may explore more for discovery that refreshes.

It has nothing to do with the quality of the content, yet again.