Gary Illyes from Google posted a brand new PSA on LinkedIn saying that the commonest cause a website unexpectedly blocks Googlebot from crawling is because of a misconfiguration of a firewall or CDN.
Gary wrote, “examine what site visitors your firewalls and CDN are blocking.” “By far the commonest challenge in my inbox is said to firewalls or CDNs blocking googlebot site visitors. If I attain out to the blocking website, within the overwhelming majority of the circumstances the blockage is unintended.”
So what are you able to do? Gary mentioned, “I’ve mentioned this earlier than, however need to emphasize it once more: make a behavior of checking your block guidelines. We publish our IP ranges so it ought to be very simple to run an automation that checks the block guidelines towards the googlebot subnets.”
Gary linked to this assist doc for extra particulars.
Briefly, do what you possibly can to check to see in case your website is accessible to Googlebot. You should use the URL inspection instrument in Google Search Console, as one technique. Additionally, verify together with your CDN or firewall firm that they’re permitting Googlebot and ask them to show it.
Discussion board dialogue at on LinkedIn.