Robots.txt fetch

  • 0
  • 1
  • Question
  • Updated 4 months ago
  • Acknowledged
  • (Edited)
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members. The community moderator provided the following reason for archiving: Archived

One of my sites on the Google webmaster tool site shows no errors during crawls. Another site https://www.drywallcarolina.com/ receives this message.

"Google couldn't crawl your site because we were unable to access your site's robots.txt file."

"Your server returned a 5xx (unreachable) error when we tried to retrieve your robots.txt file"

According to the More Info page: 
"If your site shows a 100% error rate any of the three categories, it likely means that your site is either down or misconfigured in some way."

This site showed a green and good for DNS and server connectivity but a 100% error for robots.txt. What is robots.txt and is there anything I can do to resolve the problem or is it something that has to be handled on your end?

According to Google:
Fixing robots.txt file errors
  • You don't always need a robots.txt file.
    You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file—not even an empty one. If you don't have a robots.txt file, your server will return a 404 when Googlebot requests it, and we will continue to crawl your site. No problem.
  • Make sure your robots.txt file can be accessed by Google.
    It's possible that your server returned a 5xx (unreachable) error when we tried to retrieve your robots.txtfile. Check that your hosting provider is not blocking Googlebot.  If you have a firewall, make sure that its configuration is not blocking Google.
Photo of steve4468

steve4468

  • 1,354 Points 1k badge 2x thumb

Posted 4 months ago

  • 0
  • 1
Photo of Elyzabeth

Elyzabeth , Official Rep

  • 17,578 Points 10k badge 2x thumb
Hello steve4468

Taking a look at https://www.drywallcarolina.com/robots.txt , I'm not seeing anything that would cause an issue. The few blocked items should be of no real interest to Google

There is no way you can edit it, but it is not blocking Google at all.
Photo of steve4468

steve4468

  • 1,354 Points 1k badge 2x thumb
Thank you, good to know. I'll give it a day or so to re-check crawl errors and hopefully straighten itself out. If it doesn't I'll contact you again. 
Now I'm not really saying anything, just throwing questions around but could this problem have anything to do with Cloudflare? I didn't seem to have the problem in the past but have recently switched over to https through them. Just trying to cover all bases.
Photo of Elyzabeth

Elyzabeth , Official Rep

  • 17,578 Points 10k badge 2x thumb
Good question, but no it shouldn't cause an issue. There is nothing linking to Cloudflare in your robots.txt file.
Photo of steve4468

steve4468

  • 1,354 Points 1k badge 2x thumb
Thank you. I appreciate you looking into it.

This conversation is no longer open for comments or replies.