What is the robots.txt code in Google Webmaster Tools?

  • 0
  • 2
  • Question
  • Updated 7 years ago
  • Answered
  • (Edited)
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members.

I have had a sudden drop of traffic and went into google webmaster and found this, anyone have any idea what this is?

# Block a bot that was causing issues by ignoring Disallow lines below
User-Agent: OmniExplorer_Bot
Disallow: /

# Block hotlinking of music files by projectplaylist.com due to perceived user bandwidth theft
User-agent: projectplaylist-directlink
Disallow: /

# Block all bots from the core Homestead site
User-agent: *
Disallow: /~site/Scripts_ElementMailer
Disallow: /~site/Scripts_ExternalRedirect
Disallow: /~site/Scripts_ForSale
Disallow: /~site/Scripts_HitCounter
Disallow: /~site/Scripts_NewGuest
Disallow: /~site/Scripts_RealTracker
Disallow: /~site/Scripts_Track
Disallow: /~site/Scripts_WebPoll
Photo of Russell5785

Russell5785

  • 452 Points 250 badge 2x thumb

Posted 7 years ago

  • 0
  • 2
Photo of Chris Folmer

Chris Folmer, Alum

  • 28,258 Points 20k badge 2x thumb
Hi Russell,

Thanks for posting!

What you are seeing is a robots.txt file. This file is on the backend of your website so when spiders crawl your site, they know what pages they shouldn't be going to. This wouldn't have any effect on your ranking as it doesn't contain your pages, but rather internal homestead directory pages that have no_follow attributes.

For more information, please check out: http://en.wikipedia.org/wiki/Robots_e...

Thanks!

Chris
Photo of Russell5785

Russell5785

  • 452 Points 250 badge 2x thumb
Thanks Chris

Why do you think this is in my store though? If you look it has some .com music download site? Do you think that this will harm my traffic in any way and should I be concerned about it?
Photo of Chris Folmer

Chris Folmer, Alum

  • 28,258 Points 20k badge 2x thumb
Hi Russell,

Thanks for the reply.

This definitely isn't anything you need to be concerned about. This file is on every homestead website. Here is one of my sites for example:

http://www.puppy-4-sale.net/robots.txt

The music site projectplaylist (now just playlist.com) was stealing bandwidth from other Homestead sites so they were added to the robots file to block access. All other websites in the file are elements within the website builder that don't need crawled.

Thanks!

Chris
Photo of Nicholas6170

Nicholas6170

  • 212 Points 100 badge 2x thumb
Hello Russel,
I just received a notice from Google saying it could not crawl my website
because the Block a bot was causing issues by ignoring Disalllow Lines below just like you showed above. Could you please let me know how do you fix this.
Photo of Russell5785

Russell5785

  • 452 Points 250 badge 2x thumb
Nic

Read the two replies from Chris Folmer above, It appears to be nothing to be concerned about.
Photo of Barbara6638

Barbara6638

  • 122 Points 100 badge 2x thumb
Google says my site has the following blocked URLs:

# Block a bot that was causing issues by ignoring Disallow lines below
User-Agent: OmniExplorer_Bot
Disallow: /

# Block hotlinking of music files by projectplaylist.com due to perceived user bandwidth theft
User-agent: projectplaylist-directlink
Disallow: /

# Block all bots from the core Homestead site
User-agent: *
Disallow: /~site/

It's the third one I'm concerned about since I see the others listed above. For my site, does this mean my whole site is blocked from Google? Do I have to have a robot.txt? How can I get rid of it? Google says a site doesn't need such a file in most cases.
Photo of Barbara6638

Barbara6638

  • 122 Points 100 badge 2x thumb
I wanted to add that my site is about a year old now and doesn't come up at all in Google searches unless I put in more than my one key word which I was hoping would be more successful. In fact, Google search results list sites that are under construction or very poorly constructed sites (in my opinion) whereas mine isn't even listed. What's up with that? That's why I'm wondering if I have a problem with the robot.txt file (see above).

Interestingly, my site appears on page 5 when I do a search in Bing/Yahoo.

Please help me.

This conversation is no longer open for comments or replies.