How can I edit the robots.txt file?

  • 0
  • 1
  • Question
  • Updated 5 years ago
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members. The community moderator provided the following reason for archiving: We have archived this topic as it has either reached a resolution or has become inactive.  If you believe this topic should not have been archived, please let us know, and we would be happy to review it.  You may continue to access this topic as needed fo

I am trying to modify my http://www.opc-1-2-3.com/robots.txt file and I do not know how. It is not a StoreFront as shown above. It is just a content website. It is supposed to be located in the root directory of the file manager, but it is not there. The reason to ask for this is that a Chinese search engine (baidu.com) says that this site is not accessible because it is blocked by the robots.txt file, while there is not such a problem with all other search engines. It is strange. Thanks.

Note: This topic was created from a reply on the Can I modify the robots.txt file? topic.
Photo of Antioxidant

Antioxidant

  • 316 Points 250 badge 2x thumb

Posted 6 years ago

  • 0
  • 1
Photo of Mike F.

Mike F., Alum

  • 184,698 Points 100k badge 2x thumb
Sorry, but for the SiteBuilder sites (not Storefronts), the robots.txt file is not accessible or editable. That may be a false report. If you look at the rotots.txt file there is no reference to the baidu.com search engine.

Mike
Photo of Joe7378

Joe7378

  • 124 Points 100 badge 2x thumb
so will you please tell me what this does mean....thank you

# Block a bot that was causing issues by ignoring Disallow lines below
User-Agent: OmniExplorer_Bot
Disallow: /

# Block hotlinking of music files by projectplaylist.com due to perceived user bandwidth theft
User-agent: projectplaylist-directlink
Disallow: /

# Block all bots from the core Homestead site
User-agent: *
Disallow: /~site/Scripts_ElementMailer
Disallow: /~site/Scripts_ExternalRedirect
Disallow: /~site/Scripts_ForSale
Disallow: /~site/Scripts_HitCounter
Disallow: /~site/Scripts_NewGuest
Disallow: /~site/Scripts_RealTracker
Disallow: /~site/Scripts_Track
Disallow: /~site/Scripts_WebPoll
Disallow: /~site/siteapps/showmap.action
Photo of Drew N

Drew N, Alum

  • 262,816 Points 100k badge 2x thumb
This file is telling bots not to scan certain places on the website.  Mostly scripts.  Nothing that would have a negative impact on any of the search engines bots is being blocked.  The first 2 sections were added as a result of specific sites, either ignoring the disallow or stealing bandwidth.
Photo of Joe7378

Joe7378

  • 124 Points 100 badge 2x thumb
thats what i wanted to know...thank you
Photo of Drew N

Drew N, Alum

  • 262,816 Points 100k badge 2x thumb
You're welcome

This conversation is no longer open for comments or replies.