Can I modify the robots.txt file?

  • 0
  • 3
  • Question
  • Updated 2 years ago
  • Answered
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members. The community moderator provided the following reason for archiving: Archived

To edit this file please login into the control panel and go to: > Setup & Tools > Robots.txt File
tools_robotsfile.png

- We do not recommend making changes to this file, and it will directly affect what pages are crawled by search engines.

However, this would be the place to make any changes.
robots_file.png

After you make your changes you > Save them.
robots_save.png



At default the file will contain the following:

User-agent: *

Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /finishorder.php
Disallow: /login.php
Disallow: /orderstatus.php
Disallow: /postreview.php
Disallow: /productimage.php
Disallow: /productupdates.php
Disallow: /remote.php
Disallow: /search.php
Disallow: /viewfile.php
Disallow: /wishlist.php
Disallow: /admin/
Disallow: /*sort=
Disallow: /__socialshop/


 

There is also a file located at http://www.yourdomain.com/robots_ssl.txt which contains and prevents any secure version of a webpage from being crawled.

Photo of Drew N

Drew N, Community Manager

  • 258,498 Points 100k badge 2x thumb

Posted 5 years ago

  • 0
  • 3
Photo of Antioxidant

Antioxidant

  • 316 Points 250 badge 2x thumb
I am trying to modify my http://www.opc-1-2-3.com/robots.txt file and I do not know how. It is not a StoreFront as shown above. It is just a content website. It is supposed to be located in the root directory of the file manager, but it is not there. The reason to ask for this is that a Chinese search engine (baidu.com) says that this site is not accessible because it is blocked by the robots.txt file, while there is not such a problem with all other search engines. It is strange. Thanks.
Photo of Joe7378

Joe7378

  • 124 Points 100 badge 2x thumb
ok....what would the appropriate topic be then...thanks
Photo of Drew N

Drew N, Community Manager

  • 258,468 Points 100k badge 2x thumb
Sorry. Wrong link
(Edited)
Photo of Mary-Jane2063

Mary-Jane2063

  • 120 Points 100 badge 2x thumb
I am having the same problem and can't find the robots.txt file. I am not a store front either. Where can I find this file.
Photo of William S

William S

  • 18,858 Points 10k badge 2x thumb
Hello Mary-Jane,

If you do not have a Storefront, you would not be able to edit your robots.txt file. It is in your root folder, but it is hidden and can't be edited.
Photo of Dale7733

Dale7733

  • 60 Points
I get the following message on google webmaster after verifying my site:
Googlebot cannot access your JavaScript and/or CSS files because of restrictions in your robots.txt file. These files help Google understand that your website works properly so blocking access to these assets can result in suboptimal rankings.

Why can't this be changed, I know I can't edit this on my end because I don't have access to the code.
Looking on the web I see how it can be corrected:
There’s an easy fix for it, which involves editing your site’s robots.txt file. If you’re comfortable editing that file, then go ahead with this fix.

Look through the robots.txt file for any of the following lines of code:

Disallow: /.js$*

Disallow: /.inc$*

Disallow: /.css$*

Disallow: /.php$*

If you see any of those lines, remove them. That’s what’s blocking Googlebot from crawling the files it needs to render your site as other users can see it.

Can this be done on homestead side so I can get better SEO?
Photo of Drew N

Drew N, Community Manager

  • 258,498 Points 100k badge 2x thumb
You are not able to edit this file. But our robots.txt file is very basic and blocks none of what you have posted.

# Block a bot that was causing issues by ignoring Disallow lines below
User-Agent: OmniExplorer_Bot
Disallow: /

# Block hotlinking of music files by projectplaylist.com due to perceived user bandwidth theft
User-agent: projectplaylist-directlink
Disallow: /

# Block all bots from the core Homestead site
User-agent: *
Disallow: /~site/Scripts_ElementMailer
Disallow: /~site/Scripts_ExternalRedirect
Disallow: /~site/Scripts_ForSale
Disallow: /~site/Scripts_HitCounter
Disallow: /~site/Scripts_NewGuest
Disallow: /~site/Scripts_RealTracker
Disallow: /~site/Scripts_Track
Disallow: /~site/Scripts_WebPoll
Disallow: /~site/siteapps/showmap.action

These are not going to negatively affect your SEO.

This conversation is no longer open for comments or replies.