How Do I Prevent Duplicate Content By Blocking Google From My Index Page?

  • 0
  • 2
  • Question
  • Updated 4 years ago
  • Answered
  • (Edited)
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members. The community moderator provided the following reason for archiving: Archived

Changing the robots.txt file will prevent your site from being crawled by search engines. Do not add something unless you are positive you do not want it to be seen in search results.

When building your store with Bigcommerce, there is a slim possibility that your store will appear in a search engine before it is ready to go live. This can lead to customers viewing your store while it is still under construction, and if you are recycling information from an existing store, Google may penalize you for duplicate content. You can update your robots.txt file to prevent search engines from indexing your site before it is ready. However, you must change it back once your site is live.

Modifying Robots.txt 1. In the Bigcommerce control panel, go to Setup & Tools › Advanced Tools › Robots text file.

2. Delete everything after the first Disallow:. See the example below./

Robotstxt with everything disallowed

3. Click Save when done.

4. To allow Google to index your store again, return to this page and click Revert to Default, thenSave.

The Revert to Default button at the bottom of the page

Photo of Drew N

Drew N, Community Manager

  • 252,538 Points 100k badge 2x thumb

Posted 5 years ago

  • 0
  • 2
Photo of S

S

  • 6,450 Points 5k badge 2x thumb
Is this still applicable if the BC store will be 'stores.domain.com' or under what conditions might this be applicable?
Photo of Mike F.

Mike F., Alum

  • 184,338 Points 100k badge 2x thumb
This is not related to the "stores" subdomain. The "index" file that is being references is a directory folder and not related. Google is pretty secretive about if, or how much this might affect you but to be safe you might go ahead and do the steps above.

Personally, I doubt Google will penalize you (at least not very much) as this would affect millions and millions of sites as every site has an index file.
Photo of Kenny8864

Kenny8864

  • 80 Points 75 badge 2x thumb
I have a blog with my homestead account, and in my google webmaster tools it is cited as having duplicate content - presumably because the title of the blog is automatically repeated with each entry.

Is there any way around this?
Photo of Mike F.

Mike F., Alum

  • 184,338 Points 100k badge 2x thumb
Hi Kenny8864, I have not seen this before, but with all the changes Google continuously makes this would not surprise me. I have someone from our SEO team looking into this for us now. I'll let you know as soon as I hear back.

Mike
Photo of Kenny8864

Kenny8864

  • 80 Points 75 badge 2x thumb
Thanks Mike, appreciate it. Having the ability to optimize each blog entry individually (header & meta tags) should overcome this I'd imagine?
Photo of Mike F.

Mike F., Alum

  • 184,338 Points 100k badge 2x thumb
Hi Kenny8864, I'm sure this is not the answer you were hoping to hear, but, there's not really a way to optimize the individual entries. If the posts are primarily made by yourself, you may want to just create new pages which you manually link to simulate blog functionality.

"Only way around that is to make a new page for each post instead of a blog. You could then have an articles page that links to all the posts."
Photo of Kenny8864

Kenny8864

  • 80 Points 75 badge 2x thumb
That's a good workaround Mike, I may have to start working on that.
Photo of Jill1739

Jill1739

  • 1,848 Points 1k badge 2x thumb
I have tried to research this and from what i'm reading they say NOT to disalllow index.php
Photo of Drew N

Drew N, Community Manager

  • 252,538 Points 100k badge 2x thumb
Have you already setup a Google Webmaster tools account? This will help you identify if Google is seeing duplicate content. If not, then there is no reason to do anything. If you are then the individual issues can be addressed.

Drew
Photo of Jill1739

Jill1739

  • 1,848 Points 1k badge 2x thumb
under html improvements this is the only stuff i can see.

HTML Improvements Last updated Jan 16, 2014

Addressing the following may help your site's user experience and performance.
Meta description Pages
Duplicate meta descriptions 28
Long meta descriptions 0
Short meta descriptions 63

Title tag Pages
Missing title tags 0
Duplicate title tags 13
Long title tags 0
Short title tags 0
Non-informative title tags 0

Non-indexable content Pages
We didn't detect any issues with non-indexable content on your site.

Where would i go to identify the if the index.php is considered duplicate?
Photo of Drew N

Drew N, Community Manager

  • 252,538 Points 100k badge 2x thumb
Those reported items would seem more like the problem is that you have the same page title and or description. Do you have products with the same descriptions or product name? You can check on products in the Other details section and Search engine optimization. Also, check the Title and meta description of your other pages by looking at Web content > View pages.

Drew
Photo of Jill1739

Jill1739

  • 1,848 Points 1k badge 2x thumb
Yeah I know how to do that stuff i fixed alot of them google just hasn't updated the information yet. I was just curious about the index.php whether or not it is hurting me?
Photo of Drew N

Drew N, Community Manager

  • 252,538 Points 100k badge 2x thumb
I don't see indication of it at this point. I would let it be until we know Google has updated the majority of links and pages, then you will get better reporting.

Drew
Photo of S

S

  • 6,450 Points 5k badge 2x thumb
I have not added this code and according to what I read here, it is likely not necessary. I (feeling fortunate) have only 3 duplicate title tags in Webmaster Tools and all three reference an old ProStores page (snippet below). All three pages (a sample pack...lol...one Page, one Category and one Product) have all been redirected to the new website pages since the beginning (hence the duplicate titles) - anything I should consider doing?

Thanks!
Photo of Mike F.

Mike F., Alum

  • 184,338 Points 100k badge 2x thumb
Hi Sally, Basically you have nothing to worry about. The 301 redirects tell Google that the old page is no longer relevant and that most relevant information is found on the new page. It's likely that given a little more time Google will probably just drop those old URL's from its index. However if there are any 3rd party sites that link to those particular pages they may show up for a while, but as long as the 301 redirects are in place the link authority is passed to the new page.
Photo of S

S

  • 6,450 Points 5k badge 2x thumb
FYI...my redirect errors have escalated to over 13,000. It is very misleading to tell everyone that their stores are all 301 redirected. If anyone did any changes to the search keywords in PS which created the URL - the product ID now effectively has two different URLs that need to be redirected (if they made 10 changes over the lifetime of that product ID...well, you get my drift. Just saying...

This conversation is no longer open for comments or replies.