"Submitted URL blocked by robots.txt" and "Indexed, though blocked by robots.txt"

  • 0
  • 1
  • Problem
  • Updated 1 year ago
  • Acknowledged
Archived and Closed

This conversation is no longer open for comments or replies and is no longer visible to community members.

Elizabeth asked me to secure the entire site with HTTPS, so I followed the instructions provided.
  • I made sure the setting showed Redirect no WWW to WWW.
  • I copied and pasted the provided HTTPS robots.txt.
  • I went to Google Console and made sure the property https://www.crimetech.net was included.
  • I submitted the sitemap for the above property. 
  • 830 pages have the red error "Submitted URL blocked by robots.txt." It hasn't budged an inch since May 17.
  • 690 pages have the yellow warning "Indexed, though blocked by robots.txt."
  • I double-checked using JitBit. Everything came up with errors.
I've scoured the community at Homestead and BigCommerce. I don't know what is wrong or what to do. Could someone give me a hand please?

Thanks!

Here is a copy of the robot.txt:

User-agent: AdsBot-Google
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /finishorder.php
Disallow: /login.php
Disallow: /orderstatus.php
Disallow: /postreview.php
Disallow: /productimage.php
Disallow: /productupdates.php
Disallow: /remote.php
Disallow: /search.php
Disallow: /viewfile.php
Disallow: /wishlist.php
Disallow: /admin/
Disallow: /__socialshop/

User-agent: *
Disallow: /account.php
Disallow: /cart.php
Disallow: /checkout.php
Disallow: /finishorder.php
Disallow: /login.php
Disallow: /orderstatus.php
Disallow: /postreview.php
Disallow: /productimage.php
Disallow: /productupdates.php
Disallow: /remote.php
Disallow: /search.php
Disallow: /viewfile.php
Disallow: /wishlist.php
Disallow: /admin/
Disallow: /__socialshop/
 




Photo of Leslie0014

Leslie0014

  • 1,446 Points 1k badge 2x thumb

Posted 1 year ago

  • 0
  • 1
Photo of Elyzabeth

Elyzabeth , Official Rep

  • 46,374 Points 20k badge 2x thumb
Hello,

This is exactly how it should be set up. You don't want your cart, checkout, finish order, etc. pages to be indexed and there is no reason for AdsBot to reach these pages either. Nothing is wrong with this

This conversation is no longer open for comments or replies.