How to avoid annoying Magento URL params in search engine index

by Hendrik Will. 7 Comments

For my latest Magento ecommerce project I just noticed that search engines indexed some URLs with annoying params like ?dir=asc&order=position or ?limit=9. Also some unnecessary URLs like catalog search or checkout page.

Usually we don’t want to Google to index these pages because it could be duplicated content which can be bad for our ecommerce shop‘s ranking. One option would be to have nofollow links in your template but the easier way is to build a robots.txt file your Magento ecommerce shop. I also included some other important directories we usually don’t want to have indexed by search engines.

Here is my robots.txt example:

User-agent: *
Disallow: /index.php/
Disallow: /*?
Disallow: /*.js$
Disallow: /*.css$
Disallow: /404/
Disallow: /admin/
Disallow: /api/
Disallow: /app/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product_compare/
Disallow: /catalogsearch/
Disallow: /catalogsearch/advanced/
Disallow: /catalogsearch/term/popular/
Disallow: /cgi-bin/
Disallow: /checkout/
Disallow: /checkout/cart/
Disallow: /contacts/
Disallow: /contacts/index/
Disallow: /contacts/index/post/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
Disallow: /downloader/
Disallow: /install/
Disallow: /images/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /media/
Disallow: /newsletter/
Disallow: /pkginfo/
Disallow: /private/
Disallow: /poll/
Disallow: /report/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /skin/
Disallow: /tag/
Disallow: /var/
Disallow: /wishlist/
Disallow: /anyothercontentyouwouldliketodisallow/
Sitemap: http://www.your-amazing-magento-ecommerce-shop.com/sitemap.xml

7 Responses to How to avoid annoying Magento URL params in search engine index

  1. Kaylaa says:

    Thanks so much, this is really helpful. I am a magento newbie, and I know I wanted to disallow some things in the robots.txt file, but didn’t think about this much. Now that I have seen your list, it put a lot in perspective and is really helpful. Thanks again. Great Work!
    ~Kaylaa

  2. Chandlery says:

    Thanks for a REALLY usful post. I’ve just started to notice a lot of search results pages being indexed by google, so this was a real help. Thanks…

  3. Sean says:

    @LarryE

    As long as you are generating a valid sitemap and have submitted it to Google, this should be just fine. It’s one of the least restrictive exclusion files I’ve come across.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>