Robots.txt for Shopify: A Practical Guide

Why Your Shopify Robots.txt Matters for SEO

Googlebot has a finite attention span for your site; we call this crawl budget. If the bot wastes time on faceted navigation pages (like /collections/shirts?color=blue), it might skip your actual product pages. I’ve seen stores where 80% of the crawled URLs were parameter-driven junk. A tight robots.txt Shopify store setup prevents this. It works hand-in-hand with canonical tags to consolidate signals, but blocking is faster than hoping Google respects a canonical.

Usually, Shopify handles the heavy lifting by default. The platform automatically blocks the checkout, cart, and internal search results. But the trouble starts when you install apps that generate new URLs, or when your theme creates infinite crawl spaces. That's when you need to jump in.

Shopify Robots.txt Example and Common Rules

A standard Shopify robots.txt looks something like what you see below. The first block is locked by Shopify. The second block is where you can add your custom instructions. Don’t try to override the system block; you can’t. You can only build upon it.

# System rules are managed by Shopify
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/articles/*+*
Disallow: /*/articles/*%2B*
Disallow: /*/articles/*%2b*
Disallow: /search
Disallow: /apple-app-site-association

# Custom rules you can edit below
User-agent: *
Disallow: /collections/fragrance*?*filter*
Sitemap: https://your-store.com/sitemap.xml

Notice the custom line blocking collection filters. That's a practical fix for a very common Shopify crawl issue.

How to Block Specific Pages from Indexing

Maybe you have a vendor page that creates duplicate content, or a collection page that you’d rather keep hidden from SERPs. You can’t rely on the “noindex” tag alone if bots are still wasting time crawling the URL.

To add a custom disallow rule, you need to edit the robots.txt.liquid file. Just remember, a "Disallow" line stops crawling, but it doesn’t stop indexing. This is a tricky part. If Google discovers that blocked URL through an internal link elsewhere, it might still index it without visiting it, showing that ugly "No information available" snippet. Usually, you want to pair a `noindex` meta tag with a `Disallow` rule for full protection.

Blocking the Shopify Blog (If You Don't Use It)

If your theme comes with a news section you never touched, block it. It’s thin content.

Disallow: /blogs/news/

Fixing Shopify Crawl Issues and Indexing Problems

Most crawling chaos on Shopify comes from two things: faceted navigation and internal search parameters. Even though Shopify tries to block some of these with patterns like /*sort_by*, some themes use different URL structures.

Check Google Search Console. If you see thousands of excluded URLs with long parameter strings, your robots.txt might be fine, but your internal links are out of control. Robots.txt is a band-aid here. The real fix is usually tweaking how your theme links to filtered options, but let's be real, that's a bigger dev task. A strict disallow pattern is a fast, safe fix.

Another thing worth checking: is your sitemap and robots.txt in sync? Your robots.txt should point clearly to your sitemap.xml. Never block the URL path that contains your sitemap.

Handling Duplicate Pages SEO on Shopify

Shopify is notorious for generating the same product page under different collection handles. For example: /collections/shoes/products/nike-air vs /collections/sale/products/nike-air. The canonical tag usually points back to the base product, which is good. But if you’ve got a specific collection path that’s creating a mess, you can use the robots.txt to disallow that specific product-under-collection path, though honestly, it’s often better to let Googlebot figure it out via canonicals unless the scale is massive and eating your crawl optimization for Shopify.

Protecting Your Store's Staging Domain

If your development store is on a myshopify.com domain and you’ve forwarded a custom domain, make sure the staging domain isn’t accessible. If you can't fully firewall it, you might want to use Liquid logic in your robots.txt to block everything if the domain isn't your primary one.

Best Practices for a Clean Robots.txt

• Always Test: Use Google’s robots.txt tester in Search Console before publishing changes. A misplaced slash can block your entire store.
• Don't Block CSS/JS: Shopify hosts assets on CDNs. Don’t try to control those files. Google needs them for rendering.
• Link to the Sitemap: Your custom robots.txt should explicitly state the location of your Shopify sitemap and robots.txt file relation at the very bottom.
• Use Liquid Logic Carefully: You can use {% if %} statements to hide rules from non-production environments, which is a nice trick for development stores.