How to Use the robots.txt Editor

Introduction

A robots.txt file is a simple instruction file placed on your website that tells search engines such as Google, Bing, and other crawlers which pages or sections they can access and which they should ignore.

With Muvi, you can manage the robots.txt file for your front-end website when you are using a custom domain. This helps you control how search engines crawl your website and prevent indexing of unnecessary URLs.

Why Use the robots.txt Editor

SEO & Search Visibility

• Controls which pages search engine bots can crawl.
• Prevents indexing of unnecessary or duplicate pages.
• Helps prioritize important content for better SEO.

Performance & Resource Optimization

• Reduces server load from unnecessary bot requests.
• Helps search engines use their crawl budget more efficiently.

Security & Access Control

• Prevents admin, staging, or temporary folders from being crawled.
• Reduces the chance of backend URLs appearing in search results.

Search Engine Discovery

• Allows you to declare your sitemap for faster content discovery.

How to Add Custom Domain

• First, you need to add a custom domain before making changes to the robot.txt.
• To add a custom domain, navigate to “Website & Apps” --> Settings” in the Dashboard.
• Under “Domain Name”, click on the “Edit Icon” on the right-hand side of the current domain.
• In the popup, add your custom domain and click on “Save”.

How to Block Specific Pages

You can block specific pages from the search engine crawler using the robot.txt editor.

• To block a page, navigate to “Marketing” --> SEO” in the Dashboard.
• Under “Search Engine Optimization”, scroll down to “txt Editor”.
• In the “Textbox” on the right-hand side of “Disallow:” type the page that you want to block.
• For example: if you want to block the “Admin” section of your website, then type “/admin/”. Search engines will avoid crawling URLs such as: https://yourdomain.com/admin/, https://yourdomain.com/admin/settings
• You can add multiple URLs.
• Click on “Update”.

How to Allow All Pages

You can allow the search engine bots to crawl all pages of your website.

• To allow the search engine bots to crawl all pages, navigate to “Marketing” --> "SEO” in the Dashboard.
• Under “Search Engine Optimization”, scroll down to “txt Editor”.
• In the “Textbox” on the right-hand side of “Disallow:”, keep everything blank.
• User-agent: * → applies to all search engine bots.
• Empty Disallow: → nothing is blocked.
• Click on “Update”.