Creating a WordPress Robots.txt - Every Method

Another thing you need to do to make your website rank higher in search engines is to show the way to bots. Bots play a big role in page discovery. For this you need to create the robots.txt file and configure it correctly. You can continue reading this article to learn how to do this.

What is Robots.txt?

Search engines crawl websites through bots. WordPress robots.txt file It instructs search engines on how to behave when crawling. Robots.txt file Even if you don't create one, your site will be crawled by search engines. Don't forget this.

How to Create Robots.txt

You can create a robots.txt file in WordPress manually or via a plugin.

Creating WordPress Robots.txt

Creating Robots.txt with Rank Math SEO

Rank Math SEO plugin helps you achieve internal SEO optimization in your posts and pages. This way, you can do more successful SEO work. The same goes for creating a robots.txt file. After installing and activating the plugin, click on General Settings from the Rank Math SEO menu of the admin panel. Click the Edit Robots.txt button to access the Robots.txt editing screen. If your write/read permissions are correct, you can edit your virtual robots.txt file here. If you already have a robots.txt file, this section will not apply.

Creating Robots.txt via FTP

Open Notepad and write your robots.txt rules in its content. If you do not know what to write, you can use the recommended robots.txt rules we mentioned in the following section of our article. Save the file in .txt extension and connect to the root directory of your site (usually public_html) via FTP connection and upload the file.

Creating Robots.txt with File Manager

After logging in to the hosting control panel you use (cpanel, plesk, directadmin, cwp, etc.), open the file manager. Access your site's root directory (usually public_html) from the file manager. From the new file creation menu, click create new file and set the file extension to .txt. Write robots.txt rules and save the file.

Creating Robots.txt with Robots.txt Editor Plugin

Robots.txt editor plugin It allows you to easily create and edit robots.txt files without a file manager or ftp access.

Plugin page -> https://wordpress.org/plugins/robots-txt-editor/

After downloading the plugin, activate it. Then access the Reading page under Settings using the WordPress left sidebar. Write your robots.txt rules in the box titled robots.txt at the bottom of the page and save them.

WordPress Robots.txt Rules

Recommended robots.txt rules for WordPress are as follows. You need to write your own site address instead of domain.com, and the sitemap.xml url you use instead of sitemap.xml. We would like to give you an important tip: If you are using the Rank Math SEO or Yoast SEO plugin, your sitemap will be created as sitemap_index.xml, not sitemap.xml. Don't forget to check the URL you wrote by entering it in the browser.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.domain.com/sitemap.xml

Get Free Support Creating Robots.txt

If you could not create the robots.txt file or would like to get free support about the robots.txt file, please state your problem with a detailed explanation in the comments section. We will respond to your comment within 24 hours and offer free support.

Get Professional Support to Create Robots.txt

If you could not create the robots.txt file or would like to get professional support about the robots.txt file, enter your name and phone number in the form below. Let us reach you as soon as possible.