BlendInteractive.Umbraco.RobotsTxt 17.0.0

Blend Interactive - Umbraco - RobotsTxt

This is a lightweight package that enables /robots.txt root of the umbraco website. This package is configured using appSettings.

Installation in Umbraco CMS

Command Line

donet add package BlendInteractive.Umbraco.RobotsTxt

Or Nuget

Install-Package BlendInteractive.Umbraco.RobotsTxt

Package Name Update

Previously the package was known as Our.Umbraco.Blend.RobotsTxt. The package name has been updated to BlendInteractive.Umbraco.RobotsTxt as of version 13.1.0. If you are using a version prior to 13.1.0, please continue to use the old package name.`

Setup

In the Startup.cs there is a configuration you need to add for /robots.txt path to render.

In the app.UseUmbraco() Under .WithEndpoints(u => add:

u.EndpointRouteBuilder.MapControllers();

This will use the route /robots.txt declared in the controller.

Default

If there are not any configurations in the appSettings.json file and environment is NOT Production, the default robots.txt will be:

User-agent: *
Disallow: /

If the environment is Production the default robots.txt will be:

User-agent: *
Allow: /

Configuration

In the root of your appSettings.json you can configure custom settings. You can also use appSettings.[Environment].json to have specific settings for every environment.

"Robots": [
    {
      "Sitemaps": [ "/sitemap.xml" ],
      "Groups": [
        {
          "UserAgent": "*",
          "Disallow": [ "/" ]
        },
        {
          "UserAgent": "Bingbot",
          "Allow": [ "/" ]
        }
      ]
    },
    {
      "Domains": [ "localhost:44393", "localhost:44395" ],
      "Sitemaps": [ "/sitemap.xml", "/another-sitemap.xml" ],
      "Groups": [
        {
          "UserAgent": "*",
          "Allow": [ "/" ]
        }
      ]
    }
  ]

Robots is an array of objects to be configured as needed to your use case.

Domains is an optional string array. Specify the domains that the group of robot options will apply to along with the sitemaps.

• If left blank, the group will be classified as the default and will apply for all domains. If there are more than one that are default the first is used.

Sitemaps is an optional string array. If left blank will not include.

Groups is a required array of objects. Each object will discribe the robots.txt for the domains.

UserAgent is an optional string. If left blank will use *.

Allow is an optional string array. Array of paths to allow.

Disallow is an optional string array. Array of paths not to allow.

If Allow and Disallow are both empty, will set to Allow: /