RobotsTxt 2014.2.19

A robots.txt parser for .NET

Supports ;
- Allow directives.
- Crawl-delay directives.
- Sitemap declarations.
- * and $ wildcards.

See for usage examples.

Install-Package RobotsTxt -Version 2014.2.19
dotnet add package RobotsTxt --version 2014.2.19
<PackageReference Include="RobotsTxt" Version="2014.2.19" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add RobotsTxt --version 2014.2.19
The NuGet Team does not provide support for this client. Please contact its maintainers for support.


This package has no dependencies.

Showing the top 1 GitHub repositories that depend on RobotsTxt:

Repository Stars
.NET based webcrawler

Version History

Version Downloads Last updated
2014.2.19 22,478 2/19/2014