RobotsTxt 2014.2.19

RobotsTxt : A robots.txt parser for .NET

A robots.txt parser for .NET

Supports ;
- Allow directives.
- Crawl-delay directives.
- Sitemap declarations.
- * and $ wildcards.

See https://bitbucket.org/cagdas/robotstxt for usage examples.

Install-Package RobotsTxt -Version 2014.2.19
dotnet add package RobotsTxt --version 2014.2.19
<PackageReference Include="RobotsTxt" Version="2014.2.19" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add RobotsTxt --version 2014.2.19
The NuGet Team does not provide support for this client. Please contact its maintainers for support.

Dependencies

This package has no dependencies.

Showing the top 1 GitHub repositories that depend on RobotsTxt:

Repository Stars
esbencarlsen/NCrawler
.NET based webcrawler

Version History

Version Downloads Last updated
2014.2.19 20,306 2/19/2014