NRobotsCore 1.1.3

.NET Core port of sjdirect/nrobots. A robots.txt parser written in c#. This is an unofficial fork of NRobots.txt on codeplex with extended functionality. For full documentation see https://nrobots.codeplex.com/.

Install-Package NRobotsCore -Version 1.1.3
dotnet add package NRobotsCore --version 1.1.3
<PackageReference Include="NRobotsCore" Version="1.1.3" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add NRobotsCore --version 1.1.3
The NuGet Team does not provide support for this client. Please contact its maintainers for support.

NuGet packages (2)

Showing the top 2 NuGet packages that depend on NRobotsCore:

Package Downloads
Abot
Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to process the page data. You can also plugin your own implementations of core interfaces to take complete control over the crawl process.
AbotCore
.NET Core port of sjdirect/abot. Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to process the page data. You can also plugin your own implementations of core interfaces to take complete control over the crawl process.

GitHub repositories (1)

Showing the top 1 popular GitHub repositories that depend on NRobotsCore:

Repository Stars
sjdirect/abot
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

Version History

Version Downloads Last updated
1.1.3 2,427 11/2/2020
1.1.2 36,802 10/26/2016
1.1.1-alpha1 1,021 9/26/2016
1.1.0 804 9/19/2016