NuGet Gallery | nicholasb.za

RobotsSharpParser

.NET 6.0
18,905 total downloads
last updated 12/22/2021
Latest version: 2.0.0
robots parse robots web crawling robots.txt web scraping spider sitemap More tags

This client library enables working with Robots.txt. Key Features: - Parse robots.txt into Typed object. - Lookup Allowed/Disallowed/Crawldelay based on User-Agent. - Traverse... More information

Nick.RobotsParser

by: nicholasb.za

.NET 6.0
3,741 total downloads
last updated 1/1/2023
Latest version: 2.0.6
robots parse robots web crawling robots.txt web scraping spider sitemap More tags

https://github.com/nicholasbergesen/RobotsParser/blob/master/README.md

Nick.HtmlParser

by: nicholasb.za

.NET 8.0
1,258 total downloads
last updated 1/16/2024
Latest version: 1.0.8
html parser parse html flat structure flatten html

Parses Html into a typed IEnumerable flat structure. v1.0.8 changes: - Bugfix: Handle attributes that contain tags, including malformed attributes - Upgrade to .net8.0 v1.0.7 changes: -... More information

RobotsCoreParser

by: nicholasb.za

Deprecated
.NET Core 3.0
582 total downloads
last updated 10/23/2019
Latest version: 1.0.0

Used for parsing robots files and sitemaps