SmartReader 0.3.1

There is a newer version of this package available.
See the version list below for details.
dotnet add package SmartReader --version 0.3.1
NuGet\Install-Package SmartReader -Version 0.3.1
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="SmartReader" Version="0.3.1" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add SmartReader --version 0.3.1
#r "nuget: SmartReader, 0.3.1"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install SmartReader as a Cake Addin
#addin nuget:?package=SmartReader&version=0.3.1

// Install SmartReader as a Cake Tool
#tool nuget:?package=SmartReader&version=0.3.1

<img src="https://raw.github.com/strumenta/SmartReader/master/logo.png" width="64">

SmartReader is a .NET Standard 1.3 library to extract the main content of a web page, based on a port of the Readability library by Mozilla, which in turn is based on the famous original Readability library.

Installation

You can do it the standard way, by using the NuGet package.

PM> Install-Package SmartReader

Why You May Want To Use It

There are already other similar good projects, but they don't support .NET Core and they are based on old version of Readability. The original library is already quite stable, but there are always improvement to be made. So by relying on a original library maintained by such a competent organization we can piggyback on their hard work and user base.

There are also some improvements: it returns an author and publication date, the language of the article, the featured image, a list of images and an indication of the time needed to read it.

Feel free to suggest new features.

Usage

There are mainly two ways to use the library. The first is by creating a new Reader object, with the URI as the argument, and then calling the GetArticle method to obtain the extracted Article. The second one is by using one of the static methods ParseArticle of Reader directly, to return an Article. Both ways are available also through an async method, called respectively GetArticleAsync and ParseArticleAsync. The advantage of using an object, instead of the static method, is that it gives you the chance to set some options.

There is also the option to parse directly a String or Stream that you have obtained by some other way. This is available either with ParseArticle methods or by using the proper Reader constructor. In either case, you also need to give the original URI. It will not re-download the text, but it need the URI to make some checks and modifications on the links present on the page. If you cannot provide the original uri, you can use a fake one, like https:\\localhost.

If the extraction fails, the returned Article object will have the field IsReadable set to false.

The content of the article is unstyled, but it is wrapped in a div with the id readability-content that you can style yourself.

The library tries to detect the correct encoding of the text, if the correct tags are present in the text.

On the Article object you can call GetImagesAsync to obtain a Task for a list of Image objects, representing the images found in the extracted article. The method is async because it makes HEAD Requests, to obtain the size of the images and only returns the ones that are bigger then the specified size. The size by default is 75KB. This is to exclude things such as images used in the UI.

Examples

Using the GetArticle method.

SmartReader.Reader sr = new SmartReader.Reader("https://arstechnica.co.uk/information-technology/2017/02/humans-must-become-cyborgs-to-survive-says-elon-musk/");

sr.Debug = true;
sr.Logger = new StringWriter();

SmartReader.Article article = sr.GetArticle();
var images = article.GetImagesAsync();

if(article.IsReadable)
{
	// do something with it	
}

Using the ParseArticle static method.


SmartReader.Article article = SmartReader.Reader.ParseArticle("https://arstechnica.co.uk/information-technology/2017/02/humans-must-become-cyborgs-to-survive-says-elon-musk/");

if(article.IsReadable)
{
	// do something with it
}

Options

  • int MaxElemsToParse<br>Max number of nodes supported by this parser. <br> Default: 0 (no limit)
  • int NTopCandidates <br>The number of top candidates to consider when analyzing how tight the competition is among candidates. <br>Default: 5
  • bool Debug <br>Set the Debug option. If set to true the library writes the data on Logger.<br>Default: false
  • TextWriter Logger <br> Where the debug data is going to be written. <br> Default: null
  • bool ContinueIfNotReadable <br> The library tries to determine if it will find an article before actually trying to do it. This option decides whether to continue if the library heuristics fails. This value is ignored if Debug is set to true <br> Default: true
  • int WordThreshold <br>The minimum number of words an article must have in order to return a result. <br>Default: 500

Article Model

  • Uri Uri<br>Original Uri
  • String Title<br>Title
  • String Byline<br>Byline of the article, usually containing author and publication date
  • String Dir<br>Direction of the text
  • String FeaturedImage<br>The main image of the article
  • String Content<br>Html content of the article
  • String TextContent<br>The pure text of the article
  • String Excerpt<br>A summary of the article, based on metadata or first paragraph
  • String Language<br>Language string (es. 'en-US')
  • int Length<br>Length of the text of the article
  • TimeSpan TimeToRead<br>Average time needed to read the article
  • DateTime? PublicationDate<br>Date of publication of the article
  • bool IsReadable<br>Indicate whether we successfully find an article

It's important to be aware that the fields Byline, Author and PublicationDate are found independently of each other. So there might be some inconsistencies and unexpected data. For instance, Byline may be a string in the form "@Date by @Author" or "@Author, @Date" or any other combination used by the publication.

The TimeToRead calculation is based on the research found in Standardized Assessment of Reading Performance: The New International Reading Speed Texts IReST. It should be accurate if the article is written in one of the languages in the research, but it is just an educated guess for the others languages.

The FeaturedImage property holds the image indicated by the Open Graph or Twitter meta tags. If neither of these is present, and you called the GetImagesAsync method, it will be set with the first image found.

Demo & Console Projects

The demo project is a simple ASP.NET Core webpage that allows you to input an address and see the results of the library.

The console project is a Console program that allows you to see the results of the library on a random test page.

Creating The Nuget Package

In case you want to build the Nuget package yourself you cannot use the standard nuget pack because of a bug related to .NET Core. Instaed use the dotnet pack command.

dotnet pack --configuration Release --output "..\nupkgs"

The command must be issued inside the src/SmartReader folder, otherwise it will generate nuget packages for all projects.

License

The project uses the Apache License.

Contributors

Thanks to all the people involved.

Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 was computed.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Core netcoreapp1.0 was computed.  netcoreapp1.1 was computed.  netcoreapp2.0 was computed.  netcoreapp2.1 was computed.  netcoreapp2.2 was computed.  netcoreapp3.0 was computed.  netcoreapp3.1 was computed. 
.NET Standard netstandard1.3 is compatible.  netstandard1.4 was computed.  netstandard1.5 was computed.  netstandard1.6 was computed.  netstandard2.0 was computed.  netstandard2.1 was computed. 
.NET Framework net46 was computed.  net461 was computed.  net462 was computed.  net463 was computed.  net47 was computed.  net471 was computed.  net472 was computed.  net48 was computed.  net481 was computed. 
MonoAndroid monoandroid was computed. 
MonoMac monomac was computed. 
MonoTouch monotouch was computed. 
Tizen tizen30 was computed.  tizen40 was computed.  tizen60 was computed. 
Universal Windows Platform uap was computed.  uap10.0 was computed. 
Xamarin.iOS xamarinios was computed. 
Xamarin.Mac xamarinmac was computed. 
Xamarin.TVOS xamarintvos was computed. 
Xamarin.WatchOS xamarinwatchos was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (2)

Showing the top 2 NuGet packages that depend on SmartReader:

Package Downloads
SuperMemoAssistant.Plugins.Import

Package Description

Drastic.Feed.Parser.SmartReader

Drastic.Feed.Parser.SmartReader is an implementation of IArticleParserService for Drastic.Feed, using SmartReader.

GitHub repositories (2)

Showing the top 2 popular GitHub repositories that depend on SmartReader:

Repository Stars
Richasy/Bili.Copilot
哔哩哔哩用户的个人助理
Richasy/FantasyCopilot
A new-age AI desktop tool
Version Downloads Last updated
0.9.4 10,872 8/27/2023
0.9.3 9,339 4/15/2023
0.9.2 9,912 2/7/2023
0.9.1 8,055 10/23/2022
0.9.0 18,673 8/28/2022
0.8.1 2,889 6/29/2022
0.8.0 11,733 10/19/2021
0.7.5 12,096 10/31/2020
0.7.4 14,242 9/7/2020
0.7.3 489 9/5/2020
0.7.2 1,339 5/10/2020
0.7.1 1,653 3/8/2020
0.7.0 5,546 10/29/2019
0.6.3 2,348 8/18/2019
0.6.2 1,136 5/25/2019
0.6.1 1,011 4/20/2019
0.6.0 632 4/20/2019
0.5.2 925 1/12/2019
0.5.1 858 8/27/2018
0.5.0 918 8/13/2018
0.3.1 1,845 3/3/2018
0.3.0 970 2/17/2018
0.2.0 1,084 1/15/2018
0.1.3 1,066 11/27/2017
0.1.2 1,003 10/17/2017
0.1.1 1,080 9/26/2017