DataFilters 0.9.0-beta0001

This is a prerelease version of DataFilters.
There is a newer version of this package available.
See the version list below for details.
dotnet add package DataFilters --version 0.9.0-beta0001                
NuGet\Install-Package DataFilters -Version 0.9.0-beta0001                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="DataFilters" Version="0.9.0-beta0001" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add DataFilters --version 0.9.0-beta0001                
#r "nuget: DataFilters, 0.9.0-beta0001"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install DataFilters as a Cake Addin
#addin nuget:?package=DataFilters&version=0.9.0-beta0001&prerelease

// Install DataFilters as a Cake Tool
#tool nuget:?package=DataFilters&version=0.9.0-beta0001&prerelease                

Datafilters

GitHub Workflow Status (branch) GitHub Workflow Status (branch) codecov GitHub raw issues Nuget

A small library that allow to convert a string to a generic IFilter object. Highly inspired by the elastic query syntax, it offers a powerful way to build and query data with a syntax that's not bound to a peculiar datasource.

Table of contents

The idea came to me when working on a set of REST APIs and trying to build /search endpoints. I wanted to have a uniform way to query a collection of resources whilst abstracting away underlying datasources.

Let's say your API handles vigilante resources :

public class Vigilante
{
    public string Firstname { get; set; }
    public string Lastname { get; set; }
    public string Nickname {get; set; }
    public int Age { get; set; }
    public string Description {get; set;}
    public IEnumerable<string> Powers {get; set;}
    public IEnumerable<Vigilante> Acolytes {get; set;} 
}

JSON Schema

{
  "id": "vigilante_root",
  "title": "Vigilante",
  "type": "object",
  "properties": {
    "firstname": {
      "required": true,
      "type": "string"
    },
    "lastname": {
      "required": true,
      "type": "string"
    },
    "nickname": {
      "required": true,
      "type": "string"
    },
    "age": {
      "required": true,
      "type": "integer"
    },
    "description": {
      "required": true,
      "type": "string"
    },
    "powers": {
      "required": true,
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "acolytes": {
      "required": true,
      "type": "array",
      "items": {
        "$ref": "vigilante_root"
      }
    }
  }
}

and the base URL of your API is https://my-beautiful/api.

vigilante resources could then be located at https://my-beautiful/api/vigilantes/

Wouldn't it be nice to be able to search any resource like so https://my-beautiful/api/vigilantes/search?nickname=Bat*|Super* ?

This is exactly what this project is about : giving you an uniform syntax to query resources without having to thing about the underlying datasource.

<a href='#' id='parsing'>Parsing</a>

This is the first step on filtering data. Thanks to SuperPower, the library supports a custom syntax that can be used to specified one or more criteria resources must fullfill. The currently supported syntax mimic the query string syntax : a key-value pair separated by ampersand (& character) where :

  • field is the name of a property of the resource to filter
  • value is an expression which syntax is highly inspired by the Lucene syntax

To parse an expression, simply call ToFilter<T> extension method (see unit tests for more details on the syntax)

<a href='#' id='filtering'>Filters syntax</a>

Several expressions are supported and here's how you can start using them in your search queries.

<a href='#' id='equals-expression'>Equals</a>

Search for any vigilante resource where nickname value is manbat

Query string JSON
nickname=manbat { "field":"nickname", "op":"eq", "value":"manbat" }

will result in a IFilter instance equivalent to

IFilter filter = new Filter("nickname", EqualsTo, "bat");

<a href='#' id='starts-with-expression'>Starts with</a>

Search for any vigilante resource that starts with "bat" in the nickname property

Query string JSON
nickname=bat* { "field":"nickname", "op":"startswith", "value":"bat" }

will result in a IFilter instance equivalent to

IFilter filter = new Filter("nickname", StartsWith, "bat");

<a href='#' id='ends-with-expression'>Ends with</a>

Search for vigilante resource that ends with man in the nickname property.

Query string JSON
nickname=*man { "field":"nickname", "op":"endswith", "value":"man" }

will result in a IFilter instance equivalent to

IFilter filter = new Filter("nickname", Contains, "bat");

<a href='#' id='contains-expression'>Contains</a>

Search for vigilante resources that contains bat in the nickname property.

Query string JSON
nickname=*bat* { "field":"nickname", "op":"contains", "value":"bat" }

will result in a IFilter instance equivalent to

IFilter filter = new Filter("nickname", Contains, "bat");

💡 contains also work on arrays. powers=*strength* will search for vigilantes who have strength related powers.

<a href='#' id='isempty-expression'>Is empty</a>

Search for vigilante resources that have no powers.

Query string JSON
powers=!* { "field":"powers", "op":"isempty" }

<a href='#' id='interval-expressions'>Interval expressions</a>

Interval expressions are delimited by upper and a lower bound. The generic syntax is

<field>=<min> TO <max>

where

  • field is the name of the property current interval expression will be apply to
  • min is the lowest bound of the interval
  • max is the highest bound of the interval

<a href='#' id='gte-expression'>Greater than or equal</a>

Search for vigilante resources where the value of age property is greater than or equal to 18

Query string JSON
age=[18 TO *[ {"field":"age", "op":"gte", "value":18}

will result in a IFilter instance equivalent to

IFilter filter = new Filter("age", GreaterThanOrEqualTo, 18);

<a href='#' id='lte-expression'>Less than or equal</a>

Search for vigilante resource where the value of age property is lower than 30

Query string JSON
age=]* TO 30] {"field":"age", "op":"lte", "value":30}

will be parsed into a IFilter equivalent to

IFilter filter = new Filter("age", LessThanOrEqualTo, 30);

<a href='#' id='btw-expression'>Between</a>

Search for vigilante resources where age property is between 20 and 35

Query string JSON
age=[20 TO 35] {"logic": "and", filters[{"field":"age", "op":"gte", "value":20}, {"field":"age", "op":"lte", "value":35}]}

will result in a IFilter instance equivalent to

IFilter filter = new MultiFilter
{
    Logic = And,
    Filters = new IFilter[]
    {
        new Filter("age", GreaterThanOrEqualTo, 20),
        new Filter("age", LessThanOrEqualTo, 35)
    }
}

💡 You can exclude the lower (resp. upper) bound by using ] (resp. [).

  • age=]20 TO 35[ means age strictly greater than 20 and strictly less than35
  • age=[20 TO 35[ means age greater than or equal to 20 and strictly less than35
  • age=]20 TO 35] means age greater than 20 and less than or equal to 35

💡 Dates, times and durations must be specified in ISO 8601 format

Examples :

  • ]1998-10-26 TO 2000-12-10[
  • my/beautiful/api/search?date=]1998-10-26 10:00 TO 1998-10-26 10:00[
  • ]1998-10-12T12:20:00 TO 13:30[ is equivalent to ]1998-10-12T12:20:00 TO 1998-10-12T13:30:00[

💡 You can apply filters to any sub-property of a given collection

Example : acolytes["name"]='robin' will filter any vigilante resource where at least one item in acolytes array with name equals to robin.

The generic syntax for filtering on in a hierarchical tree property["subproperty"]...["subproperty-n"]=<expression>

you can also use the dot character (.). property["subproperty"]["subproperty-n"]=<expression> and property.subproperty["subproperty-n"]=<expression> are equivalent

<a href="regex-expression">Regular expression</a>

The library offers a limited support of regular expressions. To be more specific, only bracket expressions are currently supported. A bracket expression. Matches a single character that is contained within the brackets. For example, [abc] matches a, b, or c. [a-z] specifies a range which matches any lowercase letter from a to z.

BracketExpressions can be, as any other expressions combined with any other expressions to build more complex expressions.

<a href="logic-operators">Logical operators</a>

Logicial operators can be used combine several instances of IFilter together.

<a href='#' id='and-expression'>And</a>

Use the coma character , to combine multiple expressions using logical AND operator

Query string JSON
nickname=Bat*,*man {"logic": "and", filters[{"field":"nickname", "op":"startswith", "value":"Bat"}, {"field":"nckname", "op":"endswith", "value":"man"}]}

will result in a IFilter instance equivalent to

IFilter filter = new MultiFilter
{
    Logic = And,
    Filters = new IFilter[]
    {
        new Filter("nickname", StartsWith, "Bat"),
        new Filter("nickname", EndsWith, "man")
    }
}

<a href='#' id='or-expression'>Or</a>

Use the pipe character | to combine several expressions using logical OR operator Search for vigilante resources where the value of the nickname property either starts with "Bat" or ends with "man"

Query string JSON
nickname=Bat*|*man {"logic": "or", filters[{"field":"nickname", "op":"startswith", "value":"Bat"}, {"field":"nckname", "op":"endswith", "value":"man"}]}

will result in

IFilter filter = new MultiFilter
{
    Logic = Or,
    Filters = new IFilter[]
    {
        new Filter("nickname", StartsWith, "Bat"),
        new Filter("nickname", EndsWith, "man")
    }
}

<a href='#' id='not-expression'>Not</a>

To negate a filter, simply put a ! before the expression to negate

Search for vigilante resources where the value of nickname property does not starts with "B"

Query string JSON
nickname=!B* {"field":"nickname", "op":"nstartswith", "value":"B"}

will be parsed into a IFilter instance equivalent to

IFilter filter = new Filter("nickname", DoesNotStartWith, "B");

Expressions can be arbitrarily complex.

"nickname=(Bat*|Sup*)|(*man|*er)"

Explanation :

The criteria under construction will be applied to the value of nickname property and can be read as follow :

Searchs for vigilante resources that starts with Bat or Sup and ends with man or er.

will be parsed into a

IFilter filter = new MultiFilter
{
    Logic = Or,
    Filters = new IFilter[]
    {
        new MultiFilter
        {
            Logic = Or,
            Filters = new IFilter[]
            {
                new Filter("Firstname", StartsWith, "Bat"),
                new Filter("Firstname", StartsWith, "Sup"),
            }
        },
        new MultiFilter
        {
            Logic = Or,
            Filters = new IFilter[]
            {
                new Filter("Firstname", EndsWith, "man"),
                new Filter("Firstname", EndsWith, "er"),
            }
        },
    }
}

The ( and ) characters allows to group two expressions together so that this group can be used as a more complex expression unit.

<a href='#' id='special-character-handling'>Special character handling</a>

Sometimes, you'll be looking for a filter that match exactly a text that contains a character which has a special meaning.

The backslash character (\) can be used to escape characters that will be otherwise interpreted as a special character.

Query string JSON
comment=*\! {"field":"comment", "op":"endswith", "value":"!"}

will be parsed into a IFilter instance equivalent to

IFilter filter = new Filter("comment", EndsWith, "!");

💡 For longer texts, just wrap it between quotes and you're good to go

Query string JSON
comment=*"!" {"field":"comment", "op":"endswith", "value":"!"}

<a href='#' id='sorting'>Sorting</a>

This library also supports a custom syntax to sort elements.

sort=nickname or sort=+nickname sort items by their nickname properties in ascending order.

You can sort by several properties at once by separating them with a ,.

For example sort=+nickname,-age allows to sort by nickname ascending, then by age property descending.

<a href='#' id='how-to-install'>How to install</a>

  1. run dotnet install DataFilters : you can already start building IFilter instances 😉 !
  2. install one or more DataFilters.XXXX extension packages to convert IFilter instances to various target.

<a href='#' id='how-to-use'>How to use</a>

So you have your API and want provide a great search experience ?

<a href='#' id='how-to-use-client'>On the client</a>

The client will have the responsability of building search criteria. Go to filtering and sorting sections to see example on how to get started.

<a href='#' id='how-to-use-backend'>On the backend</a>

One way to start could be by having a dedicated resource which properties match the resource's properties search will be performed onto.

Continuing with our vigilante API, we could have

// Wraps the search criteria for Vigilante resources.
public class SearchVigilanteQuery
{
    public string Firstname {get; set;}

    public string Lastname {get; set;}

    public string Nickname {get; set;}

    public int? Age {get; set;}
}

and the following endpoint

using DataFilters;

public class VigilantesController
{
    // code omitted for brievity

    [HttpGet("search")]
    [HttpHead("search")]
    public ActionResult Search([FromQuery] SearchVigilanteQuery query)
    {
        IList<IFilter> filters = new List<IFilter>();

        if(!string.IsNullOrWhitespace(query.Firstname))
        {
            filters.Add($"{nameof(Vigilante.Firstname)}={query.Firstname}".ToFilter<Vigilante>());
        }

        if(!string.IsNullOrWhitespace(query.Lastname))
        {
            filters.Add($"{nameof(Vigilante.Lastname)}={query.Lastname}".ToFilter<Vigilante>());
        }

        if(!string.IsNullOrWhitespace(query.Nickname))
        {
            filters.Add($"{nameof(Vigilante.Nickname)}={query.Nickname}".ToFilter<Vigilante>());
        }

        if(query.Age.HasValue)
        {
            filters.Add($"{nameof(Vigilante.Age)}={query.Age.Value}".ToFilter<Vigilante>());
        }


        IFilter  filter = filters.Count() == 1
            ? filters.Single()
            : new MultiFilter{ Logic = And, Filters = filters };

        // filter now contains our search criteria and is ready to be used 😊

    }
}

Some explanation on the controller's code above :

  1. The endpoint is bound to incoming HTTP GET and HEAD requests on /vigilante/search
  2. The framework will parse incoming querystring and feeds the query parameter accordingly.
  3. From this point we test each criterion to see if it's acceptable to turn it into a IFilter instance. For that purpose, the handy .ToFilter<T>() string extension method is available. It turns a query-string key-value pair into a full IFilter.
  4. we can then either :
    • use the filter directly is there was only one filter
    • or combine them using composite filter if there were more than one criterion.

You may have noticed that SearchVigilanteQuery.Age property is nullable whereas Vigilante.Age property is not. This is to distinguish if the Age criterion was provided or not when calling the vigilantes/search endpoint.

Name Package Description
DataFilters Nuget provides core functionalities of parsing strings and converting to IFilter instances.
DataFilters.Expressions Nuget adds ToExpression<T>() extension method on top of IFilter instance to convert it to an equivalent System.Linq.Expressions.Expression<Func<T, bool>> instance.
DataFilters Queries Nuget adds ToWhere<T>() extension method on top of IFilter instance to convert it to an equivalent IWhereClause instance.
Product Compatible and additional computed target framework versions.
.NET net5.0 is compatible.  net5.0-windows was computed.  net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
.NET Core netcoreapp1.0 was computed.  netcoreapp1.1 was computed.  netcoreapp2.0 was computed.  netcoreapp2.1 was computed.  netcoreapp2.2 was computed.  netcoreapp3.0 was computed.  netcoreapp3.1 was computed. 
.NET Standard netstandard1.3 is compatible.  netstandard1.4 was computed.  netstandard1.5 was computed.  netstandard1.6 was computed.  netstandard2.0 is compatible.  netstandard2.1 is compatible. 
.NET Framework net46 was computed.  net461 was computed.  net462 was computed.  net463 was computed.  net47 was computed.  net471 was computed.  net472 was computed.  net48 was computed.  net481 was computed. 
MonoAndroid monoandroid was computed. 
MonoMac monomac was computed. 
MonoTouch monotouch was computed. 
Tizen tizen30 was computed.  tizen40 was computed.  tizen60 was computed. 
Universal Windows Platform uap was computed.  uap10.0 was computed. 
Xamarin.iOS xamarinios was computed. 
Xamarin.Mac xamarinmac was computed. 
Xamarin.TVOS xamarintvos was computed. 
Xamarin.WatchOS xamarinwatchos was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (4)

Showing the top 4 NuGet packages that depend on DataFilters:

Package Downloads
DataFilters.Expressions

Converts IFilter instance to strongly typed expressions.

DataFilters.Queries

Provides extension methods to convert IFilter to IWhereClause and IOrder to ISort.

DataFilters.AspNetCore

Package Description

DataFilters.AspNetCore.Attributes

Package Description

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
0.13.1-fix0001 153 9/7/2024
0.13.1-fix.31 40 9/14/2024
0.13.1-fix.30 39 9/14/2024
0.13.1-fix.29 35 9/14/2024
0.13.1-beta0023 157 9/8/2024
0.13.1-beta0001 138 7/12/2024
0.13.0 288 7/11/2024
0.13.0-beta0001 150 7/7/2024
0.12.0 8,675 10/12/2022
0.12.0-beta0001 238 4/27/2022
0.11.0 9,731 3/13/2022
0.10.2 988 3/9/2022
0.10.0 1,028 1/12/2022
0.9.0 543 12/30/2021
0.9.0-beta0001 254 11/22/2021
0.8.0 961 10/10/2021
0.7.0 644 6/29/2021
0.6.0 843 5/3/2021
0.5.0 625 5/2/2021
0.5.0-alpha0004 335 4/3/2021
0.4.1 652 4/28/2021
0.4.0 719 4/3/2021
0.3.2 791 1/30/2021
0.3.1 4,118 1/3/2021
0.2.2 1,240 12/5/2020
0.2.1 704 12/4/2020

• Made BracketValue implement IParseable, IHaveComplexity
• Bumped Candoumbe.MiscUtilities to 0.6.3
• Bumped Queries.Core to 0.4.0
• Added DateOnly support ([#63](https://github.com/candoumbe/datafilters/issues/63))
• Added TimeOnly support ([#64](https://github.com/candoumbe/datafilters/issues/64))
• Added [NullableValueBehavior](/src/Datafilters.Expressions/NullableValueBehavior.cs) ([#68](https://github.com/candoumbe/datafilters/issues/68))

Full changelog at https://github.com/candoumbe/DataFilters/blob/release/0.9.0/CHANGELOG.md