DataFilters 0.12.0
See the version list below for details.
dotnet add package DataFilters --version 0.12.0
NuGet\Install-Package DataFilters -Version 0.12.0
<PackageReference Include="DataFilters" Version="0.12.0" />
paket add DataFilters --version 0.12.0
#r "nuget: DataFilters, 0.12.0"
// Install DataFilters as a Cake Addin #addin nuget:?package=DataFilters&version=0.12.0 // Install DataFilters as a Cake Tool #tool nuget:?package=DataFilters&version=0.12.0
Datafilters
A small library that allow to convert a string to a generic IFilter
object.
Highly inspired by the elastic query syntax, it offers a powerful way to build and query data with a syntax that's not bound to a peculiar datasource.
Disclaimer
This project adheres to Semantic Versioning.
Major version zero (0.y.z) is for initial development. Anything MAY change at any time.
The public API SHOULD NOT be considered stable.
Table of contents
- <a href='#parsing'>Parsing</a>
- <a href='#filtering'>Filters syntax</a>
- <a href='#equals-expression'>Equals</a>
- <a href='#starts-with-expression'>Starts with</a>
- <a href='#ends-with-expression'>Ends with</a>
- <a href='#contains-expression'>Contains</a>
- <a href='#isempty-expression'>Is empty</a>
- <a href='#any-of-expression'>Any of</a>
- <a href='#isnull-expression'>Is null</a>
- <a href='#interval-expressions'>Interval expressions</a>
- <a href='#gte-expression'>Greater than or equal</a>
- <a href='#lte-expression'>Less than or equal</a>
- <a href='#btw-expression'>Between</a>
- <a href='#regular-expression'>Regular expression</a>
- <a href="logic-operators">Logical operators</a>
- <a href='#and-expression'>And</a>
- <a href='#or-expression'>Or</a>
- <a href='#not-expression'>Not</a>
- <a href='#special-character-handling'>Special character handling</a>
- <a href='#sorting'>Sorting</a>
- <a href='#how-to-install'>How to install</a>
- <a href='#how-to-use'>How to use</a>
- <a href='#how-to-use-client'>On the client</a>
- <a href='#how-to-use-backend'>On the backend</a>
The idea came to me when working on a set of REST APIs and trying to build /search
endpoints.
I wanted to have a uniform way to query a collection of resources whilst abstracting away underlying datasources.
Let's say your API handles vigilante
resources :
public class Vigilante
{
public string Firstname { get; set; }
public string Lastname { get; set; }
public string Nickname {get; set; }
public int Age { get; set; }
public string Description {get; set;}
public IEnumerable<string> Powers {get; set;}
public IEnumerable<Vigilante> Acolytes {get; set;}
}
JSON Schema
{
"id": "vigilante_root",
"title": "Vigilante",
"type": "object",
"properties": {
"firstname": {
"required": true,
"type": "string"
},
"lastname": {
"required": true,
"type": "string"
},
"nickname": {
"required": true,
"type": "string"
},
"age": {
"required": true,
"type": "integer"
},
"description": {
"required": true,
"type": "string"
},
"powers": {
"required": true,
"type": "array",
"items": {
"type": "string"
}
},
"acolytes": {
"required": true,
"type": "array",
"items": {
"$ref": "vigilante_root"
}
}
}
}
and the base URL of your API is https://my-beautiful/api
.
vigilante
resources could then be located at https://my-beautiful/api/vigilantes/
Wouldn't it be nice to be able to search any resource like so
https://my-beautiful/api/vigilantes/search?nickname=Bat*|Super*
?
This is exactly what this project is about : giving you an uniform syntax to query resources without having to think about the underlying datasource.
<a href='#' id='parsing'>Parsing</a>
This is the first step on filtering data. Thanks to SuperPower,
the library supports a custom syntax that can be used to specified one or more criteria resources must fullfill.
The currently supported syntax mimic the query string syntax : a key-value pair separated by ampersand (&
character) where :
field
is the name of a property of the resource to filtervalue
is an expression which syntax is highly inspired by the Lucene syntax
To parse an expression, simply call ToFilter<T>
extension method
(see unit tests for more details on the syntax)
<a href='#' id='filtering'>Filters syntax</a>
Several expressions are supported and here's how you can start using them in your search queries.
<a href='#' id='equals-expression'>Equals</a>
Search for any vigilante resource where nickname
value is manbat
Query string | JSON | C# |
---|---|---|
nickname=manbat |
{ "field":"nickname", "op":"eq", "value":"manbat" } |
new Filter(field: "nickname", @operator : FilterOperator.EqualsTo, value : "manbat") |
<a href='#' id='starts-with-expression'>Starts with</a>
Search for any vigilante resource that starts with "bat"
in the nickname
property
Query string | JSON | C# |
---|---|---|
nickname=bat* |
{ "field":"nickname", "op":"startswith", "value":"bat" } |
new Filter(field: "nickname", @operator : FilterOperator.StartsWith, value : "bat") |
<a href='#' id='ends-with-expression'>Ends with</a>
Search for vigilante
resource that ends with man
in the nickname
property.
Query string | JSON | C# |
---|---|---|
nickname=*man |
{ "field":"nickname", "op":"endswith", "value":"man" } |
new Filter(field: "nickname", @operator : FilterOperator.EndsWith, value : "man") |
<a href='#' id='contains-expression'>Contains</a>
Search for vigilante
resources that contains bat
in the nickname
property.
Query string | JSON | C# |
---|---|---|
nickname=*bat* |
{ "field":"nickname", "op":"contains", "value":"bat" } |
new Filter(field: "nickname", @operator : FilterOperator.Contains, value : "bat") |
💡 contains
also work on arrays. powers=*strength*
will search for vigilante
s who have strength
related powers.
<a href='#' id='isempty-expression'>Is empty</a>
Search for vigilante
resources that have no powers.
Query string | JSON | C# |
---|---|---|
powers=!* |
{ "field":"powers", "op":"isempty" } |
new Filter(field: "powers", @operator : FilterOperator.IsEmpty) |
<a href='#' id='isnull-expression'>Is null</a>
Search for vigilante
resources that have no powers.
Query string | JSON | C# |
---|---|---|
N/A |
{ "field":"powers", "op":"isnull" } |
new Filter(field: "powers", @operator : FilterOperator.IsNull) or new Filter(field: "powers", @operator : FilterOperator.EqualsTo, value: null) |
<a href='#' id='any-of-expression'>Any of</a>
Search for vigilante
resources that have at least one of the specified powers.
Query string | JSON |
---|---|
powers={strength\|speed\|size} |
N/A |
will result in a IFilter instance equivalent to
IFilter filter = new MultiFilter
{
Logic = Or,
Filters = new IFilter[]
{
new Filter("powers", EqualTo, "strength"),
new Filter("powers", EqualTo, "speed"),
new Filter("powers", EqualTo, "size")
}
};
<a href='#' id='isnull-expression'>Is null</a>
Search for vigilante
resources that have no powers.
Query string | JSON | C# |
---|---|---|
N/A |
{ "field":"powers", "op":"isnull" } |
new Filter(field: "powers", @operator : FilterOperator.IsNull) or new Filter(field: "powers", @operator : FilterOperator.EqualsTo, value: null) |
<a href='#' id='interval-expressions'>Interval expressions</a>
Interval expressions are delimited by upper and a lower bound. The generic syntax is
<field>=<min> TO <max>
where
field
is the name of the property current interval expression will be apply tomin
is the lowest bound of the intervalmax
is the highest bound of the interval
<a href='#' id='gte-expression'>Greater than or equal</a>
Search for vigilante
resources where the value of age
property is greater than or equal to 18
Query string | JSON | C# |
---|---|---|
age=[18 TO *[ |
{"field":"age", "op":"gte", "value":18} |
new Filter(field: "age", @operator : FilterOperator.GreaterThanOrEqualTo, value : 18) |
<a href='#' id='lte-expression'>Less than or equal</a>
Search for vigilante
resource where the value of age
property is lower than 30
Query string | JSON | C# |
---|---|---|
age=]* TO 30] |
{"field":"age", "op":"lte", "value":30} |
new Filter(field: "age", @operator : FilterOperator.LessThanOrEqualTo, value : 30) |
IFilter filter = new Filter("age", LessThanOrEqualTo, 30);
### <a href='#' id='btw-expression'>Between</a>
Search for vigilante resources where `age` property is between `20` and `35`
| Query string | JSON | C# |
| ---------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `age=[20 TO 35]` | `{"logic": "and", filters[{"field":"age", "op":"gte", "value":20}, {"field":"age", "op":"lte", "value":35}]}` | `new MultiFilter { Logic = And, Filters = new IFilter[] { new Filter ("age", GreaterThanOrEqualTo, 20), new Filter("age", LessThanOrEqualTo, 35) } }` |
---
💡 You can exclude the lower (resp. upper) bound by using `]` (resp. `[`).
- `age=]20 TO 35[` means `age` strictly greater than `20` and strictly less than`35`
- `age=[20 TO 35[` means `age` greater than or equal to `20` and strictly less than`35`
- `age=]20 TO 35]` means `age` greater than `20` and less than or equal to `35`
💡 Dates, times and durations must be specified in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601)
Examples :
- `]1998-10-26 TO 2000-12-10[`
- `my/beautiful/api/search?date=]1998-10-26 10:00 TO 1998-10-26 10:00[`
- `]1998-10-12T12:20:00 TO 13:30[` is equivalent to `]1998-10-12T12:20:00 TO 1998-10-12T13:30:00[`
💡 You can apply filters to any sub-property of a given collection
Example :
`acolytes["name"]='robin'` will filter any `vigilante` resource where at least one item in `acolytes` array with `name` equals to `robin`.
The generic syntax for filtering on in a hierarchical tree
`property["subproperty"]...["subproperty-n"]=<expression>`
you can also use the dot character (`.`).
`property["subproperty"]["subproperty-n"]=<expression>` and `property.subproperty["subproperty-n"]=<expression>`
are equivalent
## <a href="regex-expression">Regular expression</a>
The library offers a limited support of regular expressions. To be more specific, only bracket expressions are currently supported.
A bracket expression. Matches a single character that is contained within the brackets.
For example, `[abc]` matches `a`, `b`, or `c`. `[a-z]` specifies a range which matches any lowercase letter from `a` to `z`.
`BracketExpression`s can be, as any other expressions combined with any other expressions to build more complex expressions.
## <a href="logic-operators">Logical operators</a>
Logicial operators can be used combine several instances of [IFilter][class-ifilter] together.
### <a href='#' id='and-expression'>And</a>
Use the coma character `,` to combine multiple expressions using logical AND operator
| Query string | JSON |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `nickname=Bat*,*man` | `{"logic": "and", filters[{"field":"nickname", "op":"startswith", "value":"Bat"}, {"field":"nckname", "op":"endswith", "value":"man"}]}` |
will result in a [IFilter][class-ifilter] instance equivalent to
```csharp
IFilter filter = new MultiFilter
{
Logic = And,
Filters = new IFilter[]
{
new Filter("nickname", StartsWith, "Bat"),
new Filter("nickname", EndsWith, "man")
}
}
<a href='#' id='or-expression'>Or</a>
Use the pipe character |
to combine several expressions using logical OR operator
Search for vigilante
resources where the value of the nickname
property either starts with "Bat"
or
ends with "man"
Query string | JSON |
---|---|
nickname=Bat* \| *man |
{"logic": "or", filters[{"field":"nickname", "op":"startswith", "value":"Bat"}, {"field":"nckname", "op":"endswith", "value":"man"}]} |
will result in
IFilter filter = new MultiFilter
{
Logic = Or,
Filters = new IFilter[]
{
new Filter("nickname", StartsWith, "Bat"),
new Filter("nickname", EndsWith, "man")
}
}
<a href='#' id='not-expression'>Not</a>
To negate a filter, simply put a !
before the expression to negate
Search for vigilante
resources where the value of nickname
property does not starts with "B"
Query string | JSON |
---|---|
nickname=!B* |
{"field":"nickname", "op":"nstartswith", "value":"B"} |
will be parsed into a IFilter instance equivalent to
IFilter filter = new Filter("nickname", DoesNotStartWith, "B");
Expressions can be arbitrarily complex.
"nickname=(Bat*|Sup*)|(*man|*er)"
Explanation :
The criteria under construction will be applied to the value of nickname
property and can be read as follow :
Searchs for vigilante
resources that starts with Bat
or Sup
and ends with man
or
er
.
will be parsed into a
IFilter filter = new MultiFilter
{
Logic = Or,
Filters = new IFilter[]
{
new MultiFilter
{
Logic = Or,
Filters = new IFilter[]
{
new Filter("Firstname", StartsWith, "Bat"),
new Filter("Firstname", StartsWith, "Sup"),
}
},
new MultiFilter
{
Logic = Or,
Filters = new IFilter[]
{
new Filter("Firstname", EndsWith, "man"),
new Filter("Firstname", EndsWith, "er"),
}
},
}
}
The (
and )
characters allows to group two expressions together so that this group can be used as a more complex
expression unit.
<a href='#' id='special-character-handling'>Special character handling</a>
Sometimes, you'll be looking for a filter that match exactly a text that contains a character which has a special meaning.
The backslash character (\
) can be used to escape characters that will be otherwise interpreted as
a special character.
Query string | JSON | C# |
---|---|---|
comment=*\! |
{"field":"comment", "op":"endswith", "value":"!"} |
new Filter(field: "comments", @operator: FilterOperator.EndsWith, value: "!") |
💡 For longer texts, just wrap it between quotes and you're good to go
Query string | JSON | C# |
---|---|---|
comment=*"!" |
{"field":"comment", "op":"endswith", "value":"!"} |
new Filter(field: "comments", @operator: FilterOperator.EndsWith, value: "!") |
<a href='#' id='sorting'>Sorting</a>
This library also supports a custom syntax to sort elements.
sort=nickname
or sort=+nickname
sort items by their nickname
properties in ascending
order.
You can sort by several properties at once by separating them with a ,
.
For example sort=+nickname,-age
allows to sort by nickname
ascending, then by age
property descending.
<a href='#' id='how-to-install'>How to install</a>
- run
dotnet install DataFilters
: you can already start building IFilter instances 😉 ! - install one or more
DataFilters.XXXX
extension packages to convert IFilter instances to various target.
<a href='#' id='how-to-use'>How to use</a>
So you have your API and want provide a great search experience ?
<a href='#' id='how-to-use-client'>On the client</a>
The client will have the responsability of building search criteria. Go to filtering and sorting sections to see example on how to get started.
<a href='#' id='how-to-use-backend'>On the backend</a>
One way to start could be by having a dedicated resource which properties match the resource's properties search will be performed onto.
Continuing with our vigilante
API, we could have
// Wraps the search criteria for Vigilante resources.
public class SearchVigilanteQuery
{
public string Firstname {get; set;}
public string Lastname {get; set;}
public string Nickname {get; set;}
public int? Age {get; set;}
}
and the following endpoint
using DataFilters;
public class VigilantesController
{
// code omitted for brievity
[HttpGet("search")]
[HttpHead("search")]
public ActionResult Search([FromQuery] SearchVigilanteQuery query)
{
IList<IFilter> filters = new List<IFilter>();
if(!string.IsNullOrWhitespace(query.Firstname))
{
filters.Add($"{nameof(Vigilante.Firstname)}={query.Firstname}".ToFilter<Vigilante>());
}
if(!string.IsNullOrWhitespace(query.Lastname))
{
filters.Add($"{nameof(Vigilante.Lastname)}={query.Lastname}".ToFilter<Vigilante>());
}
if(!string.IsNullOrWhitespace(query.Nickname))
{
filters.Add($"{nameof(Vigilante.Nickname)}={query.Nickname}".ToFilter<Vigilante>());
}
if(query.Age.HasValue)
{
filters.Add($"{nameof(Vigilante.Age)}={query.Age.Value}".ToFilter<Vigilante>());
}
IFilter filter = filters.Count() == 1
? filters.Single()
: new MultiFilter{ Logic = And, Filters = filters };
// filter now contains our search criteria and is ready to be used 😊
}
}
Some explanation on the controller's code above :
- The endpoint is bound to incoming HTTP
GET
andHEAD
requests on/vigilante/search
- The framework will parse incoming querystring and feeds the
query
parameter accordingly. - From this point we test each criterion to see if it's acceptable to turn it into a IFilter instance.
For that purpose, the handy
.ToFilter<T>()
string extension method is available. It turns a query-string key-value pair into a full IFilter. - we can then either :
- use the filter directly is there was only one filter
- or combine them using composite filter if there were more than one criterion.
You may have noticed that SearchVigilanteQuery.Age
property is nullable whereas Vigilante.Age
property is not.
This is to distinguish if the Age
criterion was provided or not when calling the vigilantes/search
endpoint.
Name | Package | Description |
---|---|---|
DataFilters |
provides core functionalities of parsing strings and converting to IFilter instances. | |
DataFilters.Expressions |
adds ToExpression<T>() extension method on top of IFilter instance to convert it to an equivalent System.Linq.Expressions.Expression<Func<T, bool>> instance. |
|
DataFilters Queries |
adds ToWhere<T>() extension method on top of IFilter instance to convert it to an equivalent IWhereClause instance. |
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net5.0 is compatible. net5.0-windows was computed. net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
.NET Core | netcoreapp1.0 was computed. netcoreapp1.1 was computed. netcoreapp2.0 was computed. netcoreapp2.1 was computed. netcoreapp2.2 was computed. netcoreapp3.0 was computed. netcoreapp3.1 was computed. |
.NET Standard | netstandard1.3 is compatible. netstandard1.4 was computed. netstandard1.5 was computed. netstandard1.6 was computed. netstandard2.0 is compatible. netstandard2.1 is compatible. |
.NET Framework | net46 was computed. net461 was computed. net462 was computed. net463 was computed. net47 was computed. net471 was computed. net472 was computed. net48 was computed. net481 was computed. |
MonoAndroid | monoandroid was computed. |
MonoMac | monomac was computed. |
MonoTouch | monotouch was computed. |
Tizen | tizen30 was computed. tizen40 was computed. tizen60 was computed. |
Universal Windows Platform | uap was computed. uap10.0 was computed. |
Xamarin.iOS | xamarinios was computed. |
Xamarin.Mac | xamarinmac was computed. |
Xamarin.TVOS | xamarintvos was computed. |
Xamarin.WatchOS | xamarinwatchos was computed. |
-
.NETStandard 1.3
- Candoumbe.MiscUtilities (>= 0.8.2)
- FluentValidation (>= 8.6.3)
- Microsoft.CSharp (>= 4.7.0)
- NETStandard.Library (>= 1.6.1)
- Newtonsoft.Json.Schema (>= 3.0.14)
- OneOf.Extended (>= 3.0.223)
- Optional (>= 4.0.0)
- Superpower (>= 2.3.0)
- System.Collections.Immutable (>= 5.0.0)
-
.NETStandard 2.0
- Ardalis.GuardClauses (>= 4.0.1)
- Candoumbe.MiscUtilities (>= 0.8.2)
- FluentValidation (>= 8.6.3)
- Newtonsoft.Json.Schema (>= 3.0.14)
- OneOf.Extended (>= 3.0.223)
- Optional (>= 4.0.0)
- Superpower (>= 3.0.0)
- System.Collections.Immutable (>= 5.0.0)
-
.NETStandard 2.1
- Ardalis.GuardClauses (>= 4.0.1)
- Candoumbe.MiscUtilities (>= 0.8.2)
- FluentValidation (>= 8.6.3)
- Newtonsoft.Json.Schema (>= 3.0.14)
- OneOf.Extended (>= 3.0.223)
- Optional (>= 4.0.0)
- Superpower (>= 3.0.0)
- System.Collections.Immutable (>= 5.0.0)
-
net5.0
- Ardalis.GuardClauses (>= 4.0.1)
- Candoumbe.MiscUtilities (>= 0.8.2)
- FluentValidation (>= 10.4.0)
- Newtonsoft.Json.Schema (>= 3.0.14)
- OneOf.Extended (>= 3.0.223)
- Superpower (>= 3.0.0)
- System.Collections.Immutable (>= 5.0.0)
- Ultimately (>= 2.1.1)
-
net6.0
- Ardalis.GuardClauses (>= 4.0.1)
- Candoumbe.MiscUtilities (>= 0.8.2)
- DateOnlyTimeOnly.AspNet (>= 1.0.3)
- FluentValidation (>= 10.4.0)
- Newtonsoft.Json.Schema (>= 3.0.14)
- OneOf.Extended (>= 3.0.223)
- Superpower (>= 3.0.0)
- System.Collections.Immutable (>= 6.0.0)
- Ultimately (>= 2.1.1)
NuGet packages (4)
Showing the top 4 NuGet packages that depend on DataFilters:
Package | Downloads |
---|---|
DataFilters.Expressions
Converts IFilter instance to strongly typed expressions. |
|
DataFilters.Queries
Provides extension methods to convert IFilter to IWhereClause and IOrder to ISort. |
|
DataFilters.AspNetCore
Package Description |
|
DataFilters.AspNetCore.Attributes
Package Description |
GitHub repositories
This package is not used by any popular GitHub repositories.
Version | Downloads | Last updated |
---|---|---|
0.13.1-fix0001 | 154 | 9/7/2024 |
0.13.1-fix.31 | 45 | 9/14/2024 |
0.13.1-fix.30 | 46 | 9/14/2024 |
0.13.1-fix.29 | 42 | 9/14/2024 |
0.13.1-beta0023 | 160 | 9/8/2024 |
0.13.1-beta0001 | 138 | 7/12/2024 |
0.13.0 | 309 | 7/11/2024 |
0.13.0-beta0001 | 150 | 7/7/2024 |
0.12.0 | 8,810 | 10/12/2022 |
0.12.0-beta0001 | 238 | 4/27/2022 |
0.11.0 | 9,836 | 3/13/2022 |
0.10.2 | 988 | 3/9/2022 |
0.10.0 | 1,028 | 1/12/2022 |
0.9.0 | 543 | 12/30/2021 |
0.9.0-beta0001 | 254 | 11/22/2021 |
0.8.0 | 961 | 10/10/2021 |
0.7.0 | 644 | 6/29/2021 |
0.6.0 | 843 | 5/3/2021 |
0.5.0 | 625 | 5/2/2021 |
0.5.0-alpha0004 | 335 | 4/3/2021 |
0.4.1 | 654 | 4/28/2021 |
0.4.0 | 719 | 4/3/2021 |
0.3.2 | 791 | 1/30/2021 |
0.3.1 | 4,118 | 1/3/2021 |
0.2.2 | 1,240 | 12/5/2020 |
0.2.1 | 704 | 12/4/2020 |
### New features
• Added syntax for OneOfExpression using curly braces ([#123](https://github.com/candoumbe/datafilters/issues/123))
• Added + operator to combine ConstantValueExpression with AsteriskExpression
• Added + operator to combine StartsWithExpression with EndsWithExpression
• Added + operator to combine StartsWithExpression with ContainsExpression
• Added + operator to combine StartsWithExpression with StartsWithExpression
• Added + operator to combine StartsWithExpression with StringValueExpression
• Added | operator to combine two FilterExpressions into a OrExpression
• Added & operator to combine two FilterExpressions into a AndExpression
### Breaking changes
• AsteriskExpression default constructor is now private
• Dropped filter service
• Renamed ISort<T> to IOrder<T>
• Renamed Sort<T> to Order<T>
• Renamed MultiSort<T> to MultiOrder<T>
• Renamed SortValidator<T> to OrderValidator<T>
• Renamed SortToQueries class to OrderExtensions in DataFilters.Queries
• Renamed SortExtensions class to OrderExtensions in DataFilters.Expressions
Full changelog at https://github.com/candoumbe/DataFilters/blob/main/CHANGELOG.md