Jsonata.Net.Native 2.7.0

About

.Net native implementation of JSONata query and transformation language.

• Jsonata.Net.Native
• Jsonata.Net.Native.JsonNet
• Jsonata.Net.Native.SystemTextJson

This implementation is based on original jsonata-js source and also borrows some ideas from go port.

Performance

This implementation is about 100 times faster than straightforward wrapping of original jsonata.js with Jint JS Engine for C# (the wrapping is published as jsonata.net.js package).

For measurements code see src/BenchmarkApp in this repo.

Usage

Basic

• simple case

using Jsonata.Net.Native;
...
JsonataQuery query = new JsonataQuery("$.a");
...
string result = query.Eval("{\"a\": \"b\"}");
Debug.Assert(result == "\"b\"");

Since version 2.0.0 this package does not depend on JSON.Net, instead it uses a custom implementation of JSON DOM and parser (see Jsonata.Net.Native.Json namespace). This change gave us the following benefits:

• Things got faster (see here).
• More Jsonata features wee implemented and are possible to implement in future.
• No external dependencies for the core Jsonata.Net.Native package (for those who don't use Json.Net in their projects).

Still this custom implementation is modelled on Json.Net, so the following code should look familliar

using Jsonata.Net.Native;
using Jsonata.Net.Native.Json;
...
JToken data = JToken.Parse("{\"a\": \"b\"}");
...
JToken result = query.Eval(data);
Debug.Assert(result.ToFlatString() == "\"b\"");

JSON.Net

In case you work with JSON.Net you may use a separate binding package Jsonata.Net.Native.JsonNet and its single class JsonataExtensions to:

• convert token hierarchy to and from Json.Net (ToNewtonsoft() and FromNewtonsoft(), note the overload version with formatting overrides)
• evaluate Jsonata queries via various EvalNewtonsoft() overloads
• bind values to EvaluationEnvironment (BindValue())

System.Text.Json

Same goes for when you use System.Text.Json. Separate binding package Jsonata.Net.Native.SystemTextJson provides similar JsonataExtensions class with similar wrappers for both JsonDocument/JsonElement (static DOM) and JsonNode (dynamic DOM).

Querying C# objects

It is also possible to create a JToken tree representation of existing C# object via Jsonata.Net.Native.Json.JToken.FromObject() and then query it with JSONata as a regular JSON. This may come in handy when you want to give your user some way to get and combine data from your program object model.

In case you want to go deeper and get more control over JSON representation of your objects, you may want to use Jsonata.Net.Native.JsonNet.JsonataExtensions.FromObjectViaNewtonsoft() and get all fancy stuff that Json.Net has to offer for this.

Since v2.2.0 there's also a (limited) support for converting JTokens back to C# objects via JToken.ToObject().

C# Features

• JsonataQuery objects are immutable and therefore reusable and thread-safe.
• It is possible to provide additional variable bindings via bindings arg of Eval() call.
• It is possible to provide additional functional bindings via Eval(JToken data, EvaluationEnvironment environment) call. See example
  • Functionality is same as for built-in function implementations
  • You may use a number of argument attributes to get fancy behavior if needed. Also refer to built-in function implementations
• Error codes are mostly in sync with the JS implementation, but some checkup is to be done later (TODO).

We also provide an Exerciser app with same functionality as in original JSONata Exerciser:

Parsing JSON with Jsonata.Net.Native.Json

As mentioned above, modern versions of Jsonata.Net.Native use custom implementation of JSON DOM and parsing. While re-implementation of JSON object model (JToken hierarchy) has been justified by performance and functionality reasons, writing just another JSON parser from scratch in 2022 looked a bit like re-inventing the wheel. On the other hand, forcing some specific external dependency just for the sake of parsing JSON looked even worse. So here you get just another JSON parser available via JToken.Parse() method.

This parser is being checked over the following test sets:

• JSONTestSuite — most prominent collection of corner case checks for JSON Parsers. Out implementation results are:
  • From "accepted" (y_) section: 95 out of 95 tests are passing (100%).
  • From "rejected" (n_) section: 178 out of 188 tests are passing. 2 tests are causing stackoverflow (those are ones contating 10 000 open square braces). And remaining 14 tests are considered "okay" to fail — which is to parse things that are not being expected to be parsed by strict JSON parsers (eg. numbers like -.123).
  • From "ambigous" (i_) section: all 35 tests are not causing the parser to crush miserably (and expected parsing results are not specified for those tests).
• JSON_checker — an official but small json.org's parser tests:
  • From "pass" section: 3 out of 3 tests are passing.
  • From "fail" section: 28 out of 33 tests are passing, and remaining 5 are consiered "okay" for same reasons as above.

We have implemented a number of relaxations to "strict" parser spec used in test, like allowing trailing commas, or single-quoted strings. These options are configurable wia ParseSettings class. All relaxations are enabled by default.

When facing an invalid JSON, the parser would throw a JsonParseException

We have put some effort to this parser, but still the main purpose of the package is not parsing JSON by itself, so in case you need more sophisticated parsing features, like comments (or parsing 10 000 open braces) please use some mature parser package like Json.Net or System.Text.Json and convert results to Jsonata.Net.Native.Json.JToken via routines in a binding package.

Query DOM introspection and query creation via DOM

Since v2.7.0 we provide access to query DOM via Node classes in Jsonata.Net.Native.Dom namespace.

Currently it is possible to:

• acquire (readonly) DOM representation of an existing query (via JsonataQuery.GetDom() call), and inspect it
• construct new DOM hierarchy and then create a query from it (via JsonataQuery(Node) ctor)
• check two DOM (sub)trees for equality (via Node.Equals(Node?))

Please note that right now DOM API is experimental and subject to change in backwards-incompatible way without changing major release version of a library.

Some examples may be found here.

Rationale behind this API is here and here.

JSONata language features support

The goal of the project is to implement 100% of latest JSONata version (1.8.5 at the moment of writing these words), but it's still work in progress. Here's is a list of features in accordance to manual:

• ✔️ Simple Queries with support to arrays and sequence flattening.
• ✔️ Predicate Queries, singleton arrays and wildcards.
• ✔️ Functions and Expressions.
• ✔️ Result Structures.
• ✔️ Query Composition.
• ✔️ Sorting, Grouping and Aggregation.
• ✅ Processing Model - Index (seq#$var) and Join (seq@$var) operators are not yet implemented (TODO).
• ✅ Functional Programming - Conditional operator, variables and bindings are implemented, as well as defining custom functions. Function signatures are parsed but not checked yet (TODO). Recursive functions are supported, but additional checks are needed here (TODO). Tail call optimization is not supported. Higher order functions are supported. 'Functions are closures', 'Partial function application' and 'Function chaining' features are supported.
• ✅ Regular Expressions - all is implemented, except for the unusual handling for excessive group indices in $replace() (If N is greater than the number of captured groups, then it is replaced by the empty string) which is not supported by .Net Regex.Replace(). Also match object does not yet have next property (TODO).
• ✔️ Date/Time Processing - All functions are implemented.

Operators

• ✅ Path Operators:
  • ✔️ . (Map)
  • ✔️ [ ... ] (Filter)
  • ✔️ ^( ... ) (Order-by)
  • ✔️ { ... } (Reduce)
  • ✔️ * (Wildcard)
  • ✔️ ** (Descendants)
  • ✔️ % (Parent) - done, expect for this issue
  • ✅ # (Positional variable binding) - (TODO)
  • ✅ @ (Context variable binding) - (TODO)
• ✔️ Numeric Operators - all, including .. (Range) operator.
• ✔️ Comparison Operators - all, including in (Inclusion) operator.
• ✔️ Boolean Operators.
• ✔️ Other Operators:
  • ✔️ & (Concatenation)
  • ✔️ ? : (Conditional)
  • ✔️ := (Variable binding)
  • ✔️ ~> (Chain)
  • ✔️ ... ~> | ... | ... | (Transform)

Function Library

• ✔️ String Functions:
  • ✔️ Implemented: $string(), $length(), $substring(), $substringBefore(), $substringAfter(), $uppercase(), $lowercase(), $trim(), $pad(), $contains(), $split(), $join(), $match(), $replace(), $eval(), $base64encode(), $base64decode(), $encodeUrlComponent(), $encodeUrl(), $decodeUrlComponent(), $decodeUrl()
  • ✅ There's a discrepancy when handling UTF-16 surrogate pairs. For example $length("\uD834\uDD1E") would return 2, while in original Jsonata-JS it would return 1.
• ✔️ Numeric Functions:
  • ✔️ Implemented: $number(), $abs(), $floor(), $ceil(), $round(), $power(), sqrt(), $random(), $formatNumber(), $formatBase(), $formatInteger(), $parseInteger()
  • ✅ Using C# custom and standard format strings for picture argument of $formatNumber(), $formatInteger() and $parseInteger() instead of XPath format used in JSonataJS.
• ✔️ Aggregation Functions:
  • ✔️ Implemented: $sum(), $max(), $min(), $average()
• ✔️ Boolean Functions:
  • ✔️ Implemented: $boolean(), $not(), $exists()
• ✔️ Array Functions:
  • ✔️ Implemented: $count(), $append(), $sort(), $reverse(), $shuffle(), $distinct(), $zip()
• ✔️ Object Functions:
  • ✔️ Implemented: $keys(), $lookup(), $spread(), $merge(), $sift(), $each(), $error(), $assert(), $type()
• ✔️ Date/Time functions:
  • ✔️ Implemented: $now(), $millis(), $fromMillis(), $toMillis()
  • ✅ Using C# custom and standard format strings for picture argument instead of XPath format used in JSonataJS.
• ✔️ Higher Order Functions:
  • ✔️ Implemented: $map(), $filter(), $single(), $reduce(), $sift()

Also, need to check all TODO: markers in the code (TODO).

Detailed results for the reference test suite

We use the test suite from original JSONata JS implementation to check consistency and completeness of the port. Current test results for the latest test run are:

Full and brief test reports are also in the repo. Below are current states of each test group in the suite: