Messerli.Lexer 1.1.0

.NET 6.0 .NET Standard 2.0
dotnet add package Messerli.Lexer --version 1.1.0
NuGet\Install-Package Messerli.Lexer -Version 1.1.0
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Messerli.Lexer" Version="1.1.0" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Messerli.Lexer --version 1.1.0
#r "nuget: Messerli.Lexer, 1.1.0"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Messerli.Lexer as a Cake Addin
#addin nuget:?package=Messerli.Lexer&version=1.1.0

// Install Messerli.Lexer as a Cake Tool
#tool nuget:?package=Messerli.Lexer&version=1.1.0


A simple lexer which gives you a stream of Tokens from a string and some simple to define rules.

Getting started

You have to define at least an EpsilonToken (you can name it differently, or use a class) which signals the end of the input.

public sealed record EpsilonToken : IToken;

Then you define the other token you want to use:

internal sealed record PlusToken(string Number): IToken;
internal sealed record MinusToken(string Number): IToken;
internal sealed record NumberToken(string Number): IToken;

Now you need some rules, the TokenWalker expects an IEnumerable<ILexerRule>.

internal static class ExampleRules
    public static IEnumerable<ILexerRule> GetRules()
        yield return new SimpleLexerRule<PlusToken>("+");
        yield return new SimpleLexerRule<MinusToken>("-");
        yield return new LexerRule(char.Digit, ScanNumber);

    private static Lexeme ScanIdentifier(ILexerReader reader)
        var startPosition = reader.Position;
        var stringBuilder = new StringBuilder();
        while (reader.Peek().Match(none: false, some: char.IsDigit))
            stringBuilder.Append(reader.Read().Match(none: ' ', some: Identity));

        return new Lexeme(new IdentifierToken(stringBuilder.ToString()), new Position(startPosition, reader.Position - startPosition));

We see some simple rules which just need the string, you can also define overlapping simple rules like "=" and "==" the longer strings take precedent.

We see also how you scan a more complex Lexeme, where you see how you should create a Lexeme out of your Tokens and you see how to handle the position information for the Lexem.



If you have an expression which goes over multiple lines, you usually want the position as Line number + character on that line. The default setting already handles everything as long as you declare the token which denotes a new-line with the interface ILineBreakToken.

internal sealed record NewLine(): IToken, ILineBreakToken;

Special Tasks

Overriding Line calculation

You have to instantiate the TokeWalker yourself with your own instance of ILineCalculator.

I need to do something different depending on a Lexem already parsed.

LexerRuleWithContext gives you access to all Lexems already produced.

Before you use it, think hard if you could solve it differently, maybe in the parsing phase instead of in the lexer.

Product Compatible and additional computed target framework versions.
.NET net5.0 was computed.  net5.0-windows was computed.  net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed. 
.NET Core netcoreapp2.0 was computed.  netcoreapp2.1 was computed.  netcoreapp2.2 was computed.  netcoreapp3.0 was computed.  netcoreapp3.1 was computed. 
.NET Standard netstandard2.0 is compatible.  netstandard2.1 was computed. 
.NET Framework net461 was computed.  net462 was computed.  net463 was computed.  net47 was computed.  net471 was computed.  net472 was computed.  net48 was computed.  net481 was computed. 
MonoAndroid monoandroid was computed. 
MonoMac monomac was computed. 
MonoTouch monotouch was computed. 
Tizen tizen40 was computed.  tizen60 was computed. 
Xamarin.iOS xamarinios was computed. 
Xamarin.Mac xamarinmac was computed. 
Xamarin.TVOS xamarintvos was computed. 
Xamarin.WatchOS xamarinwatchos was computed. 
Compatible target framework(s)
Additional computed target framework(s)
Learn more about Target Frameworks and .NET Standard.
  • .NETStandard 2.0

    • Funcky (>= 3.0.0 && < 4.0.0)
  • net6.0

    • Funcky (>= 3.0.0 && < 4.0.0)

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.1.0 619 9/14/2022
1.0.4 406 6/28/2022
1.0.3 351 6/28/2022
1.0.2 328 6/28/2022
1.0.1 337 6/28/2022
1.0.0 339 6/28/2022