Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Frameworks
.NET
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
37 packages returned for Tags:"Tokenizer"
Sort by
Relevance
Downloads
Recently updated
Tokenizer
by:
flipbit
242,479 total downloads
last updated
1/4/2021
Latest version:
2.2.2
tokenizer
tokens
pattern
matching
Tokenizer extracts structured information from blocks of text and reflects them onto .NET objects
Jokenizer.
Net
by:
umutozel
66,140 total downloads
last updated
2/10/2020
Latest version:
1.1.1
csharp
expression
parser
tokenizer
C# Expression parser and evaluator, inspired from jokenizer project.
OpenNLP.
NET
by:
sergey_tihon
53,814 total downloads
last updated
8/23/2022
Latest version:
1.9.4.1
NLP
OpenNLP
Tokenizer
Categorizer
Chunker
Parser
POS
NER
IKVM
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech...
More information
AI.
Dev.
OpenAI.
GPT
by:
devis
9,826 total downloads
last updated
10/20/2022
Latest version:
1.0.2
OpenAI
GPT
GPT3
Tokenizer
Tokens
AI
OpenAI GPT utils, e.g. GPT3 Tokenizer
SoftCircuits.
Parsing.
Helper
by:
SoftCircuits
17,409 total downloads
last updated
10/2/2022
Latest version:
4.2.0
text
parse
parser
parsing
text-parser
text-parsing
parsing-helper
.net
csharp
tokenizer
A .NET class library that makes it easier to parse text. The library tracks the current position within the text, ensures your code never attempts to access a character at an invalid index, and includes many...
More information
Conekta.
Xamarin
by:
bscheiman
8,941 total downloads
last updated
3/1/2015
Latest version:
1.0.6
boris
scheiman
bscheiman
conekta
xamarin
tokenizer
Tokenizador (Xamarin) para Conekta. Necesitas tener alguna libreria de servidor para usar el token.
NltkNet
by:
skeleton
10,059 total downloads
last updated
1/1/2021
Latest version:
1.0.14
nltk
nlp
parser
tokenizer
categorizer
POS
NER
chunker
IronPython
NLTK python library wrapper for .NET
BERTTokenizers
by:
NMZivkovic
3,180 total downloads
last updated
9/9/2022
Latest version:
1.2.0
BERT
Tokenizer
charp
dotnet
This package contains tokenizers for following models: · BERT Base · BERT Large · BERT German · BERT Multilingual · BERT Base Uncased · BERT Large Uncased
OpenNLP.
NET.
Signed
by:
tastybeefalo
8,992 total downloads
last updated
2/26/2020
Latest version:
1.9.1
NLP
OpenNLP
Tokenizer
Categorizer
Chunker
Parser
POS
NER
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech...
More information
VBF.
Compilers.
Scanners
by:
Ninputer
5,441 total downloads
last updated
10/24/2015
Latest version:
1.0.5
VBF
Compiler
scanner
lexer
lex
tokenizer
VBF.Compilers.Scanners is a scanner builder. It contains a regular expression to DFA engine, can generate high performance scanners for unicode source text.
Loretta.
CodeAnalysis.
Common
by:
gggkiller
4,517 total downloads
last updated
11/15/2022
Latest version:
0.2.11-nightly.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
A shared package used by Loretta. Do not install this package manually, it will be added as a prerequisite by other packages that require it.
Loretta.
CodeAnalysis.
Lua
by:
gggkiller
3,994 total downloads
last updated
11/15/2022
Latest version:
0.2.11-nightly.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
A GLua/Lua lexer, parser, code analysis, transformation and generation library.
Stringes
by:
Berkin
14,018 total downloads
last updated
11/8/2015
Latest version:
1.4.8
stringes
tokenizer
token
lexer
string
substring
parser
ast
syntax
grammar
More tags
The Stringe is a wrapper for the .NET String object that tracks line, column, offset, and other metadata for substrings.
Trl.
PegParser
by:
wcoetser
3,868 total downloads
last updated
11/26/2020
Latest version:
1.3.0
Tokenizer
PEG
Grammer
Regular_Expressions
Regex
Parser
Parser_Generator
Trl.PegParser contains a tokenizer and a parser. The tokenizer uses regular expressions to define tokens, and exposes both matched and unmatched character ranges. The PEG Parser uses parsing expression grammers...
More information
Tortoise
by:
m-wantia
1,554 total downloads
last updated
6/17/2022
Latest version:
0.1.1
lexer
lexical
tokenizer
Package Description
Loretta.
CodeAnalysis.
Lua.
Experimental
by:
gggkiller
3,055 total downloads
last updated
11/15/2022
Latest version:
0.2.11-nightly.10
parser
lexer
parsing
lexing
tokenizer
tokenizing
Experimental code that might become part of Loretta.CodeAnalysis.Lua.
owl
by:
SplittyDev
5,431 total downloads
last updated
10/21/2014
Latest version:
1.0.5407.37498
html
preprocessor
compiler
lexer
parser
scanner
tokenizer
owl
A html preprocessor which allows you to write html code using a beautiful syntax
TokenCs
by:
zeplar_exe
1,621 total downloads
last updated
8/10/2022
Latest version:
1.0.5
token
tokenizer
lexer
parse
parser
A simple library to abstract the gritty process of lexical parsing.
MetaParser
by:
dsisco11
1,740 total downloads
last updated
2/23/2023
Latest version:
1.0.7
parsing
tokenization
tokenizer
text
parser
Package Description
CEM.
Utilities.
Tokenizer
by:
dominic.proulx
1,000 total downloads
last updated
7/29/2022
Latest version:
1.0.1
CEM
Utilities
Tokenizer
Classe qui gère le découpage d'une chaine en paires de clés/valeurs et permet de la reconstruire avec des valeurs de remplacement ou des valeurs additionnelles
Previous
Next