Skip To Content
Toggle navigation
Packages
Upload
Statistics
Documentation
Downloads
Blog
Sign in
Advanced search filters
Frameworks
Include compatible frameworks
Framework Filter Mode
ALL
ANY
.NET
net8.0
net7.0
net6.0
net5.0
.NET Core
netcoreapp3.1
netcoreapp3.0
netcoreapp2.2
netcoreapp2.1
netcoreapp2.0
netcoreapp1.1
netcoreapp1.0
.NET Standard
netstandard2.1
netstandard2.0
netstandard1.6
netstandard1.5
netstandard1.4
netstandard1.3
netstandard1.2
netstandard1.1
netstandard1.0
.NET Framework
net481
net48
net472
net471
net47
net462
net461
net46
net452
net451
net45
net40
net35
net30
net20
Package type
All types
Dependency
.NET tool
Template
Options
Include prerelease
47 packages returned for Tags:"tokenizer"
Sort by
Relevance
Downloads
Recently updated
Tokenizer
by:
flipbit
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.5.2
348,527 total downloads
last updated
1/4/2021
Latest version:
2.2.2
tokenizer
tokens
pattern
matching
Tokenizer extracts structured information from blocks of text and reflects them onto .NET objects
Jokenizer.
Net
by:
umutozel
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
102,707 total downloads
last updated
2/10/2020
Latest version:
1.1.1
csharp
expression
parser
tokenizer
C# Expression parser and evaluator, inspired from jokenizer project.
AI.
Dev.
OpenAI.
GPT
by:
devis
.NET 5.0
.NET Core 3.1
88,250 total downloads
last updated
10/20/2022
Latest version:
1.0.2
OpenAI
GPT
GPT3
Tokenizer
Tokens
AI
OpenAI GPT utils, e.g. GPT3 Tokenizer
OpenNLP.
NET
by:
sergey_tihon
.NET 5.0
.NET Core 3.1
.NET Framework 4.6.1
102,113 total downloads
last updated
8/23/2022
Latest version:
1.9.4.1
NLP
OpenNLP
Tokenizer
Categorizer
Chunker
Parser
POS
NER
IKVM
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech...
More information
BERTTokenizers
by:
NMZivkovic
.NET 5.0
14,384 total downloads
last updated
9/9/2022
Latest version:
1.2.0
BERT
Tokenizer
charp
dotnet
This package contains tokenizers for following models: · BERT Base · BERT Large · BERT German · BERT Multilingual · BERT Base Uncased · BERT Large Uncased
LLMSharp.
Anthropic.
Tokenizer
by:
veerash-ayyagari
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
14,560 total downloads
last updated
10/4/2023
Latest version:
2.0.3
anthropic
tokenizer
claude
dotnet
tiktoken
Anthropic Claude BPE Tokenizer unofficial implementation
FastBertTokenizer
by:
georgjung
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
22,443 total downloads
last updated
4/30/2024
Latest version:
1.0.28
bert
tokenizer
wordpiece
llm
semantic-kernel
performance
ai
artificial-intelligence
ml
machine-learning
Fast and memory-efficient WordPiece tokenizer as it is used by BERT and others. Tokenizes text for further processing using NLP/language models.
SoftCircuits.
Parsing.
Helper
by:
SoftCircuits
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
23,543 total downloads
last updated
3/17/2024
Latest version:
5.1.0
text
parse
parser
parsing
text-parser
text-parsing
parsing-helper
.net
csharp
tokenizer
A .NET class library that makes it easier to parse text. The library tracks the current position within the text, ensures your code never attempts to access a character at an invalid index, and includes many...
More information
Conekta.
Xamarin
by:
bscheiman
.NET 6.0
12,375 total downloads
last updated
3/1/2015
Latest version:
1.0.6
boris
scheiman
bscheiman
conekta
xamarin
tokenizer
Tokenizador (Xamarin) para Conekta. Necesitas tener alguna libreria de servidor para usar el token.
Loretta.
CodeAnalysis.
Common
by:
gggkiller
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
7,852 total downloads
last updated
3/18/2024
Latest version:
0.2.13-nightly.1
parser
lexer
parsing
lexing
tokenizer
tokenizing
A shared package used by Loretta. Do not install this package manually, it will be added as a prerequisite by other packages that require it.
NltkNet
by:
skeleton
12,773 total downloads
last updated
1/1/2021
Latest version:
1.0.14
nltk
nlp
parser
tokenizer
categorizer
POS
NER
chunker
IronPython
NLTK python library wrapper for .NET
Loretta.
CodeAnalysis.
Lua
by:
gggkiller
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
7,031 total downloads
last updated
3/18/2024
Latest version:
0.2.13-nightly.1
parser
lexer
parsing
lexing
tokenizer
tokenizing
A GLua/Lua lexer, parser, code analysis, transformation and generation library.
VBF.
Compilers.
Scanners
by:
Ninputer
.NET Framework 4.5
6,081 total downloads
last updated
10/24/2015
Latest version:
1.0.5
VBF
Compiler
scanner
lexer
lex
tokenizer
VBF.Compilers.Scanners is a scanner builder. It contains a regular expression to DFA engine, can generate high performance scanners for unicode source text.
MetaParser
by:
dsisco11
3,870 total downloads
last updated
2/23/2023
Latest version:
1.0.7
parsing
tokenization
tokenizer
text
parser
Package Description
Loretta.
CodeAnalysis.
Lua.
Experimental
by:
gggkiller
.NET 5.0
.NET Core 2.0
.NET Standard 2.0
.NET Framework 4.6.1
5,423 total downloads
last updated
3/18/2024
Latest version:
0.2.13-nightly.1
parser
lexer
parsing
lexing
tokenizer
tokenizing
Experimental code that might become part of Loretta.CodeAnalysis.Lua.
Trl.
PegParser
by:
wcoetser
.NET 5.0
.NET Core 3.0
.NET Standard 2.1
5,051 total downloads
last updated
11/26/2020
Latest version:
1.3.0
Tokenizer
PEG
Grammer
Regular_Expressions
Regex
Parser
Parser_Generator
Trl.PegParser contains a tokenizer and a parser. The tokenizer uses regular expressions to define tokens, and exposes both matched and unmatched character ranges. The PEG Parser uses parsing expression grammers...
More information
Stringes
by:
Berkin
.NET Framework 4.5
15,646 total downloads
last updated
11/8/2015
Latest version:
1.4.8
stringes
tokenizer
token
lexer
string
substring
parser
ast
syntax
grammar
More tags
The Stringe is a wrapper for the .NET String object that tracks line, column, offset, and other metadata for substrings.
Tortoise
by:
m-wantia
.NET 6.0
2,257 total downloads
last updated
6/17/2022
Latest version:
0.1.1
lexer
lexical
tokenizer
Package Description
owl
by:
SplittyDev
6,419 total downloads
last updated
10/21/2014
Latest version:
1.0.5407.37498
html
preprocessor
compiler
lexer
parser
scanner
tokenizer
owl
A html preprocessor which allows you to write html code using a beautiful syntax
CEM.
Utilities.
Tokenizer
by:
dominic.proulx
.NET 6.0
1,458 total downloads
last updated
7/29/2022
Latest version:
1.0.1
CEM
Utilities
Tokenizer
Classe qui gère le découpage d'une chaine en paires de clés/valeurs et permet de la reconstruire avec des valeurs de remplacement ou des valeurs additionnelles
Previous
Next