Deckard 0.0.23
See the version list below for details.
dotnet add package Deckard --version 0.0.23
NuGet\Install-Package Deckard -Version 0.0.23
<PackageReference Include="Deckard" Version="0.0.23" />
paket add Deckard --version 0.0.23
#r "nuget: Deckard, 0.0.23"
// Install Deckard as a Cake Addin #addin nuget:?package=Deckard&version=0.0.23 // Install Deckard as a Cake Tool #tool nuget:?package=Deckard&version=0.0.23
Deckard: A Highly Scalable Cyclic Priority Queue
Deckard is a priority queue system inspired by projects such as Google Cloud PubSub, Nats, Kafka, and others. Its main distinction lies in its ability to associate a priority score with each message and have a queue that can be optionally cyclic. This means that messages can be delivered again after a user-managed time. Additionally, Deckard implements a locking mechanism to prevent message processing for a specified duration.
Briefly:
- An application inserts a message to be queued and its configuration (TTL, metadata, payload, priority score, etc).
- The message will be prioritized with a default timestamp-based algorithm if the provided score is 0 (the default value).
- A worker application pull messages from Deckard at regular intervals and performs any processing.
- When it finishes processing a message, the application must notify with the processing result.
- When notifying, the application may provide a lock time, to lock the message for a certain duration of time before being requeued and delivered again.
- It is also possible to notify a message changing its priority score.
- When the message's TTL is reached, it stops being delivered;
- For some use cases the TTL can be set as infinite.
- An application can also remove the message when notifying.
Motivation
Deckard was born from an initiative to simplify and unify applications called Frontiers which were part of STILINGUE's orchestration system for data gathering.
Several different implementations were made, but they proved to have limitations such as:
- Debugging
- Genericity
- Scalability
- Auditability
- Observability
- Prioritization
- Developer friendliness
Deckard was created to solve these problems and to be used in any application that needs to queue data and prioritize it.
The main objectives of the project are:
- Generic: any data and application should be able to use Deckard to queue messages;
- Observable: it should be easy to visualize what happened in a request and easy to investigate problems with audit;
- Developer friendly: easy to understand and pleasant to use;
- Scalable: it should be possible to support thousands of requests per second and millions of messages;
What Deckard is not?
It is important to note that Deckard is not a conventional messaging/queue system. If your use case does not involve priority, cyclic queuing, or a locking mechanism, it is recommended to consider alternatives such as GCP PubSub, Kafka, RabbitMQ, Azure Service Bus, Amazon SQS, or other messaging systems.
Project Status
Deckard has been used in a production environment since 2019, handling billions of messages and thousands of requests per second. Since its initial internal release, Deckard has undergone significant improvements and enhancements.
The performance and reliability of Deckard are directly dependent on the storage and cache services. Therefore, it is crucial to configure MongoDB and Redis correctly to ensure optimal performance. Redis is responsible for storing all priority queues and has a significant impact on Deckard's performance, while MongoDB usage is simpler and has a lower impact on performance.
Currently, we have projects that utilize MongoDB in both a virtual machine environment and a Kubernetes environment. However, it's worth noting that we have had more extensive usage of MongoDB in a virtual machine (VM) environment compared to Kubernetes. When deploying Deckard, it is important to configure MongoDB properly, following MongoDB's recommendations for production environments (documented here: MongoDB Production Notes).
While the project is being released with a 0.0.x
version due to necessary code modifications for open-sourcing, it has been thoroughly tested and proven to be reliable in production scenarios.
Please refer to our issues section for more details. Your feedback and suggestions are highly appreciated and can be shared in our discussions.
Getting Started
To quickly get started with Deckard, please consult our getting started guide, which provides step-by-step instructions. Additionally, we have provided client documentation for various programming languages to assist you in integrating Deckard into your projects.
You can also access our documentation on Deckard's components here. More details on the project structure can be found here.
Running Deckard
Here's a quick guide on how to run Deckard. You should check the getting started guide for more details.
On Linux
you can run it from sources with:
make run
You can run it with Docker:
docker run --rm -p 8081:8081 blipai/deckard
By default for Docker and Linux it will use a memory storage and a memory cache engine.
To change the default configuration see the configuration section.
You can also run it in a Kubernetes cluster using Helm:
It will deploy a MongoDB for storage and a Redis for cache.
Check the chart values.yaml to see all available configurations.
helm repo add deckard https://takenet.github.io/deckard/
helm install deckard deckard/deckard
You may also download the latest release from the releases page and execute it.
Configuration
We currently use the viper project to manage configurations and the current implementation delegates the configuration to environment variables.
All available environment variables are listed below:
Overall Configuration
Environment Variable | Default | Description |
---|---|---|
DECKARD_DEBUG |
false |
To enable debug mode to log more information. |
DECKARD_LOG_TYPE |
json |
The log type to use. Available: json, text |
DECKARD_GRPC_ENABLED |
true |
To enable the gRPC service. You can disable gRPC service if you want an instance to perform only housekeeper tasks. |
DECKARD_GRPC_PORT |
8081 |
The gRPC port to listen. |
Cache Configuration
Environment Variable | Default | Description |
---|---|---|
DECKARD_CACHE_TYPE |
MEMORY |
The cache implementation to use. Available: MEMORY, REDIS |
DECKARD_CACHE_URI |
The cache Connection URI to connect with the cache service. Currently only a Redis URI is accepted. It will take precedence over any other environment variable related to the connection Redis connection. | |
DECKARD_REDIS_ADDRESS |
localhost |
The redis address to connect while using redis cache implementation. It will be overriden by DECKARD_CACHE_URI if present. |
DECKARD_REDIS_PASSWORD |
The redis password to connect while using redis cache implementation. It will be overriden by DECKARD_CACHE_URI if present. |
|
DECKARD_REDIS_PORT |
6379 |
The redis port to connect while using redis cache implementation. It will be overriden by DECKARD_CACHE_URI if present. |
DECKARD_REDIS_DB |
0 |
The database to use while using redis cache implementation. It will be overriden by DECKARD_CACHE_URI if present. |
Storage Configuration
Environment Variable | Default | Description |
---|---|---|
DECKARD_STORAGE_TYPE |
MEMORY |
The storage implementation to use. Available: MEMORY, MONGODB |
DECKARD_STORAGE_URI |
The storage Connection URI to connect with the storage service. Currently only a MongoDB URI is accepted. It can override any other environment variable related to the connection MongoDB connection since it takes precedence. | |
DECKARD_MONGODB_ADDRESSES |
localhost:27017 |
The MongoDB addresses separated by comma to connect while using MongoDB storage implementation. It can be overridden by DECKARD_STORAGE_URI . |
DECKARD_MONGODB_AUTH_DB |
The MongoDB auth database to authenticate while using MongoDB storage implementation. It can be overridden by DECKARD_STORAGE_URI . |
|
DECKARD_MONGODB_PASSWORD |
The MongoDB password to authenticate while using MongoDB storage implementation. It can be overridden by DECKARD_STORAGE_URI . |
|
DECKARD_MONGODB_DATABASE |
deckard |
The MongoDB database to use to store messages while using MongoDB storage implementation. |
DECKARD_MONGODB_COLLECTION |
queue |
The MongoDB collection to use to store messages while using MongoDB storage implementation. |
DECKARD_MONGODB_USER |
The MongoDB user to authenticate while using MongoDB storage implementation. It can be overridden by DECKARD_STORAGE_URI . |
|
DECKARD_MONGODB_SSL |
false |
To enable SSL while using MongoDB storage implementation. It can be overridden by DECKARD_STORAGE_URI . |
DECKARD_MONGODB_QUEUE_CONFIGURATION_COLLECTION |
queue_configuration |
The MongoDB collection to use to store queue configurations while using MongoDB storage implementation. |
Housekeeper Configuration
Environment Variable | Default | Description |
---|---|---|
DECKARD_HOUSEKEEPER_ENABLED |
true |
To enable housekeeper tasks. |
DECKARD_HOUSEKEEPER_TASK_TIMEOUT_DELAY |
1s |
The delay between each timeout task execution. |
DECKARD_HOUSEKEEPER_TASK_UNLOCK_DELAY |
1s |
The delay between each unlock task execution. |
DECKARD_HOUSEKEEPER_TASK_UPDATE_DELAY |
1s |
The delay between each update task execution. |
DECKARD_HOUSEKEEPER_TASK_TTL_DELAY |
1s |
The delay between each ttl task execution. |
DECKARD_HOUSEKEEPER_TASK_MAX_ELEMENTS_DELAY |
1s |
The delay between each max elements task execution. |
DECKARD_HOUSEKEEPER_TASK_METRICS_DELAY |
60s |
The delay between each metrics task execution. |
Audit Configuration
Environment Variable | Default | Description |
---|---|---|
DECKARD_AUDIT_ENABLED |
false |
To enable auditing. |
DECKARD_ELASTIC_ADDRESS |
http://localhost:9200/ |
A ElasticSearch address to connect to store audit information. |
DECKARD_ELASTIC_PASSWORD |
A ElasticSearch password to connect to store audit information. | |
DECKARD_ELASTIC_USER |
A ElasticSearch user to connect to store audit information. |
Metrics Configuration
Environment Variable | Default | Description |
---|---|---|
DECKARD_METRICS_ENABLED |
true |
To enable exporting metrics to the metrics endpoint. |
DECKARD_METRICS_PORT |
22022 |
The metrics http port to listn. |
DECKARD_METRICS_PATH |
/metrics |
The metrics http path to expose metrics. |
DECKARD_METRICS_HISTOGRAM_BUCKETS |
0,1,2,5,10,15,20,30,35,50,100,200,400,600,800,1000,1500,2000,5000,10000 |
The histogram buckets to use to expose metrics. |
DECKARD_METRICS_OPENMETRICS_ENABLED |
true |
If true, the OpenMetrics encoding is added to the possible options during content negotiation. |
TLS Configuration
To learn more about gRPC TLS configuration please refer to gRPC Auth.
Environment Variable | Default | Description |
---|---|---|
DECKARD_TLS_CLIENT_AUTH_TYPE |
NoClientCert |
The type of client authentication TLS verification to use. Available: NoClientCert , RequestClientCert , RequireAnyClientCert , VerifyClientCertIfGiven , RequireAndVerifyClientCert . |
DECKARD_TLS_SERVER_CERT_FILE_PATHS |
A comma-delimited list of absolute file paths to PEM-encoded certificates. | |
DECKARD_TLS_SERVER_KEY_FILE_PATHS |
A comma-delimited list of absolute file paths to PEM-encoded private keys. | |
DECKARD_TLS_CLIENT_CERT_FILE_PATHS |
A comma-delimited list of absolute file paths to PEM-encoded certificates to enable mutual TLS. |
Contributing
We are always looking for new contributors to help us improve Deckard.
If you want to contribute to Deckard, please read our contributing guide which includes how to build, run and test Deckard and a complete description of our project structure.
License
Deckard is licensed under the MIT License.
Acknowledgments
We would like to thank the following people for their initial contributions building Deckard's first version:
- Lucas Soares: @lucasoares
- Gustavo Paiva: @paivagustavo
- Cézar Augusto: @cezar-tech
- Júnior Rhis: @juniorrhis
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
-
net6.0
- Google.Protobuf (>= 3.21.12)
- Grpc.Net.Client (>= 2.52.0)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
Version | Downloads | Last updated |
---|---|---|
0.0.28 | 675 | 2/5/2024 |
0.0.27 | 87 | 2/5/2024 |
0.0.26 | 167 | 8/23/2023 |
0.0.25 | 151 | 8/7/2023 |
0.0.24 | 143 | 7/20/2023 |
0.0.23 | 138 | 7/7/2023 |
0.0.22 | 143 | 7/7/2023 |
0.0.21 | 146 | 7/7/2023 |
0.0.20 | 153 | 7/7/2023 |
0.0.19 | 155 | 7/5/2023 |
0.0.18 | 154 | 7/4/2023 |
0.0.17 | 145 | 6/30/2023 |
0.0.16 | 159 | 6/27/2023 |
0.0.15 | 157 | 6/26/2023 |
0.0.10 | 192 | 4/20/2023 |
0.0.9 | 238 | 3/7/2023 |