SiLA2.Database.NoSQL 10.2.2

SiLA2.Database.NoSQL

Lightweight NoSQL Document Database for SiLA2 Applications

NuGet Package	SiLA2.Database.NoSQL on NuGet.org
Repository	https://gitlab.com/SiLA2/sila_csharp
SiLA Standard	https://sila-standard.com
License	MIT

Overview

SiLA2.Database.NoSQL is an optional component of the sila_csharp implementation that provides lightweight, embedded NoSQL document database capabilities using LiteDB. This module offers a repository pattern abstraction for document-oriented data storage, eliminating the need for external database servers while maintaining flexibility for complex data structures.

Key Value Proposition

Traditional SQL Approach:

Entity Framework → SQL Server/SQLite → Relational Tables → Join Queries

NoSQL Approach:

Repository Pattern → LiteDB → BSON Documents → Direct Object Storage

This package is ideal for:

• AnIML scientific data storage - Store experimental results in flexible document format
• Configuration and settings - Persist complex configuration objects without schema migrations
• Audit logs and event storage - Store heterogeneous log data with varying structures
• Embedded applications - No external database server required, single-file deployment
• Rapid prototyping - Schema-less design allows quick iteration without migrations

When to Use NoSQL vs SQL in SiLA2 Applications

Scenario	Use SiLA2.Database.NoSQL	Use SiLA2.Database.SQL
Storing AnIML experimental data	✅ Yes (recommended)	❌ No (poor fit for hierarchical data)
Configuration and settings	✅ Yes	⚠️ Optional
User authentication/authorization	❌ No	✅ Yes (better relational integrity)
Audit logs with flexible schemas	✅ Yes	⚠️ Optional
Complex relational queries	❌ No	✅ Yes (SQL is better)
Large datasets (>100GB)	❌ No (use SQL or specialized DB)	✅ Yes
Schema-less or evolving data models	✅ Yes	❌ No (requires migrations)
Single-file deployment requirement	✅ Yes (embedded)	⚠️ Partial (SQLite only)

Installation

Install via NuGet Package Manager:

dotnet add package SiLA2.Database.NoSQL

Or via Package Manager Console:

Install-Package SiLA2.Database.NoSQL

Requirements

• .NET 10.0+
• LiteDB 5.0.21+ (automatically installed as dependency)
• Microsoft.Extensions.DependencyInjection 10.0.2+ (for ASP.NET Core integration)

Core Concepts

1. LiteDB - Embedded NoSQL Database

LiteDB is a serverless, embedded NoSQL database for .NET, similar to SQLite but designed for document storage rather than relational data.

Key Characteristics:

• Single-file database - Entire database stored in one .db file
• BSON document format - Binary JSON for efficient serialization
• No server required - Runs in-process with your application
• ACID transactions - Supports atomic operations and rollback
• Indexed queries - Create indexes on document properties for fast lookups
• Thread-safe - Concurrent read/write with proper locking
• Small footprint - ~450KB assembly size

Comparison to Other Embedded Databases:

Feature	LiteDB	SQLite	RavenDB Embedded
Data Model	Document (BSON)	Relational (SQL)	Document (JSON)
Schema	Schema-less	Fixed schema	Schema-less
Query Language	LINQ-like	SQL	RQL/LINQ
File Size	Single file	Single file	Multiple files
Performance	Good for documents	Excellent for relational	Excellent (commercial)
License	MIT (free)	Public domain	AGPL/Commercial
Best For	SiLA2 NoSQL needs	SiLA2 SQL needs	Enterprise applications

2. Repository Pattern

The repository pattern abstracts data access logic, providing a clean separation between business logic and data persistence. This module implements a generic repository interface that works with any entity type.

Benefits:

• Testability - Easy to mock repositories for unit testing
• Consistency - Standard CRUD interface across all entity types
• Flexibility - Switch database implementations without changing business logic
• Encapsulation - Hide LiteDB-specific code from application layer

Pattern Structure:

IBaseRepository<T> (Interface)
    ↓
BaseRepository<T> (Abstract Base Class)
    ↓
MyEntityRepository (Concrete Implementation)

3. BSON Document Storage

BSON (Binary JSON) is the native storage format for LiteDB. Entities are automatically serialized to BSON when saved.

Supported Data Types:

• Primitive types: int, long, double, decimal, bool, string
• Date/time: DateTime, DateTimeOffset
• Identifiers: ObjectId, Guid, int, long
• Collections: List<T>, Dictionary<TKey, TValue>, arrays
• Nested objects: Complex types and object graphs
• Binary data: byte[] for blobs

Attributes for Customization:

using LiteDB;

public class MyEntity
{
    [BsonId]              // Marks the identifier property
    public ObjectId Id { get; set; }

    [BsonField("name")]   // Custom field name in BSON
    public string DisplayName { get; set; }

    [BsonIgnore]          // Exclude from serialization
    public string InternalCache { get; set; }

    [BsonCtor]            // Mark constructor for deserialization
    public MyEntity(string name) { ... }
}

4. When to Use NoSQL vs SQL in SiLA2 Applications

Use NoSQL (SiLA2.Database.NoSQL) when:

• Data has complex, nested, or hierarchical structures (AnIML documents)
• Schema changes frequently or is unknown at design time
• You need embedded database with no server setup
• Data is naturally document-oriented (JSON/XML equivalents)
• You want simple CRUD operations without complex queries

Use SQL (SiLA2.Database.SQL) when:

• Data has strong relational integrity requirements (foreign keys, cascades)
• You need complex queries with joins, aggregations, and reporting
• Multi-user access with high concurrency and ACID guarantees
• Data volume exceeds 100GB or requires scalability
• You need mature tooling and database administration features

Can Use Both: Many SiLA2 servers use both modules:

• SQL for user authentication, feature configuration, device state
• NoSQL for AnIML experimental data, audit logs, temporary results

Architecture & Components

Component Overview

┌─────────────────────────────────────────────────────────────┐
│              SiLA2 Application Layer                        │
│  (Feature Services, Business Logic)                         │
└───────────────────────────┬─────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│         MyEntityRepository : BaseRepository<MyEntity>       │
│  (Custom repository with domain-specific methods)           │
└───────────────────────────┬─────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│         BaseRepository<T> : IBaseRepository<T>              │
│  - Create(T) / Create(T, out object)                        │
│  - All() → IEnumerable<T>                                   │
│  - FindById(object) → T                                     │
│  - Update(T)                                                │
│  - Delete(object) → bool                                    │
│  + Collection → ILiteCollection<T>  (direct access)         │
│  + DB → ILiteDatabase                (advanced operations)  │
└───────────────────────────┬─────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                    ILiteDatabase                            │
│  (LiteDB instance - manages collections, transactions)      │
└───────────────────────────┬─────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              database.db (BSON file on disk)                │
└─────────────────────────────────────────────────────────────┘

IBaseRepository<T>

Purpose: Defines the standard CRUD interface for all entity repositories.

Interface Definition:

public interface IBaseRepository<T>
{
    T Create(T data, out object id);     // Create with ID output
    T Create(T data);                     // Create without ID output
    IEnumerable<T> All();                 // Get all documents
    T FindById(object id);                // Get by ID
    void Update(T entity);                // Update or insert (upsert)
    bool Delete(object id);               // Delete by ID
}

Design Philosophy:

• Generic type parameter - Works with any entity class
• Minimal surface area - Only essential CRUD operations
• Extensible - Inherit and add domain-specific methods
• Flexible ID types - Supports ObjectId, Guid, int, long, etc.

BaseRepository<T>

Purpose: Provides concrete implementation of IBaseRepository<T> using LiteDB as the backing store.

Key Features:

• Automatic collection management - Collection created on first access
• Virtual methods - All methods can be overridden in derived classes
• Direct collection access - Collection property for advanced queries
• Database access - DB property for transactions and multi-collection operations

Protected Members:

public abstract class BaseRepository<T> : IBaseRepository<T>
{
    public ILiteDatabase DB { get; }              // Access to database
    public ILiteCollection<T> Collection { get; }  // Access to collection

    protected BaseRepository(ILiteDatabase db)     // Constructor
}

Extension Points: Derived classes can:

1. Override virtual methods to customize behavior
2. Use Collection for advanced LINQ queries
3. Use DB for transactions spanning multiple collections
4. Add domain-specific query methods

DocumentDbServiceCollectionExtensions

Purpose: Simplifies dependency injection setup for LiteDB in ASP.NET Core applications.

Extension Methods:

public static class DocumentDbServiceCollectionExtensions
{
    // Register ILiteDatabase as singleton
    void RegisterDocumentDatabase(
        this IServiceCollection services,
        string path,
        out ILiteDatabase db);

    // Register ILiteCollection<T> as singleton
    void RegisterDocumentDatabaseTypes<T>(
        this IServiceCollection services,
        ILiteDatabase db);
}

Usage Pattern:

// In Program.cs or Startup.cs
services.RegisterDocumentDatabase("myapp.db", out var db);
services.RegisterDocumentDatabaseTypes<Customer>(db);
services.RegisterDocumentDatabaseTypes<Order>(db);

Usage Examples

Setup

All examples assume you have registered the database in your dependency injection container. See the "Dependency Injection Setup" example below.

Example 1: Basic Entity and Repository

Step 1: Define Your Entity Class

using LiteDB;

public class TemperatureReading
{
    [BsonId]
    public ObjectId Id { get; set; }

    public DateTime Timestamp { get; set; }

    public double Temperature { get; set; }

    public string SensorId { get; set; }

    public string Unit { get; set; } = "Celsius";
}

Step 2: Create a Repository

using SiLA2.Database.NoSQL;
using LiteDB;

public interface ITemperatureReadingRepository : IBaseRepository<TemperatureReading>
{
    // Add domain-specific query methods here
    IEnumerable<TemperatureReading> GetReadingsBySensor(string sensorId);
    IEnumerable<TemperatureReading> GetReadingsInRange(DateTime start, DateTime end);
}

public class TemperatureReadingRepository : BaseRepository<TemperatureReading>,
                                            ITemperatureReadingRepository
{
    public TemperatureReadingRepository(ILiteDatabase db) : base(db)
    {
        // Optional: Create indexes for better query performance
        Collection.EnsureIndex(x => x.SensorId);
        Collection.EnsureIndex(x => x.Timestamp);
    }

    public IEnumerable<TemperatureReading> GetReadingsBySensor(string sensorId)
    {
        return Collection.Query()
            .Where(x => x.SensorId == sensorId)
            .OrderByDescending(x => x.Timestamp)
            .ToList();
    }

    public IEnumerable<TemperatureReading> GetReadingsInRange(DateTime start, DateTime end)
    {
        return Collection.Query()
            .Where(x => x.Timestamp >= start && x.Timestamp <= end)
            .ToList();
    }
}

Example 2: Dependency Injection Setup

In Program.cs (ASP.NET Core):

using SiLA2.Database.NoSQL;
using LiteDB;

var builder = WebApplication.CreateBuilder(args);

// Method 1: Using extension methods (recommended)
builder.Services.RegisterDocumentDatabase(
    "Data/MyAppDatabase.db",
    out ILiteDatabase db);

// Register collections for specific types
builder.Services.RegisterDocumentDatabaseTypes<TemperatureReading>(db);
builder.Services.RegisterDocumentDatabaseTypes<AnIMLType>(db);

// Register repositories
builder.Services.AddSingleton<ITemperatureReadingRepository, TemperatureReadingRepository>();

// Method 2: Manual registration with connection string options
builder.Services.AddSingleton<ILiteDatabase>(sp =>
    new LiteDatabase("filename=Data/MyApp.db;password=mySecretPassword"));

var app = builder.Build();

Connection String Options:

LiteDB supports various connection string parameters:

// Encrypted database
"filename=data.db;password=myPassword"

// Read-only mode
"filename=data.db;readonly=true"

// Upgrade from older LiteDB version
"filename=data.db;upgrade=true"

// Custom timeout (milliseconds)
"filename=data.db;timeout=30000"

// Shared mode for multi-process access (Windows only)
"filename=data.db;mode=Shared"

// Complete example
"filename=C:\\Data\\myapp.db;password=secret;timeout=60000;upgrade=true"

Example 3: CRUD Operations

using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("api/[controller]")]
public class TemperatureController : ControllerBase
{
    private readonly ITemperatureReadingRepository _repository;

    public TemperatureController(ITemperatureReadingRepository repository)
    {
        _repository = repository;
    }

    // CREATE - Add new reading
    [HttpPost]
    public IActionResult CreateReading([FromBody] TemperatureReading reading)
    {
        reading.Timestamp = DateTime.UtcNow;
        var created = _repository.Create(reading, out object newId);

        return CreatedAtAction(
            nameof(GetReading),
            new { id = newId.ToString() },
            created);
    }

    // READ - Get all readings
    [HttpGet]
    public IActionResult GetAllReadings()
    {
        var readings = _repository.All();
        return Ok(readings);
    }

    // READ - Get by ID
    [HttpGet("{id}")]
    public IActionResult GetReading(string id)
    {
        var reading = _repository.FindById(new ObjectId(id));
        if (reading == null)
            return NotFound();

        return Ok(reading);
    }

    // UPDATE - Modify existing reading
    [HttpPut("{id}")]
    public IActionResult UpdateReading(string id, [FromBody] TemperatureReading reading)
    {
        reading.Id = new ObjectId(id);
        _repository.Update(reading);  // Upsert operation
        return NoContent();
    }

    // DELETE - Remove reading
    [HttpDelete("{id}")]
    public IActionResult DeleteReading(string id)
    {
        bool deleted = _repository.Delete(new ObjectId(id));
        if (!deleted)
            return NotFound();

        return NoContent();
    }

    // Custom query method
    [HttpGet("sensor/{sensorId}")]
    public IActionResult GetReadingsBySensor(string sensorId)
    {
        var readings = _repository.GetReadingsBySensor(sensorId);
        return Ok(readings);
    }
}

Example 4: Advanced Queries Using Collection Property

The Collection property provides direct access to LiteDB's powerful query API:

public class TemperatureReadingRepository : BaseRepository<TemperatureReading>,
                                            ITemperatureReadingRepository
{
    public TemperatureReadingRepository(ILiteDatabase db) : base(db) { }

    // Query with multiple conditions
    public IEnumerable<TemperatureReading> GetHighTemperatureReadings(
        string sensorId,
        double threshold)
    {
        return Collection.Query()
            .Where(x => x.SensorId == sensorId && x.Temperature > threshold)
            .OrderByDescending(x => x.Temperature)
            .Limit(100)
            .ToList();
    }

    // Pagination
    public IEnumerable<TemperatureReading> GetReadingsPage(int page, int pageSize)
    {
        return Collection.Query()
            .OrderByDescending(x => x.Timestamp)
            .Skip((page - 1) * pageSize)
            .Limit(pageSize)
            .ToList();
    }

    // Aggregation (count)
    public int GetReadingCountBySensor(string sensorId)
    {
        return Collection.Count(x => x.SensorId == sensorId);
    }

    // Full-text search (requires index)
    public IEnumerable<TemperatureReading> SearchBySensorName(string searchTerm)
    {
        return Collection.Query()
            .Where(x => x.SensorId.Contains(searchTerm))
            .ToList();
    }

    // Exists check
    public bool HasReadingsInRange(DateTime start, DateTime end)
    {
        return Collection.Exists(x => x.Timestamp >= start && x.Timestamp <= end);
    }

    // Delete multiple documents
    public int DeleteOldReadings(DateTime cutoffDate)
    {
        return Collection.DeleteMany(x => x.Timestamp < cutoffDate);
    }

    // Update multiple documents
    public int UpdateSensorName(string oldName, string newName)
    {
        var readings = Collection.Query()
            .Where(x => x.SensorId == oldName)
            .ToList();

        foreach (var reading in readings)
        {
            reading.SensorId = newName;
            Collection.Update(reading);
        }

        return readings.Count;
    }
}

Example 5: Using with AnIML Data (SiLA2.AnIML Integration)

This is the primary use case for the NoSQL module in SiLA2 applications.

using AnIMLCore;
using SiLA2.Database.NoSQL;
using SiLA2.AnIML.Services;
using LiteDB;

// Entity model for AnIML documents
public class AnIMLRepository : BaseRepository<AnIMLType>, IAnIMLRepository
{
    public AnIMLRepository(ILiteDatabase db) : base(db)
    {
        // AnIML documents are complex hierarchical structures
        // NoSQL is ideal for this use case
    }
}

// In a SiLA2 feature service
public class DataStorageFeatureService : DataStorageFeature.DataStorageFeatureBase
{
    private readonly IAnIMLRepository _anIMLRepository;
    private readonly ISeriesTypeBuilder _builder;
    private readonly ISeriesTypeProvider _provider;

    public DataStorageFeatureService(
        IAnIMLRepository anIMLRepository,
        ISeriesTypeBuilder builder,
        ISeriesTypeProvider provider)
    {
        _anIMLRepository = anIMLRepository;
        _builder = builder;
        _provider = provider;
    }

    public override Task<StoreData_Responses> StoreData(
        StoreData_Parameters request,
        ServerCallContext context)
    {
        // Build AnIML document from experiment data
        var animl = _builder.Build(new[] { request.SampleName.Value });
        animl.CreateAnIMLSeries(numberOfSeries: 2);

        // Add time series data
        var timestamps = request.Timestamps.Select(t => t.ToDateTime()).ToArray();
        var values = request.Values.Select(v => v.Value).ToArray();

        var seriesSet = animl.ExperimentStepSet.ExperimentStep[0].Result[0].SeriesSet;
        seriesSet.Series[0] = _provider.GetSeriesType(timestamps, DependencyType.independent);
        seriesSet.Series[1] = _provider.GetSeriesType(values, DependencyType.dependent);

        // Store in NoSQL database
        var created = _anIMLRepository.Create(animl, out object documentId);

        return Task.FromResult(new StoreData_Responses
        {
            DocumentId = new String { Value = documentId.ToString() }
        });
    }

    public override Task<RetrieveData_Responses> RetrieveData(
        RetrieveData_Parameters request,
        ServerCallContext context)
    {
        // Retrieve AnIML document from NoSQL database
        var animl = _anIMLRepository.FindById(new ObjectId(request.DocumentId.Value));

        if (animl == null)
            throw new RpcException(new Status(StatusCode.NotFound, "Document not found"));

        // Extract data from complex hierarchical structure
        var series = animl.GetAnIMLSeries(experimentStep: 0, result: 0, focusedSeries: 1);

        return Task.FromResult(new RetrieveData_Responses
        {
            Data = new String { Value = animl.GetAnIMLXml() }
        });
    }
}

Example 6: Complex Entity with Nested Objects

using LiteDB;

public class ExperimentConfiguration
{
    [BsonId]
    public ObjectId Id { get; set; }

    public string ConfigurationName { get; set; }

    public DateTime CreatedAt { get; set; }

    // Nested object
    public TemperatureSettings TemperatureSettings { get; set; }

    // List of complex objects
    public List<ProcessStep> ProcessSteps { get; set; }

    // Dictionary
    public Dictionary<string, string> Metadata { get; set; }
}

public class TemperatureSettings
{
    public double TargetTemperature { get; set; }
    public double Tolerance { get; set; }
    public string Unit { get; set; }
}

public class ProcessStep
{
    public int StepNumber { get; set; }
    public string Action { get; set; }
    public TimeSpan Duration { get; set; }
    public Dictionary<string, object> Parameters { get; set; }
}

// Repository with complex queries
public class ExperimentConfigurationRepository : BaseRepository<ExperimentConfiguration>
{
    public ExperimentConfigurationRepository(ILiteDatabase db) : base(db)
    {
        // Index on nested property
        Collection.EnsureIndex(x => x.TemperatureSettings.TargetTemperature);
    }

    // Query nested properties
    public IEnumerable<ExperimentConfiguration> GetByTargetTemperature(double temp)
    {
        return Collection.Query()
            .Where(x => x.TemperatureSettings.TargetTemperature == temp)
            .ToList();
    }

    // Query list elements
    public IEnumerable<ExperimentConfiguration> GetByStepAction(string action)
    {
        return Collection.Query()
            .Where("$.ProcessSteps[*].Action ANY = @0", action)
            .ToList();
    }
}

// Usage
var config = new ExperimentConfiguration
{
    ConfigurationName = "Standard Heating Protocol",
    CreatedAt = DateTime.UtcNow,
    TemperatureSettings = new TemperatureSettings
    {
        TargetTemperature = 80.0,
        Tolerance = 2.0,
        Unit = "Celsius"
    },
    ProcessSteps = new List<ProcessStep>
    {
        new ProcessStep
        {
            StepNumber = 1,
            Action = "Heat",
            Duration = TimeSpan.FromMinutes(30),
            Parameters = new Dictionary<string, object>
            {
                { "RampRate", 5.0 },
                { "HoldTime", 60 }
            }
        }
    },
    Metadata = new Dictionary<string, string>
    {
        { "Author", "Lab Technician" },
        { "Version", "1.0" }
    }
};

repository.Create(config);

Example 7: Identifier Strategies

LiteDB supports multiple identifier types. Choose the one that fits your use case:

using LiteDB;

// Option 1: ObjectId (MongoDB-style, recommended for new projects)
public class EntityWithObjectId
{
    [BsonId]
    public ObjectId Id { get; set; }  // Auto-generated, globally unique
}

// Option 2: Guid (Windows-friendly, globally unique)
public class EntityWithGuid
{
    [BsonId]
    public Guid Id { get; set; }
}

// Usage:
var entity = new EntityWithGuid { Id = Guid.NewGuid() };
repository.Create(entity);

// Option 3: Auto-increment integer (simple, sequential)
public class EntityWithAutoIncrement
{
    [BsonId(autoId: true)]
    public int Id { get; set; }  // Auto-incremented by LiteDB
}

// Option 4: String ID (custom identifiers)
public class EntityWithStringId
{
    [BsonId]
    public string Id { get; set; }
}

// Usage:
var entity = new EntityWithStringId { Id = "TEMP-2024-001" };
repository.Create(entity);

// Option 5: Composite key (custom logic)
public class EntityWithCompositeKey
{
    [BsonId]
    public string CompositeId => $"{SensorId}_{Timestamp:yyyyMMddHHmmss}";

    [BsonIgnore]
    public string SensorId { get; set; }

    [BsonIgnore]
    public DateTime Timestamp { get; set; }
}

Identifier Type Comparison:

Type	Auto-Generated	Globally Unique	Human-Readable	Best For
ObjectId	✅ Yes	✅ Yes	❌ No	General purpose, distributed systems
Guid	⚠️ Manual	✅ Yes	❌ No	Integration with .NET/Windows systems
int (auto)	✅ Yes	❌ No	✅ Yes	Single database, sequential IDs
string	❌ No	⚠️ Depends	✅ Yes	Custom business identifiers
Composite	❌ No	⚠️ Depends	✅ Yes	Natural keys, multi-part identifiers

Advanced Topics

Transactions

LiteDB supports ACID transactions for atomic operations across multiple collections:

public class OrderService
{
    private readonly ILiteDatabase _db;
    private readonly ILiteCollection<Order> _orders;
    private readonly ILiteCollection<Inventory> _inventory;

    public OrderService(ILiteDatabase db)
    {
        _db = db;
        _orders = db.GetCollection<Order>();
        _inventory = db.GetCollection<Inventory>();
    }

    public void PlaceOrder(Order order)
    {
        // Begin transaction
        _db.BeginTrans();

        try
        {
            // Insert order
            _orders.Insert(order);

            // Update inventory
            foreach (var item in order.Items)
            {
                var inventoryItem = _inventory.FindById(item.ProductId);
                if (inventoryItem.Quantity < item.Quantity)
                    throw new InvalidOperationException("Insufficient inventory");

                inventoryItem.Quantity -= item.Quantity;
                _inventory.Update(inventoryItem);
            }

            // Commit transaction
            _db.Commit();
        }
        catch
        {
            // Rollback on error
            _db.Rollback();
            throw;
        }
    }
}

Indexes for Performance

Create indexes on frequently-queried properties:

public class TemperatureReadingRepository : BaseRepository<TemperatureReading>
{
    public TemperatureReadingRepository(ILiteDatabase db) : base(db)
    {
        // Simple index on single property
        Collection.EnsureIndex(x => x.SensorId);
        Collection.EnsureIndex(x => x.Timestamp);

        // Unique index (enforce uniqueness)
        Collection.EnsureIndex(x => x.SensorId, unique: true);

        // Compound expression (index on calculated value)
        Collection.EnsureIndex("TemperatureRounded", "ROUND($.Temperature, 0)");
    }
}

Index Best Practices:

• Index properties used in Where() clauses
• Index properties used in OrderBy() clauses
• Don't over-index (affects write performance)
• Use unique indexes to enforce data integrity
• Indexes are automatically maintained by LiteDB

File Storage (GridFS-style)

LiteDB supports file storage for large binary data:

public class FileStorageService
{
    private readonly ILiteDatabase _db;

    public FileStorageService(ILiteDatabase db)
    {
        _db = db;
    }

    // Store file
    public string StoreFile(string filename, Stream stream)
    {
        var fileInfo = _db.FileStorage.Upload(filename, filename, stream);
        return fileInfo.Id;
    }

    // Retrieve file
    public Stream RetrieveFile(string fileId)
    {
        var memoryStream = new MemoryStream();
        _db.FileStorage.Download(fileId, memoryStream);
        memoryStream.Position = 0;
        return memoryStream;
    }

    // Delete file
    public bool DeleteFile(string fileId)
    {
        return _db.FileStorage.Delete(fileId);
    }

    // List all files
    public IEnumerable<LiteFileInfo> ListFiles()
    {
        return _db.FileStorage.FindAll();
    }
}

File Storage Use Cases:

• Storing experimental images (PNG, JPEG)
• Archiving raw instrument output files
• Storing PDF reports
• Caching large binary data

Thread Safety

LiteDB provides thread-safe operations through connection-level locking:

// Singleton database instance (thread-safe)
services.AddSingleton<ILiteDatabase>(sp =>
    new LiteDatabase("myapp.db"));

// Concurrent reads and writes are automatically synchronized
// LiteDB uses a single-writer, multiple-reader lock strategy

Thread Safety Guarantees:

• ✅ Multiple threads can read simultaneously
• ✅ Single thread can write (others block)
• ✅ ACID transactions are atomic
• ⚠️ Long-running queries can block writers

Performance Tips for Multi-Threaded Scenarios:

// Use connection pooling for high concurrency (not built-in)
public class LiteDatabasePool
{
    private readonly ConcurrentBag<ILiteDatabase> _pool = new();
    private readonly string _connectionString;

    public LiteDatabasePool(string connectionString)
    {
        _connectionString = connectionString;
    }

    public ILiteDatabase GetDatabase()
    {
        if (_pool.TryTake(out var db))
            return db;

        return new LiteDatabase(_connectionString);
    }

    public void ReturnDatabase(ILiteDatabase db)
    {
        _pool.Add(db);
    }
}

Database Backup and Maintenance

public class DatabaseMaintenanceService
{
    private readonly ILiteDatabase _db;

    public DatabaseMaintenanceService(ILiteDatabase db)
    {
        _db = db;
    }

    // Backup database to file
    public void BackupDatabase(string backupPath)
    {
        using var backup = new LiteDatabase($"filename={backupPath}");

        foreach (var collectionName in _db.GetCollectionNames())
        {
            var sourceCollection = _db.GetCollection(collectionName);
            var targetCollection = backup.GetCollection(collectionName);

            foreach (var doc in sourceCollection.FindAll())
            {
                targetCollection.Insert(doc);
            }
        }
    }

    // Shrink database (reclaim space from deleted documents)
    public long ShrinkDatabase()
    {
        return _db.Rebuild();
    }

    // Check database integrity
    public bool CheckIntegrity()
    {
        try
        {
            return _db.CheckIntegrity();
        }
        catch
        {
            return false;
        }
    }

    // Export collection to JSON
    public string ExportToJson<T>(ILiteCollection<T> collection)
    {
        var documents = collection.FindAll();
        return JsonSerializer.Serialize(documents);
    }
}

Custom BsonMapper Configuration

Customize how objects are serialized to BSON:

var mapper = BsonMapper.Global;

// Map enum to string instead of int
mapper.EnumAsInteger = false;

// Custom serialization for specific type
mapper.RegisterType<MyCustomType>(
    serialize: (obj) => new BsonDocument { ["Value"] = obj.ToString() },
    deserialize: (bson) => MyCustomType.Parse(bson["Value"].AsString)
);

// Ignore specific properties globally
mapper.Entity<TemperatureReading>()
    .Ignore(x => x.InternalCache);

// Custom field mapping
mapper.Entity<TemperatureReading>()
    .DbRef(x => x.Sensor, "sensors");  // Foreign key reference

var db = new LiteDatabase("myapp.db", mapper);

API Reference Summary

IBaseRepository<T>

Standard CRUD interface for all repositories.

public interface IBaseRepository<T>
{
    /// Creates a new document and returns the generated ID
    T Create(T data, out object id);

    /// Creates a new document without returning the ID
    T Create(T data);

    /// Retrieves all documents in the collection
    IEnumerable<T> All();

    /// Finds a document by its unique identifier
    T FindById(object id);

    /// Updates an existing document or inserts if not found (upsert)
    void Update(T entity);

    /// Deletes a document by ID, returns true if deleted
    bool Delete(object id);
}

BaseRepository<T>

Abstract base class providing LiteDB implementation.

public abstract class BaseRepository<T> : IBaseRepository<T>
{
    /// Access to the underlying LiteDB database
    public ILiteDatabase DB { get; }

    /// Direct access to the collection for advanced queries
    public ILiteCollection<T> Collection { get; }

    /// Constructor requiring ILiteDatabase instance
    protected BaseRepository(ILiteDatabase db);

    // All IBaseRepository<T> methods implemented as virtual
    // (can be overridden in derived classes)
}

DocumentDbServiceCollectionExtensions

Dependency injection extension methods.

public static class DocumentDbServiceCollectionExtensions
{
    /// Registers ILiteDatabase as singleton
    /// path: File path or connection string
    /// db: Output parameter receiving the created database instance
    void RegisterDocumentDatabase(
        this IServiceCollection services,
        string path,
        out ILiteDatabase db);

    /// Registers ILiteCollection<T> as singleton for specific entity type
    void RegisterDocumentDatabaseTypes<T>(
        this IServiceCollection services,
        ILiteDatabase db);
}

Best Practices

Entity Design Recommendations

1. Always Use [BsonId] Attribute

public class MyEntity
{
    [BsonId]  // Explicitly mark the ID property
    public ObjectId Id { get; set; }
}

2. Use Meaningful Property Names

// Good
public class TemperatureReading
{
    public DateTime MeasurementTime { get; set; }
    public double TemperatureCelsius { get; set; }
}

// Avoid
public class TemperatureReading
{
    public DateTime dt { get; set; }  // Unclear abbreviation
    public double temp { get; set; }  // Missing unit
}

3. Avoid Deep Nesting (>3 levels)

// Acceptable
public class Experiment
{
    public ExperimentSettings Settings { get; set; }  // Level 1
}

public class ExperimentSettings
{
    public TemperatureSettings Temperature { get; set; }  // Level 2
}

public class TemperatureSettings
{
    public double Target { get; set; }  // Level 3
}

// Too deep - consider flattening or splitting into separate collections
public class Experiment
{
    public Level1 { Level2 { Level3 { Level4 { Level5 } } } }  // Hard to query
}

4. Use [BsonIgnore] for Computed Properties

public class TemperatureReading
{
    public double TemperatureCelsius { get; set; }

    [BsonIgnore]  // Don't persist this
    public double TemperatureFahrenheit => TemperatureCelsius * 9/5 + 32;
}

Identifier Strategy

Choose the right identifier type for your use case:

// Use ObjectId for general purpose (recommended)
[BsonId]
public ObjectId Id { get; set; }

// Use Guid for integration with existing .NET systems
[BsonId]
public Guid Id { get; set; } = Guid.NewGuid();

// Use auto-increment int for simple sequential IDs
[BsonId(autoId: true)]
public int Id { get; set; }

// Use string for business identifiers
[BsonId]
public string Id { get; set; } = $"EXP-{DateTime.Now:yyyyMMdd}-{Guid.NewGuid():N}";

Performance Tips

1. Create Indexes on Frequently-Queried Properties

Collection.EnsureIndex(x => x.Timestamp);
Collection.EnsureIndex(x => x.SensorId);

2. Use Pagination for Large Result Sets

// Bad - loads all documents into memory
var allReadings = Collection.FindAll().ToList();

// Good - loads only needed page
var page = Collection.Query()
    .OrderByDescending(x => x.Timestamp)
    .Skip(page * pageSize)
    .Limit(pageSize)
    .ToList();

3. Use Projection to Load Only Required Fields

// Bad - loads entire document
var readings = Collection.FindAll();

// Good - loads only needed properties
var summaries = Collection.Query()
    .Select(x => new { x.Id, x.Timestamp, x.Temperature })
    .ToList();

4. Batch Operations When Possible

// Bad - multiple individual inserts
foreach (var reading in readings)
{
    Collection.Insert(reading);
}

// Good - single batch insert
Collection.InsertBulk(readings);

Connection String Best Practices

Store connection strings in configuration:

// appsettings.json
{
  "ConnectionStrings": {
    "AnIMLDocumentDatabase": "filename=Data/animl.db",
    "ConfigDatabase": "filename=Data/config.db;password=secret"
  }
}

// In Program.cs
builder.Services.RegisterDocumentDatabase(
    builder.Configuration.GetConnectionString("AnIMLDocumentDatabase"),
    out var db);

Use encrypted databases for sensitive data:

"filename=sensitive_data.db;password=YourStrongPasswordHere"

Error Handling

Handle common LiteDB exceptions:

try
{
    var entity = repository.FindById(id);
}
catch (LiteException ex) when (ex.ErrorCode == LiteException.LOCK_TIMEOUT)
{
    // Database is locked by another thread
    _logger.LogWarning("Database lock timeout, retrying...");
    // Implement retry logic
}
catch (LiteException ex) when (ex.ErrorCode == LiteException.INDEX_DUPLICATE_KEY)
{
    // Unique constraint violation
    _logger.LogError("Duplicate key error: {Message}", ex.Message);
    throw new InvalidOperationException("Entity with this ID already exists", ex);
}
catch (LiteException ex)
{
    // General LiteDB error
    _logger.LogError(ex, "Database error");
    throw;
}

Repository Method Naming Conventions

Follow consistent naming patterns:

// Query methods - start with "Get" or "Find"
IEnumerable<T> GetReadingsBySensor(string sensorId);
T FindByCustomId(string customId);

// Check methods - start with "Has" or "Exists" or "Is"
bool HasReadingsInRange(DateTime start, DateTime end);
bool ExistsBySensorId(string sensorId);

// Count methods - start with "Count"
int CountReadingsToday();

// Delete methods - start with "Delete" or "Remove"
int DeleteOldReadings(DateTime cutoffDate);
bool RemoveByCustomId(string customId);

Comparison with SiLA2.Database.SQL

Aspect	SiLA2.Database.NoSQL	SiLA2.Database.SQL
Database Engine	LiteDB (embedded)	SQLite/SQL Server (embedded/server)
Data Model	Document-oriented (BSON)	Relational (tables)
Schema	Schema-less, flexible	Fixed schema, requires migrations
Query Language	LINQ-like API	Entity Framework LINQ or SQL
Setup Complexity	Low (single file)	Medium (EF Core + migrations)
Best For	AnIML data, configs, logs	User management, relational data
Performance	Good for documents	Excellent for joins/aggregations
File Size	Compact BSON	Larger (SQL overhead)
Transactions	ACID within single DB	ACID across multiple tables
Scalability	Up to ~100GB	Up to multi-TB (SQL Server)
Type Safety	Runtime (BSON serialization)	Compile-time (EF Core)
Deployment	Single DLL + data file	Multiple assemblies + migrations

When to Use Both:

Many SiLA2 servers use both modules for different purposes:

// SQL for user authentication (relational data)
builder.Services.AddDbContext<AuthenticationDbContext>(options =>
    options.UseSqlite("Data Source=users.db"));

// NoSQL for AnIML experimental data (document-oriented)
builder.Services.RegisterDocumentDatabase("Data/animl.db", out var docDb);
builder.Services.AddSingleton<IAnIMLRepository, AnIMLRepository>();

Related Packages

• SiLA2.Core - Core SiLA2 server implementation, domain models, network discovery
• SiLA2.AnIML - AnIML scientific data format support (uses this package for persistence)
• SiLA2.Database.SQL - SQL database module with Entity Framework Core
• SiLA2.AspNetCore - ASP.NET Core integration for SiLA2 servers
• LiteDB - Underlying embedded NoSQL database engine

Contributing & Development

This package is part of the sila_csharp project.

Building from Source

git clone --recurse-submodules https://gitlab.com/SiLA2/sila_csharp.git
cd sila_csharp/src
dotnet build SiLA2.Database.NoSQL/SiLA2.Database.NoSQL.csproj

Running Tests

# Run all tests
dotnet test Tests/SiLA2.Database.NoSQL.Tests/SiLA2.Database.NoSQL.Tests.csproj

# Run integration tests
dotnet test Tests/SiLA2.IntegrationTests.Server.Tests/SiLA2.IntegrationTests.Server.Tests.csproj --filter "FullyQualifiedName~NoSQL"

Project Structure

SiLA2.Database.NoSQL/
├── IBaseRepository.cs                      # Generic repository interface
├── BaseRepository.cs                       # LiteDB-backed implementation
├── DocumentDbServiceCollectionExtensions.cs # DI extension methods
├── SiLA2.Database.NoSQL.csproj             # Project file
└── README.md                               # This file

Links & Resources

• SiLA2 Standard: https://sila-standard.com
• LiteDB Documentation: https://www.litedb.org
• Repository: https://gitlab.com/SiLA2/sila_csharp
• NuGet Package: https://www.nuget.org/packages/SiLA2.Database.NoSQL
• Documentation Wiki: https://gitlab.com/SiLA2/sila_csharp/-/wikis/home
• Issue Tracker: https://gitlab.com/SiLA2/sila_csharp/-/issues
• SiLA Slack: Join the community

License

This project is licensed under the MIT License.

Maintainer

Christoph Pohl (@Chamundi)

Security

For security vulnerabilities, please refer to the SiLA2 Vulnerability Policy.

---

Questions or Issues?

• Open an issue on GitLab
• Join the SiLA community on Slack
• Check the Wiki for additional documentation