Xrm.Persistent.Collections 2.2026.9.7

Xrm.Persistent.Collections

SQLite-backed persistent dictionary storage for Microsoft Dynamics CRM/XRM applications that survives process restarts and provides disk-based caching for long-running operations.

Built on top of SQLite with automatic JSON serialization of Dynamics 365 entities using Xrm.Json.Serialization, which now supports AliasedValue (FetchXML linked entities), OptionSetValueCollection (multi-select picklists), and BooleanManagedProperty in addition to all standard CRM data types.

🚀 Features

• ✅ Persistent Storage: Data survives application restarts
• ✅ Type-Safe: Generic dictionary implementation LocalDictionary<T>
• ✅ CRM Native: Full support for all Dynamics 365 data types via Xrm.Json.Serialization v1.2026.3.1
  • Entity, EntityReference, EntityCollection
  • OptionSetValue, Money, DateTime, Guid
  • NEW: AliasedValue (FetchXML linked entities)
  • NEW: OptionSetValueCollection (multi-select picklists)
  • NEW: BooleanManagedProperty
• ✅ High Performance: SQLite with WAL mode for concurrent access (10-15% faster than v1.x)
• ✅ Simple API: Standard IDictionary<string, T> interface
• ✅ Cache Introspection: GetAll() and GetAllKeys() methods for querying all cached items
• ✅ Thread-Safe: Built-in synchronization for multi-threaded scenarios
• ✅ Expiration Support: Automatic cleanup of expired items with configurable TTL
• ✅ .NET Framework 4.8: Latest framework with TLS 1.2/1.3 support

📦 Installation

Install-Package Xrm.Persistent.Collections

Requirements

• .NET Framework 4.8
• Microsoft.CrmSdk.CoreAssemblies 9.0.2.60+
• Dynamics 365 Online or OnPrem 9.1+

📖 Quick Start

using Xrm.Persistent.Collections;
using Microsoft.Xrm.Sdk;

// Create a persistent dictionary backed by SQLite
using (var dict = new LocalDictionary<Entity>("data.db"))
{
    // Store an entity
    var account = new Entity("account", Guid.NewGuid());
    account["name"] = "Contoso";
    account["revenue"] = new Money(1000000);

    dict["account1"] = account;

    // Retrieve it later (even after application restart!)
    var retrieved = dict["account1"];
    Console.WriteLine(retrieved["name"]); // Output: Contoso
}

💡 Use Cases & Scenarios

1️⃣ Long-Running Job Engines

Store job state, checkpoints, and progress to survive crashes or restarts:

using (var jobState = new LocalDictionary<Entity>("jobs.db"))
{
    foreach (var entity in entities)
    {
        // Process entity
        ProcessEntity(entity);

        // Save checkpoint - resume from here if job crashes
        jobState["lastProcessed"] = entity;
    }
}

Why this is useful:

• Job crashes don't mean starting from scratch
• Resume processing from exact checkpoint
• Track progress across multiple runs
• Perfect for bulk data migration, ETL processes

2️⃣ Offline-First Applications

Cache Dynamics 365 data locally for offline access:

using (var cache = new LocalDictionary<Entity>("offline-cache.db"))
{
    // Online: Fetch and cache data
    var accounts = service.RetrieveMultiple(query);
    foreach (var account in accounts.Entities)
    {
        cache[account.Id.ToString()] = account;
    }

    // Offline: Read from cache
    var cachedAccount = cache[accountId.ToString()];
    DisplayAccountDetails(cachedAccount);
}

Why this is useful:

• Work without internet connectivity
• Reduce API calls to Dynamics 365 (avoid throttling)
• Faster data access (local disk vs. network)
• Ideal for field service scenarios

3️⃣ Incremental Sync & Change Tracking

Track what's been synchronized to avoid re-processing:

using (var syncState = new LocalDictionary<DateTime>("sync-state.db"))
{
    var lastSync = syncState.ContainsKey("lastSyncDate") 
        ? syncState["lastSyncDate"] 
        : DateTime.MinValue;

    // Fetch only changed records since last sync
    var query = $@"<fetch>
        <entity name='account'>
            <filter>
                <condition attribute='modifiedon' operator='gt' value='{lastSync:yyyy-MM-dd}' />
            </filter>
        </entity>
    </fetch>";

    var changes = service.RetrieveMultiple(new FetchExpression(query));
    ProcessChanges(changes);

    syncState["lastSyncDate"] = DateTime.UtcNow;
}

Why this is useful:

• Efficient delta syncs
• Avoid processing unchanged data
• Reduce API load and improve performance
• Perfect for integration scenarios

4️⃣ Complex Entity Caching with Linked Entities (FetchXML)

Cache FetchXML query results with related entities using AliasedValue support:

using (var cache = new LocalDictionary<Entity>("fetchxml-cache.db"))
{
    // FetchXML query with linked entities
    var fetchXml = @"<fetch>
        <entity name='account'>
            <attribute name='name' />
            <link-entity name='contact' from='parentcustomerid' to='accountid' alias='primarycontact'>
                <attribute name='fullname' />
                <attribute name='emailaddress1' />
            </link-entity>
        </entity>
    </fetch>";

    var results = service.RetrieveMultiple(new FetchExpression(fetchXml));

    // Cache entities with linked data (AliasedValue preserved!)
    foreach (var entity in results.Entities)
    {
        cache[entity.Id.ToString()] = entity;
        // Entity includes "primarycontact.fullname" as AliasedValue
    }

    // Later: Retrieve with all linked data intact
    var cachedEntity = cache[accountId.ToString()];
    var contactName = cachedEntity.GetAliasedValue<string>("primarycontact.fullname");
}

Why this is useful:

• Preserve complex FetchXML query results
• Avoid expensive re-queries with joins
• Cache reports and dashboards data
• Xrm.Json.Serialization v1.2026.3+ handles AliasedValue automatically!

5️⃣ Multi-Select Picklist (OptionSetValueCollection) Support

Store entities with multi-select picklists:

using (var dict = new LocalDictionary<Entity>("multiselect.db"))
{
    var account = new Entity("account", Guid.NewGuid());
    account["name"] = "Contoso";

    // Multi-select picklist (new in Dynamics 365)
    account["industry_categories"] = new OptionSetValueCollection(new[] { 
        new OptionSetValue(1), // Manufacturing
        new OptionSetValue(3), // Technology
        new OptionSetValue(5)  // Services
    });

    dict["account1"] = account;

    // Retrieve and read multi-select values
    var retrieved = dict["account1"];
    var categories = (OptionSetValueCollection)retrieved["industry_categories"];
    Console.WriteLine($"Categories: {string.Join(", ", categories.Select(o => o.Value))}");
}

Why this is useful:

• Full support for modern Dynamics 365 multi-select fields
• Previously required custom serialization logic
• Xrm.Json.Serialization v1.2026.3+ handles this automatically!

6️⃣ Session State Persistence

Store user session data that persists across application restarts:

using (var session = new LocalDictionary<Dictionary<string, object>>("session.db"))
{
    // Store session state
    session["user123"] = new Dictionary<string, object>
    {
        { "lastActivity", DateTime.UtcNow },
        { "viewedRecords", new List<Guid> { id1, id2, id3 } },
        { "preferences", new { theme = "dark", pageSize = 50 } }
    };

    // Later (even after restart): Restore session
    var userData = session["user123"];
}

Why this is useful:

• Preserve user context across sessions
• Better user experience
• Useful for desktop applications or Windows Services

7️⃣ Error Recovery & Replay

Store failed operations for retry logic:

using (var errorQueue = new LocalDictionary<Entity>("failed-ops.db"))
{
    try
    {
        service.Update(entity);
    }
    catch (Exception ex)
    {
        // Store for later retry
        errorQueue[entity.Id.ToString()] = entity;
        LogError(ex);
    }

    // Retry logic (scheduled job or manual trigger)
    foreach (var key in errorQueue.Keys.ToList())
    {
        try
        {
            var entity = errorQueue[key];
            service.Update(entity);
            errorQueue.Remove(key); // Success - remove from queue
        }
        catch { /* Will retry next time */ }
    }
}

Why this is useful:

• Guaranteed operation retry
• Durable queue for failed operations
• No data loss during transient errors

8️⃣ Batch Processing with State Management

Process large datasets in batches with persistent state:

using (var batchState = new LocalDictionary<int>("batch-progress.db"))
{
    const int batchSize = 500;
    int currentBatch = batchState.ContainsKey("currentBatch") ? batchState["currentBatch"] : 0;

    while (true)
    {
        var entities = FetchBatch(currentBatch, batchSize);
        if (!entities.Any()) break;

        ProcessBatch(entities);

        // Save progress after each batch
        batchState["currentBatch"] = ++currentBatch;
    }
}

Why this is useful:

• Process millions of records safely
• Survive crashes without losing progress
• Throttle-aware processing (Dynamics 365 API limits)

9️⃣ Cache Introspection & Monitoring

Query all cached items without knowing keys in advance:

using (var cache = new LocalDictionary<Entity>("monitoring.db"))
{
    // Get all cached items
    var allItems = await cache.GetAll();
    Console.WriteLine($"Total cached items: {allItems.Count()}");

    // Get all keys with type information
    var allKeys = await cache.GetAllKeys();
    foreach (var keyInfo in allKeys)
    {
        Console.WriteLine($"Key: {keyInfo.Key}, Type: {keyInfo.Type?.Name}");
    }

    // Use in reporting or diagnostics
    var reportData = new Dictionary<string, object>
    {
        { "totalCached", allItems.Count() },
        { "cacheSize", allItems.Sum(item => item.Length) / 1024.0, " KB" },
        { "keyCount", allKeys.Count() },
        { "lastUpdated", DateTime.UtcNow }
    };
}

Why this is useful:

• Monitor cache health and size
• Audit what's been cached
• Generate cache statistics and reports
• Implement cache warming strategies
• Debug what's actually in the cache

🔧 Advanced Features

Thread-Safe Operations

Built-in synchronization allows safe multi-threaded access:

using (var dict = new LocalDictionary<Entity>("shared.db"))
{
    Parallel.ForEach(entities, entity =>
    {
        dict[entity.Id.ToString()] = entity; // Thread-safe
    });
}

Enumeration Support

Standard dictionary operations work as expected:

using (var dict = new LocalDictionary<Entity>("data.db"))
{
    // Count
    Console.WriteLine($"Total items: {dict.Count}");

    // Keys
    foreach (var key in dict.Keys)
    {
        Console.WriteLine(key);
    }

    // Values
    foreach (var entity in dict.Values)
    {
        Console.WriteLine(entity.LogicalName);
    }

    // Key-Value pairs
    foreach (var kvp in dict)
    {
        Console.WriteLine($"{kvp.Key}: {kvp.Value["name"]}");
    }
}

🎯 When to Use This Library

📚 Dependencies & Compatibility

Xrm.Json.Serialization v1.2026.3.1

This library uses the latest version of Xrm.Json.Serialization with major enhancements:

New Data Type Support

• AliasedValue: FetchXML queries with linked entities are now fully supported
• OptionSetValueCollection: Multi-select picklists work seamlessly
• BooleanManagedProperty: Managed properties serialize correctly

Compact JSON Format

Entities are serialized in a compact, readable format:

{
  "_reference": "account:12345678-1234-1234-1234-123456789012",
  "name": "Contoso Ltd",
  "revenue": { "_money": 1000000 },
  "industrycode": { "_option": 1 },
  "parentaccountid": { "_reference": "account:87654321-4321-4321-4321-210987654321" },
  "createdon": "2024-01-15T10:30:00Z",
  "contact.fullname": { "_aliased": "contact|fullname|John Doe" },
  "categories": { "_options": [1, 2, 3] }
}

Runtime Requirements

• .NET Framework 4.8
• Dynamics 365 Online (all versions)
• Dynamics 365 OnPrem 9.1+
• Dynamics CRM 2016+

Key Dependencies

🎓 API Reference

Constructor

var dict = new LocalDictionary<T>(string databasePath)

IDictionary<string, T> Implementation

// Add/Update
dict["key"] = value;
dict.Add("key", value);

// Retrieve
var value = dict["key"];
bool found = dict.TryGetValue("key", out var value);

// Remove
dict.Remove("key");

// Check existence
bool exists = dict.ContainsKey("key");

// Enumerate
int count = dict.Count;
ICollection<string> keys = dict.Keys;
ICollection<T> values = dict.Values;

// Iterate
foreach (var kvp in dict)
{
    Console.WriteLine($"{kvp.Key}: {kvp.Value}");
}

// Cleanup
dict.Clear();
dict.Dispose();

Cache Introspection Methods

// Get all non-expired items (raw byte arrays)
var allItems = await cache.GetAll();
var count = allItems.Count();

// Get all non-expired keys with type metadata
var allKeys = await cache.GetAllKeys();
foreach (var keyInfo in allKeys)
{
    Console.WriteLine($"Key: {keyInfo.Key}, Type: {keyInfo.Type?.Name}");
}

🛠️ Best Practices

1. Always Dispose

// Use 'using' statement to ensure proper cleanup
using (var dict = new LocalDictionary<Entity>("data.db"))
{
    // Your code here
} // Automatically disposed

2. Choose Meaningful Database Names

// Good - descriptive names
var jobQueue = new LocalDictionary<Entity>("job-queue.db");
var syncState = new LocalDictionary<DateTime>("sync-checkpoints.db");

// Avoid - generic names
var dict = new LocalDictionary<Entity>("data.db");

3. Handle Large Datasets Efficiently

// Process in batches instead of loading all values at once
using (var dict = new LocalDictionary<Entity>("large-dataset.db"))
{
    foreach (var key in dict.Keys.Take(100))
    {
        var entity = dict[key];
        ProcessEntity(entity);
    }
}

4. Use Separate Databases for Different Concerns

// Separate concerns = easier maintenance
var userCache = new LocalDictionary<Entity>("user-cache.db");
var jobQueue = new LocalDictionary<Entity>("job-queue.db");
var errorLog = new LocalDictionary<Entity>("errors.db");

📊 Performance Characteristics

• Read operations: ~0.5-2ms per item (depends on entity size)
• Write operations: ~1-5ms per item (WAL mode optimized)
• Enumeration: ~100-500ms for 1,000 items
• Storage overhead: ~15-25% JSON + SQLite indexes
• Concurrent reads: Excellent (WAL mode)
• Concurrent writes: Serialized (SQLite limitation)

Performance Tips

• Batch writes when possible
• Avoid enumerating Values for large datasets
• Use ContainsKey() instead of TryGetValue() when you only need existence check
• Keep entity sizes reasonable (<1 MB per entity)

🔄 Migration from v1.x

If upgrading from the old Innofactor.Xrm.Persistent.Collections:

// OLD (v1.x)
using Innofactor.Xrm.Persistent.Collections;

// NEW (v2.x)
using Xrm.Persistent.Collections;

That's it! Your existing .db files work without any changes. See CHANGELOG.md for full migration guide.

📘 Documentation

• CHANGELOG.md - Version history and migration guide
• UPGRADE_SUMMARY.md - Detailed upgrade information
• KNOWN_ISSUES_AND_ROADMAP.md - Future improvements
• QUICK_REFERENCE.md - Integration checklist

🤝 Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👥 Authors

• Alexey Shytikov - Original Akavache inspiration
• Jonas Rapp - Original Innofactor implementation
• Imran Akram - Current maintainer (v2.x)

🔗 Related Projects

• Xrm.Json.Serialization - Compact JSON serialization for Dynamics 365 entities (dependency)
• Akavache - Original inspiration for persistent caching

🐛 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• NuGet: NuGet Package

Version: 2.0.0+ | Framework: .NET Framework 4.8 | License: MIT | Tests: 43 passing