Mab.PipelineFramework.FileStorage.S3 1.1.0

PipelineFramework.FileStorage.S3

S3-compatible storage implementation for PipelineFramework.FileStorage. Supports AWS S3, Cloudflare R2, MinIO, and other S3-compatible services using the MinIO .NET SDK.

Installation

dotnet add package Mab.PipelineFramework.FileStorage.S3

Features

• Full IFileStore implementation for S3-compatible storage
• Supports AWS S3, Cloudflare R2, MinIO, and other S3-compatible services
• Built on the MinIO .NET SDK for broad S3 compatibility
• Fluent builder API for easy configuration
• Dependency injection extensions for ASP.NET Core
• Multi-tenant support with flexible tenant isolation modes
• Factory pattern for creating scoped file stores per bucket/tenant
• SSL/TLS support
• Custom endpoint configuration
• Base path/prefix support for organizing files within a bucket

Quick Start

ASP.NET Core Dependency Injection (Recommended)

var builder = WebApplication.CreateBuilder(args);

// Option 1: Simple configuration
builder.Services.AddS3FileStorage(options =>
{
    options.Endpoint = "s3.amazonaws.com";
    options.BucketName = "my-bucket";
    options.AccessKey = "your-access-key";
    options.SecretKey = "your-secret-key";
});

// Option 2: Fluent builder API
builder.Services.AddS3FileStorage(s3 => s3
    .WithEndpoint("s3.amazonaws.com")
    .WithCredentials("your-access-key", "your-secret-key")
    .WithBucket("my-bucket")
    .WithRegion("us-east-1")
    .WithSSL());

// Option 3: Custom MinIO client configuration
builder.Services.AddS3FileStorage(
    client => client
        .WithEndpoint("play.min.io")
        .WithCredentials("minioadmin", "minioadmin")
        .WithSSL(),
    options =>
    {
        options.BucketName = "my-bucket";
    });

var app = builder.Build();

Then inject IFileStore or IS3FileStoreFactory in your services:

public class MyService
{
    private readonly IFileStore _fileStore;

    public MyService(IFileStore fileStore)
    {
        _fileStore = fileStore;
    }

    public async Task UploadAsync(Stream content, string path)
    {
        await _fileStore.CreateFileFromStreamAsync(path, content);
    }
}

Multi-Tenant Applications (Recommended: Automatic Scoping)

The recommended approach for SaaS applications is automatic tenant scoping - the tenant is resolved automatically from your ITenantContextProvider, so developers can't forget to scope operations:

// 1. Implement ITenantContextProvider to resolve tenant from your context
public class HttpHeaderTenantProvider : ITenantContextProvider
{
    private readonly IHttpContextAccessor _httpContextAccessor;

    public HttpHeaderTenantProvider(IHttpContextAccessor httpContextAccessor)
    {
        _httpContextAccessor = httpContextAccessor;
    }

    public string? GetCurrentTenantId()
    {
        return _httpContextAccessor.HttpContext?.Request.Headers["X-Tenant-Id"].FirstOrDefault();
    }
}

// 2. Single call registers everything - tenant provider is REQUIRED via generic parameter!
builder.Services.AddHttpContextAccessor();
builder.Services.AddS3FileStorageWithTenantScope<HttpHeaderTenantProvider>(options =>
{
    options.Endpoint = "s3.amazonaws.com";
    options.BucketName = "shared-bucket";
    options.AccessKey = "your-access-key";
    options.SecretKey = "your-secret-key";
}, TenantIsolationMode.PathPrefix);

The generic parameter <HttpHeaderTenantProvider> is required - the code won't compile without it, so developers can't forget to provide the tenant resolver!

Now your services just inject IFileStore - tenant isolation is automatic:

// Clean, simple service - no manual tenant resolution needed!
public class DocumentService
{
    private readonly IFileStore _fileStore;

    public DocumentService(IFileStore fileStore)
    {
        _fileStore = fileStore;
    }

    public async Task UploadAsync(Stream content, string path)
    {
        // Automatically stored at: shared-bucket/{tenant-id}/path
        await _fileStore.CreateFileFromStreamAsync(path, content);
    }

    public async Task<Stream> DownloadAsync(string path)
    {
        // Automatically reads from: shared-bucket/{tenant-id}/path
        return await _fileStore.GetFileStreamAsync(path);
    }
}

Multi-Tenant Applications (Manual Factory Pattern)

If you need more control (e.g., cross-tenant operations for admin), use the factory pattern:

// Register with multi-tenant support (factory pattern)
builder.Services.AddS3FileStorageMultiTenant(
    options =>
    {
        options.Endpoint = "s3.amazonaws.com";
        options.BucketName = "shared-bucket";
        options.AccessKey = "your-access-key";
        options.SecretKey = "your-secret-key";
    },
    TenantIsolationMode.PathPrefix);

Then use the factory for explicit tenant control:

public class AdminFileService
{
    private readonly IS3FileStoreFactory _factory;

    public AdminFileService(IS3FileStoreFactory factory)
    {
        _factory = factory;
    }

    public async Task CopyBetweenTenantsAsync(
        string sourceTenant, string destTenant, string path)
    {
        var sourceStore = _factory.ForTenant(sourceTenant);
        var destStore = _factory.ForTenant(destTenant);

        using var stream = await sourceStore.GetFileStreamAsync(path);
        await destStore.CreateFileFromStreamAsync(path, stream);
    }

    public async Task UploadToSpecificBucketAsync(string bucketName, Stream content, string path)
    {
        var store = _factory.ForBucket(bucketName);
        await store.CreateFileFromStreamAsync(path, content);
    }
}

Tenant Isolation Modes

Multiple S3 Configurations

Register multiple S3 configurations with keyed services:

builder.Services.AddKeyedS3FileStorage("primary", options =>
{
    options.Endpoint = "s3.amazonaws.com";
    options.BucketName = "primary-bucket";
    options.AccessKey = "key1";
    options.SecretKey = "secret1";
});

builder.Services.AddKeyedS3FileStorage("archive", options =>
{
    options.Endpoint = "s3.amazonaws.com";
    options.BucketName = "archive-bucket";
    options.AccessKey = "key2";
    options.SecretKey = "secret2";
});

Inject with [FromKeyedServices]:

public class ArchiveService(
    [FromKeyedServices("primary")] IFileStore primaryStore,
    [FromKeyedServices("archive")] IFileStore archiveStore)
{
    public async Task ArchiveAsync(string path)
    {
        using var stream = await primaryStore.GetFileStreamAsync(path);
        await archiveStore.CreateFileFromStreamAsync($"archive/{path}", stream);
        await primaryStore.TryDeleteFileAsync(path);
    }
}

Manual Instantiation

using PipelineFramework.FileStorage.S3;

// Using the builder
var fileStore = new S3FileStoreBuilder()
    .WithEndpoint("s3.amazonaws.com")
    .WithCredentials("your-access-key", "your-secret-key")
    .WithBucket("my-bucket")
    .WithRegion("us-east-1")
    .WithSSL()
    .Build();

// Or using the factory method
var options = new S3StorageOptions
{
    Endpoint = "s3.amazonaws.com",
    BucketName = "my-bucket",
    AccessKey = "your-access-key",
    SecretKey = "your-secret-key",
    UseSSL = true,
    Region = "us-east-1"
};

var fileStore = S3FileStore.Create(new SystemClock(), options);

Configuration Examples

AWS S3

builder.Services.AddS3FileStorage(s3 => s3
    .WithEndpoint("s3.amazonaws.com")
    .WithCredentials("AKIAIOSFODNN7EXAMPLE", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY")
    .WithBucket("my-bucket")
    .WithRegion("us-east-1")
    .WithSSL());

Cloudflare R2

builder.Services.AddS3FileStorage(s3 => s3
    .WithEndpoint("{account-id}.r2.cloudflarestorage.com")
    .WithCredentials("your-r2-access-key", "your-r2-secret-key")
    .WithBucket("my-bucket")
    .WithRegion("auto")
    .WithSSL());

MinIO

builder.Services.AddS3FileStorage(s3 => s3
    .WithEndpoint("minio.example.com:9000")
    .WithCredentials("minioadmin", "minioadmin")
    .WithBucket("my-bucket")
    .WithSSL(false));  // Set to true if using HTTPS

Using a Base Path

Organize files within a bucket by using a base path:

builder.Services.AddS3FileStorage(s3 => s3
    .WithEndpoint("s3.amazonaws.com")
    .WithCredentials("your-access-key", "your-secret-key")
    .WithBucket("my-bucket")
    .WithBasePath("uploads/2024"));  // All files stored under this prefix

Usage Examples

Upload a File

using var stream = File.OpenRead("local-file.txt");
await fileStore.CreateFileFromStreamAsync("remote/path/file.txt", stream);

Download a File

using var stream = await fileStore.GetFileStreamAsync("remote/path/file.txt");
using var fileStream = File.Create("downloaded-file.txt");
await stream.CopyToAsync(fileStream);

List Directory Contents

await foreach (var entry in fileStore.GetDirectoryContentAsync("some/directory"))
{
    if (entry.IsDirectory)
    {
        Console.WriteLine($"Directory: {entry.Path}");
    }
    else
    {
        Console.WriteLine($"File: {entry.Path} ({entry.Length} bytes)");
    }
}

Copy a File

await fileStore.CopyFileAsync("source/file.txt", "destination/file.txt");

Move a File

await fileStore.MoveFileAsync("old/path/file.txt", "new/path/file.txt");

Delete a File

var deleted = await fileStore.TryDeleteFileAsync("path/to/file.txt");

Delete a Directory

var deleted = await fileStore.TryDeleteDirectoryAsync("path/to/directory");

API Reference

S3StorageOptions

IS3FileStoreFactory

Extension Methods

License

MIT