Stream-Serializer-Extensions 1.6.0

Stream-Serializer-Extensions

This .NET library extends any Stream object with serializing methods for writing almost any object binary serialized to a stream, and deserializing any binary stream sequence.

The built in serializer supports binary serialization of

• booleans
• numbers ((U)Int8-64, Single, Double, Decimal)
• enumerations
• strings (UTF-8)
• arrays
• lists
• dictionaries
• byte arrays
• possibly any other objects with a parameterless public constructor

NOTE: Arrays, lists and dictionaries with nullable values aren't supported.

It's possible to override the build in serializers, and to add custom type serializers, too.

Methods

The Write and WriteAsync methods will be extended with supported types, while serializing some types is being done with specialized methods:

Using the WriteObject* and ReadObject* methods you can let the library decide which method to use for the given object type.

Using the WriteAny* and ReadAny* methods, you can write and read an object with a dynamic type.

Using the WriteAnyObject* and ReadAnyObject* methods you can also (de)serialize objects which have a constructor without parameters. The serializer will process public properties which have a public getter and setter. If in the future the object changes, it's not possible to deserialize an older binary sequence, unless you work with the StreamSerializerAttribute and set an object version to the type and its properties.

NOTE: Please use the *Nullable* methods for working with nullables. They'll add an extra byte for null value detection.

NOTE: In general you should use the opposite method for reading a binary sequence that you've used for writing it!

Most methods are designed specially for one type, while other methods work more generic. This is when to choose which method:

Number serialization

The WriteNumber* and ReadNumber* methods will find the best matching serialization method for a given number. For example, if you give an Int32 value which could be fit into an UInt16, the value will be converted for serialization, and you can save one byte (because the methods will store the used numeric type in an extra byte).

NOTE: All numbers will be serialized using little endian.

Custom serializer

Using the StreamSerializerAttribute attribute

The type needs to have a constructor without parameters. Properties with a public getter and setter can be serialized:

[StreamSerializer(StreamSerializerModes.OptOut)]
public class YourType
{
	public YourType() { }
	
	public string Serialized { get; set; } = null!;
	
	[StreamSerializer]
	public string? NotSerialized { get; set; }
}

stream.WriteAnyObject(new YourType(){ Serialized = "Test1", NotSerialized = "Test2" });
stream.Position = 0;
YourType deserialized = stream.ReadAnyObject<YourType>();
Assert.AreEqual(deserialized.Serialized, "Test1");
Assert.IsNull(deserialized.NotSerialized);

Using the StreamSerializerAttribute attribute at a type, you can set the serializer mode to OptOut (the default when not using the attribute) or OptIn. OptOut includes all properties, except the ones with a StreamSerializerAttribute (if the attribute doesn't OptIn). OptIn only includes properties with a StreamSerializerAttribute, except the attribute mode was set to OptOut.

To skip property name checksums, which require one extra byte per serialized property, you can set the SkipPropertyNameChecksum property value of the attribute to true.

You can use a property versioning, which supports skipping newer properties when deserializing an older binary sequence. The type attribute defines the object version, while the property attributes define the object version in which they (dis)appear.

By setting a property value position, you can modify the order of the value within the binary sequence. This ordering will be applied:

• Position
• Name

To exclude a property depending on the object version:

• FromVersion: First object version which includes the property (optional)
• Version: Last object version which includes the property (optional)

Extending the StreamSerializer

StreamSerializer.SyncSerializer.AddOrUpdate(
    typeof(YourType),
    (stream, value) => 
    {
        // Serialize value to stream
    }
);
StreamSerializer.AsyncSerializer.AddOrUpdate(
    typeof(YourType),
    async (stream, value, cancellationToken) => 
    {
        // Serialize value to stream
    }
);
StreamSerializer.SyncDeserializer.AddOrUpdate(
    typeof(YourType),
    (stream, type, version) => 
    {
        // Deserialize value from stream
        return value;
    }
);
StreamSerializer.AsyncDeserializer.AddOrUpdate(
    typeof(YourType),
    YourType.DeserializeAsync
);

public class YourType
{
    ...

    public static async Task<YourType> DeserializeAsync(Stream stream, Type type, int version, CancellationToken cancellationToken)
    {
        // Deserialize value
        return value;
    }
}

// Then you can (de)serialize like this:
stream.WriteObject(new YourType());
stream.Position = 0;
YourType instance = stream.ReadObject<YourType>();

NOTE: The asynchronous deserializer delegate uses a Task return type, while internal the task will be converted to Task<YourType>, to get the result. Because there's a lack of support for generic delegates, this seems to be the only way to go 😦

NOTE: You can attach to the StreamSerializer.OnInit event to add your custom type serializers on start. During initialization, the StreamSerializer.SyncObject object will be thread locked, and you shouldn't use the Find* methods.

NOTE: You can use the SerializerHelper class methods for boiler plate tasks like ensuring a non-null value, or validating a deserialized length value, for example.

NOTE: You should throw a SerializerException on any (de)serializing issue!

Using the IStreamSerializer interface

Your object can implement the IStreamSerializer interface or use the StreamSerializerBase base class, which implements the IStreamSerializer interface. Then you can use the WriteSerialized*, WriteObject*, ReadSerialized* and ReadObject* methods for (de)serialization.

public class YourType : StreamSerializerBase
{
    public YourType() : base() { }

    public YourType(Stream stream, int version) : base(stream, version) { }

    protected override void Serialize(Stream stream)
    {
        ...
    }

    protected override void Deserialize(Stream stream, int version)
    {
        ...
    }
}

When deserializing using the ReadAny* methods, the target type needs to be loaded from the environment. You can add your own type loading handler using the StreamSerializer.OnLoadType event. The library uses the wan24-Core NuGet package. If you want to use the wan24-Core type helper for loading types:

StreamSerializer.OnInit += (e) => StreamSerializer.OnLoadType += (s, e) =>
{
    if(e.Type != null) return;
    e.Type = TypeHelper.Instance.GetType(e.Name);
};

CAUTION: By adding the wan24-Core type helper like this, any type may be deserialized, which may be is a security issue!

When using the StreamSerializerBase base class, you can also give a value for the parameter objectVersion to the base constructor to enable object versioning. This makes it possible that newer object versions are able to deserialize from older binary sequences, and it ensures that old object versions can't deserialize from a newer binary sequence. During deserialization you can get the serialized object version like this:

    ...
    public const int OBJECT_VERSION = 1;

    public YourType() : base(objectVersion: OBJECT_VERSION) { }

    public YourType(Stream stream, int version) : base(stream, version, objectVersion: OBJECT_VERSION) { }

    ...

    protected override void Deserialize(Stream stream, int version)
    {
        int serializedVersion = ((IStreamSerializer)this).SerializedObjectVersion!;
        if(serializedVersion < 1) throw new SerializerExeption($"Unsupported {GetType()} binary sequence version #{serializedVersion}");
        ...
    }

    ...

The SerializedObjectVersion property will have a non-null value, if the object was deserialized, and the StreamSerializerBase base constructor got a object version as objectVersion parameter. Based on the serialized object version you can switch and handle the binary sequence in the required way. To access the versioning information of an object, you can use the optional IStreamSerializerVersion interface, which is implemented by StreamSerializerBase, too.

Deserializer limitations

When deserializing variable length objects (like arrays or strings), you can limit the allowed number of items/bytes (or request a minimum count) using the specific (de)serializer methods, by giving minLen and maxLen parameter values.

Serializer version

The StreamSerializer.Version property holds the serializer version information, which you may write at the beginning of a serialized stream. The first byte (values 0-255) is used by the stream serializer internal. If you'd like custom versioning, please use bytes 2 to 4 (values 256+) for your own version number (which you can make more readable by bit shifting the value). The stream serializer extensions will only use the first 8 bits to identify a serializer version number, which can be given to all deserializer methods, while the StreamSerializer.Version value is the default serializer version number to use in case no version parameter was given to a deserializer method.

NOTE: The serializing methods will always use the latest binary sequence format version when writing, no matter which value was set to StreamSerializer.Version!

Binary sequence size

Since the serializer may be used without any configuration and contract information, it tries to create the smallest binary sequence size which still offers enough information for an also unconfigured deserialization without contracts. The currently resulting sequence size is a trade off between size and usability. By defining customized type serializers you can optimize the resulting sequence size for any type. Keep in mind, that using the most specific (de)serializing methods will result in smaller binary sequences.

Stream object enumeration

You can enumerate serialized objects from any stream like this:

foreach(AnyType obj in stream.EnumerateSerialized<AnyType>())
{
    ...
}

await foreach(AnyType obj in stream.EnumerateSerializedAsync<AnyType>())
{
    ...
}

In this example it's assumed that AnyType implements IStreamSerializer.

For enumerating any other type, you can implement an enumerator using the StreamEnumeratorBase and StreamAsyncEnumeratorBase base classes. The only thing that you'll have to do is to override the ReadObject(Async) method, which finally reads the next object to yield from the Stream. Then you can enumerate easily using the static Enumerate(Async) methods of the base types.

This is a sample bool enumerator implementation, which uses the ReadBool method:

public class StreamBoolEnumerator : StreamEnumeratorBase<bool>
{
    public StreamBoolEnumerator(Stream stream, int? version = null)
        :base(stream, version)
        { }

    protected override int ReadObject() => Stream.ReadBool(SerializerVersion);
}

To provide the enumerator as a stream extension method:

public static IEnumerable<bool> EnumerateBool(this Stream stream, int? version = null)
    => StreamBoolEnumerator.Enumerate<StreamBoolEnumerator>(stream, version);

These enumerators are implemented at present:

Security

The base serializer supports basic types and lists. Especially when deserializing lists, you should define a minimum and a maximum length.

Per default all objects that you want to deserialize using a ReadAnyObject* method, the type is required to use the StreamSerializerAttribute for security reasons. If you need to allow all types, you can set the StreamExtensions.AnyObjectRequireAttribute property value to false.

CAUTION: If you allow deserialization of any type, deserializing a manipulated input stream could harm your computer!

CAUTION: During serialization it's possible to end up in an endless recursion, if any nested property serves an object which is in the current stack already.

CAUTION: If you don't use versioning, you may end up in broken binary sequences which can't be deserialized anymore. Also a deserialization attempt could harm your computer!

The job of the serializer is to write and read objects to/from a binary sequence. There's no compression, encryption or hashing built in. If you want to compress/protect a created binary sequence, you can apply compression, encryption and hashing on the result as you want.

Object validation will be applied to deserialized objects to ensure their validity.

Roadmap

• Enumerate objects from a stream