CallingConventionDispatcher 1.5.2

<div align="center">

<img src="logo.png" alt="Logo"/>

Calling Convention Dispatcher

A managed-to-native and back solution to unsupported calling conventions (x86) for .NET

<br/>

CI/CD	Release	NuGet	Coverage	Tech Stack	Platform	License

</div>

A .NET library for dynamically generating assembly stubs to translate between different x86 calling conventions. This project is designed for advanced native interoperability scenarios, such as function hooking and interfacing with libraries that use non-standard calling conventions.

The dispatcher leverages the Iced.Intel assembler engine to create JIT-compiled x86 machine code on the fly, providing a seamless bridge between managed .NET code (which typically uses Cdecl) and native code using Stdcall, Fastcall, Thiscall, or even completely custom conventions.

The Problem It Solves

When working with native code in .NET, P/Invoke and DllImport are sufficient for standard calling conventions. However, in complex scenarios like hooking a game engine function or an native application's non-public internal API, you may encounter:

• Non-standard conventions: Functions that pass arguments in specific registers (EAX, EDI, etc.) in a custom order (commonly seen in LTCG-optimized functions)
• Mixed conventions: A native function expects Fastcall, but your managed hook is Cdecl. A direct call would corrupt the stack and crash the application.
• Complex argument types: Functions that use a mix of register-based and stack-based arguments, including structs.

This library solves these problems by creating a tiny, efficient assembly stub that acts as a translator. It correctly rearranges arguments from the source convention to the target convention before calling the destination function.

Core Concepts

The dispatcher generates two types of stubs, which can be used independently or together:

1. To-Dispatcher Stub: Translates from Cdecl (the default for managed delegates) to a specified native calling convention (Fastcall, Custom, etc.). This is ideal for calling a native function from your managed code.

   • Flow: [Managed Cdecl Caller] → [To-Dispatcher Stub] → [Native Target Function]

2. From-Dispatcher Stub: Translates from a specified native calling convention to Cdecl. This is essential for hooking, where native code calls your managed hook.

   • Flow: [Native Caller] → [From-Dispatcher Stub] → [Managed Cdecl Hook]

In a typical hooking scenario, you would use both to create a complete round-trip translation.

Features

• Multi-Convention Support: Built-in logic for MicrosoftCdecl, GCCCdecl, Stdcall, Fastcall, and Thiscall.
• Custom Convention Engine: Define any calling convention imaginable by specifying which arguments are passed in which registers (EAX, ECX, EDX, EBP, EDI, etc.).
• Full Argument Support:
  • Handles primitive types (integers, floats), enums, and pointers.
  • Correctly marshals and lays out structs on the stack according to field order.
• Stack Layout Control: Supports both RightToLeft (Cdecl, Stdcall) and LeftToRight (Pascal) argument ordering.
• Stack Cleanup Management: Automatically handles stack cleanup logic for both caller (CdeclStyle) and callee (StdcallStyle).
• Custom Return Registers: Supports functions that return values in registers other than the standard EAX.
• Safe and Efficient: Uses Iced.Intel for reliable assembly generation and allocates executable memory in isolated segments.

Requirements

• .NET framework 4.8 or .NET 8
• A 32-bit (x86) target process.
• Dependencies: Iced.Intel and Serilog (for optional logging).

ABI Support

• Windows is fully supported for x86, while support for x64 is a work-in-progress (x64 fastcall, x64 vectorcall)
• Linux x86 is technically supported, but is not tested.

Usage Guide

Using the dispatcher involves three main steps: defining your function signature with attributes, instantiating the dispatcher, and generating the executable stub.

Step 1: Define the Function Delegate

Create a delegate that precisely matches the native function's signature. Use attributes to define the calling conventions and argument locations.

Example: A custom native function that takes four integer arguments in EAX, EBP, EDX, and EDI, and returns a value in ECX.

using CallingConventionDispatcher.Attributes;
using CallingConventionDispatcher.Enums;
using Iced.Intel;

// 1. Define the dispatcher's behavior.
//    - toCallConv: The target native function's convention is Custom.
//    - stackCleanup: The target function cleans up its own stack.
[FunctionDispatcherDefinition(
    toCallConv: CallConv.Custom, 
    stackCleanup: StackCleanup.StdcallStyle)]

// 2. Define the return value location.
[return: RegisterArgument(Register.ECX)]
public delegate int CustomConventionDelegate(
    // 3. Define where each argument is passed.
    [RegisterArgument(Register.EAX)] int arg1,
    [RegisterArgument(Register.EBP)] int arg2,
    [RegisterArgument(Register.EDX)] int arg3,
    [RegisterArgument(Register.EDI)] int arg4);

Step 2: Instantiate the Dispatcher

Create an instance of X86CallingConventionDispatcher<T> with your delegate type.

// The dispatcher is a generic class typed with your delegate definition.
X86CallingConventionDispatcher dispatcher = new X86CallingConventionDispatcher<CustomConventionDelegate>();

Step 3: Generate the Stub and Get its Address

Call TryGenerateToDispatcher with the memory address of the target native function. This will assemble the stub and write it into executable memory.

// The address of the native function you want to call.
ulong nativeFunctionAddress = 0x12345678; 

if (dispatcher.TryGenerateToDispatcher(nativeFunctionAddress))
{
    // Generation was successful. Get the address of our stub.
    ulong stubAddress = dispatcher.ToDispatcherAddress;

    // The generated stub is always Cdecl, so we can cast it to an
    // unmanaged C# function pointer.
    delegate* unmanaged[Cdecl]<int, int, int, int, int> cdeclFunctionPointer;
    cdeclFunctionPointer = (delegate* unmanaged[Cdecl]<int, int, int, int, int>)stubAddress;

    // Now, call the native function through our dispatcher!
    int result = cdeclFunctionPointer(10, 20, 30, 40); 
    // The dispatcher will correctly place 10 in EAX, 20 in EBP, etc.,
    // call the native function, and retrieve the result from ECX.
}

Building the Project

The project is configured for a straightforward build process using standard .NET tooling. No special dependencies outside of the .NET SDK are required.

Prerequisites

• .NET 8 SDK (or later): The SDK is required to build, test, and pack the project. The .NET 8 SDK includes the necessary compilers and targeting packs for both .NET 8 and .NET Framework 4.8. Download .NET SDK.

Building with Visual Studio / Rider

1. Clone the repository: git clone https://gitlab.com/Rawra/calling-convention-dispatcher.git

2. Navigate to the directory: cd calling-convention-dispatcher

3. Open the Solution: Open the CallingConventionDispatcher.sln file in your IDE.

4. Build the Solution:

   • Set the solution configuration to Release.
   • Build the solution using the build command (e.g., Ctrl+Shift+B in Visual Studio or from the Build menu).

Building with the Command Line (.NET CLI)

1. Clone the repository: git clone https://gitlab.com/Rawra/calling-convention-dispatcher.git

2. Navigate to the directory: cd calling-convention-dispatcher

3. Restore NuGet Packages: Run the restore command to download all required dependencies. dotnet restore

4. Build the Project: Execute the build command. Using the Release configuration is recommended for an optimized build. dotnet build --configuration Release

Build Output

After a successful build, the compiled artifacts will be located in the bin/ folder at the root of the solution directory. The structure will be as follows:

/bin
└───/Release
    ├───/net48
    │   └─── CallingConventionDispatcher.dll
    │
    └───/net8.0
        └─── CallingConventionDispatcher.dll

License

This project is licensed under the LGPLv2 License. See the LICENSE file for details.