LanguageExt.Pipes 5.0.0-beta-36

LanguageExt.Pipes

This feature of language-ext is based on the wonderful work of Gabriella Gonzalez on the Haskell Pipes library. I have had to make some significant changes to make it work in C#, but the essence is the same, and the core types and composition of the components is exactly the same.

• If you find this feature confusing at first, and it wouldn't be surprising as it's quite a complex idea, there are some examples in the EffectsExample sample in the repo

Conventional stream programming forces you to choose only two of the following three features:

1. Effects
2. Streaming
3. Composability

If you sacrifice Effects you get IEnumerable, which you can transform using composable functions in constant space, but without interleaving effects (other than of the imperative kind).

If you sacrifice Streaming you get 'Traverse' and 'Sequence', which are composable and effectful, but do not return a single result until the whole list has first been processed and loaded into memory.

If you sacrifice Composability you write a tightly coupled for loops, and fire off imperative side-effects like they're going out of style. Which is streaming and effectful, but is not modular or separable.

Pipes gives you all three features: effectful, streaming, and composable programming. Pipes also provides a wide variety of stream programming abstractions which are all subsets of a single unified machinery:

• effectful Producer,
• effectful Consumer,
• effectful Pipe (like Unix pipes)

On top of that, Pipes has more advanced features, including bi-directional streaming. This comes into play when fusing clients and servers:

• effectful Client,
• effectful Server,

All of these are connectable and you can combine them together in clever and unexpected ways because they all share the same underlying type.

The pipes ecosystem decouples stream processing stages from each other so that you can mix and match diverse stages to produce useful streaming programs. If you are a library writer, pipes lets you package up streaming components into a reusable interface. If you are an application writer, pipes lets you connect pre-made streaming components with minimal effort to produce a highly-efficient program that streams data in constant memory.

To enforce loose coupling, components can only communicate using two commands:

• yield: Send output data
• awaiting: Receive input data

Pipes has four types of components built around these two commands:

• Producer can only yield values and they model streaming sources
• Consumer can only be awaiting values and they model streaming sinks
• Pipe can both yield and be awaiting values and they model stream transformations
• Effect can neither yield nor be awaiting and they model non-streaming components

Pipes uses parametric polymorphism (i.e. generics) to overload all operations.

You've probably noticed this overloading already:

• yield works within both Producer and a Pipe
• Consumer works within both Consumer and Pipe
• The operator | connects Producer, Consumer, and Pipe in varying ways

This overloading is great when it works, but when connections fail they produce type errors that appear intimidating at first. This section explains the underlying types so that you can work through type errors intelligently.

Producer, Consumer, Pipe, and Effect are all special cases of a single underlying type: Proxy. This overarching type permits fully bidirectional communication on both an upstream and downstream interface.

You can think of it as having the following shape:

Proxy<RT, UOut, UIn, DIn, DOut, A>

      Upstream | Downstream
          +---------+
          |         |
    UOut ◄--       ◄-- DIn   -- Information flowing upstream
          |         |
    UIn --►        --► DOut  -- Information flowing downstream
          |    |    |
          +----|----+
               |
               A

The four core types do not use the upstream flow of information. This means that the UOut and DIn in the above diagram go unused unless you use the more advanced features.

Pipes uses type synonyms to hide unused inputs or outputs and clean up type signatures. These type synonyms come in two flavors:

• Concrete type synonyms that explicitly close unused inputs and outputs of the Proxy type

• Polymorphic type synonyms that don't explicitly close unused inputs or outputs

The concrete type synonyms use Unit to close unused inputs and Void (the uninhabited type) to close unused outputs:

• Effect: explicitly closes both ends, forbidding awaiting and yield

   Effect<RT, A> = Proxy<RT, Void, Unit, Unit, Void, A>
  
            Upstream | Downstream
                +---------+
                |         |
          Void ◄--       ◄-- Unit
                |         |
          Unit --►       --► Void
                |    |    |
                +----|----+
                     |
                     A
  

• Producer: explicitly closes the upstream end, forbidding awaiting

   Producer<RT, OUT, A> = Proxy<RT, Void, Unit, Unit, OUT, A>
  
            Upstream | Downstream
                +---------+
                |         |
          Void ◄--       ◄-- Unit
                |         |
          Unit --►       --► OUT
                |    |    |
                +----|----+
                     |
                     A
  

• Consumer: explicitly closes the downstream end, forbidding yield

    Consumer<RT, IN, A> = Proxy<RT, Unit, IN, Unit, Void, A>
  
            Upstream | Downstream
                +---------+
                |         |
          Unit ◄--       ◄-- Unit
                |         |
            IN --►       --► Void
                |    |    |
                +----|----+
                     |
                     A
  

• Pipe: marks both ends open, allowing both awaiting and yield

    Pipe<RT, IN, OUT, A> = Proxy<RT, Unit, IN, Unit, OUT, A>
  
            Upstream | Downstream
                +---------+
                |         |
          Unit ◄--       ◄-- Unit
                |         |
            IN --►       --► OUT
                |    |    |
                +----|----+
                     |
                     A
  

When you compose Proxy using | all you are doing is placing them side by side and fusing them laterally. For example, when you compose a Producer, Pipe, and a Consumer, you can think of information flowing like this:

            Producer                Pipe                 Consumer
         +------------+          +------------+          +-------------+
         |            |          |            |          |             |
   Void ◄--          ◄--  Unit   ◄--         ◄--  Unit  ◄--           ◄-- Unit
         |  readLine  |          |  parseInt  |          |  writeLine  |
   Unit --►         --► string  --►          --► string --►           --► Void
         |     |      |          |    |       |          |      |      |
         +-----|------+          +----|-------+          +------|------+
               v                     v                       v
               ()                    ()                      ()

Composition fuses away the intermediate interfaces, leaving behind an Effect:

                       Effect
        +-----------------------------------+
        |                                   |
  Void ◄--                                 ◄-- Unit
        |  readLine | parseInt | writeLine  |
  Unit --►                                 --► Void
        |                                   |
        +----------------|------------------+
                        Unit