The fan-out design pattern

What if the amount of work at different steps in our pipeline is very different?

For example, it takes a lot more time to fetch the HTML than to parse it. Or what if we don't have a pipeline at all, just a lot of tasks we would like to distribute between coroutines.

That's where the fan-out design pattern kicks in. The number of coroutines may read from the same channel, distributing the work.

We can have one coroutine produce some results:

private fun producePages() = produce {
for (i in 1..10_000) {
for (c in 'a'..'z') {
            send(i to "page$c")
        }
    }
}

And have a function that would create a coroutine that reads those results:

private fun consumePages(channel: ReceiveChannel<Pair<Int, String>>) = async {
for (p in channel) {
println(p)
    }
}

This allows us to generate an arbitrary number of consumers:

val producer = producePages()

val consumers = List(10) {
consumePages(producer)
}

runBlocking {
consumers.forEach {
        it.await...