Merge, Timeouts, Retries: Reliability Patterns for Composable Pipelines

In a composable pipeline, you often run things in parallel:

• fetch price from system A
• fetch description from system B
• fetch stock from system C

Eventually, you need to bring results back together.

That’s what the Merge Processor does.

But the tricky part is: how long do you wait, and what do you do when not everything arrives?

So Qilin supports different merge policies.

1) Wait for all input processors (default)

This is the “strict correctness” mode:

• Merge waits until all parent processors finish
• No timeout policy
• Best when every input is required for a valid result

Use this when incomplete data would be worse than delayed data.

2) Wait for any input processor

This is the “fast enough” mode:

• Merge starts a timeout countdown once any input finishes
• When the timeout ends, merge proceeds with whatever results arrived

Use this when you value speed and partial results are acceptable.

3) Wait for required input processors

This is the “hybrid” mode:

• define a list of required parents
• wait for those to complete
• start a timeout countdown afterward
• proceed even if non-required inputs didn’t arrive

Use this when some inputs are critical and others are optional enhancements.

A timeout is not always a failure. Sometimes it’s just a signal.

So processors can be configured with:

• TimeoutConfig (how long is “too long”)
• RetryConfig (how many times to retry and how long to wait between tries)
• OnErrorBehavior (continue vs stop)

A simple retry policy looks like:

"retryConfig": {
  "maximumRetries": 3,
  "retryAfterInSeconds": 3
}

This gives you structured resilience without rewriting your own scheduler logic.

Let’s say you enrich products using:

• ERP (core master data) → required
• PIM (marketing text) → optional
• Pricing service (dynamic pricing) → required

A “required inputs” merge policy lets you:

• wait for ERP + pricing
• proceed even if PIM is slow or unavailable
• still produce a usable product update

That’s a pipeline that keeps the business running without silently dropping correctness.

Alongside merge reliability, we’ve also been expanding processors that work directly with object content:

• content filtering based on reusable predicate logic
• content enrichment designed to combine or augment fields

The long-term goal: keep transformations inside the pipeline, where they can be tracked, tested, and maintained.

Developers

• clear semantics instead of “mystery retries”
• consistent timeout and retry behavior across processors
• less custom glue code for parallel fan-out / fan-in patterns

Merchants & agencies

• fewer brittle syncs that collapse when one dependency hiccups
• pipelines that degrade gracefully under partial failure
• easier troubleshooting because behavior is explicit, not implied

Investors

Reliability features reduce operational cost and support load. They’re not just technical niceties – they’re economic multipliers.

Once you can merge and retry reliably, you can start making a new trade:

batching vs real-time.

July will introduce more around queue-based buffering – because sometimes the smartest pipeline isn’t the fastest one, it’s the one that ships in stable, cost-efficient chunks.

The “old way” taught us the patterns.

Qilin.Cloud is turning those patterns into configurable primitives – so your pipelines can behave like experienced engineers wrote them… even when nobody is awake at 03:00 AM.