Process Mining Data: Turning Pipeline Runs into Cost & Optimization Insights

Process Mining Data (PMD) is information we attach to pipeline execution logs so you can answer questions like:

• Which processors are the most expensive per event?
• Where do we spend time waiting (queues, retries, merges)?
• Which triggers create the biggest bursts (cronjob vs user vs system)?
• What’s the cost impact of switching from “Normal” to “Fast” processing?

In other words: PMD helps you turn “a pipeline run” into “a measurable business process”.

Qilin.Cloud already gives you Data Flow Tracking (DFT): object-level and block-level visibility into what happened.

PMD is designed to fit into that same model:

• Transaction-level fields: describe who triggered the run and with what context
• Block-level fields: describe time and cost behavior of individual execution steps

That means you don’t need a second observability system just to do process mining.
You start with the telemetry you already have.

Here are examples of PMD-style fields (conceptually):

Cost signals (block-level)

• CostPerEvent: how expensive is it to process one object in this step?
• CostPerSecond: how expensive is time spent in this step?

You can model cost in a way that matches your reality:

• compute cost (CPU-heavy transforms)
• third-party API cost (paid endpoints)
• operational cost (human attention, if you want to be cheeky)

Trigger context (transaction-level)

• TriggerAgent: who/what started this run?
• system automation
• a scheduled job
• a user action

This helps you spot patterns like:

• cronjobs cause predictable load
• users cause chaotic bursts
• system events create “silent storms”

Custom headers (transaction-level)

This is the “attach context” feature.

If you trigger pipelines via HTTP, you can pass non-standard headers that become part of the execution record (excluding standard/protected headers). That enables things like:

• “this run is for customer X”
• “this run is part of incident Y”
• “this run is part of campaign Z”

Processing speed (block-level)

Processing speed isn’t just “faster/slower”. It’s a resource policy.

Capturing the speed choice in telemetry lets you answer:

• how often do we use “Fast”?
• what does it cost?
• what does it prevent (e.g., overselling penalties)?

Imagine a pipeline like:

1. Buffer (smooth bursts)
2. Filter (policy)
3. Enrich (attributes)
4. Output connector (marketplace API)

A PMD-enriched execution log could conceptually look like:

{
  "transactionId": "dft_tx_123",
  "triggerAgent": "Cronjob",
  "customHeaders": { "tenant": "acme", "campaign": "spring-sale" },
  "blocks": [
    { "type": "BufferEntry", "durationMs": 120000, "costPerSecond": 0.01 },
    { "type": "Filter", "durationMs": 8000, "costPerEvent": 0.0002 },
    { "type": "Enrich", "durationMs": 22000, "costPerEvent": 0.0005 },
    { "type": "OutputConnectorByBatch", "durationMs": 60000, "costPerEvent": 0.001 }
  ]
}

Now you can compute things like:

• cost per pipeline run
• cost per object
• time spent waiting vs working
• which step dominates both cost and latency

That’s not just observability. That’s operational finance.

Developers

You get a quantitative feedback loop. Instead of arguing about performance, you measure and optimize.

Agencies & integrators

You can give clients something rare: transparency.
“Here’s exactly why your sync costs X and takes Y.”

Merchants

You can tie technical choices to business outcomes:

• pay for speed where it prevents penalties
• save money where delays don’t hurt revenue

Investors

PMD makes unit economics visible:

• cost per synced offer
• cost per marketplace update
• cost of reliability under load

Platforms that can measure cost can improve margins. That’s the game.

People have always optimized processes.
We just used to do it with clipboards.

PMD is the modern clipboard, built into the pipeline.