Process Mining Data (PMD) is information we attach to pipeline execution logs so you can answer questions like:

• Which processors are the most expensive per event?
• Where do we spend time waiting (queues, retries, merges)?
• Which triggers create the biggest bursts (cronjob vs user vs system)?
• What’s the cost impact of switching from “Normal” to “Fast” processing?

In other words: PMD helps you turn “a pipeline run” into “a measurable business process”.

Qilin.Cloud already gives you Data Flow Tracking (DFT): object-level and block-level visibility into what happened.

PMD is designed to fit into that same model:

• Transaction-level fields: describe who triggered the run and with what context
• Block-level fields: describe time and cost behavior of individual execution steps

That means you don’t need a second observability system just to do process mining.
You start with the telemetry you already have.

Here are examples of PMD-style fields (conceptually):

Cost signals (block-level)

• CostPerEvent: how expensive is it to process one object in this step?
• CostPerSecond: how expensive is time spent in this step?

You can model cost in a way that matches your reality:

• compute cost (CPU-heavy transforms)
• third-party API cost (paid endpoints)
• operational cost (human attention, if you want to be cheeky)

Trigger context (transaction-level)

• TriggerAgent: who/what started this run?
• system automation
• a scheduled job
• a user action

This helps you spot patterns like:

• cronjobs cause predictable load
• users cause chaotic bursts
• system events create “silent storms”

Custom headers (transaction-level)

This is the “attach context” feature.

If you trigger pipelines via HTTP, you can pass non-standard headers that become part of the execution record (excluding standard/protected headers). That enables things like:

• “this run is for customer X”
• “this run is part of incident Y”
• “this run is part of campaign Z”

Processing speed (block-level)

Processing speed isn’t just “faster/slower”. It’s a resource policy.

Capturing the speed choice in telemetry lets you answer:

• how often do we use “Fast”?
• what does it cost?
• what does it prevent (e.g., overselling penalties)?

Imagine a pipeline like:

1. Buffer (smooth bursts)
2. Filter (policy)
3. Enrich (attributes)
4. Output connector (marketplace API)

A PMD-enriched execution log could conceptually look like:

{
  "transactionId": "dft_tx_123",
  "triggerAgent": "Cronjob",
  "customHeaders": { "tenant": "acme", "campaign": "spring-sale" },
  "blocks": [
    { "type": "BufferEntry", "durationMs": 120000, "costPerSecond": 0.01 },
    { "type": "Filter", "durationMs": 8000, "costPerEvent": 0.0002 },
    { "type": "Enrich", "durationMs": 22000, "costPerEvent": 0.0005 },
    { "type": "OutputConnectorByBatch", "durationMs": 60000, "costPerEvent": 0.001 }
  ]
}

Now you can compute things like:

• cost per pipeline run
• cost per object
• time spent waiting vs working
• which step dominates both cost and latency

That’s not just observability. That’s operational finance.

Developers

You get a quantitative feedback loop. Instead of arguing about performance, you measure and optimize.

Agencies & integrators

You can give clients something rare: transparency.
“Here’s exactly why your sync costs X and takes Y.”

Merchants

You can tie technical choices to business outcomes:

• pay for speed where it prevents penalties
• save money where delays don’t hurt revenue

Investors

PMD makes unit economics visible:

• cost per synced offer
• cost per marketplace update
• cost of reliability under load

Platforms that can measure cost can improve margins. That’s the game.

People have always optimized processes.
We just used to do it with clipboards.

PMD is the modern clipboard, built into the pipeline.

A modern commerce sync is rarely one program.

It’s a chain of services:

• an API receives an update
• a workflow validates and normalizes it
• the orchestrator triggers a pipeline
• processors enrich / filter / transform data
• an output connector calls a third-party API
• status and telemetry are recorded

Tracing ties all of that together with one idea:

> Every request gets a unique “case file”, and every step in the system writes notes into it.

In practice, that means:

– a trace ID that stays the same across services
– spans (timed steps) for each meaningful operation
– correlation so logs, metrics, and errors all point to the same story

Qilin.Cloud is built around the idea that pipelines are operational assets, not just “integration code”.

That’s why we treat observability as a first-class capability:

– Data Flow Tracking (DFT) gives you object-level delivery status and block-level execution details
– Transfer Status Engine (TSE) protects idempotency and prevents duplicate transfers
– Distributed tracing connects the dots between “API call”, “pipeline run”, and “connector request”

So the question changes from:

> “Why is the sync slow?”

to

> “Which span dominates latency, and what’s the cheapest fix?”

A useful tracing strategy is opinionated. You don’t want “everything everywhere all at once”. You want the events that matter.

In Qilin.Cloud we focus on spans like:

• request received (API gateway / ingestion)
• validation and schema checks
• hashing / version comparison (change detection)
• queuing / scheduling delays (buffering, backpressure)
• processor execution (filter/enrich/transform/merge)
• connector calls (outbound requests + retries)
• persistence and audit logging (DFT/TSE updates)

That’s the critical path: the stuff that explains time, cost, and failures.

Let’s say you push a stock update for 20,000 offers.

Without tracing, you’ll typically see:

• “Accepted”
• later: “Some items failed”
• somewhere: “429 Too Many Requests”

With tracing + DFT you can break the run down into:

• queue time (did we buffer the burst?)
• processing time (CPU work in processors)
• connector time (external API latency + rate limits)
• retries / backoff time (how many attempts and why)

A “slow run” becomes a measurable composition of spans.

1) Correlate API requests with pipeline runs

When you make API calls to Qilin.Cloud (or trigger pipelines from a webhook), attach a correlation header.

Example (illustrative):

curl -X POST "https://api.qilin.cloud/<resource>" \
-H "Authorization: Bearer <token>" \
-H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" \
-H "x-qilin-processing-speed: fast" \
-d @payload.json

Now your “case file” follows the object through the system. If something fails, you don’t hunt. You follow the trace.

2) Make retries *observable* (not just automatic)

Retries are a double-edged sword:

• they increase reliability
• they can also hide instability and raise costs

Tracing makes retries explicit:

• which calls retry
• how often
• how long backoff takes
• whether we end up timing out anyway

3) Turn debugging into a feedback loop

Once traces exist, you can optimize systematically:

• find the slow processor and refine it
• split a pipeline into different speeds (critical vs routine)
• introduce buffering for bursty endpoints
• tighten timeouts and improve fallbacks

Developers

You get reproducible debugging. No more “works on my machine” when the machine is a distributed system.

Agencies & integrators

You can deliver faster because you can prove where time goes. This is a huge advantage when clients ask, “Why does this take so long?”

Merchants & operators

You get fewer incidents and faster resolution. The “why is my marketplace out of sync?” conversation becomes short and factual.

Investors

Tracing reduces MTTR (mean time to repair) and supports scalable operations. That’s the difference between “a promising prototype” and “a platform you can run profitably.”

Good operations isn’t magic. It’s instrumentation.

Distributed tracing is the modern version of what we always wanted:

• one identifier
• one timeline
• one story