commerce data | Qilin.Cloud

Flexible Attributes: Letting Your Commerce Data Be Weird (Without Losing It)

Han Ho — Mon, 31 Mar 2025 08:00:00 +0000

If commerce data were polite, life would be easy.

But commerce data is not polite.

It shows up with:

“custom fields”
“special flags”
“this one client calls it `deepLinkCode` and it’s very important”
“our ERP exports it as a string except on Tuesdays”
“Shopware puts it here, Otto puts it there, and nobody agrees on types”

In March, we’ve leaned into a reality every experienced integration team already knows:

Rigid schemas break. Flexible data wins.

So we’ve been strengthening Qilin.Cloud’s approach to Flexible Attributes – a way to carry “extra” fields safely through pipelines without turning your platform into a validation battleground.

Table of Contents

What are Flexible Attributes?

Flexible Attributes are custom properties that are not part of Qilin’s core domain models (Product, Order, Offer, etc.) but still matter to real integrations.

Instead of discarding them or forcing them into awkward workarounds, Qilin can store them in a dedicated structure (think: an `options` map) so they can:

survive ingestion
flow through pipelines
be forwarded to destinations that support them
be enriched, filtered, or transformed when needed

The core philosophy is old-school and practical:

> Don’t lose data you don’t understand yet.

The important decision: Qilin stores, not polices

One of the hardest parts of “flexible fields” is the temptation to over-validate.

It sounds responsible to enforce types and constraints, but in real integrations it often becomes a liability:

different systems define constraints differently
constraints change over time
connectors may already validate before sending data
strict validation in the middle can block otherwise useful workflows

So the platform approach is intentionally pragmatic:

Qilin.Cloud stores flexible attributes and forwards validation metadata—but does not try to re-validate every value.

That keeps Qilin.Cloud compatible across wildly different ecosystems.

A concrete example: when Shopware has a field Qilin.Cloud doesn’t

Imagine Shopware orders contain a field:

{
  "id": "order_1",
  "orderNumber": "1234",
  "deepLinkCode": "code32543343"
}

If `deepLinkCode` isn’t a native Qilin.Cloud field, the connector can represent it as a flexible attribute:

{
  "id": "order_1",
  "orderNumber": "1234",
  "options": {
    "deepLinkCode": "code32543343"
  }
}

Now the value isn’t lost.

It can be:

forwarded to another system that supports it
used in filtering logic
used to enrich downstream messages
included in auditing and tracking

We invite you to share your experiences and lessons learned with Qilin.Cloud’s innovative technology platform for composable e-commerce. Your story can inspire others and help the whole community to improve.

Share your Qilin.Cloud Success Story

Your Journey

Why agencies should quietly celebrate this

Flexible attributes are an “agency feature” even when merchants never hear the term.

Because agencies deal with:

client-specific customizations
long-tail requirements
systems with inconsistent modeling

When flexible attributes are first-class:

fewer custom connector forks are needed
implementations become more reusable
edge cases stop turning into permanent technical debt

This is the kind of platform capability that makes projects easier to deliver – and easier to maintain.

For developers: filtering & enrichment with flexible fields

Once flexible attributes are carried reliably, you can do useful things with them:

filter objects based on a flexible attribute
enrich objects by calculating new attributes
map attributes into destination-specific formats

Combined with Filter Predicates, it becomes possible to express business rules like:

“If `options.deepLinkCode` exists, route to system A, else system B”
“Only sync offers where `options.partnerTag` is in the allowed list”

Without custom code. Without duct tape.

For merchants and investors

Merchants: custom fields stop being “integration killers”. You can evolve your business model without breaking pipelines.
Investors: flexible attributes reduce per-customer customization cost while increasing platform adaptability – very good leverage.

What’s next

Flexible data is powerful, but it becomes truly useful when you can process it at scale.

Next month, we’ll cover work around bulk operations and async workflows – how Qilin.Cloud handles big imports without turning your API into a time-out lottery.

Let your data be weird (and keep it anyway)

The world isn’t tidy.

Commerce data definitely isn’t tidy.

Qilin.Cloud is building toward a platform where “we have one more custom field” isn’t a problem – it’s just Tuesday.

The post Flexible Attributes: Letting Your Commerce Data Be Weird (Without Losing It) appeared first on Qilin.Cloud.

The Egression Module: Shipping Commerce Data Without Shipping Excuses

Thien Trinh Duc — Mon, 30 Sep 2024 11:26:00 +0000

You’ve probably done this the “classic” way before: a cron job exports data, a script pushes it somewhere, and everyone crosses their fingers that categories arrive before products and nothing hits a rate limit at 02:00 AM.

It works… right up until it doesn’t.

Modern commerce isn’t a single storefront anymore. It’s a constellation of channels, each with its own quirks, dependencies, and “please don’t call us more than X times per second” constraints. That’s why September’s engineering focus has been on one of the least glamorous—but most important—parts of the platform:

Getting data out reliably.

Welcome to the Egression Module: the part of Qilin.Cloud that turns “here’s what changed” into “it arrived, in the right order, with retries, without chaos”.

Table of Contents

The Egression Module in plain English

Think of the Egression Module like a logistics hub.

Upstream systems (your ERP, PIM, shop system, marketplace feeds, NiFi flows, custom processors) don’t ship parcels directly to every destination. They hand over jobs to a hub.

The hub then answers five practical questions for every job:

Whose package is this? (subscription)
Where did it come from? (source)
Where should it go? (target channel / connector endpoint)
When should it go? (now, or scheduled later)
What is it? (object type + object IDs + action like create/update/delete)

That’s the foundation for doing commerce integrations the way seasoned teams have always wanted to do them: predictable, traceable, and recoverable.

Why “jobs” beat “payload dumps”

Old-school integrations love to ship massive payloads and hope the destination sorts it out.

But e-commerce data has dependencies:

Categories and brands often need to exist before products reference them.
Offers might depend on products.
Orders might require customer and shipping context.

So instead of blindly dumping big blobs, Qilin.Cloud models the “work to do” explicitly as jobs, and lets the platform orchestrate ordering and parallelism safely.

Under the hood: conveyor belts, partitions, and rate limits

A logistics hub needs conveyor belts.

In Qilin.Cloud, the Event Processor pulls jobs from storage and processes them:

Sequentially when ordering matters (same partition key).
In parallel when independence allows it (different partition keys).

A partition key is basically: “these jobs must stay in order together”.

Example: all product updates for the same storefront could share a partition key, while different storefronts run concurrently.

Then comes the real-world pain point: connector rate limits.

Before sending a job to an output connector, the processor checks whether that connector is currently over its allowed throughput. If it is, the job gets delayed and rescheduled. No heroics, no drama, no accidental DDoS.

The delivery loop: from job to connector call

At a high level, the delivery loop looks like this:

Upstream sends an HTTP request to create jobs (object type, object IDs, subscription, action).
Jobs are stored and then pushed into the Event Processor’s workflow.
The Event Processor resolves:
– the object payload via Metadata APIs,
– the correct connector endpoint based on the target channel,
– the authentication method configured for that subscription + channel.
The processor calls the output connector with:
– the action (create/update/delete),
– the object bodies,
– the authentication context.
The job is marked success or failed.
– On failure, it can be rescheduled automatically based on response codes and retry rules.

This is the unsexy part of a platform that makes the sexy part possible: shipping data without shipping excuses.

Share your Qilin.Cloud Success Story

Your Journey

Real-world example: “Catalog sync without the midnight panic”

Scenario: a merchant is launching a new collection across multiple channels.

They push a batch of:

Categories
Manufacturers / brands
Products
Offers

The Egression Module can process these with controlled ordering:

Belt 1: Categories and brands first
Belt 2: Products next
Belt 3: Offers once products exist

And it can do it across multiple destinations in parallel—without mixing up ordering where it matters.

For developers

If you build integrations for a living, the Egression Module should feel like coming home:

Explicit orchestration beats implicit side effects.
Partitioning gives you concurrency and determinism.
Rate-limit aware scheduling prevents the classic “works in staging, melts in production” story.
Retry as a first-class concept means fewer custom “retry-daemon” scripts.

In short: the platform does the boring parts so you can do the clever parts.

For merchants, agencies, and investors

Merchants: fewer partial syncs, fewer “why is it missing?” mysteries.
Agencies: predictable delivery semantics mean faster implementations and fewer late-night support calls.
Investors: reliable egression is compounding platform value—it enables more connectors, more volume, and lower operational cost as usage grows.

What’s next

Now that delivery is getting sharper, the next step is controlling what should be delivered—without having to write custom code for every rule. Next month’s topic will dive into Filter Predicates: routing logic you can version, audit, and reuse across pipelines.

Ready to put it to work?

If you’ve ever had to rebuild the same “delivery engine” in three different client projects… you already know why this matters.

Qilin.Cloud is where that hard-earned integration wisdom becomes reusable infrastructure.

The post The Egression Module: Shipping Commerce Data Without Shipping Excuses appeared first on Qilin.Cloud.