The Decision-Making Process Behind the Qilin Event Processor

Every problem-solving journey begins with the recognition of a challenge. At Qilin.Cloud, often involves a meticulous examination of our system’s performance, user feedback, or an evolving technological landscape. The key is to proactively identify areas that need improvement or potential roadblocks that could hinder the seamless operation of our services.

Our platform thrives on transferring vast amounts of “objects” (think musical notes) between various channels. Each transfer involves a series of crucial “jobs” (like instrument solos), demanding flawless execution in the right order. On average, each flow handles roughly 100 object creations/updates/deletions every 15 minutes, synchronizing across 5 output channels (like different sections of the orchestra). With Qilin’s algorithms, this translates to millions of jobs daily, a veritable cacophony begging for organization.

To orchestrate this data concerto, we needed a maestro capable of both parallel and sequential processing based on specific conditions. Enter the Qilin EP, the conductor ensuring each note hits its mark at the perfect time.

Once the challenge is on the table, the next crucial step is framing the problem. This involves breaking down the overarching issue into manageable components. By defining the problem clearly, we create a shared understanding among team members, laying the groundwork for effective problem-solving discussions.

One challenge within the module involved delivering processed data to designated “output connectors” (think various concert halls). The data resides in Object Storage (think sheet music in a library), and we needed a system to:

• Scan Object Storage: Identify and send relevant data to connectors.
• Track Transfer Status: Ensure each object reaches its designated connector.
• Retry Failed Transfers: Offering automatic retries and easy logging for manual intervention to guarantee seamless delivery even if hiccups occur.
• Scalability: Adapting to an ever-growing orchestra of data, without missing a beat.
• Cost Optimization: Prioritize cost-effective solutions without compromising performance.

Decision-making at Qilin.Cloud is a team effort. As soon as a challenge is identified, the first step involves bringing together a group of team members. This interdisciplinary collaboration ensures that various perspectives, from engineering to user experience, are considered. Our team thrives on collective intelligence, and every member’s input is valued.

Rather than diving straight into technical intricacies, our team first explores solutions. This phase is about envisioning different approaches that could address the problem. Each solution is weighed not only in terms of technical feasibility but also in terms of its alignment with our overarching goals and user expectations. Our team faced a critical decision: How to orchestrate this symphony?

One distinctive aspect of our decision-making process is the emphasis on comparative analysis. We often find ourselves at a crossroads where multiple solutions could be viable. In such cases, our team meticulously compares options, considering factors beyond technical efficiency. This includes aspects like ease of implementation, scalability, and potential impact on user experience.

Our team gathered, buzzed with diverse ideas and perspectives, then carefully considered two main solutions:

This option utilized a database to store data and a message broker to schedule jobs. It seemed familiar and potentially scalable.

This bolder approach involved building our job scheduler within the database. While initially complex, it promised cost-effectiveness and easier scaling.

Understanding the benefits and disadvantages of each potential solution is pivotal. Our decision-making process involves a holistic evaluation, considering short-term gains and long-term implications. We prioritize solutions that not only resolve the immediate challenge but also contribute positively to the overall system architecture and user satisfaction.

We meticulously weighed the pros and cons of each solution:

• Pros: Familiar technology, potentially easier initial implementation.
• Cons: Complex retry logic due to FIFO blocking, debugging challenges due to the intricate interplay between components, high cost and increased maintenance overhead.

This option relied on well-established technologies, similar to experienced actors comfortable in their roles. Its advantages included high reliability and familiarity for our team. However, the setup resembled a complex backstage maze, potentially leading to high costs and performance bottlenecks when transferring high-volume data.

• Pros: Tailored to our specific needs, simpler retrying and debugging, potentially more scalable and cost-effective.
• Cons: Requires development effort because the initial complexity of implementing a custom scheduler may be more difficult than using existing tools.

This option resembled a fresh-faced playwright crafting a tailored script. We will build the Qilin EP from scratch, ensuring it perfectly fits our platform’s needs. While requiring more development effort, it promised enhanced scalability and cost-effectiveness, like a streamlined production with minimal set changes.

We did not rush into a choice. Our collaborative process involved:

• Brainstorming sessions: Diverse perspectives from developers, architects, and product managers were heard and valued.
• Technical evaluation: We meticulously analyzed the pros and cons of each solution, considering performance, scalability, and cost.
• Prototyping and testing: We built and tested small-scale prototypes of both solutions to assess their feasibility and performance in real-world scenarios.

Before committing to a solution, our team believes in the power of testing. This involves running pilot programs, conducting simulations, or employing other testing methodologies to gauge how a proposed solution performs in a real-world or controlled environment. Testing provides valuable insights and often reveals nuances that might not be apparent in theoretical discussions. To make an informed decision, we conducted thorough testing. We evaluated both solutions based on:

• Partitioning: Ensuring flawless order for complex data transfers.
• Performance: How quickly and efficiently could each handle the data load?
• Fault Tolerance: Guaranteeing a seamless performance even when unexpected cues arise.
• Scalability: Adapting gracefully to an ever-expanding workload.
• Cost-effectiveness: How much would each option cost to implement and maintain?

After analyzing the rehearsals and feedback, the curtain rose on our decision. While Solution 1 offered comfort and familiarity, Solution 2’s adaptability, cost-effectiveness, and custom-built efficiency ultimately stole the show. Qilin EP was born through rigorous collaboration and testing.

• Cost-effectiveness: The significant cost savings compared to Solution 1, especially for scaling needs.
• Scalability: The custom job scheduler offered a more flexible and scalable solution for future growth.
• Simplicity in the long run: While initial implementation might be marginally more complex, the long-term maintenance and debugging would be easier due to the reduced number of components.

Every decision, whether successful or not, is an opportunity for learning. At Qilin.Cloud, we prioritize the documentation of lessons learned from each problem-solving journey. This knowledge repository serves as a valuable resource for continuous improvement, enabling our team to evolve and adapt in an ever-changing landscape.

The Qilin EP is a living, breathing entity, constantly evolving to meet the demands of our growing platform. While we chose Solution 2, the decision-making process was not a one-time event. We continuously monitor the EP’s performance, gather feedback, and explore new possibilities. This dedication to optimizing our data flow symphony ensures that Qilin delivers its performance with the precision and grace of a well-conducted orchestra.

This blog post has offered a glimpse into the intricate decision-making process behind the Qilin Event Processor. We hope it has shed light on the challenges we faced, the solutions we considered, and the rationale behind our final choice. By sharing our backstage story, we aim to inspire a culture of collaboration and continuous improvement, where every member of the team can contribute to the next showstopping innovation. Remember, the symphony of data never truly ends, and we remain committed to perfecting our maestro for the benefit of our users.