Introducing... CupidDB!

Hello, world! I’m excited to share with you my latest project: CupidDB, an in-memory database designed for performance and efficiency. After eight years of coding professionally and working with numerous open-source projects, I felt the urge to give back to the community. For a long time, I was unsure about what and how I should contribute to the open-source community until...

The Challenge

Recently, I faced a challenge that many developers encounter when working with data-intensive applications. I needed to cache a large pandas DataFrame and share it among multiple clients. Initially, I turned to Redis, known for its speed and reliability as a caching solution. While Redis excelled at caching, it quickly became apparent that it is not ideal for my problem.

The crux of the issue was that my Python clients required only a portion of the DataFrame. This meant they were not only consuming more bandwidth than necessary but also utilizing extra memory to load the entire DataFrame, only to discard a significant portion of it once it was in memory. This inefficiency was frustrating and prompted me to think: wouldn’t it be great if there was an in-memory database that could filter DataFrames on the server side, allowing clients to retrieve only the data they need?

Enter CupidDB

This realization sparked the idea for CupidDB. My goal was to create a solution that combines the speed of Redis with the ability to filter DataFrames server-side, effectively eliminating the need for clients to load unnecessary data. I decided to write CupidDB in Rust, a language known for its performance and safety, to ensure that the database would be both fast and reliable.

To facilitate efficient communication between the server and clients, CupidDB uses the Apache Arrow columnar format. This choice allows for blazing-fast serialization and deserialization of data, making it incredibly efficient for transmitting data between the database and clients.

The Name

As I was brainstorming names for this project, I wanted something that encapsulated the essence of what CupidDB does. Given that the database stores and sends data in the Apache Arrow format, I envisioned it as a system that "shoots" arrow data to its clients. This concept led me to the name "CupidDB".

The Authors

CupidDB is a collaborative effort. I had the pleasure of working with Anon Ongsakul to bring this project to life. Anon was instrumental in developing the data filtering capabilities that allow CupidDB to serve clients efficiently.

A Contribution to the Open Source Community

With CupidDB, I hope to contribute a valuable tool to the open-source community. It’s been rewarding to develop a solution that not only addresses my needs but can also help others facing similar challenges. As developers, we constantly encounter obstacles that inspire us to innovate, and CupidDB is my answer to an issue in data caching.

I’m excited to see how CupidDB can be utilized and improved by fellow developers. I encourage anyone interested in enhancing their data processing workflows to check it out, provide feedback, and contribute to its development. Together, we can make CupidDB even better and more useful.

I’m looking forward to sharing more updates and developments in the future!

Watt Iamsuri
2024-10-31