Company has rearchitected Apache Pulsar for a cost-conscious data streaming era
StreamNative, the cloud-native data streaming company founded by the original creators of Apache Pulsar, today launched Ursa, a Kafka-compatible data streaming engine built on top of lakehouse storage. Ursa is advancing Apache Pulsar with added functionality in public and private cloud environments to take data streaming to its next-generation.
Although cloud computing reduced operational cost in the last decade, there has been increased cost in infrastructure, especially networking. For technologies like data streaming and analytics where networking costs roughly 80% of entire IT spend, optimizing networking became a primary concern for many companies. Additionally, most cloud vendors charge high fees for egress traffic which makes it harder to aggregate data from multiple sources. These compounding factors drove the emergence of the “lakehouse” paradigm for unified batch and streaming data destinations. Data streaming capabilities must continue advancing to meet modern demands for building real-time analytics on large-scale cloud data platforms.
Ursa leverages Apache Pulsar’s architectural principles and benefits, such as separate compute from storage, a rebalance-free architectural design, and cloud-first storage. Yet, Ursa has three significant updates:
- Kafka API Compatibility. Ursa is completely compatible with Kafka API. This allows users to migrate to StreamNative without rewriting applications and continue to use the entire Kafka ecosystem out-of-the box.
- A Departure from ZooKeeper and BookKeeper. Users no longer have to manage Zookeeper and have the ability to opt out Bookkeeper for cost optimized workloads, enabling easier deployments in modern containerized environments, such as Kubernetes. Ursa leverages commodity object storage for cost-effective data replication, eliminating the excessive inter-AZ traffic traditionally seen in other data streaming technologies.
- Lakehouse First. Lakehouse is the destination of all the analytical data, and streaming data will eventually land in a lakehouse. With the Ursa engine, data can be made immediately available in lakehouse formats, eliminating the complex and costly data movement between Pulsar, Kafka and your lakehouse. This allows for faster and cost-effective data processing and analysis with other tools, in addition to no longer needing data tiering between different storage systems. The use of lakehouse storage formats such as Delta Lake, Apache Hudi and Apache Iceberg for long term retention keeps costs to a minimum.
“We are thrilled to see StreamNative adopt the Delta Lake format in lakehouse storage, a testament to their forward-thinking approach to data management. Delta Lake’s growing popularity reflects its robust capabilities, and StreamNative’s integration of this format signifies a significant advancement for its users. This move not only enhances their offerings but also aligns with industry-leading standards for data reliability and scalability,” said Himanshu Raja, Director of Product Management at Databricks.
“When we founded StreamNative, our vision was to provide a truly modern data streaming service in the cloud, leveraging the architectural benefits pioneered in open-source, Apache Pulsar. While our initial cloud offering based on Pulsar was an industry leader, we recognized that realizing our vision of a streaming data cloud required rethinking system architecture, operations, and the user experience from the ground up for the cost conscious time,” said Sijie Guo, Co-founder & CEO at StreamNative. “With Ursa, we’ve advanced Pulsar’s core data streaming engine across several dimensions, representing a major step toward fulfilling StreamNative’s mission of creating the world’s premier cloud-native data streaming platform.”
The URSA engine is available through ONE StreamNative Platform as of today. For more information visit, visit the materials listed below:
- URSA blog from Sijie Guo, Co-founder & CEO at StreamNative
- Lakehouse storage blog post
- Pulsar Virtual Summit EMEA 2004
Pulsar Virtual Summit Europe 2024
Today’s announcement is made in tandem with the Pulsar Virtual Summit Europe 2024, an event dedicated to Apache Pulsar and the messaging and event streaming community. The conference gathers an international audience of CTOs, CIOs, developers, data architects, data scientists, Apache Pulsar committers and contributors, and the data streaming community, to share experiences, exchange ideas and knowledge.
Prior Pulsar Summit events have featured 200+ interactive sessions by tech leads, open-source developers, software engineers, and software architects from Cisco, Salesforce, Splunk, Verizon Media, Iterable, Yahoo! JAPAN, TIBCO, Intuit, and more. The conferences have garnered 3,000+ attendees around the globe, including attendees from top tech, fintech and media companies, such as AWS, Google, Uber, Microsoft, AMEX, Salesforce, Disney, and Paypal.
Explore AITechPark for the latest advancements in AI, IOT, Cybersecurity, AITech News, and insightful updates from industry experts!
