Simplified connection management, improved upgrade support, and cost-savings enhancements for Apache Airflow empower organizations to accelerate data, analytics and AI
Astronomer, the leader in modern data orchestration, today announced the latest Astro platform release to streamline the data pipeline journey and help teams unleash the power of their data. This release features a suite of powerful enhancements designed to secure and optimize data platforms, simplify connectivity, and empower teams to leverage the full potential of their data. Three key features introduced in this release include a Connection Management system, new tools to upgrade with confidence including Upgrade Utilities and Deployment Rollbacks, and Scale-to-Zero Development Deployments.
As data grows in complexity and volume, distributed across numerous teams and applications, the imperative for enhanced security and increased efficiency in modernizing data workflows has never been greater. The ability to effectively streamline data pipelines and processing is crucial for organizations aiming to fully leverage AI and ML capabilities. Astro’s latest platform is designed to meet these challenges, offering data engineers and teams robust security measures and optimized processes.
“As the data landscape becomes increasingly dynamic, data teams rely on data pipelines to ultimately materialize & deliver data assets on time –– a critical piece of effective AI and LLM strategy,” said Pete DeJoy, Senior Vice President of Product at Astronomer. “Today’s platform updates, including connection management, rollbacks, and scale-to-zero environments enable teams to unlock the power or Airflow with improved developer productivity, security, and cost efficiency.”
New innovations in the latest Astro Platform include:
- Connection Management: A cornerstone of this release, Connection Management changes how businesses handle securing credentials to critical data services. An Astro-native managed secrets backend, Connection Management centralizes connection setup to reduce operational complexity, enhance security, and improve efficiency. With an intuitive built-in inheritance model, Connection Management allows secure sharing of connections across deployments, reducing setup time and providing flexibility. Its impact on accelerating onboarding processes and ensuring flexible, secure data management is a game-changer for businesses looking to scale.
- Upgrade Airflow with Confidence: Deployment Rollbacks and Upgrade Utilities: Addressing a critical need in data management, the Deployment Rollback feature provides businesses with the assurance to upgrade Airflow environments confidently. Coupled with Upgrade Utilities that proactively test for compatibility issues, these enhancements minimize business disruptions, ensuring continuous and secure data operations.
- Scale to Zero Development Deployments: In a move to optimize operational costs, the Scale to Zero Development Deployment feature enables businesses to effectively manage resources. In a user-specified timeframe, all components of the Development Deployment will be scaled down to zero – from the Airflow system components to scheduler to workers – and incur zero costs. By allowing deployments to scale down to zero during non-peak periods, this feature presents a cost-effective solution for managing development environments, directly impacting the bottom line.
“During an Airflow upgrade at Zepz, we were able to easily revert when conflicts arose,” said Jafar Amin, Senior Data Engineer at Zepz. “Knowing that we can easily revert has given us more confidence to upgrade Airflow.”
Visit AITechPark for cutting-edge Tech Trends around AI, ML, Cybersecurity, along with AITech News, and timely updates from industry professionals!
