dbt now supports data transformation in Python, to help teams solve new types of problems with data
dbt Labs, the pioneer in analytics engineering, today announced during the keynote of dbt Labs’ Coalesce 2022, that it has added support for data transformation in Python to dbt. This will supplement existing SQL capabilities and allow data teams to tackle new categories of problems, including statistical analysis and predictive modeling. dbt users will now be able to deploy Python and SQL from within the same workflow without the need for additional infrastructure.
dbt has emerged as an industry standard for data transformation in the cloud. With this release, the 16,000 organizations using dbt today can leverage the Python capabilities available on major cloud data platforms. Snowflake’s Snowpark for Python and BigQuery’s Serverless Spark, as well as Databricks’ expansion into data analytics workflows with Databricks SQL, have elevated the role of Python in the modern data stack. dbt users will have the option of choosing the best language — SQL or Python — for the task at hand.
“Snowflake’s partnership with dbt Labs has been instrumental for modern analytics as we work towards enabling data teams to securely and collaboratively deploy SQL code to production,” said Torsten Grabs, Director of Product Management, Snowflake. “With dbt Labs’ introduction of Python models and Snowflake’s Snowpark for Python, joint customers can now effortlessly combine the power of SQL and Python for modern analytics, and benefit from the wealth of data processing innovation in the Python community. This will make it even easier for analytics, data engineering, and data science teams to be productive and collaborative in the Data Cloud.”
“We created dbt to harness the power of the modern cloud data platform and empower all data practitioners to participate in the data transformation process. Six years ago, that meant working exclusively in SQL, the native language of the warehouse,” said Tristan Handy, Founder and CEO of dbt Labs. “Today, with advancements across data platforms, we’re excited to bring the power and accessibility of dbt to a new set of data workloads.”
“dbt has proven to be a flexible workflow for BigQuery customers to manage and help drive their data transformations,” said Sudhir Hasbe, Sr. Director of Product Management from Google Cloud. “We’re proud to work with dbt Labs and offer support for Python processing in BigQuery so customers and the data community have even more ways to solve business challenges with data.”
“Data teams are adopting a data lakehouse for all of their analytics and AI use cases, leveraging multiple programming languages to solve their data challenges,” said Adam Conway, SVP of Products at Databricks. “That’s why we’re excited to build on our partnership with dbt Labs to bring Python capabilities to joint customers, offering dbt users access to not only SQL transformations but also the entire lakehouse ecosystem including ML.”
For teams that are doing analytics work today in SQL, the inclusion of support for Python unlocks important new capabilities, giving them the ability to:
- Do more through dbt: Data teams can now use Python to run advanced statistical analysis or create simple predictive models, to do things like predict churn or customer lifetime value, taking advantage of an enormous ecosystem of pre-built Python packages.
- Deploy Python and SQL code from a single place: With support for Python models in dbt, users no longer need to manage an additional, separate set of tooling to deploy Python code.
- Take advantage of your cloud data platforms: Users can take advantage of all that cloud data platforms supporting Python workloads have to offer, as part of the same dbt workflow they already know and love.
The potential impact of Python support in dbt extends far beyond its immediate value to data analytics teams. Today, there is a significant gulf between analytics teams and data science teams, resulting from their use of different tooling that frequently leads to differing assumptions. In the long term, support for Python in dbt opens the door for more data scientists to collaborate in the same tooling as their analytics counterparts, bridging that gap. This would mirror the way that dbt has reduced silos between analysts and data engineers over the past several years by providing them a shared framework for collaboration.
Visit AITechPark for cutting-edge Tech Trends around AI, ML, Cybersecurity, along with AITech News, and timely updates from industry professionals!
