Research from Ascend.io reflects the shift toward a postmodern data stack as data teams seek to address tool sprawl, burnout, and workload inefficiencies
Ascend.io, the leader in data pipeline automation, today released the results from its fourth annual DataAware Pulse Survey, unveiling trends in data tool adoption, productivity rates, and automation’s central role in the post-modern data stack. While the number of data tools is increasing, productivity and workload capacity declines are putting a strain on data teams and prompting them to implement automation to work faster and smarter.
The number of tools is increasing, but team productivity does not track
Data teams now use five tools on average in their data stack and over two-thirds (68%) plan to maintain or increase this number in the next year. Despite this trend, data teams report stagnation in their overall capacity. Virtually all data professionals surveyed (95%) report being at or above their work capacity for the fourth year in a row, with data engineers reporting being significantly overworked more than three times as often as their peers in data analytics and data architecture. The majority of data engineers report spending 50% or more of their time just maintaining existing programs.
Nine percent of data engineers and 8% of data analysts actually report being less productive this year compared to last year. The data reveals why: despite the surge in tools, the average data professional spends an equivalent of two business days per week simply trying to access data to do their job. This reveals a significant shortcoming of their fragmented data pipeline technology which was intended to close this gap.
These insights reflect the immense pressure and inefficiencies data teams face at a time when corporate data has skyrocketed to an all time high, and is expected to double by 2026 according to IDC.
“The reality is that data teams are not equipped to contend with the brittle, manual data pipelines that are required to deliver analytics and AI projects today,” said Sean Knapp, co-founder and CEO of Ascend. “It’s no surprise that teams are feeling this strain as the number of tools they need to manually integrate and coordinate across steadily increases. Fortunately, there’s a solution in consolidated platforms like Ascend.io that can use metadata paired with an intelligent controller to automate up to 90% of the construction and maintenance of these pipelines.”
Data teams view automation as the answer
To address tool sprawl and strained capacity, data teams are overwhelmingly turning to automation. Ninety-one percent of teams have either implemented data automation technologies or plan to in the next 12 months, including a 20% increase in teams who are “very likely” to implement automation in the next year compared with the 2022 results. More than one-quarter (27%) report plans to adopt automation as their primary mitigation strategy for data team capacity issues, second only to a desire for continued hiring (28%).
Additionally, data professionals are incorporating generative AI into their workflows for tasks such as test automation (50%), code generation (45%) and documentation (42%).
Managers must serve as the bridge between data engineers and executives
The data shows misalignment between data engineers, who are overwhelmingly stretched thin, and executives, who experience the pain of brittle data pipelines in other ways. The survey reveals that executives are twice as likely to perceive basic tasks with data as taking too long and spend nearly six more hours per week than individual contributors simply trying to get access to data (17.8 hours vs. 12.2 hours). Both of these represent opportunities for middle management to help their leaders understand why brittle data pipelines cause basic tasks to take longer, and result in them having to do more work to find the required data. Executives were overwhelmingly supportive of plans to implement data automation technologies in the next 12 months (74%). Middle management has the opportunity to help the C-Suite understand how these automation projects affect their daily experiences with data.
Misalignment was also found in the following areas:
- Individual contributors prefer a reduction of tools, and are nearly twice as likely as executives to favor removing tools from their data stack (22% vs. 12%). Team leads were more than twice as likely as individual contributors to favor consolidating onto a single platform rather than removing functionality from the stack.
- Time to complete tasks is perceived differently, with executives twice as likely as individual contributors to perceive basic data tasks as taking too long (53% vs. 26%).
- Key performance indicators vary across the team, with executives three times as likely to rank presentations created, and one-and-a-half times more likely to cite a number of new dashboards created as their top impact measurement compared with individual contributors, who rank errors fixed and tickets closed as their top metrics.
- Individual contributors are the least bullish on AI: they are 200% less likely to believe AI will increase their impact on their organizations’ business outcomes compared to the rest of their teams.
“Managers and Directors are the connective tissue between the doers in the engine room and the planners in the C-Suite. This gap in alignment speaks to the need for them to do more to bridge between both worlds,” said Knapp. “They need to help executives understand why these things are hard, and where they can invest to help it go faster. They need to help their team understand the connection between errors & tickets and the overall objectives of the business.”
Visit AITechPark for cutting-edge Tech Trends around AI, ML, Cybersecurity, along with AITech News, and timely updates from industry professionals!
