2022Big DevBuild Times

We launched this survey with a single question in mind.
What is the true state of the software build process today, and what impact does it have on the broader business?
To uncover the answer, we spoke to respondents across a wide array of industries, roles, and company sizes, and with varied amounts of hands-on vs managerial experience. We asked them whether they suffer from long
build times, and whether this problem is getting worse year over year. We delved into the frustrations they face, the technologies they use, and how much time and
productivity they could regain if they had the right processes and tools in place.
The results reveal the nature of the build experience for today’s enterprises and paint a clear picture of how organizations can translate faster build times into true ROI.

Methodology

To collect these insights we commissioned a survey of 300 senior managers or above from North America, APAC, and the EU in companies with 1,000 employees or more that implement C++ software projects. The survey was completed by Global Surveyz, an independent survey company, and took place during Q4 2021. The titles of our respondents ranged from CTOs to software engineers and architects, as well as firmware engineers, embedded systems, build/release developers or those in configuration management.
The respondents were recruited through a global B2B research panel and invited via email to complete the survey. The average amount of time spent on the survey was 5 minutes.
The answers to the majority of the non-numerical questions were randomized to prevent order bias in the answers.

Running a build or a CI/Release build takes its toll. Companies are spending 20 minutes on average to run each build (figure 1) and 23 minutes to run a CI/Release build (figure 2).
69% of companies are spending 10 or more minutes to run a build, and 24% of companies are spending more than 30 minutes (figure 1).

82% of companies are spending 10 or more minutes to run a CI/Release build, and 32% of companies are spending more than 30 minutes (figure 2).
Instead of being able to build quickly, test and get results, builds are taking a long time, slowing down the whole development cycle and having a direct impact on productivity.

98% of respondents admitted they waste time waiting for builds to finish (figure 3). The average time spent waiting for builds to finish is just under an hour at 57 minutes (figure 4). It varies by industry, with defense/military holding the slowest build times (72 minutes) and banking the fastest build times (38 minutes).
However, even 38 minutes is a lot of wasted time in terms of productivity. In many cases developers need to wait for builds to finish before they can move onto the next task, so this wait is literally time down the drain.

96% of survey respondents said build time increased by an average of 15.9% in 2021 compared to 2020 (figure 5).
This can probably be attributed to increase in codebase size.
Only 4% reported that build time decreased. The top reasons they give for the decrease are better hardware (44%) and a decrease in codebase size (33%). Better hardware might seem like a quick fix, but it is a very costly solution which organizations will pay for whether it’s utilized or not. It’s also not sustainable, as a team can’t keep adding more hardware within a single machine to meet growing needs.
Decreasing the codebase size might be helpful at first, but will provide diminishing returns over time, and have marginal impact once an organization has made initial fixes.

The most used technologies and methodologies for accelerating or improving the development cycle are dev in the cloud (49%), static code analysis (46%) and dynamic code analysis (28%). Just 8% of survey respondents have no plans to use dev in the cloud, likely because of security/compliance issues.
About one quarter (23%) of respondents plan to start using build acceleration in 2022.

Slow build times are a problem for 91% of companies. Only 33% said this isn’t a big problem and 58% said it’s a significant problem or a problem to some extent.
When comparing by industry (figure 9), defense/military suffers the most from slow build times (73%), followed by IT services (67%) and automotive (63%).

97% of survey respondents report that they experience frustrations in their day-to-day work.
The most frustrating issues for respondents are analyzing crashes from production (35%), learning new technologies (35%), and long build times (31%). While crashes and new technologies are somewhat necessary evils, long build times can be solved with the right technology.

Static code analysis is one method of reducing crashes, but this adds time to builds, which is likely to increase frustration overall.
This requires a solution since adoption of both static and dynamic code analysis are growing in the market (figure 7) as methods to improve testing and quality.

The top time-wasters are creating internal documentation that no one will read (82%), unnecessary changes in requirements (78%), and slow builds (69%). Also on the list are slow tests, highlighted by 67% of respondents. Slow builds and slow testing feed into one another. If a company can speed up builds, they can speed up tests.

97% of respondents admit they have areas they wish they could improve in their day-to-day work which would save on development time.
The top requested areas are moving to new build systems (35%), optimizing build scripts (31%), and being able to accelerate their build times (28%). All of these top requests are aligned with improving build times, showing the significance of this need.

91% of survey respondents have hundreds of thousands of lines of code or more. 76% have millions of lines of code or more (figure 13).
Codebases are expected to grow by 18.6% in 2022. This will grow most at large companies with 5K+ employees (21.6%), while in companies of up to 5K employees, it is expected to grow 16% (figure 14). Bigger companies face bigger problems.
As codebases grow, build times will get even longer.

95% of survey respondents indicated that with the right tools and processes, they can save (average weight) 65 minutes per day. Based on 261 working days per year (in 2021) and 8 working hours per day, that translates into approximately 36 days per year. Translating this into cost savings, 36 days is almost $20,000 per year if we take $60/hour as a typical dev cost. Productivity in this area could realize serious ROI even for those who say it will only save 30 minutes or an hour per day. Saving two hours per day would mean a company could add another full-time developer to its team.
Faster builds also allow the adoption of best practices such as feature flagging, working in parallel on a few code branches and more, and allow for faster bug resolution, faster time to market and better customer success opportunities – all of which impact an organization’s bottom line.

When looking at the impact on productivity of having the right tools and processes by role, we see that while the average time saved for all respondents is 65 minutes per day. The number grows by 30% when it comes to R&D, and the right tools and processes will save them 85 minutes per day.
In DevOps and R&D there are much higher issues with recruitment, so when translating the time savings to cost savings (figure 16), it’s clear to see how impactful the right tools and processes could be on these teams.

The cloud is already used throughout the entire development lifecycle. The top stages for using the cloud overall are in development (43%) and deployment (42%). When looking at the industry breakdown, banking has the highest rate of cloud usage. When we connect cloud usage to slow build times, we can see a direct correlation to costs. Time-consuming builds cost much more on the cloud in terms of compute utilization.
In some industries, it’s impossible to use money-saving options such as spot instances, and therefore costs can spiral quickly. Organizations need to look for ways to keep cloud costs low while optimizing for faster builds.