Join us for an insightful webinar where we...
Gartner Data Warehouse Automation
![documentation-blog Agile Documentation through Automation](https://www.wherescape.com/wp-content/uploads/2021/02/documentation-blog.jpg)
Gartner Magic Quadrant Data Warehouse
The fact that WhereScape does not fit exactly in any one category of the Gartner Magic Quadrants, and in fact spans several categories as an end-to-end Data Automation tool, meant that our software did not get too much coverage from the leading analyst voice in IT.
This changed recently with the publication of a paper entitled Automating Data Warehouse Development, written by Gartner analyst Henry Cook (Gartner subscribers can download the full report from their website here). We feel this publicity reflects a shift in the data world from seeing automation as a nice to have, to a necessity for companies’ ability to compete in the digital age.
Henry’s technical background means he can describe the gains Data Warehouse Automation (DWA) creates from a business angle, but also go into granular detail about what DWA means for developers day to day.
Data Warehouse Automation
“Few companies can implement all their potential requirements — limits on resources typically preclude this. Therefore, a DWA tool is often used as a productivity enhancer, which helps to meet more requirements and thus deliver more benefits. The dependence on external staff — such as contractors — can also be reduced through the use of DWA.
“Developers gain agility through the ability to generate code, and — equally important — the ability to regenerate it to make changes. Aside from generating code for the original state and for the new state, the DWA tool can also generate the special code to move from one state to the other.
“For example, consider the process of adding columns to a presentation layer data mart. The DWA tool generated the data description for the original mart, then the ETL to load it. Done manually, the developers will also need to write some special, one-off code to unload the mart, reshape it, and then reload the data in its new form, merging in data for the new columns. The DWA can automatically generate this transitional code too. This is faster and much more productive than having to do this manually. Over the lifetime of a DW, the developers will repeatedly perform this type of process. The principle is to build once and then repeatedly reuse.”
Later in the report, Henry describes how DWA affects the Data Warehouse Development lifecycle in the long-term, not just in its initial development.
“On the left-hand side of Figure 3, the DWA developers are defining the system and the linkages between the components in metadata.
“Once defined, the DWA tool will then automatically generate all the components shown on the right. Generating those components manually, then integrating and testing them (and doing this repeatedly) will require significantly more effort than using the DWA tool. Organizations using these tools commonly see productivity gains of 400%. Of course, productivity gains will vary depending on the tool and the nature of the system.
“Developers can change metadata and then quickly deploy the generated changes in the subsequent automated routines. This means that your development staff can devote the time that they had allocated for administrative work to address much more valuable and leveraged design work. Equally importantly, developers are much less likely to introduce mistakes, since the automatic generation of changes is always exact.”
ETL Automation vs Data Warehouse Automation
A few years ago, the only way to quickly explain Data Automation was to compare it to an ETL tool, which is not a straight comparison given the holistic reach of tools such as WhereScape. WhereScape extracts, transforms and loads but in a different order. ELT uses the power of modern databases to transform, often using MPP to complete the process in a fraction of the time of traditional ETL tools. So data is loaded 5-6x faster than with an old-school ETL tool, but this is just the tip of the iceberg.
Ten or even five years ago we had to really explain what we do from scratch, but now the conversations we have with data professionals are changing. The industry is much more educated on the role automation has to play in the development process. Stats of the leaps forward in productivity companies are making catch the eye of IT and business staff at all levels looking to do more with fewer resources.
Data Automation isn’t just a tool to plug in and forget about. It’s not just a quick fix for a small problem. It is a conduit for data warehouse modernization and a new way of working altogether. Recently a customer told us that, “WhereScape will force you to think differently,” (Will Mealing, Head of Data & Analytics at Legal & General – read the full case study here). “Your acceleration is huge once you rework how you historically have done things.”
WhereScape Data Automation
WhereScape software has been around since the turn of the century and was designed as an automated code generator and integrated development environment to take the repetitive manual work out of the 90s ETL coding methodologies. It enables you to design and build data infrastructures with a drag and drop GUI, then writes 100s of lines of codes in seconds and documents everything at the click of a button. WhereScape merely sits on top of your existing infrastructure and writes the SQL code to make it fit together faster and more efficiently so developers can focus on higher-value work.
Some ambitious companies have been automating their data infrastructure for years and achieving rapid time to value. But only now, 20 years later, are the majority of companies catching up and DWA is quickly becoming the norm. The days of IT staff sitting in siloed offices, staring at screens all day through to evenings and weekends are gone. Today we have stand up meetings, SCRUM sessions and sprints, then let automation do the grunt work because we don’t have to anymore.
Torgil Hellman, Chief Architect at Atea Sweden – watch the video here, talks about life BA (before automation) and AA (after automation). Before WhereScape he used to spend 90% of his working life at a screen and 10% collaboration with the business, now it is the other way around. He doesn’t have to wake up early when he’s on holiday to check his emails because he trusts the automation tool.
Data Vault Automation
Data Vault has gone from the latest fad to a mainstream modeling style in just a few years. The agility it enables has endless implications for new streams of actionable data, but its complexity needs automation to implement and maintain. Today around half of our new customers are doing Data Vault, and it could be said that this is pulling DWA into the mainstream. Many customers also want to migrate data and architecture onto Cloud platforms for further agility, such as Aptus Health who operate a Data Vault on Snowflake.
In order for companies to succeed today, the puzzles and firefighting that accompany a continued reliance on hand coding must be removed. Modern data architectures are complex enough without also having to deal with problems arising from outdated 1990s methodologies. Only then can companies focus forward on innovation and agile collaboration to supply reliable data to the business at the right time.
Once lost between categories of data tools that address very specific challenges, end-to-end data warehouse automation tools such as WhereScape are now finally getting recognition from top analysts like Gartner, and we hope to bring many more reports like Henry’s to your attention over the next couple of years.
WhereScape Data Automation
WhereScape eliminates the risks in data projects and accelerates time to production to help organizations adapt better to changing business needs. Book a demo to see what you can achieve with WhereScape.
Experience the Power of WhereScape 3D 9.0.3: New Features and Improvements
We’re thrilled to introduce our latest iteration of WhereScape 3D! Version 9.0.3 brings a host of new features and enhancements designed to make your data warehousing journey smoother, faster, and more efficient. Let’s dive into the details of what you can expect from...
Ahead of the Curve: Future Trends in Data Automation and WhereScape’s Pioneering Solutions
The Evolving Landscape of Data Automation As new technologies emerge and existing tools constantly change and improve, the world of data automation transforms rapidly. Even the most well-versed data teams find themselves disoriented and overwhelmed in the face of...
Investing in Data Automation: A Strategic Approach to Business Growth
Unlocking Growth: The Strategic Advantage of Data Automation Organizations reaping the benefits of data automation stay ahead of industry trends and improve the efficiency of their operations and decision-making. Data automation tools offer a strategic advantage for...
Data + AI Summit 2024: Key Takeaways and Innovations
The Data + AI Summit 2024, hosted by Databricks at the bustling Moscone Center in San Francisco, has concluded with remarkable revelations and forward-looking innovations. Drawing over 16,000 attendees in person and virtually connecting over 60,000 participants from...
WhereScape RED 10.1 is Here: Enhanced Scheduling and Customization
We’re proud to announce the highly anticipated WhereScape RED 10.1 is now available, and it’s packed with exciting new features and enhancements designed to make your data warehousing experience more efficient and enjoyable. Let's take a closer look at what’s new and...
Supercharging Data Integration: The WhereScape and Databricks Advantage
The demand for robust data management systems has never been higher, and Databricks has quickly become a favored choice for cloud-based solutions. Its powerful capabilities make it a top contender for managing large-scale data, but when combined with WhereScape's...
Empowering Customer Success: WhereScape’s Comprehensive Support and Training Resources
Enhancing Operational Success with WhereScape’s Support Systems At WhereScape, we understand that a data warehouse is only useful to the extent that it is understood. In order to drive your organization closer to your key goals and objectives, you need full mastery of...
Revolutionizing Day-to-Day Operations: The Power of Automated Data Integration
The Transformational Role of Automation in Data Management Across industries and business stages, organizations of all types manage data in their daily operations. Whether that data entails patient appointments and reminders in a healthcare clinic, student performance...
Gartner® Insights: Microsoft Fabric as a Unified Data & Analytics Platform
Are you ready to revolutionize your data management strategy with a platform that promises to simplify and enhance your operations? According to a Gartner poll, 43% of respondents believe that the data and analytics ecosystem will significantly influence their choice...
WhereScape and YellowFin Attending World of Data in Munich
We are excited to announce that WhereScape and YellowFin will be attending the World of Data conference in Munich on June 6, 2024. This event will bring together data professionals, industry leaders, and technology enthusiasts from around the globe to explore the...
Related Content
Experience the Power of WhereScape 3D 9.0.3: New Features and Improvements
We’re thrilled to introduce our latest iteration of WhereScape 3D! Version 9.0.3 brings a host of new features and enhancements designed to make your data warehousing journey smoother, faster, and more efficient. Let’s dive into the details of what you can expect from...
Ahead of the Curve: Future Trends in Data Automation and WhereScape’s Pioneering Solutions
The Evolving Landscape of Data Automation As new technologies emerge and existing tools constantly change and improve, the world of data automation transforms rapidly. Even the most well-versed data teams find themselves disoriented and overwhelmed in the face of...
Investing in Data Automation: A Strategic Approach to Business Growth
Unlocking Growth: The Strategic Advantage of Data Automation Organizations reaping the benefits of data automation stay ahead of industry trends and improve the efficiency of their operations and decision-making. Data automation tools offer a strategic advantage for...
Data + AI Summit 2024: Key Takeaways and Innovations
The Data + AI Summit 2024, hosted by Databricks at the bustling Moscone Center in San Francisco, has concluded with remarkable revelations and forward-looking innovations. Drawing over 16,000 attendees in person and virtually connecting over 60,000 participants from...