Join us for an insightful webinar where we...
BI Built to Order, On-Demand: Automating Data Warehouse Delivery
This week, Dr. Barry Devlin published a provocative new paper on data warehouse automation – “BI Built to Order, On-Demand: Automating Data Warehouse Delivery.”
You can grab it here, if you’re curious. And you should be. Because in the paper Devlin does two things: first, he considers a few Inconvenient Truths about how data warehouses are built and managed – or misbuilt and mismanaged – and, second, he makes the case for data warehouse automation as a common-sense fix for today’s often mismanaged data warehouse development.
When Devlin described his vision for the business information system he called a “data warehouse” – back in early 1988 – we just didn’t have the tools to efficiently design, build, and manage warehouse systems. Everything, or almost everything, had to be done by hand: there weren’t any ETL tools, data integration suites, studios, platforms or workbenches. But even once we got primitive versions of these tools – starting in 1993 or thereabouts – things didn’t magically get better. In fact, by 2003, we were already starting to come to grips with the empirical fact that data warehouse projects took too long to build, failed to deliver on many of the promises Devlin had outlined in his paper, and, most important, were too hard to change. We know: WhereScape-the-company grew out of the integration experiences of our founders, who specialized in fixing just these problems.
But a great point that Devlin makes is that most of these problems were byproducts of what might be called an “out of phase” development process. Simply put: building data warehouse systems was and to some degree still is a disintegrated affair. In larger organizations, it is performed by separate teams or groups of developers, each working with their own set of tools, each using their own methodology, and each building at their own pace. According to Devlin, this is one of the biggest impediments to traditional analytic development.
“Modeling, database design and development of population routines required multiple, disconnected iterations involving business users, modelers, database administrators and ETL programmers at different times, each using different and unconnected tools. These gaps and tool transitions slowed the process and gave rise to design errors and inconsistencies,” Devlin writes.
The upshot is that this model compromises both the consistency of data and the timeliness of application delivery. Devlin sees data warehouse automation software – which centralizes data warehouse and analytical development in a single tool – promoting an iterative, agile development methodology, and implements a shared metadata repository – as a prescriptive Rx for this problem.
“The common environment and shared metadata repository offered by data warehouse automation overcomes this … by integrating the design and delivery of the data model, database structure, and the population process in one place – whether for a warehouse or mart,” he writes. “All the design and population metadata is stored together in a single repository, allowing development to flow smoothly and iteratively from user requirements, through database design, to creation of population routines. By integrating all the steps of the design and development process, consistent and quality data can be delivered quickly to the business for immediate review and early acceptance.”
Data warehouse automation software isn’t a turnkey fix. Devlin recognizes this. All the same, it’s a way to eliminate out-of-phase development, centralize the development process, and enforce a consistent, delivery-focused development paradigm. It gives you a solid foundation on which to build your data warehouse. Data warehouse automation software has other benefits that aren’t at all confined strictly to development. As Devlin notes, it promotes collaboration between business and IT, making it possible to produce data-driven – or business-data-driven – apps.
I’ll say more about this in a follow up post.
Experience the Power of WhereScape 3D 9.0.3: New Features and Improvements
We’re thrilled to introduce our latest iteration of WhereScape 3D! Version 9.0.3 brings a host of new features and enhancements designed to make your data warehousing journey smoother, faster, and more efficient. Let’s dive into the details of what you can expect from...
Ahead of the Curve: Future Trends in Data Automation and WhereScape’s Pioneering Solutions
The Evolving Landscape of Data Automation As new technologies emerge and existing tools constantly change and improve, the world of data automation transforms rapidly. Even the most well-versed data teams find themselves disoriented and overwhelmed in the face of...
Investing in Data Automation: A Strategic Approach to Business Growth
Unlocking Growth: The Strategic Advantage of Data Automation Organizations reaping the benefits of data automation stay ahead of industry trends and improve the efficiency of their operations and decision-making. Data automation tools offer a strategic advantage for...
Data + AI Summit 2024: Key Takeaways and Innovations
The Data + AI Summit 2024, hosted by Databricks at the bustling Moscone Center in San Francisco, has concluded with remarkable revelations and forward-looking innovations. Drawing over 16,000 attendees in person and virtually connecting over 60,000 participants from...
WhereScape RED 10.1 is Here: Enhanced Scheduling and Customization
We’re proud to announce the highly anticipated WhereScape RED 10.1 is now available, and it’s packed with exciting new features and enhancements designed to make your data warehousing experience more efficient and enjoyable. Let's take a closer look at what’s new and...
Supercharging Data Integration: The WhereScape and Databricks Advantage
The demand for robust data management systems has never been higher, and Databricks has quickly become a favored choice for cloud-based solutions. Its powerful capabilities make it a top contender for managing large-scale data, but when combined with WhereScape's...
Empowering Customer Success: WhereScape’s Comprehensive Support and Training Resources
Enhancing Operational Success with WhereScape’s Support Systems At WhereScape, we understand that a data warehouse is only useful to the extent that it is understood. In order to drive your organization closer to your key goals and objectives, you need full mastery of...
Revolutionizing Day-to-Day Operations: The Power of Automated Data Integration
The Transformational Role of Automation in Data Management Across industries and business stages, organizations of all types manage data in their daily operations. Whether that data entails patient appointments and reminders in a healthcare clinic, student performance...
Gartner® Insights: Microsoft Fabric as a Unified Data & Analytics Platform
Are you ready to revolutionize your data management strategy with a platform that promises to simplify and enhance your operations? According to a Gartner poll, 43% of respondents believe that the data and analytics ecosystem will significantly influence their choice...
WhereScape and YellowFin Attending World of Data in Munich
We are excited to announce that WhereScape and YellowFin will be attending the World of Data conference in Munich on June 6, 2024. This event will bring together data professionals, industry leaders, and technology enthusiasts from around the globe to explore the...
Related Content
Experience the Power of WhereScape 3D 9.0.3: New Features and Improvements
We’re thrilled to introduce our latest iteration of WhereScape 3D! Version 9.0.3 brings a host of new features and enhancements designed to make your data warehousing journey smoother, faster, and more efficient. Let’s dive into the details of what you can expect from...
Ahead of the Curve: Future Trends in Data Automation and WhereScape’s Pioneering Solutions
The Evolving Landscape of Data Automation As new technologies emerge and existing tools constantly change and improve, the world of data automation transforms rapidly. Even the most well-versed data teams find themselves disoriented and overwhelmed in the face of...
Investing in Data Automation: A Strategic Approach to Business Growth
Unlocking Growth: The Strategic Advantage of Data Automation Organizations reaping the benefits of data automation stay ahead of industry trends and improve the efficiency of their operations and decision-making. Data automation tools offer a strategic advantage for...
Data + AI Summit 2024: Key Takeaways and Innovations
The Data + AI Summit 2024, hosted by Databricks at the bustling Moscone Center in San Francisco, has concluded with remarkable revelations and forward-looking innovations. Drawing over 16,000 attendees in person and virtually connecting over 60,000 participants from...