Join us for an insightful webinar where we...
Big Data Analytics WhereScape RED
Big Data and Advanced Analytics using WhereScape RED
We at WhereScape have been making significant investments recently integrating our automation software WhereScape RED with big data platforms. There has been a lot of interest from customers who recognise the value that automation can bring to these powerful yet complex solutions.
We have recently implemented a financial forecasting solution at an enterprise customer to prove the value of big data technology within the organisation. The solution developed in partnership between WhereScape and the customer delivered multiple benefits:
- Delivered a high value forecasting model to the business
- Used big data technology to deliver a model where traditional relational technology failed
- Proved the value of big data technology within the organisation
- Highlighted the potential of big data technologies to solve new and interesting problems
Problem
The customer needed to generate an accurate set of key performance indicators each month. The process required large daily volumes (approx. 20 million rows per day) of customer and detailed product revenue data. Because of storage and processing limitations, data was only available for the current month which impacted the forecast accuracy.
Solution
To solve the storage and processing problem the customer decided to implement a Cloudera Hadoop big data platform to store full historical datasets, along with the WhereScape RED for Big Data Adaptor to enable data lake automation. The completed solution delivered the following functionality:
- An automated process to extract large volumes of daily transactions from source each day and save them to the Cloudera platform using Hive
- Based on the extracted data, a forecast model was built in Hive and SQL Server. This model was built using WhereScape RED via rapid, iterative development cycle
- Once the forecast numbers were prepared, an automated process saved the forecast as an incremental snapshot in Hive and refreshed Cloudera Impala for interactive querying via Tableau
Some of the key features / benefits of using WhereScape RED for Big Data are:
- Ability to transfer large amounts of data from source system in to Hive seamlessly
- Common metadata across the Extended Data Warehouse environment (Hive, SQL etc)
- Consistent tools for developers
- Easily generate DDL and ELT SQL for Hive with data movement using Sqoop
- Centralised audit and error logging
- Integrated documentation across the full environment (i.e. SQL Server and Hive)
- Automated and integrated scheduling and workflow engine across the full environment
Solution Overview
Outcome
The project sponsors (both Business and IT) were impressed with how easily and quickly WhereScape RED could deliver a solution to solve their big data problem. Now that data is easily accessible for several months, business stakeholders are excited about their ability to easily generate accurate forecasts in less than 60 minutes as opposed to several days. Financial analysts can also query the big data platform directly via Tableau to rapidly gain insight, without the need to wait for data to be transferred to a relational data warehouse and enterprise reporting suite.
IT have learned that the WhereScape Big Data Adaptor and Cloudera Big Data platform can be used to solve complex and valuable business problems. It was also proved that big data technologies from WhereScape and Cloudera are functional and robust, and means that that projects previously deemed impossible can be looked at again.
Technologies Used
To build this solution, following technologies were used:
- Oracle
- SQL Server
- WhereScape RED
- WhereScape RED Big Data Adaptor
- Cloudera Big Data Platform
- Hive
- Sqoop
- Impala
About WhereScape RED for Big Data
WhereScape RED is data warehouse and big data automation software for building, deploying and renovating your analytic solutions whatever the size of your data. WhereScape RED sets the standard for delivery speed using familiar industry standards, frameworks and best practices to dramatically accelerate time to value.
WhereScape RED customers are able to fully manage their Apache Hive™ big data environments through the WhereScape RED data automation platform. This centralises development of the entire decision support infrastructure into one integrated platform and toolset.
There is no need to license separate ETL, data integration, or data modelling tools because WhereScape RED supports industry standard SQL. Customers can leverage their existing resources and training rather than having to rely on tool or platform-specific expertise.
Experience the Power of WhereScape 3D 9.0.3: New Features and Improvements
We’re thrilled to introduce our latest iteration of WhereScape 3D! Version 9.0.3 brings a host of new features and enhancements designed to make your data warehousing journey smoother, faster, and more efficient. Let’s dive into the details of what you can expect from...
Ahead of the Curve: Future Trends in Data Automation and WhereScape’s Pioneering Solutions
The Evolving Landscape of Data Automation As new technologies emerge and existing tools constantly change and improve, the world of data automation transforms rapidly. Even the most well-versed data teams find themselves disoriented and overwhelmed in the face of...
Investing in Data Automation: A Strategic Approach to Business Growth
Unlocking Growth: The Strategic Advantage of Data Automation Organizations reaping the benefits of data automation stay ahead of industry trends and improve the efficiency of their operations and decision-making. Data automation tools offer a strategic advantage for...
Data + AI Summit 2024: Key Takeaways and Innovations
The Data + AI Summit 2024, hosted by Databricks at the bustling Moscone Center in San Francisco, has concluded with remarkable revelations and forward-looking innovations. Drawing over 16,000 attendees in person and virtually connecting over 60,000 participants from...
WhereScape RED 10.1 is Here: Enhanced Scheduling and Customization
We’re proud to announce the highly anticipated WhereScape RED 10.1 is now available, and it’s packed with exciting new features and enhancements designed to make your data warehousing experience more efficient and enjoyable. Let's take a closer look at what’s new and...
Supercharging Data Integration: The WhereScape and Databricks Advantage
The demand for robust data management systems has never been higher, and Databricks has quickly become a favored choice for cloud-based solutions. Its powerful capabilities make it a top contender for managing large-scale data, but when combined with WhereScape's...
Empowering Customer Success: WhereScape’s Comprehensive Support and Training Resources
Enhancing Operational Success with WhereScape’s Support Systems At WhereScape, we understand that a data warehouse is only useful to the extent that it is understood. In order to drive your organization closer to your key goals and objectives, you need full mastery of...
Revolutionizing Day-to-Day Operations: The Power of Automated Data Integration
The Transformational Role of Automation in Data Management Across industries and business stages, organizations of all types manage data in their daily operations. Whether that data entails patient appointments and reminders in a healthcare clinic, student performance...
Gartner® Insights: Microsoft Fabric as a Unified Data & Analytics Platform
Are you ready to revolutionize your data management strategy with a platform that promises to simplify and enhance your operations? According to a Gartner poll, 43% of respondents believe that the data and analytics ecosystem will significantly influence their choice...
WhereScape and YellowFin Attending World of Data in Munich
We are excited to announce that WhereScape and YellowFin will be attending the World of Data conference in Munich on June 6, 2024. This event will bring together data professionals, industry leaders, and technology enthusiasts from around the globe to explore the...
Related Content
Experience the Power of WhereScape 3D 9.0.3: New Features and Improvements
We’re thrilled to introduce our latest iteration of WhereScape 3D! Version 9.0.3 brings a host of new features and enhancements designed to make your data warehousing journey smoother, faster, and more efficient. Let’s dive into the details of what you can expect from...
Ahead of the Curve: Future Trends in Data Automation and WhereScape’s Pioneering Solutions
The Evolving Landscape of Data Automation As new technologies emerge and existing tools constantly change and improve, the world of data automation transforms rapidly. Even the most well-versed data teams find themselves disoriented and overwhelmed in the face of...
Investing in Data Automation: A Strategic Approach to Business Growth
Unlocking Growth: The Strategic Advantage of Data Automation Organizations reaping the benefits of data automation stay ahead of industry trends and improve the efficiency of their operations and decision-making. Data automation tools offer a strategic advantage for...
Data + AI Summit 2024: Key Takeaways and Innovations
The Data + AI Summit 2024, hosted by Databricks at the bustling Moscone Center in San Francisco, has concluded with remarkable revelations and forward-looking innovations. Drawing over 16,000 attendees in person and virtually connecting over 60,000 participants from...