If you’ve ever had the joy of watching 8-year-olds play soccer, you would've also witnessed a particular parent on the sidelines. Not content with simple encouragement, they bellow instructions to their child to do things that, frankly, the world’s greatest players might struggle with. The gap between where their child’s skills lie and where they want them to be is wildly misaligned.
It's similar to a phenomenon that I see when talking to organizations about their move to the cloud. So enamored are they with the potential that the cloud offers that they are blind to the reality of where their current infrastructure really is. Their desire to get to the cloud immediately is understandable, but in most cases -- like the parent who wants their child to play like Ronaldo -- it's totally unrealistic.
Quite simply, a combination of available skills, investment and ingrained processes means that moving to the cloud overnight isn’t going to happen, and, in some cases, a complete migration to the cloud is not realistic, practical or even desired. Accepting that while the move to a cloud-based environment is the goal, working in a hybrid environment for the short-, medium- or even long-term is a reality and therefore crucial to ensuring success. I believe that success is determined by one key objective that the business needs to benefit from: time to value, or TTV.
So, as you start your transition to the cloud, how can you improve the all-critical TTV? The answer lies in retooling your processes to take advantage of new technologies while also leveraging the best practices in agile data warehousing that your existing environment and historical approach have held you back from implementing. OK, I hear you saying, "I get the theory, but how do I turn this into practice?" Let me take a step back and explain.
As we know, the cloud has changed this dynamic by allowing us to only pay for what we need. Cloud-based infrastructure enables you to do a special project for a few months or maybe a proof-of-concept trial and simply fire up the necessary capacity for the duration of the project. Once it is complete, you can just shut down the machine(s). And when you don’t have to purchase new hardware that is only going to be used for a short period of time for a particular project, it is far easier to make a compelling business case.
Now with increased security and trust, we are shifting more critical workloads onto the cloud, but we still often spend more money than is needed. While storage is cheap, compute power is expensive and often virtual machines sit unused, racking up huge bills, which is money that can now be spent on growing the business elsewhere. People often talk of the cloud being expandable to grow with a company based on their needs, but its ability to contract so we only spend what we need to is equally, if not more, important.
If you plan on moving to the cloud, you should look to take advantage of the capabilities it provides, particularly in the area of elastic compute: the ability to scale up and down as workload demands. No longer do you need to purchase hardware based on the peak workload requirements (e.g., end-of-month processing, nightly batch ETL jobs); rather, you can increase and decrease the compute as needed and for the time it is needed. Overnight batch loads and month-end end processing can be accommodated by adding more compute capacity to perform the processing, and then, once completed, you shut down the unneeded capacity.
Despite the long list of advantages of cloud platforms from a sysadmin standpoint, however, you still have to build and manage that data warehouse. Cloud platforms are not going to help you there. Without automating the development and DevOps aspects of your data warehouse, you are only solving half of the TTV problem. Data warehouse automation software allows you to maximize your development resources by reducing the time it takes to design, develop, deploy and operate the data warehouse while also reducing the cost and risk associated with that development. Your resources can then spend their valuable time working with the business users to deliver new business value in a much shorter timeframe while letting the automation software do the tedious and time-consuming work of generating the code, using built-in best practices and industry standards to ensure that the code is optimized for the platform in use.
While finding data warehouse automation software that can work with cloud-native platforms such as Amazon Redshift, Azure SQL Data Warehouse or Snowflake is a must, many companies will find that a hybrid environment is going to be the norm for a period of time and, in some cases, forever. Therefore, in this scenario, it is critical to have software that can manage these hybrid environments seamlessly and treat them as a single logical data warehouse. This will allow you to manage your data warehouse ecosystem more effectively, manage the transition over time, control workloads and provide a consistent view of the system regardless of location of data. For hybrid environments, this will be a significant factor in improving TTV.