Join us for an insightful webinar where we...
Agile Documentation through Automation
![documentation-blog Agile Documentation through Automation](https://www.wherescape.com/wp-content/uploads/2021/02/documentation-blog.jpg)
Documentation is an essential component of any sustainable data architecture, yet for practical reasons, it is often neglected by teams still working with manual processes. Developers are under constant pressure to meet deadlines and are judged more on the code they produce than how well they document it.
The Need for Automated Documentation
Pseudo Agile strategies are now common in data warehousing, and we often hear of teams working in sprints with SCRUM meetings and so on. However, most developers still type out every line of code by hand and work with 90s ETL technology. While the deadlines and iterations they have to meet are set on Agile timeframes, the tools and methodologies they use are not Agile, which puts them at a huge disadvantage even before they start. It is rare that sufficient consideration is given to documentation when sprints are defined. Under these constraints, developers will always be under stress and need to take shortcuts to deliver code. Documentation is always the first thing to be neglected or dropped completely.
The need to write documentation inhibits productivity in a number of ways. Firstly, if you take regular breaks to document what you have done, you break your flow and write less code. If you leave it until you have finished, you forget what you have done. Even leaving it until a Monday to document last week’s code means you will forget a decent amount of what you did. The more productive you were, the more you have to forget.
Data Automation for Sustainable Architectures
In the short-term world of delivering code just in time for the next deadline, documentation is seen as a nice-to-have and may not even be examined by those responsible for data governance. It is only further down the line, when our architectures are suffering from clutter and are hard to maintain, or one of the developers leaves the company and someone comes in to replace them, that we realize the value of accurate, standardized documentation.
Data Automation enables us to be flexible and switch target databases much easier than in the past. A choice of database is no longer a ten-year decision. This also means that we no longer have to rip up and start again if we want to switch modeling styles. The code we write now can be tweaked depending on needs, and can be sustainable for many years. For this reason, it needs to be reliable to stand the test of time, which also applies to how it is documented. Architectures need to supersede individuals and databases, so when one developer leaves, the code must pass on to colleagues seamlessly.
Automated Documentation Benefits
WhereScape removes the documentation issue from a developer’s schedule entirely, enabling them to focus on higher-value tasks and meet Agile deadlines without cutting corners. All they have to do is click one button and all their work will be documented to a level of detail that would take many hours to do by hand.
WhereScape is metadata-driven. The WhereScape GUI is a simplified manifestation of all the metadata that sits behind it and makes the data warehouse work as it is shown to. This out-of-the-box documentation feature means that each action taken by users and every item and structure within the architecture is stored, such as:
- The code itself.
- Lists of columns and objects, who created them, whether they are included in certain jobs and so on.
- Transformations.
- Lineage backwards and forwards (where did data come from and where did it go after this point?).
- Data types and all information about the current object you are looking at.
- Interactions between the various objects within the architecture.
WhereScape documentation is a roadmap that goes back through the project you have been working on, providing hyperlinks to every stage of the process so you can click in and see its code and structure. All the documentation that developers should be writing by hand, but often don’t for the various reasons outlined above, is generated automatically at a much higher level of detail. To do this for an entire architecture would be months of work; too much to do whilst maintaining productivity in writing code.
Improve Developer Productivity
Such comprehensive documentation means bugs can be caught and fixed much faster. Rather than having to go through badly documented code by hand to find the fault, WhereScape highlights faulty code in red. As well as finding the problem quickly, this prevents you from working on good code that is mistakenly diagnosed as incorrect.
“The turnaround of our bugs has been incredible. It takes two hours now where it used to take us two weeks. The autonomy the developers have over deployments means a quicker turnaround for performing re-tests.” – James Gardiner, Data Warehouse Technical Lead at Admiral Group.
When discussing Data Automation, it is important to remember that human creativity is still paramount. It is augmented by automation tools that eradicate repetitive manual work, enabling developers’ brains to be less fatigued and so more creative. However, in many areas of data warehousing, creativity and flair are undesirable and even harmful. Documentation is exactly the kind of mundane, repetitive work suited to automation. All we need is a reliable account of what was done in a language that other developers can understand, nothing more, nothing less.
“The documentation is generated which saves us a lot of time and it’s always accurate.” – Marleen Gerbraad, Delivery Manager Datalogistics at Rabobank.
See our automated documentation capabilities in action in this video.
Experience the Power of WhereScape 3D 9.0.3: New Features and Improvements
We’re thrilled to introduce our latest iteration of WhereScape 3D! Version 9.0.3 brings a host of new features and enhancements designed to make your data warehousing journey smoother, faster, and more efficient. Let’s dive into the details of what you can expect from...
Ahead of the Curve: Future Trends in Data Automation and WhereScape’s Pioneering Solutions
The Evolving Landscape of Data Automation As new technologies emerge and existing tools constantly change and improve, the world of data automation transforms rapidly. Even the most well-versed data teams find themselves disoriented and overwhelmed in the face of...
Investing in Data Automation: A Strategic Approach to Business Growth
Unlocking Growth: The Strategic Advantage of Data Automation Organizations reaping the benefits of data automation stay ahead of industry trends and improve the efficiency of their operations and decision-making. Data automation tools offer a strategic advantage for...
Data + AI Summit 2024: Key Takeaways and Innovations
The Data + AI Summit 2024, hosted by Databricks at the bustling Moscone Center in San Francisco, has concluded with remarkable revelations and forward-looking innovations. Drawing over 16,000 attendees in person and virtually connecting over 60,000 participants from...
WhereScape RED 10.1 is Here: Enhanced Scheduling and Customization
We’re proud to announce the highly anticipated WhereScape RED 10.1 is now available, and it’s packed with exciting new features and enhancements designed to make your data warehousing experience more efficient and enjoyable. Let's take a closer look at what’s new and...
Supercharging Data Integration: The WhereScape and Databricks Advantage
The demand for robust data management systems has never been higher, and Databricks has quickly become a favored choice for cloud-based solutions. Its powerful capabilities make it a top contender for managing large-scale data, but when combined with WhereScape's...
Empowering Customer Success: WhereScape’s Comprehensive Support and Training Resources
Enhancing Operational Success with WhereScape’s Support Systems At WhereScape, we understand that a data warehouse is only useful to the extent that it is understood. In order to drive your organization closer to your key goals and objectives, you need full mastery of...
Revolutionizing Day-to-Day Operations: The Power of Automated Data Integration
The Transformational Role of Automation in Data Management Across industries and business stages, organizations of all types manage data in their daily operations. Whether that data entails patient appointments and reminders in a healthcare clinic, student performance...
Gartner® Insights: Microsoft Fabric as a Unified Data & Analytics Platform
Are you ready to revolutionize your data management strategy with a platform that promises to simplify and enhance your operations? According to a Gartner poll, 43% of respondents believe that the data and analytics ecosystem will significantly influence their choice...
WhereScape and YellowFin Attending World of Data in Munich
We are excited to announce that WhereScape and YellowFin will be attending the World of Data conference in Munich on June 6, 2024. This event will bring together data professionals, industry leaders, and technology enthusiasts from around the globe to explore the...
Related Content
Experience the Power of WhereScape 3D 9.0.3: New Features and Improvements
We’re thrilled to introduce our latest iteration of WhereScape 3D! Version 9.0.3 brings a host of new features and enhancements designed to make your data warehousing journey smoother, faster, and more efficient. Let’s dive into the details of what you can expect from...
Ahead of the Curve: Future Trends in Data Automation and WhereScape’s Pioneering Solutions
The Evolving Landscape of Data Automation As new technologies emerge and existing tools constantly change and improve, the world of data automation transforms rapidly. Even the most well-versed data teams find themselves disoriented and overwhelmed in the face of...
Investing in Data Automation: A Strategic Approach to Business Growth
Unlocking Growth: The Strategic Advantage of Data Automation Organizations reaping the benefits of data automation stay ahead of industry trends and improve the efficiency of their operations and decision-making. Data automation tools offer a strategic advantage for...
Data + AI Summit 2024: Key Takeaways and Innovations
The Data + AI Summit 2024, hosted by Databricks at the bustling Moscone Center in San Francisco, has concluded with remarkable revelations and forward-looking innovations. Drawing over 16,000 attendees in person and virtually connecting over 60,000 participants from...