WhereScape Q&A: Your Top Questions Answered on Data Vault and Databricks

During our latest WhereScape webinar, attendees had fantastic questions about Data Vault 2.0, Databricks, and metadata automation. We’ve compiled the best questions and answers to help you understand how WhereScape streamlines data modeling, automation, and integration with modern cloud platforms.

From schema evolution to Business Vault transformations, here’s everything you need to know—straight from the webinar’s Q&A session! 

1. Can I create PIT and Bridge tables in the Business Vault?

Yes, WhereScape provides built-in wizards to automate the creation of PIT (Point-in-Time) and Bridge tables. These tables are essential for improving query performance and historical tracking in a Data Vault model.

WhereScape RED includes a PIT wizard, where you simply select the necessary options, define the business keys, and specify the ghost record settings. This wizard will generate the PIT table, which helps track historical changes efficiently.

The Bridge table wizard allows users to define relationships between multiple tables, making queries faster by reducing the number of joins needed in reporting. These tables are especially useful in Databricks, where complex joins can impact performance.

Additionally, WhereScape automatically generates the necessary SQL code based on templates, so users don’t have to manually write or maintain these tables.

2. Does WhereScape support schema evolution in Databricks?

Yes, WhereScape and Databricks Auto Loader work together to manage schema evolution when data structures change.

WhereScape handles schema evolution in two key ways:

1. Automatic Schema Detection
   • Databricks Auto Loader can detect schema changes in incoming data and apply updates dynamically.
2. Metadata Validation in WhereScape
   • WhereScape provides a metadata validation tool that compares the source schema with the current data model.
   • If a column is added, removed, or changed, WhereScape will alert users and offer two options:
     • Update metadata in the UI.
     • Issue an ALTER statement to modify the schema in the database.

This ensures that schema changes are controlled and validated before affecting downstream processes, preventing unexpected errors.

3. How does WhereScape support the Medallion Architecture?

WhereScape aligns directly with the Bronze, Silver, and Gold layers of Databricks’ Medallion Architecture:

• Bronze Layer → Load & Stage Tables
  • WhereScape loads raw data into staging tables that act as a landing zone for incoming files.
• Silver Layer → Data Vault (Raw Vault)
  • WhereScape generates a Data Vault model where raw data is structured into hubs, links, and satellites.
• Gold Layer → Business Vault & Reporting Schema
  • Users can create additional business rules, PIT, and Bridge tables in the Business Vault before loading fact and dimension tables for reporting.

WhereScape allows flexibility—users can choose to build Data Vault, 3NF, or Star Schema models within this framework.

4. Can WhereScape integrate with Azure Purview?

Yes, WhereScape 3D supports importing metadata from Azure Purview.

Users can bring metadata from Purview into WhereScape 3D to create data models based on cataloged data.

Additionally, Purview can discover the WhereScape 3D PostgreSQL repository, enabling governance and lineage tracking.

This integration helps centralize metadata management, allowing organizations to track data lineage, apply compliance rules, and improve governance across multiple platforms.

5. Can WhereScape support serverless architectures in Databricks?

Yes, WhereScape fully supports Databricks serverless compute by leveraging:

• Delta Live Tables (DLT) Pipelines
  • Users can define a continuously running or scheduled pipeline.
  • WhereScape automatically generates the necessary scripts to run jobs in serverless mode.
• Notebook-Based Processing
  • Users can configure Databricks Notebooks instead of ODBC connections to process data dynamically.
  • WhereScape will generate Python code and push it to Databricks notebooks automatically.

This ensures a fully automated, serverless architecture that eliminates the need for managing compute infrastructure manually.

6. How are business rules applied in the Business Vault?

Business rules should not be applied in the Raw Vault, but rather in the Business Vault, which supports:

• Views on top of Satellites – Users can create custom SQL views to filter, aggregate, or transform data.
• Transformation Satellites – These store calculated columns derived from raw attributes.
• Fact Table Transformations – Business rules can be applied directly within fact tables before being exposed to a reporting tool.

The Business Vault acts as an extension of the Raw Vault, allowing users to apply additional calculations and derive new insights before data is used in BI tools.

7. Does WhereScape automatically generate Delta Tables in Databricks?

Yes, all tables created in WhereScape RED for Databricks are Delta Tables by default.

• Users can modify properties to create:
  • Delta Live Tables (DLT) for real-time ingestion.
  • Parquet or Iceberg tables if needed.

WhereScape ensures that every table is optimized for Databricks’ native storage formats, improving performance and reducing the need for additional configurations.

Final Thoughts

This webinar provided valuable insights into how WhereScape supports Data Vault, Databricks, and metadata-driven automation. If you have further questions, feel free to reach out to us at [email protected] or Request a Demo Here.

Watch the Full Webinar On-Demand

Want to see these concepts in action? You can watch the full WhereScape webinar on-demand to get a detailed walkthrough of Data Vault automation in Databricks, complete with live demos and Q&A insights. Watch it here.

Additional Resources

Data Vault and Databricks: Automation Techniques, Best Practices, and Use Cases

Watch an exclusive webcast featuring Kevin Marshbank, Principal Consultant of The Data Vault Shop, where he demonstrates how Data Vault, Databricks, and WhereScape automation tools transform data warehousing strategies.

Unifying WhereScape with Databricks – Databricks White Paper

Explore how organizations overcome complex data challenges using WhereScape and Databricks, showcasing real-world solutions, measurable outcomes, and strategic insights. Download Now.

Webcast: The Benefits of Combining WhereScape with Databricks

Learn how to integrate WhereScape’s automation tools with Databricks’ Medallion Architecture for enhanced data processing and management.

10 Pro Tips to Enhance Databricks Performance with WhereScape

Discover expert strategies for optimizing WhereScape’s capabilities in Databricks, including Delta Live Tables, Structured Streaming, and AutoML. Read more here.