WhereScape 3D: How to handle model conversion rules

This blog was written by Marcel Schwarze, Consultant at b.telligent, one of our trusted DACH implementation partners. Click here to read the original blog post in German and here to go to b.telligent’s partner page on our website.

Model conversion rules in WhereScape 3D are used to automatically generate data models and customize them according to individual requirements. Due to the high level of standardization and automation capacity required, model conversion rules are particularly useful for Data Vault models.  

With growing demands from the business, customization and development of model conversion rules can become more complex and complicated. This blog provides you with some best practices, tips and tricks to handle your model conversion rules successfully and simplifies their usage for beginners.

Tip #1: Look for recurring patterns (assessment of necessity)

An essential precondition for automating with model conversion rules is the availability of repeatable patterns. Process execution with model conversion rules only makes sense if certain patterns and regularities are in place for various process steps. Only then can the advantages of automation can be fully realized.

If you have a very individual process, i.e. the process is either executed only once or each new process execution significantly differs from the previous execution, then there aren’t any patterns and regularities available and it is difficult to standardize the process. In this case, the effort to implement a process with model conversion rules exceeds the actual benefit of automation and we recommend a manual implementation.

Tip #2: Implement model conversion manually before automating

Before starting to develop a new model conversion set, i.e. the automation of a process, we recommend that you execute your required process steps and have the process properly documentation first.

This has three key benefits:

1. Assessment of feasibility: You can only automate the desired requirements if they are manually feasible. This prior manual execution will probably save you time as you don’t have to work on an automated, in most cases more time-consuming solution first.
2. Simplification of rule selection: Due to the prior documentation of your process execution, you gain an overview of the process steps you need to map with model conversion rules. Even though not every process step equals a certain rule type, it facilitates your selection of required rule types in terms of type and extends them significantly. In this context, consider that for one process step you need multiple rule types and that multiple process steps can be covered by one rule type.
3. Optimization of rule development: Gaining knowledge about your individual process steps, the relations and mutual interactions, will ultimately simplify your development (templates, matching criteria, …). 

Tip #3: Simple but useful: use the help function

During your rule development, the best support is provided by WhereScape’s help function. Check out the navigation path Help > Help. The sections named “Model Conversions” and “Pebble Templates” offer detailed descriptions of individual rule types and syntax of the Pebble template language.

Tip #4: Use the preview function

in WhereScape 3D you can see the result of your template code application during development and therefore reduce the development time and error rate of your implemented template snippets.

You will find the preview function to the right of the code editor. To use it, go to “Choose entity to preview template”, click on the “…” button and choose an entity. Then the Pebble code from the preview window to the left is applied to the selected entity and the result is displayed on the right-hand side.

Special tip:

By activating the auto-refresh, each change on the left-hand screen will automatically refresh the preview window. Without auto-refresh you have to click on the refresh symbol to refresh the preview window.

Tip #5: Debug your model conversions by using the snapshot functionality

If the execution of a model conversion set fails, an error message appears. This gives you information about failed rules, the object on which the rule has been applied in case of failure, and an error description.

If the information for debugging is not sufficient or a successful but incorrect model has been created, we recommend you debug by using the snapshot functionality: A snapshot during rule execution is like a snapshot of a model after a certain rule has been applied to this model. Especially before an incorrect rule (“How does a model look like before something went wrong?”) and for the incorrect rule itself (“How does a model look like after something went wrong?”) the use of snapshots is reasonable.

You can create a snapshot in just two steps:

1. The first step is to select the rules after whose successful execution a snapshot should be created. Navigate to the model conversion rules, click on the left-hand field of one or multiple rules, and select “Snapshot”.
2. In the second step, the corresponding rules must be executed. Here you have to ensure that the field “Create snapshots” is activated during the selection of the model conversion sets to be executed.

After running through the rules you can view the models created based on the snapshots and identify any aspects that may have been implemented incorrectly.

Tip #6: Define clear development rules

To ensure a unified development and obtain consistent and clearly structured model conversion sets, we recommend defining development guidelines. The following considerations will help:

1. Creation of new versus extending existing model conversion sets: For clarity, you should avoid creating a new model conversion set for each new rule. Conversely, not all rules can be combined into one single model conversion set. In fact, for each new development, you should evaluate whether creating a new or extending an existing model conversion set makes more sense. Ask yourself the following questions: Can you assign a new rule to an already existing range of tasks or should it have its own topic area? Does the complexity of all new rules justify a different model conversion set?
2. Prevention of negative interactions: Often several users develop in parallel. To avoid negative interactions due to simultaneous modifications of existing rules you should work with duplicates and transfer them after successful testing into the original model conversion sets.
3. Compliance of naming convention: Finally, create a naming convention to ensure you keep a clear overview. More on this in the next section. 

Tip #7: Create and use a naming convention

To ensure consistency and standardization as well as to simplify the search for developed rules, we recommend creating and using a naming convention. Including the following name components in the name of a rule set applied during the transition from model category A to model category B has proven effective:

[IndividualToken]_[AbbreviationModelcategoryA][AbbreviationModelcategoryB] – [Short description of rule sets]

[IndividualToken] corresponds to an in-house abbreviation. This is to distinguish from the default rules supplied by WhereScape.  [AbbreviationModelcategoryA] and [AbbreviationModelcategoryB] are the categories of the initial model as well as the newly generated model.

As an example, let’s look at rule set “ws3d_rvls – Create loads”: This is a rule set (ws3d) provided by WhereScape which is applied during the transition from Data Vault category (RV=RawVault) to load and staging category (ls) and is used to generate load tables (Create loads).

For the rules within a rule set, we recommend you use meaningful wording and maintain the description fields properly. Additionally, you can define semantic sections via rule type “Separator” within the model conversion set.

Tip #8: Regularly export existing moel conversion sets

To reduce the risk of an accidental loss and to import legacy development stages, we recommend you regularly export existing model conversion sets in the form of XML files.

To do so, navigate to the model conversion rules and click on the displayed disk icon. Afterward, you can select the Model Conversion Sets of the current repository to be exported. This means that an entire model conversion set is exported instead of only a single rule. Use the arrow next to the disk to import model conversion sets.

Alternatively, the entire repository can be backed up automatically using CLI calls and the WhereScape RED scheduler. The import of individual model conversion sets is no longer possible. Instead, the entire repository backup must always be imported.

Special tip:

Usually, in 3D you would run different rulesets one after another. If you want to combine rulesets you can export them and copy the rules of one XML data file into the other. You can then import these results into 3D. 

Tip #9: Introduce external model conversion documentation

Often several people work on current model conversion rules at the same time. As well as inserting new rules or deleting old ones, the modification of existing rules is a critical operation. It often involves small changes to the template code or the matching criteria that may remain undiscovered or are quickly forgotten by other team members if they are not discussed.

To avoid future confusion and guarantee efficient testing, we recommend you introduce an external model conversion documentation. The following table shows a sample structure for a model conversion set named “Generate Data Vault”. You can also use it for the documentation of other rule sets and can complement backups if necessary.

In this table, the rule “Create satellites” is changed and later deleted, whereas the rule “Create hubs” is recreated.

For large DWH projects with individual requirements, we recommend sticking to certain standards to guarantee efficient and effective use of model conversion rules. In this blog article we showed you sample tips and tricks to help you get started easier.

This blog was written by Marcel Schwarze, Consultant at b.telligent, one of our trusted DACH implementation partners. Click here to read the original blog post in German and here to go to b.telligent’s partner page on our website.