Building a Data Management Sequence for Data Ingress and Egress
Introduction
When delivering a productionized SensibleAI Forecast use case to a customer, it is generally required to provide data management jobs that run data ingress and data egress business rules that extract, load, and transform customer data from its source and into databases within OneStream or manipulate SensibleAI Forecast forecast outputs to consumable formats.
Typically, these data management sequences can be incorporated into a “wrapper dashboard” that allows users to run these jobs from the SensibleAI Forecast workflow. As such, it is encouraged to incorporate input parameters to these data management steps, so that end-users can maintain some control over how the data ingress or egress sequence is run.
Creating the Data Management Group
Navigate to the Data Management page via Application > Tools > Data Management within the OneStream environment. Create a new Data Management group by selecting “Create Group”.
Then, enter the Data Management Group name, a description, and configure its Security settings (if applicable). It is recommended to name the group “SensibleAI Forecast [UseCase] [Ingress or Egress]”, where the use case is a short description of the purpose of SensibleAI Forecast Forecasting. This could be demand, a specific division or region in the company, or a P&L, for example. Save the group when finished.
The created group will now appear in the Data Management Groups dropdown and have an empty Sequence and Step below it. At this point, it is recommended to create the business rule(s) that should be run.
Configuring the Step and Creating the Sequence
Generally, data ingress and egress groups should be designed with a single sequence that performs a single step. Create the step by selecting the Steps section below the Data Management group and clicking “Create Step”. This step should be Execute Business Rule.
Fill out the name of the Step, recommended to be identical to the Data Management group name, and specify the business rule it should execute. By selecting the ellipsis (…) on the right side of the business rule text field, a list of selectable business rules will appear. Parameters are optional and will be covered in a later section. Save the step when completed.
Next, select the Sequences section below the Data Management group and click “Create Sequence”. In the Sequence Properties tab, name the Sequence identical to the Data Management group name. For common Data Ingress and Data Egress jobs, it isn’t required to fill out the Application Servers, or Parameter Substitution sections.
Then, select the Sequence Steps tab and add a step by clicking the plus button. Select the Step that was just created. Since most Data Ingress and Egress sequences will just be composed of a single step, no additional steps should be added. If there is the need for them, the order of the steps can be changed with the up and down arrows next to the Add Step button. When run, the Sequence will execute all Steps in the order specified on this page.
Save the Step after completion. The Data Management job can now be executed by selecting the “Run” button on the top ribbon. In a production setting, it is not recommended to require the users to run the Ingress or Egress statements from the Data Management page, but rather build a dashboard from which they can be run. Rather, use the “Run” button for testing and debugging.
Linking Input Parameters
One of the important aspects of a data management job is the ability to feed parameters into the executed business rule. Common examples of this could include:
- Specifying forecast start date - Imagine the customer continuously updates their source data. If a partial month has been uploaded, SensibleAI Forecast may think that the forecast start date is the following month, rather than the current month. By having the user enter the intended forecast start date, the business rule can be designed to remove any data past that date, guaranteeing the SensibleAI Forecast prediction aligns with expectations.
- Specifying time horizon or granularity - Sometimes project segmentation is required in a use case that overlays a short term and a long term forecast, or a weekly with a monthly forecast. Therefore, predictions are run at different times. When the Data Egress job attempts to align these forecasts, it may require direction on which long-term or short-term forecast should be processed.
- Scenario Modeling - Users may want to run processing on specific scenarios or forecasts in a Deployed Model Forecast table.
In these examples, it is more efficient to build a single business rule that slightly changes based on an input parameter. These input parameters can be accessed in the Business Rules by the ExtenderArgs args parameter, and can be programmed into the Data Management Step under the “Parameters” field.
Since these parameters will likely be defined from a dashboard, the “Parameters” field can be filled in as follows:
param_ForecastStartPeriod=|!param_ForecastStartPeriod!|, param_Scenario="Long Term"
Parameters and their values are separated by commas, and the parameter name (param_ForecastStartDate, param_Scenario, in this case) is exactly how the business rule will look up the value.
string startPeriod = "";
args.NameValuePairs.TryGetValue("param_ForecastStartPeriod", out startPeriod);
The |!param_name!| syntax for parameter value is how a user can tie the business rule parameter to a dashboard parameter. The idea behind this is that a user can select a month and year from a dropdown box, for example, and those values can be passed into the business rule by aligning the data management step with the dashboard parameter name. Alternatively, parameters can be hardcoded with a value. Perhaps instead of a dropdown box specifying running the long-term or short-term forecast export, the user wants two different buttons. In this case, two data steps could be created that reference the same business rule, with hardcoded “Long Term” or “Short Term” parameters.
Summary Recommendations
Best practices for building data ingress and egress Data Management sequences are as follows:
-
Each Group should contain a single sequence that runs a single step, executing the ingress or egress business rule.
-
Name the Group, Step, and Sequence “SensibleAI Forecast [UseCase] [Ingress or Egress]”.
-
Incorporate parameters so that user can specify certain aspects of how the job should run.
- At a bare minimum, the intended forecast start date should always be included to guarantee that weeks or months are not partially filled.