ModelForecastStage

Versions

• 1.0.0

v1.0.0

Basic Information

Class Name: ModelForecastStage

Title: Model Forecast Stage

Version: 1.0.0

Author: Evan Rasmussen

Organization: OneStream

Creation Date: 2024-04-14

Default Routine Memory Capacity: 2.0 GB

Tags

Sensible AI Forecast, Data Transformation, Interpretability, Data Preprocessing

Description

Short Description

Stage a SensibleAI Forecast Model Forecast table for Forecast Value Add (FVA) Dashboard ingestion.

Long Description

Model Forecast Staging transforms a traditional Deployed Model Forecast (DMF) table or a Backtest Model Forecast (BMF) table from SensibleAI Forecast into a format conducive for FVA Dashboard ingestion. The Cascading Stage method of the routine offers the ability to filter the table to include only the top-ranked models for each target prediction, based on the user's specified hierarchy.

Use Cases

1. Cascading Stage Best ML Models

A user has recently updated their SensibleAI Forecast (FOR) predictions and compiled them into a Deployed Model Forecast (DMF) data table, which presents an array of predictions from 1-20 different models for each business target prediction. The user is only interested in the best ranked model for each prediction, so they will use the Model Forecast Stage routine to pick these top-ranked models based on their preferences. The user will specify that models will be selected based on the following hierarchy: Best ML, Best Intelligent, Best, and Best Baseline. This means that if there is an ML model available for a prediction, it will be selected first, followed by the Best Intelligent model, etc. The user will additionally specify that they want to copy the actuals per forecast version (actuals are duplicated for every PredictionCallID), trim the forecast ranges to the global bounds of the actuals, and have no overlap merge on the forecasts (depending on the forecast horizon and forecast regeneration frequency , there may be multiple forecasts for the same target ~ date). This approach will allow the user to filter out the noise from the worse models and focus on the most reliable and accurate predictions. With the resulting table, the user will compare the FOR predictions with the customer's benchmark forecasts, yielding a Forecast Value Add (FVA) table that provides a comprehensive analysis of accuracy improvements that FOR provides over the customer's current forecasting methods.

2. Backtest Model Forecast

A consultant is in the Rapid Project Experimentation phase of their client engagement, so they have created several projects with different configurations of feature sets and dimensionality grain. Now the consultant needs to evaluate the performance of these projects in comparison with one another to determine the best project configuration. The consultant has three different Backtest Model Forecast (BMF) tables, one for each project. Taking one BMF table at a time, they will use the Model Forecast Stage routine to filter the table to include the most accurate models. Their first choice is to select the Ensemble Best Two ML models for each prediction, which will take the average value of the two best ML models combined into one row. If there are no ML models available, they will select the Best Intelligent model, which is the single best ML or Statistical model. The third choice is the Best model, which will take the best model available, regardless of type. The consultant will repeat this process for each BMF table, resulting in three Forecast Value Add (FVA) tables that they will manipulate to generate a Forecast Snapshot dashboard. This dashboard will allow the consultant to compare the performances of each project configuration directly. At this point, the consultant may be satisfied with the results they have found and elect to continue with the best project configuration to compare with the client's benchmark forecasts.

Routine Methods

1. Cascading Stage (Method)

• Method: cascading_stage

  • Type: Method

  • Memory Capacity: 3.0 GB

  • Allow In-Memory Execution: No

  • Read Only: Yes

  • Method Limits: This method is highly variable to the corresponding SensibleAI Forecast configuration. A project of 15K targets over a 30 week prediction horizon utilizing 19 ML models, 6 selected convert types, and including actuals completed in 40 minutes with a 40GB memory allocation. With the utilization of batching, this method can be assumed to scale well to highly complex projects with additional memory allocation and an extended runtime.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • Cascading Stage a Deployed Model Forecast table as a Forecast Value Add table.

  • Detailed Description:

    • Forecast Value Add (FVA) dashboards are used to compare the performance of forecasts generated by FOR models with benchmark forecasts provided by clients. The Model Forecast Stage routine is used to filter Backtest Model Forecast (BMF) or Deployed Model Forecast (DMF) tables to a format conducive for FVA Dashboard ingestion. The routine filters the table to include only the top-ranked models for each business target prediction, based on the user's specified hierarchy. The routine also allows the user to select the handling for overlapping forecasts, forecasts, and actuals. Multiple runs of the method produce tables that are combined prior to FVA ingestion and used in conjunction with other routines to create an FVA dashboard, allowing the client to bring their benchmark forecasts and FOR predictions together.

  • Inputs:

    • Required Input
      • Source Connection: The connection information source data.
        • Name: data_connection
        • Tooltip:
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: Must be an instance of Tabular Connection
        • Nested Model: Tabular Connection
          • Required Input
            • Connection: The connection type to use to access the source data.
              • Name: tabular_connection
              • Tooltip:
                • Validation Constraints:
                  • This input may be subject to other validation constraints at runtime.
              • Type: Must be one of the following
                • SQL Server Connection
                  • Required Input
                    • Database Resource: The name of the database resource to connect to.
                      • Name: database_resource
                      • Tooltip:
                        • Validation Constraints:
                          • This input may be subject to other validation constraints at runtime.
                      • Type: str
                    • Database Name: The name of the database to connect to.
                      • Name: database_name
                      • Tooltip:
                        • Detail:
                          • Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
                        • Validation Constraints:
                          • This input may be subject to other validation constraints at runtime.
                      • Type: str
                    • Table Name: The name of the table to use.
                      • Name: table_name
                      • Tooltip:
                        • Validation Constraints:
                          • This input may be subject to other validation constraints at runtime.
                      • Type: str
                • MetaFileSystem Connection
                  • Required Input
                    • Connection Key: The MetaFileSystem connection key.
                      • Name: connection_key
                      • Tooltip:
                        • Validation Constraints:
                          • This input may be subject to other validation constraints at runtime.
                      • Type: MetaFileSystemConnectionKey
                    • File Path: The full file path to the file to ingest.
                      • Name: file_path
                      • Tooltip:
                        • Validation Constraints:
                          • This input may be subject to other validation constraints at runtime.
                      • Type: str
                • Partitioned MetaFileSystem Connection
                  • Required Input
                    • Connection Key: The MetaFileSystem connection key.
                      • Name: connection_key
                      • Tooltip:
                        • Validation Constraints:
                          • This input may be subject to other validation constraints at runtime.
                      • Type: MetaFileSystemConnectionKey
                    • File Type: The type of files to read from the directory.
                      • Name: file_type
                      • Tooltip:
                        • Validation Constraints:
                          • This input may be subject to other validation constraints at runtime.
                      • Type: FileExtensions_
                    • Directory Path: The full directory path containing partitioned tabular files.
                      • Name: directory_path
                      • Tooltip:
                        • Validation Constraints:
                          • This input may be subject to other validation constraints at runtime.
                      • Type: str
      • Convert Types: Select the hierarchical types of transformations to make for the model forecast table.
        • Name: convert_types
        • Tooltip:
          • Validation Constraints:
            • The input must have a maximum length of 6.
            • This input may be subject to other validation constraints at runtime.
        • Type: list[ModelForecastConvertType_]
      • Overlapped Forecasts Handling: Determine how to handle overlapping forecasts.
        • Name: overlapped_forecasts_handling
        • Long Description: Use Latest: remove overlapping predictions from prior forecasts and keep most recent one. Use Oldest: remove overlapping predictions from more recent forecasts and keep oldest one. No Overlap Merge: no action on DMF table.
        • Tooltip:
          • Detail:
            • Use Latest: remove overlapping predictions from prior forecasts and keep most recent one. Use Oldest: remove overlapping predictions from more recent forecasts and keep oldest one. No Overlap Merge: no action on DMF table.
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: OverlappedForecastsHandling_
      • Forecast Bounds Handling: Determine how to remove forecast values relative to actual values.
        • Name: forecast_handling
        • Long Description: Trim to Actuals Bounds: remove forecasts values for targets whose dates are outside of the date bounds of the actuals values for that target. Trim to Global Actuals Bounds: remove forecasts values for targets whose dates are outside of the global date bounds of the actuals values. No Action: take no action, keep forecasts that may be outside of these date ranges.
        • Tooltip:
          • Detail:
            • Trim to Actuals Bounds: remove forecasts values for targets whose dates are outside of the date bounds of the actuals values for that target. Trim to Global Actuals Bounds: remove forecasts values for targets whose dates are outside of the global date bounds of the actuals values. No Action: take no action, keep forecasts that may be outside of these date ranges.
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: ForecastHandling_
      • Actuals Handling: Determine how actuals from the DMF table should be handled.
        • Name: actuals_handling
        • Long Description: Remove: remove all actuals from the DMF table. No Action: take no action, keep actuals as is in the DMF table. Copy per Forecast Version: duplicate the actuals for each forecast (PredictionCallID).
        • Tooltip:
          • Detail:
            • Remove: remove all actuals from the DMF table. No Action: take no action, keep actuals as is in the DMF table. Copy per Forecast Version: duplicate the actuals for each forecast (PredictionCallID).
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: ActualsHandling_

  • Artifacts:

    • Cascading Staged Data: A model forecast table with hierarchical selections of top-ranking models staged as a Forecast Value Add table.
      • Qualified Key Annotation: cascading_staged_data
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@cascading_staged_data/data_/data_<int>.parquet
          • A partitioned set of parquet files where each file will have no more than 1000000 rows.

Interface Definitions

No interface definitions found for this routine

Developer Docs

Routine Typename: ModelForecastStage

Method Name	Artifact Keys
cascading_stage	cascading_staged_data