TargetFlagRoutine

Versions

1.0.0

v1.0.0

Basic Information

Class Name: TargetFlagRoutine

Title: Target Flagging Analysis

Version: 1.0.0

Author: Sebi Kulyckyj

Organization: OneStream

Creation Date: 2024-08-27

Default Routine Memory Capacity: 2.0 GB

Description

Short Description

Calculate various metrics and flag them based on certain values

Long Description

This routine is used to flag target dimension intersections in a dataset based on calculated metrics values. We generate metrics tables based on the source data as well as the model forecast data. Using both of these datasets we are able to calculate the following metrics: Source Data Metrics: Actuals Summation, Target Start Date, Target End Date, Collection Lag Days, Collection Lag Periods, Start Up Lag Days, Start Up Lag Periods, IsForecastable (how forecastable a target dimension is), Local Density, Local Density with Zeros Treated as Missing Values, Global Density, Global Density with Zeros Treated as Missing Values, Local Range. Model Forecast Data Metrics: Forecast Start Date, Forecast Name, Forecast Number, Target Name, Model, Model Category, Model Rank, Forecast Summations, Actuals Summations, Mean Absolute Error (MAE), MAE Percentage, Root Mean Squared Error, Bias Error (forecast totals - actuals totals), Bias Error Percentage, Score Percentage (MAE% + Bias%) and Growth Rate. Growth Rate corresponds to the forecast value of a target compared to the actuals of that same period from the year before (Ex: Jan 2020 - Feb 2020, compared to Jan 2019 - Feb 2019). We also generate two plots from the model forecast dimension data which compare the actuals against MAE% and Score% (MAE + Bias Error). The routine also generates an HTML template that offers an interactive interface for users to analyze the metrics data easier. Once we calculate the metrics, customers are able to create certain flags, such as if MAE% is greater than 0.7, flag the target and return a severity level of the flag.

Use Cases

1. Generate All Metrics Without Flags

This routine is useful for customers who want to understand the metrics of the source data and the model forecast data. The routine will create two separate tables, one based on the source time series data and one based on the combination of the source time series data and the forecasted target data for each specific target dimension intersection. A target dimension intersection can be thought of as a unique combination of dimensions, such as a store and department. The calculated metric tables will provide a summary of the time series data for each target dimension intersection, including the total summations value for the target dimension, the local density, global density and other useful metrics when looking at source data. We also then generate forecast dimension metrics table which calculates important metrics such as the mean absolute error, mean squared error, and so on. This routine method also generates the MAE vs Actuals and Score vs Actuals plots. No flags will be generated as an output to this routine method.

2. Generate Source Metrics Without Flags

This routine caters to a wide audience, ranging from data scientists to analysts to business professionals seeking to extract meaningful insights from their source time series data table. Generating the source data table by itself will allow users to attain metrics about their data's local density, global density, start up lag days, start up lag periods, collection lag days, collection lag periods, whether or not their targets are forecastable and other simple metrics. This method does not offer our MAE vs Actuals or Score vs Actuals plots. This routine method also does not offer target flagging, no flags will be generated as an output.

3. Generate Forecast Metrics Without Flags

This routine is useful for customers who want to understand the metrics of the forecasted data. The routine will create a table based on the deployed model forecast data that will provide a summary for each target dimension intersection, including the forecast totals, the actual totals for that same period (if available), the mean absolute error, mean squared error, and other useful metrics. This routine method also generates the MAE vs Actuals and Score vs Actuals plots. This information can be very helpful to see which targets are being forecasted well and which targets are not being forecasted well. However this routine method does not provide target flagging.

4. Generate All Metrics With Flags

This routine method is useful for users who would like to understand more about their source data as well as the deployed model forecast metrics and be able to flag these calculated metrics based on certain thresholds. This routine method will calculate all the same metrics as the Generate All Metrics Without Flags method, but will also provide the ability to flag these metrics based on certain thresholds. For example if a user wants to flag all targets with a MAE% greater than 0.7, they can do so with this routine method. This routine method will also generate the MAE vs Actuals and Score vs Actuals plots. The output of this routine method will be the source data metrics table, the forecast data metrics table, a flag definition table (all metrics that are flagged and what the value and the HTML template with the flags applied.

5. Generate Source Metrics With Flags

This routine method is useful for users who would like to understand more about their source data and be able to flag these calculated metrics based on certain thresholds. This routine method will calculate all the same metrics as the Generate Source Metrics Without Flags method, but will also provide the ability to flag these metrics based on certain thresholds. For example if a user wants to flag all targets with a local density greater than 0.7, they can do so with this routine method. This routine method will not generate the MAE vs Actuals and Score vs Actuals plots. The output of this routine method will be the source data metrics table, a flag definition table (all metrics that are flagged and what the value) and the HTML template with the flags applied.

6. Generate Forecast Metrics With Flags

This routine method is useful for users who would like to understand more about their deployed model forecast metrics and be able to flag these calculated metrics based on certain thresholds. This routine method will calculate all the same metrics as the Generate Forecast Metrics Without Flags method, but will also provide the ability to flag these metrics based on certain thresholds. For example if a user wants to flag all targets with a MAE% greater than 0.7, they can do so with this routine method. This routine method will also generate the MAE vs Actuals and Score vs Actuals plots. The output of this routine method will be the forecast data metrics table, a flag definition table (all metrics that are flagged and what the value) and the HTML template with the flags applied.

7. Generate Flags Only

This routine method is useful for users who only want to generate the certain flags but do not care with the output of actual metric tables. This routine method will not output the source metrics table, the forecast metrics table, the MAE vs Actuals plot or the Score vs Actuals plot. The output of this routine method will be the Flag Definition table (all metrics flags provided by the user) as well as the Dimension Flag table (the results of the flags as well as which targets violate these flags). The source data and the forecast data will still need to be inputted for the flags to be generated.

Routine Methods

1. Init (Constructor)

Method: __init__
- Type: Constructor
- Allow In-Memory Execution: No
- Read Only: No
- Method Limits: N/A
- Outputs Dynamic Artifacts: No
- Short Description:
  - Constructor for the TargetFlagRoutine
- Detailed Description:
  - This constructor will initialize the flag definition table with the user inputs. The flag definition table holds all flags the user wants to check with metric name, value to compare to, comparator and other information.
- Inputs:
  - Required Input
    - Flagging Setup: The metrics to flag based on user inputted thresholds.
      - Name: flag_inputs
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: list[TargetFlaggingDefinition]
- Artifacts: No artifacts are returned by this method

2. All Metrics Analysis (Method)

Method: all_metrics_analysis
- Type: Method
- Memory Capacity: 2.0 GB
- Allow In-Memory Execution: No
- Read Only: No
- Method Limits: N/A
- Outputs Dynamic Artifacts: No
- Short Description:
  - Generate source and forecast metrics table without target flagging
- Detailed Description:
  - This method will take a source dataset as well as the dimension columns, date column name, value column name and a data frequency. With these values we will then create a source dimension metrics table with the following columns: Target Dimension Columns, Value column, Local Density, Local Density With Zeros, Global Density, Global Density With Zeros, Start Up Lag Days, Start Up Lag Periods, Collection Lag Days, Collection Lag Periods, Local Start Date, Local End Date, Summations (actuals summations), IsForecastable, Global Start Date, Global End Date and the Local Range. We will also create a forecast dimension metrics table with the following columns: Target Dimension Columns, Forecast Start Date, Date, Value, ProjectID, XperimentBuildID, XperimentKernelID, TargetName, Model, ModelCategory, ModelRank, ForecastStartDate, ForecastName, ActualsSummations, ForecastSummations, BiasError, MeanSquaredError, MeanAbsoluteError, MeanAbsolutePercentageError, PercentBiasError, RootMeanSquaredError, GrowthRate.
- Inputs:
  - Required Input
    - Source Data Definition: The source data definition.
      - Name: source_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Model Forecast Data Definition: The model forecast data definition. Expects a SensibleAI Forecast deployed model forecast table.
      - Name: model_forecast_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Data Frequency: The frequency of the source data.
      - Name: data_frequency
      - Tooltip:
        
        Detail:
        
        Keeping the frequency as auto will result in the data frequency being determined automatically.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: DataFrequency_
- Artifacts:
  - Source Dimension Metrics Parquet: Parquet file containing the data about the source data provided. This data contains a multitude of metrics that are used to determine the quality of the data. This Parquet includes simpler metrics such as total value, local start date, etc.
    - Qualified Key Annotation: source_data_artifact
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@source_data_artifact/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Forecast Dimension Metrics Parquet: Parquet file containing the data about the forecast model data provided. This data contains a multitude of metrics that are used to determine the quality of the data. This Parquet includes more in depth metrics such as squared error, bias error, etc.
    - Qualified Key Annotation: forecast_data_artifact
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@forecast_data_artifact/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - All Metrics Summary HTML report: An HTML file that provides an interactive interface which the user can leverage in order to analyze both the source dimension metrics as well as the model forecast dimension metrics. On top of that, this HTML file provides two plots displaying the actuals of each target against that target's MAE% (Mean Absolute Error) and another for Score% (MAE + Bias Error where Bias Error is the total forecast - total actuals for a given forecast range).
    - Qualified Key Annotation: metrics_data_summary
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@metrics_data_summary/data_/html_content.html
        
        The html content.
  - Dimension Columns: A dictionary containing the dimension columns that were used to generate the source data dimension metrics.
    - Qualified Key Annotation: dimension_columns
    - Aggregate Artifact: False
    - In-Memory Json Accessible: True
    - File Annotations:
      - artifacts_/@dimension_columns/data_/data.json
        
        Stored json data. The schema is not known until runtime.

3. All Metrics Flagging (Method)

Method: all_metrics_flagging
- Type: Method
- Memory Capacity: 2.0 GB
- Allow In-Memory Execution: No
- Read Only: No
- Method Limits: N/A
- Outputs Dynamic Artifacts: No
- Short Description:
  - Generate source and forecast metrics table without target flagging
- Detailed Description:
  - This method will take a source dataset as well as the dimension columns, date column name, value column name and a data frequency. With these values we will then create a source dimension metrics table with the following columns: Target Dimension Columns, Value column, Local Density, Local Density With Zeros, Global Density, Global Density With Zeros, Start Up Lag Days, Start Up Lag Periods, Collection Lag Days, Collection Lag Periods, Local Start Date, Local End Date, Summations (actuals summations), IsForecastable, Global Start Date, Global End Date and the Local Range. We will also create a forecast dimension metrics table with the following columns: Target Dimension Columns, Forecast Start Date, Date, Value, ProjectID, XperimentBuildID, XperimentKernelID, TargetName, Model, ModelCategory, ModelRank, ForecastStartDate, ForecastName, ActualsSummations, ForecastSummations, BiasError, MeanSquaredError, MeanAbsoluteError, MeanAbsolutePercentageError, PercentBiasError, RootMeanSquaredError, GrowthRate.
- Inputs:
  - Required Input
    - Source Data Definition: The source data definition.
      - Name: source_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Model Forecast Data Definition: The model forecast data definition. Expects a SensibleAI Forecast deployed model forecast table.
      - Name: model_forecast_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Data Frequency: The frequency of the source data.
      - Name: data_frequency
      - Tooltip:
        
        Detail:
        
        Keeping the frequency as auto will result in the data frequency being determined automatically.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: DataFrequency_
- Artifacts:
  - Source Dimension Metrics Parquet: Parquet file containing the data about the source data provided. This data contains a multitude of metrics that are used to determine the quality of the data. This Parquet includes simpler metrics such as total value, local start date, etc.
    - Qualified Key Annotation: source_data_artifact
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@source_data_artifact/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Forecast Dimension Metrics Parquet: Parquet file containing the data about the forecast model data provided. This data contains a multitude of metrics that are used to determine the quality of the data. This Parquet includes more in depth metrics such as squared error, bias error, etc.
    - Qualified Key Annotation: forecast_data_artifact
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@forecast_data_artifact/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - All Metrics Flagging HTML report: An HTML file that provides an interactive interface which the user can leverage in order to analyze both the source dimension metrics as well as the model forecast dimension metrics. On top of that, this HTML file provides two plots displaying the actuals of each target against that target's MAE% (Mean Absolute Error) and another for Score% (MAE + Bias Error where Bias Error is the total forecast - total actuals for a given forecast range).
    - Qualified Key Annotation: metrics_data_summary
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@metrics_data_summary/data_/html_content.html
        
        The html content.
  - Dimension Columns: A dictionary containing the dimension columns that were used to generate the source data dimension metrics.
    - Qualified Key Annotation: dimension_columns
    - Aggregate Artifact: False
    - In-Memory Json Accessible: True
    - File Annotations:
      - artifacts_/@dimension_columns/data_/data.json
        
        Stored json data. The schema is not known until runtime.
  - Flag Definitions: A Parquet file containing the definitions of the flags that were used to flag the data.
    - Qualified Key Annotation: flags_definition
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@flags_definition/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Dimension Flags: A Parquet file containing the flags that were generated for each dimension as well as the severity levels of the flags.
    - Qualified Key Annotation: dimension_flags
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@dimension_flags/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.

4. Forecast Metrics Analysis (Method)

Method: forecast_metrics_analysis
- Type: Method
- Memory Capacity: 2.0 GB
- Allow In-Memory Execution: No
- Read Only: No
- Method Limits: N/A
- Outputs Dynamic Artifacts: No
- Short Description:
  - Generate forecast metrics table without target flagging
- Detailed Description:
  - This method will take a model forecast dataset as well as the dimension columns, date column name, and value column name. With these values we will then create a forecast dimension metrics table with the following columns: Target Dimension Columns, Forecast Start Date, Date, Value, ProjectID, XperimentBuildID, XperimentKernelID, TargetName, Model, ModelCategory, ModelRank, ForecastStartDate, ForecastName, ActualsSummations, ForecastSummations, BiasError, MeanSquaredError, MeanAbsoluteError, MeanAbsolutePercentageError, PercentBiasError, RootMeanSquaredError, GrowthRate
- Inputs:
  - Required Input
    - Source Data Definition: The source data definition.
      - Name: source_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Model Forecast Data Definition: The model forecast data definition. Expects a SensibleAI Forecast deployed model forecast table.
      - Name: model_forecast_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
- Artifacts:
  - Forecast Dimension Metrics Parquet: Parquet file containing the data about the forecast model data provided. This data contains a multitude of metrics that are used to determine the quality of the data. This Parquet includes more in depth metrics such as squared error, bias error, etc.
    - Qualified Key Annotation: forecast_data_artifact
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@forecast_data_artifact/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Forecast Dimension Metrics HTML report: An HTML file that provides an interactive interface for the user to view the forecast dimension metrics data. This HTML file provides two plots that plot the actuals of each target against that target's MAE% (Mean Absolute Error) and another for Score% (MAE + Bias Error where Bias Error is the total forecast - total actuals for a given forecast range).
    - Qualified Key Annotation: forecast_data_summary
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@forecast_data_summary/data_/html_content.html
        
        The html content.
  - Dimension Columns: A dictionary containing the dimension columns that were used to generate the source data dimension metrics.
    - Qualified Key Annotation: dimension_columns
    - Aggregate Artifact: False
    - In-Memory Json Accessible: True
    - File Annotations:
      - artifacts_/@dimension_columns/data_/data.json
        
        Stored json data. The schema is not known until runtime.

5. Forecast Metrics Flagging (Method)

Method: forecast_metrics_flagging
- Type: Method
- Memory Capacity: 2.0 GB
- Allow In-Memory Execution: No
- Read Only: No
- Method Limits: N/A
- Outputs Dynamic Artifacts: No
- Short Description:
  - Generate forecast metrics table with target flagging
- Detailed Description:
  - This method will take a model forecast dataset as well as the dimension columns, date column name, and value column name. With these values we will then create a forecast dimension metrics table with the following columns: Target Dimension Columns, Forecast Start Date, Date, Value, ProjectID, XperimentBuildID, XperimentKernelID, TargetName, Model, ModelCategory, ModelRank, ForecastStartDate, ForecastName, ActualsSummations, ForecastSummations, BiasError, MeanSquaredError, MeanAbsoluteError, MeanAbsolutePercentageError, PercentBiasError, RootMeanSquaredError, GrowthRate. We will also create a flag definition table with the following columns: Dimension Columns 1-8, Target, MetricSource, MetricName, MetricDataType, MetricValue, FlagDefintionID, FlagSystemComment.
- Inputs:
  - Required Input
    - Source Data Definition: The source data definition.
      - Name: source_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Model Forecast Data Definition: The model forecast data definition. Expects a SensibleAI Forecast deployed model forecast table.
      - Name: model_forecast_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
- Artifacts:
  - Forecast Dimension Metrics Parquet: Parquet file containing the data about the forecast model data provided. This data contains a multitude of metrics that are used to determine the quality of the data. This Parquet includes more in depth metrics such as squared error, bias error, etc.
    - Qualified Key Annotation: forecast_data_artifact
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@forecast_data_artifact/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Forecast Dimension Flagging HTML report: An HTML file that provides an interactive interface for the user to view the forecast dimension metrics data. This HTML file provides two plots that plot the actuals of each target against that target's MAE% (Mean Absolute Error) and another for Score% (MAE + Bias Error where Bias Error is the total forecast - total actuals for a given forecast range).
    - Qualified Key Annotation: forecast_data_summary
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@forecast_data_summary/data_/html_content.html
        
        The html content.
  - Dimension Columns: A dictionary containing the dimension columns that were used to generate the source data dimension metrics.
    - Qualified Key Annotation: dimension_columns
    - Aggregate Artifact: False
    - In-Memory Json Accessible: True
    - File Annotations:
      - artifacts_/@dimension_columns/data_/data.json
        
        Stored json data. The schema is not known until runtime.
  - Flag Definitions: A Parquet file containing the definitions of the flags that were used to flag the data.
    - Qualified Key Annotation: flags_definition
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@flags_definition/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Dimension Flags: A Parquet file containing the flags that were generated for each dimension as well as the severity levels of the flags.
    - Qualified Key Annotation: dimension_flags
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@dimension_flags/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.

6. Generate Flags Only (Method)

Method: generate_flags_only
- Type: Method
- Memory Capacity: 2.0 GB
- Allow In-Memory Execution: No
- Read Only: No
- Method Limits: N/A
- Outputs Dynamic Artifacts: No
- Short Description:
  - Generate source and forecast metrics table without target flagging
- Detailed Description:
  - This method will take a source dataset as well as the dimension columns, date column name, value column name and a data frequency. With these values we will then create a source dimension metrics table with the following columns: Target Dimension Columns, Value column, Local Density, Local Density With Zeros, Global Density, Global Density With Zeros, Start Up Lag Days, Start Up Lag Periods, Collection Lag Days, Collection Lag Periods, Local Start Date, Local End Date, Summations (actuals summations), IsForecastable, Global Start Date, Global End Date and the Local Range. We will also create a forecast dimension metrics table with the following columns: Target Dimension Columns, Forecast Start Date, Date, Value, ProjectID, XperimentBuildID, XperimentKernelID, TargetName, Model, ModelCategory, ModelRank, ForecastStartDate, ForecastName, ActualsSummations, ForecastSummations, BiasError, MeanSquaredError, MeanAbsoluteError, MeanAbsolutePercentageError, PercentBiasError, RootMeanSquaredError, GrowthRate.
- Inputs:
  - Required Input
    - Source Data Definition: The source data definition.
      - Name: source_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Model Forecast Data Definition: The model forecast data definition. Expects a SensibleAI Forecast deployed model forecast table.
      - Name: model_forecast_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Data Frequency: The frequency of the source data.
      - Name: data_frequency
      - Tooltip:
        
        Detail:
        
        Keeping the frequency as auto will result in the data frequency being determined automatically.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: DataFrequency_
- Artifacts:
  - Flagging Summary HTML report: An HTML file that provides an interactive interface for the user to view the target flagging only.
    - Qualified Key Annotation: summary
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@summary/data_/html_content.html
        
        The html content.
  - Flag Definitions: A Parquet file containing the definitions of the flags that were used to flag the data.
    - Qualified Key Annotation: flags_definition
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@flags_definition/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Dimension Flags: A Parquet file containing the flags that were generated for each dimension as well as the severity levels of the flags.
    - Qualified Key Annotation: dimension_flags
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@dimension_flags/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Dimension Columns: A dictionary containing the dimension columns that were used to generate the source data dimension metrics.
    - Qualified Key Annotation: dimension_columns
    - Aggregate Artifact: False
    - In-Memory Json Accessible: True
    - File Annotations:
      - artifacts_/@dimension_columns/data_/data.json
        
        Stored json data. The schema is not known until runtime.

7. Source Metrics Analysis (Method)

Method: source_metrics_analysis
- Type: Method
- Memory Capacity: 2.0 GB
- Allow In-Memory Execution: No
- Read Only: No
- Method Limits: N/A
- Outputs Dynamic Artifacts: No
- Short Description:
  - Generate source metrics table without target flagging
- Detailed Description:
  - This method will take a source dataset as well as the dimension columns, date column name, value column name and a data frequency. With these values we will then create a source dimension metrics table with the following columns: Target Dimension Columns, Value column, Local Density, Local Density With Zeros, Global Density, Global Density With Zeros, Start Up Lag Days, Start Up Lag Periods, Collection Lag Days, Collection Lag Periods, Local Start Date, Local End Date, Summations (actuals summations), IsForecastable, Global Start Date, Global End Date and the Local Range
- Inputs:
  - Required Input
    - Source Data Definition: The source data definition.
      - Name: source_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Data Frequency: The frequency of the source data.
      - Name: data_frequency
      - Tooltip:
        
        Detail:
        
        Keeping the frequency as auto will result in the data frequency being determined by our function.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: DataFrequency_
- Artifacts:
  - Source Dimension Metrics HTML report: An HTML file that provides an interactive interface for the user to view the source dimension metrics data.
    - Qualified Key Annotation: source_data_summary
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@source_data_summary/data_/html_content.html
        
        The html content.
  - Source Data Dimension Metrics Parquet: Parquet file containing the data about the source data provided. This data contains a multitude of metrics that are used to determine the quality of the data. This Parquet includes simpler metrics such as total value, local start date, etc.
    - Qualified Key Annotation: source_data_artifact
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@source_data_artifact/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Dimension Columns: A dictionary containing the dimension columns that were used to generate the source data dimension metrics.
    - Qualified Key Annotation: dimension_columns
    - Aggregate Artifact: False
    - In-Memory Json Accessible: True
    - File Annotations:
      - artifacts_/@dimension_columns/data_/data.json
        
        Stored json data. The schema is not known until runtime.

8. Source Metrics Flagging (Method)

Method: source_metrics_flagging
- Type: Method
- Memory Capacity: 2.0 GB
- Allow In-Memory Execution: No
- Read Only: No
- Method Limits: N/A
- Outputs Dynamic Artifacts: No
- Short Description:
  - Generate source metrics table with target flagging
- Detailed Description:
  - This method will take a source dataset as well as the dimension columns, date column name, value column name and a data frequency. With these values we will then create a source dimension metrics table with the following columns: Target Dimension Columns, Value column, Local Density, Local Density With Zeros, Global Density, Global Density With Zeros, Start Up Lag Days, Start Up Lag Periods, Collection Lag Days, Collection Lag Periods, Local Start Date, Local End Date, Summations (actuals summations), IsForecastable, Global Start Date, Global End Date and the Local Range. We will also create a flag definition table with the following columns: FlagDefinitionId, FlagName, FlagDescription, MetricSource, MetricName, MetricComparator, MetricCompareValue, SeverityLevel.
- Inputs:
  - Required Input
    - Source Data Definition: The source data definition.
      - Name: source_data_definition
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: Must be an instance of Time Series Source Data
      - Nested Model: Time Series Source Data
        
        Required Input
        
        Connection: The connection to the source data.
        
        Name: data_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be an instance of Tabular Connection
        
        Nested Model: Tabular Connection
        
        Required Input
        
        Connection: The connection type to use to access the source data.
        
        Name: tabular_connection
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: Must be one of the following
        
        SQL Server Connection
        
        Required Input
        
        Database Resource: The name of the database resource to connect to.
        
        Name: database_resource
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Database Name: The name of the database to connect to.
        
        Name: database_name
        
        Tooltip:
        
        Detail:
        
        Note: If you don’t see the database name that you are looking for in this list, it is recommended that you first move the data to be used within a database that is available within this list.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Table Name: The name of the table to use.
        
        Name: table_name
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Path: The full file path to the file to ingest.
        
        Name: file_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Partitioned MetaFileSystem Connection
        
        Required Input
        
        Connection Key: The MetaFileSystem connection key.
        
        Name: connection_key
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: MetaFileSystemConnectionKey
        
        File Type: The type of files to read from the directory.
        
        Name: file_type
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: FileExtensions_
        
        Directory Path: The full directory path containing partitioned tabular files.
        
        Name: directory_path
        
        Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Dimension Columns: The columns to use as dimensions.
        
        Name: dimension_columns
        
        Tooltip:
        
        Validation Constraints:
        
        The input must have a minimum length of 1.
        
        This input may be subject to other validation constraints at runtime.
        
        Type: list[str]
        
        Date Column: The column to use as the date.
        
        Name: date_column
        
        Tooltip:
        
        Detail:
        
        The date column must in a DateTime readable format.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
        
        Value Column: The column to use as the value.
        
        Name: value_column
        
        Tooltip:
        
        Detail:
        
        The value column must be a numeric (int, float, double, decimal, etc.) column.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
        
        Type: str
    - Data Frequency: The frequency of the source data.
      - Name: data_frequency
      - Tooltip:
        
        Detail:
        
        Keeping the frequency as auto will result in the data frequency being determined by our function.
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: DataFrequency_
- Artifacts:
  - Source Dimension Metrics Flagging HTML report: An HTML file that provides an interactive interface for the user to view the source dimension metrics data.
    - Qualified Key Annotation: source_data_summary
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@source_data_summary/data_/html_content.html
        
        The html content.
  - Source Data Dimension Metrics Parquet: Parquet file containing the data about the source data provided. This data contains a multitude of metrics that are used to determine the quality of the data. This Parquet includes simpler metrics such as total value, local start date, etc.
    - Qualified Key Annotation: source_data_artifact
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@source_data_artifact/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Dimension Columns: A dictionary containing the dimension columns that were used to generate the source data dimension metrics.
    - Qualified Key Annotation: dimension_columns
    - Aggregate Artifact: False
    - In-Memory Json Accessible: True
    - File Annotations:
      - artifacts_/@dimension_columns/data_/data.json
        
        Stored json data. The schema is not known until runtime.
  - Flag Definitions: A Parquet file containing the definitions of the flags that were used to flag the data.
    - Qualified Key Annotation: flags_definition
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@flags_definition/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.
  - Dimension Flags: A Parquet file containing the flags that were generated for each dimension as well as the severity levels of the flags.
    - Qualified Key Annotation: dimension_flags
    - Aggregate Artifact: False
    - In-Memory Json Accessible: False
    - File Annotations:
      - artifacts_/@dimension_flags/data_/data_<int>.parquet
        
        A partitioned set of parquet files where each file will have no more than 1000000 rows.

9. Update Flags (Method)

Method: update_flags
- Type: Method
- Memory Capacity: 2.0 GB
- Allow In-Memory Execution: No
- Read Only: No
- Method Limits: N/A
- Outputs Dynamic Artifacts: No
- Short Description:
  - Update the flags in the flag_df based on the user inputs.
- Detailed Description:
  - In this method the users will give input on what flags they want to add, update or delete. The inputs include metric names, value to compare to, the comparator and the severity level.
- Inputs:
  - Required Input
    - Flagging Operations: The flags to update, add or delete.
      - Name: changes
      - Tooltip:
        
        Validation Constraints:
        
        This input may be subject to other validation constraints at runtime.
      - Type: list[JointOperations]
- Artifacts: No artifacts are returned by this method

Interface Definitions

No interface definitions found for this routine

Developer Docs

Routine Typename: TargetFlagRoutine

Method Name	Artifact Keys
`__init__`	N/A
`all_metrics_analysis`	source_data_artifact, forecast_data_artifact, metrics_data_summary, dimension_columns
`all_metrics_flagging`	source_data_artifact, forecast_data_artifact, metrics_data_summary, dimension_columns, flags_definition, dimension_flags
`forecast_metrics_analysis`	forecast_data_artifact, forecast_data_summary, dimension_columns
`forecast_metrics_flagging`	forecast_data_artifact, forecast_data_summary, dimension_columns, flags_definition, dimension_flags
`generate_flags_only`	summary, flags_definition, dimension_flags, dimension_columns
`source_metrics_analysis`	source_data_summary, source_data_artifact, dimension_columns
`source_metrics_flagging`	source_data_summary, source_data_artifact, dimension_columns, flags_definition, dimension_flags
`update_flags`	N/A

Versions​

v1.0.0​

Basic Information​

Tags​

Description​

Short Description​

Long Description​

Use Cases​

1. Generate All Metrics Without Flags​

2. Generate Source Metrics Without Flags​

3. Generate Forecast Metrics Without Flags​

4. Generate All Metrics With Flags​

5. Generate Source Metrics With Flags​

6. Generate Forecast Metrics With Flags​

7. Generate Flags Only​

Routine Methods​

1. Init (Constructor)​

2. All Metrics Analysis (Method)​

3. All Metrics Flagging (Method)​

4. Forecast Metrics Analysis (Method)​

5. Forecast Metrics Flagging (Method)​

6. Generate Flags Only (Method)​

7. Source Metrics Analysis (Method)​

8. Source Metrics Flagging (Method)​

9. Update Flags (Method)​

Interface Definitions​

Developer Docs​