NumericDataFillRoutine

Versions

• 1.0.0

v1.0.0

Basic Information

Class Name: NumericDataFillRoutine

Title: Numeric Data Fill

Version: 1.0.0

Author: Evan Rasmussen

Organization: OneStream

Creation Date: 2024-04-05

Default Routine Memory Capacity: 2.0 GB

Tags

Data Transformation, Interpretability, Data Preprocessing, Data Cleansing

Description

Short Description

A routine to fill missing values in a dataset.

Long Description

This routine fills the missing values in a column or columns with the specified fill strategy. The fill strategy can be one of the following: fill with zero, fill with mean, fill with median, fill with mode, fill with minimum value, fill with maximum value, fill with a custom value, fill forward, or fill backward. The forward fill strategy will fill the missing values with the last non-null value for the matching dimensions. Note that this is the last non-null value for those dimensions, not merely the most recent entry in the value column overall. The same applies to the backward fill strategy. Because of this, it is possible that the final filled values may still contain null values. The custom fill value can only be specified when the fill with custom value strategy is selected. This routine is meant to be used only on numeric columns.

Use Cases

1. Preprocess Data

Data preprocessing is a critical step in the data analysis and machine learning pipeline, serving to ensure that datasets are primed for optimal performance. A common challenge encountered during this phase is the presence of missing values, which can significantly impede the effectiveness of analytical models and algorithms. To address this issue, data preprocessing includes techniques for accurately imputing or filling missing values, thereby creating a more complete and meaningful dataset. This routine provides a range of solutions for filling null values including zero, mean, median, mode, min, max, custom value, forward fill, and backward fill. For example, one may have a dataset containing values missing from a 3 week period when a new ERP system was implemented. Using the numeric data fill routine, one may elect to fill these missing values with the mean value of the column, or with the value from the most recent entry using a forward fill. By leveraging these strategies, analysts can ensure that their datasets are complete and prepared for further analysis and modeling.

Routine Methods

1. Numeric Data Fill (Method)

• Method: numeric_data_fill

  • Type: Method

  • Memory Capacity: 2.0 GB

  • Allow In-Memory Execution: No

  • Read Only: Yes

  • Method Limits: This method has been tested with three datasets of different sizes. The first dataset with 10k targets and 1.1M rows completed in 29 seconds with 2GB of memory. The second dataset with 50k targets and 7.5M rows completed in roughly 4 minutes with 10GB of memory. The third dataset with 40k targets and 29.2M rows completed in about 5 hours with 10 GB of memory. About 15% of the rows in selected column for each dataset were to be filled.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • This routine method is used to fill null values in a dataset with the specified fill strategy.

  • Detailed Description:

    • The fill strategy can be one of the following: fill with zero, fill with mean, fill with median, fill with mode, fill with min, fill with max, fill with custom value, fill forward, or fill backward. The custom value can only be specified when the Fill with custom value strategy is selected.

  • Inputs:

    • Required Input
      • Source Data Definition: The source data definition.
        • Name: source_data_definition
        • Tooltip:
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: TimeSeriesTableDefinition
      • Data Fill Definition: List of columns and fill strategies.
        • Name: data_fill_definition
        • Tooltip:
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: NumericDataFillDefinition

  • Artifacts:

    • Filled Data Table: The data after filling in the missing values.
      • Qualified Key Annotation: filled_data
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@filled_data/data_/data_<int>.parquet
          • A partitioned set of parquet files where each file will have no more than 1000000 rows.

Interface Definitions

No interface definitions found for this routine

Developer Docs

Routine Typename: NumericDataFillRoutine

Method Name	Artifact Keys
numeric_data_fill	filled_data