TimeSeriesCleaning

Versions

• 1.0.0

v1.0.0

Basic Information

Class Name: TimeSeriesCleaning

Title: Time Series Cleaning

Version: 1.0.0

Author: Evan Rasmussen

Organization: OneStream

Creation Date: 2024-04-28

Default Routine Memory Capacity: 2.0 GB

Tags

Data Cleansing, Time Series, Anomaly, Data Preprocessing

Description

Short Description

Clean anomalies from time series data with this routine.

Long Description

This routine is used to clean certain datapoints within a time series dataset. Some of the options to clean the anomalous data include cleaning the values to zero, cleaning the values to the mean, cleaning them with a Kalman filter, or using a statistical ARIMA model to predict values. The dataframe returned from this routine includes two columns representing the original value. One column will be appended with '_cleaned' (representing the column with all new values along with original values that were not changed) and another with '_original' (representing the value column exactly as it previously was).

Use Cases

1. Clean Anomalous Data

A user recently completed a run of the Anomaly Arena Routine on their dataset and successfully identified several true anomalies. The user wishes to update these values such that they are no longer considered anomalous. The user may navigate to the cleanse page of ADR, the Time Series Anomaly Detection solution, in OneStream to configure the cleaning definitions for the identified anomalies. Since the user wishes to replace these bad datapoints with more representative values, they decide to utilize the ARIMA cleaning method for every identified anomaly (or every row of the cleaning manifest table they have configured within the solution). This ARIMA method will learn from datapoints prior to a given anomaly (identified by the cleaning manifest table) and predict a new value. Upon completion of this routine run, the user then navigates to the visualize page of ADR to analyze the results from the cleaning. It's possible the user notices some of the predictions do not align with what they expected. In this case, the user may revisit the cleaning definition page to make adjustments to the cleaning manifest table. Potential solutions may include changing the start or end dates of values to clean, changing the cleaning method, or even removing the row entirely. The user then clicks the cleanse button to kick off a new run of the Time Series Cleaning Routine. When the results are acceptable, the user can take the cleansed dataset and utilize it in their downstream processes, assured that it is a more accurate representation of their metrics.

Routine Methods

1. Clean Anomalies (Method)

• Method: clean_anomalies

  • Type: Method

  • Memory Capacity: 2.0 GB

  • Allow In-Memory Execution: No

  • Read Only: Yes

  • Method Limits: This method depends on both the source dataset and the cleaning manifest, and its runtime is primarily determined by the size of each. With 10,000 targets, 7.3 million source rows, and a manifest of ~20,000 cleaning actions, the method completes in a couple hours, but it fails when scaled to 15,000 targets, 10.95 million source rows, and ~30,000 manifest actions. All tests were conducted with 100 GB of memory. The error occurs roughly 2 hours after the data is successfully partitioned into subsets and displays as a worker lost error. Simple cleaning methods—such as mean, zero, and interpolate—run quickly, whereas statistical methods like ARIMA and Kalman are far slower and become the dominant factor in overall execution time as dataset size increases.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • A method to clean anomalies from a time series dataset.

  • Detailed Description:

    • This method takes a source data connection to clean values on and an accompanying cleaning manifest table definition containing relevant information about the values to clean and cleaning mechanisms to use. These mechanisms can be completely mixed and matched as the user sees fit. The key is that each row of the cleaning manifest table contains a value or range of values to clean, identified by date ranges and dimension values.

  • Inputs:

    • Required Input
      • Source Data Connection: The connection information source data.
        • Name: source_data_definition
        • Tooltip:
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: TimeSeriesTableDefinition
      • Cleaning Manifest Table Connection: The connection to the table defining the data to clean and the cleaning methods to apply.
        • Name: cleaning_manifest_definition
        • Long Description: This table is expected to have the following columns: 'CleaningMethod', 'StartDate', 'EndDate', 'Dim1', 'Dim2', 'Dim3', 'Dim4', 'Dim5', 'Dim6', 'Dim7', and 'Dim8'.
        • Tooltip:
          • Detail:
            • This table is expected to have the following columns: 'CleaningMethod', 'StartDate', 'EndDate', 'Dim1', 'Dim2', 'Dim3', 'Dim4', 'Dim5', 'Dim6', 'Dim7', and 'Dim8'.
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: TabularConnection
      • Dimension Column Mapping: A mapping of the 'Dim1'-'Dim8' columns in the cleaning manifest table to the source data dimension columns.
        • Name: dimension_column_mapping
        • Tooltip:
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: CleaningDimensionColumnMapping
      • Incremental Cleaning: Whether to perform incremental cleaning. If False, previously cleaned values are ignored in downstream calculations.
        • Name: incremental_cleaning
        • Long Description: Example: If set to True and a value is cleaned to the mean, this cleaned value will be included in any mean calculations to clean values from later dates, rather than the original value.
        • Tooltip:
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: Optional[bool]
      • Custom Cleaning Definitions: Custom cleaning definitions are not currently supported.
        • Name: custom_cleaning_definitions
        • Tooltip:
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: list[str]
      • Data to Return: The data to return after cleaning.
        • Name: data_to_return
        • Tooltip:
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: DateSpecificData | CleanedDataOnly | FullSourceAndCleanedData

  • Artifacts:

    • Cleaned Dataframe: The dataframe with updated values based on the cleaning definitions.

      • Qualified Key Annotation: cleaned_dataframe
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@cleaned_dataframe/data_/data_<int>.parquet
          • A partitioned set of parquet files where each file will have no more than 1000000 rows.

    • Cleaning Summarization Report: A PDF summarization report overlaying original and cleaned values.

      • Qualified Key Annotation: cleaning_report
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@cleaning_report/data_/document.pdf
          • A pdf variant of the html file. Please note the interactivity that may be found in the html is lost within the pdf variant.
        • artifacts_/@cleaning_report/data_/html_content.html
          • The html content.

Interface Definitions

No interface definitions found for this routine

Developer Docs

Routine Typename: TimeSeriesCleaning

Method Name	Artifact Keys
clean_anomalies	cleaned_dataframe, cleaning_report