Skip to main content

PrincipalComponentAnalysisRoutine

Versions​

v1.0.0​

Basic Information​

Class Name: PrincipalComponentAnalysisRoutine

Title: Principal Component Analysis

Version: 1.0.0

Author: Josh Liu

Organization: OneStream

Creation Date: 2024-07-15

Tags​

ML, Time Series, Data Transformation, Dimensionality Reduction, Feature Generation

Description​

Short Description​

The Principal Component Analysis algorithm for dimensionality reduction.

Long Description​

Principal Component Analysis is a statistical method leveraged for dimensionality reduction, data compression, and feature extraction. The goal of PCA is to capture the greatest variances in the data along the axes which are called principal components. This transformation of the data via PCA reduces the dimensionality of the data while retaining as much variability as possible.

Use Cases​

1. Anomaly Detection in Fraudulent Transactions​

In time series data, anomalies are data points that deviate significantly from the normal pattern of the data. Detecting fraudulent transactions involves identifying unusual patterns in transaction volumes or values. PCA can be used to reduce the dimensionality of the data to identify the principal components that capture the majority of the variability. With this transformation of the data, data points that do not fit well within this reduced-dimensional space can be identified as anomalous data.

2. Forecasting Electricity Demand​

Forecasting electricity demand involves data with various features such as temperature, day of the week, time of day, and historical usage patterns. Each independent variable may contribute to the overall pattern in different ways. PCA can identify the most significant components that explain the majority of the variance in the data. This will reduce the complexity of the forecasting model. The principal components derived from PCA can be used as input features for forecasting models such as ARIMA, neural networks, and other regression models.

Routine Methods​

1. Run PCA (Method)​
  • Method: run_pca

    • Type: Method

    • Memory Capacity: 2.0 GB

    • Allow In-Memory Execution: No

    • Read Only: Yes

    • Method Limits: This method has been tested with various datasets. With a dataset containing 5K targets and 550K rows, this method completed in 1 minute with 2 GB of memory allocated. With a dataset containing 10K targets and 1.1M rows, this method completed in 1 minute with 2 GB of memory allocated. With a dataset containing 40K targets and 29M rows, this method completed in 2 minutes with 10 GB of memory allocated. With a dataset containing 15K targets and 7.5M rows, this method completed in 1 minute with 5 GB of memory allocated.

    • Outputs Dynamic Artifacts: No

    • Short Description:

      • Main method for the Principal Component Analysis routine.

    • Detailed Description:

      • This method will deseasonalize the data, scale the data, and run Principal Component Analysis.

    • Inputs:

      • Required Input
        • Source Data Definition: The source data definition.
          • Name: source_data_definition
          • Tooltip:
            • Validation Constraints:
              • This input may be subject to other validation constraints at runtime.
          • Type: TimeSeriesTableDefinition
        • Number of Principal Components: Choose to select a single number of components value or a range of number of components values.
          • Name: n_components_option
          • Tooltip:
            • Detail:
              • Recommend 2-3 principal components if the goal is data visualization. Number of principal components must be less than number of dataset dimensions.
            • Validation Constraints:
              • This input may be subject to other validation constraints at runtime.
          • Type: SingleNumComponents | RangeNumComponents
        • advanced_pca_parameters: Specify optional advanced PCA settings.
          • Name: set_advanced_settings
          • Tooltip:
            • Detail:
              • If set to False, advanced settings will be set to default values. For more information on PCA advanced settings, read scikit learn PCA documentation.
            • Validation Constraints:
              • This input may be subject to other validation constraints at runtime.
          • Type: Optional[AdvancedPcaParameters]
        • Data Stationarity: Specify if the time series data is considered stationary.
          • Name: stationarity_data_state
          • Tooltip:
            • Detail:
              • A time series dataset is considered stationary if its statistical properties, such as the mean or variance, do not change over time. Non-stationary time series data statistical properties change over time.
            • Validation Constraints:
              • This input may be subject to other validation constraints at runtime.
          • Type: bool
        • Perform Seasonal Decomposition: Specify if the time series data should be seasonally decomposed.
          • Name: seasonal_decomposition
          • Tooltip:
            • Detail:
              • Seasonal decomposition is a method of stationarizing the dataset by deseasonalizing. Many statistical and machine learning models often assume data is stationary.
            • Validation Constraints:
              • This input may be subject to other validation constraints at runtime.
          • Type: bool
        • Data Scaling Method: Specify the type of scaling or standardization to transform the dataset.
          • Name: scale_standardize_data
          • Tooltip:
            • Detail:
              • Data scaling/standardization is recommended if difference in scale between dimension values are large.
            • Validation Constraints:
              • This input may be subject to other validation constraints at runtime.
          • Type: ScaleType_

    • Artifacts:

      • Preprocessed Dataframe: The state of the data after preprocessing. This is the state of the data that is input into the PCA model.

        • Qualified Key Annotation: preprocessed_data
        • Aggregate Artifact: False
        • In-Memory Json Accessible: False
        • File Annotations:
          • artifacts_/@preprocessed_data/data_/data_<int>.parquet
            • A partitioned set of parquet files where each file will have no more than 1000000 rows.

      • Principal Components Data: Dataset containing timepoints, value column, and principal component columns.

        • Qualified Key Annotation: principal_component_data
        • Aggregate Artifact: False
        • In-Memory Json Accessible: False
        • File Annotations:
          • artifacts_/@principal_component_data/data_/data_<int>.parquet
            • A partitioned set of parquet files where each file will have no more than 1000000 rows.

      • Components Dataframe: The directions of maximum variance in the data.

        • Qualified Key Annotation: components_data
        • Aggregate Artifact: False
        • In-Memory Json Accessible: False
        • File Annotations:
          • artifacts_/@components_data/data_/data_<int>.parquet
            • A partitioned set of parquet files where each file will have no more than 1000000 rows.

      • PCA Report: A comprehensive PDF report of the dataset along with the HTML content used to generate the PDF.

        • Qualified Key Annotation: pca_report
        • Aggregate Artifact: False
        • In-Memory Json Accessible: False
        • File Annotations:
          • artifacts_/@pca_report/data_/document.pdf
            • A pdf variant of the html file. Please note the interactivity that may be found in the html is lost within the pdf variant.
          • artifacts_/@pca_report/data_/html_content.html
            • The html content.

Interface Definitions​

No interface definitions found for this routine

Developer Docs​

Routine Typename: PrincipalComponentAnalysisRoutine

Method NameArtifact Keys
run_pcapreprocessed_data, principal_component_data, components_data, pca_report
Last updated on by jrobinsonDSS

Was this page helpful?