FeatureEngineeringAnalysis

Versions

• 2.0.0

v2.0.0

Basic Information

Class Name: FeatureEngineeringAnalysis

Title: Feature Engineering Analysis

Version: 2.0.0

Author: Kendall Haddigan

Organization: OneStream

Creation Date: 2025-09-23

Default Routine Memory Capacity: 2.0 GB

Tags

Data Preprocessing, Data Transformation

Description

Short Description

Comprehensive feature engineering and data preprocessing routine.

Long Description

This routine provides end-to-end feature engineering capabilities including null value handling, categorical encoding, numerical normalization, and transformation tracking. It supports various preprocessing strategies and maintains transformation metadata for reproducible data pipelines.

Use Cases

1. Data Preprocessing Pipeline

Use this routine to clean and prepare raw data for machine learning models and analytics workflows. It provides comprehensive data preprocessing capabilities including intelligent null value handling using statistical methods like mean, median, and mode for numerical data, and categorical mode for text data. The routine automatically detects feature types and applies appropriate transformations, ensuring data quality and consistency. It handles missing values systematically, encodes categorical variables using various methods, normalizes numerical features for optimal model performance, and tracks all transformations for reproducible preprocessing pipelines. This makes it ideal for preparing training and testing datasets with consistent preprocessing steps, enabling reliable model development and deployment workflows.

2. Feature Engineering Automation

Automate the feature engineering process for data science workflows by leveraging advanced transformation techniques and intelligent feature type detection. This routine streamlines the creation of engineered features that improve model performance through systematic data transformation processes. It automatically analyzes feature characteristics, determines optimal transformation strategies, and applies sophisticated preprocessing techniques including robust scaling, min-max normalization, and categorical encoding methods. The routine maintains detailed transformation metadata and history, enabling reproducible feature engineering pipelines and consistent data processing across different datasets. It supports complex workflows where multiple transformations need to be applied in sequence, tracks the order of operations for reversibility, and ensures data lineage throughout the transformation process, making it essential for production-ready machine learning systems.

Routine Methods

1. Init (Constructor)

• Method: __init__

  • Type: Constructor

  • Memory Capacity: 2.0 GB

  • Allow In-Memory Execution: No

  • Read Only: No

  • Method Limits: Memory usage scales with dataset size. For a dataset with 2M rows and 4 feature columns, this method is expected to complete with ~5GB of memory allocated. For 5M rows and 3 feature columns, ~12GB of memory allocated. For 10M rows and 3 feature columns, ~25GB of memory allocated.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • Initialize the feature engineering routine with actual data.

  • Detailed Description:

    • Analyzes and stores metadata about the data that must hold true over the routine lifecycle.

  • Inputs:

    • Required Input
      • Data Source Selection: Data source to transform.
        • Name: datasource
        • Tooltip:
          • Detail:
            • Select the data source for feature engineering
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: TabularConnection
      • Dimension Selection: Dimension columns for row tracking.
        • Name: dimensions
        • Tooltip:
          • Detail:
            • Select columns that uniquely identify rows
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: list[str]
      • Feature Selection: Feature columns to transform.
        • Name: feature_columns
        • Tooltip:
          • Detail:
            • Select the columns to apply transformations to
          • Validation Constraints:
            • The input must have a minimum length of 1.
            • This input may be subject to other validation constraints at runtime.
        • Type: list[str]

  • Artifacts: No artifacts are returned by this method

2. Encode (Method)

• Method: encode

  • Type: Method

  • Memory Capacity: 2.0 GB

  • Allow In-Memory Execution: No

  • Read Only: No

  • Method Limits: Memory usage scales with dataset size. For a dataset with 2M rows, this method is expected to complete in around 10 seconds with 2GB of memory allocated. For 5M rows, this method is expected to complete in around 8 seconds with 5GB of memory allocated. For 10M rows, memory requirements are expected to scale proportionally with approximately 10GB of memory allocated.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • Encode categorical features using the specified encoding methods.

  • Detailed Description:

    • This method transforms categorical features into numerical representations that can be used by machine learning algorithms. It supports various encoding strategies including label encoding, one-hot encoding, and target encoding. The method automatically identifies categorical features and applies the chosen encoding methods sequentially while preserving the ability to reverse the transformation later. Supports multiple sequential encoding operations in a single step (e.g., label encode then cosine/sine encode).

  • Inputs:

    • Required Input
      • Configure Encoding Method: Configure multiple encoding operations. Add configurations to apply encodings in sequence (e.g., label encode then cosine/sine encode).
        • Name: encoding_configs
        • Tooltip:
          • Detail:
            • Add encoding configurations to apply in sequence. Maximum of 2 encoding steps allowed (e.g., label then one-hot or cosine/sine).
          • Validation Constraints:
            • The input must have a maximum length of 2.
            • This input may be subject to other validation constraints at runtime.
        • Type: list[FeatureEncodingConfig]

  • Artifacts:

    • Encoded Data: DataFrame with categorical features encoded using the specified method.

      • Qualified Key Annotation: transformed_data
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@transformed_data/data_/data_<int>.parquet
          • A partitioned set of parquet files where each file will have no more than 1000000 rows.

    • Encode Column Mapping: Maps each original feature name to the list of column names it produced after encoding. Identity (1:1) for label/ordinal; one-to-many for one-hot and cosine/sine.

      • Qualified Key Annotation: column_mapping
      • Aggregate Artifact: False
      • In-Memory Json Accessible: True
      • File Annotations:
        • artifacts_/@column_mapping/data_/data.json
          • Stored json data. The schema is not known until runtime.

3. Fill Null (Method)

• Method: fill_null

  • Type: Method

  • Memory Capacity: 2.0 GB

  • Allow In-Memory Execution: No

  • Read Only: No

  • Method Limits: Memory usage scales with dataset size. For a dataset with 2M rows, this method is expected to complete in around 8 seconds with 2GB of memory allocated. For 5M rows, this method is expected to complete in around 11 seconds with 5GB of memory allocated. For 10M rows, this method is expected to complete in around 14 seconds with 10GB of memory allocated.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • Fill null values in the dataset using the specified statistical method.

  • Detailed Description:

    • This method processes the transformed dataset to identify and fill null values across all feature columns using the specified filling method. It supports various strategies including mean, median, mode, zero, and none (no filling). The method automatically detects whether features are categorical or numerical and applies appropriate filling strategies. All transformations are tracked in the feature column metadata to maintain a complete history of applied operations for reproducibility and potential reversal.

  • Inputs:

    • Required Input
      • Configure Fill Null Method: Configure multiple fill null operations. Add configurations to apply in sequence (e.g., mode for categorical, mean for numerical).
        • Name: fill_null_configs
        • Tooltip:
          • Detail:
            • Add fill null configurations to apply transformations in sequence.
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: list[FillNullConfig]

  • Artifacts:

    • Transformed Data: DataFrame with null values filled using the specified method.
      • Qualified Key Annotation: transformed_data
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@transformed_data/data_/data_<int>.parquet
          • A partitioned set of parquet files where each file will have no more than 1000000 rows.

4. Inverse Transform (Method)

• Method: inverse_transform

  • Type: Method

  • Memory Capacity: 2.0 GB

  • Allow In-Memory Execution: No

  • Read Only: Yes

  • Method Limits: Memory usage scales with dataset size. For a dataset with 2M rows and 4 feature columns, this method is expected to complete in around 8 seconds with 5GB of memory allocated. For 5M rows and 3 feature columns, this method is expected to complete in around 8 seconds with 15GB of memory allocated. For 10M rows and 3 feature columns, this method is expected to complete in around 4-22 seconds with 20GB of memory allocated.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • Reverse all applied transformations to return data to its original form.

  • Detailed Description:

    • This method undoes all feature engineering transformations that have been applied to the transformed dataset, returning it to its original state. It processes the transformation history in reverse order, applying inverse operations for normalization, encoding, and null filling. This capability is essential for interpreting model results in terms of the original data features and values.

  • Inputs:

    • Required Input
      • Transformed Data Source: Transformed dataset to reverse transformations on.
        • Name: transformed_datasource
        • Tooltip:
          • Detail:
            • Select the transformed data source to reverse transformations
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: TabularConnection

  • Artifacts:

    • Inverse Transformed Data: DataFrame with transformations reversed to original form.
      • Qualified Key Annotation: original_data
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@original_data/data_/data_<int>.parquet
          • A partitioned set of parquet files where each file will have no more than 1000000 rows.

5. Normalize (Method)

• Method: normalize

  • Type: Method

  • Memory Capacity: 2.0 GB

  • Allow In-Memory Execution: No

  • Read Only: No

  • Method Limits: Memory usage scales with dataset size. For a dataset with 2M rows, this method is expected to complete in around 5 seconds with 2GB of memory allocated. For 5M rows, this method is expected to complete in around 15 seconds with 5GB of memory allocated. For 10M rows, this method is expected to complete in around 1 second with 10GB of memory allocated.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • Normalize numerical features using the specified normalization method.

  • Detailed Description:

    • This method scales numerical features to ensure they have similar ranges and distributions, which improves the performance and stability of machine learning algorithms. It supports various normalization techniques including standard scaling, min-max scaling, robust scaling, and max-abs scaling. The method automatically identifies numerical features and applies the chosen normalization while maintaining transformation metadata for reversibility.

  • Inputs:

    • Required Input
      • Configure Normalization Method: Configure multiple normalization operations. Add configurations to apply in sequence.
        • Name: normalize_configs
        • Tooltip:
          • Detail:
            • Add normalization configurations to apply transformations in sequence.
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: list[NormalizeConfig]

  • Artifacts:

    • Normalized Data: DataFrame with features normalized using the specified method.
      • Qualified Key Annotation: transformed_data
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@transformed_data/data_/data_<int>.parquet
          • A partitioned set of parquet files where each file will have no more than 1000000 rows.

6. Transform (Method)

• Method: transform

  • Type: Method

  • Memory Capacity: 2.0 GB

  • Allow In-Memory Execution: No

  • Read Only: Yes

  • Method Limits: This method has been tested with various dataset sizes and feature column configurations. For a dataset with 2M rows and 3 feature columns configured in the constructor method, this method completed in approximately 1 minute with 10 GB of memory allocated, completing 3 distinct transformations for each feature column. For a dataset with 5M rows and 3 feature columns configured in the constructor method, this method completed in approximately 1 minute with 20 GB of memory allocated, completing 3 distinct transformations for each feature column. For a dataset with 10M rows and 3 feature columns configured in the constructor method, this method completed in approximately 3 minutes with 25 GB of memory allocated, completing 3 distinct transformations for each feature column.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • Apply all configured transformations to new data using learned transformation parameters.

  • Detailed Description:

    • This method takes new data and applies the previously learned transformations to transform it consistently with the original training data. It ensures that the new data has the same structure and feature types as the original data, validating compatibility before applying transformations.

  • Inputs:

    • Required Input
      • New Data Source: New dataset to apply transformations to.
        • Name: new_datasource
        • Tooltip:
          • Detail:
            • Select the new data source to transform using learned transformations
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: TabularConnection

  • Artifacts:

    • Fully Transformed Data: DataFrame with all transformations applied in sequence.
      • Qualified Key Annotation: transformed_data
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@transformed_data/data_/data_<int>.parquet
          • A partitioned set of parquet files where each file will have no more than 1000000 rows.

7. Weight (Method)

• Method: weight

  • Type: Method

  • Memory Capacity: 2.0 GB

  • Allow In-Memory Execution: No

  • Read Only: No

  • Method Limits: Memory usage scales with dataset size. For a dataset with 2M rows, this method is expected to complete in around 5 seconds with 2GB of memory allocated. For 5M rows, this method is expected to complete in around 12 seconds with 5GB of memory allocated. For 10M rows, this method is expected to complete in around 17 seconds with 10GB of memory allocated.

  • Outputs Dynamic Artifacts: No

  • Short Description:

    • Apply feature weighting to adjust the importance of multiple features.

  • Detailed Description:

    • This method processes a list of feature weights collected via the Add/Continue workflow pattern. Each feature weight is applied to multiply the feature values and stored as metadata for downstream algorithms (such as clustering) to use for emphasizing or de-emphasizing features.

  • Inputs:

    • Required Input
      • Configure Feature Weighting: Configure weights for multiple features.
        • Name: feature_weighting
        • Tooltip:
          • Detail:
            • Add feature weights to adjust importance in analysis.
          • Validation Constraints:
            • This input may be subject to other validation constraints at runtime.
        • Type: list[FeatureEngineeringWeightingV2]

  • Artifacts:

    • Weighted Data: Dataset with weighted features

      • Qualified Key Annotation: transformed_data
      • Aggregate Artifact: False
      • In-Memory Json Accessible: False
      • File Annotations:
        • artifacts_/@transformed_data/data_/data_<int>.parquet
          • A partitioned set of parquet files where each file will have no more than 1000000 rows.

    • Feature Weights: Dictionary of feature names and their weights

      • Qualified Key Annotation: feature_weights
      • Aggregate Artifact: False
      • In-Memory Json Accessible: True
      • File Annotations:
        • artifacts_/@feature_weights/data_/data.json
          • Stored json data. The schema is not known until runtime.

Interface Definitions

No interface definitions found for this routine

Developer Docs

Routine Typename: FeatureEngineeringAnalysis

Method Name	Artifact Keys
__init__	N/A
encode	transformed_data, column_mapping
fill_null	transformed_data
inverse_transform	original_data
normalize	transformed_data
transform	transformed_data
weight	transformed_data, feature_weights