Skip to main content

Routine Storage Structure

Author: Drew Shea, Created: 2025-06-23

This article outlines key information about how Routine data is stored within the Routine MetaFileSystem.

The Routine system that powers SensibleAI Studio is built on storing all data and metadata related to Routine Instances and Routine Runs in the file-based storage system, MetaFileSystem.

Both the Xperiflow AI engine and OneStream have the ability to programmatically interact with the MetaFileSystem.

info

There are two personas that will be referred to throughout this document

  • Routine Developers: Personas that actually write and develop Routines (from inside of the AI Engine)
  • Power Users: Personas that programmatically interact with routines (from inside of OneStream)

Routine Instance Directory Structure​

When a Routine Instance is created and one or Routine Runs are created, the directory structure within the “Shared” MetaFileSystem will look like this.

instances_/
|-- <instance_name>-<rand_chars>/
|  |-- internalvars_/
|  |-- shared_/
|  |-- runs_/
|  |  |-- <run_name>-<rand_chars>/
|  |  |  |-- shared_/
|  |  |  |-- artifacts_/
|  |  |  |  |-- <artifact_key>/
|  |  |  |  |  |-- data_/ ...
|  |  |  |  |  |-- metadata_/ ...
|  |  |  |  |  |-- previews_/ ...
|  |  |  |  |  |-- statistics_/ ...
info

Key to the directory structure:

  • <var> = a variable portion of a name
  • <rand_chars> = Randomly generated characters
  • … = Directory likely has additional contents within it
note

With the exception of shared_ directories, each directory within the Routine MetaFileSystem is read-only. This means that only xperiflow (the system itself) can write to these directory locations.

Power Users can ONLY write data to the shared_ directories.

Routine Instance​

When a Routine Instance is created, a new folder will be created within the instances_ folder. The name of the folder will be of max 20 characters of the form <routine_instance_name>-<random_chars>, where:

  • Your routine instance name will be truncated after 13 characters
  • random_chars is of length 6

Within this newly created folder, three folders will be created: internalvars_, runs_, and shared_

Example:

  • Instance Name = “My Routine Name”
instances_/
|-- my-routine-na-39jfm1/
|  |-- internalvars_/ ...
|  |-- shared_/
|  |-- runs_/ ...
info

This structure gets generated BEFORE you run the constructor of the Routine Instance. This is useful for situations where you want to store the inputs that are going to be necessary for the Routine Instance in the shared_ directory.

internalvars_​

The internalvars_ directory holds additional sub-folders for each member variable (MemberVar) defined within the Routine class.

The Routine Instance itself will write data here. The Routine Developer may make use of this folder to store internal state data that is needed to make the Routine Instance function properly.

This data should not be accessed from OneStream (by Power Users) as this data is used to maintain internal state of the Routine Instance.

shared_​

The shared_ directory is a place where Power Users can read and write any file-based data they want as it relates to the Routine Instance. In other words, this is the “instance-level” shared directory.

tip

This makes it easy to “self-contain” data that needs to exist only for same duration for the existence of the routine itself.

Below are some common use cases for storing data in the shared_ Routine Instance directory:

  • Storing Tabular Data Inputs: Many of the Routines available require some form of tabular data. You can write C# DataTable objects as partitioned parquet files to the shared_ directory and then “reference” this data for the Routine Run inputs that require that tabular data.
    • Why: This allows you to “self-contain” the inputs directly on the Routine Instance. That way, if you ever need to delete this Routine Instance, the data will also get deleted and don’t “pollute” your database or other areas of the MetaFileSystem with “orphaned” data.

runs_​

The runs_ directory holds all the data and metadata generated for each Routine Run that was executed. There is more on this section below.

Routine Run​

When a Routine Run is created, a new directory will be created within the Routine Instances runs_ folder. The name of the folder will be of max 20 characters, but of the form **<routine_run_name>-<random_chars>**.

Within this newly created folder, two folders will be created: artifacts_, and shared_.

Example:

  • Instance Name = “My Routine Name”
  • First Routine Run Name = “First Run”
  • Second Routine Run Name = “Second Run”
instances_/
|-- my-routine-na-39jfm1/
|  |-- internalvars_/ ...
|  |-- shared_/
|  |-- runs_/
|  |  |-- first-run-bndiw81/
|  |  |  |-- shared_/
|  |  |  |-- artifacts_/ ...
|  |  |-- second-run-ew7190c/
|  |  |  |-- shared_/
|  |  |  |-- artifacts_/ ...

shared_​

The shared_ directory is a place where Power Users can read and write any file-based data they want as it relates to the Routine Run. In other words, this is a “run-level” shared directory.

tip

This makes it easy to “self-contain” data that needs to exist only for same duration for the existence of the run itself.

Below are some common use cases for storing data in the shared_ Routine Instance directory:

  • Storing Tabular Data Inputs: Many of the Routines available require some form of tabular data. You can write C# DataTable objects as partitioned parquet files to the shared_ directory and then “reference” this data for a specific Routine Run input that require that tabular data.
    • Why: This allows you to “self-contain” the inputs directly on the Routine Run. That way, if you ever need to delete this Routine Run, the data will also get deleted and don’t “pollute” your database or other areas of the MetaFileSystem with “orphaned” data.

artifacts_​

The artifacts_ directory holds information about the Routine Run’s Artifact. There is more on this section below.

Routine Artifacts​

When Routine methods create artifacts, the artifacts_ directory will be populated. It will hold one or more folders, depending on the Artifacts returned by the Routine method. Each attribute defined on the Artifact will result in a new directory, with that Artifact Key Annotation as the name of the directory.

For example, many of the time series related anomaly detectors return the following artifacts (as defined within the Routine Developer code):

class AnomalyDetectionArtifacts(ArtifactBaseModel):
    anomaly_snapshot: ArtifactDef[PolarsDataFrame[AnomalySnapshotArtifactSchema], PolarsParquetArtifactIOFactory] = ArtifactMetadataDef(
        title="AnomalySnapshot",
        description="Parquet file containing data about your anomaly detection run.",
    )
    anomaly_dates: ArtifactDef[PolarsDataFrame[AnomalyDatesArtifactSchema], PolarsParquetArtifactIOFactory] = ArtifactMetadataDef(
        title="Specific Anomaly Dates",
        description="Parquet file containing data about the specific dates an anomaly was detected.",
    )
    anomaly_instance: ArtifactDef[PolarsDataFrame[AnomalyInstanceArtifactSchema], PolarsParquetArtifactIOFactory] = ArtifactMetadataDef(
        title="Specific Anomaly Instances",
        description="Parquet file containing data about the specific anomaly instances that were detected.",
    )

This results in the following sub-folder being created under the Routine Run’s artifacts_ directory. For example:

instances_/
|-- my-routine-na-39jfm1/
|  |-- internalvars_/ ...
|  |-- shared_/
|  |-- runs_/
|  |  |-- first-run-bndiw81/
|  |  |  |-- shared_/
|  |  |  |-- artifacts_/
|  |  |  |  |-- anomaly_snapshot/ ...
|  |  |  |  |-- anomaly_dates/ ...
|  |  |  |  |-- anomaly_instance/ ...

Now, within each Artifact Key Annotation sub-directory, four more directories are created which are data_, metadata_, previews_, and statistics_.

metadata_​

The metadata_ directory holds the artifact_metadata.json file which carries metadata information about the Artifact.

See an example artifact_metadata.json file below:

artifact_metadata.json
{
    "key": "report_content",
    "qualified_key": "report_content",
    "title": "Data Analysis Report",
    "description": "An HTML report presenting analysis of a tabular dataset",
    "path": "instances_/data-analysis-lwkx7b/runs_/asdg-1tty62/artifacts_/report_content",
    "routine_instance_id": "data-analysis-lwkx7b",
    "routine_run_id": "asdg-1tty62",
    "routine_type_name": "SourceDataAnalysis",
    "method_name": "source_data_analysis",
    "is_aggregate": false,
    "creation_time": "2025-07-20T19:15:10.020942",
    "artifact_python_data_type": "builtins.str",
    "io_factory_typename": "HtmlArtifactIOFactory",
    "io_factory_kwargs": {},
    "io_factory_serialized_instance_data": {
        "serialized_metadata": {
            "version_number": "0.0.1",
            "object_metadata": {
                "class_name": "HtmlArtifactIOFactory",
                "import_path": "HtmlArtifactIOFactory",
                "mro": [
                    "HtmlArtifactIOFactory",
                    "IArtifactIOFactory",
                    "ISerializerMixin",
                    "ABC",
                    "object"
                ]
            },
            "serializer_typename": "DictSerializer"
        },
        "serialized_object": {
            "include_pdf": false,
            "pdfkit_options": {}
        }
    },
    "artifact_annotation": {
        "file_annotations": [
            {
                "file_annotation": "artifacts_/@report_content/data_/html_content.html",
                "file_annotation_description": "The html content."
            }
        ],
        "key_annotation": "report_content",
        "title": "Data Analysis Report",
        "description": "An HTML report presenting analysis of a tabular dataset",
        "is_aggregate": false,
        "is_artifact_json_serializable": false
    },
    "statistic_factory_typename": null,
    "statistic_factory_kwargs": null,
    "statistic_factory_serialized_instance_data": null,
    "preview_factory_typename": "HtmlArtifactPreviewFactory",
    "preview_factory_kwargs": null,
    "preview_factory_serialized_instance_data": {
        "serialized_metadata": {
            "version_number": "0.0.1",
            "object_metadata": {
                "class_name": "HtmlArtifactPreviewFactory",
                "import_path": "HtmlArtifactPreviewFactory",
                "mro": [
                    "HtmlArtifactPreviewFactory",
                    "IArtifactPreviewFactory",
                    "ISerializerMixin",
                    "ABC",
                    "object"
                ]
            },
            "serializer_typename": "DictSerializer"
        },
        "serialized_object": {
            "include_pdf": false,
            "pdfkit_options": {}
        }
    }
}

previews_​

Summary information about an Artifact is written here. This is meant to provide some additional detail about the Artifact.

This may include describe and top_n_rows sub-directories, and any additional Artifact Previews implemented by the Routine Developer.

info

It is possible that this directory may not be populate if either:

  1. This Artifact Type does not generate Artifact Previews
  2. The Power User turned off Artifact Previews for this specific Routine Run.

statistics_​

Additional summary statistics about the Artifact.

info

It is possible that this directory may not be populate if either:

  1. This Artifact Type does not generate Artifact Statistics
  2. The Power User turned off Artifact Statistics for this specific Routine Run.

data_​

This is where the actual Artifact for the given Artifact Key Annotation is written. The type of data found within the data_ directory is entirely dependent on type of Artifact.

For example, an Artifact that returns tabular data is likely written as partitioned parquet files in the form of data_/data_<partition_id>.parquet.

tip

Check out the File Annotations that exist for within the Routine Documentation found in SensibleAI Studio for a given Routine Method for a given Artifact to know what the file data will look like.

For example, we can go look at the “Forecast Allocation” Routine and take a look at the Signature → Simple Spread” Routine Method definition and scroll down to “Artifacts” and see the following:

alt text

As a result, if we were to run this “Simple Spread” Routine, we could expect our Routine Instance storage structure to look like the following:

instances_/
|-- my-routine-na-39jfm1/
|  |-- internalvars_/
|  |-- shared_/
|  |-- runs_/
|  |  |-- first-run-bndiw81/
|  |  |  |-- shared_/
|  |  |  |-- artifacts_/
|  |  |  |  |-- output_data/
|  |  |  |  |  |-- data_/
|  |  |  |  |  |  |-- data_0.parquet  // Fits up to 1M records per partition
|  |  |  |  |  |  |-- data_1.parquet  // Fits up to 1M records per partition
|  |  |  |  |  |-- metadata_/
|  |  |  |  |  |  |-- artifact_metadata.json
|  |  |  |  |  |-- statistics_/ ...
|  |  |  |  |  |-- previews_/ ...

It’s also important to note that when you have the situation where you have Nested Artifacts (Artifacts inside of Artifacts), there will be another Artifact found within the outermost data_ directory.

As an example, the “Anomaly Arena Routine” has Nested Artifacts.

alt text Anomaly Arena Routine - Signature → Predict → Artifacts showcasing Nested Artifacts

Notice how in the image above, the documentation showcases how the “Anomaly Detection Artifacts” has inner Artifacts like “AnomalySnapshot”, “Specific Anomaly Dates” (not shown in screenshot) and “Specific Anomaly Instances” (not shown in screenshot).

If we were to run the “Anomaly Arena Routine” Run Method of “Predict”, we could expect our storage structure to look like the following.

instances_/
|-- my-anomaly-ar-2ng311/
|  |-- internalvars_/
|  |-- shared_/
|  |-- runs_/
|  |  |-- my-predict-abk1df/
|  |  |  |-- shared_/
|  |  |  |-- artifacts_/
|  |  |  |  |-- anomaly_artifacts/  # Outer Artifact
|  |  |  |  |  |-- data_/
|  |  |  |  |  |  |-- anomaly_snapshot/  # Inner Artifact
|  |  |  |  |  |  |  |-- data_/
|  |  |  |  |  |  |  |  |-- data_0.parquet
|  |  |  |  |  |  |  |-- metadata_/
|  |  |  |  |  |  |  |  |-- artifact_metadata.json
|  |  |  |  |  |  |  |-- statistics_/ ...
|  |  |  |  |  |  |  |-- previews_/ ...
|  |  |  |  |  |  |-- anomaly_dates/  # Inner Artifact
|  |  |  |  |  |  |  |-- data_/
|  |  |  |  |  |  |  |  |-- data_0.parquet
|  |  |  |  |  |  |  |-- metadata_/
|  |  |  |  |  |  |  |  |-- artifact_metadata.json
|  |  |  |  |  |  |  |-- statistics_/ ...
|  |  |  |  |  |  |  |-- previews_/ ...
|  |  |  |  |  |  |-- anomaly_instance/  # Inner Artifact
|  |  |  |  |  |  |  |-- data_/
|  |  |  |  |  |  |  |  |-- data_0.parquet
|  |  |  |  |  |  |  |-- metadata_/
|  |  |  |  |  |  |  |  |-- artifact_metadata.json
|  |  |  |  |  |  |  |-- statistics_/ ...
|  |  |  |  |  |  |  |-- previews_/ ...
|  |  |  |  |  |-- metadata_/
|  |  |  |  |  |  |-- artifact_metadata.json
|  |  |  |  |  |-- statistics_/ ...
|  |  |  |  |  |-- previews_/ ...
Last updated on by Jeremy Meiss

Was this page helpful?