MetaFileSystemClient SDK

Author: Drew Shea, Created: 2024-02-26

The MetaFileSystemClient is an API wrapper interface around the Python-based Xperiflow MetaFileSystem module that allows OneStream developers and consultants to read file-based data from the Xperiflow Backend Storage system.

This MetaFileSystemClient SDK is written in Csharp and is a module within the Xperiflow Business Rules Solution.

note

Be sure to pay close attention to the difference between MetaFileSystem and MetaFileSystemClient.

MetaFileSystem = The Python-based Xperiflow backend module that manages all stored files in Xperiflow

MetaFileSystemClient = The C# Class that wraps the MetaFileSystem APIs that allows OneStream developers and consultants to read files and folders from the Xperiflow MetaFileSystem

Qualities

The MetaFileSystemClient exhibits the following attributes for OneStream developers and consultants:

Easy to Use

Given that this Class is expected to be used by consultants who may not have rich experience with C#, this MetaFileSystemClient must be intuitive and very well-documented.

Configurable

Given that the Xperiflow MetaFileSystem can store files in multiple different storage backends, it is important to notice that there are multiple different instances of the MetaFileSystem within Xperiflow. Notably, there are:

• Project Level MetaFileSystem: Files and Folders existing for a specific SensibleAI Forecast Project.

• Framework Level MetaFileSystem: Files and Folders existing at the overall “AI Services” level.

You can choose which MetaFileSystem you talk to using the MetaFileSystem parameter metadataConnectionKey. See more details on this metadataConnectionKey below.

Rich Data Converter Helpers

A considerable part of this MetaFileSystemClient being successful is the rich set of file-to-datatype converters. There is a rich set of functions that can easily plug and play with the MetaFileSystemClient to convert data to different formats returned like:

• .csv file to OneStream Database Table

• .parquet file to OneStream Database Table

• .parquet file to System.Data.DataTable object

• .parquet file to Microsoft.Data.Analysis.DataFrame object

• .json file to string object

• and many more

Posix Path Naming

The MetaFileSystem uses posix file path conventions (aka forward-slashes) to separate file paths (ex: fileshare/my/folder/my_data.txt) instead of windows files paths which uses backward slashes.

Note, there is a PathUtils class that exists within the MetaFileSystem namespace that can be used to help clean and parse file paths for the MetaFileSystem.

Use Cases

SensibleAI Forecast Data Pipelines

As of the v3.1.0 Xperiflow release, the MetaFileSystem can now hold SensibleAI Forecast data that could be useful in supporting future SensibleAI Forecast Data Pipelines for customers. Although at the time of writing 15 Dec 2023, there are no such datasets, in the future, new data that may be accessible or included in the SensibleAI Forecast project may be found in the MetaFileSystem that is now accessible by consultants. This may include data such as new explainable insights (like cross-correlations, similarity explanations, master data management data, etc.)

info

If you are a consultant doing SensibleAI Forecast work, don’t be afraid to put in enhancement requests for new explainable insights or new forms of data that you cannot find within SensibleAI Forecast Consumption Groups that could be surfaced through the Project Level MetaFileSystem.

You can then pull in this new data via the MetaFileSystemClient and surface this within new dashboards for SensibleAI Forecast Engagements.

SensibleAI Studio Data Extraction

The new AI Services product SensibleAI Studio heavily relies on the MetaFileSystem to store any artifacts and other data spun off by SensibleAI Studio Routines.

If you are unfamiliar with the SensibleAI Studio Routines, think of these as callable data science functions like anomaly detectors or fuzzy matchers that will save .parquet, .json, .csv, and any other file-based data to the MetaFileSystem, which can then be accessed via the MetaFileSystemClient.

To demonstrate the entire round trip through an example, you could:

1. (User) Run a SensibleAI Studio Anomaly Detector Routine against a SensibleAI Forecast Source Dataset.
2. (System) After the Anomaly Detector Routine finishes running, it will output a file called anomalies.parquet to the MetaFileSystem.
3. (User) Use the MetaFileSystemClient to read the newly created anomalies.parquet stored at a path like dsl/routines/runs/anomaly_detector/anomalies.parquet file into a System.Data.DataTable object (or even directly into a BiBlend Database Table).
4. Build a OneStream dashboard on top of this data to create a nice user experience around the SensibleAI Studio Routine for the customer.

Future AI Services Solutions

The data behind the MetaFileSystem is projected to become pivotal for future development of AI Services solutions. For example, an AI Services Developer or MarketPlace Developer could use a series of SensibleAI Studio Routines to build a Data Quality Monitor AI Services solution that ends up calling Anomaly Detectors, Data Thresholders, and Data Cleanser Routines found in the SensibleAI Studio. Similar to the use case mentioned above, the developer can then use the MetaFileSystemClient to pull back data generated from the SensibleAI Studio functions.

Stakeholders

SensibleAI Forecast Consultants

As called out in some of the use cases, consultants on SensibleAI Forecast implementations should be comfortable with the MetaFileSystemClient. They will find many of the prebuilt data converter functions helpful and dramatically simplify the interchanging of data formats (ex: .parquet to DB Table).

AI Services Developers

AI Services developers are expected to begin to rely heavily on the MetaFileSytemClient as all future AI Services solutions are expected to rely on “SensibleAI Studio” functionality first, then on custom backend builds second. This will become as common as working with SQL.

OneStream MarketPlace Developers

OneStream MarketPlace developers can also incorporate the MetaFileSystemClient into their solutions via the SensibleAI Studio.

Usage Guide

Instantiating the MetaFileSystemClient

There are two main ways to create instances of the MetaFileSystemClient

• Option 1: Entrypoint (Factory) Based.
• Option 2: Dependency Injection Based.

Entrypoint (Consulting Persona)

If you are a consultant looking to build custom solutions or one-off dashboards for customers, it is recommended that you use the Entrypoints class found within Xperiflow Business Rules. This can be found at the following namespace Workspace.AIS.XBRV310.Entrypoints;

There are factory methods that exist in this namespace that will instantiate a IMetaFileSystem concrete implementation. Here are the function signatures available to you:

public I`MetaFileSystemClient` Get`MetaFileSystemClient`(SessionInfo si, string connectionKey, OISToken oisToken = null);

public I`MetaFileSystemClient` Get`MetaFileSystemClient`(SessionInfo si, MetaFileSystemLocation metaFileSystemLocation, OISToken oisToken = null);

public I`MetaFileSystemClient` Get`MetaFileSystemClient`(SessionInfo si, string connectionKey, IHttpClientManager httpClientManager, IXperiflowStorageClient xperiflowStorageClient)

caution

If you are building one-off dashboards for customers, do NOT import the Startup.cs of another solution and attempt to use the Dependency Injection object creation approach details in the next section. If you truly want to want to avoid using the Entrypoints.cs, you should develop your dashboards as an AI Services Solution using the DSTS AI Services Solution template.

Using with Dependency Injection (Developer Persona)

If you are a solution developer, it is expected that you will register the IMetaFileSystemClient with the dependency injection in the Startup.cs like you would any of the other services that we use. At the time of writing 27 Mar 2024, it is expected that the constructor parameter metadataConnectionKey is set to “sql-server-framework"

Below is an example of service registration:

services.AddTransient(sp =>
{
    var httpClientManager = sp.GetRequiredService();
    var client = sp.GetRequiredService();
    string metadataConnectionKey = "sql-server-framework";

    return new `MetaFileSystemClient`(SessionInfoAccessor.si, httpClientManager, client, metadataConnectionKey);
});

caution

If you are solution developer, you should avoid using the Entrypoint approach given that there are a lot of additional objects that need to be created every time you call the entrypoint. The dependency injection approach is set up to avoid additional object creation calls that are unnecessary if you know you are working within the confines of a Solution.

Common Operations

Making Directories

The MetaFileSystem allows you to create one or more directories within a single method call to MakeDirectoriesAsync. This is useful for better organizing the data that you plan to write for the MetaFileSystem. Note that there are runtime restrictions on where you can create directories. This is controlled on the Xperiflow side.

`MetaFileSystemClient`.MakeDirectoriesAsync("fileshare/folder/folder5/folder7", existsOk: true).Wait();

Basic Creating and Writing of Files

The MetaFileSystem allows you to easily create and write files using the overloaded WriteFileAsync method.

Below is an example of how to write a text file where the content of the text file will be: “My Text File Data to Write"

`MetaFileSystemClient`.WriteFileAsync("fileshare/folder/folder5/folder7/test.txt", "My Text File Data to Write", true).Wait();

note

The MetaFileSystem does not support the ability to update files at this time. If you need to update a file, it is recommended that you first read the file into memory, make your edits, then write the file back to the same location.

Read File Metadata

The following illustrates how to read metadata about a file stored within the MetaFileSystem. Note that this does not actually read the file's content, but its metadata.

// Make the directory path
`MetaFileSystemClient`.MakeDirectoriesAsync("fileshare/folder/folder5/folder7", existsOk: true).Wait();

// Write the file
`MetaFileSystemClient`.WriteFileAsync("fileshare/folder/folder5/folder7/test.txt", "My Text File Data to Write", false).Wait();

// Pull back file metadata. Note, this will throw a FileNotFoundException if file does not exist.
FileMetadata? fileMetadata = `MetaFileSystemClient`.GetFileMetadataAsync("fileshare/folder/folder5/folder7/test.txt").Result;

Console.WriteLine($"File Full Name: {fileMetadata?.FullName}");  // File Full Name: /fileshare/folder/folder5/folder7/test.txt
Console.WriteLine($"File Name: {fileMetadata?.Name}");  // File Name: test.txt
Console.WriteLine($"File Extension: {fileMetadata?.Extension}"); // File Extension: .txt
Console.WriteLine($"File Size in Bytes {fileMetadata?.Size}"); // File Size in Bytes 26

In the code snippet above, we are attempting to read metadata about a file stored in the MetaFileSystem at the file path of fileshare/folder/folder5/folder7/test.txt. Note that it will throw a FileNotFoundException if the provided file path does not exist.

If I do not want the function to throw an exception if no file exists, then I can use the optional parameter on the GetFileMetadataAsync called errorWhenDoesNotExist, and I can turn it to false. Now, in the event that the file does not exist, null will be returned from GetFileMetadataAsync(...)

note

You will find that many of the MetaFileSystemClient methods have this errorWhenDoesNotExist parameter. It exhibits the same behavior as mentioned above.

info

Notice how the fileMetadata?.FullName gives back the fully qualified path.

Read Directory Metadata

Similarly to reading metadata about a file, we can also read metadata about a directory (aka a folder).

DirectoryMetadata? directoryMetadata = `MetaFileSystemClient`.GetDirectoryMetadataAsync("fileshare/folder/folder5/folder7", false).Result;

Console.WriteLine($"Directory Full Name: {directoryMetadata?.FullName}");
Console.WriteLine($"Directory Name: {directoryMetadata?.Name}");
Console.WriteLine($"Directory Size: {directoryMetadata?.Size}");
Console.WriteLine($"Directory Creation Time: {directoryMetadata?.CreationTime}");

Note that the above code will tell me about the directory, not any of its subcontents.

Read Directory

Reading a directory allows you to bring back metadata of its subfiles and subdirectories. For example, if I have a directory structure that looks like this:

/fileshare
  /folder
    /folder5
      test3.json
      /folder7
        test.txt
        test2.txt

I can use the MetaFileSystemClient.GetDirectoryAsync("root_folder/my_folder", ...) to pull back metadata about all the subfiles and subfolders found within fileshare/folder. I will get back lists of FileMetadata objects for subfiles (test3.json, test.txt, test2.txt) and DirectoryMetadata objects for subdirectories (folder5, folder5/folder7).

// Create some folders and files
`MetaFileSystemClient`.MakeDirectoriesAsync("fileshare/folder/folder5/folder7", existsOk: true).Wait();

// Create some files (some json, some txt, and at different levels)
`MetaFileSystemClient`.WriteFileAsync("fileshare/folder/folder5/folder7/test.txt", "My Text File Data to Write", false).Wait();
`MetaFileSystemClient`.WriteFileAsync("fileshare/folder/folder5/folder7/test2.txt", "My Text File Data to Write", false).Wait();
`MetaFileSystemClient`.WriteFileAsync("fileshare/folder/folder5/test3.json", @"{""key"":""value""}", false).Wait();

// ATTENTION: This actual method that gets the directory content
DirectoryContent? directoryContent = `MetaFileSystemClient`.GetDirectoryAsync("fileshare/folder", -1, true, false).Result;

// Ensure the directory content is not null
if (directoryContent == null)
{
    throw new NullReferenceException(nameof(directoryContent));
}

// Inspecting the "folder" directory
Console.WriteLine($"Directory Full Name: {directoryContent?.DirectoryMetadata.FullName}"); // Directory Full Name: /fileshare/folder/folder5
Console.WriteLine($"Directory Name: {directoryContent?.DirectoryMetadata.Name}"); //Directory Full Name: /fileshare/folder/folder5/folder7
Console.WriteLine();

// Inspecting the files in the "folder" directory
if (directoryContent.SubFiles != null)
{
    // Print the files in the directory
    foreach (FileMetadata file in directoryContent.SubFiles)
    {
        Console.WriteLine($"File Full Name: {file.FullName}");
    }
}

// For loop File Output
// File Full Name: /fileshare/folder/folder5/folder7/test.txt
// File Full Name: /fileshare/folder/folder5/folder7/test2.txt
// File Full Name: /fileshare/folder/folder5/test3.json

Console.WriteLine();

// Inspecting the subdirectories in the "folder" directory
if (directoryContent.SubDirectories != null)
{
    // Print the subdirectories in the directory
    foreach (DirectoryMetadata dir in directoryContent.SubDirectories)
    {
        Console.WriteLine($"Directory Full Name: {dir.FullName}");
    }
}

// Directory Full Name: /fileshare/folder/folder5
// Directory Full Name: /fileshare/folder/folder5/folder7

Additionally, I can control the amount of FileMetadata and DirectoryMetadata that I want to pull back from the MetaFileSystem with this GetDirectoryAsync function.

For example:

DirectoryInfoDto? directory1 = `MetaFileSystemClient`.GetDirectoryAsync("/fileshare/folder", maxDepth:1, includeDirectories:false).Result;

Setting maxDepth to 1 and includeDirectories to false would yield only test3.json in our example.

Reading File Contents

Reading file contents allows you to retrieve the raw contents of the file and return within a Stream object. The Stream object comes with a rich set of subclasses and other implementations that many 3rd party packages will integrate with. Stream is considered the most basic implementation and works with most things.

Reading Text Data Into a String

To read a file's content as text, we can do the following:

Stream test1OutputStream = `MetaFileSystemClient`.GetFileContentStreamAsync("myfolder/folder5/folder7/test.txt").Result;

// Check if the Stream is Null
if (test1OutputStream == null)
    throw new ArgumentNullException(nameof(test1OutputStream));

// Assuming the stream contains text data
string test1Output = null;
using (StreamReader reader = new StreamReader(test1OutputStream, Encoding.UTF8))
{
    try
    {
        // Read the stream into the string variable
        test1Output = reader.ReadToEnd();
    }
    catch (Exception ex)
    {
        // Handle any exceptions that may occur
        throw new InvalidOperationException("Error reading from the stream.", ex);
    }
}

Reading a Parquet Into a DataTable

Similar to the example above, we can chain together MetaFileSystemClient operations to work nicely with other libraries and functions that leverage the Stream object. Below is an example showing how to read a parquet files into a DataTable object.

var parquetFilePath = "routines/instances/14c5b2b0-f038-1337-0165-d48031bc3bfd/runs/4b7a7f1c-a1cf-d6fd-0408-d5c031bc3c11/artifacts/predictions/data_/data-0.parquet";

if (`MetaFileSystemClient`.FileExistsAsync(parquetFilePath).Result)
{
    // Get back a file stream object of the parquet file.
    Stream parquetFileStream = `MetaFileSystemClient`.GetFileContentStreamAsync(parquetFilePath).Result;

    // Check if the Stream is Null
    if (parquetFileStream == null)
        throw new ArgumentNullException(nameof(parquetFileStream));

    // Leverage the Xperiflow Business Rules UtilParquetReader
    DataTable dt = UtilParquetReader.ReadParquetToDataTableAsync(parquetFileStream).Result;
}

Deleting a Directory

Similar to deleting a file, the user has the option to delete directories and their inner subdirectories and subfiles as long as the directory itself, its subdirectories, and subfiles were “externally-generated”.

// (Operation 1) Make some directories
`MetaFileSystemClient`.MakeDirectoriesAsync("fileshare/test_folder/another_folder", existsOk: true).Wait();
`MetaFileSystemClient`.MakeDirectoriesAsync("fileshare/test_folder/another_folder_2", existsOk: true).Wait();

// (Operation 2) Add a file to the directory
`MetaFileSystemClient`.WriteFileAsync("fileshare/test_folder/test.txt", "My Text File Data to Write", false).Wait();

// (Operation 3) Delete the another_folder_2
`MetaFileSystemClient`.RemoveFileAsync("fileshare/test_folder/another_folder_2", errorWhenDoesNotExist: true).Wait();

// (Operation 4) Delete multiple folders and files using recursive = true
`MetaFileSystemClient`.RemoveDirectoryAsync("fileshare/test_folder", recursive: true).Wait();

Let’s analyze the example above through the folder structure at each operation.

Operation 1: Make some directories

fileshare
  test_folder
    another_folder
    another_folder_2

Operation 2: Add a file to the directory

fileshare
  test_folder
    test.txt
    another_folder
    another_folder_2

Operation 3: Delete the another_folder_2

fileshare
  test_folder
    test.txt
    another_folder

Operation 4: Delete multiple folders and files using recursive = true

fileshare

Additional

Relative MetaFileSystem

Another MetaFileSystem implementation exists that uses relative paths rather than fully qualified paths. This filesystem is called the RelativeMetaFileSystem. This version of the IMetaFileSystem takes in a root path into its constructor and then allows the users of this file system instance to just provided relative paths. For examples, I recommend checking out its usage within the IRoutineClient and its respective concrete implementations and its sub-objects.