Skip to main content
Author: Drew Shea, Created: 2025-04-06

Author: Drew Shea
Date Created:

Overview

The purpose of this document is to provide common code recipes for interactions with Routine C# objects (XBRApi.Routines) including RoutineInstance, Run, Artifact, InMemoryJsonArtifact, ArtifactInfo, and more.

Object: Routine Instance

Use Case: Create a Routine Instance​

var routineClient = XBRApi.Routines.GetRoutineClient(si);

var routineInstance = routineClient.CreateRoutineInstanceAsync(
    "InMemExecutionTestRoutine",
    null
).Result;

Explanation:

  • Line 1: Retrieve RoutineClient

    • We are retrieving the RoutineClient which provides the ability to manage RoutineInstance objects.

  • Line 3 - 6: Create RoutineInstance object with the least amount of configurations

    • Line 4: This is the Routine Name that we want to create an instance of. The names of Routines can be found in the SensibleAI Studio → Explore Page.
    • Line 5: We can optionally provide a name for the RoutineInstance that we want to create or we can jsut provide null and a name will be automatically generated.

Use Case: Store Arbitrary Json Attributes on the Routine Instance​

It is not uncommon to want to have a Routine Instance take in additional metadata about the solution that it is operating within while also not wanting this metadata exposed to the user of the Routine Instance (often from a UI perspective). Storing arbitrary metadata as JSON on the Routine Instance allows the ability to handle these situations.

var routineInstance = routineClient.CreateRoutineInstanceAsync(
    "InMemExecutionTestRoutine", // Routine Type Name
    null, // Routine Name (null means the name will be automatically generated)
    labels: new List<string> { "MyCustomLabel", "AnotherCustomLabel" },  // User defined labels you can filter/categorize Routine Instances by
    attributes: new JObject()
    {
        {"RoutineAttribute", "RoutineAttributeValue" },
        { "RoutineAttribute2", "RoutineAttributeValue2" }
    }  // Arbitrary json attributes that will live at the routine instance level. These are accessible on the python side as well
).Result;

Explanation:

  • Line 5 - 9: Arbitrary JSON Attributes

    • Arbitrary json attributes can be attached and stored on the Routine Instance that can then be retrieved later on the C# side and python side (see below)

C#: Access via RoutineClient​

var rehydratedRoutineInstance = routineClient.GetRoutineInstanceAsync("my-routin-j52801l").Result;

var attributes = rehydratedRoutineClient.Attributes;
BRApi.ErrorLog.LogObject(si, "Rehydrated Attributes", attributes);
// Log Output: 
Rehydrated Attributes

{
  "RoutineAttribute": "RoutineAttributeValue",
  "RoutineAttribute2": "RoutineAttributeValue2"
}
info

Note: There is a BRApi.ErrorLog.LogObject extension method located at the namespace using Workspace.XBR.Xperiflow.Utilities.Logging.Extensions;

Python: Access via the RoutineApi​

For Routine Developers, you may access this Routine Instance Attributes wherever you have access to the RoutineApi object as a python dict at the api.routine.attributes property. For example, you can see the ability to interact with the attributes.

@routine
class InMemExecutionTestRoutine(BaseRoutine):

    definition = RoutineDefinitionContext(...)

    @rmethod(memory_capacity=2, allow_in_memory_execution=True)
    def concat(self, api: RoutineApi, parameters: SimplePbm) -> InMemExecutionArtifact:
      # Access Routine Instance Json Attributes via 'api'
      my_value = api.routine.attributes["RoutineAttribute"]
      # my_value will equal "RoutineAttributeValue"

Use Case: Store Data in the Routine Instance File System Directory​

There are situations where you want to store data at the Routine Instance level that doesn’t fit into the model of input parameters and arbitrary json attributes. For example, you may have a situation where you want to store tabular data on the C# side and you want this data to be contained within the Routine Instance itself, so that when the Routine Instance is deleted, this data is wiped as well.

There is a property on the RoutineInstance object called SharedDataStore which provides a key-value pair abstraction over the MetaFileSystemClient to allows the developer user to read and write data to the following Routine MetaFileSystem root directory location: instances_/<routine_instance_id>/shared_/store_

C#: Write and Read Text Data to the SharedDataStore​

var routineInstance = routineClient.GetRoutineInstanceAsync("my-routine-3823t2").Result;

// Case 1a: Write key-value pair where the value is text (behind the scenes it gets serialized to a .txt file)
routineInstance.SharedDataStore.WriteText("my_string_key", "hello world");

// Case 1b: Read the text back into memory
var rehydratedText = routineInstance.SharedDataStore.ReadText("my_key");

C#: Write and Read DataTable to the SharedDataStore​

var routineInstance = routineClient.GetRoutineInstanceAsync("my-routine-3823t2").Result;

// A DataTable I want to save
var dt = new DataTable("MyDataTable");
dt.Columns.Add("Column1", typeof(string));
dt.Columns.Add("Column2", typeof(string));
dt.Rows.Add("Row1Column1", "Row1Column2");
dt.Rows.Add("Row2Column1", "Row2Column2");    

// Case 2a: Write key-value pair where the value is a DataTable (behind the scenes it gets serialized as partitioned parquet files)
routineInstance.SharedDataStore.WriteDataTable("my_datatable", dt);

// Case 2b: Read the DataTable back into Memory
var rehydratedDt = routineInstance.SharedDataStore.ReadDataTable("my_datatable");

C#: Write and Read Json Data to the SharedDataStore​

var routineInstance = routineClient.GetRoutineInstanceAsync("my-routine-3823t2").Result;

// Some json I want to save
var json = new JObject(
  new JProperty("key1", "value1"),
  new JProperty("key2", "value2")
);

// Case 3a: Write key-value pair where the value is a JObject (behind the scenes it serialized to a .json file)
routineInstance.SharedDataStore.WriteJson("my_json_key", json);

// Case 3b: Read the JObject back into memory
var rehydratedJson = routineInstance.SharedDataStore.ReadJsonAsJObject("my_json_key");

C#: Write and Read Json Serializable Objects to the SharedDataStore​

public class SimpleObject
{
    public string key1 { get; set; }
    public string key2 { get; set; }
}

public void main() 
{
    var routineInstance = routineClient.GetRoutineInstanceAsync("my-routine-3823t2").Result;
    
    // Some json serializable object I want to save
    var simpleObject = new SimpleObject()
    {
        key1 = "value1",
        key2 = "value2"
    };
    
    // Case 4a: Write key-value paire where the value is a Json Serializable object
    routineInstance.SharedDataStore.WriteJson<SimpleObject>("my_json_object_key", simpleObject);
    
    // Case 4b: Read the Simple Object back into Memory
    var rehydratedSimpleObject = routineInstance.SharedDataStore.ReadJsonAsObject<SimpleObject>("my_json_object_key");
}

This SharedDataStore object also exists on the Run object. However, the data saved will be scoped to the Run at this Routine MetaFileSystem root directory location: instances_/<routine_instance_id>/runs_/<run_id>/shared_/store_

Object: Run

Use Case: Store Arbitrary Json Attributes on the Run​

It is not uncommon to want to have a Run take in additional metadata about the solution that it is operating within while also not wanting this metadata exposed to the user of the Routine Instance (often from a UI perspective). Storing arbitrary metadata as JSON on the Run allows the ability to handle these situations.

var run = routineInstance.CreateRunAsync(
    "concat",
    null,
    false,
    false,
    InvocationMethodType.Direct,
    new JObject(
        new JProperty("first", "Hello"),
        new JProperty("second", "World")
    ),
    "My Description",
    storeArtifacts: true,
    executionType: "in-memory",
    attributes: new JObject()
    {
        {"RunAttribute", "RunAttributeValue" },
        { "RunAttribute2", "RunAttributeValue2" }
    }
).Result;

Explanation:

  • Line 5 - 9: Arbitrary JSON Attributes

    • Arbitrary json attributes can be attached and stored on the Routine Instance that can then be retrieved later on the C# side and python side (see below)

C#: Access via RoutineClient​

var rehydratedRun = routineInstance.GetRunByIdentifierAsync(run.RunIdentifier).Result;

var attributes = rehydratedRun.Attributes;
BRApi.ErrorLog.LogObject(si, "Rehydrated Attributes", attributes);
// Log Output: 
Rehydrated Attributes

{
    {"RunAttribute", "RunAttributeValue" },
    { "RunAttribute2", "RunAttributeValue2" }
}
info

Note: There is a BRApi.ErrorLog.LogObject extension method located at the namespace using Workspace.XBR.Xperiflow.Utilities.Logging.Extensions;

Python: Access via the RoutineApi​

You may access this Run Attributes wherever you have access to the RoutineApi object as a python dict at the api.run.attributes property. For example, you can see the ability to interact with the attributes.

@routine
class InMemExecutionTestRoutine(BaseRoutine):

    definition = RoutineDefinitionContext(...)

    @rmethod(memory_capacity=2, allow_in_memory_execution=True)
    def concat(self, api: RoutineApi, parameters: SimplePbm) -> InMemExecutionArtifact:
      # Access Routine Instance Json Attributes via 'api'
      my_value = api.run.attributes["RunAttribute"]
      # my_value will equal "RunAttributeValue"

Use Case: Run a Routine Method at the REST API Layer (“in-memory”)​

Now available with the Xperiflow v4.0.0 release, there is the ability to execute Routine Methods “in-memory”. When invoking a Routine Method “in-memory” from the C# side, it will run the Routine Method on the Xperiflow Caller Server (web server). This method of execution allows for some powerful benefits over the traditional way of execution (now dubbed “background”) including:

  • Directly Return Json Serializable Artifacts

    • Given that the Routine Method runs within the process that received the invocation, certain Artifact Types can directly return a Json Serializable version of the Artifact

  • Faster Runtimes

    • There is no 10 - 15 second Python process spin-up penalty like there when doing “background” execution

      • Note: We have seen a Routine Method Run go from 30 seconds to under a second as part of the AI Account Reconciliation project that occurred

    • These faster runtimes allow to even run a Routine Method behind a UI interaction (a button click and screen refresh)

Python: In-Memory Executable Routine Methods​

It is important to note not all Routine Methods are “in-memory” executable. A Routine Method is only “in-memory” executable if it has the argument allow_in_memory_execution=True within the @rmethod decorator.

@pbm
class InMemExecutionArtifact(ArtifactBaseModel):
    concat_str: ArtifactDef[str, StrTextArtifactIOFactory] = ArtifactMetadataDef(
        title="Concatenated String", description="The concatenated string based on input strings."
    )
    dataframe: ArtifactDef[pl.DataFrame, PolarsParquetArtifactIOFactory] = ArtifactMetadataDef(
        title="DataFrame", description="The dataframe based on the concatenated string."
    )

@routine
class InMemExecutionTestRoutine(BaseRoutine):

    definition = RoutineDefinitionContext(
        title="In Memory Execution Testing Routine",
        tags=[RoutineTags_.ml, RoutineTags_.supervised, RoutineTags_.data_integration],
        short_description="Testing routine for testing in memory execution",
        long_description="Testing routine for testing in memory execution" * 10,
        use_cases=[],
        creation_date=dt.datetime(25, 3, 25),
        author="Luke Heberling",
        organization=OrganizationNames_.ONESTREAM,
        version="0.0.1",
    )

    @rmethod(memory_capacity=2, allow_in_memory_execution=True)  # Marked to Allow In-Memory Execution
    def concat(self, api: RoutineApi, parameters: SimplePbm) -> InMemExecutionArtifact:
        ...

Explanation:

  • Line 3 - 5: A Json Serializable Artifact

    • On our ArtifactBaseModel implementation, the concat_str Artifact is Json Serializable. This is because the StrTextArtifactIOFactory implements a special method called write_json_artifact that we will see in more detail in a moment.

  • Line 6 - 8: Not Json Serializable Artifact

    • On our ArtifactBaseModel implementation, the concat_str Artifact is not Json Serializable. This is because it does not implement the special method write_json_artifact. This means that if we want access to this object on the C# side, we will have use the standard methodology for pulling artifacts data.

  • Line 25: rmethod decorator turns on in-memory execution

    • We can see that we have allow_in_memory_execution set to True

Python: Json Serializable Artifacts​

For Routine Developers, as briefly mentioned above in the benefits section, only certain Artifact Types are JSON serializable and therefore can be directly return from the “in-memory” execution.

The technical explanation here is: IArtifactWriter (python side) concrete implementations that implement the IJsonArtifactWriterMixin (ex: StrTextArtifactWriter) will have their Json representation of their Artifacts accessible as the response to an invocation of an “in-memory” Routine Method Run.

Let’s take a look at the StrTextArtifactWriter implementation:

class StrTextArtifactWriter(IArtifactWriter[str], IJsonArtifactWriterMixin[str]):
    @property
    def file_annotations(self) -> list[FileAnnotationContext]:
        return [FileAnnotationContext(file_annotation="string.txt", file_annotation_description="text file of string")]

    def write(self, fs: AbstractFileSystem, dirpath: str, data: str):
        full_path = pathutil.join_parent_and_name(dirpath, "string.txt")
        with fs.open(path=full_path, mode="w") as f:
            f.write(data)

    def write_json_artifact(self, data: str) -> dict[str, str | int | float | list[str]]:
        return {"string_data": data}

Explanation:

  • Line 1: Inherits a Mixin

    • See how it inherits a mixin called IJsonArtifactWriterMixin[str] This defines a method called def write_json_artifact on it

  • Line 11 - 12: The write_json_artifact method

    • This is the method that gets called when an “in-memory” execution is finishing if the Routine Method Run were to return an Artifact that implements this IJsonArtifactWriterMixin

C#: Creating an “in-memory” Run​

Now that we have an appreciation for what it means for a Routine Method to support “in-memory” execution and have a JSON serializable Artifact that can be directly returned as a response to our Run invocation, let’s invoke an “in-memory” Run using the RoutineClient:

// Get the Routine Client
var routineClient = XBRApi.Routines.GetRoutineClient(si);

// Create a Routine Instance
var routineInstance = routineClient.CreateRoutineInstanceAsync(
    "InMemExecutionTestRoutine",
    null,
    labels: new List<string> { "RoutineLabel", "Another" },
    attributes: new JObject()
    {
        {"RoutineAttribute", "RoutineAttributeValue" },
        { "RoutineAttribute2", "RoutineAttributeValue2" }
    }
).Result;

// Create a Run of the Routine Method "concat"
var run = routineInstance.CreateRunAsync(
    "concat",
    null,
    false,
    false,
    InvocationMethodType.Direct,
    new JObject(
        new JProperty("first", "Hello"),
        new JProperty("second", "World")
    ),
    "My Description",
    storeArtifacts: true,
    executionType: "in-memory",
    attributes: new JObject()
    {
        {"RunAttribute", "RunAttributeValue" },
        { "RunAttribute2", "RunAttributeValue2" }
    }
).Result;

Explanation:

  • Lines 1 - 14: Nothing Special

    • We are just simply created a Routine Instance of our InMemExecutionTestRoutine that was shown above in the python code snippets.

  • Line 17: We Create a Run

  • Line 28: Store Artifact Boolean

    • This determines whether or not you want to save artifacts to the metafilesystem. By default, you should always do this. However, it can be smart to turn this off when executionType = "in-memory" if there is no reason to refer back to the Artifacts again.
    • This can save shave roughly 1 second of the execution when executionType = "in-memory"

  • Line 29: Execution Type

    • This is what actually controls whether you execute “in-memory” or as a “background” job.
    • By default, this is set to “background”

C#: Invoking the Run​

// Start the run, wait for successful completion, and get a RunResult object back
var runResult = run.StartAndWaitForSuccessfulCompletionResultAsync().Result;
BRApi.ErrorLog.LogObject(si, "Run Result Object", runResult);

Given that the RunResult is a new object, it can be helpful to see all of its properties in the context of a real Run

Description: Run Result Object
Error Time: 4/6/2025 10:54:16 PM
Error Level: Information
Tier: AppServer
User: Administrator
Application: AISDevelopment
App Server: RMTHO-L6759
App Server Version: 9.0.0.17121
App Server OS Version: Microsoft Windows NT 10.0.22631.0
Session ID: 420419b8-b69b-4387-80da-31a959241c96
Error Log ID: 2aaa3457-4bf0-4716-b934-c72c1d3e4bc0

Total Memory: 34,010,619,904 (31.67 GB)
Memory In Use: 846,118,912 (0.79 GB)
Private Memory In Use: 1,547,788,288 (1.44 GB)
Peak Memory In Use: 3,767,492,608 (3.51 GB)
Maximum Data Records In RAM: 7,971,239
Maximum Data Units In RAM: 100,000
Number Of Threads: 110

----------------------------------------
Exception Type: Unknown
Thread Id: 138
{
  "ExecutionMetadata": {
    "ExecutionIdentifier": "52",
    "ExecutionType": "in-memory",
    "InputData": "",
    "ExecutionStatus": {
      "ExecutionIdentifier": "52",
      "ExecutionType": "in-memory",
      "ActivityStatus": "completed",
      "QueuedTime": "2025-04-06T22:54:03.099439",
      "StartTime": "2025-04-06T22:54:15.299556",
      "LastUpdateTime": "2025-04-06T22:54:16.775672",
      "EndTime": "2025-04-06T22:54:16.775672",
      "Message": "Created Run",
      "PercentComplete": 1.0,
      "LastStatusCheck": "2025-04-06T22:54:03.099439",
      "Extra": {}
    },
    "Extra": {}
  },
  "RunMetadata": {
    "InstanceIdentifier": "inmemexecu-20-du4mbu",
    "RunIdentifier": "concat-202504-xhncxw",
    "ExecutionIdentifier": "52",
    "ExecutionType": "in-memory",
    "MethodName": "concat",
    "RoutineCallableType": "method",
    "RunName": "concat_20250406225403012",
    "RunDescription": "My Description",
    "CreationTime": "2025-04-06T22:54:03.127",
    "ModifiedTime": "2025-04-06T22:54:03.127",
    "CreatedByUserId": "2",
    "CreatedByUserName": "Administrator",
    "ModifiedByUserId": "2",
    "ModifiedByUserName": "Administrator",
    "ArtifactPath": "instances_/inmemexecu-20-du4mbu/runs_/concat-202504-xhncxw/artifacts_",
    "InputParams": {
      "first": "Hello",
      "second": "World"
    },
    "InvocationMethod": {
      "InvocationMethodType": "direct",
      "WorkflowId": -1
    },
    "MemoryCapacity": 2.0,
    "IncludeStatistics": false,
    "IncludePreviews": false,
    "StoreArtifacts": true,
    "Labels": [],
    "Attributes": {
      "RunAttribute": "RunAttributeValue",
      "RunAttribute2": "RunAttributeValue2"
    }
  },
  "ArtifactInfos": [
    {
      "HasArtifactStored": true,
      "HasInMemoryArtifact": true,
      "QualifiedKey": "concat_str"
    },
    {
      "HasArtifactStored": true,
      "HasInMemoryArtifact": false,
      "QualifiedKey": "dataframe"
    }
  ]
}

Explanations:

  • Line 25 - 41: Execution Metadata

    • As you can see, the RunResult object gives us direct access to the ExecutionMetadata of the Run

  • Line 44 - 77: The Run Metadata

    • The RunResult object also have the RunMetadata object attached to it.

  • Line 78: The Artifact Info Collection

    • This maintain a collection of ArtifactInfo objects.

  • Line 79 - 83: An Artifact Info object

    • There is more than just these three attributes you see. There are methods on an ArtifactInfo object for retrieve the standard Artifact object and retrieving the InMemoryJsonArtifact object as well. We will see this in the next code block
    • Line 80: Tells us if the concat_str Artifact has it’s data stored in the MetaFileSystem
    • Line 81: Tell us if the concat_str Artifact was also returned to us as an InMemoryJsonArtifact
    • In short, in this case, we see that because we had storeArtifact:true back on routineInstance.CreateRunAsync and because the concat_str artifact key that was defined on our python InMemExecutionArtifact is a json serializable artifact, we have the concat_str artifact available In Memory AND in the MetaFileSystem

C#: Interacting with the In-Memory Artifacts​

var artifactResult = runResult.GetArtifactInfo("concat_str");

// Check that the Artifact Info exists
if (artifactResult != null)
{
    // See the In Memory Version
    if (artifactResult.HasInMemoryArtifact)
    {
        InMemoryJsonArtifact inMemoryArtifact = artifactResult.GetInMemoryJsonArtifactAsync().Result;

        // Get the Artifact Data
        JObject? jsonArtifactData = inMemoryArtifact?.GetArtifactData();
        BRApi.ErrorLog.LogMessage(si, $"In Memory Artifact Data: {jsonArtifactData.ToString()}");
       
    }

    // See the Standard Version
    if (artifactResult.HasArtifactStored)
    {
        // Get the Artifact
        Artifact artifact = artifactResult.GetArtifactAsync().Result;
        
        // Get the Artifact Data
        var artifactData = artifact.GetDataAsObjectAsync<string>().Result;
        BRApi.ErrorLog.LogMessage(si, $"Artifact Data: {artifactData}");
    }
}

Console Output:

Artifact Data: HelloWorld

In Memory Artifact Data: {
  "string_data": "HelloWorld"
}

Summary and Additional Notes​

In summary and additional notes to be aware of:

  • The “in-memory” option allows you to invoke a Run fast enough to be behind a UI interaction on behind a dashboard.
  • Set storeArtifact:False for an additional speed boost if you are certain you don’t need to stored artifacts after executing the Run when using executionType:"in-memory"
  • The Web Service will kill your “in-memory” execution if it runs for longer than 2 minutes.
  • Don’t default set all Routine Methods to allow_in_memory_execution=True. This should be used very sparingly and mainly for Routine Methods that do simple CRUD operations. Rule of Thumb: If you would feel comfortable running the code in your Routine Method behind a standard Xperiflow REST API, then it’s probably safe to turn allow_in_memory_execution=True
  • There are no plans to support json serializers for tabular artifact writers (ex: PolarsParquetArtifactWriter). This is to protect developers from themselves and avoid returning large dynamic column json datatables over a REST Api.

Use Case: Retrieving Artifact Data from a Run​

In the prior use case, there was heavy focus on “in-memory” execution. In this use case, we will quickly hit on what some of the new methodologies for gathering Artifact Data without needing to write a lot of code.

There are two main ways in which you can retrieve Artifact Data from a Run.

Option 1: Get Data as Object Method​

This is the new method of how you can retrieve certain types of artifacts.

// Retrieve the run
var routineRun = routineInstance.GetRunByIdentifierAsync("sv-cs-predict-hq3gze").Result;

////////////////////////////
// CASE A: Tabular Artifact
////////////////////////////
// Retrieve Tabular (parquet-based) Artifact Data (using the Artifact Qualified Key "predictions")
var tabularArtifact = routineRun?.GetArtifactAsync("predictions").Result;

// Pull out the tabular artifact as a DataTable
DataTable dtPredictions = tabularArtifact?.GetDataAsObjectAsync<DataTable>().Result;

////////////////////////////
// CASE B: Text-Based Artifact
////////////////////////////
// Retrieve (.txt file-based) Artifact Data (using the Artifact Qualified Key "text_data")
var textArtifact = routineRun?.GetArtifactAsync("text_data").Result;

// Pull out the text artifact as a string
string text = textArtifact?.GetDataAsObjectAsync<string>().Result;

////////////////////////////
// CASE C: Json-Based Artifact
////////////////////////////
// Retrieve (.json file-based) Artifact Data (using the Artifact Qualified Key "json_data")
var jsonArtifact = routineRun?.GetArtifactAsync("json_data").Result;

// Pull out the text artifact as a string
JObject json = jsonArtifact?.GetDataAsObjectAsync<JObject>().Result;


Explanations:

  • Line 8: Get the Artifact Object

    • We get the Artifact Object associated with the Artifact Qualified Key of “prediction” which in this case is associated parquet-based Artifact Data.

  • Line 11: Extract as DataTable

    • The Artifact object has this special method on it called GetDataAsObject<Generic> that abstracts away all the hydration logic of pulling back the Artifact Data found within the Artifact.
    • In this case, behind the scenes, this method recognizes that the underlying Artifact Data is parquet data which is compatible with the C# DataTable object type. After it recognizes it’s compatible, it will automatically pull out the partitioned parquet data into the C# DataTable for you.

  • Line 20: Extract Text Based Artifact Data

    • In this case, the textArtifact object corresponds to an underlying Artifact Data of data.txt file.
    • With GetDataAsObject<string>(), it will automatically pull the string based data out of the underlying data.txt file for you and put it into the text C# variable

  • Line 29: Extract Json Based Artifacts

    • In this case, the jsonArtifact object corresponds to an underlying Artifact Data to data.json file.
    • With GetDataAsObject<JObject>(), it will automatically pull the json (string) based data out of the underlying data.json file for you and put it into the json C# variable

Option 2: Read as File Data​

This option involves “directly” reading the files of the Artifact Data.

// Retrieve the run
var routineRun = routineInstance.GetRunByIdentifierAsync("sv-cs-predict-hq3gze").Result;

////////////////////////////
// CASE A: Tabular Artifact
////////////////////////////
// Retrieve Tabular (parquet-based) Artifact Data (using the Artifact Qualified Key "predictions")
var artifact = routineRun?.GetArtifactAsync("predictions").Result;

// Loop through the partitioned parquet files and extract them as DataTables
var dts = new List<DataTable>();
foreach (var file in files)
{
    var dt = XBRApi.Etl.ExtractMetaFileSystemParquetAsDataTable(si, MetaFileSystemLocation.Routine, file);
    dts.Add(dt);
}

// Concatenate the partitions into one
var dtPredictions = DataTableUtil.Concat(si, dts);

Explanations:

  • Line 8: Get the Artifact Object

    • We get the Artifact Object associated with the Artifact Qualified Key of “prediction” which in this case is associated parquet-based Artifact Data.

  • Line 10 - 19: Extract as DataTable

    • First important point to call out - These lines of code are equivalent to “Option 1’s” Line 11. As you can see Option 1 abstracts away a lot of the underlying annoyance of working with files and loading out data.
    • Line 11: Create a list of DataTable objects (one for each file - don’t assume you know how many partitioned files there are)
    • Line 12 - 16: Loop through each file and read the file into a DataTable object using the ETL function.
    • Line 19: Concatenate the list of DataTable objects (there is a helper in DataTableUtil class).
Last updated on by Jeremy Meiss

Was this page helpful?