Interface ILoader<T>

Defines the contract for loading (storing) data into various destinations in ETL (Extract, Transform, Load) operations.

Namespace: Workspace.XBR.Xperiflow.Etl

Assembly: Xperiflow.dll

Declaration

public interface ILoader<T>

Examples

Database loader implementation:

public class SqlBulkLoader : ILoader
{
    private readonly string _connectionString;
    private readonly string _tableName;

    public SqlBulkLoader(string connectionString, string tableName)
    {
        _connectionString = connectionString;
        _tableName = tableName;
    }

    public void Load(DataTable payload)
    {
        using var connection = new SqlConnection(_connectionString);
        connection.Open();

        using var bulkCopy = new SqlBulkCopy(connection);
        bulkCopy.DestinationTableName = _tableName;
        bulkCopy.WriteToServer(payload);
    }
}

// Usage
var loader = new SqlBulkLoader(connectionString, "ProcessedData");
loader.Load(transformedDataTable);

JSON file loader implementation:

public class JsonFileLoader : ILoader
{
    private readonly string _directoryPath;
    private readonly bool _appendTimestamp;

    public JsonFileLoader(string directoryPath, bool appendTimestamp = true)
    {
        _directoryPath = directoryPath;
        _appendTimestamp = appendTimestamp;
    }

    public void Load(T payload)
    `{
        var fileName = _appendTimestamp 
            ? $"data_{DateTime.UtcNow:yyyyMMdd_HHmmss}`.json"
            : "data.json";

        var filePath = Path.Combine(_directoryPath, fileName);
        var json = JsonConvert.SerializeObject(payload, Formatting.Indented);

        Directory.CreateDirectory(_directoryPath);
        File.WriteAllText(filePath, json);
    }
}

// Usage
var loader = new JsonFileLoader>(@"C:\OutputData");
loader.Load(processedOrders);

Complete ETL pipeline using extractor and loader:

// Create ETL components
var extractor = new SqlDataTableExtractor(sourceConnectionString, "SELECT * FROM RawData");
var loader = new SqlBulkLoader(targetConnectionString, "ProcessedData");

// Execute ETL pipeline
var rawData = extractor.Extract();
var transformedData = TransformData(rawData); // Your transformation logic
loader.Load(transformedData);

Console.WriteLine($"Loaded {transformedData.Rows.Count} records successfully");

Remarks

The Workspace.XBR.Xperiflow.Etl.ILoader%601 interface represents the "Load" phase of ETL operations, providing a standardized way to store processed data into various destinations such as databases, files, APIs, or other data repositories. This interface complements Workspace.XBR.Xperiflow.Etl.IExtractor%601 to form complete ETL pipelines.

Key Design Principles:

• Type SafetyGeneric type parameter ensures compile-time type safety for loaded data

• Destination AgnosticAbstracts the destination, allowing implementations for databases, files, APIs, etc.

• Transactional SupportImplementations can support transactional loading for data consistency

• Batch ProcessingCan be designed to handle both single records and batch operations efficiently

Common Implementation Scenarios:

• Database inserters/updaters for DataTable, DataSet, or custom objects

• File writers for CSV, JSON, XML, or other structured formats

• Web API clients that POST data to REST endpoints

• Message queue publishers for event-driven architectures

• Cache loaders for performance optimization

Error Handling and Reliability:

Implementations should handle destination-specific errors appropriately, such as connection failures, constraint violations, or capacity limitations. Consider implementing retry logic, dead letter queues, or rollback mechanisms for robust data loading operations.

Performance Optimization:

For high-volume data loading, consider implementing bulk insert operations, connection pooling, and parallel processing strategies. The design should balance between throughput and resource utilization.

Methods

Load(T)

Loads the provided data payload into the configured destination.

Declaration

void Load(T payload)

Remarks

This method performs the actual data loading operation to the configured destination. The implementation should handle all aspects of connecting to the destination, preparing the data for storage, and executing the load operation.

Implementation Guidelines:

Performance Considerations:

For large payloads, consider implementing:

Data Integrity:

Ensure that the loading operation maintains data integrity by implementing appropriate validation, constraint checking, and rollback mechanisms in case of partial failures.

Parameters

Type	Name	Description
<T>	payload	The data payload to load into the destination

Exceptions

System.InvalidOperationException Thrown when the loading operation fails due to destination connectivity issues, data validation problems, or other operational errors System.UnauthorizedAccessException Thrown when the loader lacks necessary permissions to write to the destination System.ArgumentNullException Thrown when the payload parameter is null