Skip to main content

Xperiflow Routine Execution

Author: Drew Shea, Created: 2025-11-16

Xperiflow Routine Execution is the mechanism by which Routines—custom, reusable workflows—are executed in the Xperiflow engine. Routines are essentially a special type of Task that can be wrapped in a Job for asynchronous execution, or executed directly in memory for synchronous, immediate results. This dual execution model provides flexibility to choose the right approach based on your needs.

Info: Routine Execution bridges the gap between custom user-defined workflows and the standard Job/Task execution framework, allowing you to leverage the same orchestration infrastructure for both built-in operations and Routines.

Overview​

Routine Execution provides a way to run custom workflows (Routines) through the same execution infrastructure used by standard Jobs and Tasks. When you execute a Routine, you're essentially creating a specialized Task that runs the specific Routine code, which is then wrapped in a Job for full integration with the Conduit Orchestration Framework.

Routines can be executed in two ways:

  • Job-Based Execution: The Routine runs as a Task within a Job, providing full integration with the orchestration framework, queuing, monitoring, and resource management
  • In-Memory Execution: The Routine executes immediately in the current process, providing instant results without the overhead of job creation and queuing

Core Concepts​

Routines as Tasks​

At its core, a Routine is simply bundled as a specialized type of Task. When you execute a Routine through the Conduit Orchestration Framework:

  1. A RoutineExecution Job is created
  2. The Job creates a RoutineExecutionTask (an atomic Task)
  3. The Task executes the Routine's custom logic
  4. The execution is tracked like any other Job/Task

This means Routines benefit from all the same features as standard Tasks:

  • Progress tracking
  • Error handling and retries
  • Resource management (memory, CPU)
  • Status monitoring
  • Integration with the broader execution framework
info

**Note**: Routines “return” *Artifact* (output data) the Routine MetaFileSystem upon the completion of the Task.

Job-Based Execution​

When a Routine is executed as a Job:

The Job Layer:

  • A RoutineExecution Job is created
  • The Job follows standard Job lifecycle (QUEUED → RUNNING → COMPLETED)
  • The Job can be scheduled, prioritized, and monitored like any other Job
  • The Job appears in job lists and can be queried like standard Jobs

The Task Layer:

  • The Job creates a single RoutineExecutionTask
  • This Task is an atomic Task (execution type: ATOMIC)
  • The Task is a background Task, meaning it runs on worker processes
  • The Task enforces memory capacity requirements
  • The Task executes your Routine's method

Benefits:

  • Full integration with the orchestration framework
  • Asynchronous execution (doesn't block the invoker’s process)
  • Resource management and limits enforcement
  • Progress tracking and status monitoring
  • Error handling with retries
  • Can be scheduled for later execution
  • Appears in standard job monitoring interfaces

In-Memory Execution​

Some Routine methods can be executed directly in memory, bypassing the Job/Task framework entirely:

How It Works:

  • The Routine method executes immediately in the current process
  • No Job or Task is created
  • Execution is synchronous (your code waits for completion)
  • Artifacts are returned directly in a JSON compatible format.

When Available:

  • Only methods that explicitly allow in-memory execution can use this mode
  • The Routine method must be marked as allowing in-memory execution
  • If not allowed, the system automatically uses job-based execution

Benefits:

  • Instant execution (no queuing or scheduling overhead)
  • Immediate results (synchronous execution)
  • Simpler for quick operations
  • Lower overhead for lightweight operations
  • Direct return values

Limitations:

  • Only available for certain methods
  • Blocks the current process until completion
  • No built-in progress tracking
  • Limited error recovery options
  • Cannot be scheduled or prioritized

Choosing Execution Mode​

The system automatically chooses the execution mode based on:

Automatic Selection:

  • If the method allows in-memory execution and no explicit mode is specified, in-memory execution is used
  • If the method doesn't allow in-memory execution, job-based execution is used automatically
  • You can explicitly request a specific execution mode

When to Use Job-Based Execution:

  • Long-running operations that shouldn't block your process
  • Operations that need to be scheduled for later
  • Operations that benefit from progress tracking
  • Operations that need resource management and limits
  • Operations that should appear in job monitoring interfaces
  • Operations that may need retries or error recovery

When to Use In-Memory Execution:

  • Quick, lightweight operations
  • Operations where you need immediate results
  • Operations that are part of a larger synchronous workflow
  • Operations that don't require the overhead of job management
  • Operations that complete quickly (seconds, not minutes)

Routine Execution Properties​

Job Properties​

When executed as a Job, Routine Execution inherits standard Job properties:

  • Priority: Default priority is 1 (high priority) given a Routine Method only executes as a single Atomic Task.
  • Session Type: Framework-level (not tied to a specific project)
  • Status Tracking: Full Job lifecycle tracking
  • Resource Management: Memory and CPU limits enforced

Task Properties​

The RoutineExecutionTask has specific characteristics:

  • Execution Type: Atomic (performs actual work, no subtasks)
  • Background Task: Runs on worker processes
  • Memory Enforcement: Enforces memory capacity requirements
  • Retries: Supports automatic retries on failure
  • Session Type: Framework-level execution

Memory Management​

Routine Execution respects memory requirements:

  • Each Routine specifies its memory capacity needs
  • The system validates sufficient memory is available
  • Memory limits are enforced before execution begins
  • In-memory execution uses the current process's memory
  • Job-based execution allocates memory on worker processes

Monitoring Routine Execution​

Job-Based Monitoring​

When executed as a Job, you can monitor:

  • Job Status: Standard Job activity statuses
  • Task Status: The underlying Task's execution status
  • Progress: Progress tracking through the Job/Task framework
  • Timestamps: Creation, start, and completion times
  • Error Information: Detailed error logs if execution fails

In-Memory Monitoring​

In-memory execution provides:

  • Immediate Feedback: Success or failure is known immediately
  • Direct Results: Results are returned directly
  • Error Handling: Exceptions are raised immediately
  • No Separate Tracking: No separate job or task to monitor
Last updated on by Jeremy Meiss

Was this page helpful?