Pipeline lifecycle and interface

An EDD pipeline combines a KATCP control server, a container and Ansible role, and the processing program or device it manages. All pipelines implement the same lifecycle even when their internal processing is entirely different.

Pipeline flavours

Measuring pipelines

These pipelines act on individual scans. After capture starts they enter ready and use measurement prepare, start, and stop transitions for each scan. Writers and pulsar processing are typical examples.

Streaming pipelines

These pipelines run continuously across scan boundaries. After capture starts they enter streaming. Measurement commands may update metadata or be acknowledged, but do not leave that state. A correlator-beamformer is a typical example.

The core can infer the flavour from overridden measurement methods, but new pipelines should pass an explicit PipelineFlavor to make their behaviour unambiguous.

State model

digraph edd_state_machine { rankdir=LR; graph [nodesep=".35", ranksep=".6"]; node [shape="rectangle", fontcolor="white", fillcolor="#18038d", style="filled", fontname="Arial"]; edge [fontsize=10, fontname="Arial"]; idle -> configured [label="configure"]; configured -> ready [label="capture-start\n(measuring)"]; configured -> streaming [label="capture-start\n(streaming)"]; ready -> set [label="measurement-prepare\nactive=true"]; ready -> inactive [label="measurement-prepare\nactive=false"]; set -> measuring [label="measurement-start"]; measuring -> ready [label="measurement-stop"]; inactive -> ready [label="measurement-stop"]; streaming -> streaming [label="measurement commands"]; configured -> idle [label="deconfigure"]; ready -> idle [label="deconfigure"]; streaming -> idle [label="deconfigure"]; set -> idle [label="deconfigure"]; measuring -> idle [label="deconfigure"]; inactive -> idle [label="deconfigure"]; stalled -> idle [label="deconfigure"]; error -> idle [label="deconfigure"]; error -> panic [label="deconfigure failed"]; error [tooltip="Reachable when an operational transition fails."]; stalled [tooltip="The current scan cannot be processed."]; panic [tooltip="Cleanup failed and may require manual intervention."]; }

Stable EDD pipeline states. Short-lived transitional states are omitted.

idle

The server is running and accepts configuration changes.

configured

Long-lived resources, processes, and stream descriptions are configured.

ready

A measuring pipeline is capturing and can prepare a scan.

set

Scan metadata has been accepted and the pipeline can start measuring.

measuring

The pipeline is processing the current scan.

streaming

A streaming pipeline is processing continuously.

inactive

The pipeline was disabled for the current scan by measurement_prepare_default.active or the scan configuration.

stalled

The current scan cannot be processed, but a later scan may recover.

error

An operational transition failed. Deconfigure is the normal recovery path.

panic

Deconfiguration failed and manual cleanup may be required.

The implementation also reports transitional states such as configuring, capture-starting, measurement-preparing, measurement-starting, measurement-stopping, deactivating, and deconfiguring. These are not safe points from which to issue another lifecycle command.

Common KATCP requests

Common requests

Request

Valid source state

Effect

?set <json>

idle

Merge a partial configuration without changing state. Unknown keys are rejected; type changes currently produce warnings rather than complete schema validation.

?configure [json]

idle

Optionally merge configuration, allocate resources, and enter configured.

?capture-start

configured

Activate ingest and enter ready or streaming according to flavour.

?measurement-prepare [json]

ready

Apply scan metadata and enter set or inactive. A stalled pipeline may result if the scan cannot be processed.

?measurement-start

set

Begin scan processing and enter measuring.

?measurement-stop

measuring, stalled, or inactive

Finish the scan and return to ready.

?deconfigure

Any state

Stop pipeline resources and return to idle. Failure enters panic.

?capture-stop remains available for compatibility but is ignored by the current core state model. Use ?deconfigure to stop capture and release resources.

Common configuration

The base class adds these keys to every pipeline configuration:

id and type

Unique product identifier and pipeline type.

input_data_streams and output_data_streams

Stream descriptors. Inputs may use source references that the Master Controller resolves during configure.

data_store

Redis host and port used for registration and shared state.

measurement_prepare_default.active

Default scan activation flag. Setting it to false places a measuring pipeline in inactive for a scan.

ignore_pipeline_errors

Controls how the Master Controller treats a product error. Use only when an observing mode is deliberately able to continue without that product.

Developer responsibilities

A pipeline implementation must:

  • release every resource acquired during configure when deconfigure is called, including after partial failure;

  • validate scientific and hardware constraints not covered by the base class;

  • declare final output stream metadata before downstream configuration;

  • expose actionable status, throughput, buffer, and error sensors;

  • write all application logs to standard output;

  • register generated files so measurement stop can report them; and

  • supply unit, lifecycle, and provision tests.

See Developing EDD pipelines and plugins for a current implementation template.