Operating an EDD observation

Normal operation is performed through the Master Controller. A telescope control interface may issue the same commands automatically; direct KATCP interaction is useful for commissioning and diagnosis.

Pre-observation checks

Before provisioning:

confirm the requested provision description and site-repository version;
confirm that no other observation owns the selected hosts, multicast ranges, GPUs, FPGAs, or output directories;
check time synchronization and telescope metadata;
check free storage and expected data rate;
check the Grafana overview and recent Loki errors; and
confirm that the Master Controller is unprovisioned.

Observation lifecycle

The usual sequence is:

provision
    configure
        capture-start
            measurement-prepare
            measurement-start
            measurement-stop
            ... repeat for each scan ...
        deconfigure
    deprovision

Use the command spelling reported by the KATCP client. KATCP wire requests are normally written with hyphens, for example ?capture-start and ?measurement-prepare; Python methods use underscores.

Provision

provision selects the paired playbook and base configuration. For a mode named example, the Master Controller normally loads example.yml and example.json. Provisioning launches containers and registers pipelines, leaving the system in idle.

Check every expected product is registered before continuing. A missing product usually indicates an Ansible, image, container-startup, or KATCP registration failure.

Configure and start capture

configure resolves the stream graph, configures products in dependency order, allocates buffers and resources, and starts required processing applications. The system reaches configured when all products succeed.

capture-start activates ingest. Measuring pipelines enter ready; streaming pipelines enter streaming. Confirm packet-loss, buffer-fill, and throughput sensors before starting a scan.

Run scans

measurement-prepare supplies scan-specific metadata and moves measuring pipelines from ready to set. Verify source, project, frequency setup, output directory, and generated EDD identifier before proceeding.

measurement-start begins the scan and moves measuring pipelines to measuring. measurement-stop ends it and returns them to ready. Streaming pipelines remain streaming throughout these commands.

Stop cleanly

After the final scan, issue deconfigure and wait for idle before deprovision. Deprovisioning stops observation-specific containers and returns the Master Controller to unprovisioned. Confirm output files are closed and visible before releasing storage or moving data.

Failure handling

stalled: The pipeline cannot process the current scan but may recover for the next one. Stop the measurement, correct scan metadata or inputs, and prepare the next scan. Some pipelines can be configured to promote a stall to an error.
error: The current control loop failed. Capture logs and sensors first, then use deconfigure to return to idle. Do not simply override the state unless a developer has established that the pipeline resources are clean.
panic: Deconfiguration failed, so resources may still be active. Stop the observation, preserve logs, inspect processes, containers, buffers, and network transmitters, and involve an EDD administrator before retrying.

See Troubleshooting for symptom-based checks.