Architecture

EDD separates reusable backend infrastructure from observing-mode software and from observatory-specific configuration. Keeping these boundaries explicit is the central design rule of the system.

System layers

Core

The EDD core provides the Master Controller, the common pipeline state model, KATCP control clients and servers, Ansible roles, monitoring and logging services, data-stream descriptions, process management, DADA buffer helpers, and provision-test infrastructure.

Plugins

An EDD plugin is an Ansible Galaxy collection that provides one or more processing pipelines or auxiliary services. A plugin may combine Python control software, C++ or CUDA processing applications, container templates, dashboards, and tests. Plugins are installed and versioned independently from the core.

Site repository

A site repository contains the inventory, network and storage settings, enabled plugin versions, site playbooks, provision descriptions, and any telescope-specific interfaces. Site configuration must not be embedded in a reusable plugin.

Services and pipelines

Services are normally long-lived components, such as the Master Controller, Redis, Grafana, Loki, a Telescope Metadata Interface, or a telescope-control protocol bridge. Services do not participate in the observing state model.

Pipelines are provisioned for an observing mode and implement a shared KATCP interface. They may control a device, transform a data stream, write a data product, or perform several of these tasks. Pipelines are either measuring pipeline or streaming pipeline instances.

Control and data planes

EDD deliberately separates control traffic from bulk astronomical data:

• Ansible uses SSH for installation, host configuration, and starting or stopping containers.

• The Master Controller uses KATCP for runtime control, state transitions, sensors, and error reporting.

• Pipelines exchange bulk data through declared stream interfaces. Common transports are SPEAD over UDP multicast, PSRDADA shared-memory buffers, and VDIF streams or files.

• Redis stores current configuration, telescope metadata, and larger live diagnostic values. InfluxDB stores time-series metrics. Grafana displays both, and Loki collects container logs.

Configuration graph

A provision description contains a list of products. An input stream can refer to an upstream output with source: product-id:stream-name. The Master Controller resolves these references into a directed acyclic graph and configures upstream products before their consumers. This permits an upstream pipeline to finalize addresses or derived stream metadata during configuration.

Configuration is applied at four different timescales:

Configuration timescales

Timescale	Mechanism	Typical content
Site installation	Inventory and site playbook	Hosts, networks, storage, services, plugin versions
Observing-mode provisioning	Provision YAML and JSON	Pipeline placement, stream graph, base configuration
Capture configuration	configure and capture-start	Buffers, processes, network subscriptions, output streams
Individual scan	measurement-prepare	Source, project, scan identifier, output destination

See Pipeline lifecycle and interface for the state model and Provisioning and site configuration for the configuration files.