EDD core services

Core services are deployed for the lifetime of an EDD site and do not participate in the per-observation pipeline state model.

Master Controller

The Master Controller is the single control point for provisioning and the observational lifecycle. It:

  • runs provision and deprovision playbooks;

  • tracks registered products and their KATCP endpoints;

  • builds and validates the data-stream dependency graph;

  • configures upstream products before downstream consumers;

  • fans lifecycle commands out to pipelines;

  • aggregates pipeline state, errors, notes, and output files; and

  • generates unique EDD identifiers and measurement summaries.

The Master Controller itself extends the common pipeline interface with unprovisioned, provisioning, deprovisioning, and maintenance states. All telescope-control commands should pass through it rather than addressing individual processing pipelines.

Ansible interfaces

Runtime provisioning needs to launch containers on remote hosts. An Ansible interface container on each managed node supplies the Master Controller with SSH access and controlled access to the container daemon. Site SSH keys belong in the encrypted site inventory. Restrict the service to the management network: access to it is effectively administrative access to the host.

Redis

Redis stores current backend and product configuration, registered endpoints, telescope metadata, Ansible facts, and live values such as diagnostic images. Redis is operational state, not the sole archival provenance record. Persist the site revision, provision, configuration, and measurement summary with scientific data.

Redis Commander may be deployed for administrative inspection. It must not be exposed to untrusted networks.

InfluxDB

InfluxDB stores scalar sensor time series such as data rates, packet loss, buffer fill, and processing latency. Configure retention in the site inventory according to operational and provenance requirements and monitor database disk usage.

Grafana

Grafana combines Redis and InfluxDB sources into a site overview and pipeline-specific dashboards. Product dashboards are added when a provision is loaded and removed when it is torn down. See Monitoring and logging for the operator workflow and dashboard contribution rules.

Loki and external logging

Pipelines log to standard output. Container logs are collected centrally in Loki and queried through Grafana. Sites may additionally forward logs to an external service such as Graylog. Retention must cover the interval in which an observer can report a data-quality problem.

Telescope Metadata Interface

The TMI adapts observatory-specific metadata to EDD. Depending on the site it may subscribe to a multicast or ZeroMQ feed, query an observatory service, or accept values supplied by the telescope-control system. Pipelines consume the normalized metadata through the EDD data store. See Telescope integration.