Monitoring and logging

Monitoring has two purposes: detecting operational faults quickly and retaining enough evidence to assess data quality later. A green pipeline-state sensor alone is not sufficient; data rates, loss, buffers, timestamps, metadata, and output progress also matter.

Operator workflow

At the start of a shift:

  1. Open the EDD overview dashboard and confirm persistent services are healthy.

  2. Check recent warning and error logs for the Master Controller.

  3. Confirm database and storage retention have free space.

  4. Verify time, telescope metadata, and the active site version.

After provisioning:

  1. Confirm every expected product appears with the intended host and image.

  2. Open product dashboards and check packet loss and buffer progress.

  3. Compare measured throughput with the rate calculated for the mode.

  4. Confirm no development product or multicast stream overlaps production.

During a scan, watch the stable pipeline state, notes, receiver loss, DADA fill, processing throughput, output rate, writer progress, and free storage. Record the UTC time before investigating a transient anomaly so the same interval can be found in Loki and InfluxDB.

Sensor destinations

The Redis sidecar is suitable for current state and larger values such as PNG diagnostics. The Influx sidecar is suitable for numeric time series. Plugin roles should exclude image or high-cardinality sensors from InfluxDB and should not emit unbounded labels or sensor names.

Logs

Every application must write structured, timestamped messages to standard output. Include the pipeline ID, state transition, configuration summary, and actionable exception context. Do not log credentials, vault contents, private keys, or access tokens.

When diagnosing a failure, search the Master Controller and the first failing upstream product over the same UTC interval. Downstream errors often only report that their input never became available.

Modifying dashboards

Provisioned dashboards are generated from version-controlled templates. Edits made only in the Grafana UI are lost on the next provisioning cycle.

  1. Copy the live dashboard before experimenting.

  2. Make and test the change against representative sensor data.

  3. Export the dashboard JSON.

  4. Remove site-specific data-source UIDs and temporary values.

  5. Set the dashboard id field to null.

  6. Apply the change to the appropriate core or plugin role template.

  7. Provision and deprovision it on a development site to test both creation and cleanup.

Add product-specific panels to the plugin that owns the sensors. Put genuinely universal state, receive, DADA, or host panels in the core.