Troubleshooting

Collect evidence before restarting a component. At minimum record the UTC time, site-repository revision, provision name, EDD identifier, affected product, pipeline state and notes, relevant sensors, container image tag, and the corresponding Loki log interval.

Pipeline does not register

  1. Inspect the Ansible result for the product host.

  2. Check that the container exists and has not exited.

  3. Inspect its standard output for import, device, permission, and bind errors.

  4. Confirm the selected KATCP port is reachable from the Master Controller.

  5. Confirm the product ID is unique and matches the JSON configuration.

Configure fails

  1. Read the first failing product in the Master Controller log; downstream failures may only be consequences of it.

  2. Inspect the configuration graph for missing or cyclic stream references.

  3. Compare stream format name and version between producer and consumer.

  4. Check multicast address allocation, interface selection, and NUMA/GPU access.

  5. Check that DADA keys are not left by an earlier failed instance.

Packet loss or buffer pressure

Check receiver packet counters, DADA fill level, CPU affinity, NUMA locality, NIC link state, switch multicast configuration, MTU, socket or DPDK settings, and the sender’s output rate. Confirm that the configured data rate does not exceed the NIC, PCIe, memory, GPU, disk, or downstream capacity. Do not hide a sustained loss condition by increasing buffers without identifying the bottleneck.

No output data

Confirm the pipeline is active for the scan, output mode is not null, the measurement metadata selected the intended directory, the container can write the mounted path, storage is available, and the writer reports an open output. For network output, confirm the resolved destination rather than only the base configuration, because an upstream pipeline may alter it during configure.

Metadata or timestamps are wrong

Compare host time, digitizer synchronization time, sample clock, packet timestamps, telescope metadata, and the output header. Timezone strings are for presentation; observation timestamps and synchronization epochs must be handled as UTC. Stop the observation if provenance cannot be reconstructed.

Deconfigure or deprovision hangs

Preserve logs, then identify the product awaiting a task or subprocess. Check for blocked disk I/O, an unresponsive device, lingering network processes, and containers that ignore termination. A panic state means cleanup is not known to be complete. Verify transmitters and shared-memory buffers manually before forcing container removal.

Escalation record

An actionable issue report contains:

  • exact core, plugin, image, and site-repository versions;

  • the smallest provision description that reproduces the problem;

  • commands and state transitions performed;

  • logs and sensor snapshots with UTC ranges;

  • relevant hardware and network topology; and

  • expected versus observed behavior.