Provisioning and site configuration

Provisioning starts and stops the pipelines needed for one observing mode. The Master Controller runs Ansible against the same site inventory used for core deployment, then applies a JSON base configuration to the registered products.

Installation versus provisioning

Installation: An infrequent administrative operation that deploys persistent core services, prepares hosts, and installs pinned plugin collections.
Provisioning: A routine operational action that starts a selected set of pipeline and auxiliary-service containers on assigned hardware. Deprovisioning tears them down again.

Provision descriptions

A provision named example normally consists of two files in provision_descriptions/:

example.yml: An Ansible playbook that maps fully qualified plugin roles to inventory hosts and sets instance-level role variables.
example.json: A base configuration containing product IDs, pipeline settings, and stream connections.

The names are paired automatically. The Master Controller can also accept a semicolon-separated playbook and configuration name when a site deliberately combines different files; prefer paired files for normal operation because they are easier to test and review.

Ansible placement file

This example starts two spectrometers and their upstream packetizer controller:

---
- hosts: gpu_server[0:1]
  vars:
    edd_group: 50000
  roles:
    - role: EDD.gated_spectrometer.gated_spectrometer
      container_name: gated_spectrometer_{{ play_hosts.index(inventory_hostname) }}
      container_env: "EDD_ALLOWED_NUMA_NODES=0"

- hosts: digitizer_controller[0]
  roles:
    - role: EDD.digitizer_packetizer_controller.digitizer_packetizer_controller
      container_name: dig_pack_controller
      device: receiver_packetizer

Use the collection and role names from the versions pinned in EDD_PLUGINS. Do not use short role names such as common because they depend on the local Ansible search path.

Product IDs

Every running pipeline needs a unique product ID. The ID must agree in three places:

the effective container_name used by the role;
the product’s id in the JSON base configuration; and
any source reference to that product.

A role’s default container name is sufficient only when there is one instance of that role. Set container_name explicitly for repeated instances.

Base configuration and stream graph

{
  "products": [
    {
      "id": "dig_pack_controller",
      "bit_depth": 8,
      "sampling_rate": 4000000000.0,
      "output_data_streams": {
        "polarization_0": {"format": "MPIFR_EDD_Packetizer:2"},
        "polarization_1": {"format": "MPIFR_EDD_Packetizer:2"}
      }
    },
    {
      "id": "gated_spectrometer_0",
      "fft_length": 262144,
      "naccumulate": 16384,
      "input_data_streams": [
        {
          "source": "dig_pack_controller:polarization_0",
          "format": "MPIFR_EDD_Packetizer:2"
        }
      ],
      "output_data_streams": {
        "P0_ND0": {
          "format": "GatedSpectrometer:2",
          "ip": "auto",
          "port": 7152
        },
        "P0_ND1": {
          "format": "GatedSpectrometer:2",
          "ip": "auto",
          "port": 7152
        }
      }
    }
  ]
}

source uses product-id:output-name. The Master Controller resolves the reference and merges the producer’s final output description into the consumer input. Products are configured in dependency order. Cycles and references to unknown products are errors.

Stream descriptors

A stream descriptor is a versioned contract, not merely an IP address. Depending on the format it may include:

format name and version;
address and port;
sample rate, bit depth, dimensions, polarization, channels, and bandwidth;
synchronization epoch and timestamp interpretation;
samples or bytes per heap;
index-vector or SPEAD item definitions; and
subdivisions that allow consumers to select channels or polarizations.

The subscribing pipeline is responsible for rejecting incompatible formats. Always include format in a source reference so incompatibilities are visible during review. ip: auto requests an address from the site’s configured multicast range.

Role-level settings

The following variables are commonly set on a product role in the YAML file:

container_name: Unique container and product name.
container_env: Environment variables injected into the container. The EDD_ALLOWED_NUMA_NODES variable restricts the NUMA nodes from which a pipeline may allocate CPUs, memory, and devices; the pipeline remains responsible for enforcing the restriction.
edd_user and edd_group: UID/GID or names used inside the container. The defaults are root; production data writers should use the least privilege compatible with the device and filesystem permissions.
pipeline_data_dir: Per-instance data directory. Prefer overriding this derived variable only when a site has an explicit storage layout requirement.
extra_volumes: Additional reviewed container mounts.

Common role contract

Plugin roles normally include EDD.core.common. The current common role provides image build or pull, container launch, free KATCP port selection, registration, data directories, monitoring sidecars, dashboards, and provision tests.

A minimal plugin role resembles:

- name: Example pipeline
  include_role:
    name: EDD.core.common
  vars:
    image_name: edd_example
    default_container_name: example_pipeline
    container_cmd: >-
      edd_example_pipeline --host={{ edd_subnet }} --port={{ bind_port }}
    docker_remote_registry: registry.example.org/edd/example
    dashboard_panels:
      - pipeline_status.json
    provision_test_imports: edd.example.provision_tests
  tags:
    - always

Important common variables include:

Common role variables
Variable	Default	Purpose
`image_name`	required	Container image name.
`container_cmd`	required	KATCP pipeline server command.
`default_container_name`	`image_name`	Default product name.
`bind_port`	`0`	KATCP port; zero selects a free port from the configured range.
`launch_redis_sidecar`	true	Copy suitable live sensor values into Redis.
`launch_influx_sidecar`	true	Copy scalar time-series sensors into InfluxDB.
`install_dashboard`	true	Install the product’s Grafana dashboard.
`register_as_pipeline`	true	Register the container with the Master Controller state model.
`dashboard_panels`	empty	Dashboard panel templates supplied by the role or core.
`measurement`	`image_name`	InfluxDB measurement name.
`version_tag`	`latest`	Image version built or pulled. Production inventories must override it with a pinned version policy.

Container data layout

The common role mounts four scopes:

Data scopes
Container path	Host path below `data_base_path`	Scope
`/mnt` and `/EDD/pipeline`	`pipeline_data/<container_name>`	One pipeline instance
`/EDD/role`	`role_data/<pipeline_type or image_name>`	All instances of a role
`/EDD/plugin`	`plugin_data/<plugin_type>`	All roles in one plugin
`/EDD/global`	`global_data`	Site-wide, mounted read-only by default

Write products to the narrowest appropriate scope. Do not make global data writable merely to bypass an ownership problem.

Inventory

An inventory defines hosts, groups, and variables for one deployment target. Common group names include gpu_server, registry, redis, influx, grafana, mastercontroller, and interface, but only the groups used by the pinned site and core playbooks are mandatory.

Core and plugin pins

EDD_PLUGINS records the installed Galaxy collections:

EDD_PLUGINS:
  - name: EDD.core
    source: https://gitlab.example.org/edd/edd-core.git
    version: "260601.0"
  - name: EDD.gated_spectrometer
    source: https://gitlab.example.org/edd/gated_spectrometer.git
    version: "260603.1"

Pin immutable tags or commits in production. A plugin collection version, its container image, and its Python or Debian package version must describe the same implementation.

Network and service variables

Typical site variables include:

edd_subnet: Management subnet from which pipeline KATCP servers accept connections. Avoid 0.0.0.0 unless network isolation is enforced elsewhere.
high_speed_data_subnet and high_speed_data_subnetmask: Network used to select high-rate interfaces.
multicast_network: Address range from which the Master Controller allocates automatic stream destinations. It must be isolated from other production and development EDD instances.
edd_katcp_port_range_start and edd_katcp_port_range_stop: Host port range available to pipeline KATCP servers.
docker_registry and docker_registry_port: Local image registry endpoint.
data_base_path: Root of the EDD data layout on the host filesystem.
redis_host, influx_host, and grafana_host: Service endpoints, often derived from the first host in the corresponding inventory group.

Hardware devices

Store packetizers, digitizers, FPGA interfaces, and other controlled hardware in inventory variables. Associate a controller role with a device using its device variable. This keeps management addresses, ports, MAC addresses, firmware details, and data interfaces out of reusable provision descriptions.

Validation checklist

Before committing a provision description:

Parse the YAML, JSON, and Ansible inventory.
Confirm every role exists in a pinned collection.
Confirm container names and product IDs are unique and identical.
Confirm every source resolves and the graph is acyclic.
Confirm stream format versions and units match on both sides.
Calculate aggregate network, GPU, memory, and storage rates.
Confirm NUMA and device assignments do not overlap unintentionally.
Run unit and lifecycle tests, then a complete provision test on development hardware.
Review produced files, monitoring dashboards, logs, and cleanup after deprovisioning.