Application Information¶
Architecture¶
rucio-register registers Butler data with Rucio for transfer. Hermes which sends messages about data transfers was modified to send Kafka events with Butler information. This code is in lsst-dm/ctrl_rucio_ingest and is where modifications to the Rucio Hermes container are developed.
ctrl_ingestd is a Butler/Rucio ingest daemon that listens to Kafka messages from the Rucio Hermes daemons and performs butler ingests. It is used to automatically ingest registered data to a Butler. The code for this is in lsst-dm/ctrl_ingestd
The France and UK Data Facilities connect to the Rucio Kafka cluster with MirrorMaker to Rucio Butler ingest events to perform locally.
Architecture Diagram¶
See DMTN-198
Associated Systems¶
The Rucio-Butler integration is installed in Rucio. Events are sent to the Rucio Kafka.
Configuration Location¶
Config Area |
Location |
|---|---|
Configuration |
|
Vault Secrets Dev |
Uses Rucio secrets. No additional secrets. |
Vault Secrets Prod |
Uses Rucio secrets. No additional secrets. |
Data Flow¶
Dependencies - S3DF¶
Butler Main
Dependencies - External¶
Network connection to France and UK Data Facilities