Service Information¶
Architecture¶
The Ceph S3 bucket notification feature is used to send notifications as Kafka events when Objects are created in the rubin-summit bucket in production and the rubind-pp-dev bucket in development. The bucket notification is configured to send to the S3 File Notifications Kafka cluster. An instance of S3 File Notifications is deployed in both Prompt Processing prod and dev.
S3 File Notifications Kafka is installed with Phalanx and ArgoCD. Three brokers are configured for redundancy. Authentication, Authorization, and SSL are not enabled as these did not work when setting up Ceph notifications. Load Balancer IPs are provisioned from the sdf-rubin-ingest network for the Brokers and External Bootstrap to provide connectivity outside of the cluster in S3DF.
Architecture Diagram¶
Associated Systems¶
The S3-File-Notifications Kafka cluster is used by Prompt Processing to identify when files are uploaded to the USDF. The Data Transfer Monitoring application which tracks files transfers from the USDF to the Summit uses file notifications to count the files transferred.
Configuration Location¶
Config Area |
Location |
|---|---|
Configuration |
|
Vault Secrets Dev |
N/A |
Vault Secrets Prod |
N/A |
Data Flow¶
Ceph S3 creates file notifications when objects are created. Prompt Processing uses these notifications to identify when files are uploaded to the USDF. The Data Transfer Monitoring application which tracks files transfers from the USDF to the Summit uses file notifications to count the files transferred.
Dependencies - S3DF¶
Ceph
Dependencies - External¶
No external dependencies.
Disaster Recovery¶
Data does not need to be retained in a Disaster Recovery event. If data is lost or unavailable redeploy the Kafka cluster.