Service Information¶

Architecture¶

Prompt Kafka is deployed with Strimzi in the Prompt Processing vCluster. The Strimzi Schema Registry is not used. Schemas are managed with Pydantic in Butler Writer.

Prompt Processing sends events containing dataset records to Prompt Kafka as Kafka messages. The Butler Writer Services batches up these messages and inserts into the Embargo Butler.

The Prompt Publication services reads Butler Ingestion events.

Further details in DMTN-310 Reducing Butler database contention in Prompt Processing

Architecture Diagram¶

        graph TD
   %% Node Definitions
   PPS[Prompt Publication Service]
   PKJ[prompt-keda-lsstcam<br/>KEDA Scaled Jobs]
   PK(prompt-kafka)
   BWS[Butler Writer Service]
   EB[(Embargo Butler)]

   %% Connections and Labels

   PKJ -- "Sends Dataset Records" --> PK
   PPS -- "Reads Butler Ingestion Events" --> PK
   PK --> BWS
   BWS -- "Batch Insert Dataset Records" --> EB

Associated Systems¶

The Butler Writer Service in the Butler Writer Service Namespace in the Prompt Processing vClusters sends events to the butler-writer and butler-writer-ingestion-events topics.

The Prompt Publication Service consumes events from the butler-writer-ingestion-events topic.

Configuration Location¶

Config Area	Location
Configuration	lsst-sqre/phalanx
Vault Secrets Dev	secret/rubin/usdf-prompt-processing-dev/prompt-kafka
Vault Secrets Prod	secret/rubin/usdf-prompt-processing/prompt-kafka

Data Flow¶

Details in Architecture diagram above.

Dependencies - S3DF¶

Weka for Persistent Volume Storage

Dependencies - External¶

No external dependencies