Procedures¶
Intended audience: Anyone who is administering the S3 File Notifications Kafka cluster.
Deployment¶
Deployment is with Phalanx and ArgoCD.
Maintenance¶
Maintenance should be announced on the #lsstcam-prompt-processing Slack channel.
Backup¶
No backups required. The file notifications are not needed after they are processed.
Cold Startup¶
If the Brokers start before the Controllers restart the Controllers.
Cold Shutdown¶
No cold shutdown procedures needed.
Reproduce Service¶
Create another environment in Phalanx and deploy to a different namespace.
View Messages¶
Topics and messages can be viewed in Kafdrop at:
Pulling Messages¶
To manually download messages for examination run the below command. Update the IP to list the topics for dev.
apptainer exec kafka_0.34.0-kafka-3.4.0.sif /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 172.24.10.54:9094 --topic rubin-summit-notification-6 --group test-1 --from-beginning > prompt-file-notifications.txt
Creating File Notifications¶
The rubin-pp-dev bucket is used in Prompt Processing Dev. The rubin-summit bucket is used for Prompt Processing Prod. To create run the commands in the shell scripts at slaclab/usdf-embargo-deploy
Upgrading Kafka¶
See Upgrading Kafka