Procedures

Intended audience: Anyone who is administering the service.

Deployment

Deployment is with Phalanx and ArgoCD.

Maintenance

The production hours are during observing. Maintenance can be performed during the day and should be announced in the lsstcam-prompt-processing Slack channel. For development announce on the dm-prompt-processing Slack channel if performing maintenance.

Backup

No Kafka backups are taken. Snapshots are done at the Weka storage level.

Cold Startup

Validate that both the controllers and brokers are running. Depending on when nodes are started controllers and brokers may need to be restarted.

Cold Shutdown

No specific procedures are needed to cleanly shutdown.

Reproduce Service

Create another values file to deploy a new environment.

View Messages

Topics and messages can be viewed in Kafdrop at:

• Kafdrop S3 Prod

• Kafdrop S3 Dev

Create New Kafka Topics

Kafka topics are created with Helm subcharts. To create a new Kafka topic perform the following.

1. Create a new folder in the Phalanx repo under prompt-kafka/charts. An existing folder like butler-writer or prompt-publication could be reviewed for an example and copied.

2. Configure the topic and users Kubernetes manifests

3. Add a dependency to the strimzi-kafka chart. Add in template. Replace with name of subchart.

   - name: <name of subchart>
       condition: <name of subchart>.enabled
       version: 1.0.0
   

4. Enable for the environment. Add in template. Replace with name of subchart.

   - <name of subchart>:
       enabled: true
   

Restarting Kafka

See Restart Kafka

Upgrading Kafka

See Upgrading Kafka

Upgrading Strimzi Operator

See Upgrading Strimzi Operator