Procedures¶
Intended audience: Anyone who is administering Prompt Redis.
Deployment¶
Deployment is with Phalanx and ArgoCD.
Maintenance¶
The production hours are during observing. Maintenance can be performed during the day and should be announced in the lsstcam-prompt-processing Slack channel.
For development announce on the dm-prompt-processing Slack channel if performing maintenance.
Backup¶
The Redis instance does not need to be backed up currently. This may change if there is a catch up service deployed in the future. The configuration is in Phalanx to redeploy.
Cold Startup¶
No specific cold startup procedures.
Cold Shutdown¶
No specific procedures are needed to cleanly shutdown.
Reproduce Service¶
Deploy with Phalanx into a different vCluster or namespace. Follow the Bootstrapping Prompt Redis procedure to prep the Redis keys and consumer groups. Instances of Next Visit Fan Out and Prompt Processing will need to be deployed alongside to test application functionality.
Redis Command Line¶
Redis has a CLI to view and manage Redis Streams. From the appropriate dev or production Prompt Processing environment access the CLI with kubectl exec -it prompt-redis-0 -n prompt-redis -- redis-cli. Below displays a successful connection validated with a ping.
kubectl exec -it prompt-redis-0 -n prompt-redis -- redis-cli
127.0.0.1:6379> ping
PONG
Bootstrapping Prompt Redis¶
Once a new Redis instance is created with Phalanx. Follow the procedures in Creating Redis Streams
Creating Redis Streams¶
Redis Streams are created using the Redis Command Line. The XGROUP CREATE command is used to create the Redis Stream and Consumer Group. If the Redis Cluster is destroyed the streams need to be recreated with the below commands. For Dev:
XGROUP CREATE instrument:hsc hsc_consumer_group $ mkstream
XGROUP CREATE instrument:latiss latiss_consumer_group $ mkstream
XGROUP CREATE instrument:lsstcam lsstcam_consumer_group $ mkstream
For Prod:
XGROUP CREATE instrument:lsstcam lsstcam_consumer_group $ mkstream
Clear Redis Stream¶
To delete all the events in a Redis stream the DEL command can be used. Below is an example with LSSTCam to delete and recreate the Redis Stream. Please note there will be errors generated by Keda any Scaled Jobs connected to the stream when the stream is deleted.
127.0.0.1:6379> DEL instrument:lsstcam
(integer) 1
127.0.0.1:6379> XGROUP CREATE instrument:lsstcam lsstcam_consumer_group $ mkstream
OK
Viewing Message Statistics¶
Prompt Processing is configured to ignore messages that have already been read by another consumer. To view messages statistics for a consumer group enter XINFO GROUPS <consumer_group_name> with the Redis Command Line. An example below with the LSSTCam consumer group. The lag is 9 so 9 messages have not been acknowledged. To manually clear these messages see the Clear Redis Stream section.
127.0.0.1:6379> XINFO GROUPS instrument:lsstcam
1) 1) "name"
2) "lsstcam_consumer_group"
3) "consumers"
4) (integer) 704
5) "pending"
6) (integer) 0
7) "last-delivered-id"
8) "1740691320501-1"
9) "entries-read"
10) (integer) 72
11) "lag"
12) (integer) 9
In the above example there are 72 entries-read which is 72 messages acknowledged. pending is how many messages are being actively processed by consumers, but not acknowledged yet.
Viewing Pending messages¶
Pending messages are messages that have been delivered to a consumer within a consumer group but have not been acknowledged as processed. There should not usually be a build up of pending messages. To view if there are pending messages run XPENDING instrument:lsstcam lsstcam_consumer_group with the Redis Command Line. Replace if the instrument name and the consumer group for a different instrument.
Upgrading Redis¶
- To upgrade the Redis perform the following.
The Square team Redis chart is used. Review the Releases page for the correct version.
Update the Redis Chart version in Phalanx with the release number.
Refresh and Sync in ArgoCD
Restarting Redis¶
To restart the keda operator run kubectl rollout restart statefulset prompt-redis -n prompt-redis