Troubleshooting

Intended audience: Anyone who is administering Prompt Redis.

Known Issues

Issue	Description	Workaround
None

Monitoring

No Such Group or Key Error

Symptoms: A redis.exceptions.ResponseError: NOGROUP No such key 'instrument:lsstcam' or consumer group 'lsstcam_consumer_group' in XREADGROUP with GROUP option error is reported by Prompt Processing.

Cause: This can happen is the Redis Cluster is recreated and the keys and consumers groups are not bootstrapped.

Solution: Follow the Creating Redis Streams procedure to create the keys and consumer groups.

Prompt Processing Cannot Connect to Redis

Symptoms: Prompt Processing is reporting errors that it cannot connect to Redis.

Cause: The internal Kubernetes DNS name of prompt-redis.prompt-redis is used to connect to Prompt Processing.

Solution: Validate that the name of the service with the command below.

kubectl get svc -n prompt-redis

The issue could be DNS related. Restart the coredns deployment in the kube-system namespace with the kubectl rollout restart deployment coredns -n kube-system command which will restart the coredns pods. If neither fixes the issue contact usdf-infr-support channel on Slack for assistance.