Troubleshooting¶
Intended audience: Anyone who is administering Prompt Processing.
Known Issues¶
Issue |
Description |
Workaround |
|---|---|---|
Missing data at USDF |
No prompt processing can be done if raw data did not arrive USDF embargo bucket within the waiting time. This is an upstream issue. |
Prompt Processing workers would time out after waiting a configurable amount of time. |
Missing nextVisit events |
No fanned out nextVisit are received after their publication. This is an upstream issue. |
See Next Visit Fan Out. |
Missing dataset types in the |
Typically new dataset types are registered in the |
Register manually: lsst-dm/prompt_processing |
Missing file arrival notifications |
The bucket is configured to send Kafka notifications when new objects are created. It has got disabled for unknown reasons in the past. Prompt Processing relies on these notifications together with the raw presence microservice to know that a raw file is available. This is a storage issue. |
If this is confirmed not to be a wider embargo storage issue and no other storage issues are present, one can try to reconfigure the bucket notifications. |
Images rejected as unprocessable |
Sometimes the nextVisit and the actual exposure are too inconsistent, or necessary exposure metadata is missing; these exposures are unprocessable. This is an upstream issue. |
Prompt Processing raises RuntimeError and skips the image. |
Incompatible stack versions between auto-ingest and prompt service. |
It could cause problems for Prompt Processing to write data to the embargo repo. |
Auto-ingest may need to update the stack version. |
Prompt service software bugs |
Major stack regression, middleware interface or activator bugs that are not discovered during testing and release process. |
Rollback to the previously working version and report to the Prompt Processing team. |
PipelineExecutionError |
Errors from pipeline payload execution, but the framework is correctly executing. |
Non-urgent but report to the Pipelines team |
Monitoring¶
Prompt Processing Production Overview
Prompt Processing Production output
Sample Troubleshooting¶
Symptoms:
Cause:
Solution: