Application Information¶

Architecture¶

The central component of APDB is Apache Cassandra database. The database serves approximately 200 clients of Prompt Processing system concurrently reading and writing to the database. Cassandra stores its data on fast local storage (NVMe disks in raidz2 array). Presently there are 12 nodes at USDF (sdfk8sk001-012) split between two Cassandra clusters (prod and dev). Additional integration cluster (int) uses five extra nodes (sdfcasdev001-003 and sdfcasdev101-102).

Backups of the Cassandra files are managed by cassandra-medusa service running on each cluster node. Backups are stored in S3 bucket, off-site storage of backups is not implemented yet.

A separate application reads data from APDB and replicates it to PPDB, which will likely be implemented on top of Google BigQuery.

Architecture Diagram¶

        flowchart TD
        pp@{ shape: procs, label: "Prompt<br/>Processing" }
        daytime@{ shape: procs, label: "Daytime<br/>Catchup" }
        dedup@{ shape: proc, label: "Dedup" }
        subgraph Cassandra Node x N
          c8@{ shape: proc, label: "Cassandra"}
          disk@{ shape: lin-cyl, label: "Local<br/>storage" }
          medusa@{ shape: proc, label: "Medusa<br/>backup" }
          c8 <--> disk
          disk --> medusa
        end
        pp <--> c8
        daytime <--> c8
        dedup <--> c8
        medusa <--> S3disk@{ shape: lin-cyl, label: "S3 Bucket" }
        c8 --> replication["Replication"]
        replication --> ppdb[("PPDB")]

Replication process is an application separate from APDB.

Associated Systems¶

Prompt Processing

Configuration Location¶

Config Area	Location
Configuration	lsst-dm/dax_apdb_deploy
Vault Secrets Dev	rubin/usdf-apdb-dev/cassandra
Vault Secrets Dev (superuser)	rubin/usdf-apdb-dev/cassandra-super
Vault Secrets Prod	rubin/usdf-apdb-prod/cassandra
Vault Secrets Prod (superuser)	rubin/usdf-apdb-prod/cassandra-super
S3 bucket for backups	usdf-apdb-prod
Vault Secrets for S3	rubin/usdf-apdb-prod/s3

Data Flow¶

Prompt Processing jobs query APDB for the pre-existing data in the region covered by visit-detector.
Based on the images and pre-existing data Prompt Processing generates a set of records that it saves to Cassandra.
This repeats for every visit during the night, approximately 1k visits are expected during each night.
Daytime catchup processing works similarly to Prompt Processing working on images that failed to process during the night.
Deduplication process also runs during daytime, it analyzes records generated during the night and performs cleanup on some of them.
Newly generated data are copied from Cassandra periodically by a replication process.
Backups are taken periodically once a day and are uploaded to an S3 bucket.
Off-site backup is not implemented yet.

Dependencies - S3DF¶

Vault (for client and superuser passwords).
S3 for backups.

Dependencies - External¶

Off-site backup, which is not implemented yet, will likely depend on some external S3 service.

Disaster Recovery¶

In case of disaster the content of database can be restored from the latest backup. Recovery procedure consists of running medusa-cassandra service in a special recovery mode which copies data from S3 to the Cassandra hosts. Cassandra cluster needs to be configured with the same number of nodes and preferably with the same IPs. Recovery of the data from S3 depends on the amount of data and location of backup (on-site vs off-site) and it could take anywhere between few hours and a day or longer. dax_apdb_deploy README documents recovery procedure.