Database Information¶
The following sections provide detail on the Minor Planet Center Replica.
Architecture¶
The mpcorb Postgres database is installed in the usdf-minor-planet-survey Kubernetes vCluster in the mpcorb-replica namespace with the Cloud Native Postgres Operator. Some of the data comes from Rubin, but is sent to the MPC Annex first via HTTP, loaded into the database, then replicated back to Rubin.
Postgres Logical Replication is used to replicate data from the MPC annex to the USDF. mpcorb depends on Internet connectivity to receive updates from the MPC Annex. The SLAC NAT IP was shared with the MPC Annex as the NAT addresses used for outbound connectivity with SLAC. If this address changes the MPC Annex will need to be notified.
mpcorb is also setup as a Postgres publication to replicate data to the EPO. Further details are in the Architecture Diagram and Data Flow sections of this page.
mpc_orbits table is largest table with 1 million rows. It will grow to 5 million rows over the course of the survey.
Architecture Diagram¶
Associated Systems¶
A cronjob is run to each night before observing with moving objects objects in the Solar System. This is used by Prompt Processing.
Pipelines code interacts with this database to review for potential new objects to submit to the the Minor Planet Center.
Configuration Location¶
Config Area |
Location |
|---|---|
Kubernetes Configuration |
|
SQL Configuration |
|
Vault Secrets Dev |
secret/rubin/usdf-minor-planet-survey/postgres |
Vault Secrets Prod |
N/A |
Data Flow¶
The Minor Planet Center Annex is the Postgres Logical Replica publication. The USDF Minor Planet Center replica is configured with subscription to the Obs table and to the other tables at the Annex.
Rubin EPO also needs a copy of this data. It was not an option to connect directly from EPO to the Minor Planet Center Annex so a double hop Postgres Replication is setup. The USDF Minor Planet Center replica is also setup as a Postgres publication to replicate the same tables that it subscribes to. EPO has a development, integration, and production environments setup as subscriptions in the Google Cloud with the Cloud Native Postgres (CNPG) Operator. Currently only dev is connected.
A separate pipeline is run to identify new objects and their orbits. These are sent to an HTTP endpoint at the Minor Planet Center which is the clearinghouse for new discoveries. Once the data is accepted it sent back to the USDF through logical replication.
Dependencies - S3DF¶
Below are S3DF Dependencies.
Kubernetes
SLAC LDAP to authenticate to the vCluster
DNS resolution for the SBN address
Weka storage for Kubernetes. The database uses a persistent volume claim.
Dependencies - External¶
Below are External Dependencies.
Internet connectivity to receive logical replication updates. Access is tied to the SLAC NAT IP.
Disaster Recovery¶
The sync for the sbn146_rubin_usdf_obs_table_sub took seven hours and forty minutes when the table size was 234 GB. There is one replication slot from the Minor Planet Center Annex.