NCSS Wiki

Bruce Worden developed the first version of the History of ShakeMap Developmentmodule which retrieves event and amp values from the database, and passes these elements onto the Shakemap queue module via files. His version is https://github.com/cbworden/shakemap-aqms

Gary Gann, who use to work at USGS Pasadena, took ownership of the code. He corrected some bugs, and contributed the aftershock exclusion feature. This version is available at https://github.com/shakemap-ci/shakemap-aqms

Gary's version got incorporated into the aqms-swg Git repo and is now maintained by Li Zhang, USGS Pasadena. https://gitlab.com/aqms-swg/aqms-shakemap/-/tree/master/shakemap_aqms This version works with the latest versions of Shakemap. Plus, Li introduced code to retrieve the 'reviewed origin' flag from the origin table.

Before Li took over the code, Renate, UW, cloned the module over to https://github.com/pnsn/shakemap-aqms-1/tree/postgres Renate added Postgres support. But this version lacks the inclusion of 'review origin' and is not compatible with the latest versions of Shakemap. Although, with some minor changes(cited below), it can become compatible. Renate also implemented the aqms_notify feature, which is used to execute a command when Shakemap runs for an event.

So, Renate's version is the one to use. However, Li made changes to his version for reviewed origin inclusion. The two versions still need to be merged.

Here's a brief guide on setting up the PNSN shakemap-aqms module.

1 After installing Shakemap, activate the environment conda activate shakemap

2 Retrieve the shakemap-aqms postgres branch from PNSN wget -w postgres.zip https://github.com/pnsn/shakemap-aqms-1/archive/refs/heads/postgres.zip

3 Install prereq packages for supporting Oracle and Postgres db connections, and impactutils (which replaced shakelib).

conda install psycopg2
conda install cx_Oracle
conda install impactutils

4 Intstall Oracle Instant Client and set LD_LIBRARY_PATH Refer to https://cx-oracle.readthedocs.io/en/latest/user_guide/installation.html An Oracle account may be needed to download the InstantClient or copy the client directories from another system (eg. rodgers).

The modified versions of aqms_db2xml.py and aqms_eq2xml.py, for supporting Postgres via psycopg2 module, still requite cx_Oracle, which in turn require the instantclient. Refer to aqms_db2xml as an example of “import cx_Oracle” shakemap-aqms-1/shakemap_aqms/coremods/aqms_db2xml.py at postgres · pnsn/shakemap-aqms-1 · GitHub

Similar to try:

  import psycopg2

except Exception as e:

  print("postgresql not enabled")

You could wrap the import cx_Oracle' in a try/except clause - then you may not have to install cx_Oracle or the instantclient replace:       import cx_Oracle with try:

  import cx_oracle

except Exception as e:

  print("oracle not enabled")

This may or may not work - there may be other areas of the code that need to be modified; though I don't think so.

5 Install shakemap-aqms unzip postgres.zip pip install ./shakemap-aqms-1-postgres

6 Copy the shakemap-aqms binaries/scripts to the desired location. For example, where the shakemap binaries are. eg. cp shakemap-aqms-1-postgres/bin/* /home/shake/miniconda/envs/shakemap/bin/

7 If using the PNSN Postgres version, modifications are needed for shakemap-aqms to work with the latest Shakemap https://github.com/shakemap-ci/shakemap-aqms/issues/4

Use find “eg, find $HOME -name aqms_eq2xml.py -ls” to locate these files within the conda environment directory.

In the files …./shakemap_aqms/coremods/aqms_eq2xml.py Change from shakelib.rupture.origin import write_event_file to from impactutils.rupture.origin import write_event_file

…./shakemap_aqms/coremods/aqms_db2xml.py Change from shakelib.rupture.origin import Origin to from impactutils.rupture.origin import Origin

…./shakemap_aqms/coremods/aqms_notify.py Change from shakelib.rupture.origin import Origin to from impactutils.rupture.origin import Origin

8 Edit aqms.conf Add/edit database stanzas, and specify either driver = postgres or driver=oracle for each active stanza. Also at the end, add a new element ([notify]) which can be left without a definition and implemented later. Refer to the code on the expected definition. https://github.com/pnsn/shakemap-aqms-1/blob/postgres/shakemap_aqms/coremods/aqms_notify.py

[notify]

9 Try starting the aqms_queue script which listens for alarms. Confirm that the system_ip:2345 is listed in the shake_hosts_ports file for the ShakeMap alarm on the rt and pp systems, and that port 2345 is open. python /home/shake/miniconda/envs/shakemap/bin/aqms_queue

I think the above will get close to a working shakemap_aqms.

Jeff

Here's an overview the ShakeMap systems that are running

mnlodd1new; active Oracle ShakeMap 4.0.2+386, does not support aftershock exclusion distributes products to USGS production PDL hubs

rodgers; standby Oracle ShakeMap 4.0.2+442, does not support aftershock exclusion distributes products to USGS development PDL hub (may be temporarily disabled)

ucbrt-test; test Oracle Shakemap 4.1.5+87, supports aftershock exclusion, and currently being used for testing reviewed-origin as a PDL property distributes products to USGS development PDL hub

mnlosm1; test Postgres ShakeMap 4.1.5+4, supports aftershock exclusion, and currently being used for testing reviewed-origin as a PDL property Upgraded to 4.1.5 on 11/6/2024 This instance will likely migrate to aqms-nc-aux, after some testing.

mnlodd1; deprecated Oracle ShakeMap v3.5, to be shutdown

I think we need a Postgres ShakeMap at BSL that can operate in active (publish products to USGS production PDL) or standby (publish to USGS dev PDL). And we might want to go back to the ShakeMap resiliency discussion, now that there's a higher bandwidth private connection between Menlo and Berkeley. One idea proposed was to distribute ShakeMap products to the NCSS PDL hubs, similar to how origin and other products are distributed. NEIC has the choice of pulling products from either ucbrt1 or mnlort. This approach may not work out - maybe because the ShakeMap products consist of many files.

Shakemap products can be produced from the command line - eg

conda activate shakemap

shake 60101616

If you don't want the products transferred to pdl, make sure that transfer_pdl isn't listed in the autorun_modules configuration within the shake.conf file. I think the default location for output is $HOME/shakemap_profiles/default/data/<evid> If you enable transfer_pdl, make sure the transfer.conf file references a PDL config.ini file, which specifies the pdldevel.cr.usgs.gov system as the PDL hub, rather than the production hubs.

This should also test the aqms_eq2xml and most importantly aqms_db2xml modules - if they're listed in autorun_modules. Within the output directory $HOME/shakemap_profiles/default/data/<evid>/current There should be a set of dat files which contain the ground motions retrieved from the database. These are a little easy to peruse than the $HOME/shakemap_profiles/default/data/<evid>/products/stationlist.json file. Ofcourse, publishing to staging-earthquake provides an easier way to evaluate the products generated. Just keep track of the version before and after running 'shake' - particularly if a shakemap was already generated and posted for the event. https://staging-earthquake.usgs.gov/earthquakes/eventpage/nc60101616/shakemap/intensity

To test aqms_queue, the alarm listener

telnet localhost 2345 

and enter a line like

shake_alarm 60101616 2 

Some notes:

• I think by default aqms_queue runs on port 2345.
• The last # in the shake alarm string is map version. With version 1 already at staging-earthworm, this value can be incremented when generating new shakemaps.
• I think malformed strings issued to aqms_queue will crash it. It should follow the convention “shake_alarm <evid> <version>”
• After issuing the shake_alarm, view the shake.log within $HOME/shakemap_profiles/default/logs
• If you make any changes to shake.conf, restart 'queue'. The same may apply for any config changes.