===== ShakeMap For NCSS ===== NCSS operates two systems for generating [[https://earthquake.usgs.gov/data/shakemap/|ShakeMaps]]: rodgers.geo.berkeley.edu and mnlodd1.wr.usgs.gov. This page describes how ShakeMap production is handled in the NCSS. See the [[https://usgs.github.io/shakemap/|ShakeMap Manual]] for general information. The NCSS ShakeMap systems are currently running version 3.5 and should be kept up to the most recent SVN release. When ShakeMap version 4 is released perhaps within the next year, most of the below information will need to changed! ----------------------------------------------------------- == ShakeMap Version 4.0 == In December 2020, MP and UCB switched production of ShakeMaps from version 3.5 to version 4.0. In this subsection we will begin to document information regarding running and maintaining ShakeMaps in version 4.0 on both the MP and UCB systems. The information on this wiki for running ShakeMap 3.5 is maintained below. == ShakeMap v4.0 - Where Is ShakeMap in NCSS? == info on accessing systems here Connect to VPN and use SSH to login to MP or UCB systems to access ShakeMap ShakeMap 4.1.1 on Virtual Machine (currently in testing mode) MP - login to mnlosm1 > ssh -l shake mnlosm1 ShakeMap 4.0 MP - login to mnlodd1new > ssh -l shake mnlodd1new UCB - login to seiche > ssh -l ncss seiche.geo.berkeley.edu ShakeMap 3.5 MP- login to mnlodd1 > ssh -l shake mnlodd1 UCB - login in to rodgers (may have been migrated due to server change?) > ssh -l ncss rodgers.geo.berkeley.edu === ShakeMap v4.0 - Steps to Run Map === Auto Run All Modules Example example to re-run a map > shake -a 73584926 The above will run the autorun modules set in shake.conf. The current modules are (updated 12/6/2022 to implement select): autorun_modules = aqms_eq2xml aqms_db2xml associate dyfi select assemble model mapping contour coverage gridxml info kml shape plotregr raster uncertaintymaps rupture stations transfer_pdl transfer_scp transfer_email Here is an example on how to re-run one of the aftershocks of the Antelope Valley earthquake that occurred July 2021: > conda activate shakemap > shake -a 73585021 The terminal will output showing it is processing like below and ask for a comment. Just click enter to continue processing. All the modules above will run, including the transfer to NEIC and update the map. (shakemap) [shake@mnlodd1new data]$ shake -a 73585021 In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The text.latex.preview rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The mathtext.fallback_to_cm rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: Support for setting the 'mathtext.fallback_to_cm' rcParam is deprecated since 3.3 and will be removed two minor releases later; use 'mathtext.fallback : 'cm' instead. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The validate_bool_maybe_none function was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The savefig.jpeg_quality rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The keymap.all_axes rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The animation.avconv_path rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The animation.avconv_args rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. WARNING -- 2021-07-14 17:37:11 -- shake.main -- The -a (--autorun) option is deprecated. It is now the default behavior to use the autorun_modules in shake.conf when no modules are specified. INFO -- 2021-07-14 17:37:11 -- shake.main -- No modules specified, checking for autoun_modules... INFO -- 2021-07-14 17:37:11 -- shake.main -- Autorun modules found: ['aqms_eq2xml', 'aqms_db2xml', 'associate', 'dyfi', 'assemble', 'model', 'mapping', 'contour', 'coverage', 'gridxml', 'info', 'kml', 'shape', 'plotregr', 'raster', 'uncertaintymaps', 'rupture', 'stations', 'transfer_pdl', 'transfer_scp', 'transfer_email'] INFO -- 2021-07-14 17:37:12 -- shake.main -- Running command aqms_eq2xml INFO -- 2021-07-14 17:37:12 -- shake.main -- Finished running command aqms_eq2xml: Elapsed 0.69 secs INFO -- 2021-07-14 17:37:12 -- shake.main -- Running command aqms_db2xml INFO -- 2021-07-14 17:37:31 -- shake.main -- Finished running command aqms_db2xml: Elapsed 18.55 secs INFO -- 2021-07-14 17:37:31 -- shake.main -- Running command associate INFO -- 2021-07-14 17:37:31 -- shake.main -- Finished running command associate: Elapsed 0.00 secs INFO -- 2021-07-14 17:37:31 -- shake.main -- Running command dyfi INFO -- 2021-07-14 17:37:43 -- dyfi.execute -- Wrote 34 DYFI records to /home/shake/shakemap_profiles/default/data/73585021/current/dyfi_dat.xml INFO -- 2021-07-14 17:37:43 -- shake.main -- Finished running command dyfi: Elapsed 12.33 secs INFO -- 2021-07-14 17:37:43 -- shake.main -- Running command assemble Please enter a comment for this version. comment: INFO -- 2021-07-14 17:37:55 -- shake.main -- Finished running command assemble: Elapsed 11.75 secs INFO -- 2021-07-14 17:37:55 -- shake.main -- Running command model INFO -- 2021-07-14 17:38:59 -- shake.main -- Finished running command model: Elapsed 64.01 secs INFO -- 2021-07-14 17:38:59 -- shake.main -- Running command mapping INFO -- 2021-07-14 17:39:49 -- shake.main -- Finished running command mapping: Elapsed 50.16 secs INFO -- 2021-07-14 17:39:49 -- shake.main -- Running command contour INFO -- 2021-07-14 17:39:53 -- shake.main -- Finished running command contour: Elapsed 3.78 secs INFO -- 2021-07-14 17:39:53 -- shake.main -- Running command coverage INFO -- 2021-07-14 17:39:55 -- shake.main -- Finished running command coverage: Elapsed 2.10 secs INFO -- 2021-07-14 17:39:55 -- shake.main -- Running command gridxml INFO -- 2021-07-14 17:39:58 -- shake.main -- Finished running command gridxml: Elapsed 2.44 secs INFO -- 2021-07-14 17:39:58 -- shake.main -- Running command info INFO -- 2021-07-14 17:39:58 -- shake.main -- Finished running command info: Elapsed 0.00 secs INFO -- 2021-07-14 17:39:58 -- shake.main -- Running command kml INFO -- 2021-07-14 17:40:08 -- shake.main -- Finished running command kml: Elapsed 9.98 secs INFO -- 2021-07-14 17:40:08 -- shake.main -- Running command shape INFO -- 2021-07-14 17:40:13 -- shake.main -- Finished running command shape: Elapsed 4.77 secs INFO -- 2021-07-14 17:40:13 -- shake.main -- Running command plotregr INFO -- 2021-07-14 17:40:21 -- shake.main -- Finished running command plotregr: Elapsed 7.70 secs INFO -- 2021-07-14 17:40:21 -- shake.main -- Running command raster WARNING -- 2021-07-14 17:40:21 -- warnings._showwarnmsg -- /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/mapio/gdal.py:490: DataSetWarning: 'Down-casting double precision floating point to single precision' warnings.warn(DataSetWarning('Down-casting double precision ' INFO -- 2021-07-14 17:40:22 -- shake.main -- Finished running command raster: Elapsed 0.87 secs INFO -- 2021-07-14 17:40:22 -- shake.main -- Running command uncertaintymaps INFO -- 2021-07-14 17:40:42 -- shake.main -- Finished running command uncertaintymaps: Elapsed 20.81 secs INFO -- 2021-07-14 17:40:43 -- shake.main -- Running command rupture INFO -- 2021-07-14 17:40:43 -- rupture.execute -- Writing rupture.json file... INFO -- 2021-07-14 17:40:43 -- shake.main -- Finished running command rupture: Elapsed 0.00 secs INFO -- 2021-07-14 17:40:43 -- shake.main -- Running command stations INFO -- 2021-07-14 17:40:43 -- shake.main -- Finished running command stations: Elapsed 0.50 secs INFO -- 2021-07-14 17:40:43 -- shake.main -- Running command transfer_pdl INFO -- 2021-07-14 17:40:43 -- transfer_base.execute -- Making backup directory... INFO -- 2021-07-14 17:41:31 -- transfer_pdl.execute -- 70 files sent. Message from sender: "70 files sent successfully: resulting in output: "urn:usgs-product:nc:shakemap:73585021:1626284444255 "" INFO -- 2021-07-14 17:41:31 -- shake.main -- Finished running command transfer_pdl: Elapsed 47.46 secs INFO -- 2021-07-14 17:41:31 -- shake.main -- Running command transfer_scp INFO -- 2021-07-14 17:41:32 -- transport._log -- Connected (version 2.0, client OpenSSH_7.4) INFO -- 2021-07-14 17:41:32 -- transport._log -- Authentication (publickey) successful! Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/shake_result.hdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/mmi_legend.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pin-thumbnail.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity_overlay.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity_overlay.pngw... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_mmi.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_mi.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_pga.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_pgv.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_psa0p3.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_psa1p0.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_psa3p0.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_mmi_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_mmi_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_mmi_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pga_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pga_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pga_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pgv_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pgv_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pgv_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa0p3_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa0p3_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa0p3_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa1p0_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa1p0_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa1p0_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa3p0_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa3p0_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa3p0_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/grid.xml... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/uncertainty.xml... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/info.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/shakemap.kmz... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/shape.zip... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/mmi_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/attenuation_curves.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/raster.zip... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/rupture.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/stationlist.json... INFO -- 2021-07-14 17:42:15 -- transfer_scp.execute -- 69 files sent. Message from sender: "69 files sent to remote host eoc1.cisn.org" INFO -- 2021-07-14 17:42:16 -- transport._log -- Connected (version 2.0, client OpenSSH_7.4) INFO -- 2021-07-14 17:42:16 -- transport._log -- Authentication (publickey) successful! Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/shake_result.hdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/mmi_legend.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pin-thumbnail.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity_overlay.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity_overlay.pngw... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_mmi.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_mi.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_pga.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_pgv.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_psa0p3.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_psa1p0.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/cont_psa3p0.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_mmi_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_mmi_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_mmi_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pga_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pga_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pga_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pgv_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pgv_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_pgv_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa0p3_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa0p3_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa0p3_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa1p0_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa1p0_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa1p0_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa3p0_high_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa3p0_medium_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/coverage_psa3p0_low_res.covjson... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/grid.xml... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/uncertainty.xml... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/info.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/shakemap.kmz... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/shape.zip... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/mmi_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0_regr.png... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/attenuation_curves.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/raster.zip... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/intensity_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pga_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/pgv_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa0p3_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa1p0_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0_uncertainty.pdf... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/psa3p0_uncertainty.jpg... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/rupture.json... Copying /home/shake/shakemap_profiles/default/data/73585021/current/products/stationlist.json... INFO -- 2021-07-14 17:43:00 -- transfer_scp.execute -- 69 files sent. Message from sender: "69 files sent to remote host www2.cisn.org" INFO -- 2021-07-14 17:43:00 -- shake.main -- Finished running command transfer_scp: Elapsed 88.16 secs INFO -- 2021-07-14 17:43:00 -- shake.main -- Running command transfer_email INFO -- 2021-07-14 17:43:00 -- transfer_email.execute -- Mail has already been generated for this event. Returning. INFO -- 2021-07-14 17:43:00 -- shake.main -- Finished running command transfer_email: Elapsed 0.00 secs INFO -- 2021-07-14 17:43:00 -- shake.main -- shake finished with event 73585021 (shakemap) [shake@mnlodd1new data]$ In some cases you may want to run individual modules for testing, etc. Steps To Run Each Module: First activate shakemap >conda activate shakemap create using us evid (or use aqms_eq2xml aqms_db2xml as above to acquire from aqms drop in module) >sm_create --force 73584926 manually run each module (can do one at a time and selectively, for example to avoid auto transfer when testing) >shake --force 73584926 associate >shake --force 73584926 dyfi >shake --force 73584926 assemble >shake --force 73584926 model >shake --force 73584926 mapping >shake --force 73584926 contour >shake --force 73584926 coverage >shake --force 73584926 gridxml >shake --force 73584926 info >shake --force 73584926 kml >shake --force 73584926 shape >shake --force 73584926 plotregr >shake --force 73584926 raster >shake --force 73584926 uncertaintymaps >shake --force 73584926 rupture >shake --force 73584926 stations === Staging vs. Production sites === To push to staging vs. production: edit the pdl config /home/shake/run/params/config.ini There are two lines near the beginning defining the receiving hubs - confusingly named senders senders = prod01, prod02 #senders =pdldevel All three prod01, prod02, pdldevel are defined later in the configuration. I commented the first 'senders' line and uncommented the second line so products would get sent through to the staging earthquake website. I then run shake manually with the eventid. After the run, I reverse the comment changes in config.ini. staging website links https://jmfee-usgs.github.io/comcat-timeline/timeline.html#nc73584926 https://jmfee-usgs.github.io/comcat-timeline/timeline.html#nc73498145|staging-earthquake.usgs.gov === ShakeMap v4.0 - Finite Fault Descriptions for ShakeMap === Acquire finite fault in json format. For example, we got this rupture.json file from Dave Wald used in the US (ShakeMap Central) ShakeMap and included it in the NC shakemap https://earthquake.usgs.gov/realtime/product/shakemap/nc73584926/us/1625968974704/download/rupture.json (note: Checking on details for how rupture.json was resolved, for Antelope Valley we had InSAR data - see email from Bill Barnhardt) 4.4.6. Rupture Specification describes the format of the rupture.json file https://usgs.github.io/shakemap/manual4_0/sg_input_formats.html Place rupture.json file in the event directory. For nc73584926, that's something like /home/shake/shakemap_profile/default/data/73584926/current After the rupture module runs, a similar file to the input is written to the subdir products. === ShakeMap v4.0 - How to manually exclude stations from ShakeMap === Navigate to the model.conf configuration file and search for "bad_stations" > cd /home/shake/shakemap_profiles/default/install/config Use vi or other editor to add station to exclude under the bad_station section. For example, to exclude NN.WAK following the July 2021 Antelope Valley earthquake: > vi model.conf (do escape i to insert/edit in vi and escape :x! to save or escape :q! to quit without editing) #--------------------------------------------------------------------------- # Bad stations: Stations may be manually flagged as "bad", and will be # excluded from processing. Each line in the bad_stations section # takes the form of "net.sta = ondate:offdate. E.g: # # CI.XYZ = 2017-04-29:2018-06-01 # # The on date and off date take the form of YYYY-MM-DD and are compared # with the earthquake date to determine if the station should be # excluded or not. If there is no off date (i.e., the station is still # bad), then the off date should be excluded but the ':' should be # included: # # CI.WXY = 2017-04-29: # # If the start date is arbitrarily early, use 1970-01-01 and the station # will be excluded for all dates prior to the stop date (if supplied). #--------------------------------------------------------------------------- [[bad_stations]] #excluded starting at 07-08-2021 Antelope Valley CA M6.0 earthquake (according to NN operator, bad station) NN.WAK = 2021-07-07:2021-07-11 Note: as of 2021-07-12 NN had replaced the sensor at WAK so we adjusted the timeframe to include this station in future ShakeMap runs. Some info on manual flagging is on the SM v4.0 wiki here: https://usgs.github.io/shakemap/manual4_0/sg_input_formats.html (note: in SM3.5 grind.conf was used to exclude stations. This function is no longer included in SM4, so the steps above are the new approach to manually exclude stations) The ShakeMap can be re-run to exclude the recently excluded station with the auto run command, i.e.: > conda activate shakemap > shake -a 73584926 You can check if station was manually flagged and excluded by looking at the station.json product file, i.e.: {"type": "Feature", "id": "NN.WAK", "properties": {"code": "WAK", "name": "Walker, California w84", "instrumentType": "UNK", "source ": "NN", "network": "NN", "commType": "UNK", "location": "", "intensity": "null", "intensity_flag": "", "intensity_stddev": "null", "pga": "nul l", "pgv": "null", "distance": 9.556, "elev": null, "vs30": 648.35, "channels": [{"name": "HNE", "amplitudes": [{"name": "pga", "value": 0.005, "units": "%g", **"flag": "ManuallyFlagged,Outlier", "ln_sigma": 0.0}**, {"name": "pgv", "value": 0.0012, "units": "cm/s", **"flag": "ManuallyFlagged ,Outlier",** "ln_sigma": 0.0}, ... === ShakeMap v4.0 - Configuration === info on fault source, Vs30, etc here link to central github for CISN here (USGS gitlab?) === Canceling a ShakeMap === > shake --cancel   Example. With the Dec 2020 M6.2 Petrolia Event, later in Jan 2022 the event was separated into a foreshock and mainshock. We wanted to remove the M5.7 foreshock shakemap that was auto generated We had to modify the config shake.conf to remove commented out cancel # cancel_modules: If the -c (--cancel) option is used, this # parameter specifies teh modules that will be run and if the # --cancel option will be passed onto these other modules. # # cancel_modules = module1 --cancel module2 --cancel # #cancel_modules = transfer_pdl --cancel transfer_email --cancel to cancel_modules = transfer_pdl --cancel command is > shake --cancel 71127029 The cancellation was successful and example output for this run was (Note: shakemap may need to be removed manually on CISN and backup pages, contact Jennifer Taggart). Right now this event appeared on the new and old CISN shakemap pages, respectively - https://www.cisn.org/shakemap/ and https://www.cisn.org/shakemap/nc/shake/archive/ (shakemap) [shake@mnlodd1new config]$ shake --cancel 71127029 In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The text.latex.preview rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The mathtext.fallback_to_cm rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: Support for setting the 'mathtext.fallback_to_cm' rcParam is deprecated since 3.3 and will be removed two minor releases later; use 'mathtext.fallback : 'cm' instead. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The validate_bool_maybe_none function was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The savefig.jpeg_quality rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The keymap.all_axes rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The animation.avconv_path rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. In /home/shake/.conda/envs/shakemap/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: The animation.avconv_args rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later. INFO -- 2022-01-14 22:35:44 -- shake.main -- Cancelling shakemap, using cancel_modules... INFO -- 2022-01-14 22:35:44 -- shake.main -- Cancel modules found: ['transfer_pdl', '--cancel'] INFO -- 2022-01-14 22:35:44 -- shake.main -- Running command transfer_pdl INFO -- 2022-01-14 22:36:24 -- shake.main -- Finished running command transfer_pdl: Elapsed 38.82 secs INFO -- 2022-01-14 22:36:24 -- shake.main -- shake finished with event 71127029 === ShakeMap v4.0 - How to Switch Production and Backup Status between EMP and UCB === To switch the ShakeMap instance from MP and UCB and vice-versa, simply remove/add the transfer module to the list of modules to be run for an incoming event in the configuration file shake.conf. shake.conf can be found and edited under the directory /home/shake/shakemap_profiles/default/install/config When the shakemap instance is in production mode (not backup) it will include the transfer_pdl, transfer_scp, and transfer_email modules like below (select implemented in CA 12/6/2022): autorun_modules = aqms_eq2xml aqms_db2xml associate dyfi select assemble model mapping contour coverage gridxml info kml shape plotregr raster uncertaintymaps rupture stations transfer_pdl transfer_scp transfer_email (double checking on this step) ----------------------------------------------------------- == ShakeMap Version 3.5 == One of the purposes of ShakeMap is to show as good a representation of ground shaking as possible for an event. The earthquake parameters (time, location, magnitude) are the starting point. But ShakeMap is not much good without measurements of ground shaking. For NCSS, that means ground motion amplitudes, which are stored in the //amp// table of NCSS AQMS databases. The NCSS RT systems determine ground motion amplitudes for all the broadband and accelerometer waveforms that are processed by those RT systems. There are many other stations whose waveforms do not go through the NCSS RT systems. Various systems (Caltech's AQMS system, CGS, NSMP, etc.) process those non-NCSS waveforms and submit ground motion amps to the [[postproc:amp_import|amp import]] system running on the post-processing systems. The result is that ground motion amps can trickle into the NCSS AQMS databases over hours, days, months after significant earthquakes. There is no easy way for ShakeMap or the ShakeMap operator to determine whether new amps are in the database since the last time a ShakeMap was produced. If in doubt, run //retrieve// by hand and see if more amps are collected than the previous time. == Where Is ShakeMap in NCSS? == * rodgers.geo.berkeley.edu: * user: //ncss// * base directory: /home/ncss/shake * run script: /home/ncss/run/bin/run_shakecontrol, called by /home/ncss/run/bin/run_all, which is called by the //ncss// init script. * ShakeMap commands (in /home/ncss/shake/bin) are NOT in PATH. * mnlodd1.wr.usgs.gov * user: //shake// * base directory: /home/shake * run script: /home/shake/run/bin/run_shakecontrol, called by /home/shake/run/bin/start_shake, which is a cron job. * ShakeMap commands (in /home/shake/bin) ARE in PATH. In both systems are the following directories under the ShakeMap base directory: * bin holds all the ShakeMap executable programs * config holds configuration files * data holds directories for each event and scenario containing their ShakeMaps and intermediate files. * lib contains a number of files used by various programs * logs contains the log files created by ShakeMap programs * perl: perl modules specific to ShakeMap * pw: password file for each of the databases used by ShakeMap, Oracle and MySQL. The main ShakeMap components that are of interest are: * shake: the primary program for generating a ShakeMap for one event. It's configuration file lists all the subsidiary programs that must be run, in order, to create and publish the maps. //shake -help// will give you a summary. * retrieve: runs a couple of other programs to //retrieve// the input data for ShakeMap. It runs: * eq2xml retrieves event parameters from Oracle databases, writes //event.xml// in the event's input directory. * db2xml retrieves ground motion amps from Oracle databases, writes //dbX_dat.xml// (X is 0, 1, ..., one for each database returning data) in the event's input directory. This is because amps written to the RT database by ampgen take up to several minutes to replicate to the DC databases, while amps imported from other sources will only ever be in the DC databases. ShakeMap (grind, mostly) uses the union of these multiple *dat.xml files. * grind: Process input ground motion parameters and estimate ground motions in map area. If you need to adjust the scale or centering of a ShakeMap, grind offers several command line options. Run //grind -help// for details of this. * tag: minor program for adding tags to the ShakeMap catalog list, which is visible only at http://www.cisn.org/shakemap/nc/shake/. Has no effect on ShakeMaps submitted to USGS EHP. * mapping generates PostScript GMT maps of computed ground motion parameters. * genex generates customized files for export. * transfer: Transfers data files to web, ftp, pdl, and push destinations. * shakemail: Mail shakemap alert to user list. * setversion: Manipulate the version information for an event. This version information is stored in a MySQL database exclusively for ShakeMap. * plotregr: Plot site-corrected station ground motion parameters and regression values. The plots are stored in each event's regression directory. * queue: Program to receive event alarms from the real time system and run //shake// with the appropriate event ID. * set_shake: Used by NCSS ShakeMap operators to change their system's configuration between //production// and //backup// modes. In backup mode, ShakeMaps are produced but not published. In production mode, the maps are published to all the destinations listed in transfer.conf-prod. To see which mode your ShakeMap system is in, see the comments are the top of set_shake.conf. set_shake is unique to NCSS, not part of the ShakeMap distribution. * cpdb is an NCSS addon for ShakeMap. It is used on the //production// ShakeMap system (configured in shake.conf) to make a dump of some mysql database tables to a file. The file gets sent using Earthworm sendfile/getfile to the backupShakeMap system. If needed, this dump file couldbe loaded into the mysql database to get its tables upto date. This might be useful if the backup SHakeMap system were out of service while the production ShakeMap system created new ShakeMaps. * shake_alarm, shake_cancel can be used to signal the local //queue// to run or cancel a ShakeMap for one event. * missamps can be used to list stations whose amp data is missing from a ShakeMap. This is experimental code that has been shared with other CISN ShakeMap operators. === Queue === //queue// is the program for controlling the automatic running of the ShakeMap programs. It runs all the time, listening on TCP port 2345 (hard-coded). * Started by the run script listed above * Configured by queue.conf in the ShakeMap //config// directory. * Accepts socket connections from the configured list of servers. These are the RT and post-processing systems from which an an alarm action would call for ShakeMap. Connections from other hosts are immediately closed and logged. * Expects a message of ''ACTION EVENT_ID REPEAT'' where ACTION is **shake_alarm** or **shake_cancel**. Apparently REPEAT is ignored by queue. * If ACTION is //shake_cancel//: * the event ID is added to the //cancelled_events// file in the logs directory. This prevents queue from ever producing a ShakeMap for this event again! If you need to **un-cancel** this event, you have to manually delete the event ID from the //cancelled_events// file. * Runs //shake// to cause the ShakeMap to be removed from servers. * If ACTION is //shake_alarm//: * Check that the event is not in the //cancelled_events// file; if it is, no further action is taken. * Query one or more Oracle databases listed in config/db.conf for event information, in the configured order, stopping when it finds the event. * For most of Northern California, the magnitude must be at least 3.5. * For the Mendocino offshore area, the magnitude must be at least 4.0: smaller offshore events won't produce any shaking on land. This is controlled by a ''Metro box'' in queue.conf. * If the event meets these limits, it is added to a queue of events to be processed. Priority is higher for larger magnitudes, for events in the SF Bay area, and for newer events. Details, with lots of comments, are in queue.conf. * Runs the //shake// program on this event as soon as queue has available time. This causes a ShakeMap to be produced for this event. //queue// is configured with a **repeat** schedule based on event magnitude; see queue.conf. Larger events have longer repeat schedules, which causes queue to run //shake// in case more ground motion amplitude data becomes available in the AQMS database. The repeat schedule is based on the event origin time, not the time at which queue was notified of the event. Note that queue will get signaled by an event several times after the event is detected by the RT system. These signals would result from both instances of ampgen (early and late), from TMTS, from the Duty Review Page when the event is accepted, from TMTS web interface for a manual moment tensor, and when the event is finalized in jiggle. Because of all these signals from AQMS, we have reduced the number of scheduled repeats in queue.conf. We have had complaints from USGS EHP when we generate ''too many'' versions of a ShakeMap, so now we try to limit the number. Manual signaling of queue may be needed when new ground motion amps get added to the database. === CISN Ad Hoc List === CISN maintains and ad hoc list of station parameters useful for some ShakeMap related programs. On the ShakeMap systems this file is **lib/grind/adhoc.lis**. On the post-processing systems, the file is /home/ncss/ncpp/ampexc/import/conf/adhoc.lis, where it is used by the [[postproc:amp_import|amp import]] system. The fields are Station code, Network code, SEED Channel and location codes, [[http://www.strongmotioncenter.org/NCESMD/reports/cosmos_format_1_20.pdf|COSMOS]] Table 6 Site code, latitude, longitude, elevation, and the SNCL's descriptive name. This file is built from smaller files maintained by UC Berkeley, Caltech, and CGS (California Geological Survey). The script **getalladhocs** (in the ShakeMap bin directory and in /home/ncss/ncpp/bin/) to query these three sources for their parts of the ad hoc list and then merges this into the adhoc.lis file //in the current directory//. The ShakeMap operator must run periodically to keep the adhoc.lis files up to date, say every 2 weeks. The UC Berkeley portion of the ad hoc list is maintained on rodgers.geo.berkeley.edu in /home/ncss/run/StationLists. In this directory, the script **genadhoc.pl** (no options needed) writes to stdout, which can be redirected to adhoc_NCSS.lis-new. Compare this with adhoc_NCSS.lis; if you are satisfied with the changes, move adhoc_NCSS.lis-new to adhoc_NCSS.lis. At the same time, you should use the script **genStaList.pl** to update //ncss_stalist//. This second list is used by some process at Caltech to generate a KML map of CISN stations. These two files are served to the web by www.ncedc.org. There are symbolic links to the files on rodgers in /data/dc6/ftp/pub/doc/CISN.info/. The Berkeley portion of the ad hoc list contains data for these networks: BG,BK,GM,GS,LB,NC,NN,NP,PG,TA,UO,US,UW,WR. Thus there is no separate list for Menlo Park stations. === Finite Fault Descriptions for ShakeMap === For earthquakes up to about magnitude 6, the fault source can be considered a point. For larger earthquakes, ShakeMap makes better maps when provided with a description of the fault rupture surface. This is done by creating a file named //ANYTHING_fault.txt// in the event input directory, where ''ANYTHING'' can be whatever you want. To remind you how to do this, in the ShakeMap config directory there is a file named **std_fault.txt** containing comments about this. Here's a sample file used for the South Napa M6.0 event of 2014/08/24: # Jack Boatwright's interpolation of Doug Dreger's finite fault # For SM 3.5 and later, each line must consist of lines containing # latitude, longitude, depth points to form one or more closed polygons # Disconnected polygons should be separated by lines containing only ">". 38.220 -122.313 2 38.220 -122.313 11 38.310 -122.333 11 38.310 -122.333 2 38.220 -122.313 2 **ShakeMap operators need to seek out the fault coordinates from seismologists for any events magnitude 6 and larger.** This should be done within a few hours of the earthquake, not later than 24 hours later! === Where does NCSS send ShakeMaps? === Whichever is the //production// ShakeMap system for NCSS sends ShakeMaps to the following locations. This is controlled by //transfer.conf// on the production ShakeMap system. * USGS PDL hubs using the [[network:pdl|ProductClient]] program. The maps become connect with their event pages at https://earthquakes.usgs.gov/earthquakes/. This USGS site does not provide a convenient catalog of ShakeMaps. * www.cisn.org using scp. These are actually two separate systems: www1.cisn.org at UC Berkeley; and www2.cisn.org at Caltech. DNS serves both of them as www.cisn.org. These web servers provide the old-style html ShakeMap pages that are produced by genex. This results in a complete catalog of ShakeMaps, e.g. for Northern California at http://www.cisn.org/shakemap/nc/shake/archive/. * Weather Central wxc.com using ftp. I don't know what happens to these maps. transfer.conf gives the last known contact there, probably ten years ago! === End of Year Changes === As mentioned above, ShakeMap (genex) generates html pages containing the fill catalog of ShakeMaps. The information about the catalog is maintained in the ShakeMap MySQL database. The html for the catalog is generated using a template file: **lib/genex/web/config/archivepages.xml**. This file needs to manually updated at the start of each year to close out the section for the previous year and start a new section for the new year. Scroll down to the line ''''. You should be able to figure out what to do! === MySQL Database === ShakeMap needs a MySQL database to store information about earthquakes it has processed and about the command flags used for each run of ShakeMap. This last feature allows you to manually run a ShakeMap program like //grind// with special options to adjust the map size and location. These options will be remembered in the database so that future runs of //shake// (manually or drive by //queue//) will use the same options for that event. There is a detailed discussion of this in the ShakeMap manual. For the NCSS, MySQL is started by system init scripts at boot time. The two databases are NOT replicated between each other. The MySQL database on rodgers.geo.berkeley.edu is backed up nightly using a cron job /home/ncss/mysql.backup/bin/dump.csh. The backup data (text files) are stored in /home/ncss/mysql.backup/data/ where they are backed up to tape by the BSL backup system.