After the AQMS RT system has detected and located an earthquake, we need to collect waveforms for the relevant SNCLs to make them available for human event review. As explained in Real-Time Processing Systems in the NCSS, request cards are generated on the active RT system. The request card generators are the programs that decide which SNCLs should be collected and for what time periods. The requests are written to the request_card table on local RT database. From there the request_card table is replicated to each of the “data center” or “archive” databases.
The basic information in a request card is: event ID, station, network, channel, location (SNCL), start and end time, request ID, retry count, date of last retry.
On whichever is the mast post-processing system, waveform archiver program called “wanc” processes these requests. There are several instances of wanc running at once, one for each of groups of networks of seismic stations. Breaking the work load up by networks provides an easy way to have several archivers working in parallel while ensuring they do not interfere with each other.
It is important that each instance of wanc is configured to operate on a unique list of networks. Otherwise the waveform tables are likely to become inconsistent.
The waveform archivers must be run under the dcmgr user so that they can create files and directories in the NCEDC archive.
Waveform archivers run continuously, doing the following actions in a loop:
Note that wanc works on lists of request cards, not on lists of events.
wanc.cfg has the following arguments:
path_with_prefix The path with a logfile prefix where daily logs will be stored.n Set the level of logging the program will do. 0 = off, higher numbers are more verbosity. Wave_Archiver Does not appear to be used, but still required.Triggered Sets the type of request card to be processed: Triggered or Continuous. Currently only Triggered is supported.authority The 2 character code of the authority running wanc (e.g., NC etc). source_id Specify the Source ID for wanc to insert into the database tables. age in days Request cards older than this are deleted. Age is based on the time when the request card was first written. The idea is that waveforms will reside in most wave servers for a limited time; after that there is no point in asking for expired waveforms. It would make more sense if the request card were based on the event origin time: if you write request cards for a very old event, wanc will keep requesting waveforms when there is no chance of them being in most wave servers. Oh, well…host:port Specify the host name and port number for a AQMS-style wave server. Use as many of these commands as you want, one host:port per line.time in seconds How long to sleep after a query for request cards returns nothing; i.e. when wanc is idle. If the query does return request cards, they will be processed and the query repeated immediately.Oldest Specify the order to process request cards for a given priority and retry count. Choices are Oldest or Newest.time in seconds Specify how long to wait for a wave server to accept a network connection.n Specify how many request cards to process in one batch.time in sample intervals Specify how large the interval between samples can be before a gap is declared. This is one factor that makes wanc special compared to the standard AQMS waveform archiver: The NCEDC makes separate entries in the Waveform database table for each segment of waveforms separated by a gap. Other AQMS databases make a single entry for one SNCL per event, regardless of gaps within the waveform.wave-directory Specify the root directory for storing event waveforms. The actual directory for an event will the the directory active/NC.<EVID> appended to this root.net,net… Provide a comma-separated list of one or more network codes on which this instance of wanc should work. It is very important that a given network code appear in only one wanc.cfg. Otherwise competing wancs will make a mess! If this option is not given in the configuration file, then wanc will work on all request cards.xxx Specify the name of the database being used.xxx Specify the name of the database user account.xxx Specify the password for the database user account specified by DBUser.n Specify the interval (in arbitrary units) for attempts to reconnect to the database. While this command is required, it has no effect on the behavior of wanc; more vapor-ware.n Specify the number of times to attempt to reconnect to the database. While this command is required, it has no effect on the behavior of wanc.Sample configuration file:
Logfile ../logs/wa.log LoggingLevel 2 ProgramName Wave_Archiver # # Application configuration parameters # RequestType Triggered Auth NC Subsource RTTEST MaxValidTime 7 WaveServer transfrm:6500 PollInterval 60 Order Oldest ServerConnectTimeout 10 MaxRequests 128 GapThresh 0.1 WaveRoot /data/ncedc/events # optional list of one or more network codes to handle; default is to handle # all network codes. Separate multiple nets by commas, no spaces. NetList BK,BP DBService service DBUser user DBPasswd passwd DBConnectionRetryInterval 5 DBMaxConnectionRetries 5
The waveforms collected for an event may not always be what jiggle users want to see. For example, the event parameters (origin time, location, magnitude) available in the database when the request card generator first ran may be incorrect.
To address such problems, the NCSS developed some tools to effectively remove the existing waveform collection and replace it with a (hopefully) better collection. This starts with the Catalog maintenance web page. This lists all the earthquakes in the database for the last seven days, including events with their selectflag set to “0”, i.e. “deleted” from the catalog.
The user selects an event of interest. The event page shows the current status of event information as well as the status of collected waveforms and pending request cards. If the event information is sufficient (must have a valid location and magnitude) and the event is not too old (max is 12 days), the user is offered a “Repopulate Waveforms” button.
Pressing the “Repopulate Waveforms” button causes the event to be inserted into the PCS table in the TPP-TPP-REPOP state. From there the repopwfcont script takes action. When wanc sees the new request cards that are generated by repopwfcont, the requests are treated just like any other requests.
All the waveform archiving processes must be run as user dcmgr so that they have permission to write files on the NCEDC waveform filesystems.
Each instance of wanc has its own run script, e.g. run_wanc_NC. These scripts take one argument: either start or stop. There are also scripts start_all_wancs and stp_all_wancs. All these scripts are in the /home/dcmgr/run/bin directory on the post-processing systems.
wanc will not stop immediately. If it is processing a batch of request cards, it will continue until that batch is finished. Sometimes when wanc is idle, it will fail to stop. It's OK to give it a kill -9.
The wancs are NOT started automatically during the boot process. The consequences of having two sets of wancs running (on each of the post processing systems) are so bad that we only start the wanc processes by hand.
See post-proc active-standby switch for details on switching the running set of wancs between post-processing systems.
Provided that the wancs are configured correctly, they generally work well. When people complain about missing waveforms for one or more events, it usually is a problem with wave servers. Is the problem waveform available on the wave server(s) that wanc is configured to use for that network? Is that wave server configured correctly to serve that SNCL?