NCSS Wiki

After the AQMS RT system has detected and located an earthquake, we need to collect waveforms for the relevant SNCLs to make them available for human event review. As explained in Real-Time Processing Systems in the NCSS, request cards are generated on the active RT system. The request card generators are the programs that decide which SNCLs should be collected and for what time periods. The requests are written to the request_card table on local RT database. From there the request_card table is replicated to each of the “data center” or “archive” databases.

The basic information in a request card is: event ID, station, network, channel, location (SNCL), start and end time, request ID, retry count, date of last retry.

On whichever is the mast post-processing system, waveform archiver program called “wanc” processes these requests. There are several instances of wanc running at once, one for each of groups of networks of seismic stations. Breaking the work load up by networks provides an easy way to have several archivers working in parallel while ensuring they do not interfere with each other.

It is important that each instance of wanc is configured to operate on a unique list of networks. Otherwise the waveform tables are likely to become inconsistent.

The waveform archivers must be run under the dcmgr user so that they can create files and directories in the NCEDC archive.

Waveform archivers run continuously, doing the following actions in a loop:

• Query the database for stale request cards, ones that are older than a configurable age. Delete any requests that are found to be stale.

• Query the database for up to 128 request cards that are “due” for processing. This is a complex query that prioritizes the newest request cards first, and then tries older request cards at intervals that increase with age, up to about daily for older requests.

• If no eligible request cards are found, sleep for a minute and try again.

• Query the AssocWFRC and waveform tables for information about requests that have been partially satisfied but could be improved.

• Make requests to the configured list of wave servers for each SNCL and time interval.

• Evaluate the returned waveforms to see how completely is fills the requested time interval.

• If the new waveform is more complete than the previously saved waveform (if any), then prepare to save the new data to file. Otherwise discard the newly retrieved waveform.

• Determine the directories in which the waveforms are to be saved. The base directory is /data/ncedc/events/active, which currently is a symbolic link to /data/dc22/ncedc/events/active. Within that directory, there is a directory for each event of the form “NC.<EVID>”. Query the database subdir table for directory names. If there is not one for the event, add one to the subdir table and create the new directory on the file system.

• Write each waveform into a file in the appropriate directory. The file name is “STA.NET.CHAN.LOC.D.ms”.

• For waveforms that are replacing previously archived but incomplete, delete their Waveform, Filename, AssocWaE and AssocWfRC rows.

• Write information to database tables: Waveform, Filename, Subdir, AssocWaE about the newly saved waveforms. If any of these waveforms are incomplete, also write AssocWfRC rows.

• For requests that are completely satisfied, delete their request cards and AssocWfRC rows, if any.

• For requests that are not completely satisfied, update their retry count and date of last retry.

Note that wanc works on lists of request cards, not on lists of events.

wanc.cfg has the following arguments:

• Logfile path_with_prefix The path with a logfile prefix where daily logs will be stored.

• LoggingLevel n Set the level of logging the program will do. 0 = off, higher numbers are more verbosity.

• ProgramName Wave_Archiver Does not appear to be used, but still required.

• RequestType Triggered Sets the type of request card to be processed: Triggered or Continuous. Currently only Triggered is supported.

• Auth authority The 2 character code of the authority running wanc (e.g., NC etc).

• Subsource source_id Specify the Source ID for wanc to insert into the database tables.

• MaxValidTime age in days Request cards older than this are deleted. Age is based on the time when the request card was first written. The idea is that waveforms will reside in most wave servers for a limited time; after that there is no point in asking for expired waveforms. It would make more sense if the request card were based on the event origin time: if you write request cards for a very old event, wanc will keep requesting waveforms when there is no chance of them being in most wave servers. Oh, well…

• WaveServer host:port Specify the host name and port number for a AQMS-style wave server. Use as many of these commands as you want, one host:port per line.

• PollInterval time in seconds How long to sleep after a query for request cards returns nothing; i.e. when wanc is idle. If the query does return request cards, they will be processed and the query repeated immediately.

• Order Oldest Specify the order to process request cards for a given priority and retry count. Choices are Oldest or Newest.

• ServerConnectTimeout time in seconds Specify how long to wait for a wave server to accept a network connection.

• MaxRequests n Specify how many request cards to process in one batch.

• GapThreshold time in sample intervals Specify how large the interval between samples can be before a gap is declared. This is one factor that makes wanc special compared to the standard AQMS waveform archiver: The NCEDC makes separate entries in the Waveform database table for each segment of waveforms separated by a gap. Other AQMS databases make a single entry for one SNCL per event, regardless of gaps within the waveform.

• WaveRoot wave-directory Specify the root directory for storing event waveforms. The actual directory for an event will the the directory active/NC.<EVID> appended to this root.

• NetList net,net… Provide a comma-separated list of one or more network codes on which this instance of wanc should work. It is very important that a given network code appear in only one wanc.cfg. Otherwise competing wancs will make a mess! If this option is not given in the configuration file, then wanc will work on all request cards.

• DBService xxx Specify the name of the database being used.

• DBUser xxx Specify the name of the database user account.

• DBPasswd xxx Specify the password for the database user account specified by DBUser.

• DBConnectionRetryInterval n Specify the interval (in arbitrary units) for attempts to reconnect to the database. While this command is required, it has no effect on the behavior of wanc; more vapor-ware.

• DBMaxConnectionRetries n Specify the number of times to attempt to reconnect to the database. While this command is required, it has no effect on the behavior of wanc.

Sample configuration file:

Logfile		../logs/wa.log
LoggingLevel	2
ProgramName	Wave_Archiver


#
# Application configuration parameters
#
RequestType			Triggered
Auth				NC
Subsource			RTTEST
MaxValidTime			7
WaveServer			transfrm:6500
PollInterval			60
Order				Oldest
ServerConnectTimeout		10
MaxRequests			128
GapThresh			0.1
WaveRoot			/data/ncedc/events
# optional list of one or more network codes to handle; default is to handle
# all network codes. Separate multiple nets by commas, no spaces.
NetList				BK,BP
DBService			service
DBUser				user
DBPasswd			passwd
DBConnectionRetryInterval	5
DBMaxConnectionRetries		5

The waveforms collected for an event may not always be what jiggle users want to see. For example, the event parameters (origin time, location, magnitude) available in the database when the request card generator first ran may be incorrect.

To address such problems, the NCSS developed some tools to effectively remove the existing waveform collection and replace it with a (hopefully) better collection. This starts with the Catalog maintenance web page. This lists all the earthquakes in the database for the last seven days, including events with their selectflag set to “0”, i.e. “deleted” from the catalog.

The user selects an event of interest. The event page shows the current status of event information as well as the status of collected waveforms and pending request cards. If the event information is sufficient (must have a valid location and magnitude) and the event is not too old (max is 12 days), the user is offered a “Repopulate Waveforms” button.

Pressing the “Repopulate Waveforms” button causes the event to be inserted into the PCS table in the TPP-TPP-REPOP state. From there the repopwfcont script takes action. When wanc sees the new request cards that are generated by repopwfcont, the requests are treated just like any other requests.

All the waveform archiving processes must be run as user dcmgr so that they have permission to write files on the NCEDC waveform filesystems.

Each instance of wanc has its own run script, e.g. run_wanc_NC. These scripts take one argument: either start or stop. There are also scripts start_all_wancs and stp_all_wancs. All these scripts are in the /home/dcmgr/run/bin directory on the post-processing systems.

wanc will not stop immediately. If it is processing a batch of request cards, it will continue until that batch is finished. Sometimes when wanc is idle, it will fail to stop. It's OK to give it a kill -9.

The wancs are NOT started automatically during the boot process. The consequences of having two sets of wancs running (on each of the post processing systems) are so bad that we only start the wanc processes by hand.

See post-proc active-standby switch for details on switching the running set of wancs between post-processing systems.

Provided that the wancs are configured correctly, they generally work well. When people complain about missing waveforms for one or more events, it usually is a problem with wave servers. Is the problem waveform available on the wave server(s) that wanc is configured to use for that network? Is that wave server configured correctly to serve that SNCL?