User Tools

Site Tools


operations:db_ncedc:bg_import

DART BG Network Import

Geysers proxy waveservers:

  • aqms-nc-pp.wr.usgs.gov:9325
  • aqms-nc-rt.wr.usgs.gov:9325

Programs to download data from BG waveserver:

  • cron_start_bg_get:
Cron job runs once per minute and checks to see that run_bg_get is running.  If not, starts run_bg_get.
  • run_bg_get:
Shell script that sets up logging and runs bg_get. Job to download BG data in hourly chunks.
  • bg_get:
Shell script that runs bg_dl to perform hourly BG channel files from Geysers onsite Winston Waveserver.
Imposes 7 hour delay on downloading data to allow Nanometrics out-of-order data telemetry to create as
complete a copy of the data as possible in the Winston Waveserver at the Geysers.
	set HOMEDIR = /dart/work
	set BINDIR = $HOMEDIR/bin
	set DLTMPDIR = $HOMEDIR/dltmp
	set STATEFILE = $BINDIR/bg.state
Script runs continuously, sleeping if needed to wait for 7 hour delay to download next hour of data. 
Runs in DLTMPDIR.
Save state info (last hour completed) in STATEFILE.
  • bg_dl:
Perl script that uses wvc and proxy waveserver to download hourly BG channel files from Geysers onsite
Winston Waveserver.   Works in current directory or optional "-w workdir" directory.
Outputs hourly channel files in directory
	/data/work/dldata

Programs used to create DART channel-day files:

  • run_bg_merge_day:
Shell script run from cron periodically (4 times per day) to setup logfile and run bg_merge_day.
Will run bg_merge_day for each days of any hourly files in directory
	/dart/work/dldata
  • bg_merge_day:
Process all new files for a channel-day at once rather than one file at a time. Perl script that
merges the data for a SNCL day into a daily DART file.
Input from current directory:
	/dart/work/dldata
	DART channel directory (if it exists)
Output to directory:
	/data/dart/bdstage (intermediate results)
	DART channel directory
Moves processed hourly files to:
	/dart/work/done

Programs used to backfill missing data:

Look for any hourly files that are empty in the “done” directory. Try to download that hour of data again. Experience has shown that we either get all of the data available for the hour, or no data (which creates an empty file).

  • run_bg_backfill:
Perl script run by cron periodically find empty hourly files in
	/data/work/done
and run bg_backfill on that hourly file.
  • bg_backfill:
Run bg_dl to attempt to download again the channel-hour of data.

Cleanup of BG work space:

  • bg_cleanup:
Shell script run periodically by cron to delete files older than 10 days in the "done" directory
	/dart/work/done

BG channel list Generation:

  • run_gen_bg_sncl: (run as cron job)
Generates list of BG channels (file sncl.list). Changes are emailed. If changes are present,
bg_get process needs to be killed.
operations/db_ncedc/bg_import.txt · Last modified: 2024/05/29 22:13 by stephane