====== DART Archiving Procedures ======

<code>
DART organization:
    DART directory structure:
	$DARTHOME/bin		(symlink to dir on backed up dir)
	$DARTHOME/config	(symlink to dir on backed up dir)
	$DARTHOME/data		
	$DARTHOME/logs
    $DARTHOME = /data/dart

    DART data directory structure:
	Net/Station.Net/Channel.Location.type/files 
    Filenames:
	Station.Net.Channel.Location.type.year.doy[.time]
    Files sort by data time order.
    Files are MiniSEED files.
    Files are currently daily files (with non-exact day boundaries).

    Automated DART archiving is performed by a separate process for each 
    network.

Policy-based archiving from DART:

    Archiving of data from DART is determined by policies.
    Policies apply at the N, NS, NSC, or NSCL level.
    For each data channel, the most specific policy specified is used for that channel.
    In general most policies are specified at the N or NS level.
    Each archive config file has a DEFAULT policy.

    Currently implemented policies:

    0.	abort:
	Do not archive this channel, and abort further archiving.
	Primarily used to catch misfiled data channels in the DART>

    1.	QC:
	Extract data in daily QC directories, and process with calqc.
	SKIP the data channel with automated DART archiving.

    2.	Channel:
	For each SNCL, archive data from that channel directory.
	Archive any day of data that meets any of the following criteria:
	    INCLUDE any day that meets one of the following conditions:
	    1.  Day is contained within an "eligible" file:
	 		$file_tdtm > ttime_last_archived
	    2.  Day is more recent than last_day_archived:
	 		$ydoy > $ latest_doy_archived
	 	
	    EXCLUDE any day that meets the following condition:
	    1.  ANY day within an "active" file.
	    Active file = last_file_in_sort_order && 
	 		  dtm_of_file > tnow() - $MAX_ARCHIVE_DELAY
	Used any specified options during channel archiving:
		repack		Repack data into full MiniSEED records.
		mssieve=N	Sieve (discard) and data segments < N seconds log.
		blksize=N	Aggregate data into MiniSEED records of specified size.
		msqual=R|D|Q	Specify the MiniSEED quality flag for the archived data.

Rule file:

Foreach network, create a file $DARTHOME/config/dart_archive.$net.rules with:

	##################################################
	# Rules for archiving data in DART
	# N.S.C.L	Rule	Options
	##################################################
	DEFAULT		abort
	NC		channel	repack:mssieve=60:blksize=4096:msqual=D
	##################################################
	NC.AFD		qc
	NC.BBGB		qc
	NC.GDXB		qc
	NC.KBO		qc
	NC.KCPB		qc
	NC.KCT		qc
	NC.KEB		qc
	NC.KHBB		qc
	NC.KHMB		qc
	NC.KMPB		qc
	NC.KMR		qc
	NC.KRMB		qc
	NC.KRP		qc
	NC.KSXB		qc
	NC.LDH		qc
	NC.LMC		qc
	NC.MCB		qc
	NC.MDH1		qc
	NC.MDPB		qc
	NC.MMLB		qc
	NC.PAGB		qc
	NC.PMPB		qc
	##################################################

   You are required to have a DEFAULT policy for each network.

Automated dart_archive procedure for each network:

Each day, an dart_archive process is run by cron to archive eligible data.
	$DARTHOME/bin/run_dart_archive
The process iterates over each network and runs
	$DARTHOME/bin/dart_archive -N $net

Dart_archive uses the $DARTHOME/config/dart_archive.$net.rules file to
determine how to archive the data:

    Channel-based archive:
	Traverse the network directory.
	For each channel {
		Find the most specific rule for the channel.
		If (rule==channel) {
			foreach channel in station directory {
				archive data for any elegible days from dart;
				update last_timear_chived(SNCL);
			}
		}
	}

    QC-based archive:
	Foreach network:
		Foreach QC-ed Station {
			Extract data from DART or alternate source
		}
		Analyst used calqc to determine missing data or data problems,
		backfill missing data, correct data problems and archive the 
		QC-ed network-day of data.

    Future logic for QC-based archive:
    (Currently skipped by dart_archive; QC archiving performed by $QCDIR/newday -N $net)
	Traverse the network directory.
	For each channel, look up rule.
		If (rule==QC) {
			push SN onto QC list;
			}
		}
	}
	Compare SN QC list to pre-defined SN QC list.
	Complain if the lists are not identical.
	if (QC list is not empty) {
		QC (net, SN_list) for the day;
	}
</code>