* Connect as the oracle user on each machine (ucbrt, [ncdb1], ncdb2, mnlort1, mnlodb1).

* To switch to the private T1 line:

% cd $ORACLE_HOME/network/admin/
% \rm tnsnames.ora; ln -s tnsnames.ora.cisn tnsnames.ora

* To switch to the public internet:

% cd $ORACLE_HOME/network/admin/
% \rm tnsnames.ora; ln -s tnsnames.ora.public tnsnames.ora

* Connect as replication administrator: (mviewadmin on RT DB's - repadmin on DC DB's):

sqlplus repadmin@db_name

* Unschedule ONLY the broken job(s) by using the appropriate command(s):

EXEC DBMS_DEFER_SYS.UNSCHEDULE_PUSH ('publicdb.ncdb1.geo.berkeley.edu');
EXEC DBMS_DEFER_SYS.UNSCHEDULE_PUSH ('dcucb.ncdb2.geo.berkeley.edu');
EXEC DBMS_DEFER_SYS.UNSCHEDULE_PUSH ('dcmp2.mnlodb1.mp.cisn.org');

* Reschedule the broken job(s) by using the appropriate command(s):

BEGIN
DBMS_DEFER_SYS.SCHEDULE_PUSH (
destination => 'publicdb.ncdb1.geo.berkeley.edu',
interval => 'SYSDATE + 1/21600',
next_date => SYSDATE,
stop_on_error => FALSE,
delay_seconds => 0,
parallelism => 1);
END;
/

BEGIN
DBMS_DEFER_SYS.SCHEDULE_PUSH (
destination => 'dcucb.ncdb2.geo.berkeley.edu',
interval => 'SYSDATE + 1/21600',
next_date => SYSDATE,
stop_on_error => FALSE,
delay_seconds => 0,
parallelism => 1);
END;
/

BEGIN
DBMS_DEFER_SYS.SCHEDULE_PUSH (
destination => 'dcmp2.mnlodb1.mp.cisn.org',
interval => 'SYSDATE + 1/21600',
next_date => SYSDATE,
stop_on_error => FALSE,
delay_seconds => 0,
parallelism => 1);
END;
/

* Check that there are no more broken job(s):

SELECT dblink FROM defschedule WHERE job IN (SELECT job FROM dba_jobs WHERE broken='Y');

* Connect as replication administrator: (mviewadmin on RT DB's - repadmin on DC DB's):

sqlplus repadmin@db_name

* Find the affected object(s):

SELECT DISTINCT SCHEMANAME,PACKAGENAME,PROCNAME FROM defcall;

* Select error(s) information:

select DEFERRED_TRAN_ID, DESTINATION, ERROR_MSG from deferror;

* Take care of the problem.

* Re-execute the deferred transaction(s):

EXEC DBMS_DEFER_SYS.EXECUTE_ERROR ('deferred_tran_id','destination');

* Make sure that there are no more errors:

select DEFERRED_TRAN_ID, DESTINATION, ERROR_MSG from deferror;

* RTMP2 (mnlort1) or RTUCB2 (ucbrt) machine:

• No action is required.

* PUBLICDB (ncdb1) machine:

• Before taking down ncdb1, unschedule push jobs to PUBLICDB:

sqlplus repadmin@dcucb
EXEC DBMS_DEFER_SYS.UNSCHEDULE_PUSH ('publicdb.ncdb1.geo.berkeley.edu');

sqlplus repadmin@dcmp2
EXEC DBMS_DEFER_SYS.UNSCHEDULE_PUSH ('publicdb.ncdb1.geo.berkeley.edu');

• When ncdb1 is back up, reschedule push jobs to PUBLICDB:

sqlplus repadmin@dcucb
BEGIN
DBMS_DEFER_SYS.SCHEDULE_PUSH (
destination => 'publicdb.ncdb1.geo.berkeley.edu',
interval => 'SYSDATE + 1/21600',
next_date => SYSDATE,
stop_on_error => FALSE,
delay_seconds => 0,
parallelism => 1);
END;
/

sqlplus repadmin@dcmp2
BEGIN
DBMS_DEFER_SYS.SCHEDULE_PUSH (
destination => 'publicdb.ncdb1.geo.berkeley.edu',
interval => 'SYSDATE + 1/21600',
next_date => SYSDATE,
stop_on_error => FALSE,
delay_seconds => 0,
parallelism => 1);
END;
/

* DCUCB (ncdb2) machine:

• Before taking down ncdb2, unschedule push jobs to DCUCB:

sqlplus mviewadmin@rtucb2
EXEC DBMS_DEFER_SYS.UNSCHEDULE_PUSH ('dcucb.ncdb2.geo.berkeley.edu');

sqlplus repadmin@dcmp2
EXEC DBMS_DEFER_SYS.UNSCHEDULE_PUSH ('dcucb.ncdb2.geo.berkeley.edu');

• When ncdb2 is back up, reschedule push jobs to DCUCB:

sqlplus mviewadmin@rtucb2
BEGIN
DBMS_DEFER_SYS.SCHEDULE_PUSH (
destination => 'dcucb.ncdb2.geo.berkeley.edu',
interval => 'SYSDATE + 1/21600',
next_date => SYSDATE,
stop_on_error => FALSE,
delay_seconds => 0,
parallelism => 1);
END;
/

sqlplus repadmin@dcmp2
BEGIN
DBMS_DEFER_SYS.SCHEDULE_PUSH (
destination => 'dcucb.ncdb2.geo.berkeley.edu',
interval => 'SYSDATE + 1/21600',
next_date => SYSDATE,
stop_on_error => FALSE,
delay_seconds => 0,
parallelism => 1);
END;
/

* DCMP2 (mnlodb1) machine:

• Before taking down dcmp2, unschedule push jobs to MNLODB1:

sqlplus mviewadmin@rtmp2
EXEC DBMS_DEFER_SYS.UNSCHEDULE_PUSH ('dcmp2.mnlodb1.mp.cisn.org');

sqlplus repadmin@dcucb
EXEC DBMS_DEFER_SYS.UNSCHEDULE_PUSH ('dcmp2.mnlodb1.mp.cisn.org');

• When mnlodb1 is back up, reschedule push jobs to MNLODB1:

sqlplus mviewadmin@rtmp2
BEGIN
DBMS_DEFER_SYS.SCHEDULE_PUSH (
destination => 'dcmp2.mnlodb1.mp.cisn.org',
interval => 'SYSDATE + 1/21600',
next_date => SYSDATE,
stop_on_error => FALSE,
delay_seconds => 0,
parallelism => 1);
END;
/

sqlplus repadmin@dcucb
BEGIN
DBMS_DEFER_SYS.SCHEDULE_PUSH (
destination => 'dcmp2.mnlodb1.mp.cisn.org',
interval => 'SYSDATE + 1/21600',
next_date => SYSDATE,
stop_on_error => FALSE,
delay_seconds => 0,
parallelism => 1);
END;
/