====== CMS - CISN Message System ======

CMS is the message system used by many parts of AQMS. It implements the
**publish-subscribe** paradigm: A publisher issues messages with a subject but
has no knowledge about who, if anyone, is listening. Zero or more subscribers
tell CMS that they are interested in a subject, but have no knowledge about
who publishes the messages.

This page makes no effort to describe how to write CMS client code. The best
way to learn CMS coding is to read existing code.

CMS uses code written by Caltech along with their contractor
[[http://www.isti.com|ISTI]]. This replaces an earlier message system based on
Telarian Smart Sockets, a high-cost commercial publish-subscribe message
system. CMS is based on
[[https://en.wikipedia.org/wiki/Common_Object_Request_Broker_Architecture|CORBA]].
The CMS server programs are written in java; the client libraries are C++ code
currently maintained in the CISN svn repository maintained at Caltech.

The CMS server consists of two java programs: QWServer.jar and
ISTINotifServer.jar. The general documentation for these programs is
[[http://www.isti2.com/QWIDS/current_dist/QWServer/doc/QWServer.html|here]].
Note that these two programs are used in other parts of AQMS: they form part
of the [[rtem:pdl|PDL hubs]] and they are used in the
[[http://www.cisn.org/software/cisndisplay.html|CISNDiplay]] servers:
  * Caltech: quakewatch1.cisn.org and quakewatch2.cisn.org
  * Menlo Park: quakewatch4.cisn.org
  * Berkeley: quakewatch3.cisn.org

All CMS messages are passed in XML format. Various facilities handle the
conversion into and out of XML; this includes the C++ client libraries for
AQMS. 

For the NCSS AQMS, there are CMS servers on each of the RT systems
ucbrt.geo.berkeley.edu and mnlort1.wr.usgs.gov; and on each
of the post-processing systems ucbpp.geo.berkeley.edu and mnlodb1.wr.usgs.gov. Each of the servers on the post-processing systems is
configured to pull messages from both of the RT system servers. Messages **do
not** flow back from post-processing CMS servers to the RT systems.

=== Directories and Files ===

CMS servers are located in **~ncss/run/CMS**. In this directory you will find:
  * jar files
  * scripts for starting and stopping the servers. Of these, **runAll** is the one of primary interest.
  * copies of the Linux init script **cms.init** and for RHEL7, the systemd **cms.service** file.
  * directories:
    * conf:  configuration files QWServerConfig.xml and NotifConfig. Each entry has a comment; see the http://www.isti2.com/QWIDS/current_dist/QWServer/doc/QWServer.html|manual]] for more detail.
    * fileOutData, outputterdata: not currently used
    * log: log files
    * pssdata: temporary files used bu ISTINotifServer
    * storage: daily files containing each of the messages received by the server. These can be cryptic but reading them is useful to find where messages are getting dropped.
    * MainMessageService.loc: This is a CORBA [[https://en.wikipedia.org/wiki/Interoperable_Object_Reference|IOR]] file used to establish CMS connections. The only thing that users can do with this file is read it and have it [[http://www2.parc.com/istl/projects/ILU/parseIOR/|decoded]]. This can be useful when trying to find configuration errors.

All CMS client programs (publishers and subscribers) write messages in queue
files. These files are normally kept in **/home/ncss/run/qfiles/**. For each
client, there usually will be one queue file for outgoing messages: its name
will end in SEND.q. And there will be another queue file for incoming
messages: its name will end in RECV.q. The remaining parts of each queue file
name include the Project and Program names, read from various configuration
files.

The AQMS client programs each must have a CMS client configuration file named
in their own config files. This file is normally
**/home/ncss/run/params/cms.cfg**. Here's the one currently found on ucbrt:

<code>
Project            ncss_rtucb
Host               ucbrt.ucb.cisn.org:40003
CorbaVersion       1.2
MaxQFileSize       200
MaxConnRetry       100
QueueFileHomeDir   /home/ncss/run/qfiles
SubscriberHost     ucbrt.ucb.cisn.org
SubscriberPort     36688
SleepBetweenRetries 20
</code>

The parameters here are:
  * Project: effectively a namespace to group the various message subjects.
  * Host: the CMS server host and port number to which clients publish messages. The port number should be the same as **feederPortNumber** in the QWFeederMod section of QWServerConfig.xml.
  * CorbaVersion: identifies the CORBA version being implemented in the server.
  * MaxQFileSize: specifies the maximum number of messages that can be kept in a queue file. Supposedly the client library will delete a queue file if it ever reaches this limit.
  * MaxConnRetry: number of times a program should attempt to (re)establish connection to the server. Usually a client will exit with an error if this limit is reached.
  * SleepBetweenRetries: Number of seconds for a client to sleep before attempting to reconnect to the server.
  * QueueFileHomeDir: full path to the queue file directory.
  * SubscriberHost: hostname of the server for subscribing client
  * SubscriberPort: port number for server subscriptions. This should be the same as **outputPortNumber** in the QWOutputterMod section of QWServerConfig.xml.

Each CMS client will have a few parameters in its own configuration file that
are related to CMS. There will always be a **CMSConfig** parameter, to specify
the cms configuration file. There will also be one or more parameters to
specify CMS subjects under which to publish messages or to which to
subscribe. The names and uses of these **subject** parameters are specific to
each application.

Most CMS clients mention some special subjects, usually in **common.cfg** that
is listed as an **Include** file in a client configuration. These subjects are
SystemSubject, StatusSubject, DebugSubject, InfoSubject and ErrorSubject. As
far as I can tell these are remnants of very old code and are not actually
used by AQMS. The code probably still requires that they be present in
configuration files.


=== Maintenance ===

Normally CMS servers are started up and shut down by operating system scripts:
init script on older Linux and Solaris systems; systemd file on RHEL7 and
similar systems. These startup scripts should be arranged so that CMS starts
before AQMS programs are started.

If there are problems that might be in CMS, look in the CMS server logs,
especially QWServerConsole.txt. Occasionally java memory problems are reported
there. These may be cleared up by stopping and restarting the CMS servers
using the runAll script:
   runAll stop
   runAll start
Because the clients will reconnect themselves, this method is less intrusive
than rebooting the entire system.

If messages don't seem to be getting from publisher to subscriber the way you
think they should, look in the queue files and the CMS storage directory. The
queue files have a small amount of non-readable ASCII text, but each message
can easily be seen with a text viewer. A CMS publisher first writes its
outgoing message in its SEND queue file. If the client is connected to the CMS
server, the message will be sent to the server where it can be found in the
current day's storage file and also in the current QWMessages_*log
file. Subscribing clients will write the message into their RECV queue file:
this is done by the CMS library before the message is passed to the main part
of the client code (through its registered massage handler function.)