CMS - CISN Message System

CMS is the message system used by many parts of AQMS. It implements the publish-subscribe paradigm: A publisher issues messages with a subject but has no knowledge about who, if anyone, is listening. Zero or more subscribers tell CMS that they are interested in a subject, but have no knowledge about who publishes the messages.

This page makes no effort to describe how to write CMS client code. The best way to learn CMS coding is to read existing code.

CMS uses code written by Caltech along with their contractor ISTI. This replaces an earlier message system based on Telarian Smart Sockets, a high-cost commercial publish-subscribe message system. CMS is based on CORBA. The CMS server programs are written in java; the client libraries are C++ code currently maintained in the CISN svn repository maintained at Caltech.

The CMS server consists of two java programs: QWServer.jar and ISTINotifServer.jar. The general documentation for these programs is here. Note that these two programs are used in other parts of AQMS: they form part of the PDL hubs and they are used in the CISNDiplay servers:

• Caltech: quakewatch1.cisn.org and quakewatch2.cisn.org
• Menlo Park: quakewatch4.cisn.org
• Berkeley: quakewatch3.cisn.org

All CMS messages are passed in XML format. Various facilities handle the conversion into and out of XML; this includes the C++ client libraries for AQMS.

For the NCSS AQMS, there are CMS servers on each of the RT systems ucbrt.geo.berkeley.edu and mnlort1.wr.usgs.gov; and on each of the post-processing systems ucbpp.geo.berkeley.edu and mnlodb1.wr.usgs.gov. Each of the servers on the post-processing systems is configured to pull messages from both of the RT system servers. Messages do not flow back from post-processing CMS servers to the RT systems.

CMS servers are located in ~ncss/run/CMS. In this directory you will find:

• jar files
• scripts for starting and stopping the servers. Of these, runAll is the one of primary interest.
• copies of the Linux init script cms.init and for RHEL7, the systemd cms.service file.
• directories:
  • conf: configuration files QWServerConfig.xml and NotifConfig. Each entry has a comment; see the http://www.isti2.com/QWIDS/current_dist/QWServer/doc/QWServer.html|manual]] for more detail.
  • fileOutData, outputterdata: not currently used
  • log: log files
  • pssdata: temporary files used bu ISTINotifServer
  • storage: daily files containing each of the messages received by the server. These can be cryptic but reading them is useful to find where messages are getting dropped.
  • MainMessageService.loc: This is a CORBA IOR file used to establish CMS connections. The only thing that users can do with this file is read it and have it decoded. This can be useful when trying to find configuration errors.

All CMS client programs (publishers and subscribers) write messages in queue files. These files are normally kept in /home/ncss/run/qfiles/. For each client, there usually will be one queue file for outgoing messages: its name will end in SEND.q. And there will be another queue file for incoming messages: its name will end in RECV.q. The remaining parts of each queue file name include the Project and Program names, read from various configuration files.

The AQMS client programs each must have a CMS client configuration file named in their own config files. This file is normally /home/ncss/run/params/cms.cfg. Here's the one currently found on ucbrt:

Project            ncss_rtucb
Host               ucbrt.ucb.cisn.org:40003
CorbaVersion       1.2
MaxQFileSize       200
MaxConnRetry       100
QueueFileHomeDir   /home/ncss/run/qfiles
SubscriberHost     ucbrt.ucb.cisn.org
SubscriberPort     36688
SleepBetweenRetries 20

The parameters here are:

• Project: effectively a namespace to group the various message subjects.
• Host: the CMS server host and port number to which clients publish messages. The port number should be the same as feederPortNumber in the QWFeederMod section of QWServerConfig.xml.
• CorbaVersion: identifies the CORBA version being implemented in the server.
• MaxQFileSize: specifies the maximum number of messages that can be kept in a queue file. Supposedly the client library will delete a queue file if it ever reaches this limit.
• MaxConnRetry: number of times a program should attempt to (re)establish connection to the server. Usually a client will exit with an error if this limit is reached.
• SleepBetweenRetries: Number of seconds for a client to sleep before attempting to reconnect to the server.
• QueueFileHomeDir: full path to the queue file directory.
• SubscriberHost: hostname of the server for subscribing client
• SubscriberPort: port number for server subscriptions. This should be the same as outputPortNumber in the QWOutputterMod section of QWServerConfig.xml.

Each CMS client will have a few parameters in its own configuration file that are related to CMS. There will always be a CMSConfig parameter, to specify the cms configuration file. There will also be one or more parameters to specify CMS subjects under which to publish messages or to which to subscribe. The names and uses of these subject parameters are specific to each application.

Most CMS clients mention some special subjects, usually in common.cfg that is listed as an Include file in a client configuration. These subjects are SystemSubject, StatusSubject, DebugSubject, InfoSubject and ErrorSubject. As far as I can tell these are remnants of very old code and are not actually used by AQMS. The code probably still requires that they be present in configuration files.

Normally CMS servers are started up and shut down by operating system scripts: init script on older Linux and Solaris systems; systemd file on RHEL7 and similar systems. These startup scripts should be arranged so that CMS starts before AQMS programs are started.

If there are problems that might be in CMS, look in the CMS server logs, especially QWServerConsole.txt. Occasionally java memory problems are reported there. These may be cleared up by stopping and restarting the CMS servers using the runAll script:

 runAll stop
 runAll start

Because the clients will reconnect themselves, this method is less intrusive than rebooting the entire system.

If messages don't seem to be getting from publisher to subscriber the way you think they should, look in the queue files and the CMS storage directory. The queue files have a small amount of non-readable ASCII text, but each message can easily be seen with a text viewer. A CMS publisher first writes its outgoing message in its SEND queue file. If the client is connected to the CMS server, the message will be sent to the server where it can be found in the current day's storage file and also in the current QWMessages_*log file. Subscribing clients will write the message into their RECV queue file: this is done by the CMS library before the message is passed to the main part of the client code (through its registered massage handler function.)