Introduction

IBM Content Manager OnDemand (IBM "CMOD") is a complex product that integrates the features of many different software components to build a fast, efficient, massively scalable Enterprise Report Management ("ERM") tool. By choosing to integrate with leading-edge tools like IBM DB2, Oracle, and Tivoli Storage Manager, IBM OnDemand benefits from the massive amount of development, features, and performance of these enterprise-class tools. However, this reliance on other tools means that finding the source of problems can be frustrating, as issues with individual components can produce strange errors in CMOD.

Components

These are the components of CMOD, and the functionality they provide.

Content Manager OnDemand

Content Manager OnDemand has a built-in 'System Log' that you can query for information. As almost every action is logged into the OnDemand System Log, it can be difficult to find what you're looking for. By choosing from the pop up list for the 'Severity' field, you can quickly find the most important error messages stored in the System Log.

Also, in the most recent versions of Content Manager OnDemand, almost all command-line utilities (arssockd, arsload, arsadmin) have 'tracing' features that can be turned on by specifying the -1 and -2 parameters.

 arssockd -I ARCHIVE -S -1 /tmp/arssockd.trace -2 ALL=15

... specifies maximal tracing for arssockd. It's highly recommended that you only run in this mode while specifically doing debugging, as enabling tracing in Content Manager OnDemand reduces performance of the server.

There are a lot of articles on the IBM CMOD support portal for enabling tracing: Enabling ODWEK Tracing Enabling tracing in Content Manager OnDemand server Formatting trace files on Windows Trace Facility for PDF indexer

If arssockd doesn't produce a trace file, on platforms that include the strace utility, prefixing your command with: strace -s 256 -f will produce diagnostically valuable output. Search the output for the word 'arslog' in order to find error messages that are unable to be written to the System Log or trace file.

 strace -s 256 -f arssockd -I ARCHIVE -S

... then search the output for the word 'arslog', and view the error messages that occur after it, as in the example below:

 [pid 900] execve("/opt/ibm/ondemand/V9.5/bin/arslog", ["/opt/ibm/ondemand/V9.5/bin/arslog", "ARCHIVE", "2017-11-27 11:38:06.247480", "0", "ARSMSG", "", "2", "216", "ARS0216E Unable to determine SMS/DMS tablespace type or filesystem is invalid >/arsdb/SMS<. The error number is 2", ""],

Operating Systems

One of the common issues that manifests in strange, odd, and puzzling ways is a lack of resources (CPU / Memory). On the UNIX-style platforms, this is controlled by the 'ulimit' command (AIX and Linux). Given that IBM CMOD is installed on servers, the ulimits should be set to 'unlimited' for all parameters. The only reason to establish limits on OnDemand's access to server resources is if it's installed on a shared system -- but this generally only happens in the smallest of Content Manager OnDemand installations.

You can view your current ulimits with:

   ulimit -a

AIX

Advanced Interactive eXecutive ("AIX") is IBM's unique flavor of the popular UNIX operating system. Common places to search for diagnostic information is the Error Report (errpt -a) or the standard logging directory, /var/log. Also, logging to the console is a popular option. If you have root access to the server, you can run:

   swcons `tty`

... which SWitches the CONSole to the currently session you're connected to. Pay special attention to the 'backticks', rather than quote characters that surround the "tty" command. The backticks tell AIX to run the command "tty", and place the output of the "tty" command into the command line as an argument to the swcons command.

After you've successfully switched the console to your local login session, try the command that is giving your problems again, and you may see an error message pop up on your display, which may provide some explanation as to what the problem is.

You can also get hardware and operating system level error messages from AIX's "Error Report" command, errpt.

   errpt -a

If you don't have strace on your AIX server, you can use truss instead.

   truss arssockd -I CMOD_DEV -S

Search the output in reverse order, looking for errors, permissions issues, or missing files.

Linux

Linux is a popular open source operating system. You can get system-level information or errors by running the 'dmesg' command -- but you'd be well-advised to use the 'more' or 'less' or 'pg' utilities to get a single page worth out output at a time, as the dmesg output can be thousands of lines long.

 dmesg |more

Also, as Linux takes several of its concepts for system administration from UNIX, you can also find logging information in the /var/log directory.

Database Engines: IBM DB2, Oracle, MSSQL

The database component of a Content Manager OnDemand installation provides for the storage of configuration data, as well as the storage and indexing of document-related metadata. OnDemand processes documents, and loads the index data for those documents into the database, so end users can query it later. The database is responsible for maintaining tables, building indexes (to make queries faster), and at a lower level, performing backups.

IBM DB2

IBM DB2 is provided by default as a 'limited use' license with Content Manager OnDemand at no cost, and as such, is the most popular database for IBM CMOD. DB2 has a concept called 'instances', which is like a complete installation of DB2 that is unique and managed at the operating system's user account level. If users Alice and Bob both have accounts on a single server, they can each have their own instance, and that instance is complete and wholly functional and independent of each other. To facilitate this user-level instance, DB2 creates a directory called 'sqllib' inside the user's home directory (in our example, Alice would have /home/alice/sqllib, and Bob would have /home/bob/sqllib). Inside the sqllib directory is the 'db2dump' subdirectory, which contains a file called 'db2diag.log'. It contains the majority of the diagnostic information for DB2.

Log in as the database instance owner, then run these commands:

   cd sqllib/db2dump  # Change into the directory where log files are kept.
   more db2diag.log   # View the contents of the DB2 Diagnostic Log a page at a time.

If you can't find the db2diag.log, or the data in it is old, your organizations Database Administrator may have moved the diagnostic data files. You can find the new path with the following command:

 $ db2 "get dbm cfg" |grep DIAGPATH
 Diagnostic data directory path               (DIAGPATH) = /db2/db2dump/
 Alternate diagnostic data directory path (ALT_DIAGPATH) =

Alternately, you can see the last 500 lines of the log with tail:

   tail -n 500 ~/sqllib/db2dump/db2diag.log  # Show the last 500 lines -- may be different on other Operating Systems!

Or of you want to monitor messages as they're written into the file by the database:

   tail -f ~/sqllib/db2dump/db2diag.log  # Prints new messages as they're added to the log file.

In order to exit from the tail command, press Control-C.

Real time performance monitoring

All modern versions of IBM DB2 include a tool called 'db2top' that is a performance monitoring tool just like Linux's 'top' and AIX's 'topas'. Use it to monitor what DB2 is doing in near-real-time. Here are two articles to get you started using it: Ember Crooks DB2 Basics IBM Developerworks

Tivoli Storage Manager

On all platforms, Tivoli Storage Manager ("TSM") is available to be utilized to store data for long-term archiving. In TSM, there is an internal 'activity log' that records all activity and errors. You can use the "query activity log" command with a variety of options to get information about any issues that TSM has experienced in the past.

 query activity log

... will provide the last several hours of messages.

 query activity log begintime=23:59

... will show all activity log messages starting at 11:59pm. The specified time my be formatted in 24-hour or "military" time.

 query activity log begindate=12/31/2014

... will show all activity beginning on a particular date.

There are many more options to the "query activity log" command, which can be viewed online with:

 help query activity

Additionally, if you want to view a 'console' of all messages in TSM as they happen, you can log in to the TSM Administrative Client in 'Console Mode':

 dsmadmc -consolemode

... and after entering a valid User ID and password, you will see all messages in TSM, as they happen. To exit from Console Mode, simply press Control-C.

Return Codes

As with DB2, a lot of the IBM CMOD error messages are 'passed through' from Tivoli Storage Manager.

 ARS0429E   TSM Error: ANS0266I (RC2302) The dsmEndTxn vote is ABORT, so check the reason field., Return Code=2302, Reason=41, File=arssmsms.cpp

In addition to getting the TSM activity log data, you can look up the return code in the TSM v7.1 API Return Codes page, to give you a better understanding of where the error might be.

Troubleshooting Content Manager OnDemand

Contents