I've mentioned before that I like having the ITNM control scripts on every server in an distributed ITNM solution. I like to be able to start with "itnm_status" on any server.
At present, the ITNM installer doesn't put those scripts on a server where the only thing installed is OMNIbus stuff.
Here's a quick way to move what you need:
On a server with the ITNM core processes, do the following:
$ cd $NCHOME
$ tar cf ITNM_control_scripts.tar precision/bin/*_control* precision/bin/itnm_st* precision/bin/tip_server.sh
Now copy the tar file to the target server and untar it relative to the $NCHOME directory. Add $NCHOME/precision/bin
to your PATH and you should be able to run the "itnm_status" command.
If the script complains with a message like this:
nco_pad RUNNING PID=12345 nco_pa
But /opt/netcool/etc/itnm,cfg says it is installed elsewhere
The problem is probably a missing $NCHOME/etc/itnm.cfg file.
For a quick fix on an OMNIbus-only host do this:
$ echo "nco=$(hostname)" > $NCHOME/etc/itnm.cfg
That should stop the complaints.
The itnm_status command is certainly handy - So handy, in fact, that even on distributed ITNM installs I will put it on systems that have only OMNIbus components (i.e. only nco_pad running, no ncp_ctrl and no TIP)
I much prefer just typing "itnm_status" and seeing what's running vs "nco_pa_status" and having to deal with authentication.
So imagine my annoyance when I found that it wasn't displaying the status of the syslog probe in my latest ITNM deployment. (OK actually, it was the "syslogd" probe - and syslog vs. syslogd is worth another post someday.) At any rate, I traced the misbehavior down to this code in nco_control.sh:
for my_pid in $seen; do
#ticket 248834 APAR - IV01170 Display only ITNM related process
if [ $my_pid = "nco_objserv" -o $my_pid = "nco_g_objserv_uni" -o $my_pid = "nco_p_tivoli_eif" -o $my_pid = "nco_p_mttrapd" ]; then
I have no idea why someone thought it was a bad idea to show the status of processes deemed "unrelated" to ITNM. Seems like a dubious proposition in the first place, as it's highly likely that if a process is managed by the same PA daemon as the ObjectServer, the SNMP trap probe and so forth, it's probably "related". What's really not debatable is that a syslog probe would certainly be lbe "related". Here's how I "fixed" it for the time being:
for my_pid in $seen; do
#ticket 248834 APAR - IV01170 Display only ITNM related process
if [ $my_pid = "nco_objserv" -o $my_pid = "nco_g_objserv_uni" -o $my_pid = "nco_p_tivoli_eif" -o $my_pid = "nco_p_mttrapd" \
-o $my_pid = "nco_p_syslog" -o $my_pid = "nco_p_syslogd" \
This begs the question about other possibly "related" processes (e.g. nco_g_objserv_bi" in a fail-over situation)
I've mentioned in a number of blog entries that I consider non-root installation and operation of ITNM to be "Best Practice". One area where the "Out of the box" ITNM product doesn't support this are the system startup scripts. These are the scripts thar reside in /etc/init.d (the path varies slightly by *NIX variant). Of course it you are running the ITNM installer as a non-root user, the installer won't be able to put scripts in the protected system directory, but it will still create versions in $NCHOME/precision/custom/control/init.d. The three scripts it creates are:
- ncp - The startup script for all of the "core" precision processes
- nco - The startup script for OMNIbus processes
- tip - The startup script for the Tivoli Integrated Portal
N.B. These scripts are built as part of the process of building and customizing the various ITNM control scripts (e.g itnm_start, itnm_status, and so forth.) There is a master script, $NCHOME/precision/install/scripts/create_all_control.sh that does this - it's easy to figure out the chain of subordinate scripts from there.
The problem with the stock startup scripts is that they provide no mechanism for starting their process as something other than root. I have added that feature to the scripts along with a few other improvements.
You can get them: here
. (UPDATE 1/23/13 - Fixed export of NC_RULES_HOME in the "nco" script.)
These scripts can be run both by the root user (either at startup or later) or by the user reserved to be the owner of the ITNM processes (I use "netcool" unless my customer provisioned a different name.)
EDIT: One more thing to note - these startup scripts do require the ITNM control scripts to be present (which is not the default if the server only has OMNIbus stuff). Here's how to fix that
EDIT: And another thing to remember - Change the modes to r_xr_x_r_x after copying them to the appropriate startup directory, and don't forget to make the appropriate symlinks.
For example, installing the nco script on Linux would be done like:
# cp nco /etc/init.d
# chmod 755 /etc/init.d/nco
# /sbin/chkconfig --add nco
Other operating systems have variations of the above.
Modified by RogerPowell
What a head-scratcher! I'd been working on refining the discovery of a Customer's network, when one day all ITNM Network Views started appearing as if they were empty! What was particularly strange is that this change came on suddenly, during a period where I had made no changes to the Customer ITNM deployment. The only thing that had "changed", so to speak, was that my laptop had been forcibly rebooted by our Tivoli Endpoint Management software. I don't know if it was an update to the IBM java client itself, or something related to it, but the same Network Views that worked the day before now showed up empty.
The first thing I did was check the NCIM database, to make sure the entities were still there. I could still report on the entities via SQL, so it had to be something with Topoviz or the actual Java client.
One nice thing about Topoviz is that if you change the logging level in the properties file, you don't have to restart anything. I edited the
$NCHOME/precision/profiles/TIPProfile/etc/tnm/topoviz.properties file, and changed this property: topoviz.log.level=ALL
Right away, the $NCHOME/precision/profiles/TIPProfile/logs/tnm/ncp_topoviz.0.log file started to have much more information about what the client was doing. But it still didn't make sense to me.
It was time for help from Support, and I was pointed to this Technote:
The recent re-release of ITNM 3.9 is a welcome thing. It's a huge timesaver for a new install, not having to deal with installing ITNM and them immediately applying Fix Pack 2. So hooray for that!
Here's some other benefits:
ITNM 3.9 now supports IE 9 and Firefox ESR 10 (although in my experience, FF ESR 10 renders some portlets wrong the first time and you have to refresh the page). Another nice thing is that an explicit DB2 entitlement is included, so you can substitute DB2 for Informix (which is looking more and more like a good idea.)
Lastly, there are a lot of updates to bundled software
- The package contains an updated OMNIbus Web GUI
- The package upgrades Informix (but consider DB2 or MySQL)
- The package contains a newer TIP
- The package contains a newer TCR
So what's to complain about? These two things:
- The package includes a older version of the SNMP trap probe
- The package includes an older version of the Netcool Include Library
Seems like an oversight to me. This means that after running the ITNM installer, a conscientious installer needs to upgrade both the SNMP probe and the NcKL stuff. The former should be done
via the usual nco_install_integration, while the latter is done via a straight overwrite of the rules tar.
Note that NcKL 3.6 did make a minor (but important) change to the Advanced Correlation automations. A "when" clause was added to each automation that makes sure that the automation only runs when the ObjectServer is the acting primary. Obviously this only matters when dealing with a pair of ObjectServers in a fail-over configuration, but it's good practice and will hopefully be standard soon. For the purposes of installing the newer NcKL, it's easy to either re-run the SQL file for the automations (ignoring errors for columns already built) or just update them by hand.
Here's the clause:
By the way, the part numbers (for Linux) that you would download are:
Netcool/OMNIbus Probe for SNMP (nco-p-mttrapd 13_0) for Linux English (CI5G7EN)
Netcool Knowledge Library (NcKL) Enhanced Probe Rules V3.6 Multiplatform English (CI9AZEN)
Disclaimer: The part numbers were current as of this writing (11/2/2012), it is always wise to check for updates
Modified by RogerPowell
I am a strong proponent of installing and running products from the Netcool suite under a dedicated user ID that is not the root user. Of course this means that certain processes that require root access will need to be set up as "setuid root". When I install the ITNM product, there's a handy script that does just that for the core ITNM processes (things like ncp_df_ping, ncp_dh_ping, ncp_poller and so forth.)
Curiously, the ITNM scripts don't do anything about the SNMP trap probe (aka the "mttrapd" probe), even though the ITNM installer will happily install this probe as part of a unified ITNM/OMNIbus install. For the record, here's how to do it: (HT to Alex Greenbank for the 64-bit linux info)
For linux: (First you must check for 32 vs 64 bit versions)
# $OMNIHOME/probes/nco_p_mttrapd -version
Netcool/OMNIbus MTTrapd probe - Version 7.4.0 64-bit <-- Notice that this example is the 64 bit version
For linux (32 bit version):
# cd $OMNIHOME/probes/linux2x86
# chown root nco_p_mttrapd
# chmod u+sx nco_p_mttrapd
# echo “$NCHOME/platform/linux2x86/lib” >> /etc/ld.so.conf
For linux (64 bit version):
# cd $OMNIHOME/platform/linux2x86/probes64
# chown root nco_p_mttrapd
# chmod u+sx nco_p_mttrapd
# echo “$NCHOME/platform/linux2x86/lib64” >> /etc/ld.so.conf
# echo “$OMNIHOME/platform/linux2x86/lib64” >> /etc/ld.so.conf
# cd $OMNIHOME/probes/aix5
# chown root nco_p_mttrapd
# chmod 4555 nco_p_mttrapd
# cd /usr/lib
# ln -s $NCHOME/platform/aix5/lib/libOpl_r.so
# ln -s $NCHOME/platform/aix5/lib/libnetcool_r.so
# ln -s $NCHOME/platform/aix5/lib/libPa_r.so
# ln -s $NCHOME/platform/aix5/lib/libslclient_r.so
# ln -s $NCHOME/platform/aix5/lib/libsybct_r.so
# ln -s $NCHOME/platform/aix5/lib/libsybcs_r.so
# ln -s $NCHOME/platform/aix5/lib/libsybtcl_r.so
# ln -s $NCHOME/platform/aix5/lib/libsybcomn_r.so
# ln -s $NCHOME/platform/aix5/lib/libsybintl_r.so
# ln -s $NCHOME/platform/aix5/lib/libtre.a
# ln -s $NCHOME/platform/aix5/lib/libicuuc40.a
# ln -s $NCHOME/platform/aix5/lib/libicui18n40.a
# ln -s $NCHOME/platform/aix5/lib/libicudata40.a
All of the links are necessary because AIX requires setuid programs to use shared libraries from the default search path (/lib or /usr/lib).
Note: The SNMP Trap Probe gives away its permissions once it has opened the UDP port specified by its properties. When examining the running probe with the "ps" command, don't be alarmed to see it running as an ordinary user, even though you configured it for "setuid root" operation.
A Further Note: It's high time we Netcoolers stopped using the term "Multi-threaded trap probe" for this probe. That term is a legacy from years ago when most probes were single-threaded and the new "multi-threaded" trap probe represented a significant performance boost. Now all our probes are multi-threaded and it makes no sense to call this out in the name. Let's get back to calling it what it is: The SNMP Trap probe.
Update for Solaris: I haven't vetted this personally, but here's a link for doing this on Solaris.
I always try to install the various Netcool products as a dedicated user (usually "netcool") vs installing as root. There are a lot of good reasons for this - worth a blog post sometime, but not at the moment,
After installing the refreshed version of ITNM 3.9, I had to run a script (as root) to install Informix. This script is in $NCHOME/precision/install/scripts and it is named "install_ids_root.ksh". Unfortunately, the documentation does not contain sufficient warnings about this script, so here's my hard-earned knowledge:
(Note: Updated 10/10/12 as I learned more)
Assume for the moment that your NCHOME is /opt/netcool/ibm owned by "netcool" with group ownership of "ncoadmin", and your precision domain is called TEST.
1) Make sure (at least on AIX) that /usr/sbin is in your PATH since the script will attempt to create users (informix, ncim) with the "useradd" command.
If you create these users yourself, make sure that they have valid passwords that matched what you gave the installer for the database password
(if you've forgotten that password, it's in ../data/ids.properties as explained in step 4)
2) Make sure the path to $NCHOME/platform is at least r-xr-xr-x
3) Make sure to have the precision domain set in your environment, e.g. export PRECISION_DOMAIN=TEST
4) Run the script with "-f ../data/ids.properties" - that file has various settings (like NCHOME) taken from the installer session.
If things go wrong, you can use: "$NCHOME/Uninstall_ITNM -i" to wipe out Informix and try again. If you are prebuilding the informix and ncim users you'll have to do it again.
By the way, there are several important environmental variables to set for Informix, Here's a snippet from
the .profile (or .bashrc) file for the "netcool" and "informix" users:
# If Informix Dynamic Server is installed, add commands to PATH
if [ -x $NCHOME/platform/$PLATFORM/informix ]; then
inpath $INFORMIXDIR/bin || PATH=$PATH:$INFORMIXDIR/bin
Once the Informix database is built, you should be able to connect to it with Informix's "dbaccess" command. For starters,
log in (or su) as the Informix user (make sure it has the environment variables set as shown above.) Informix uses system identification - if you are on the same host and have already authenticated
yourself as "informix" it will automatically authenticate you to the database.
Try "dbaccess itnm" - If all goes well, you'll be in the client and able to access basic information such as the dbspaces that have been built. In my case, there was another glitch....
The host (we'll call it "itnmhost") was multi-homed. The Informix install created the etc/sqlhosts.ITNM file with a line that looked like:
ITNM onsoctcp *itnmhost 9088
I could not authenticate to the database (Error 951) - Neither the "informix" user nor "ncim" user worked. However once I removed the asterisk before the host name and made the line:
ITNM onsoctcp itnmhost 9088
This fixed the problem. Now I could connect to the database and populate it with the $NCHOME/precision/scripts/sql/informix/populate_informix_database.sh script. (The installation guide does not do a good job of explaining that you need to do this if you are installing Informix after running the ITNM installer.)
Lastly, while still authenticated as the "informix" user, I gave the regular "netcool" connection privileges:
$ dbaccess itnm -
grant connect to 'netcool';
This allows me to run the dbaccess command as the netcool user without have to use a "connect' statement to authenticate to Informix (while on the same host.)
I've begun a project installing the refreshed ITNM v3.9 on an AIX system for a customer. This is my first look at the refresh and it didn't take long before the first surprise.
First off - it was time to find the software part number so the customer could download it. A little searching on Xtreme Leverage yielded:
- IBM Tivoli Network Manager IP Edition 3.9 AIX Multilingual (updated to Network Manager 3.9 Fix Pack 2 equivalent) (CI7F1ML) - posted 13-Sep-2012
My customer downloaded the part, but when I looked for CI7F1ML.tar.gz in the specified directory it was nowhere to be seen. Instead there was "ITNP_IP_AIX.tar.gz" - What the heck? Well it certainly wasn't my customer's fault, because when I tried downloading it in XL, Download Directory suggested that same name.
Then there is the mystery of the OMNIbus Part Number. Depending on how you search, you will find either
- IBM Tivoli Netcool/OMNIbus Core 7.3.1 for AIX Multilingual (CZW01ML) - posted 25-Feb-2011
- IBM Tivoli Netcool OMNIbus V7.3.1 Core for AIX Multilingual (CI3JLML) - posted 21-Oct-2011
They are almost the same size (358,993,920 vs 359,086,080 byte) - I sure hope they are the same except for something like header information.
The interesting thing is if you download the first one, Download Director will try to create a file named "CZW01ML.tar" - as expected, while the second one wants to create a file named "NCO_CORE_AIX_ML.tar". Among other things, this latter name will not work with the ITNM installer, which looks for either CI3JLML.tar or OMNIbus-v7.3.1-aix5-5.21.36.tar (which is what I had to rename the file to, in order to let the ITNM installer do OMNIbus for me too.)
I've never been a big fan of part numbers for file names. Once I dump it in a directory, who would know that CI3JLML.tar is really OMNIbus 7.3.1 for AIX ? But this change is worse - the suggested file names have no revision information in them at all. Wouldn't it make a lot more sense if the suggested downloaded file name be (in this case): "OMNIbus-v7.3.1-aix5-5.21.36.tar" ?
Imagine, if you will, the cyberspace equivalent of a modest room with a circle of chairs in the center. The chairs are occupied by a variety of technical types, brought together to this support group because they share the same problem. Yours Truly, gets up, faces his peers and says:
"Hi. My name is Roger Powell and I am an Information Hoarder. I work with the Netcool/OMNIbus and Tivoli Network Manager products, and I've got lots of helpful tidbits of information, various utilities, and plenty of experience that I've been meaning to share ... BUT ... I don't share anything unless it's perfect! If I've written a utility script, it has to be clean and general purpose with help text. If I decide to respond to a question on the International User's Group mailing list, my answer must be 100% correct as if the voice of God himself (or at least Alex Greenbank) had spoken. Because of this, I've likely deprived people of the chance to profit from my experience."
At this point in my imaginary scenario, all my peers offer encouragement and tell me that they value me as a person even if I am an Information Hoarder. The imaginary group leader though, prescribes what he thinks is the perfect remedy: "You need to blog!"
Um yeah - active imagination indeed! So this perfectionist stands before you (or whatever the cyberspace equivalent is) and expressly disavows perfection. I will not wait to make stuff "exactly right" before I post it. I'll be happy to fix errors once they are pointed out to me. And ... I promise that no other post on this blog will be as dull as this first one!
OK - A little more housekeeping ... The blog will focus on Tivoli's Netcool/OMNIbus and Network Manager products because that's what I do for a living. The perspective will be from what I do now - post-sales consulting for IBM - although I've done other things. I've worked with Netcool/OMNIbus since late 1997 (that's almost 15 years if you do the math) so I've seen a few installations in my time. My aim is to be interesting and useful - at least for those who work with OMNIbus, the OMNIbus Web-based GUI, and ITNM. Everybody else should have surfed away by now!
One final point today - If you are still reading this, you are a member of the INUG, right? (That's the International Netcool User's Group.) It was formed back when the Netcool products all came from Micromuse, so it remains focused on those products as opposed to others in the Tivoli Suite. If you're not a member, get thee hence and join! It's a great resource.
I should make it clear right now - I have no connection with the INUG, other than being a member. In turn, the INUG has no official relationship with IBM, but several IBMers do answer questions there (the aforementioned Alex Greenbank being a superstar IMO.)
And lastly this disclaimer: I am an IBM employee, however the statements in this blog are my own and do not necessarily represent the opinions or official positions of IBM. I'm solely responsible for the content of the posts.
The posts should be much more fun from now on.