Tivoli Network Manager Collector Layer Recipe
raffaee 270001GU7S Comments (2) Visits (4249)
have been asked on numerous occasions by our IBM Services team to
explain how to build connectivity using the Perl Collector. In this
article, I’m going to share with you the secret recipe that goes
into building a Perl Collector and how the data it retrieves are
processed by our Collector layer discovery agents.But
before we start cooking, you should familiarize with the concepts
related to Collector development.
Let's recap, a Collector is an independent integration tool in Network Manager which interfaces with EMS and you build one using the Perl language. How would you know when to build a Collector as opposed to developing Discovery Agents? In simple terms, you should first identify your target discovery source. If you intend to communicate directly with your target device, then Collectors are no longer applicable. On the other hand, if you are not allowed to communicate directly with the devices and must go through an EMS, then developing Collector is the way forward.
Are Collectors and Collector discovery agents similar? The answer is no as these components are distinctive. A Collector is a component which replicates EMS data into a format understood by Collector discovery agents and listens to XML-RPC calls from Network Manager. In other words, Collector also acts as an XML-RPC server. Collector discovery agents are components which retrieve data from Collectors and use XML-RPC to interface with Collectors via the Network Manager XML-RPC Helper.
To better illustrate this, the components described interface with each other in the following manner:
There are a few network models available in Collector framework which allow you to model your managed elements. Among the models available are:
(1) Physical entities modeling for asset management,
(2) MPLS Layer 2 VPN, MPLS Layer 3 VPN for MPLS management,
(3) Connectivity data for Layer 2 and Layer 3 topology management,
this article, I will only emphasize on building connectivity data in
Collector and ensuring the data is propagated to discovery agent
returns table and Network Manager OQL Layer tables.Throughout
this article, I
make references to IBM Tivoli Network Manager IP Edition, Collector
Developer Guide document which the document is available for download
at this link,
Now on to the FUN stuff!
Suppose we name our EMS target ACME EMS. To start building a Collector that communicates to ACME EMS, there are 3 cooking stages involved namely:
(1) Feed connectivity data to Perl Collector;
(2) Verify data in Collector Store and;
(3) Verify Connectivity data in returns table.
Each stage has several steps which I will describe the details as below.
(i) Source Network Element.
(ii) Destination Network Element,
(iii) Source Port (or
Source Interface) and Destination Port (or Destination Interface)
Where 1001 is a sample of an Interface Index, PortA, Port123 is a sample of an Interface Id.
Most EMSes are usually not SNMP based and therefore Interface Index (IfIndex) may not be present. For cases like this, use Interface Id field instead.
Step 2: Map EMS connectivity data retrieved from Step 1 to the Collector fields.
Section 18.104.22.168 of the
Collector Developer Guide details the AddL
Step 3: Develop Collector code
Pre-requisites before building Layer 2 or Layer 3 connectivity data are Device Chassis data and Interface data presence in Collector Store. Unfortunately, I will not touch the topics of building Device Chassis data and Interface data in this article. I advise you instead to refer to IBM Tivoli Network Manager IP Edition, Collector Developer Guide document for more details.
Next step requires us to build a hash table with Collector fields and use Perl Collector API call to push hash table data to Collector Data Store.
In Example A, I use Interface Id as an
alternative to IfIndex for the case where IfIndex is not present in
the target EMS. InterfaceId is more versatile than IfIndex because
it accepts alphanumeric values, typically port name such as
'port1/1/1' can be used to populate this field. Below is an example of Perl Collector
code for building Layer 2 using Interface Id :
** Note: AddLayer2Connection is Perl API in Store module which add Layer 2 connectivity data into Collector::Store
Example A : Building Layer 2 using Interface Id (Click above image to enlarge)
In Example B, I use IfIndex for the case where IfIndex is present in the target EMS. IfIndex is quite rigid because it is in numeric and its value for an interface must be consistent throughout each discovery cycle. In the following example, I build Layer 2 connectivity for Perl Collector using IfIndex :
** Note: AddLayer2Connection is Perl API in Store module which add Layer 2 connectivity data into Collector::Store
Example B : Building Layer 2 using IfIndex (Click above image to enlarge)
Building Layer 3 Connectivity data to Collector does not differ much with Layer 2 Connectivity. The main difference is Perl API call where AddLayer3Connection method is called for building Layer 3 Connectivity data. In the next 2 examples, I share 2 code snippets which build Layer 3 connectivity data using Interface Id and IfIndex respectively.
In Example C, I build Layer 3 connectivity data using Interface Id.
** Note : AddLayer3Connection is Perl API in Store module which add Layer 3 connectivity data into Collector::Store
Example C : Building Layer 3 using Interface Id (Click above image to enlarge)
In Example D, I build Layer 3 connectivity for Perl Collector using IfIndex :
** Note: AddLayer3Connection is Perl API in Store module which add Layer 3 connectivity data into Collector::Store
Example D : Building Layer 3 using IfIndex (Click above image to enlarge)
Step 4: Call the methods in LoadData.
Make calls to the methods we've just created in Example A, Example B, Example C and Example D in LoadData.
Example E : Add LoadL2xxx and LoadL3xxx methods to main LoadData (Click above image to enlarge)
Stage 2: Verify data in Collector Store
In Stage 1, we have built the Collector code to add data to Collector Store. This Stage is about verifying that we've successfully added Connectivity record in Collector Store.
Step 1: Verify Perl Collector data store using Dumper.
Perl has a useful module called Data::Dumper which allows us to inspect data in Collector::Store. Therefore, let's use print Dumper for Collector Store which is very useful especially when you want to ascertain whether or not your Collector has successfully consumed Layer data built described from Step 1 to Step 4.
To print Dumper, add the following line number 1 and line number 10 in LoadData method for the Collector.
Example F : Print Dumper for Collector Store at Line 10 (Click above image to enlarge)
Step 2: Run the Perl collector.
Run Perl collector via command line. During collector execution, depending on your standard output, you will be getting lots of Perl hashes and arrays. Perl arrays which we are interested in are Layer2Connectivity and Layer3Connectivity.
Example G below illustrates expected output from Perl Collector Store in standard output.
Example G : Collector Store output (Click above image to enlarge)
Next compare the output in Example G with Stage 1 Step 2: Map EMS connectivity data retrieved from Step 1 to the Collector fields. I expect the output in Example G to match with your manual map in Stage 1 Step 2.
Stage 3: Verify connectivity data in returns table
Should you successfully complete Steps 1-3 from Stage 1, then kudos! You have ascertained that layer connectivity data is present in Collector Store for the Collector. Now, the Collector is ready to feed data to CollectorLayer2 or CollectorLayer3 discovery agents when Network Manager Collector discovery is invoked.
Step 1: Enable all Collector discovery agents in Network Manager and perform a discovery.
After discovery completes, use OQL to
query the backend's discovery agent returns tables. The table to
query in this case is Coll
Screen Snapshot A (Click above image to enlarge)
And the expected output from the OQL backend for CollectorLayer2 discovery agent returns table is as follows :
Example H : Coll
Data in CollectorLayer2 discovery agent returns table basically contains field values in Example G. They are located at the following line number :
(a) line 13 contains value from SourceIfIndex field in Collector Store,
(b) line 20 contains value from DestinationIfIndex field in Collector Store,
(c) line 21 contains value from DestinationName field in Collector Store,
(d) line 23 contains value from SourceName field in Collector Store,
(e) line 39 contains value from SourceInterfaceId field in Collector Store,
(f) line 46 contains value from Dest
(g) line 47 contains value from DestinationName field in Collector Store,
(h) line 49 contains value from SourceName field in Collector Store.
For each line stated above, its value
corresponds to Collector Store field in Example G. Now, we roughly
know fields that we've mapped in Stage 1 Step 2 have sunk into OQL
backend at discover agent returns table. I would expect similar
output from Coll
Let's take a step back. Remember that in Stage 1 Step 3, we have prepared Collector hash table with SourceName, SourceInterfaceId, DestinationName and Dest
Table A : Mapping between Collector Field, XML-RPC field, OQL backend Discovery agent table (Click above image to enlarge)
At this point, you must be wondering about the relationship between XML-RPC fields with Collector and Collector Discovery agents. Recall in the introduction section, I mentioned Collector Discovery agents interface with Collector via XML-RPC helper.
When an XML-RPC method say
Do I not confuse you so far? I hope not :) In my opinion, this paragraph contains the main secret to understanding Collectors better. By getting to know source field, intermediate field and destination fields, you can build your first Perl Collector with ease!
Step 2: Verify total number of Layer 2 or Layer 3 connectivity data in Collector Store match with row count in CollectorLayerX returns table.
Perform a row count Layer 2 connectivity data in Collector Store and observe whether or not it matches with OQL result
for OQL query “select count(*) from
Repeat for Layer 3 connectivity data verification. You are on right track should these row count match. Otherwise, you will need to revisit your Collector code.
Step 3: CollectorLayerX returns table will then feed data to Layer OQL database tables.
This step is the final step for this
recipe. Should either one of these steps not return the expected
values, you will need to take a step back, understand the data flow
and perform corrections if needed. At the OQL backend, data from the
Collector Discovery agents (i.e. Coll
At GUI frontend, when “Correlating Connections” phase on Monitoring bar from Network Manager GUI shows “Completed”, it means our OQL Layer tables are already populated with connectivity data.
Next step, at OQL backend, let's check rowCount of Source OQL tables using filter “where m_RemoteNbr <> NULL” and compare it with rowCount of Target OQL Layer tables. Use table below which maps Source OQL table to Target OQL Layer table.
Example I : Table mapping between Discovery agent returns tables to Layer tables (Click above image to enlarge)
Typically, I do a rowCount for Source
OQL table and it must match with rowCount for Target OQL table. For example, data from
This article is one of many Collector articles which I plan to write. I hope by now, you have a good grasp of understanding Collector data structure and Collector data flow.
Stay tuned for more articles from me. Should you have any question, do drop a comment and I’ll be happy to answer them to the best of my knowledge. Should you have any questions or article maybe of an interest to you, do drop your comment as well.
<< Raffaee >>
Acronyms / Abbreviations :ACME = A Company that Makes Everything
API = Application Programming Interface
EMS = Element Management System
GUI = Graphical User Interface
IfIndex = Interface Index
IP = Internet Protocol
MPLS = Multiprotocol Label Switching
OQL = Object Query Language
SNMP = Simple Network Management Protocol
XML-RPC = Extensible Markup Language-Remote Procedural Call