Topic
  • 8 replies
  • Latest Post - ‏2012-04-02T23:17:17Z by mendell
Jim Sharpe
Jim Sharpe
98 Posts

Pinned topic Can UDBSource be used to ingest port 162 SNMP traps?

‏2012-03-30T20:54:15Z |
Sorry in advance if this is a silly question, but can the standard toolkit UDPSource operator be used to ingest port 162 SNMP traps?

I just tried a simple test and although the bytes do seem to be getting to the port and I have validated that the UDP operator is sucessfully binding to that port, it doesn't seem to realize that there is anything there, even if set to binary format?
  • mendell
    mendell
    219 Posts

    Re: Can UDBSource be used to ingest port 162 SNMP traps?

    ‏2012-03-31T02:22:08Z  
    Have you tried block format, and reading the packet as a *blob*?

    Mark
  • Jim Sharpe
    Jim Sharpe
    98 Posts

    Re: Can UDBSource be used to ingest port 162 SNMP traps?

    ‏2012-04-02T15:26:16Z  
    • mendell
    • ‏2012-03-31T02:22:08Z
    Have you tried block format, and reading the packet as a *blob*?

    Mark
    Unfortunately that didn't work either.

    An example code snip is below. Although port 162 is showing as bound and traps are getting there, Streams doesn't outputs anything to the console. (I've also tried writing it to a file but nothing goes there either.)
    stream<blob blk> TrapData = UDPSource()
    {
    param
    address : "10.13.161.2"; // the IP for the host where Streams is running
    port : 162u ;
    format : block ;
    }
    () as TrapData_FileWriter = Custom(TrapData) { logic onTuple TrapData : println ("got something") ; }
    Assuming I could get the UDPSource operator to work, my next thought was to something like the code below to convert it to strings which I could then punctuate and parse. However, I'm not sure where I can get the string representing the encoding. Are there standard functions available from within Streams to look-up an encoding string (e.g., something like getEncoding("iso-8859-1") or perhaps even just a header file somewhere listing the valid encoding strings.

    stream<rstring txt> TrapStrings = Functor(TrapData )
    {
    output
    TrapStrings : txt = convertToUtf8((list<uint8>)blk, "what goes here?") ;
    }
  • mendell
    mendell
    219 Posts

    Re: Can UDBSource be used to ingest port 162 SNMP traps?

    ‏2012-04-02T15:47:24Z  
    Unfortunately that didn't work either.

    An example code snip is below. Although port 162 is showing as bound and traps are getting there, Streams doesn't outputs anything to the console. (I've also tried writing it to a file but nothing goes there either.)
    stream<blob blk> TrapData = UDPSource()
    {
    param
    address : "10.13.161.2"; // the IP for the host where Streams is running
    port : 162u ;
    format : block ;
    }
    () as TrapData_FileWriter = Custom(TrapData) { logic onTuple TrapData : println ("got something") ; }
    Assuming I could get the UDPSource operator to work, my next thought was to something like the code below to convert it to strings which I could then punctuate and parse. However, I'm not sure where I can get the string representing the encoding. Are there standard functions available from within Streams to look-up an encoding string (e.g., something like getEncoding("iso-8859-1") or perhaps even just a header file somewhere listing the valid encoding strings.

    stream<rstring txt> TrapStrings = Functor(TrapData )
    {
    output
    TrapStrings : txt = convertToUtf8((list<uint8>)blk, "what goes here?") ;
    }
    Could the problem be that there is another process attempting to read the port? Can you write a small Perl script to list at that port, and confirm that it gets the information?

    Can you run a standalone program with options '-d3 -c' to see if the console messages give any clues?

    Mark
  • mendell
    mendell
    219 Posts

    Re: Can UDBSource be used to ingest port 162 SNMP traps?

    ‏2012-04-02T15:52:27Z  
    Unfortunately that didn't work either.

    An example code snip is below. Although port 162 is showing as bound and traps are getting there, Streams doesn't outputs anything to the console. (I've also tried writing it to a file but nothing goes there either.)
    stream<blob blk> TrapData = UDPSource()
    {
    param
    address : "10.13.161.2"; // the IP for the host where Streams is running
    port : 162u ;
    format : block ;
    }
    () as TrapData_FileWriter = Custom(TrapData) { logic onTuple TrapData : println ("got something") ; }
    Assuming I could get the UDPSource operator to work, my next thought was to something like the code below to convert it to strings which I could then punctuate and parse. However, I'm not sure where I can get the string representing the encoding. Are there standard functions available from within Streams to look-up an encoding string (e.g., something like getEncoding("iso-8859-1") or perhaps even just a header file somewhere listing the valid encoding strings.

    stream<rstring txt> TrapStrings = Functor(TrapData )
    {
    output
    TrapStrings : txt = convertToUtf8((list<uint8>)blk, "what goes here?") ;
    }
    Forgot the other question:

    I believe that iso-8859-1 is a subset of UTF-8, so that isn't a problem. If you did want to do conversions, you would have to write a C++ native function that uses iconv to implement them.

    A C++ native function is also needed to convert a blob to an rstring. I will see if we can add some support for rstring<->blob conversions in the standard library in the future.

    Mark
  • Jim Sharpe
    Jim Sharpe
    98 Posts

    Re: Can UDBSource be used to ingest port 162 SNMP traps?

    ‏2012-04-02T16:39:54Z  
    • mendell
    • ‏2012-04-02T15:52:27Z
    Forgot the other question:

    I believe that iso-8859-1 is a subset of UTF-8, so that isn't a problem. If you did want to do conversions, you would have to write a C++ native function that uses iconv to implement them.

    A C++ native function is also needed to convert a blob to an rstring. I will see if we can add some support for rstring<->blob conversions in the standard library in the future.

    Mark
    Thanks Mark,

    I'll take a shot at a Perl listener when I get back in the office in the morning, and would also welcome any additional future support in this area, either by way of example operators/code or additional functionality in Streams itself. I'm guessing it could be a fairly useful capability for others besides my client.
  • RockClimb
    RockClimb
    8 Posts

    Re: Can UDBSource be used to ingest port 162 SNMP traps?

    ‏2012-04-02T20:40:21Z  
    Thanks Mark,

    I'll take a shot at a Perl listener when I get back in the office in the morning, and would also welcome any additional future support in this area, either by way of example operators/code or additional functionality in Streams itself. I'm guessing it could be a fairly useful capability for others besides my client.
    I believe for the address parameter you should use the address that is sending the traps, or leave it out if you want to accept traps from anywhere.
  • Jim Sharpe
    Jim Sharpe
    98 Posts

    Re: Can UDBSource be used to ingest port 162 SNMP traps?

    ‏2012-04-02T21:11:18Z  
    • RockClimb
    • ‏2012-04-02T20:40:21Z
    I believe for the address parameter you should use the address that is sending the traps, or leave it out if you want to accept traps from anywhere.
    That is exactly correct. As soon as we made the change we started seeing data come through. Now all we need to do is figure out the best way to decode it. :) FYI we had correctly set the IP earlier, but during the course of many experiments that resolved other issues, had not set it back.
  • mendell
    mendell
    219 Posts

    Re: Can UDBSource be used to ingest port 162 SNMP traps?

    ‏2012-04-02T23:17:17Z  
    That is exactly correct. As soon as we made the change we started seeing data come through. Now all we need to do is figure out the best way to decode it. :) FYI we had correctly set the IP earlier, but during the course of many experiments that resolved other issues, had not set it back.
    I am glad that you got this working. In order to decode the data, I suggest that you write a C++ native function that accepts the data and a tuple with the right fields, and then uses the reflective interface to tuples to assign to the fields.

    The alternate is to write a C++ primitive function that takes the data from the UDPSource, and outputs tuples. This has the advantage that you can use the code generation template to generate efficient C++ code that can directly address the fields in the tuple being generated. This would be more efficient, but may not be necessary for your uses.

    This all assumes that you can figure out the format of the SNMP messages, and that you can craft an SPL tuple that represents the parts of the data that you wish to extract.

    Mark