Going through the Perly Gates
GohBoonYeow 270004BU3V Visits (3933)
Got your attention there with the odd title. My virgin entry so an introduction is in order. My name is Goh Boon Yeow. Most folks address me by my surname "Goh" because it's easier. I've been working on performance management products for more than half a decade; first starting with the currently EOL NPR product, then onto Perl Gateways, and finally touching on Technology Packs. Today, I manage and lead a team of 20 engineers covering wireless Technology Pack development, Lab Services, and Level 3 support.
With introductions out of the way, let's get down to business. The subject today is the general term of data mediation (or transformation). To me, data mediation has always been a game of fitting square pegs into round holes. The square peg being data from any given source. The round hole is the data warehouse. In the case of TNPM Wireless, the mediation piece would be the Perl Gateways. For Wireline, you have SNMP and UBA. In other instances, you have DataStage. All of these perform one key task. Transforming and massaging data in varying data formats into a standard format that is readable by a core platform to be loaded into a data warehouse.
Some of you might be scratching your head and wondering what is this guy talking about. Allow me give you an example in a real world context. Let's say you were designing a database and you expect to get data into the database. You wrote a simple SQL insert statement and wrap it into an application that reads a simple CSV file for input. The CSV file needs to be very specific in terms of content and structure because the SQL application is rigid. Now let's say you had two of your buddies provide you with input for your database. Unfortunately, the guys didn't know that your program expects very specific input files. One guy gave you a spreadsheet with the required columns while the other lady exported her input into an XML file. When you tell your buddies you need it in a specific format, they look at you funny and say it's your problem to deal with.
In such a situation, you have two options. First would be to extend your simple application to be able to understand the new input formats. Second is data mediation by transforming the given input into the expected CSV format. Whether you do this manually or via an automated application is secondary. The act of transforming that data into a strict format is what is specifically term as data mediation. Data mediation also encompasses the transformation of the content itself. This includes but is not limited to topics like data enrichment, combining multiple sources, data grouping and aggregation.
Over the next few entries, I'll be zooming into the Perl Gateways (or what I prefer to call the Perly Gates), a sub-component of the TNPM Wireless platform. Among other things, I'll touch on and share some trade secrets about (in no specific order):
Thanks for the attention. Feel free to leave your thoughts and comments. I'll take them into consideration and try to touch on topics that are of interest to the audience.