Modified by PeggyZ
I'm really enjoying being back working at IBM's Silicon Valley Lab, after spending some time on assignment in IBM's Global Business Services division. Since November, I've been working as one of the product architects for IBM's Big Insights product, focusing on Text Analytics.
This week we were working closely with our top-notch design team, and it occurred to me that the four big areas of data actifity we were discussing spelled out LOVE.
Load, Organize, Visualize, Export.
Since we started just after Valentine's day, we decided this was appropriate. And for me, this has all truly been a labor of LOVE. I'm trying to make it stick as a code name. You heard it here first!
I thought I'd share a nice little AQL output example that might be of help in your Big Insights text analytics programming.
Consider that you've created a very complex AQL view, one which extracts a very detailed concept from text. This might take into account many different text constructs and match a large variety of different text in documents.
If you want to take that view and use it to simply identify which documents have an occurance of this advanced concept, you can do it like this. In this case, I've assumed the complex view name is called "Division'.
create view DivisionCount as
select Count(*) as dc from Division D;
create view DivisionBoolean as
when Equals(DC.dc,0) then 'no'
from Document R,DivisionCount DC;
To test this using the Text Analytics tutorial example, I created a new document called text.txt which doesn't have any matches for Division in it. When I run it and select to see DivisionBoolean in the output, I get this as a result:
We now have the capability in DB2 9 for z/OS to search text data that is stored in DB2 for z/OS using SQL statements. Wahoo!
You mean you missed the announcement?
And you just followed that link and still couldn't find it? It's under "utilities", no, it's not that kind of utility, but still, that's where it is.
What is added is built-in functions for contains() and score(), and also shipping a text search server which runs on a separate, non-z/OS server. For more details, see the announcements!
One prerequisite for this is to have a WLM application environment set up to run a java user-defined function. The early customers I've been working with have had the most stumbling with this part of it. So, this is something you can set up even if you are not quite to DB2 9 for z/OS yet. I'll post some more on the setup steps for that.
So, what kind of data are you going to search, and what kinds of searches are you going to do?[Read More]
I had the pleasure this week of participating in a couple of live IBM Academy of Technology events - the first for me in several years. It was great to reconnect with some of my colleagues, and get a chance to talk over IBM technology and client engagements.
Our academy president, Rashik Parmar, had arranged for a session with Bill Gajda from Visa. Bill's role at Visa includes mobile strategy as well as global innovation. We had a frank and open discussion about Visa's use of data, who their customers are, and IBM's technology role.
Please note that any errors in here are likely mine, I didn't take notes, I was far too engaged and inspired thinking about all the data (!) during the discussion.
Clearly, credit card transactions are "big data". I don't recall the exact numbers, but here's an article from over 5 years ago which references 300 million transactions a day. Bill Gajda told us that Visa keeps around 5-7 years of past transactions. They primarily use this information for real-time fraud scoring of individual transactions. So, when you charge something, Visa gets the approval request, and attaches a score which indicates the likelihood that the transaction is fraudulent. This is based on several factors, such as your usual spending patterns and the location of the transaction. That is the primary use for the historical data. Bill also told us of a partnership they did with The Gap, where if a Gap customer in their loyalty program opted in with their phone number, Visa would tell the Gap when a purchase was being made nearby a Gap store, and then The Gap could text an offer to their customer for a discount at the Gap. I was able to find an article describing that, and was surprised to see it was from 2011! Now, if you're a geek like me, you'll think for a second about how the data flows. (This is just Peggy speculating, I have no further knowledge of the internals of this...) So say you charge your lunch on your Visa card, and the approval data flows to Visa, then after Visa processes the approval, it also looks at the zip code and compares it to a list of zip codes it has from The Gap of their store locations (I'm making this easy by Zip code rather than a lattitude/longitude based proximity lookup). When there's a match, Visa initiates a message to The Gap to tell them you're close to a Gap store... and Gap then sends you a text message with a promotion. Phew! Do you think Visa also has a indicator on your credit card that you're a Gap customer? They must... I can't imagine they do this on "every" Visa transaction... ! Actually, I guess there is "Gap Visa" card, so it's probably those which are targeted.
I don't know about you, but I have fun thinking about this kind of stuff. Some other interesting facts I learned are that Visa doesn't have your name or personal information - that data all resides with the issuing bank. One of my colleagues asked Bill Gajda whether if a person has multiple Visa cards, does Visa have information across the cards. At first, he answered "yes", but then with another colleague's question about matching the names, he said "no" - Visa has no idea that you are the same person when you have multiple cards. Interesting thought about whether we could perhaps match spending patterns to identify who might be the same person? That might not be something Visa wants to do, of course, I was thinking of it more as a theoretical exercise. Also, Visa doesn't really consider cardholders customers, despite its advertising budget - its customers are the issuing banks and the accepting retailers, so they are the ones who would be likely consumers of the volume of data or aggregated insights from it. Tho I guess the banks already have it, too.
In any case, again, data about people and their habits is big data and big business. I personally don't find any of this "scary" but I guess some people might. I just like people, so data about people is interesting data to me. I might those who do find it scary might just have to take a closer look at some of those tiny print privacy agreements!
Hi all, and welcome to my new blog!
All this week, I am busy at the International DB2 User's Group conference here in Athens, Greece. The two topics I'm presenting are "Native SQL stored procedures in DB2 9 for z/OS" and "Java stored procedures".
I was very glad to get the chance to get across this key point in yesterday's presentation on native SQL stored procedures -- yes, when they execute they are eligible for redirect to the zIIP processor, but only when they are invoked from a remote client, and then at the same percentage as other DDF work. Lots of presentations lately have stated this a bit more broadly, leaving people with the wrong impression. So, let's be clear - when a remote thread comes into DB2, it executes on an enclave SRB, and DB2 dials the zIIP redirect to a certain percentage. A DB2 9 native SQL procedure executes on the invoking execution block and not on a WLM-SPAS TCB - that's one of their big advantages. Thus, when a native SQL stored procedure is invoked from a remote client over a TCP/IP connection, it runs on the enclave SRB and thus picks up that same DDF zIIP redirect percentage. On the other hand, when a native SQL stored procedure is invoked locally on z/OS, it is executed on the TCB that PC'd in from CICS or batch, and that is not eligible for zIIP redirect.
Lots of other good stuff going on here - tomorrow is a DB2 for z/OS Special Interest Group as well as our IBM query panel.[Read More]
Last week in Athens I attended a presentation by Julian Stuhler of Triton Consulting
. I was of course aware of the support, but more from a DB2 internals point of view. It was great to get an external perspective on it from someone who has been working with it.
The key information is that spatial data can be points, lines, or polygons (including multi-part polygons). If you think about it, this is really powerful. One example that Julian used is that an address is a point, and a flood zone is a polygon. So now you can ask "is the house in a flood zone", which is "is this point inside the polygon"? Cool stuff! I can really imagine how this could be used by some situational applications to use data in DB2 alongside other data.
Complete documentation on the spatial features can be found in this book
We are often asked where to find the sample files for our text analytics tutorial. It's a small set of IBM quarterly reports. I've uploaded them here, just click on that to download them. Happy coding!!
I found this article
online today, which highlights the importance of enterprise search.
Company networks contain mountains of structured and unstructured data archived in numerous formats, some of them decades old and stored in secure servers.
IBM also is building a portfolio of enterprise search tools and services, under the OmniFind brand.
Of course you know that DB2 for z/OS data contains mountains of information! This is what our just-released text search support addresses for DB2 for z/OS data - character, binary, and XML. And it's built on OmniFind technology. With this support, you can do text search queries using the built-in CONTAINS() function. It's provided with DB2 9 for z/OS and the no-charge accessories suite.
Now, I know that this is just one piece of enterprise search. In fact, I joke with my colleagues that all of the work that we've put into this is "just an SQL statement". :-) But hey, it's an important piece - it can keep the DB2 for z/OS data where it is and "let the searches come to us".[Read More]
I think that most of you reading this work for large companies, and our U.S. large companies tend to have pretty active legal departments. One of the hot topics these days around litigation is the investigation of email to answer legal requirements for evidence. Yep, they're likely keeping all of your email, and are required to comply when asked to provide the relevant ones as part of a lawsuit. Getting that set right is a big deal.
Now, I'm not a lawyer. I do happen to come from a family of lawyers, but that's not really here nor there for this discussion. The group where I work in IBM's Information Mangement has just produced a pretty cool part of the eDiscovery puzzle. It's called eDiscovery Analyzer. As you can see in the announcement letter, it works in conjunction with other IBM products to analyze email content in a repository.
The cool part is what's under the hood here. Based on the open, unstructured information management architecture-based search and text analytics (known as UIMA to those who know and love it), this product processes the text inside as well as the associated information about all the emails. This processing in turn allows a legal email analyzer person to work with and filter based on extracted entities from the email, such as people and company names, and stuff like sender, recipient and date. Combine that with powerful free-text search and you really have some amazing capability to categorize, gather, flag... this really helps a legal staff when they're asked to provide exactly what's needed and no more.
Now... what if you had this kind of capability on other information besides legal email repositories in your enterprise. What would you do with it? What other business problems could this kind of technology solve for you?
I know that many of you are getting ready to attend the IOD conference
next week in Las Vegas. Alas, I will not be there, but that doesn't stop me from giving you advice about how to get the most out of a conference. After all, you are spending your time there, and your company's money, so you might as well make the most of it.
Earlier this month, I was fortunate to attend the Grace Hopper Celebration of Women in Computing in Keystone, Colorado. I was notified of my participation only two weeks before, and I wasn't aware of this conference at all. I made my reservations and printed out the conference agenda, which I reviewed on the plane on the way there.Most conferences give you access to the agenda ahead of time, and it's a good idea to print that out and have a look at it.
As for me, after one quick pass through it, I started by putting a "dot" next to anything I might be interested in, and a "star" next to anything that was a 'must see'. Then I started to notice some trends emerging. I came up with these goals for the conference, based on what was in the agenda:
- meet other IBM women (this was, after all, a Women in Computing conference, and IBM was sending me)
- text analytics technologies
- women in technology issues, including attracting more students
- social networking and collaboration
- cool stuff other than the above
To that end, I circled the name of any IBMer, and labeled each of the rest of what I noted as one of the above. Now, I realize, this approach might seem a bit, well, organized -- particularly for me. But it really worked, particularly to keep my attention and also keep a balance of different topics, as well as provide a tiebreaker when there were multiple sessions at the same time - I just look at the balance of the other sessions I've been attending.
Another thing you want to do while you're there is talk to other people about what they've seen, what they are going to see, etc. It's so easy to miss something or misinterpret something in an agenda.
If you keep a good balance between "use right now" and "good to know for the future" topics, you can help keep your brain from overflowing - that's always a danger at a multi-day conference![Read More]
Some of my IBM colleagues have created a pretty cool idea - that we, the community of folks with an interest in IBM's information management technology, should designate a day to connect virtually online. This means not just reading content, but actually taking a step further and participating.
I've always seen online social networking tools as extensions of what is done better in person, and a pretty good substitute for when it's just not practical to be in person. This goes back years and years, to online forums, prodigy (remember that?), etc. If you think of your participation online as much like an in-person event as you possibly can, you'll benefit the most possible.
Say, for example, if you attended a talk at a conference, and you gained a lot of useful knowledge from it, and then found your self face-to-face with the speaker right afterwards, you'd say "thanks, I learned a lot from your talk". And if you were sitting at lunch and someone said "Do you know anyone here who can help me with an SQL issue", you'd point them across the room to where your favorite SQL expert sat. Or, you'd do your best to answer the SQL issue yourself.
What we're thinking is that perhaps if we picked a day and asked everyone to speak up in just one small way, we might get some folks more comfortable with participating online, and everyone would benefit - make some contacts, get some questions answered, reconnect with someone you met in person, etc.
So, this Wednesday October 1, get out there to your favorite Information Management online sites and find a way to speak up. There are more ideas and links mentioned here.[Read More]
Announced today: New pricing options for DB2 for z/OS running new workloads! All you data center folks who lament to us that pricing for "other" databases can't be compared to DB2 for z/OS - rejoice!!
Announcing today, and already found here is this gem of a news item tidbit:
IBM is also announcing the immediate availability of DB2 for z/OS Value Unit Edition, which provides a new one-time-charge offering that enables the deployment of new application workloads. This offering strengthens the role of System z as a cornerstone for key business initiatives such as SOA, Data Warehousing, Business Intelligence and packaged applications such as SAP. DB2 for z/OS Value Unit Edition and IBM Information Server enable System z clients to further deliver trusted information for their dynamic warehousing requirements.
Just updated: Here is where you can find the gory details.
Is this cool or what? Doesn't this just remove the last and final objection that the application architects have for leaving DB2 for z/OS out of the running for those new applications?
Now, lest you think I am somehow reflecting a non-developer perspective, look, I have spent most of my efforts in DB2 for z/OS developing the kinds of new technologies designed to attract new workloads, and since even I have heard the pricing objection, isn't it perfectly fair for me to mention this in my DW space? And heck, since I am a developer, not a pricing person by any stretch of the imagination, if this has gotten my attention, you know it's big news!
Bring on those new workloads! And then come to us in development and tell us what you need to bring more work onto z, OK?
I found out about this one last week, and I think it sounds pretty interesting. IBM is holding a data management 'virtual conference' on February 25th. This one sounds like it's a lot more than just a webinar, as there is a show floor with virtual Expo Pedestals as well as speakers and the chance to chat with experts.
The screen shots that I saw look intriguing, although I admit that I don't yet have an exact sense of what it will be like to "be there". What I do know is that travel budgets are likely to range from tight to non-existent for everyone, but the need for knowledge and personal technical contacts is greater than ever.
Here's the agenda:
8:00 AM ET Show floor opens
11:00 Understanding the Foundations of The Information Agenda
12:00 Noon Chat with the experts
1:30 PM Integrated Data Management Revolution with Merv Adrian of Forrester Research
2:30 PM Chat with the experts
6:00 PM Show floor closes
If you haven't seen much on the Information Agenda, it's worth a look. It's all about trusted information and getting the right information out of the silos and working for the business. Over my years in the Information Management area of IBM, I've seen this message evolve and it continues to make more sense to me - how about you? The way I see it, if you're responsible for one silo of information, making that available for the business to benefit from should be one of your main objectives, well, that along with ensuring the security and reliability of that information.
Here's what the Expo Solution Pedestals will have:
Information AgendaLower Costs with IBM Data ServersOptim Data GrowthArchitect and Developer ProductivityDBA Efficiency and Autonomics
Along with the fact that there's no travel cost, the virtual conference is free to attend! Interested? Register now!
. And let me know how you liked it, did it work? It's easy to say that this can't fully replace a real in-person conference experience (after all there's no beer), but I like the idea of supplementing the real ones with other ways to stay connected and informed. This should be a step up from a webinar, as there seem to be many opportunities to interact real-time.
Now... how are they going to handle free swag from those pedestals? :-)[Read More]
We made a change in DB2 9 for z/OS in order to better package java(TM)code. We now ship DB2 java code such as that required for our XML schema registration and text search password encyrption, to be installed in an HFS/ZFS directory like /usr/lpp/db2/db2910_base
If your installation lets SMP/E default to that directory, then the same set up you use for Java stored procedures in DB2 for z/OS V8 will continue to work. But if you change that, then you need to set a new ENVAR in your JAVAENV dataset such as "DB2_BASE=/usr/lpp/db9a/db2910_base" so we can find our code. Otherwise, you'll see this error when the WLM-SPAS tries to start up: java.lang.NoClassDefFoundError: com.ibm.db2.dsnx9.JARLoader
I know, there are an awful lot of "moving parts" to setting up for running java stored routines. You need the DB2 universal java driver, the z/OS JVM, and JCL and a JAVAENV dataset. The stored procedures redbook has a good chapter on setup. It's a complex environment, but a very powerful one, too![Read More]
I heard an interesting story on the news last week, about how the individual states of the U.S. were graded on how they use information. The state I live in, California, got a C+. How can this be, with our advanced technologycenters in Silicon Vallley?
I found the article online here and found some interesting things, although nothing specific about California.
The article says:
When all is said and done, a state’s skillwith information is found at the intersectionof three distinct operations: the willingnessto share data, the capacity to generategood information, and the ability toget those who should use the data to do so.
Well, that sounds a lot like stuff that I have talked about when describing IBM's Information on Demand strategy. Is your organization good at doing this? I particularly noted the last point in the article, because some of the states complain that their legislators just aren't interested in using the data! Maybe we information professionals have to make that easy (and fun?) to do.
What about the highest-graded states? The article had this to say about one of them:
In Washington State, Governor ChristineGregoire held a series of town hallmeetings on the budget to communicate resultsto citizens and follow up on the budgetarypriorities she had previously establishedwith much citizen input. “We wantto give concrete information about whethera difference has been made or hasn’t"
Yep... this is what everyone wants to know. What did we say we'd do? Did it make a difference? In fact, I've been trying to get this type of information from my financial analyst for some time!
What about states that were graded worse than California?
Some state employees in Rhode Island arestill operating with typewriters—electric, ofcourse, but still a far cry from the ability toshare information in a database. NewHampshire has such weak data-sharing systemsthat it doesn’t know how much itspends each month—kind of like an averageJoe who’s lost his checkbook.
At the opposite end of the spectrum, there’s Wyoming. Itstransportation department has linked geographicinformation systems to financialsystems and now knows with exact specificityhow money is being spent, down to thecost of the salt used between each milemarker on the state’s snowy roads.
OK, well, perhaps that is an example of too much information! :-)[Read More]