Managing the data lifecycle
updated 6/16/ to fix minor typo.
Following up on Curt's blog about the new releases in June, let's take a deeper look at what is new in InfoSphere Data Architect V7.5.2.
Building on top of the privacy specifications for generating test data that was already built in the product from December 2008 you will now be able to pick from a predefined list of categories for specific data privacy information. It's probably best to explain this with an example. Let's say you have a credit card column that you want to mark as private by generating a random number where you maintain the first 4 digits of the card. Within InfoSphere Data Architect you can specify that you want to use the credit card masking policy, and IDA will be able to connect to Optim Test Data Management and Data Privacy solutions to get the appropriate masking method that should be used. Not only can you generate this in the design phase of your model you can now share this with Optim Development Studio so now when developing applications you have the ability to view what data is private and even look at the SQL that accesses the sensitive data.
Also new in IDA 7.5.2 is the capability to size storage requirements and estimate for data capacity and growth. This is often called volumetrics support and as per customer requests we have implemented this in the new release.
Finally, building on the fact that InfoSphere Data Architect is more than just a data modeling tool, we have leveraged all the different use cases that customers have implemented to improve on the different integration scenarios that we provide with IDA. We already know that Data Architect is built on top of the Rational Software Delivery platform (reminder, this product used to be called Rational Data Architect) and we continue to improve in those areas, but we have also enhanced integration scenarios related to Information Management as well. Since most of the Optim Solutions for Integrated Data Management are built on Eclipse you can utilize the sharing of connection information feature that was introduced in the June releases. Also new in IDA 7.5.2 is improved integration with IBM Industry Models and glossary information. All Industry Models and the newly added glossary information can now be managed in InfoSphere Data Architect.
The trial of this release will be available in a few weeks at the current trial download location. The announcement letter is here. Oh, by the way, the announcement letter also contains information about the updated Learning Services course for IDA that has been enhanced to cover more product capabilities. I always strongly recommend that new users get education, and this new and improved course can help you get what you need to get started.
-- Anson Kokkat
Greetings from Berlin! I’m very pleased today to tell you about several new announcements we are making in the portfolio previously known as Data Studio. With these new releases, we are taking great strides toward the vision of Integrated Data Management -- An integrated, modular environment to manage enterprise application data and optimize data-driven applications, from requirements to retirement across heterogeneous environments.
While delivering on an Integrated Data Management vision is a very broad value proposition and one that will involve all aspects of IBM Software Group, you’ll see with this release that we are adopting the Optim name as a rallying point for this technology emphasizing our focus on optimizing the value of your data assets by managing them across their lifecycle. This announcement represents another major step in delivering on our vision focusing both on extending heterogeneity and portfolio integration that provides the basis for cross-role, cross-lifecycle collaboration, efficiency, and alignment.
In future posts, we’ll take you through a couple of scenarios enabled by the new releases, but to get you started, here is a summary of the announcement and links to announcement letters and web pages:
Here’s a link to the announcement letter or visit the Web sites for Optim Development Studio , Optim pureQuery Runtime, Optim Database Administrator.
Many of you may be wondering about the no-charge capabilities. We heard loud and clear that our customers want a stand-alone download package for the no-charge capability. We are reverting to that packaging model with this release. More on that as it becomes available in the near future.
I’m asking some of my technical leaders and product managers to go into more detail on these announcements in future blog posts. In the meantime, I must go back to the conference. Lots to talk about…
OK - I'm finally contributing to our Data Studio blog. Yes, I should have done this a long time ago and yes, now that I've posted this, I've pretty much committed myself to regularly contributing. I'm the product manager for the Data Studio portfolio and oversee the direction of the products within Data Studio.
So my first blog will be about a topic near and dear to my heart, "A Day in the Life of a DBA". I used to be a DBA prior to working for software companies, and I remember being pulled in all different directions during the day. From design sessions with development on new applications, to restoring databases when things go wrong, a DBA's life is never boring. So how does this relate to Data Studio?
We've been working hard on providing functionality for the DBA. Most people that think Data Studio is only for the developer, which may have been true a year ago, but that has changed with the recent 2.1 release (Dec. '08), and the upcoming release will offer even more functionality for the DBA.
With Data Studio and Data Studio Administrator did you know you can:
-- Deb Jenson
I wanted to let you know about a new two-part article series written by Alice Ma and me which is intended to help you use DB2 Performance Expert for Linux, UNIX, and Windows to your best advantage.
I hope that the concepts described in the articles will help you to more effectively use DB2 Performance Expert. Note that although there is not a downloadable trial version of Performance Expert, if you are interested you can contact your IBM Sales Representative to get it.
-- Ute Baumbach
I just got back from IDUG in Denver and wanted to dash off a quick blog about it. I sat in on several sessions where Data Studio and pureQuery were being presented (some by our team, some by consultants, some as BOF sessions). We really couldn't have asked for things to go better:
See you at both conferences.
-- Curt Cotner
Hi, this is my first time blogging here. I'm an architect in the Data Studio development team, and I work on integrations, heterogeneous database access, and more. I wanted to use this opportunity to tell you about some work I did with pureQuery and Enterprise Generation Language.
Enterprise Generation Language (EGL) is a modern programming language specifically designed to help the business-oriented developers quickly write full-function applications and services based on Java and modern Web technologies. Business-oriented developers write their business logic in EGL source code using the powerful development facilities of Rational Business Developer Extension, Rational Developer for System z with EGL, or Rational Developer for i for SOA Construction. From there, the tools then generate Java or COBOL code, along with all the runtime artifacts you need to deploy the application to the desired execution platform.
Data Access is one of the key components of EGL. You can access your database data using EGL SQL Records which provides a very high level of abstraction and allows you access to the data using simple verbs or you can write your own data access logic. Below are simple examples showing both scenarios.
Figure 1. SQL Records
Figure 2. A basic data access program
If you are a regular reader of this forum, you probably already know that pureQuery is IBM's, high-performance data access platform focused on simplifying, developing, securing, managing, and optimizing applications that access data. You may have read about the benefits of using pureQuery client optimization with Hibernate, JPA, and even .NET applications. You can also use pureQuery technology with the Java code generated from EGL to
Kathy Zeidenstein and I have put together a tutorial on Rational Cafe that shows how this integration works and how the technologies can be used by EGL customers writing applications with DB2 data servers.
Enjoy, and let me know if you have any questions.
Hi, I haven't written a blog entry for a couple of months, so if you're new to our blog, I'm the team lead and architect for Data Studio Administrator.
One of the features in the 2.1 release is a quick and easy way to copy object(s) from one database to another. You can read all about it in this developerWorks article that members of my team recently published. But let me summarize here.
Suppose I've just made some changes to my development database and application, and I've completed my testing. Now I want to copy the database changes to a test environment for further validation. Let's see how I can do that using Data Studio Administrator 2.1 with Fix Pack 1.
I connect to my source database in the Data Source Explorer (what used to be called the Database Explorer). I like to drill down using the flat folder view (new in 2.1, which presents objects in folders by type) versus the hierarchical mode, so I toggle to that view using the icon in the Data Source Explorer tool bar. I expand the database and select the Tables folder. In the Object List View I copy the tables that I've changed and want to move to the test system.
Now I'm ready to paste these objects into my target.
To paste the objects into a new database, I connect to the target database, expand the database and select the Schemas folder. I click on the schema I'd like these objects pasted into, right click and choose paste.
When I do this, I have the option to also paste dependent objects and data. The wizard helps me to create a change management script to implement these changes. Optionally I can customize the change further. Finally I can deploy them along with my application changes.
Copy and paste is a quick and convenient way to move objects between environments like development and test.
We talk about the benefits of static SQL execution as a benefit of using Data Studio Developer and pureQuery. And those benefits are great. But if you're an Informix developer or DBA, you may still be wondering if Data Studio Developer can help you improve application performance, since IDS doesn't currently support DB2's static execution model.
When we announced Data Studio Developer 2.1, we added many features that benefit Informix database developers and DBAs. Guy Bowerman covered some of the key 'base' features for both developers and DBAs, including support for UPDATE STATISTICS, improved support for triggers and tablefragmentation. But if you're a Java database developer or DBA who cares about performance, take a look at pureQuery capabilities for Informix as well, including:
I was looking at Scott Ambler’s surveys on IT project success rate. It is very interesting how project success as seen through Scott’s surveys present a more hopeful picture for project success than from the Standish Group’s Chaos Report, which in its 2006 refresh reported a 35% success rate and a 46% “challenged” rate. (Nice blog entry summarizing a variety of research on the topic in Dan Galorath’s blog and 2006 Standish numbers from an SD Times article.) Standish defined success as “on time, on budget, meeting the spec”, while challenged means they had cost or time overruns or didn’t fully meet the user’s needs. But I digress…
Scott’s data indicates that projects that use evolutionary development methodologies, e.g. Agile or Rational Unified Process, fare better than those using traditional waterfall or ad-hoc processes. That’s not surprising given the emphasis on tight collaboration among stakeholders and continuous evolution and validation. Really, it’s pretty intuitive. So I was thinking about key characteristics of iterative methodologies and how they relate to database and data access development. (I know, Scott has already thought about this too.
See his Agile Data site. And Rafael did a Webcast on it earlier in the year.) But more specifically, I wanted to look at how our Data Studio portfolio supports evolutionary development methodologies. Yes, there’s more to do, but I think what we offer goes a long way towards accelerating solution delivery with high quality results. Vijay and I are going to do a Webcast on this April 28th titled Accelerating Solution Delivery for Data-Driven Applications. Hope you’ll join us.
In some ways, this is also the companion Webcast to Rafael’s Performance Optimization webcast. In his blog, he talked about how from a lifecycle perspective performance optimization can broken down into doing it right the first time or fixing it after that fact. His Webcast focused on the latter and this one on the former.
What are your stories about evolutionary methodologies and database development? Have you used Data Studio software in this context?
I’ve been working on DBA solutions lately -- in particular, performance optimization topics. You’ve heard some cool things about our offerings already if you follow the Data Studio blog, such as Alice’s blog on performance diagnostics or Jeff’s blog on pureQuery from a systems programmer perspective. One of the things I want to do is bring our different offerings together in a more cohesive context.
Performance optimization can be broken down into 2 big categories:
Then you have to fix the problem. Of course, there’s always adding more resources – more CPU, more memory, more storage. But if that’s not an option, for database performance problems you’re going to want to:
-- Rafael Coss
Hello everyone - first let me introduce myself. I joined the Data Studio development team more than 2 years ago and since then I've been working on the Data Web Services capability. During that time I gained a lot of knowledge about connecting database assets to the Web, Java and J2EE development as well as XML processing ... until one day a strange blue box named "DataPower" crossed my way. This blog is about the results of this encounter ...
Information as a Service
A lot has been said about making your data accessible to an SOA environment, the Web, the Cloud, and so on... and IBM, as well as other vendors, provides a wide variety of software tools and frameworks to make all that possible.
But let's look at a common case where data is represented as XML and stored in a relational database:
Very often data is represented as XML messages which flow over the wire. The XML may contain data which comes from, or is going to be written to, a database. It may also contain data and metadata to perform remote procedure calls (RPC) as we see with SOAP. There might be complex business logic at the remote server associated with a message flow - for example in form of a stored procedure in DB2 - but we may also just have a simple mapping from XML into relational structures, or we may just store the XML "as-is" in the database - thanks to the pureXML capabilities in DB2.
The bottom line is that there are many cases where you need nothing else but a simple mapping definition from XML to an SQL statement or a stored procedure call, and another mapping definition to map result sets and output parameters to XML structures.
Make it fast and secure with WebSphere DataPower Appliance
And there are again many approaches on how such a mapping and network wiring can be done, but today I want to introduce you to a fairly new way of doing that by using the IBM WebSphere DataPower Integration Appliance XI50. Because DataPower has - among many other features - the capability of connecting to databases, which includes superior support for DB2 on all platforms.
DataPower allows you to define mappings from XML to database calls via XSL. Furthermore, you can define message flows, network protocol endpoints, security, quality of service properties, WS-* features and much, much more via the award-winning Web GUI interface. And all that without writing a line of code or even code generation! Everything is well contained and controlled inside the DataPower appliance without the need of changing your database server, your legacy data or your application/business logic contained in stored procedures.
Scenario 1: Enrich messages with database content
In this case, messages passing through a DataPower appliance can be enriched with data stored in a database. It's also possible to write parts of or the complete message into the database - for example using the database as an audit log.
Scenario 2: DataPower as an SOA Gateway
Here we declare the database as the target for the message flow. The message data can be stored into or retrieved from the database, or more complex business logic can be invoked in form of a stored procedure.
Scenario 3: High-performance batch INSERT with DB2
DataPower supports the DB2 batch INSERT feature which can speed up the insertion of multiple records into the database. This can be leveraged for XML shredding. You can easily break up a large XML document into multiple records (rows) by defining the shredding rules in your XSL script. The batch INSERT takes care of writing the individual records quickly into the database.
Find out more in this developerWorks article on "Using WebSphere DataPower SOA Appliances to enable the Information as a Service pattern".
The native DataPower XML processing stack provides you wire-speed XML parsing, validation and transformation. DataPower supports a wide variety of different network protocols which allows connectivity to many different back-end and front-end systems - but at the same time you can be sure that your data is kept safe and reliable with the many security capabilities provided with the appliance.
Last but not least, a well designed management API as well as connectivity to registries like the WebSphere Service Registry and Repository (WSRR) allows easy life-cycle management of your artifacts.
Make it even easier by using IBM Data Studio Developer
At this point you may say - "All that sounds great, but why in the world is he posting this in the Data Studio blog?".
Well, you may already be familiar with the Data Web Services feature in Data Studio Developer (if not, see links below) where you can easily expose stored procedure calls and SQL statements as Web service operations by creating the appropriate J2EE runtime artifacts. But did you know that you can now also create service runtime artifacts for DataPower? The Data Studio Developer tooling generates all artifacts for a Data Web service - including the WSDL file and - in case of DataPower - the XSL scripts which perform the mapping from the XML message into database calls. There is no need to code the XSL by hand. All that's left do to is the deployment of the artifacts on the DataPower appliance.
I wrote a tutorial on this topic to show you how easy it is to use Data Studio Developer to create the Data Web service artifacts and how to configure DataPower to host the generated services. Check it out.
To get some more ideas on using DataPower and DB2 pureXML I recommend the following two articles:
WebSphere DataPower SOA Appliances on DeveloperWorks
Data Web services in Data Studio Developer
After 5 years of use on my existing cell phone I decided to upgrade to one of the more feature-rich models that was available on the market. Once I purchased my phone, I really didn't know where to get started. I had all this data from my old cell phone that I wanted to transfer to my new one and start using right away. I was happy to find a seamless way to migrate the data over and even more surprised at how simple and quick it was. Now that the data was migrated over, I learned about new features that really helped to make my life more productive and started wondering how I did things before.
The new ERwin to InfoSphere Data Architect (IDA) migration guide gives customers a cookbook approach to transferring assets from ERwin to IDA and start being productive right away. ERwin has been around for about 20 years so it's difficult to suddenly pick up a new tool and expect things to be the same. This guide helps by mapping terminology and concepts between the products and by offering best practices guidelines, including:
-- Anson Kokkat
P.S. IDA doesn't come with GPS or a 5 MP camera, but it does come with tools that would turn a data architect's head.[Read More]
I totally love performance, and I like the whole process of trying to find the issues. There is nothing more cool than being the hero-of-the-hour by finding the extreme I/O usage, the I/O hot spots, the file system contention, the logical control unit bottlenecks, the horrible buffer pool management, the poor SQL, or any combination of the above.
But what makes it so interesting is the tooling surrounding performance. To be more specific, the fact that I can now tune DB2 for LUW in the fashion as I would tune DB2 for z/OS. Heresy, you say? Please hear me out.
All through the 1990's and on through today, I've been using Tivoli OMEGAMON XE for tuning DB2 for z/OS. This tool is great for finding just about any issue or bottleneck, including looking at historical data. In 2003, I had to support DB2 for LUW, and I felt like I was stepping back in time. My biggest complaint was the fact that I had to set up scripts to pull down activity. And finding historical information on DB2 for LUW activity? Forget about it.
But that was then.
What we have now for DB2 for LUW is, quite simply, amazing. DB2 Performance Expert is a monitoring tool any DB2 person would just love. I have my main console, which is strikingly similar to the OMEGAMON classic view. There is the drill down capability in DB2 Performance Expert where I look for troubleshooting issues. But what do I really like? The fact that I can now look at historical information without a script or agent running on the monitored server!
Doing the Monday morning tasks - you on-call DBAs know what I'm talking about here - are made so much easier with DB2 Performance Expert. I can look at what happened over the weekend so that neither I nor my manager get blindsided by applications or by my end users. In addition, I can find problems in the here-and-now without physically issuing snapshots either at the command line processor or within a script. I can find the long running SQL and easily find the tables it's going after. By knowing the tables, I can go to the next level - is this a one-off SQL situation or is this a table that needs an index, or is this a table that needs to be isolated either in a bufferpool or using file system placement? DB2 for LUW will not be the product that you "poke a stick" at for tuning.
Will DB2 Performance Expert give you the desire to hang on to your turn in the on-call rotation for another week? I can't answer that. But finding the issues simply and easily with this tool will make you the hero-of-the-hour.
The fix packs for products in the Data Studio family 2.1 have arrived!
The fix packs includes enhancements and fixes to the Version 2.1 release of IBM Data Studio Developer, IBM Data Studio Administrator, IBM Data Studio pureQuery Runtime and of InfoSphere Data Architect 7.5.1. These fix packs are intended to fix problems you may have experienced in the 2.1 release. For additional information or detail on included fixes please check out these links.
Data Studio Administrator Fix Pack 1
InfoSphere Data Architect Fix Pack 1
Data Studio Developer Fix Pack 1
For those who already have these products installed on Windows, you will use IBM Installation Manager to apply the fix pack:
For those who don't have the products installed, go ahead and download the trial versions of the products from the following links:
Data Studio Developer
Data Studio Administrator
InfoSphere Data Architect
(By they way, you can find all these links together on the Data Studio Community Space.)
Now you are ready to go!
We also have fix packs for DB2 Performance Expert and DB2 Performance Expert Insight Feature:
DB2 Performance Expert Version 3.2 Fix Pack 1
DB2 Performance Expert Extended Insight Feature Version 3.2 Fix Pack 1
-- Tina Chen
As you may recall, when we announced Data Studio pureQuery Runtime 2.1 for LUW, one of the new features in the release was ability to use pureQuery with .NET applications. This support is available for all of the DB2 servers; however, I want to focus a bit on this from a z/OS perspective, since it is primarily these customers who let us know that they heard and liked the pureQuery for Java story we were telling, but that they needed something like this for .NET as well. They wanted the advantages of static SQL - for security, manageability, and performance reasons.
So we did add that support for .NET in the 2.1 release of the Data Studio pureQuery Runtime and in the latest IBM ADO .NET provider. Although it doesn't have all the rich tools support that Java does, it provides many of the key benefits that Java shops can get - static SQL performance and consistency, static SQL authorization model, and the ability to create uniquely named packages that can help DBAs and system programmers isolate performance problems to a particular application and particular SQL statements. And, since it's using client optimization, that means your applications can get these benefits without having to change source code.
To validate the performance benefit, I'm very happy to announce that we've published the results of our performance study (using the IRWW benchmark) of the pureQuery support for .NET. I don't want to spoil the surprise, but the numbers are very impressive with huge increases in throughput and dramatic reductions in CPU per transaction.
Also, be sure to see this developerWorks tutorial. It's a good step-by-step guide to the process of enabling .NET applications to use pureQuery.
It's been a while since I've blogged. I've been spending a lot of time talking to customers about what they want and need from our integrated data management portfolio, and now the whole team is working on some great new capabilities and offerings to help address some of the key pain points I've been hearing about. More on that later.
Right now, I just wanted to draw your attention to our recent announcement of the pureQuery Runtime 2.1 on z/OS. This version of the product has been available on LUW since December and we are happy to announce its availability on z/OS for those shops who run their apps natively on z/OS. There are some excellent capabilities in the 2.1 Data Studio pureQuery Runtime and Data Studio Developer releases -- you can read this What's New article for a good overview, and these videos also show you many of these enhancements through Data Studio Developer. And don't forget Jeff Sullivan's blog entry which gives a lot of good reasons from a z/OS system programmer perspective why he likes pureQuery.
Use pureQuery Runtime for z/OS with stand-alone applications, applications deployed on WebSphere Application Server or other application servers, or with DB2 stored procedures. Data Studio pureQuery runtime supports both type 2 database drivers and type 4 database drivers for DB2 for z/OS V8 and V9.
Hi, all. It's been a while since I've written. I've been really, really busy doing pureQuery POCs, training our technical sales guys and whatnot. My job is to focus on data access development through many different architectures and technologies. Last year I was spending a lot of my time on SOA -- and how data plays such a key role in it. I just noticed that there's an upcoming webcast that talks about SOA and information management that focuses on strategies to return business value from an SOA project.
As more and more companies go from exploring and evaluating SOA projects to actual implementation, I think it's really important to learn from these experiences so that you can be more likely to deliver on a successful project. Check out the webcast - it's being sponsored by the System z team and includes insights from David Linthicum, an independent consultant who has lots of experience in this area.
While SOA as a whole is a much larger topic, I like to think that during that "tiny phase" of implementation, one can use all the help they can get to develop the most efficient data access. Web Services support in Data Studio is one of the many components that can help deliver on the SOA promise. Take a look at this video that shows how tooling has become sophisticated enough to jumpstart some of this implementation.
This tooling makes it easy for you to re-use existing DML operations that you know are efficient and well-tested (including COBOL-based stored procedures).
If you're interested in doing some exploration with the Data Web Services technologies yourself, I recommend this IBM Redpaper which is a condensed version of the DB2 for z/OS V9 SOA Redbook. It's z/OS focused,uses WebSphere Community Edition for the app server (however, this can be WebsphereApplication Server if needed), and Data Studio Developer for tooling. Try it out and let me know what you think.
--- Vijay Bommireddipalli
Happy Friday the 13th, everyone! Hope your day was lucky and your weekend is good.
Today I want to dig down a little bit on how data modeling can help DBAs be better at their jobs and become more valuable to the organization. You have probably heard about the key benefits of the IBM Integrated Data Management portfolio, including improving productivity, saving time, reducing costs, and improving collaboration among roles. I’d like to propose that as much as any data development or performance monitoring tool, using InfoSphere Data Architect for data design and discovery can provide tangible (and not so tangible) benefits to DBAs and the organization both in the short term (rapid response to changes) and long term (improved skills, better quality data models, etc.).
Let’s look at skills and productivity in a heterogeneous environment first. Data Architect is a heterogeneous tool. If your shop is like many I see, you have to deal with multiple databases, such as Oracle and DB2 for z/OS. Or DB2 and SQL Server. Or Informix Dynamic Server and DB2 and MySQL. Whatever combination you may have, what you really want is a single tool that works with all these different data servers.
Having a single tool that generates the DDL from several different database vendors means you don't need to be an expert in all of them. It allows the DBAs to become more skilled across more products and thus allows DBAs to do more with less, and as their skills grow they become more productive and valuable to the enterprise.
Even if you don't have multiple database systems to deal with, InfoSphere Data Architect can help DBAs be more responsive to ‘agile’ development scenarios (or any scenario in which requirements come in late – all of them?). Designing databases from scratch is always difficult, but having a data modeling tool to can really ease the pain. So, if a developer says "I just remembered, we forgot to take the modules from Sally's group into account," it’s pretty easy to manage that kind of change with InfoSphere Data Architect. My recommendation is to manage all design changes through the logical data model, and then transform as appropriate to the physical model required for the target database, which, by the way, might very well be different vendors between development, test and production. You don’t have to follow this approach, but it is a best practice.
Another scenario: Let's say that your company has recently acquired a company and it’s your job to integrate their database into the main corporate database. And by the way, they have no or little documentation on what is in their databases. No problem. Use the Data Source Explorer in InfoSphere Data Architect to reverse engineer the existing database to get a nice diagram of the physical data model to see what they have already. Then you can compare what they have with what you have (for example, do we really need another customer table? Or can we just migrate their data to ours by maybe adding or modifying a column?). If you need to make changes to accommodate their data model, you can generate the necessary DDL to do so and can even deploy directly from the tool if you like.
Speaking of inheriting database, another DBA might pass you a database model that has cryptic table names or cryptic schema names. Just looking at the table name you can’t tell what it means and it doesn’t match anything else you have seen before. InfoSphere Data Architect can help you "decipher" the cryptic database model against a glossary that defines your enterprise standards. The glossary is a way for you to enforce naming conventions or decipher a table you created in the past. If there are any discrepancies or a newly created table doesn’t match the glossary, the tool will help report these back to you so you can fix them.
The nice thing about Data Architect that is different from other modeling tools is that it does include some cool capabilities in the Data Source Explorer such as being able to sample live data. And because it integrates seamlessly with products like Rational Software Architect for WebSphere Software, it’s that much easier to collaborate with the application architects and exchange models with them.
Anyway, this is a lot of words, I know. I recommend that you check out this great new 'how to' demo on developerWorks that goes through some basic scenarios so you can see the tool in action. If you want more details on the capabilities, go to the web site, post a question using the comment link below, or send me a mail at ansonk at ca.ibm.com.
-- Anson Kokkat
Presentations of software products can sometimes be downright hard.
I recently participated in a customer lunch-and-learn seminar called "Recession Busting Data Management Software". The seminar was absolutely brilliant. We talk to potential customers on how to save money, right now in this down economy, with IBM software. The presentations were 20-minute brief talks on IBM Optim Data Growth Solution and Test Data Management Solution, Data Studio Administrator, Data Studio pureQuery Runtime, and DB2 Storage Optimization Feature.
These are all particularly well suited for saving money and are really cool tools. My part was on Data Studio and pureQuery. It started nicely by describing pureQuery Runtime, which also included my reasons for why it is such a money saver. The ability to have WLM schedule the DDF work into other service classes with statically-bound "named" packages that are easily managed with Data Studio Developer resonates with DB2 on z/OS customers.
All went well until I ran into slides that were not coherent in the aggregate. Simply put, it was not a smooth flow.
But, quoting a very famous radio legend, here is the "rest of the story".
I had created the presentation from several other presentations and combined the slides very quickly while in a hotel room in Austin. Then, I proceeded to give a different presentation later that afternoon at SHARE in Austin. But that wasn't the end of it. The next morning, I went to the airport at 8 AM thinking my flight was at 10 AM but in reality it was boarding as I arrived at baggage check-in. Needless to say, I did not arrive home until 7 PM instead of 11 AM. In my original plan, the day I lost waiting for the next flight was the day I was going to get to know my slides and practice them.
In retrospect, there is a bigger answer to the poorly presented presentation and missing the flight - have a plan B but ALWAYS be prepared.
Until next time,
IBM_Optim 27000269HS Tags:  design infosphere metadata model glossary kokkat rda ida 5,576 Views
In my last blog I talked about the tools associated with InfoSphere Foundation Tools, including my product, InfoSphere Data Architect. However I wanted to really show you that most of what I was talking about has substance, and that there is true integration among the tools – it’s not just marketing!
In Denis Vasconcelos's latest article, Understanding leads to Trust: Sharing a Common Vocabulary across InfoSphere Foundation tools, he has really hit home the message about how a common understanding of business terms can help improve communication and enforce standards across IT and business organizations. His article shows you how to import your existing business concepts into a business glossary (InfoSphere Business Glossary with InfoSphere Metadata Workbench) and then use that glossary within InfoSphere Data Architect to do such things as enforce naming standards in data models, which of course will mean that applications built on the resulting database will also be using correct terms that are meaningful to the business.
I like how the article shows how all of these products are interconnected, and how the various technologies have been designed to make sure that you are doing the most with your metadata.
Read the article and let me know your thoughts... I am especially interested to know if this set of tools meets your objective of managing metadata effectively. If there is something missing, let me know. I really think we have a unique offering with these set of tools, and something that really stands out from the rest of the crowd.
Thanks to those of you who responded to my previous blog asking for feedback on using Java on z/OS for your database tools. It was really helpful.
I need your help again.
We are having internal discussions about plans for the Data Studio administration console, a no-charge download that includes both a replication dashboard and high-level monitoring of database health and availability. It is the database health and availability monitor that I need your feedback on. I need to hear from both DB2 for LUW and DB2 for z/OS users, so don’t be shy!
If you aren’t familiar with the health and availability monitor, there's a good tutorial here. Just as a reminder, health and availability monitoring enables you to easily assess the high-level health of DB2 for LUW and DB2 for z/OS systems. It includes a health overview, which lets you look over a landscape of database servers, and a dashboard that lets you focus on a single server. In addition, there is a time slider that lets you view changes over time in both the dashboard and an alert history. Health and availability monitoring also includes quick analysis and suggests possible resolutions for many database server conditions and scenarios.
The intended purpose of the health and availability monitor is to serve as a first level, “at a glance” type monitor. It’s not intended to provide the deep diagnostics that a monitor like DB2 Performance Expert or Tivoli OMEGAMON XE Performance Expert for DB2 provides, but it can allow you to quickly glance over your systems and immediately spot whether one of your databases needs attention. Our thought that was customers would most likely want to use this in their test environments, as they may not want to invest heavily in monitoring non-production servers.
OK, so here are my questions (these aren’t formal survey-style questions, so feel free to improvise):
Please take a few minutes to dash off an email with as much information as you can. Don’t forget to tell us a bit about your environment and if and how that influences your answers. I really appreciate your help.
-- Bryan Smith
IBM_Optim 27000269HS Tags:  correlation purequery sullivan wlm z/os workload_manager 3 Comments 7,776 Views
I've been around the block enough times to see that the bricks look the same. I have seen the same performance issues repeated at untold companies with the biggest issue being identification and performance tuning for Java applications.
To be specific, it is the Java communications with DB2 for z/OS that has profoundly perplexed me and my z/OS colleagues. Java, from my perspective, has sometimes been a performance nightmare on z/OS systems. By using the Distributed Data Facility (DDF) for communication, we use a one-size-fits-all approach by using a single Workload Manager service policy. This isn’t because we want to, but because we have to. To make matters worse, these connections come in sufficiently generic that there is NO WAY to figure out what Java application created this thread or multiple threads.
I can't tell you the number of times I would hear a Java programmer wander into my cube and tell me that they have a problem but they don't know where it is. "..Jeff, can you look at...” in which case I'd say, "STOP! Look at what? A thread? I have hundreds of them. Which one?". And you wonder why DBAs get a reputation as being difficult to work with! But that is another story...
Recently, I joined the IBM Data Studio Enablement team. One of our charters is to articulate the value of Data Studio, pureQuery, DB2 Optimization Expert, and a few other tools. OK, I am an old "green screen" guy. A teammate accused me of not embracing our products to which I answered that I am reluctant to jump on any bandwagon unless I see a true value - Not just as an IBMer, but as a z/OS DB2 System Programmer. How can I "sell" a product if I am not "sold" on the value myself?
The tooling in Data Studio Developer allows the programmer to quickly develop the Java structures necessary to access DB2 for z/OS. In addition, Java application developers can now easily bind a package, which accesses DB2 for z/OS, used by the Java application. Finally!! We’ve had this DB2/application access technique forever with COBOL using CICS. Now we have it with Java, too.
As a result of having a statically-bound package for Java code with a type 4 driver, I can now set up a Workload Manager service policy for Java DB2 calls as it passes through subsystem type DDF using a PK rule. To go one step further, WLM could be set up with a JAVAHIGH and a JAVALOW service classes. Then, these classes can be prioritized, have a time goal applied for period 1 and a velocity goal applied for period 2. Then, using the naming rule PK for package name, these service classes can be referenced to specific service classes.
Not too many products really flip my switch, but this combination of Data Studio Developer and Data Studio pureQuery Runtime is one of them. Dynamic SQL tuning with Java and DB2 for z/OS has been my nemesis for a very long time. pureQuery gets Java code as close to "well-tuned" as I have ever seen. I would recommend Data Studio Developer and pureQuery to anyone struggling with an out-of-control distributed environment going against DB2 for z/OS as the DB-tier.
-- Jeff Sullivan
You might want to check out a new article Tina Chen wrote about What's New in Data Studio Administrator 2.1 The article highlights how many database administration functions have been added to Data Studio. For those new to Eclipse, I think this article will help you cut through and start tackling real database tasks. If you're a Data Studio Administrator 1.2 or DB2 Change Management user, you'll notice the radical UI improvements, making the product easier to learn and easier to use. Check it out.
I just got off the phone with a DBA who just started to use DB2 Performance Expert (PE) a couple of months ago. This customer, whom I will call “Milo,” after my cat, called me to tell they were able to finally diagnose a pesky performance problem that had been hiding for all these months.
Let me back up to give you some history. When I first talked to Milo, they were having performance problems on all their environments. Before using PE, they usually ran a series of scripts to diagnose performance problems. These scripts would help them collect performance metrics for each partition. To add to the complexity they had a bunch of different environments they were monitoring. This means that Milo would end up with reports all over the place --in different systems and in different directory structures. Then when he downloaded them to his workstation he would need to print them out.
Milo's desk was covered with printouts. He had stacks of printouts on the left, stacks of printouts on the right -- they were even stacked on his bookcase. Milo's baseball bobble heads were holding up printouts because Milo ran out of room in his cube. Each area of his cube represented a different environment. Milo even made jokes that he was drowning in his own printouts, and he felt like the performance reports were multiplying if you left them alone. We both got a good chuckle out of that.
Fast forward to life after getting PE. With PE, Milo was easily able to pinpoint the performance problem in their large warehousing environments. PE was able to monitor all their different environments, which helped them diagnose a ton of problems because they were able to see all the performance metrics and how they correlated. For example, PE allowed them to view each partition and compare the partitions.
However, there was one sticky problem in their large warehouse environment (environment A) which they were unable to diagnose. But since they began using PE, that slowdown didn’t show up for months. Then one day, that pesky performance problem finally revealed itself again. Apparently one of the DBAs ran the old performance scripts on environment A, at which time the problem reappeared so the team could finally work on isolating the problem.
Milo told me laughing We couldn't figure it out….. the slowdown happened almost like clockwork. We had our scripts scheduled to capture the performance data and somehow it just happened. We used PE to diagnose the problem, only to discover it was our own poorly written performance scripts that caused the problem. You see, environment A was built on smaller UNIX boxes, with less memory and other resources. When they ran the performance scripts they caused they system to run out of memory, thus impacting their DB2 system. Since PE was able to show them what else is using resources on the system outside of DB2 they were able to see the problem immediately. Their scripts didn't check OS resources, but PE does.
The person who wrote the original scripts has been gone for a long time. The scripts ran fine and never caused performance problems on the other larger environments, so there was never any reason to examine them. Ooops :-)
Cheers - Alice Ma
Edited on 2/6/2009 to correct the query and to acknowledge that the existence of Java stored procedures may not necessarily mean you have the SDK. Thanks for keeping me honest, folks.
We are investigating implementing some server-side functions in our data tools that would run in a Java runtime on z/OS, and I would appreciate getting your feedback to help us with this planning work.
SELECT SCHEMA, NAME, CREATEDBY, LANGUAGE, ROUTINETYPE, SPECIFICNAME, WLM_ENVIRONMENT
WHERE LANGUAGE = 'JAVA';
You can send your feedback directly to me at bfsmith at us.ibm.com.
Thanks a lot!
--- Bryan Smith
Did you know….. you can reduce the CPU consumption of RUNSTATS by using the Statistics Advisor of InfoSphere Optim Query Workload Tuner for DB2 for z/OS
IBM_Optim 27000269HS Tags:  optimization_expert workload statistics willoughby runstats query_tuner cpu 1 Comment 6,887 Views
…and at the same time maintain or improve query performance.
Previously I have blogged about how DB2 Optimization Expert can help developers produce better performing queries early on during development. So with good query tuning it should be possible to drive down CPU costs for database applications.
However, there’s another way that DB2 Optimization Expert can help drive down CPU costs and it has to do with the general maintenance that DBAs are tasked to perform.
A common practice in many DB2 for z/OS shops is to execute RUNSTATS with the TABLESPACE <database-name.table-space-name> TABLE ALL option. If you have ever attended one of Bryan Smith’s presentations on utilities, you have heard him say using this option is expensive and wastes CPU resources.
Why is this option expensive? The TABLE ALL option gathers statistics on all columns of the table(s) in the named table space. The CPU resources required increases as the table size increases and is also dependent on the number of columns defined. Are all of the column statistics really needed? Probably not, because if the column is not referenced in the WHERE clause of a query, then unneeded statistics have been gathered and unnecessary CPU resources have been consumed. Also the TABLE ALL option does not gather COLGROUP or histogram statistics, which might improve filter factor estimates and improve query performance.
The DB2 for z/OS V9.1 Utilities Guide has a paragraph in the Improving RUNSTATS performance section:
“Run RUNSTATS on only the columns or column groups that might be used as search conditions in a WHERE clause of queries. Use the COLGROUP option to identify the column groups. Collecting additional statistics on groups or columns that are used as predicates improves the accuracy of the filter factor estimate and leads to improved query performance. Collecting statistics on all columns of a table is costly and might not be necessary.”
Easier said than done. How does one perform the analysis required to only gather the statistics that are truly required to maintain query performance and reduce the CPU requirements of RUNSTATS? Without a tool, it’s a very manually intensive process and can be even more of a challenge when working with dynamic SQL. Just to give you a feel for this effort, here are the steps you might need to do:
Fortunately, the Statistics Advisor in DB2 Optimization Expert for z/OS can make this analysis less daunting – a lot less daunting. Statistics Advisor will analyze a single statement or group of statements (aka workload) and provide a set of recommended RUNSTATS statements.
To illustrate how easy it is to gather the necessary statistics in OE, I captured a workload from the DB2 catalog for a specific collection. I was presented with 852 queries and then invoked Workload Statistics Advisor on this set of queries to receive the recommended RUNSTATS statements for objects in the workload. The process was completed in less than thirty minutes. For dynamic SQL, I would follow basically the same process, but the workload would have been captured from the dynamic statement cache, or from the profile monitor for DB2 for z/OS V9.1 subsystems.
To validate the CPU usage assertions, I actually executed RUNSTATS twice on one of the table spaces used in the above workload. The first execution was with the commonly used TABLE ALL option:
RUNSTATS TABLESPACE DBASE1.TSPACE1
SHRLEVEL CHANGE REPORT YES
The second execution used the statement recommended by OE’s Statistics Advisor:
RUNSTATS TABLESPACE DBASE1.TSPACE1
INDEX(QUAL1.TB1X3 HISTOGRAM NUMCOLS 1 NUMQUANTILES 20,
SHRLEVEL CHANGE REPORT YES
There was a 28% reduction in CPU time between the two runs. Not only has the CPU consumption been reduced, but additional filter factor statistics have been gathered that should improve query performance.
One topic I have not covered is SAMPLING. It is a technique that can be used to further reduce RUNSTATS CPU consumption by limiting the number of rows evaluated. Statistics Advisor supports SAMPLING and its use is controlled via user-managed preferences.
So in summary, DB2 Optimization Expert’s Statistics Advisors can help DBAs gather the right statistics resulting in less CPU consumption and improved maintenance window throughput. And the additional stats gathered may improve query performance, which is the ultimate goal of running RUNSTATS after all.
I work with LOTS of customers who use or want to use DB2 Performance Expert for LUW. The first question I get asked usually has to do with the basic architecture - what pieces run where and on what platform?
The first thing to understand is that DB2 PE is a client/server application. The PE server runs on LUW platforms (AIX, SUN, Linux, Windows, etc). Most customers install the PE client on Windows then connect to the PE server running on AIX, SUN, Linux, Windows, etc.
Look at the diagram below.
As shown there, we recommend that you install the PE server on its own DB2 instance -- not on the monitored box -- and have that server remotely monitoring the other DB2 instances on the network. Why? Well, you don't want to impact the monitored DB2 instances. I like to call this window shopping... you look but you don't want to touch. Note that in the 3.2 release, you get a license of DB2 ESE for just this purpose so you don't need to use one of your own DB2 installations for this.
PE can monitor different DB2 levels all from a single PE Server. You can also mix and match your operating systems, too. For example, the PE server can be installed on AIX and remotely monitor your DB2 instance running on AIX, SUN, Linux, Windows, etc..
If you want the gory details on specific levels of hardware and software server support for both the client piece and the server piece of DB2 PE, see this technote.
I'll be back soon with answers to more FAQs. Feel free to submit your questions here using the comment link below or send an email to dstudio at us.ibm.com.
-- Alice Ma
In these trying economic times, we have to find ways to reduce costs everywhere we can. And if you can reduce costs while at the same time improving staff productivity, well, that looks pretty good.
For organizations running WebSphere and DB2 applications on z/OS, and there are quite a few, we think we have just that opportunity in pureQuery software. Stephen Brodsky and others have been writing about the value of pureQuery on this blog and we published some good numbers on pureQuery performance as well. In case you prefer listening to reading, I recently recorded a podcast on pureQuery. You can find the podcast on iTunes by searching the podcast directory for “Did you say mainframe?" which will bring up the Did you say mainframe? category. My podcast is the one entitled "Enhancing Java environments with pureQuery."
Also, Stephen and I are doing a webcast together on February 4th focused on pureQuery in the mainframe environment. You can register here. I hope you’ll join us and that this creates an opportunity for your organization to save money.
A question that keeps coming up again and again.... and again is:
"Should we use stored procedures, or SQLJ or pureQuery for z/OS database access? "
The answer, as usual, is - "It depends."
Let me try to give some perspective on the question about stored procedures, SQLJ, and pureQuery.
Stored procedures are primarily meant to reduce network traffic and encapsulate a series of SQL operations with logic, but have also been used heavily to lock down the SQL issued from applications - similar to static SQL.
One thing to consider is the additional overhead one would incur for invoking stored procedures, This cost can be pretty high from what I understand. If there are a bunch of statements grouped together, along with business logic, or if results from one statement need to be used as input to another, then stored procedures would be more appropriate. If you only have one or very few SQL statements, where there is no logical need to group them together, then the overhead of the stored procedure call becomes more of a burden on the pathlength. There are more efficient ways to lock down SQL statements.
pureQuery and SQLJ provide options to lock down statements (using static SQL) without having to incur the additional stored procedure overhead.
The other consideration is cost. With pureQuery and SQLJ, the workload is eligible to run on zIIP or zAAP processors. Stored procedures will run on the general purpose CPU, unless they are native SQL procedures, which became available in V9 of DB2 for z/OS.
So, in cases where you have considered and ruled out stored procedures, you now have to think about SQLJ vs. pureQuery. Ok, so pureQuery is not free. However, from a total cost of ownership point of view, there are advantages to pureQuery. For example, Data Studio Developer has some slick tooling for pureQuery. The code generation, content assist etc., which are popular with developers, are designed around the pureQuery API. Also, if you have existing JDBC applications that you want to optimize and/or provide additional security for, you can get to static SQL without having to change any code using the pureQuery Runtime and Data Studio Developer tools.
Deployment is also significantly simplified with pureQuery. With SQLJ, emergency changes to the application and redeployment were challenging because both developers and DBAs need to get involved in altering the SQL, rebinding, and redeploying the application. With pureQuery there is no back and forth between development and DBAs. The DBA can edit the package directly and the tooling ensures that the modified SQL provides equivalent results.
My colleague Holly Hayes came up with this summary, which I like.
Achieving static SQL execution to increase z/OS capacity, lock down access path, enhance security:
Characteristics of COBOL stored procedures
Characteristics of SQLJ
Characteristics of pureQuery
Added license costs, but...
-- Vijay Bommireddipalli
Hi! It’s been a while since I posted an entry, and with the announcement of DB2 Optimization Expert for z/OS V2.1, I wanted to give you some highlights of what this release offers.
The biggest news about this release is the ability to run in an integrated desktop with IBM Data Studio, Rational, InfoSphere, and other products built on Eclipse V3.4.1, which includes products such as:
• IBM Data Studio Developer 2.1
• Rational Application Developer V18.104.22.168
• Rational Software Architect V22.214.171.124
• InfoSphere Data Architect V7.5.1
This integration is accomplished with shell sharing, and Michael Hsing has a recent entry on this topic, and it also includes links to other shell sharing articles.
So what does that integration provide? If you are a user of any of the above products, you can now invoke DB2 Optimization Expert without leaving the environment of those products. To me this is ideal for developers who want to perform query analysis on SQL contained within their code without leaving the IDE. My last blog entry provided information on those tools and advisors and can be reviewed here. You can also see the advisors in action in this demo that Thuan blogged about earlier.
One important note: The ability to shell-share with other Eclipse-based products such as Data Studio is only available with DB2 Optimization Expert. The Optimization Service Center included with the DB2 z/OS Accessories Suite does not include this capability.
For those of you who are DBAs and not developers, I encourage you to try DB2 Optimization Expert 2.1, since the integrated desktop is our stated direction. However, since this release contains the same features and functions as V1.2.2, you can continue using it and slowly migrate to the new release.
The primary focus of this release was shell sharing; our future release will focus on enhancements to our tools and advisors, as well as improved usability and workflow.
Stay tuned, more is coming.
Hi all! If you have been reading this blog for a while, you must remember Steve Brodsky’s entry on pureQuery and pureXML. If you are a new reader of this blog or don’t remember Steve’s post, you can find it here.
In his entry, Steve described how both technologies were born and why both of them got the “pure” in the name. He finished the post by describing some of the integration points between pureQuery and pureXML. That’s where I jump in! Motivated by his post, I decided to create some code snippets to show you how you can plug pureQuery and pureXML together to create Java applications that persist data into a DB2 pureXML database. My initial plan was to put that in a blog post, but as I started writing it down, more ideas were flowing in my mind than I could actually fit a single blog entry, so I decided to work on a more complete article.
The article contains the code samples (available for download) that will get you started developing with pureQuery and pureXML, but its main focus is on the different approaches that one can use when developing such applications.
In a typical application development scenario with three layers – SQL, data access API and business logic - I suggest three different approaches to handle the XML data, each one focusing on a different layer. There are certainly more approaches you can use, and you can even mix and match some of them, but my main goal was to get you started with these two great technologies and to open your mind to different ways of thinking when it comes to integrating XML into your Java applications.
Without further ado, here are the approaches I suggest in the article:
I hope you find it a good read! We continuously get questions from you, our customers, regarding this topic, so I hope I have answered them. After you are done reading it, go play with pureQuery (you can get it by downloading the trial version of Data Studio Developer) and pureXML yourself and make sure you give us your feedback, either here on the blog or on the Data Studio forum.
-- Vitor Rodrigues
Hi, I belong to the DB2 Performance Expert development team, and one of my roles is to support customers and help them to get the most out of the product.
DB2 Performance Expert for Linux, UNIX, and Windows 3.2 and the new DB2 Performance Expert Extended Insight Feature for end-to-end database monitoring of Java applications is now available. The Performance Expert Extended Insight Feature gives DBAs a new view into the performance of Java database applications. It helps to:
It’s sometimes difficult to explain the real value of what we call end to end database monitoring in words (although you can read about it in this article), so I wanted to use this as an opportunity to introduce you to a short demo that I think is worth more than a thousand words.
Thuan mentioned this video in Monday's posting. It's a live demo given by the lead architect of DB2 Performance Expert, Torsten Steinbach. In the demo you learn how easy it is to identify SQL statements that are responsible for a bad response time and to identify who issued the SQL statement and where the SQL statement spent it is time ( application, driver, network or data server ). Take a look at the video to get a first impression on this great new feature of DB2 Performance Expert.
Please let me know by adding a comment to this blog or sending an email to firstname.lastname@example.org if you have any specific questions about DB2 Performance Expert you would like me to address in future blog entries.
IBM_Optim 27000269HS Tags:  optimization_expert client_optimization purequery demo z/os bui 4,508 Views
Just want to let you know that we recently posted another demo for Data Studio on DemoZone. This is a two-part, scenario-based demo focused on a DB2 for z/OS environment. In part 1, we wanted to show some cool features available in DB2 Optimization Expert for z/OS for tuning and optimizing query performance. The second part is an extension to the previous demo set, with details of steps in setting up and using the client optimization feature in Data Studio Developer and pureQuery Runtime for performance stability. Let us know your comments on this video and what specific demos you’d like to see.
I found DB2 Optimization Expert for z/OS easy to use. Of course, when you’re trying to create a demo that shows off the features, it can be challenging. We wanted to come up with a query that would enable us to get tuning recommendations from all the advisors in the product to showcase its capabilities. That was a bit difficult but we managed to do it. You can read more about the various advisors in Ray’s blog entry.
DB2 Optimization Expert is currently available only for z/OS. As query tuning capabilities are a key skill for all database professionals, you might expect to see more database platforms supported in the future.
In addition to the demos on DemoZone, we are posting other less formal, more feature-oriented demos on Channel DB2. In particular, there is a series of demo for new features in Data Studio 2.1 including What’s New for Data Studio Developer and DB2 Performance Expert Extended Insight Feature.
We’re looking forward to hearing from you – just either add a comment to this blog or send an email to email@example.com.
Howdy! In case you missed it, we just announced a new release of HPU (High Performance Unload) for DB2 for LUW... V4.1. In case you've never looked at our HPU products (for DB2 for z/OS and DB2 for LUW), they can be great productivity enhancers and possibly even save you some resources.
One of the great things about HPU is that it has both a utility-like interface and an SQL interface. The SQL interface is perfect for application developers since they aren't used to invoking utilities. Once invoked, HPU can access the underlying table space or backup / image copy directly, producing multiple data type conversions and unload file formats suitable for most any target data store. When extracting a high volume of data in this way, or by sampling the source, the elapsed time and CPU savings are humongous versus using SQL (or Export or DSNTIAUL).
HPU for DB2 for LUW is also partition-aware, allowing you to unload from multiple partitions with a single execution of HPU into a single output file/pipe or multiple files/pipes. It also provides a re-partitioning capability that unloads and re-partitions the output for new data distribution on the same or different system.
The hot new feature in the 4.1 release for DB2 for LUW adds the ability to migrate data directly (unloading, transferring, and loading) from one database to another without the need for intermediate disk storage. This capability delivers the fastest way to migrate your data. The new release also has other usability improvements and now supports Windows 64-bit platforms.
For more information on HPU, visit http://www.ibm.com/software/data/studio/high-performance-unload/
-- Bryan Smith
Same product, same features, same organization, different name
I often say the above phrase when I have to explain to everyone what the difference is between 'RDA' and 'IDA'. We announced the renaming of IBM Rational® Data Architect to IBM InfoSphere™ Data Architect today, December 16, 2008. See the announcement letter. It really is the same product, still part of the Data Studio family, and even built on top of the same Eclipse level.
So why the name change now?
The name change features the InfoSphere Data Architect role in IBM InfoSphere Foundation Tools, an open set of tools that help prepare an organization to adopt an information agenda. Read more about the Foundation Tools in this executive brief.
There has always been integration between our data architect product and the InfoSphere Family and its predecessors. The very first release featured function to assist in data integration design, but at that time the DB2 Information Integrator branding was too limiting for our offering. Given the broad database support, we opted to give it a Rational brand featuring its integration with the Rational Software Development Platform. InfoSphere Data Architect still is and will continue to be fully integrated with the Rational portfolio, and in particular with the architectural components including Rational Software Architect, WebSphere Business Modeler and other Rational products.
What exactly is InfoSphere?
I was talking with one of the InfoSphere reps to about how he explains InfoSphere to clients when I was at the IOD conference in Las Vegas this year. He started to explain to me about moving. When I say "moving" I mean like you've bought a new house and need to physically move your items from one location to another. So let's describe the process of what you do when you move.
Same product, same features, same organization, different name
-- Anson Kokkat
I wanted to bring your attention to an article that was recently published on developerWorks that describes the pureQuery / OpenJPA integration that I discussed in my earlier blog post, which is a new feature in WebSphere Application Server v7 that enables developers to generate SQL from their JPA application entities and named queries, which can then be bound into static DB2 packages, providing a fast path to the security and performance benefits of static access for DB2 data. In addition, when you use this capability, your JPA app can take advantage seamlessly of optimizations provided in pureQuery such as the ability to update multiple tables in a single network call.
This integration is our first step toward providing a strong, integrated stack among DB2 and WebSphere, using pureQuery as the "glue". (As an aside, the Performance Expert Extended Insight Feature also uses pureQuery to provide new insights into the interactions between Java applications and DB2, with the most capabilities being provided for Java applications in WebSphere.)
From a tooling perspective, this initial JPA/pureQuery integration is fairly light, but you will notice that the static binder utility is now invocable from the WebSphere Application Server console, so WebSphere admins won't have to switch to another tool to do the bind. In addition, you'll be able to use the new capability in Data Studio Developer 2.1 to visualizeelapsed time for SQL statements directly from the pureQuery outline.
Nevertheless, there is still much more we can do to make this process easier and less command-line driven. WebSphere plans to ship enabling technology that will "turn on" capability in Data Studio Developer 2.1 to invoke the wsdb2gen utility. In addition, you'll be able to use the output from wsdb2gen within Data Studio Developer and take advantage of other pureQuery outline capabilities, including the ability to correlate SQL statements with specific OpenJPA queries and the relations between the SQL and the associated tables and columns.
Check out the article if you get a chance.
-- Steve Brodsky
What if there were no walls between the DBA and developers?
I'm not really talking about your cubicle walls but those implicit barriers we create when we are so focused on our own thing and work using our own tools. But to develop higher quality applications with more agility, we all need to work together and even cross-pollinate our skills more.
I'm the Team Lead for the Data Studio Developer tooling. Earlier this year, with Data Studio Developer 1.2, we provided tools that help teams break down barriers between developers and DBAs:
As always, if you use DB2, you can use static SQL to reduce CPU consumption in some cases. With this release, you'll find the experience from developing static SQL applications to deploying them to be significantly improved.
I've written more on developerWorks about What’s new and exciting in IBM Data Studio Developer 2.1.
BTW: I sincerely appreciate the feedback that some of you provided during my IOD sessions where I previewed Data Studio Developer 2.1. Your feedback is really important as we build our new releases.
Watch this space for the announcement of the Data Studio Developer 2.1 download shortly... Then let me know either here or on the Data Studio Forum what you think of the new release.
Ever since I bloggedmy experience with Shell Sharing, people (ok, that will be exactly one) startedthinking I were some sort of expert and asking me questions. Luckily, Ihave found a newly published Shell Sharing article (written by real experts mayI add) on IBM developerWorks that should have all the answers. To save you thetime for searching, you can get it from http://www.ibm.com/developerworks/db2/library/techarticle/dm-0811khatri/index.html.
This article uses DSD 1.2 as an example, which is just a bitdifferent from my previous experience with Data Studio 1.1.2. First of all, thedefault installed directories have been changed to:
1) Installation directory:C:\Program Files\IBM\DSDEV1.2
2) Shared ResourcesDirectory: C:\Program Files\IBM\DS12Shared
The default package name has also changed to “IBM DataStudio” instead of the more general “IBM Software Development Platform” in lastrelease.
I then followed the instructions to download a trial versionof Rational Data Architect (RDA,) and used Installation Manager to install itas Shell Shared with Data Studio. The only funny thing was “InstallationManager” displayed a message that a newer version of “Installation Manger” mustbe installed in order to continue. After I clicked OK, it just went ahead andinstalled a new version of itself. Don’t you wish every product would upgradeitself like this?
Here I’ll share a secret: developerWorks has a special spacefor “Data Studio” at http://www.ibm.com/developerworks/spaces/datastudio.Click on the “Trials and downloads” tab will not only bring you to the trialdownload page, but it will also tell you what will happen to you after thetrial period ends. You may also find thefollowing information useful:
The following products can shell-share with Data StudioDeveloper Version 1.2.
Data Studio Developer 2.1 has been announced and will beavailable soon. Since it’s an Eclipse 3.4 based product, it is safe to assumeit won’t shell share with the Eclipse 3.2 based products. My secret source toldme one nice improvement is there will be a common splash screen with a list ofthe products being shell-shared upon launch. I can’t wait to try it out.
-- Michael Hsing
New Data Studio releases bring us one step closer to realizing the integrated data management vision
IBM_Optim 27000269HS Tags:  administrator cotner announcements purequery performance_expert 2 Comments 6,696 Views
It's been less than 5 months since we announced our 1.2 releases of Data Studio, which I blogged about back in July.
Since then, we have talked to thousands of people, provided demonstrations to hundreds, and visited dozens of customers. People are starting to understand Data Studio and the value of Integrated Data Management better.
With this latest release, announced today, we are really targeting the DBA with enhancements across the portfolio to help DBAs improve application performance, security, manageability, and TCO. In this release, the enhancements are particularly targeting Java applications that access DB2 data, but you'll see we're starting to branch into .NET as well.
The announcements today are for:
Data Studio Administrator 2.1, in which we've really focused on both usabilty and functionality. We've done lots of usability testing with DBAs and have provided a more natural approach for doing many tasks, including copy and paste of database changes, flatter traversal of the data source explorer, better sorting and filtering of objects, and new task assistants for utilities, commands and configuration parameters, so you won't have to leave your environment to go out to the command line or control center to perform those tasks.
Data Studio Developer and Data Studio pureQuery Runtime 2.1, which extends the power of pureQuery for developers and DBAs to collaborate together to:
If you extend DB2 Performance Expert with the Extended Insight feature (separate PID and separately priced but prereqs DB2 PE), you can enable new end-to-end database monitoring for Java applications for DB2 servers on Linux, UNIX, and Windows. This monitoring capability will really help improve availability of mission-critical database applications by making it much easier to detect performance issues and figure out whether the problem is one in the database or somewhere else in the software stack.
Also, you can set thresholds (your SLAs, so to speak) so you can easily see how the application is performing against those targets. If you haven't read it yet, I encourage you to see the article that the Germany team who develops this feature wrote. It's a great introduction to this new capability, and it's really just our first step. This whole concept of providing greater insight to DBAs and developers is planned to be rolled out across more databases and more data access environments.
Just a head up. We're not done. We have more announcements coming soon!
Is "agile data" just another buzzphrase? Does it even make sense to try to apply agile development principles to the database?
An expert in agile development, Scott Ambler, sees agile data as an essential component for application development that goes against a database. You can learn more about agile data here: http://www.agiledata.org/
I think one of the classic challenges that agile data faces is about dealing with a "brittle" database. What do I mean by brittle? Basically, I am talking about how difficult and time consuming it can be to refactor the database schema to improve software. Check out the results of this survey question: "How long does it take to safely rename a column in a production database?"
Source of this survey:
Source: Data Quality Techniques survey by Ambysoft, September 2006.
The database and/or your software development techniques around the database are "brittle" if it takes longer than one week to make a simple rename change. Almost half of these respondents fell into that category. I would venture to say that more interesting refactoring would therefore take most shops much longer than a week.
Another part of the agile data challenge is about being able to quickly tell what the impact of a change is going to be. If we want to rename a column, what are all the database objects (tables spaces, views, stored procedures, etc ...) that will be impacted, and is there a tool to help me automate a script to make these changes?
If this sounds interesting to you and you want to learn more about agile data and how Data Studio can help, come listen to a replay (until May 09) of a webcast I did last week on how Data Studio can help make data more agile.
If you listen to the replay or are exploring agile data I am very curious to get your feedback. Just call me an agile guy. What do you think of applying agile techniques to the database? Are you doing it? If so, what is your experience? What tools are you using? What tools do you need?
What do you think?
-- Rafael Coss
Recently I finished conducting a day long Proof of Technology session in New York on Data Studio Developer and pureQuery and I thought I'd share my experience.
For those of you who have never attended an IBM Proof of Technology, it is usually a day long event at an IBM location and is a combination of presentations and hands-on exercises designed to help attendees learn and play with the technology. The computers at these sites are pre-loaded with the software and exercises that complement the presentations. Your IBM sales rep or tech sales contact is the one who would nominate you to attend one of these.
Back to the pureQuery PoT -
It walks attendees through some of the basics of Data Studio Developer, all the way to advanced pureQuery concepts.
Here are details of some of the modules:
I always like the feedback and validation (and sometimes invalidation) of our ideas.
Things I learnt during this trip:
There were two sessions on using pureQuery with Groovy at IOD. One of the sessions was a joint one that I presented with my friend Vladimir Bacvanski at InferData, who, by the way, offers a pureQuery course.
Here's a description of the session we gave at IOD:
Project Zero and WebSphere® sMash represent a new wave of technologies for agile development of dynamic Web applications based on scripting runtime environment for Groovy and PHP, REST-style interfaces, mashups and rich Web interfaces. In this talk we show the united power of Project Zero with pureQuery . a high-performance technology that accesses relational data and which is the foundation for the data access in Project Zero.
This session was due to be held on Friday morning.Vladimir and I created the first ever (we think!) Groovy, pureQuery and DB2 Static SQL application on Thursday night, 10/30/08 at 11:14 pm PDT at a casino in Las Vegas.
In getting prepared for the Friday session Vladimir and I started experimenting on what is possible. Groovy is Java, and pureQuery is also Java-based, so hypothetically we should be able to completely mix and match it. So, we thought, let's try using sMash's Groovy to quickly build a database web 2.0 application that leverages the power of DB2 static SQL execution. After some hacking, at 11:14 we built a small Dojo app that received data using JSON from DB2 that was using static execution. Waaaayyyyyy coool!!!!! We got so excited about how cool, easy and quick it was we ended up playing with the app till 2 am in a Las Vegas Casino cranking out code.
-- Rafael Coss
So we are almost near the end of the conference and attendees of the conference are starting to head back.
I am still amazed at the scale of these events, especially watching lunch being served to so many people. I heard there are around 7000+ attendees!
I am done with two presentations on Data Studio Developer, and have just one more lab to go. The presentations went well, however, for me, the post presentation conversations are always the most interesting part.
Some of the questions / comments:
How do I deploy pureQuery applications and merge it with existing deployment processes
- Need to pass along some examples
How does Data Studio Developer integrate with RAD?
- I explained and showed how shell sharing works on my laptop
One random comment:
An attendee came up to me and said, I like how you present ( I say - Thank You). Then he says, I like how you do the 1 - 2 punch ! ... ( I am trying to figure out that comment .. I am still clueless) :)
I also spent quite a bit of time at the Data Studio pedestal. Every customer that I talked to seemed to have pain points associated with Java and data access that pureQuery is addressing. We had a significant number of sessions, so the conversations with quite a few of them usually started with some level of familiarity and that was refreshing.
Attendees, what was your experience like ?
p.s. The background music at the conference hallways is super-annoying ! Agree?
We had the Data Studio Customer Advisory Council today. One of our toughest customers gave Torsten a standing ovation upon completing his demo of the E2E database monitoring that is planned for delivery soon. I never saw Torsten blush before today.
One of the Toronto user-centered design guys, Rick, came up with a clever way (aka Vegas style) for the CAC members to vote on future functions... They were given poker chips to put into feature function cups. It was really clever.
I finally won tonight at the tables after the Rock the Mainframe party.
-- Bryan Smith
Here are my impressions from Monday through Wednesday.
Curt almost missed his session yesterday. Someone called him 10 mins before the session to remind him -- he recovered quickly and was only 5 mins late. Full room with heads bobbing up and down when talking about problems with supporting Java applications.
Most folks liked Dana Carvey much better last year than Martin Short.
Listening now to Jim Pickel on DB2 Security.
Lost more money at craps and blackjack table. I'm now starting to feel this financial crisis that I keep hearing about on the news.
-- Bryan Smith
I am sitting in the awesome developer den where there is a roomful of colorful bean bags! Along with 2 Wiis that they will be giving away at the end of the week. IOD attendees should stop by to enter into the Wii contest and there are many laptops set up with Data Studio developersWorks articles. Visitors can check out articles on Data Studio and its family of products and also ask the experts about Data Studio. We are located in Breakers G from 10am - 5pm.
Bad news, we had connectivity problems in the Data Studio for DB2 for z/OS labs. But we have a re-do on Thursday, and I hope everyone who really wanted to do this lab with a DB2 for z/OS server will come:
Session: HOL-2670B Data Studio and DB2 for z/OS
Time: Thu, 30/Oct, 02:00 PM - 05:00 PM
Location: Mandalay Bay South Convention Center - Lagoon F
-- Tina Chen
This is actually day 4 for me as I spent the weekend with DB2 LUW customers attending the DB2 Customer Advisory Council, a group of DB2 customers that provide feedback to the DB2 Toronto team on upcoming releases and future strategies to help shape DB2 LUW. We had a three hour window with these customers on Data Studio and the feedback was tremendous. The big change I'm seeing from 6 months ago at IDUG is customers are now downloading and using Data Studio throughout their developer communities. They're seeing the value Data Studio has to add over and above what they get from either Developer Workbench or other 3rd party development tools. Also, not only are they using this for their DB2 LUW environments, but several users indicated that they're using this for DB2 z/OS, as it enables their developers to easily build and debug SQL stored procedures for their DB2 z/OS environments. This is a big change from 6 months ago when customers didn't even know Data Studio existed.
Yesterday I held a session called "Empowering DBAs with Data Studio", the room was full with standing room only. Goes to show that DBAs are always looking for the latest and greatest technology to manage their databases. I demonstrated the new Data Studio coming soon. This upcoming release has added a ton of functionality for the DBA; including utilties, commands and more DDL management. The audience was very excited and really wants to see Data Studio become their tool for managing both DB2 LUW and DB2 z/OS databases. There's definitely a buzz around Data Studio at this conference.
Tonight I arranged a podcast with YL&A consultants and Curt Cotner on Data Studio. I'll let you know how that goes....
-- Deb Jenson
Expo area is large. It was nice to see familiar faces there.
Lost money at the craps table last night.
Opening session: Ambuj announced over 7000 registered. It looked like they all attended the opening session. Martin Short was hilarious. His no-holds-barred routine was great.
I gave my session with Holger this am on Administration tooling. It went well. I took some questions about some upcoming branding changes that were discussed during the opening session. As we've been saying over time, the tools will evolve to support more than one database platform. In addition, over time the thought is to rebrand to Optim, which has good heterogeneous support.
Sitting now in Torsten and Holger's session on end to end database monitoring. There are probably150 folks in attendance. If you want to know more about it... see this hot off the press article in IBM Database Magazine.
I wonder what it's like outside... wish I could get some time to just sit and veg....
-- Bryan Smith
Definitely growing momentum. Tina did a Data Studio Developer for DB2/z hands on lab that was at capacity and had 15 people waiting outside to get into the lab.
The Gold consultants heard about next generation Web 2.0 and data-oriented cloud computing topics. Then, Curt talked about Data Studio plans, which was mostly stuff that is going to come in 2009, and he was asked three times when exactly is that feature going to be available -- just goes to show how compelling the Data Studio vision is.
I did a session on Data Studio and Rational integration. And a few common interesting thoughts came out. It seems like folks are very hungry to be able to tie the software requirements using Rational Req Pro to both the application and data model. Folks are so very happy about the shell sharing concept, about being able to install a series of tools into a common Eclipse in which the value of the whole is greater than the sum of the individual parts.
I was also working the Data Studio demo booth, and it was interesting to see the reaction of DBAs on being re-empowered to manage the data access layer via pureQuery for both existing JDBC application and new applications based on pureQuery.
-- Rafael Coss
"RU Ready?" This is what I was thinking when I was on the plane to China.
Being that this was my first time in China, I didn’t even know what to expect much less how many people would show up at my Data Studio session. I was on my way to attend the Rational Software Development Conference in China, which was hosted in the Shanghai Convention Center in September.
When I finally arrived I was completely surprised to see how many people showed up at the one day conference!
Around 900-plus people packed the 2nd floor to listen to “R Heroes” discuss what Rational had to offer this year. Due to the larger than expected audience, the auditorium was packed with attendees for the keynote speech, and since so many people showed up they had to open a separate hall to televise the speech! Did I mention the entire conference was also streamed online live? Talking about using technology to its fullest advantage!
To my surprise, I had around 110 people show up at my talk, and the conference hall was so packed that there were people standing on chairs outside the door trying to get a glimpse of the slides.
By the request of the local team at the last minute, I was asked to present my Data Studio slides in Chinese. Now I really had to put my 10-plus years of Chinese school learned in the US to work!
When I speech was over, there was silence. I could hear people scribbling on their notepad, jogging down notes. Was my Chinese so bad that no one knew what I was saying?
And all of sudden, hands were popping into the air.
This was the first time many of the attendees were introduced to Data Studio and its family of products. Attendees were asking great questions like how to use pureQuery with their existing Hibernate applications. And how Data Studio could help them become productive.
On my plane ride back to the states I exhaled a sigh of relieve as I boarded the plane. My mission was complete.
-- Tina Chen
Hi, I'm a Program Director with IBM Data Studio software. I've had a long history with IBM and, in particular, database and database tools spanning DB2 for z/OS, data replication, business intelligence, information integration, and database management tools.
I recently attended the CIO Forum in Pittsburgh, Pennsylvania. The CIO Forum is a 2 day regional event held in various cities across the United States focused on developing executive IT leadership, networking with peers, and leveraging technology to succeed in the global economy. There were around one hundred attendees which made it a great size for interacting.
A few of things stand out for me:
First, of course, our panel on Integrated Data Management with Deborah Hurley, Chief Warehouse Architect for Constant Contact, and Joe Farrugia, VP of Sales and Consulting for Dynamic Systems Solutions. Deborah is leading the way towards effective alignment of business and IT at Constant Contact. Using Rational Data Architect, IBM InfoSphere Information Server, and Business Glossary together, Constant Contact is driving ownership of the glossary terms with the subject matter experts in the line of business, but keeping IT in synch through integrations with Rational Data Architect and Information Server. Linking business and IT is key to developing trusted information assets, and business analysts are going to be front and center in this evolution.
The increasing importance of the business analyst role was also echoed in an analyst research note I'd read on the plane on the way to Pittsburgh. And at the event, one of the attendees spoke of their business curriculum that was morphing to focus on developing the cross-business-and-IT skill set needed to perform these emerging roles. Check out Deborah's session at Information on Demand.
Constant Contact Manages Metadata: RDA, Cognos BI, & Information Server , Thursday, Oct 30 at 8:30 AM.
I'd really like to hear from you... How is your organization managing alignment across business and IT?
The next thing I noticed is the degree of interest in data archiving solutions (even more than data privacy solutions). Our IBM table garnered lots of interest, particularly about our data archiving solutions (or maybe it was the ever-popular THINK ball caps). Folks we spoke to were looking to data archiving to improve performance for both online and batch processing and to retain information off-line for regulatory purposes while providing fast access in case of discovery requirements. They thought IBM Optim solutions sounded very promising, particularly given their cross-platform, cross-database, and application-aware characteristics. I think Joe Farrugia's description of client successes and savings drew them our way, too. If this is an area of interest to you, too, you might want to check out another session at IOD:
Mid-American Energy: DB2 Archiving for Performance and Cost Control, Wednesday, Oct 29th at 11:30 AM.
Are archiving or data privacy top issues for your organization? What are your key business drivers?
I'm really interested in hearing from you all about issues around business and IT. It's always good to be able to feed real scenarios back to our engineering teams to ensure that we hit the mark. Feel free to comment using the Comment link below, or send an email to firstname.lastname@example.org and tell us your story!
(Updated 10/15/2008 with updated Guide to IOD)
These Information on Demand conferences that we have every year really keep our lab and sales people busy. But all the work leading up to them is really worth it in the end, because it gives us an opportunity to talk directly to so many of our customers and get insight into the problems they face (which hopefully we can help solve). Last year’s IOD is where Data Studio was launched, and I’m very pleased that at this year’s IOD we have so much more to share with you – lots of sessions, demos, labs, community events, usability sandboxes, BOFs, etc. It’s also a great opportunity to get a sneak peek of upcoming enhancements.
I’m very honored that State Farm Insurance will be co-presenting with me on their experiences using pureQuery and Data Studio software, and I hope you can make that session.
We like working with customers on early releases to help us drive the product vision forward. This is the same basic pattern that we’ve used in DB2 for our entire 25 year history for each of the major steps in our technology journey (initial release of DB2, sysplex technology, DRDA and DDF, stored procedures, and now Data Studio and pureQuery). At a high level, the pattern is pretty simple:
It really helps our team deliver products that will fulfill your needs if they actually spend time talking to you. This is why I encourage you to come to our sessions, our birds of a feather sessions, our demos and tell us what you think about what we’re saying and showing. Effective product management and development relies on conversations, not on lectures.
For those of you who can’t make the conference in person, our team will be blogging from the conference. Hopefully that will give you the opportunity to get some idea of what is happening and provide you with the opportunity in this blog to add your comments and feedback.
I’m attaching a document here that highlights key sessions around Data Studio and Integrated Data Management and provides a pretty comprehensive list of sessions by day and product/topic area. I hope you find it useful, and I hope to see you there.
-- Curt Cotner
As I mentioned in my previous blog on the RDA 7.5 announcement, I promised to let you know when the trial code is ready. Well, it's ready now. You can download the 30-day trial off developerWorks.
For the highlights of the new release, check out the What's New documentation and check out my earlier blog on this topic that focuses on the data privacy and integration aspects of the new release, or even better, listen to my webcast.
I can show you all the new features in person. Come meet me at the IOD 2008 conference. I'll be busy at the conference - you can find me at either of the following sessions:
Get your hands dirty and join me for the following hands on labs:
Mandalay Bay South Convention Center - Breakers A Tue, 28/Oct, 10:00 AM - 01:00 PM
Mandalay Bay South Convention Center - Breakers C Wed, 29/Oct, 10:00 AM - 01:00 PM (This one looks full but come by anyway since sometimes people don't show up)
And you don't have to take my word about it. Don't miss this customer session:
-- Anson Kokkat