IBM Open Data Analytics for z/OS, V1.1.0 enables support for Apache Spark, Python, and Anaconda stack natively on IBM z/OS

IBM Latin America Software Announcement LP17-0288
July 17, 2017

Table of contents

	Overview		Publications
	Key prerequisites		Technical information
	Planned availability date		Ordering information
	Description		Terms and conditions
	Statement of general direction		Prices
	Program number		Latin America distribution

At a glance

IBM® Open Data Analytics for z/OS®, V1.1.0 offers an integrated, optimized runtime foundation of industry-leading open source technologies that can be used to gain analytic insights at the source of data origin, by leveraging Apache Spark as well as Anaconda and Python environments.

Open Data Analytics for z/OS benefits include:

Higher value insights: Use the most current data, incorporate a wide variety of data on and off IBM z/OS™ to reduce latency of insights, and derive higher value from analytics.
Reduced costs: Keep data in place and bring the analytics to the data by leveraging the use of open source technologies, and reduce costly Extract-Transform-Load (ETL) processes.
Flexible: Integrate across a heterogeneous environment using modern and consistent analytic capabilities commonly used by data scientists everywhere.
Simplify data analytics: Leave your data right where it resides today and use common analytics programming languages and open source technologies with access to both z/OS and distributed data.
Security - data governance: Protect business data and insights with the advantage of IBM Z crypto cards that zero out when tampered with, encrypt all your data and analyze it at the source of origin, and configure specific data obfuscation for varying levels of privacy within an organization.
Interoperability: Integration with enterprise business applications through unique capabilities to persist and share insights through standard interfaces.
Federated analytics with unified access to data: Eliminate the need to consolidate all data prior to analytics, leverage abstractions of data from multiple disconnected sources on z/OS as well as other platforms, and virtually integrate to perform analytics to derive insights using open source technologies.

Overview

The large volume and variety of data available today offers the possibility of seeing things in new ways. Value comes from data-driven insights, not data alone. Analytic algorithms can combine and extract insights from a wide range of data, from transactional systems and data on the mainframe to customer data collected from mobile and social applications and data streaming in from the Internet of Things and sensors, in order to identify trends, model possible scenarios, and predict future results. These insights can provide business advantage by identifying emerging opportunities, improving the customer experience, enhancing operational efficiencies, reducing risks, and more.

Open Data Analytics for z/OS (IzODA) extends the IBM Z open analytics beyond Apache Spark on z/OS to include Python and Anaconda Python packages for data science (z/OS IzODA Anaconda). By combining industry standard open source analytics frameworks with the Optimized Data Layer, data scientists are enabled to access mainframe data-in-place for analytics without requiring mainframe expertise. This creates a unified experience and expands the ability to extract insights from a wide range of data to a new generation of users who previously may not have had the skills needed to analyze data in z/OS. With this combined solution, the computations can be brought to the data, thereby avoiding costly processing and security considerations associated with data movement.

The Optimized Data Layer also enables access to and use of off-platform data as well as z/OS data, enabling an integrated view. Eliminate the need to consolidate all data prior to analytics, leverage abstractions of data from multiple disconnected sources on z/OS as well as other platforms, and virtually integrate to perform analytics to derive insights using open source technologies.

Apache Spark is an open source engine that is built specifically to accelerate the time from analytics to insight by providing a full framework of capability for data science. It helps simplify algorithm development and accelerate analytics results. With Apache Spark, you can better extract value from data, conducting deeper analysis and delivering results faster, all while reducing the time and effort required for coding.

Open Data Analytics for z/OS enables clients to take advantage of the latest Apache Spark features and extends the capabilities with Python and Anaconda Python packages (z/OS IzODA Anaconda) through a continuous delivery model. The continuous enhancement delivery model provides rapid delivery of new functionality without depending on formal version upgrades while also protecting your ability to control the deployment of any specific enhancement.

The Open Data Analytics for z/OS product structure is delivered in three Function Modification Identifiers (FMIDs):

HSPK120 - z/OS IzODA Spark
HMDS120 - z/OS IzODA MDS
HANA110 - z/OS IzODA Anaconda

Key prerequisites

Open Data Analytics for z/OS requires:

IBM z/OS V2.1, or later
IBM 64-bit SDK for z/OS, Java™ Technology Edition, Version 8 Refresh 4 Fix Pack 6, or later
An IBM z14, IBM z13™, IBM z13s, IBM zEnterprise® EC12, or IBM zEnterprise BC12 server

Planned availability date

September 8, 2017

Description

What is it?

Open Data Analytics for z/OS offers an enterprise-grade platform to enable data scientists to use Apache Spark and develop Python applications directly on z/OS.

What does it do?

Open Data Analytics for z/OS includes Apache Spark open source capabilities consisting of the Apache Spark core, Spark SQL, Spark Streaming, Machine Learning Library (MLib), Graphx, and support for Python and Anaconda packages (z/OS IzODA Anaconda).

Open Data Analytics for z/OS provides Python on z/OS, along with many Anaconda Python packages, to provide data scientists with a comprehensive solution for integrating computations to the data. The optimized data abstraction and virtualization layer enable efficient access to a broad set of structured and unstructured data sources through Spark APIs. With this capability, traditional z/OS data sources such as IMS™, VSAM, DB2® z/OS, PDSE, PS, or SMF data can be accessed in combination with data from other platforms in a performance-optimized manner with Apache Spark or Python/Anaconda applications.

Improved VSAM data access

With the ability to read data in parallel already in Mainframe Data Service for Apache Spark V1.1, optimization has been enhanced to write data in parallel for VSAM ESDS files in Open Data Analytics for z/OS , V1.1, thereby providing optimized data access on both reads and writes. zIIP-enabled parallel reads and writes ensures your data is accessed in an optimized manner for an improved analytics experience.

Optimized data access for IMS sources

Optimize your data access with the support of compression exits for IMS databases. This enhancement zIIP-enables multiple calls to IMS data compression exits using parallel data threads, further reducing elapsed time and CPU consumption. Accessing IMS data can be done directly for HDAM and HIDAM databases. OSAM database types can be called directly, thereby speeding data access and reducing elapsed time.

Access IDMS data and dashDB™ data using standard SQL

Open Data Analytics for z/OS V1.1 introduces SQL access to IDMS record and set information managed by IDMS central versions running on z/OS. Easily map IDMS records and sets to virtual tables that can then be joined using standard SQL to enable your applications to navigate IDMS data.

Open Data Analytics for z/OS, V1.1 also introduces the ability to access IBM dashDB data sources with DRDA® using standard SQL processing. Easily join dashDB tables to other data sources using standard SQL for enterprise-wide access to your data.

Workload management integration

Analytics workloads on z/OS can automatically respond to dynamically changing conditions with workload management (WLM) integration in Spark on z/OS, ensuring not only analytics at scale, but enterprise-level availability for mission-critical workloads. A system administrator can classify different analytics workloads based on their business value, ensuring system resources are used in support of business goals. Spark with WLM enables you to take advantage of scaling up inside a z/OS image, while also inheriting the security advantages of keeping the data in a Spark cluster within a z/OS system.

Security enhancements / Client authentication

Help seamlessly safeguard analytics with Open Data Analytics for z/OS integration of enterprise-level mainframe security features for identity management. It is important to note that without the integration of authentication, the open source level of Apache Spark running on any platform, does not provide user authentication when work is submitted to a stand-alone Spark cluster. When running on z/OS, access to a Spark cluster can be restricted and secured through z/OS configuration settings, such as port controls. Integrated user authentication enables a Spark user to be additionally validated by the z/OS external security manager, such as RACF®. This not only enhances security, but greatly simplifies auditing. Additionally, when cluster workloads run under the identity of the end user, the memory and processor resources consumed can be controlled by Workload Management policies set for that user, allowing both high-priority and low-priority Spark workloads sharing a cluster to be balanced based on priorities set by the z/OS system programmer.

Dynamic workload management

One of the strengths of the Z platform and the z/OS operating system is the ability to run multiple workloads at the same time within one z/OS image or across multiple images. Such workloads have different, often competing, resource requirements. These requirements must be balanced in order to make the best use of the resources of an installation, maintain the highest possible throughput, and achieve the best possible system responsiveness. The function that makes this possible is dynamic workload management, which is implemented in the Workload Management component of z/OS. With z/OS Workload Management (WLM), you define performance goals and assign a business importance to each goal. You define the goals for work in business terms, and the system decides how much resource, such as CPU or memory, should be given to it to meet the goal. Workload Management will constantly monitor the system and adapt processing to meet the goals.

Open Data Analytics for z/OS leverages Z technology for exceptional performance and scalability, giving clients the ability to reliably run multi-workload Spark Clusters with optimal performance. Analytic workloads on z/OS can automatically respond to dynamically changing conditions with WLM integration in Spark on z/OS, ensuring not only analytics at scale, but enterprise-level availability for mission-critical workloads. With WLM, analytic workloads can be classified based on their business value, ensuring system resources are used in support of business goals. WLM will automatically scale the compute and memory resources based on business importance and Apache Spark's changing resource requirements for user jobs. z/OS WLM capabilities provide analytic applications with consistent run times that optimally use system resources without requiring the data scientist or application developer to have a deep understanding of system resources and their current utilization.

Open Data Analytics for z/OS with WLM helps you to take advantage of scaling up inside a z/OS image, while also inheriting the security advantages of keeping the data within a z/OS system.

In October 2016, z/OS announced the intent to stage support for new capabilities in z/OS for metering and capping workloads over CPU and memory consumption, with the initial focus on workloads that run only on specialty engines. Availability of the first Metering and Capping Workloads solution for z/OS V2.2 and z/OS V2.1 comprises the following APARs.

For z/OS V2.2, the support is delivered through PTFs for APARs:

OA51171 (RSM)
OA50953 (Supervisor)
OA50845 (WLM/SRM)
OA50760 (RMF™)
PI71118 (z/OSMF)

For z/OS V2.1, the support is delivered through PTFs for APARs:

OA51171 (RSM)
OA50953 (Supervisor)
OA50845 (WLM/SRM)
PI71118 (z/OSMF)
OA50760 (RMF)
z/OSMV V2.1 with APAR PI71084 (z/OSMF)

PTFs are identified by the Spark fix category of IBM.Function.ApacheSpark.

This support provides additional WLM controls that enable the system capacity planner to specify that the resource consumption of selected workloads should not exceed the desired resource consumption limits. In particular, for Spark workloads, this capability provides the ability to specify that all the applications/jobs submitted to a particular Spark cluster do not exceed a specified real memory limit, and that they do not receive help from standard processors when the zIIP capacity is exceeded. To accomplish this, WLM supports a new attribute for resource groups that allows specifying an upper limit for real memory consumption through the "Memory Limit" attribute for WLM resource groups. The system will enforce this collective upper limit among all address spaces that are associated with the resource group.

In addition, to allow more flexibility over zIIP workloads, a new Honor Priority option is provided on a WLM service class basis that allows a service class to be excluded from the system-wide Honor Priority defaults to prevent the specialty processor eligible work from receiving help from general purpose processors. These controls can also be used for other analytic workloads or cloud workloads for controlling their memory and CPU usage.

Accessibility by people with disabilities

A US Section 508 Voluntary Product Accessibility Template (VPAT), containing details about accessibility compliance, can be found on the Product accessibility information website.

Statement of general direction

IBM intends to replace IBM z/OS Platform for Apache Spark (5655-AAB) with IBM Open Data Analytics for z/OS (5655-OD1) and to only provide new functions to IBM Open Data Analytics for z/OS.

IBM's statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM's sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remain at our sole discretion.

Program number

Program number	VRM	Program name
5655-OD1	1.1.0	IBM Open Data Analytics for z/OS, V1.1.0

Product identification number

IBM Open Data Analytics for z/OS, V1.1.0

Program PID number	Subscription and Support PID number
5655-OD1	5655-OD2

Education support

IBM training provides education to support many IBM offerings. Descriptions of courses for IT professionals and managers can be found on the IBM Training and Skills website.

Contact your IBM representative for course information.

Offering Information

Product information is available on the IBM Offering Information website.

Publications

The product documentation includes these publications:

Title	Order number
Program Directory for IBM Open Data Analytics for z/OS	GI13-4348-00
License Information for IBM Open Data Analytics for z/OS	GI13-4349-00
IBM Open Data Analytics for z/OS Installation and Customization Guide	SC27-9033-00
IBM Open Data Analytics for z/OS User's Guide	SC27-9034-00
IBM Open Data Analytics for z/OS Administrator's Guide	SC27-9035-00
IBM Open Data Analytics for z/OS Solutions Guide	SC27-9036-00

IBM Knowledge Center will provide access to the documentation on September 8, 2017, at the IBM Open Data Analytics for z/OS home page.

The IBM Publications Center is a worldwide central repository for IBM product publications and marketing material with a catalog of over 70,000 items. Extensive search facilities are provided. A large number of publications are available online in various file formats, which can currently be downloaded.

Technical information

Specified operating environment

Hardware requirements

Open Data Analytics for z/OS requires one of the following Z servers:

IBM z14
IBM z13
IBM z13s
IBM zEnterprise EC12
IBM zEnterprise BC12

Software requirements

Open Data Analytics for z/OS requires:

IBM z/OS V2.1, or later
IBM 64-bit SDK for z/OS, Java Technology Edition, Version 8 Refresh 4 Fix Pack 6, or later

Planning information

The Open Data Analytics for z/OS is available in CBPDO and ServerPac.

Customer responsibilities

The customer is responsible for evaluation, selection, and implementation of security features, administrative procedures, and appropriate controls in application systems and communication facilities.

Packaging

The Open Data Analytics for z/OS package is available in CBPDO and ServerPac.

Security, auditability, and control

Open Data Analytics for z/OS uses the security and auditability features of host hardware, host software, and application software.

Ordering information

Charge metric

Program name	Part number or PID number	Charge metric
IBM Open Data Analytics for z/OS, V1.1.0	5655-OD1	Per Install
IBM Open Data Analytics for z/OS, V1.1.0 S&S	5655-OD2	Per Install

Install

Install is a unit of measure by which the program can be licensed. An install is an installed copy of the program on a physical or virtual disk made available to be executed on a computer. Licensee must obtain an entitlement for each install of the program.

Basic license

To order, specify the program product number and the appropriate license or charge option. Also, specify the desired distribution medium. To suppress shipment of media, select the license-only option in CFSW.

Program name: IBM Open Data Analytics for z/OS, V1.1.0

Program PID: 5655-OD1

Entitlement identifier	Description	License option/Pricing metric
S0187CS	Open Data Analytics for z/OS	Basic OTC, per Install
	Open Data Analytics for z/OS	MultiVersion Measurement NC

Orderable supply ID	Description	Language	Distribution medium
S0187D5	Open Data Analytics for z/OS	English US	1/2 Inch Data Cartridge - IBM 3590 - 30 GB - 128 Track

Subscription and Support PID: 5655-OD2

Entitlement identifier	Description	License option/Pricing metric
S0187CW	Open Data Analytics for z/OS	Basic MSC, per Install SW S&S
	Open Data Analytics for z/OS S&S	No charge, decline SW S&S
	Open Data Analytics for z/OS S&S	MultiVersion Measurement NC

Orderable supply ID	Description	Language	Distribution medium
S0187CX	Open Data Analytics for z/OS S&S	English US	Paper

Customized Offerings

Product deliverables are shipped only through CBPDO and ServerPac. These customized offerings are offered for internet delivery in countries where Shopz product ordering is available. Internet delivery reduces software delivery time and allows you to install software without the need to handle tapes. For more details on Internet delivery, go to the Help section on the Shopz website.

You choose the delivery method when you order the software. IBM recommends internet delivery. In addition to internet and DVD, the supported tape delivery options include:

3590
3592

Most products can be ordered in ServerPac the month following their availability in CBPDO. z/OS can be ordered through CBPDO and ServerPac at general availability. Many products will also be orderable in a Product ServerPac without also having to order the z/OS operating system or subsystem.

Shopz and CFSW will determine the eligibility based on product requisite checking. For more details on the product ServerPac, go to the Help section on the Shopz website.

For additional information about the Product ServerPac option, refer to Software Announcement LP12-0371, dated July 31, 2012.

Production of software product orders will begin on the planned general availability date.

CBPDO shipments will begin one week after general availability.
ServerPac shipments will begin four weeks after general availability.

Terms and conditions

The information provided in this announcement letter is for reference and convenience purposes only. The terms and conditions that govern any transaction with IBM are contained in the applicable contract documents such as the IBM International Program License Agreement, IBM International Passport Advantage® Agreement, and the IBM Agreement for Acquisition of Software Maintenance.

Licensing

IBM International Program License Agreement including the License Information document and Proof of Entitlement (PoE) govern your use of the program. PoEs are required for all authorized use.

This software license includes Software Subscription and Support (also referred to as Software Maintenance).

Agreement for Acquisition of Software Maintenance

The following agreement applies for Software Subscription and Support (Software Maintenance) and does not require customer signatures:

IBM Agreement for Acquisition of Software Maintenance (Z125-6011)

These programs are licensed under the IBM Program License Agreement (IPLA) and the associated Agreement for Acquisition of Software Maintenance, which provide for support with ongoing access to releases and versions of the program. These programs have a one-time license charge for use of the program and an annual renewable charge for the enhanced support that includes telephone assistance (voice support for defects during normal business hours), as well as access to updates, releases, and versions of the program as long as support is in effect. IBM z Systems Operational Support Services - Support Line is an option if you desire added services.

License Information number

L-MAMY-AMVMQN

See the License Information documents page on the IBM Software License Agreement website for more information.

Limited warranty applies

Yes

Limited warranty

IBM warrants that when the program is used in the specified operating environment, it will conform to its specifications. The warranty applies only to the unmodified portion of the program. IBM does not warrant uninterrupted or error-free operation of the program or that IBM will correct all program defects. You are responsible for the results obtained from the use of the program.

IBM provides you with access to IBM databases containing information on known program defects, defect corrections, restrictions, and bypasses at no additional charge. For further information, see the IBM Software Support Handbook.

IBM will maintain this information for at least one year after the original licensee acquires the program (warranty period).

Program technical support

Technical support of a program product version or release will be available for a minimum of two years from the general availability date, as long as your Software Subscription and Support (also referred to as Software Maintenance) is in effect.

This technical support allows you to obtain assistance (by telephone or electronic means) from IBM for product-specific, task-oriented questions regarding the installation and operation of the program product. Software Subscription and Support (Software Maintenance) also provides you with access to versions, releases, and updates (CD releases, Long Term Support Releases or fixes) of the program. You will be notified, through an announcement letter, of discontinuance of support with six months' notice. If you require additional technical support from IBM, including an extension of support beyond the discontinuance date, contact your IBM representative or IBM Business Partner. This extension may be available for a fee.

For additional information on the IBM Software Support Lifecycle Policy, see the IBM Software Support Lifecycle Policy website.

Money-back guarantee

If for any reason you are dissatisfied with the program and you are the original licensee, you may obtain a refund of the amount you paid for it, if within 30 days of your invoice date you return the program and its PoE to the party from whom you obtained it. If you downloaded the program, you may contact the party from whom you acquired it for instructions on how to obtain the refund.

For clarification, note that for programs acquired under any of IBM's On/Off Capacity on Demand (On/Off CoD) software offerings, this term does not apply since these offerings apply to programs already acquired and in use by you.

Volume orders (IVO)

Passport Advantage applies

Software Subscription and Support applies

Yes. During the Software Subscription and Support period, for the unmodified portion of a program, and to the extent problems can be recreated in the specified operating environment, IBM will provide the following:

Defect correction information, a restriction, or a bypass.
Program updates: Periodic releases of collections of code corrections, fixes, functional enhancements and new versions and releases to the program and documentation.
Technical assistance: A reasonable amount of remote assistance by telephone or electronically to address suspected program defects. Technical assistance is available from the IBM support center in the organization's geography.

Additional details regarding Technical Assistance, which includes IBM contact information, are provided in the IBM Software Support Handbook.

Software Subscription and Support does not include assistance for:

The design and development of applications.
Your use of programs in other than their specified operating environment.
Failures caused by products for which IBM is not responsible under the IBM Agreement for Acquisition of Software Maintenance.

Software Subscription and Support is provided only if the program is within its support timeframe as specified in the Software Support Lifecycle policy for the program.

Yes. All distributed software licenses include Software Subscription and Support (also referred to as Software Maintenance) for a period of 12 months from the date of acquisition, providing a streamlined way to acquire IBM software and assure technical support coverage for all licenses. Extending coverage for a total of three years from date of acquisition may be elected.

While your Software Subscription and Support is in effect, IBM provides you assistance for your routine, short duration installation and usage (how-to) questions, and code-related questions. IBM provides assistance by telephone and, if available, electronic access, only to your information systems (IS) technical support personnel during the normal business hours (published prime shift hours) of your IBM support center. (This assistance is not available to your end users.) IBM provides Severity 1 assistance 24 hours a day, every day of the year. For additional details, go to the IBM Support Handbooks page.

Software Subscription and Support does not include assistance for the design and development of applications, your use of programs in other than their specified operating environment, or failures caused by products for which IBM is not responsible under this agreement.

For more information about the Passport Advantage® Agreement, go to the Passport Advantage and Passport Advantage Express® website.

IBM Operational Support Services - Support Line

Variable charges apply

Educational allowance available

Yes. A 15% education allowance applies to qualified education institution customers.

Multi-Version Measurement

Multi-Version Measurement (MVM) replaces the previously announced Migration Grace Period time limit of six months and allows unlimited time for clients to run more than one eligible version of a software program. Clients may run multiple versions of a program simultaneously for an unlimited duration during a program version upgrade. Clients may also choose to run multiple versions of a program simultaneously for an unlimited duration in a production environment. MVM does not extend support dates for programs withdrawn from service.

For more information about MVM, including requirements for qualification, see the MVM web page. For a list of eligible programs, see the IPLA Execution-Based web page.

Statement of good security practices

IT system security involves protecting systems and information through intrusion prevention, detection, and response to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed, or misappropriated or can result in misuse of your systems to attack others. Without a comprehensive approach to security, no IT system or product should be considered completely secure and no single product or security measure can be completely effective in preventing improper access. IBM systems and products are designed to be part of a regulatory compliant, comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems, products, or services to be most effective.

Important: IBM does not warrant that any systems, products, or services are immune from, or will make your enterprise immune from, the malicious or illegal conduct of any party.

Prices

For all local charges, contact your IBM representative.

IBM Global Financing

IBM Global Financing offers competitive financing to credit-qualified customers to assist them in acquiring IT solutions. Offerings include financing for IT acquisition, including hardware, software, and services, from both IBM and other manufacturers or vendors. Offerings (for all customer segments: small, medium, and large enterprise), rates, terms, and availability can vary by country. Contact your local IBM Global Financing organization or go to the IBM Global Financing website for more information.

IBM Global Financing offerings are provided through IBM Credit LLC in the United States, and other IBM subsidiaries and divisions worldwide to qualified commercial and government customers. Rates are based on a customer's credit rating, financing terms, offering type, equipment type, and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension, or withdrawal without notice.

Latin America distribution

All Latin American countries except Cuba.

Trademarks

IBM z, IBM z13, IMS, dashDB, RMF and IBM z Systems are trademarks of IBM Corporation in the United States, other countries, or both.

IBM, z/OS, zEnterprise, Passport Advantage, DB2, DRDA, RACF and Express are registered trademarks of IBM Corporation in the United States, other countries, or both.

Oracle and Java are trademarks of Oracle and/or its affiliates in the United States, other countries, or both.

Other company, product, and service names may be trademarks or service marks of others.

Terms of use

IBM products and services which are announced and available in your country can be ordered under the applicable standard agreements, terms, conditions, and prices in effect at the time. IBM reserves the right to modify or withdraw this announcement at any time without notice. This announcement is provided for your information only. Reference to other products in this announcement does not necessarily imply those products are announced, or intend to be announced, in your country. Additional terms of use are located at

For the most current information regarding IBM products, consult your IBM representative or reseller, or visit the IBM worldwide contacts page

IBM Directory of worldwide contacts