With the availability of the IBM Storage Driver for OpenStack, the IBM DS8870 can offer a range of capabilities that enable more effective storage automation deployments in to private or public clouds. The driver is an IBM proprietary solution supporting OpenStack Block Storage on top of the OpenStack and Cinder open source technologies.
Enabling OpenStack with DS8870 allows for storage to be made available whenever it is needed without the traditional associated cost of highly skilled administrators and infrastructure
You can learn about the implementation and see illustrated examples of the supported functions, in the IBMRedBooks publication:
The functions currently supported in Cinder with the IBM Storage Driver for OpenStack and DS8870 are listed hereafter:
Create Volume to create a DS8870 volume.
Delete Volume to delete a DS8870 volume.
Attach Volume to attach a DS8870 volume to an instance (host) and includes the management of the internal DS8870 LUN mapping.
Detach Volume to detach a DS8870 volume from an instance (host).
Volume Retype to change the volume type of a existing DS8870 volume. As retype a volume from non-replicated to replicated will create the IBM Metro Mirror relationship. The other way around will drop this relationship.
Snapshot / Remote copy operations
Create Snapshot to create a snapshot of a DS8870 volume. Creating snapshots uses the IBM FlashCopy® functionality.
Delete Snapshot to delete a snapshot of a DS8870 volume.
List Snapshots to list snapshots of DS8870 volumes.
Create Volume from Snapshot to create a DS8870 volume from an existing snapshot by utilizing the IBM FlashCopy functionality.
Create Volume from Image to create a DS8870 volume out of an image. DS8870 will create a volume and the OpenStack Image Service (Glance) provides the data of the image.
Create Image from Volume to create an image out of a DS8870 volume. This action is also performed by the OpenStack Image Service (Glance) service.
Create Volume from Volume (Clone) to create a new DS8870 volume from another already existing volume. This is also supported by the DS8870 internal IBM FlashCopy function when the source volume already resides on the same DS8870.
Volume Replication to replicate DS8870 volumes. In OpenStack, replication can be either host based replication or it can be performed by the backend storage. IBM DS8870, as storage backend for OpenStack, supports the synchronous replication method, IBM Metro Mirror.
Consistency Groups for Snapshots to create a consistent snapshot if data is spread over multiple DS8870 volumes. Therefore internal IBM FlashCopy operations are also used.
From inception, what is known today as IBM z Systems always had a close and unique relation to its storage. Over the years. improvements to the z Systems processors and storage software, the disk storage systems and their communication architecture have constantly reinforced the synergy.
We have compiled in a new IBM RedPaper publication, the various aspects, advanced functions and technologies often pioneered by IBM, that make the IBM z Systems and the IBM DS8870 an ideal combination.
A major outcome of that synergy is obviously the impact on performance improvement and enhanced resiliency of an overall mainframe storage infrastructure.
To name a few, the paper reviews topics such as:
Parallel Access Volumes and HyperPAVs
I/O priority queuing
MIDAWs (Modified Indirect Data Access Word)
Caching optimized for z System
IO Priority Manager and Work Load Manager (WLM) for z/OS
High Performance FICON for IBM z Systems (zHPF)
It also includes a discussion of the latest features enabled by the availability of 16 GFC Host Adapter for the DS8870 and z13 I/O enhancements, including:
Forward Error Correction (FEC)
Read Diagnostic Parameters (RDP)
FICON Dynamic Routing (FIDR)
Fabric IO Priority
For a comprehensive overview read IBM RedPaper , IBM DS8870 and z Systems Synergy , REDP-5186.
When I started to write this blog to share the highlights of the OpenStack Summit May 2015 presentation about Docker containers being a good platform for Big Data analytics I asked some of my peers if they new about the Thrilla in Manila eye catcher. 100% of my very small data sample said they had no idea. In 1975 Muhammad Ali and Joe Frazier fought their third boxing match for the world championship. This epic fight was held in Manila and Muhammad Ali used rhymes to make his opponents mad. The Thrilla in Manila was from one of his rhymes.
Also, in 1975 the IBM 3350 Direct Access Storage Facility with the advanced disk technology known as "Winchester." had a Head Disk Assembly (HDA) storage capacity of 317,498,850 bytes. Fast forward 40 years and now we are talking about OpenStack Manila: the new OpenStack file share service along with IBM Spectrum ScaleTM the high performance, scale-out, clustered file system that can span Yottabytes (1 Yottabyte = 1 000 000 000 000 000 000 000 000 bytes!) of storage.
The presentation describes the use case of 30+ universities using SuperVessel Cloud built on the IBM Power platform with advanced technologies like OpenStack Manila, Docker, Spark/Hadoop, Big data Analytics, Cognitive Computing and Spectrum Scale.
Docker containers are an ideal platform for the Big Data analytics use case because of their unique security and performance advantages. Big Data work loads typically use distributed file systems to provide shared access to both input and output data sets. OpenStack Manila is an excellent way to manage distributed file systems in
the Cloud environment. Applications like Spark and Hadoop add the analytics engine to complete the picture. In this talk, the team describes how they use Manila, Nova-docker and the distributed file system to allow Big Data clients in OpenStack, distributed across various compute nodes, to access large amounts of data in a shared, load-balanced and multi-tenant fashion. The presentation includes the experiences in building and deploying this cloud service, in use today by 30+ universities, and shows a brief demonstration: https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/big-data-analytics-and-docker-the-thrilla-in-manila
The presentation includes:
What is the use case?
What are the building blocks?
How does everything fit together?
Demo showing what the university students see
Challenges faced by integrating all of the components inside OpenStack and lessons learned
The two figures from the presentation highlight the Spectrum Scale architecture vision supported by Openstack components and the overall OpenPOWER SuperVessel Unified Management Platform with Spectrum Scale as a single scale-out data plane. Take a look at the presentation for the complete Thrilla in Manilla.
Figure 1 shows the IBM vision for the high performance, scale-out, clustered file system Spectrum Scale
The video presentation explains how Spectrum Scale and Manila are integrated with the Spectrum Scale Manila driver.
Figure 2 architecture describes how everything fits together. For this project the pieces are:
IBM Spectrum Scale
Figure 2 shows how Spectrum Scale fits in to the larger architecture of the OpenPOWER-SuperVessel Unified Management Platform solution with the OpenStack components.
I was reviewing some of the IBM Spectrum Scale™ related presentations from the May 2015 OpenStack Summit in Vancouver, Canada and thought I would highlight this session From Archive to Insight: Debunking Myths of Analytics on Object Stores. Spoiler alert: Here is the link to the session if you want the myths to be a surprise.
Background for the session: Many people believe running analytics on a Swift object store has to be a painful experience. Either the analysis can be run directly on the Swift cluster using specialized architectures, which never match native performance, or the data must be migrated out into a separate analytics cluster, delaying time to insight.
Here are the myths that are addressed in the 30 minutes session.
Myth 1: Data must migrate from Swift to HDFS
Myth 2: Swift should only be used with in-memory analytics (Spark)
Myth 3: Swift cannot efficiently support frameworks such as Hive and HBase that require appending to a file
Myth 4: Object Stores are slow for analytics
The session steps you through a demonstration that addresses each of the myths. Here is the diagram of the system used for the demonstration.
IBM FlashSystem® V9000 is a comprehensive all-flash high-speed enterprise storage solution that delivers the full capabilities of IBM FlashCore™ technology (Thesecret sauce). It delivers microsecond response times plus a rich set of software-defined storage features including: IBM Real-time Compression™, dynamic tiering, thin provisioning, snapshots, cloning, replication, data copy services and high-availability configurations.
It introduces you to IBM FlashSystem V9000 and provides in-depth knowledge of the product architecture, software and hardware, its implementation, and hints and tips. It provides use cases that demonstrate real-world solutions, as well as examples of the benefits gained by integrating FlashSystem storage into business environments.
IBM FlashCore™ technology is built on hardware accelerated I/O, advanced flash management, and IBM MicroLatency® modules with IBM-enhanced Micron technology.
Also part of the sauce is the integrated management in a FlashSystem V9000 scalable environment of components using Fibre Channel connections: All manageable from one central V9000 GUI.
A FlashSystem V9000 fixed or scalable building block consists of:
Two FlashSystem V9000 Control Enclosures
One FlashSystem V9000 Storage Enclosure
The FlashSystem V9000 scalable solution provides scalable performance. You can scale-up and scale-out independently, up to 2.5 million IOPS and 19.2 GB/s bandwidth, with deterministic IBM MicroLatency, and Quality of Service.
The scalable building blocks also allow for for the addition of up to 4 individual FlashSystem Storage Enclosures to be added to the storage system. If 228 TB from 4 building blocks isn't enough capacity, up to 4 additional Storage Enclosures (SEs) can then be added.
In total, a FlashSystem V9000 Storage System can contain a maximum of 8 FlashSystem V9000 Storage Enclosures, offering a potential storage capacity of 456TB, and up to 2.2 PB effective capacity is available at 80% compression. Real-time Compression is available as a software feature that allows users to elect to deploy Real-time Compression where it is desired.
The following image illustrates the scalable capacity of FlashSystem V9000 . It also shows that additional SEs can be added to a single Building Block, as well as to 2, 3, or 4 Building Blocks.
Read IBM Redbooks publication; Introducing and Implementing IBM FlashSystem V9000 for how-to guidance and examples to effectively deploy and integrate IBM FlashSystem V9000 into your enterprise environment. Topics include Encryption, Concurrent Code Load and Concurrent Maintenance, Performance guidelines and more.
Available since March 13, 2015, IBM Spectrum Accelerate is a key member of the IBM Spectrum Storage portfolio.
IBM Spectrum Accelerate enables rapid deployment of block data storage across existing or purpose-built data infrastructure.
IBM Spectrum Accelerate runs as a virtual machine on vSphere ESXi hypervisor, enabling you to build a server-based SAN from commodity hardware that includes x86 servers, Ethernet switches, SSDs, and direct-attached, high-density disks.
Spectrum Accelerate essentially acts as an operating system for your self-built SAN storage, grouping virtual nodes and spreading the data across the entire grid.
To get detailed technical information on IBM Spectrum Accelerate and learn how to deploy and use its many features, download the IBM RedBooks publication:
In a recent posting , we introduced the upcoming support for VMware Virtual Volumes (VVOL) with the IBM XIV Storage System. With the announcement of vSphere 6.0, VMware also officially released support for the vSphere Virtual Volumes (VVOL) architecture
Since March 12 , 2015 , the VVOL and XIV integration is effectively supported through IBM Spectrum Control Base, and based on the VMware API for Storage Awareness (VASA). IBM Spectrum Control Base is a follow on to the IBM Storage Integration Server.
Here are 5 interesting facts to know about IBM FlashSystem 900:
1. Implement FlashSystem 900 when speed is critical.
IBM FlashSystem 900 is designed to accelerate the applications that drive business. Powered by IBM FlashCore Technology, FlashSystem 900 delivers high performance at lower cost:
90us/155us read/write latency
Up to 1.1 million random read 4 K IOPS
Up to 10 GB/second read bandwidth
2. Employ FlashSystem 900 for high capacity business needs.
IBM FlashSystem 900 has 12 hot-swappable IBM MicroLatency™ storage modules: 1.2 TB. 2.9 TB, and 5.7 TB IBM. This is a 40% increase in capacity per module with up to 10% lower cost per TB over the IBM FlashSystem 840.
3. FlashSystem 900 provides higher density and the same or greater performance than its predecessor.
IBM FlashSystem 900 employs 20nm MLC chips with IBM-enhanced Micron MLC technology for higher storage density and improved endurance.
4. FlashSystem 900 is highly scalable.
FlashSystem 900 is configurable 2.4 - 57 TB of capacity for increased flexibility. Many granular capacity points are possible due to the 3 choices in flash module capacity:
Using feature code AF23 (1.2 TB)
Start with 2.4 TB and grow to 4.8 TB 7.2 TB 9.6 TB, or 12 TB
Using feature code AF24 (2.9 TB)
Start with 5.8 TB and grow to 11.6 TB, 17.4 TB, 23.2 TB, or 29 TB
Using AF25 (5.7 TB)
Start with 11.4 TB and grow to 22.8 TB, 34.2 TB, 45.6 TB, or 57 TB
5. FlashSystem 900 is easy to integrate in VMware environments.
FlashSystem 900 is easy to integrate with VMware VASA via IBM Storage Integrated Server to exploit the following features:
Greater communication between vSphere and FlashSystem
Ability of vSphere to monitor and directly manage FlashSystem allowing greater efficiencies
Integration of VASA Unmap for greater storage efficiency
For more details on the new IBM FlashSystem 900, powered by IBM FlashCore technology, read the IBM Redbooks Product Guide : IBM FlashSystem 900 TIPS1261
At the beginning of the year I blogged about the updates to the LTFS EE V188.8.131.52 Redbooks pub. that included support for Elastic Storage 4.1. IBM has announced the IBM Spectrum StorageTM Family. An important part of the announcement is that LTFS provides the functionality for IBM Storage Archive™ and IBM Storage Scale™ is based upon IBM General Parallel File System™ or GPFS, also formerly known as code name Elastic Storage. The LTFS EE V184.108.40.206 changes were key to building a stronger connection between Spectrum Archive and Spectrum Scale which opens up new opportunities for tape as tier in Software Defined Environments.
IBM Spectrum Archive™ enables you to automatically move infrequently accessed data from disk to tape to lower costs while retaining ease of use and without the need for proprietary tape applications. The functionality of IBM Spectrum Archive is provided by IBM Linear Tape File System.
IBM Spectrum Scale™ is flash accelerated, industrial strength, highly scalable software defined storage that enables global shared access to data with extreme scalability and agility for cloud and analytics.
IBM Linear Tape File System Enterprise Edition V220.127.116.11 Installation and Configuration Guide Redbooks Publication published January 29. This edition is only two months after the final publication of the V18.104.22.168. Spectrum Archive Enterprise Edition (formerly LTFS EE) now supports Spectrum Scale and the new TS1150 tape drive. The TS1150 tape drive with JD/JZ cartridges (native capacity of 10 TB (.9 TiB)), 8 Gbps Fibre connection speed, and 360 MB/s native data rate will make your tape tier in the cloud faster and denser.
This revision reflects the addition, deletion, or modification of new and changed information described below.
January 2015, Third Edition
Elastic Storage 4.1 now IBM Spectrum Scale
IBM TS1150 tape drive and media
Added -u option to reconcile command to skip pretest to check the necessity to reconcile before mounting the tapes
When FlashSystem V840 Real-time Compression is used on a compressible workload, the resulting solution has up to a 5:1 reduction in cost and capacity, while maintaining up to a 5x reduction in latency compared to disk drive based systems. Redpaper Accelerate with FlashSystem V840 Compression discusses “In a nutshell” how to effectively implement FlashSystem V840 with Real-time Compression (RtC). It walks you through planning, set up and operations, and provides performance guidance needed to exploit V840 performance and capacity savings. Candidate data sets and workloads best suited for compression are discussed. Examples are also provided on using the Comprestimator Utility to estimate expected compression rates for V840 with various workloads.
The Operations and analysis chapter explains the V840 7.4 software stack, performance monitoring, using synthetic workloads with RtC, V840 with RtC compared with disk, as well as topics on analysis and verification.
It is important to understand the FlashSystem V840 software stack and the flow of read and write requests as it pertains to Real-time Compression. The FlashSystem V840 software stack is shown in the figure below. Compression is transparently integrated with existing system management design. All of the FlashSystem V840 advanced features are supported on compressed volumes. You can create, delete, migrate, map (assign), and unmap (unassign) a compressed volume as though it were a fully allocated volume. In addition, you can utilize Real-time Compression along with Easy Tier on the same volumes. This compression method provides non disruptive conversion between compressed and decompressed volumes. This conversion provides a uniform user-experience and eliminates the need for special procedures when dealing with compressed volumes.
The Real-time Compression software component sits below the upper-level fast write cache and above the lower-level advanced read/write cache. There are several advantages to this dual-level model regarding Real-time Compression:
Host writes, whether to compressed or decompressed volumes, are still serviced directly via the upper-level write cache, preserving low host write I/O latency. Response time can improve with this model as the upper cache flushes less data to Real-time Compression component more frequently.
The performance of the destaging of compressed write I/Os to storage is improved because these I/Os are now destaged via the advanced lower-level cached, as opposed to directly to storage.
The existence of a lower-level write cache below the Real-time Compression component in the software stack allows for the coalescing of compressed writes, and as a result, a reduction in back-end I/Os due to the ability to perform full-stride writes for compressed data.
The existence of a lower-level read cache below the Real-time Compression component in the software stack allows the temporal locality nature of Real-time Compression to benefit from pre-fetching from the backend storage.
The main (lower level) cache stores compressed data for compressed volumes, increasing the effective size of the lower-level cache.
Read the publicly available draft of Redpaper, Accelerate with FlashSystem V840 Compression for more detail and use cases that demonstrate how to implement and immediately start gaining the benefits of FlashSystem V840 with Real-time Compression.
IBM SmartCloud Virtual Storage Center (VSC) is a key component of the IBM Software Defined Storage (SDS) Control Plane. It provides efficient virtualization, management of heterogeneous storage systems, end-to-end storage management, and data protection through application aware snapshots. These capabilities are all tightly integrated with advanced analytics functions such as optimization and provisioning to deliver a robust storage cloud-based solution The newly published IBM Redbook, IBM SmartCloud Virtual Storage Center provides practical use cases to assist clients in transforming traditional storage into storage cloud environments.
One of many step-by-step topics in this book demonstrates how VSC can be used to give VMware administrators the capability to provision storage with minimal interaction of the storage administrator.
As of IBM SmartCloud Virtual Storage Center V5.2, there are two functions that will help VMware administrators work with the data that VSC Storage Management has collected in the environment. The functions can be used independently:
VMware vCenter Web Client Extension: Tivoli Storage Productivity Center has created an extension that enables VMware administrators to view end-to-end information about storage and fabrics in new reports. The extension allows them to start the Tivoli Storage Productivity Center provisioning task from the VMware Web Client. This extension is also referred to as the Tivoli Storage Productivity Center Plug-in.
vSphere API for Storage Awareness (VASA): The Tivoli Storage Productivity Center VASA provider enables storage data to be accessible in the existing vCenter Server reports and views, as well as receive Tivoli Storage Productivity Center alerts.
Through the VMware vCenter Web Client Extension it is possible to allow VMware administrators to provision their own storage, which assists them in achieving even faster response times when demands for new machines arrive. The VMware vCenter Web Client extension also gives VMware administrators access to see performance and asset related information about the storage usage, enabling them to better understand the environment, and to optimize the resources they use.
VMware VASA integration with IBM Storage Systems
This integration enables VMware administrators to independently and centrally manage their storage resources on IBM storage systems. These resources include IBM SAN Volume Controller (SVC), IBM Storwize products; V7000, V5000, V3700; and FlashSystem V840. The Tivoli Storage Productivity Center Plug-in runs as a Microsoft Windows Server service on the vCenter server. Any VMware vSphere Client that connects to the vCenter server detects the service on the server. The service then automatically enables the IBM storage management features on the vSphere Client.
VSC and VMware are integrated using the Tivoli Storage Productivity Center plug-in. In the diagram below, we see the following integrated activities:
Probes: Tivoli Storage Productivity Center collects information about VMware data sources.
Reports and Provisioning: Function of the Tivoli Storage Productivity Center Plug-in.
Storage Capabilities and Events/Alarm:VMware vSphere API for Storage Awareness (VASA) functions.
VSC and VMware integration topology
The arrows in in the above diagram show where a connection is started. For example a probe is started from Tivoli Storage Productivity Center to the vCenter or to an ESX server. The box labeled with VMware vCenter Storage Monitoring Service enables the communication with a VASA provider. Events and Alarms are VMware terms, which correspond to alerts in Tivoli Storage Productivity Center.
Want to learn more? Read the IBM Redbooks publication; IBM SmartCloud Virtual Storage Center for a wealth of practical use cases that provide step-by-step instruction and demonstrate storage management, provisioning, optimizing, application aware data protection, and how to easily transform your traditional storage to storage cloud.
The updated version of A Deployment Guide for Elastic Storage Object, REDP5113 is available. There is now an Install tool to help streamline the deployment of your Elastic Storage Object. The new information highlights to the Redpaper are:
Introduced a new automated installation procedure using the Chef configuration management toolset
Described in more detail the steps that are required to back up your object store configuration and data to protect against disaster scenarios and how to recover from that saved data
Tip: Access to the install tool is described in the Redpaper.
If you have updated your storage capacity with the new TS4500 tape library that can hold up to 5.5 PBs of storage in one frame, you can now monitor it from your iphone. You will need to install IBM Storage Mobile Dashboard from the app store or update the app if you already have it installed.
The TS4500 tape library is now supported as shown in the list below under IBM Storage Mobile Dashboard. Select the cog wheel icon (in the upper left of the display) --> Systems --> Add System to get the log in options for your TS4500.
The free Quick Reference for IBM System Storage mobile app, now available for iPhone, Android and Blackberry, is designed to let you easily access the latest storage product information, success stories and Storage experts. Here is the link to the itunes preview for the IBM System Storage app.
IBM Storage Mobile Dashboard is a free application that provides basic monitoring capabilities for IBM storage systems. Storage administrators can securely check the health and performance status of their IBM Storage systems by viewing events as well as real time performance metrics.
Supported storage platforms:
• IBM Storwize V7000/V7000 Unified
• IBM Flex System V7000 Storage Node
• IBM Storwize V5000, V3700, V3500
• IBM SAN Volume Controller (SVC)
• IBM FlashSystem
• IBM XIV Storage System
• IBM TS4500 Tape Library
The numbers with the red background identify how many alerts there are on that device. Tap the number and it will show you the alerts.
Tip: Make sure you have SSL enabled on your TS4500.
The TS4500, IBM delivers the density that today’s and tomorrow’s data growth requires—along with the cost efficiency and the manageability to grow with business data needs while preserving existing investments in IBM tape library products. Now, organizations can achieve both a low cost per terabyte (TB) and a high TB density per square foot. In fact, TS4500 can store up to 5.5 PBs of data in a single 10 square foot library frame, more than three times more capacity than the IBM TS3500 tape library. Here an image of the minimum and maximum TS4500 tape library configurations.
If your data center is running out space the TS4500 tape library with TS1150 tape drives can give you tremendous floor space savings to handle storing Big Data files, archives and backups.
LTFS EE supports the TS4500 tape library with TS1150 tape drives.