I was reviewing some of the IBM Spectrum Scale™ related presentations from the May 2015 OpenStack Summit in Vancouver, Canada and thought I would highlight this session From Archive to Insight: Debunking Myths of Analytics on Object Stores. Spoiler alert: Here is the link to the session if you want the myths to be a surprise.
Background for the session: Many people believe running analytics on a Swift object store has to be a painful experience. Either the analysis can be run directly on the Swift cluster using specialized architectures, which never match native performance, or the data must be migrated out into a separate analytics cluster, delaying time to insight.
Here are the myths that are addressed in the 30 minutes session.
Myth 1: Data must migrate from Swift to HDFS
Myth 2: Swift should only be used with in-memory analytics (Spark)
Myth 3: Swift cannot efficiently support frameworks such as Hive and HBase that require appending to a file
Myth 4: Object Stores are slow for analytics
The session steps you through a demonstration that addresses each of the myths. Here is the diagram of the system used for the demonstration.
IBM FlashSystem® V9000 is a comprehensive all-flash high-speed enterprise storage solution that delivers the full capabilities of IBM FlashCore™ technology (Thesecret sauce). It delivers microsecond response times plus a rich set of software-defined storage features including: IBM Real-time Compression™, dynamic tiering, thin provisioning, snapshots, cloning, replication, data copy services and high-availability configurations.
It introduces you to IBM FlashSystem V9000 and provides in-depth knowledge of the product architecture, software and hardware, its implementation, and hints and tips. It provides use cases that demonstrate real-world solutions, as well as examples of the benefits gained by integrating FlashSystem storage into business environments.
IBM FlashCore™ technology is built on hardware accelerated I/O, advanced flash management, and IBM MicroLatency® modules with IBM-enhanced Micron technology.
Also part of the sauce is the integrated management in a FlashSystem V9000 scalable environment of components using Fibre Channel connections: All manageable from one central V9000 GUI.
A FlashSystem V9000 fixed or scalable building block consists of:
Two FlashSystem V9000 Control Enclosures
One FlashSystem V9000 Storage Enclosure
The FlashSystem V9000 scalable solution provides scalable performance. You can scale-up and scale-out independently, up to 2.5 million IOPS and 19.2 GB/s bandwidth, with deterministic IBM MicroLatency, and Quality of Service.
The scalable building blocks also allow for for the addition of up to 4 individual FlashSystem Storage Enclosures to be added to the storage system. If 228 TB from 4 building blocks isn't enough capacity, up to 4 additional Storage Enclosures (SEs) can then be added.
In total, a FlashSystem V9000 Storage System can contain a maximum of 8 FlashSystem V9000 Storage Enclosures, offering a potential storage capacity of 456TB, and up to 2.2 PB effective capacity is available at 80% compression. Real-time Compression is available as a software feature that allows users to elect to deploy Real-time Compression where it is desired.
The following image illustrates the scalable capacity of FlashSystem V9000 . It also shows that additional SEs can be added to a single Building Block, as well as to 2, 3, or 4 Building Blocks.
Read IBM Redbooks publication; Introducing and Implementing IBM FlashSystem V9000 for how-to guidance and examples to effectively deploy and integrate IBM FlashSystem V9000 into your enterprise environment. Topics include Encryption, Concurrent Code Load and Concurrent Maintenance, Performance guidelines and more.
Available since March 13, 2015, IBM Spectrum Accelerate is a key member of the IBM Spectrum Storage portfolio.
IBM Spectrum Accelerate enables rapid deployment of block data storage across existing or purpose-built data infrastructure.
IBM Spectrum Accelerate runs as a virtual machine on vSphere ESXi hypervisor, enabling you to build a server-based SAN from commodity hardware that includes x86 servers, Ethernet switches, SSDs, and direct-attached, high-density disks.
Spectrum Accelerate essentially acts as an operating system for your self-built SAN storage, grouping virtual nodes and spreading the data across the entire grid.
To get detailed technical information on IBM Spectrum Accelerate and learn how to deploy and use its many features, download the IBM RedBooks publication:
In a recent posting , we introduced the upcoming support for VMware Virtual Volumes (VVOL) with the IBM XIV Storage System. With the announcement of vSphere 6.0, VMware also officially released support for the vSphere Virtual Volumes (VVOL) architecture
Since March 12 , 2015 , the VVOL and XIV integration is effectively supported through IBM Spectrum Control Base, and based on the VMware API for Storage Awareness (VASA). IBM Spectrum Control Base is a follow on to the IBM Storage Integration Server.
Here are 5 interesting facts to know about IBM FlashSystem 900:
1. Implement FlashSystem 900 when speed is critical.
IBM FlashSystem 900 is designed to accelerate the applications that drive business. Powered by IBM FlashCore Technology, FlashSystem 900 delivers high performance at lower cost:
90us/155us read/write latency
Up to 1.1 million random read 4 K IOPS
Up to 10 GB/second read bandwidth
2. Employ FlashSystem 900 for high capacity business needs.
IBM FlashSystem 900 has 12 hot-swappable IBM MicroLatency™ storage modules: 1.2 TB. 2.9 TB, and 5.7 TB IBM. This is a 40% increase in capacity per module with up to 10% lower cost per TB over the IBM FlashSystem 840.
3. FlashSystem 900 provides higher density and the same or greater performance than its predecessor.
IBM FlashSystem 900 employs 20nm MLC chips with IBM-enhanced Micron MLC technology for higher storage density and improved endurance.
4. FlashSystem 900 is highly scalable.
FlashSystem 900 is configurable 2.4 - 57 TB of capacity for increased flexibility. Many granular capacity points are possible due to the 3 choices in flash module capacity:
Using feature code AF23 (1.2 TB)
Start with 2.4 TB and grow to 4.8 TB 7.2 TB 9.6 TB, or 12 TB
Using feature code AF24 (2.9 TB)
Start with 5.8 TB and grow to 11.6 TB, 17.4 TB, 23.2 TB, or 29 TB
Using AF25 (5.7 TB)
Start with 11.4 TB and grow to 22.8 TB, 34.2 TB, 45.6 TB, or 57 TB
5. FlashSystem 900 is easy to integrate in VMware environments.
FlashSystem 900 is easy to integrate with VMware VASA via IBM Storage Integrated Server to exploit the following features:
Greater communication between vSphere and FlashSystem
Ability of vSphere to monitor and directly manage FlashSystem allowing greater efficiencies
Integration of VASA Unmap for greater storage efficiency
For more details on the new IBM FlashSystem 900, powered by IBM FlashCore technology, read the IBM Redbooks Product Guide : IBM FlashSystem 900 TIPS1261
At the beginning of the year I blogged about the updates to the LTFS EE V18.104.22.168 Redbooks pub. that included support for Elastic Storage 4.1. IBM has announced the IBM Spectrum StorageTM Family. An important part of the announcement is that LTFS provides the functionality for IBM Storage Archive™ and IBM Storage Scale™ is based upon IBM General Parallel File System™ or GPFS, also formerly known as code name Elastic Storage. The LTFS EE V22.214.171.124 changes were key to building a stronger connection between Spectrum Archive and Spectrum Scale which opens up new opportunities for tape as tier in Software Defined Environments.
IBM Spectrum Archive™ enables you to automatically move infrequently accessed data from disk to tape to lower costs while retaining ease of use and without the need for proprietary tape applications. The functionality of IBM Spectrum Archive is provided by IBM Linear Tape File System.
IBM Spectrum Scale™ is flash accelerated, industrial strength, highly scalable software defined storage that enables global shared access to data with extreme scalability and agility for cloud and analytics.
IBM Linear Tape File System Enterprise Edition V126.96.36.199 Installation and Configuration Guide Redbooks Publication published January 29. This edition is only two months after the final publication of the V188.8.131.52. Spectrum Archive Enterprise Edition (formerly LTFS EE) now supports Spectrum Scale and the new TS1150 tape drive. The TS1150 tape drive with JD/JZ cartridges (native capacity of 10 TB (.9 TiB)), 8 Gbps Fibre connection speed, and 360 MB/s native data rate will make your tape tier in the cloud faster and denser.
This revision reflects the addition, deletion, or modification of new and changed information described below.
January 2015, Third Edition
Elastic Storage 4.1 now IBM Spectrum Scale
IBM TS1150 tape drive and media
Added -u option to reconcile command to skip pretest to check the necessity to reconcile before mounting the tapes
When FlashSystem V840 Real-time Compression is used on a compressible workload, the resulting solution has up to a 5:1 reduction in cost and capacity, while maintaining up to a 5x reduction in latency compared to disk drive based systems. Redpaper Accelerate with FlashSystem V840 Compression discusses “In a nutshell” how to effectively implement FlashSystem V840 with Real-time Compression (RtC). It walks you through planning, set up and operations, and provides performance guidance needed to exploit V840 performance and capacity savings. Candidate data sets and workloads best suited for compression are discussed. Examples are also provided on using the Comprestimator Utility to estimate expected compression rates for V840 with various workloads.
The Operations and analysis chapter explains the V840 7.4 software stack, performance monitoring, using synthetic workloads with RtC, V840 with RtC compared with disk, as well as topics on analysis and verification.
It is important to understand the FlashSystem V840 software stack and the flow of read and write requests as it pertains to Real-time Compression. The FlashSystem V840 software stack is shown in the figure below. Compression is transparently integrated with existing system management design. All of the FlashSystem V840 advanced features are supported on compressed volumes. You can create, delete, migrate, map (assign), and unmap (unassign) a compressed volume as though it were a fully allocated volume. In addition, you can utilize Real-time Compression along with Easy Tier on the same volumes. This compression method provides non disruptive conversion between compressed and decompressed volumes. This conversion provides a uniform user-experience and eliminates the need for special procedures when dealing with compressed volumes.
The Real-time Compression software component sits below the upper-level fast write cache and above the lower-level advanced read/write cache. There are several advantages to this dual-level model regarding Real-time Compression:
Host writes, whether to compressed or decompressed volumes, are still serviced directly via the upper-level write cache, preserving low host write I/O latency. Response time can improve with this model as the upper cache flushes less data to Real-time Compression component more frequently.
The performance of the destaging of compressed write I/Os to storage is improved because these I/Os are now destaged via the advanced lower-level cached, as opposed to directly to storage.
The existence of a lower-level write cache below the Real-time Compression component in the software stack allows for the coalescing of compressed writes, and as a result, a reduction in back-end I/Os due to the ability to perform full-stride writes for compressed data.
The existence of a lower-level read cache below the Real-time Compression component in the software stack allows the temporal locality nature of Real-time Compression to benefit from pre-fetching from the backend storage.
The main (lower level) cache stores compressed data for compressed volumes, increasing the effective size of the lower-level cache.
Read the publicly available draft of Redpaper, Accelerate with FlashSystem V840 Compression for more detail and use cases that demonstrate how to implement and immediately start gaining the benefits of FlashSystem V840 with Real-time Compression.
IBM SmartCloud Virtual Storage Center (VSC) is a key component of the IBM Software Defined Storage (SDS) Control Plane. It provides efficient virtualization, management of heterogeneous storage systems, end-to-end storage management, and data protection through application aware snapshots. These capabilities are all tightly integrated with advanced analytics functions such as optimization and provisioning to deliver a robust storage cloud-based solution The newly published IBM Redbook, IBM SmartCloud Virtual Storage Center provides practical use cases to assist clients in transforming traditional storage into storage cloud environments.
One of many step-by-step topics in this book demonstrates how VSC can be used to give VMware administrators the capability to provision storage with minimal interaction of the storage administrator.
As of IBM SmartCloud Virtual Storage Center V5.2, there are two functions that will help VMware administrators work with the data that VSC Storage Management has collected in the environment. The functions can be used independently:
VMware vCenter Web Client Extension: Tivoli Storage Productivity Center has created an extension that enables VMware administrators to view end-to-end information about storage and fabrics in new reports. The extension allows them to start the Tivoli Storage Productivity Center provisioning task from the VMware Web Client. This extension is also referred to as the Tivoli Storage Productivity Center Plug-in.
vSphere API for Storage Awareness (VASA): The Tivoli Storage Productivity Center VASA provider enables storage data to be accessible in the existing vCenter Server reports and views, as well as receive Tivoli Storage Productivity Center alerts.
Through the VMware vCenter Web Client Extension it is possible to allow VMware administrators to provision their own storage, which assists them in achieving even faster response times when demands for new machines arrive. The VMware vCenter Web Client extension also gives VMware administrators access to see performance and asset related information about the storage usage, enabling them to better understand the environment, and to optimize the resources they use.
VMware VASA integration with IBM Storage Systems
This integration enables VMware administrators to independently and centrally manage their storage resources on IBM storage systems. These resources include IBM SAN Volume Controller (SVC), IBM Storwize products; V7000, V5000, V3700; and FlashSystem V840. The Tivoli Storage Productivity Center Plug-in runs as a Microsoft Windows Server service on the vCenter server. Any VMware vSphere Client that connects to the vCenter server detects the service on the server. The service then automatically enables the IBM storage management features on the vSphere Client.
VSC and VMware are integrated using the Tivoli Storage Productivity Center plug-in. In the diagram below, we see the following integrated activities:
Probes: Tivoli Storage Productivity Center collects information about VMware data sources.
Reports and Provisioning: Function of the Tivoli Storage Productivity Center Plug-in.
Storage Capabilities and Events/Alarm:VMware vSphere API for Storage Awareness (VASA) functions.
VSC and VMware integration topology
The arrows in in the above diagram show where a connection is started. For example a probe is started from Tivoli Storage Productivity Center to the vCenter or to an ESX server. The box labeled with VMware vCenter Storage Monitoring Service enables the communication with a VASA provider. Events and Alarms are VMware terms, which correspond to alerts in Tivoli Storage Productivity Center.
Want to learn more? Read the IBM Redbooks publication; IBM SmartCloud Virtual Storage Center for a wealth of practical use cases that provide step-by-step instruction and demonstrate storage management, provisioning, optimizing, application aware data protection, and how to easily transform your traditional storage to storage cloud.
The updated version of A Deployment Guide for Elastic Storage Object, REDP5113 is available. There is now an Install tool to help streamline the deployment of your Elastic Storage Object. The new information highlights to the Redpaper are:
Introduced a new automated installation procedure using the Chef configuration management toolset
Described in more detail the steps that are required to back up your object store configuration and data to protect against disaster scenarios and how to recover from that saved data
Tip: Access to the install tool is described in the Redpaper.
If you have updated your storage capacity with the new TS4500 tape library that can hold up to 5.5 PBs of storage in one frame, you can now monitor it from your iphone. You will need to install IBM Storage Mobile Dashboard from the app store or update the app if you already have it installed.
The TS4500 tape library is now supported as shown in the list below under IBM Storage Mobile Dashboard. Select the cog wheel icon (in the upper left of the display) --> Systems --> Add System to get the log in options for your TS4500.
The free Quick Reference for IBM System Storage mobile app, now available for iPhone, Android and Blackberry, is designed to let you easily access the latest storage product information, success stories and Storage experts. Here is the link to the itunes preview for the IBM System Storage app.
IBM Storage Mobile Dashboard is a free application that provides basic monitoring capabilities for IBM storage systems. Storage administrators can securely check the health and performance status of their IBM Storage systems by viewing events as well as real time performance metrics.
Supported storage platforms:
• IBM Storwize V7000/V7000 Unified
• IBM Flex System V7000 Storage Node
• IBM Storwize V5000, V3700, V3500
• IBM SAN Volume Controller (SVC)
• IBM FlashSystem
• IBM XIV Storage System
• IBM TS4500 Tape Library
The numbers with the red background identify how many alerts there are on that device. Tap the number and it will show you the alerts.
Tip: Make sure you have SSL enabled on your TS4500.
The TS4500, IBM delivers the density that today’s and tomorrow’s data growth requires—along with the cost efficiency and the manageability to grow with business data needs while preserving existing investments in IBM tape library products. Now, organizations can achieve both a low cost per terabyte (TB) and a high TB density per square foot. In fact, TS4500 can store up to 5.5 PBs of data in a single 10 square foot library frame, more than three times more capacity than the IBM TS3500 tape library. Here an image of the minimum and maximum TS4500 tape library configurations.
If your data center is running out space the TS4500 tape library with TS1150 tape drives can give you tremendous floor space savings to handle storing Big Data files, archives and backups.
LTFS EE supports the TS4500 tape library with TS1150 tape drives.
The IBM DS8870 Release 7.4 was made generally available on December 5th. Among several hardware and software enhancements, the DS8870 now supports Mutliple Target Peer-to-Peer Remote Copy.
IBM Multiple Target Peer-to-Peer Remote Copy (Multiple Target PPRC) enhances a multi-site disaster recovery environment by providing the capability to have two PPRC relationships on a single primary volume, giving the additional data protection of another remote site.
Multiple Target PPRC provides the following enhancements:
Mirrors data from a single local primary site to two remote secondary sites
Increased capability and flexibility for disaster recovery solutions using:
Combination of synchronous replication and asynchronous replication configurations
Improves a cascaded Metro/Global Mirror (MGM) configuration and simplifies recovery procedures
Enhanced migration capabilities
You can read all the details in the just released IBMRedbooks publication:
Virtualization technology is transforming businesses and organizations. Current storage capacity, performance, and Independent Software Vendor (ISV) integration requirements are growing faster than ever before while the infrastructure budget is shrinking in the same proportion.
IBM Redpaper REDP-5148 Deploying IBM FlashSystem V840 in a VMware Environment describes and demonstrates how to address IBM FlashSystem V840 business performance and capacity requirements by creating a Software Defined Storage (SDS) Control Plane to more efficiently integrate with VMware and Independent Software Vendor (ISV).
The IBM developed solution, IBM Storage Integration Server, addresses this requirement bringing automation, elasticity, Storage as a service capabilities and operations management for Storage management.
IBM Storage Integration Server provides the ability for IBM disk arrays and IBM FlashSystem V840 flash arrays to be automatically orchestrated and monitored in a cloud environment.
The IBM Storage Integration Server is a centralized server system that consolidates a range of IBM storage provisioning, virtualization, cloud, automation, and monitoring solutions through a unified server platform.
The IBM Storage Integration Server facilitates the integration of IBM Storage System resources by using options (user credential, storage system, and storage pool management) with supported independent software vendor (ISV) platforms and frameworks. It provides a foundation for integration with IBM systems and ISV solutions, and can be downloaded at no extra cost!
You can download the latest version at any time from the IBM Fix Central Website:
IBM Storage Integrated Server provides a single server back-end location and enables centralized management of IBM storage resources for different virtualization and cloud platforms, including:
VMware vCenter Server
VMware vSphere Web Client
VMware vSphere APIs for Storage Awareness (VASA)
As shown below, IBM Storage Integration Server works as an abstraction layer between the VMware environment and the FlashSystem V840, thereby consolidating and reducing the VMware components generated requests against the storage system:
This year there have been several tape related announcements that increase your enterprise storage capacity without increasing your storage floor space. The TS4500, IBM delivers the density that today’s and tomorrow’s data growth requires—along with the cost efficiency and the manageability to grow with business data needs while preserving existing investments in IBM tape library products. Now, organizations can achieve both a low cost per terabyte (TB) and a high TB density per square foot. In fact, TS4500 can store up to 5.5 PBs of data in a single 10 square foot library frame, more than three times more capacity than the IBM TS3500 tape library.
The IBM TS1150 tape drive gives organizations an easy way to deliver fast access to data, improve security and provide long-term retention—and for less expense than disk solutions. TS1150 offers high-performance, flexible data storage with support for data encryption. This fifth-generation drive can help protect investments in tape automation by offering compatibility with existing automation. Plus, an upgrade model is available for existing IBM TS1140 tape drives. What’s more, TS1150 supports the LTFS format for direct, intuitive and graphical access to data. Up to 7 TB native capacity on an IBM 3592 JC/JY cartridge with a data transfer rate up to 700 MBps with compression.
IBM Linear Tape File System Enterprise Edition (LTFS EE) gives organizations an easy way to use cost-effective IBM tape drives and libraries within a tiered storage infrastructure. By using tape libraries instead of disks for Tier 2 and Tier 3 data storage—data that is stored for long-term retention—organizations can improve efficiency and reduce costs. In addition, LTFS EE seamlessly integrates with the scalability, manageability and performance of IBM Elastic Storage, an IBM enterprise file management platform that enables organizations to move beyond simply adding storage—to optimizing data management.
Take a look at the IBM Tape storage web page (shown in the figure below) and look at the how the higher density enterprise tape drive with the TS4500 can lower your costs by replacing your Tier 2 and Tier 3 disk storage with the scalability, manageability and performance of IBM Elastic Storage. When your space is constrained and your energy costs are growing or not able to expand using IBMs latest tape solutions with LTFS EE can solve your storage retention problems with a greener solution.
The IBM Storwize family of storage systems is ideal for organizations needing a robust and flexible technology infrastructure they can rely on, providing cloud-ready storage components that can access, manage, back up and archive the vast amounts of data generated by cloud workloads.
Organizations that hope to take full advantage of cloud benefits to address these imperatives need a robust and flexible technology infrastructure they can rely on, including a cloud-ready storage component that can address the need to access, manage, back up and archive the vast amounts of data that cloud workloads generate. They need look no further than the broad IBM portfolio of cloud-ready storage, including the IBM® Storwize® family of storage systems.
The Edison Group has created a white paper that compares IBM Real-time Compression for Storwize V7000 versus EMC and NetApp compression for Block Storage. This paper contains a detailed competitive evaluation of IBM’s Storwize V7000’s Real-time Compression versus EMC’s VNX and NetApp's FAS compression.
Of course, we are not accusing anyone of being a dummy per se ! However, this book examines data storage and management challenges and explains software-defined storage, an innovative solution for high-performance, cost-effective storage using Elastic Storage, based on IBM General Parallel File System (GPFS).
This Storage Switzerland paper examines how software defined storage extends the concept of storage virtualization beyond the mere aggregation of storage assets and basic volume management. Software defined storage adds enhanced capabilities like data protection, storage tiering and storage efficiency.
The next generation of software defined storage will evolve from proprietary services into a platform. This will continue to provide core data services like volume management, snapshots, data tiering but at the same time be open to additions from independent third party solutions such as Bridgeworks SANSlide.
Storage Switzerland explains how IBM Storwize family is a mature software defined storage platform that now is being enhanced with greater functionality to meet the needs of customers deploying virtualized infrastructures. They also example the benefits of adding third-party software to the platform and specifically how Bridgeworks SANSlide can help make remote replication more efficient.
As I type this we are in the final throes of updating the IBM Redbooks on the IBM Storwize V5000 and the IBM Storwize V3700 which we hope to have available by mid-December. As part of the research behind the writing of the book, we found this quite useful and it is worth a read.
Written by the Taneja Group, this analyst report delves into the Storwize V5000 and describes how IBM is integrating a broad swath of new capabilities while building those capabilities on a field proven and deeply architected storage foundation.