In honor of today's many holidays, I have some fix packs for you. These will be available by end of day March 21, 2017, US Eastern time.
- SLES users: we've become aware of a file system corruption issue with xfs on SLES 12, including fix packs 1 and 2, that (when it hits) leaves your guest unbootable. We recommend that you avoid using xfs in your captured images until SLES releases a fix and you re-capture them. Symptoms below.
- Liberty FP4: VM66002 contains a short list: it fixes the "applying service cripples ZHCP" problem you had to manually bypass with FP3, cleans up IP addresses properly after a new instance deployment request fails, and creates reliable drive mappings for application use (/dev/vdx drives will be consistent).
- Newton FP1: VM65996 is a more varied bunch, but at a high level: it adds Ubuntu boot-from-volume support, allows more varieties of guests to use IUCV instead of ssh (which makes deployments faster, and can simplify networking setup), improves the xCAT UI for customizing the automated Installation Verification Program (IVP) runs (think: on-going health checks), allows you to deploy guests when the instance name template expands to <8 characters, and reduces log file consumption inside of CMA (reducing the likelihood of the file system filling up).
SLES xfs corruption details:
- Our testers have intermittently reproduced a xfs metadata corruption problem on our test systems when deploying SLES 12. We have opened a problem with SUSE.
- It appears to be a timing issue (race condition) in shutdown processing. The more often your guests are re-booted, the more likely they are to experience this.
- Once the xfs metadata has been corrupted, the next boot will fail, leaving messages like the following on the boot console:
Modified on by JohnArwe
The PTFs for z/VM's OpenStack Newton support are expected to close by end of day January 20 US time ("today", for much of the world). Those PTFs will update z/VM's Cloud Manager Appliance (CMA) and other SMAPI servers, so there are several of them (details below).
If you're already running z/VM 6.4
- Congratulations, you're eligible to run Newton! Once you do, you get:
- Ubuntu 16.04 guest deployment support - boot from volume has been delayed, but should be coming soon. All other functions are ready.
- Substantially faster guest deployment times
- Substantially faster CMA restart time when running OpenStack services
- CMA configuration wizard - much simpler way to configure CMA, either new/from scratch or for incremental changes like adding another LVM disk
- Simpler CMA health monitoring (geeky details further down, for those interested)
- Guest consoles are readable through the OpenStack horizon dashboard GUI
- Existing 6.4 orders will have Liberty "in the box"/on DVD. New orders will have Newton. There is no supported way to down-grade when ordering or after receipt.
- Starting now, Liberty will receive only security fixes; Liberty fix pack 3 should be the last of any functional fixes.
- You have 6 months to upgrade to Newton; after July 2017, you should expect no further Liberty updates, not even security fixes. This is aligned with the expectations set in the Statement of Direction included in the z/VM 6.4 announcement materials.
If you're currently running z/VM 6.3
- Sorry, no Newton for you z/VM's Newton support runs only on z/VM 6.4.
- Gentle reminder, only 11 months left before 6.3 goes out of service! Hope you are building your migration plan.
- You should expect only security updates, no functional updates, for OpenStack Liberty from here on out. Unlike z/VM 6.4 though, you'll get those security updates until the end of 2017 when 6.3's support ends.
- In order to migrate to Newton:
- If you have data that you care about migrating, then you must install the latest CMA Liberty service first - either before or after migrating z/VM itself.
- If you have nothing you're worried about keeping, you can treat Newton as a new installation once you have migrated to 6.4.
- VM65893 updates CMA. The installation instructions are in the CMA140 FILE on MAINT 400 once you apply the APAR. Follow them, or you won't be running the code afterward, Promise!
- VM65954 updates other SMAPI servers and adds new files for the CMA configuration wizard.
- VM65955 updates SMAPI's caching server, LOHCOST
- As I'm writing this blog post, the APARs are orderable but the PTFs are not closed. That just means it may take a few hours to receive the PTFs, based on when you order and whether or not they're closed when you do order them.
- As usual, the z/VM OpenStack maintenance page will have a service list file that you can download and run the SERVICE command against to get the complete order list, including pre-requisites.
- VM65970 will come "soon" to provide help files for the new configuration wizard. No need to wait for this one to get started ordering the others though.
- PDF copies of the publications (Enabling Newton, SMAPI 6.4) are now available on the publibz site and (z/VM Migration Guide excerpts specific to CMA Newton) on the wiki. The full Migration Guide that incorporates the excerpts will come to publibz later; best current guess, early 2Q2017.
- The Knowledge Center publications will not be available immediately; best current guess, early 2Q2017 for those.
- There were some documentation updates that missed the deadline for the PDFs. We're using the wiki to make the really important (as in: not just editorial) ones available immediately. Fair warning, it will likely be several months before the PDFs are updated.
- Selected clarifications on the configuration wizard are also on the documentation updates page.
- Checksums for all the VM65893 files are on the wiki, so if you're a tin foil hat wearer like me you can verify that they're unscathed after all the download, upload, and unpacking commands. FTP is notoriously finicky, perfectly innocent-looking variations from CMA140's instructions have resulted in the wrong bits coming out the other end of the process, with no visible errors before that.
- If you're running a production-like environment, have a look at our tuning wiki page. We've learned things from other clients as well as our regression test system.
- Less obvious changes included in Newton:
- HTTPS is required by default now when calling OpenStack APIs. In Liberty (starting with FP2), you had to run a procedure to secure them.
- CMA uses IUCV instead of ssh to manage guests by default. Much more on this in the Migration Guide.
- The chef client has been removed, so CMA tends to use noticeably less virtual storage and CPU to run equivalent work. Scripts are installed to handle optional functions that used to require chef.
- Logging and serviceability improvements.
- Installation verification tests (IVP) are run when the CMA is first started, and are scheduled to run on an on-going basis so you have more time to react to conditions like DirMaint storage pools filling up, and any other "slow creep" problems. The xCAT GUI allows you to control the periodic runs.
- When an OpenStack-requested guest deployment fails because CMA was unable to connect to the guests before OpenStack times out, a CP MSG is sent to the notify userid to facilitate automated or live monitoring.
| Edited 15:30 ET 2017-01-20 ... PTFs closed, added VM65970, fix sentence fragment
| Edited 09:40 ET 2017-01-23 ... re-order words in one of the "no more functional fixes for Liberty" lines
We recently discovered a somewhat subtle conflation of "related but distinct" concepts in the Enabling [z/VM for OpenStack
(Support for OpenStack insert-your-favorite Release)] publication. I've gotten this corrected in the Newton GA pubs, but this applies to ALL earlier levels as well.
- An xCAT management node is capable of managing a set of z/VM systems. There's no specific name for this set; I'll call it a xCAT cluster in this post for consistency.
- A z/VM system, since 6.2, can be a member of a Single System Image (SSI) cluster.
In most cases (but not all, of course - that would be too simple), where Enabling talks about SSI it's really intending to talk about a xCAT cluster. The rare cases where SSI is intended are those like Live Guest Relocation (LGR), base z/VM function that's scoped to an SSI cluster regardless of how it's triggered. When a function like LGR is triggered through xCAT (whether manually, or via the xCAT REST API using scripts or OpenStack), then both concepts come into play.
That distinction can be important, of course; some installations don't use SSI, but they can still happily use xCAT clustering. Installations choosing to abstain from SSI won't be able to use z/VM LGR (or any other function that has dependencies on SSI), but they can use anything else that xCAT offers... including clustering. The moral is: don't read the pre-Newton Enabling references to SSI and conclude (since you don't use SSI) that it's off the table for you.
Once the Newton-level pubs are available ("sooner rather than later"), the simplest way to clarify things will be to look in those. Neither kind of cluster will change in Newton.
If you have a specific case in mind and the Newton pubs are not available yet, leave a comment or send me an email and I'll update this post to cover it.
Modified on by JohnArwe
If you're running xCAT and/or OpenStack, especially on z/VM 6.3, a set of new fixes is available.
- VM65953, if you're on the current code base ("CMA") ... this is Liberty Fix Pack 3, for anyone who's counting.
- VM65864, if you're on the old code base ("classic")
- You have a migration plan, right?
- You have less than 12 months left to move to the new code, where you get goodies like RHEL 7 and SLES 12 support.
- The code base will not run on z/VM 6.4. It's only getting critical security fixes at this point.
- If you're unsure which code you're running, link and access MAINT 193, open DMSSICMO COPY, and see what value you have in property openstack_system_role ... if it's the empty string (""), you're on the old code base, otherwise you're on the new one.
If you're running OpenStack, you should expect Newton support in the first quarter, and probably sooner rather than later. So if your change windows are limited and you're running on z/VM 6.4 already, you should consider skipping Liberty FP3 and upgrading to Newton when it arrives. Newton will only be supported on z/VM 6.4.
| Updated to add the nested bullet under VM65953.
| 2017-02-27 Added the red PE warning.
In case you're not into reading the full text of announcement letters, you might be interested in the following.
- The systems management code needs to move faster than the base hypervisor code. OpenStack, for example, releases a new version every six months. z/VM intends to ship every other one (one release per year), as it has been doing of late (Juno, Liberty, Newton, ...).
- Usually a given version of the systems management code, delivered via the Cloud Manager Appliance (CMA), only runs on a single z/VM release. Hence, eventually to get new function you may have to move to a new z/VM release. Example: to get Newton, you'll have to be on z/VM 6.4.
- Within a z/VM release, you'll have 6 months to upgrade to a new CMA release once it's available. During that time, the older release will only get security fixes. Example: CMA Liberty was available on z/VM 6.3 in March 2016, so CMA Juno gets security fixes only, and Juno only gets those security fixes until September 2016. After September 2016, Juno must be assumed to have unpatched vulnerabilities.
- Once a new z/VM release is available, the CMA running on the older level will only get security fixes, but will get them until end of service for the z/VM release. Example: once z/VM 6.4 is available, z/VM 6.3's (Liberty) CMA will only get security fixes, but it will get them for more than 6 months since the current end of service date for z/VM 6.3 is more than 6 months later.
Excerpts from the z/VM 6.4 announcement (links to the full letter for each geography are on the z/VM 6.4 news page). It should be the same regardless of geography, but if in doubt verify against the relevant full letter.
IBM OpenStack for z Systems. ...skip 2 paragraphs... italics mine
With the PTF for APAR VM65893, expected to be available first quarter 2017, the OpenStack Cloud Management Appliance (CMA) packaged with z/VM V6.4 will be enhanced to include the Newton level of OpenStack. Additional enhancements will include provisioning of Ubuntu guests, FlashCopy support for provisioning, and additional RAS and security support.
...skip 1 paragraph...
IBM intends to upgrade the z/VM OpenStack CMA to a new level approximately once per year. This new level of the CMA would be based on every other release of OpenStack. IBM intends to make the new CMA level available only on the most recent release of z/VM. After the CMA upgrade to a new level of OpenStack on a particular release of z/VM, the previous CMA level on that z/VM release will be supported only for another 6 months. During that 6 month period, the previous CMA level will receive only security fixes. Any functional bug fixes and enhancements will only be provided on the latest CMA level available on a z/VM release. After that 6 month period, no service will be provided for the previous CMA levels and users should upgrade to the latest CMA level.
For example, when the Newton level ships that will be only for z/VM V6.4 and there would not be an update for z/VM V6.3 for Newton. The Liberty level on z/VM V6.3 will continue to be supported until the end of service date for z/VM V6.3. On z/VM V6.4, when the Newton-based CMA becomes available, only security related fixes will be made available for Liberty and for only the next 6 months.
When a new release of z/VM is made available, IBM intends to freeze the level of the CMA available on the previous z/VM release. This frozen level of the CMA will receive only security fixes until service ends for that z/VM release.
Basically, the systems management code needs to move faster than the base hypervisor code.
Modified on by JohnArwe
APAR VM65881/PTF UM34873 is how you order the code. As usual, follow the installation instructions in the CMA120 FILE found on MAINT's 400 disk after applying the APAR.
Summary of content
- New function
- Increased security
- Passwords can have non-alphanumeric characters (anything aside from single and double quotes).
- SHA-256 used to sign certificates.
- Bug fixes (30-ish, in round numbers, many of which you'd have to be poring over log files to observe)
As usual, the easy way to deal with this is to use the SERVICE STATUS LIST command with the SERVICE files downloadable from the z/VM OpenStack service page (if you're running CMA).
If you're still running the non-OpenStack xCAT code base, "nothing to see here, move along"... but do have a migration plan. The older code base will never get new function, and before too long it won't get security fixes either. CMA120 FILE and pages on the wiki (see earlier blog posts too) contain information to help you migrate. Note that z/VM 6.3 is the last release of z/VM likely to run this older code base, too.
Modified on by JohnArwe
A quick doorknob post before I disappear into the Wind Rivers for my annual walkabout. (If you find pepper-spray-scented grizzly bear scat with bells in it someday, might be what's left of me ;-) ).
It's not easy being smart
Sometimes in our quest to be smart, or efficient, we complicate apparently simple things with non-obvious behavior. This shows through in at least these points when configuring a CMA:
- Admin passwords
- LVM disks
- Virtual switches
The key point in each case is that CMA has some memory, and it uses that memory during initialization. The contents of DMSSICNF and DMSSICMO COPY files are simply not the whole story in these cases.
The value of cmo_admin_password from CMO may (or may not) be the current password in the OpenStack UI(s) (plural to cover the Juno case, only). That property is used to give the admin password an initial value. Basically, it only gets used when all the bits inside the LVM first get initialized. Some of those places are human-facing, like the GUIs, and so we recommend you immediately change them at first logon. In effect, once you take our advice, the value of cmo_admin_password is irrelevant... it's just not used, change it all you like. It will only be used again if you format all the LVM disks (losing your images and other data, unless you backed it up - you do back up those volumes, right?).
Edited 2016-10-06 to add: In Liberty fix pack 2 and earlier, cmo_admin_password actually is still used in one place, and that's a bug to be fixed in a future fix pack. Every time CMA boots, its value is written into mnadmin's openrc file. Certain xCAT functions depend on the value in openrc being correct; so for now, update the admin password as above and, at the same time, update the new value in the openrc file and in cmo_admin_password.
- Change human-facing passwords once the UI is up.
- Manage those passwords like any others. If you forget them, you cannot use the cmo_admin_password property to update them unless you wipe all the LVM disks, which will have lots of other (probably worse) side effects.
The value of cmo_data_disks from CMO tells CMA which disks you want it to add to an OpenStack controller's LVM ... it is not the full list of disks currently part of the LVM, and removing a volume ID from this list will not remove it from the LVM. Once a volume is added to the LVM, it will be listed as an MDISK in the directory entry of the VM where OpenStack services run. The virtual device address used is a contiguous range starting with 0200. The volume will also have a region defined for it in DirMaint, and it will be added to the XCAT1 volume group.
- Removing a volume ID from cmo_data_disks will not remove it from the LVM.
- You cannot tell the full set of LVM disks in use by looking at CMO, if you have ever changed the value of cmo_data_disks.
The values of xcat_vswitch and xcat_mn_vswitch from CNF tells xCAT services which virtual switches to use, for "internal" and "external" access respectively. The associated values may or may not be used ... some are used only when creating the vswitches, some are also used to create networking interfaces on the virtual machine where xCAT services run. The xcat_vlan property, for example, falls into the first category. The xcat_addr property, on the other hand falls into the second. Since the interfaces are always created during xCAT services initialization but the vswitches are re-used if they already exist, they can get out of synch with what you see in DMSSICNF.
- xCAT services only create the virtual switches configured in xcat_vswitch and xcat_mn_vswitch when they do not already exist. When they already exist, xCAT services simply use them as-is.
- If you change the networking parameters in CNF, you may have to manually update the vswitches (if you've made other manual modifications). If you're letting xCAT services manage those vswitches, then you can destroy them and they'll be re-created using the then-current CNF property values the next time xCAT services initialize.
Obligatory xckd (a Juno joke, since I mentioned Juno above): http://xckd.com/1703/
If you're running xCAT on your z/VM system, you'll want one of the following:
- VM65834, if you're running CMA Liberty (you have this if VM65780 is applied already and DMSSICMO property openstack_system_role has a non-blank value)
- VM65833, if you're still running OpenStack Juno (you should upgrade, really ... Juno ONLY gets critical security fixes now)
- VM65835, if you're still running the non-OpenStack xCAT code base that came with z/VM 6.3 (you should upgrade, really ... this ONLY gets critical security fixes now)
As usual, the easy way to deal with this is to use the SERVICE STATUS LIST command with the SERVICE files downloadable from the z/VM OpenStack service page (if you're running CMA) or the z/VM xCAT service page (if you're still running the non-OpenStack xCAT code base), since that will show you all related APARs you need to order.
The bulk of the fixes are security-related, but there are several in the Liberty APAR related to migration and image capture that it are simply best avoided by pro-actively installing this service.
Modified on by JohnArwe
tl;dr: if you're running z/VM 6.3.0 with the XCAT or ZHCP virtual machines on your z196 or later system today, you should upgrade to CMA Liberty.
z/VM 6.3.0 refreshed its xCAT and OpenStack support Monday, March 28, 2016; see the linked-to service pages for the APAR numbers. Both are now packaged in z/VM's Cloud Manager Appliance (CMA). Matching copies of Enabling z/VM for OpenStack (PDF, web) and SMAPI (PDF, web) publications are live. You'll want to hold off installing until one more APAR (VM65809-11, depending on your environment) is orderable, since it's going to include a new xCAT image, but you can order anything else on the service pages now and beat the rush.
What's in it for me?
, you get...
- Simpler: manage fewer VMs - ZHCP's services will run inside the XCAT virtual machine once you migrate
- Leaner: get onto a path to reduce the frequency of service delta disk overflows (z/VM 6.4 will get you the rest of the way)
using CMA Juno, you get...
- Currency: OpenStack Liberty support replaces Juno; CMA Liberty ships the same OpenStack projects that CMA did, and adds ceilometer
- Currency: Provisioning support for RHEL 7 and SLES 12
- Monitoring: OpenStack instrumentation support via a new ceilometer plugin
- Security: integrating your enterprise LDAP server no longer requires back-porting fixes from the OpenStack Kilo release
- All the xCAT benefits listed above
of course, you keep the things you know and love from Juno
- Discovery: manage guests created outside of OpenStack
- Automation: APARs applied using the z/VM service process are automatically installed when the CMA is restarted
- Integration: CMA fits into larger "bring your own OpenStack" solutions such as vRealize Automation
- Automation: a chef client and IBM-provided cookbooks let you configure CMA using chef
, you get...
- All the benefits listed above
of course, you keep the things you know and love from Juno
- Choice: you choose which (if any) z/VM OpenStack enablement components to use or not, which combinations deployment topologies to support for how long, and so on.
What's the fine print?
- Sorry, but there is no free lunch. Unless you're installing a new z/VM system, you need to migrate from what you're using now, whether that's just xCAT or CMA Juno. I'll be filling in those pages soon, and updating them over time based on experiences with client systems.
- Provisioning RHEL 7 and SLES 12 is only supported on the OpenStack path, not when using xCAT without OpenStack. If you have your own ways to provision guests outside of xCAT, they are unchanged by this support.
- CMA Liberty is smaller than CMA Juno, in part, because we left out the chef server, the ICM self-service portal and ICM's cross-platform deployer. The only UI CMA provides is the OpenStack dashboard (horizon), and it is not customized for z/VM. Some solutions, like IBM Cloud Orchestrator, never used ICM's self-service portal anyway.
- CMA Liberty only manages z/VM guests. It can still be part of a multi-platform solution, it just is not one by itself.
- CMA Liberty only supports the OpenStack authentication (keystone) version 3 API; CMA Juno only supported version 2.
Modified on by JohnArwe
Many of you are probably in a change freeze right now, but here's the information so you can unwrap this present after the New Year begins (or whenever your change freeze expires/lifts).
VM65769 includes the following content plus other bug fixes. Remember that in addition to VMSES/applying the APAR (which delivers no code, only instructions for obtaining and installing the code), you must look at CMA42 FILE for the instructions to follow, including the URL for Fix Central and how to navigate to the download files.
- z/VM's CMA can be one region in a larger OpenStack cloud (each region has its own controller, but a single cloud authentication service covers all regions)
- z/VM's CMA can be one compute node in a larger OpenStack cloud (the controller runs on another platform, and the controller does not need to have the z/VM OpenStack plugins for nova/compute or neutron/network installed)
- z/VM's CMA, when running as an OpenStack controller, can allow users to manage existing z/VM guests through OpenStack-based UIs like ICM for z Systems and OpenStack horizon. This function is called discovery, and not all OpenStack operations can be used. Discovery was first delivered Sept 4 with VM65724, this APAR enhances it and fixes some bugs on error paths.
- Several security fixes
- Multi-path DASD support.
- ICM for z Systems 4.2 Fix Pack 3
- Each time the CMA starts, it will print its code version to the console log. If a notify_userid is configured in DMSSICMO COPY, XCAT will MSG the same information each time it starts. (I'm a strong proponent of positive reinforcement, so I like this a lot. Give me a hint that things are going right!)
- One bug fix worth mentioning: if you changed the IP address of the XCAT management node in DMSSICNF COPY but did not reset the CMA, in the some cases you would not be able to logon to the ICM for z Systems UI but you could logon to the OpenStack horizon UI. That's sorted now.
- When using the UI to stop an instance, z/VM now uses signal shutdown, so the instance has the z/VM system default shutdown internal to gracefully quiesce I/O and so on, with fallback to force immediate after that interval. Prior to this APAR, z/VM always used force immediate, which was a bit unforgiving.
- The -03 edition of the Enabling z/VM for OpenStack (Juno) publication is live.
Some bits of advice:
- The downloads are a full replacement of the entire CMA. Everything in the /install sub-tree of XCAT's file system will be preserved; that's reserved for user data. Everything else, gone.
- You will probably need at least 20GB of free space in XCAT's /dev/mapper/xcat-repo in order to get through the install; CMA42 FILE has details, exact amount depends on the sub-tasks you need to run but 20GB is a safe bet. This is in addition to any free space you have for deploying guests (the install can dip into that space, but then IVP will complain the next time you run it unless you remove the source files after installing them). Review the common problems list before starting (they're "common" for a reason).
- Be patient with the ICM Deployer - it took 15 minutes to run on my system.
- Follow the instructions EXACTLY.
- Feel free to delete the files you get from fix central once things are installed and you see the positive evidence (updated versions, usually). The install source files are never used after installation.
FWIW, when I did the test install of the final APAR (starting from Fix Central, doing it all myself, no shortcuts), it went quite smoothly. It's easier than it used to be, since VM65724 included a better service installation process. With some false starts because I was intentionally using a very constrained system  and I was recording all the checksums etc. on the wiki, I needed 2 days end to end. Without my false starts, 2 hours would have been enough. Your system and your guests can stay up while this is going on; the only "outages" are recycling SMAPI and XCAT. That typically takes well under five minutes, unless you're adding disks to the LVM... the I/O to format them takes a few minutes itself.
When you're done, you should expect to see this in the XCAT console (this excerpt came from a notify_userid):
09:45:06 * MSG FROM XCAT : CMA VERSION IS: 1.1.3-20151119
09:45:06 * MSG FROM XCAT : XCAT VERSION IS: 220.127.116.11
XCAT's version will be updated once you DDR the new image and recycle SMAPI. CMA's version will be updated a bit later when the "small" 5GB tar file is found and installed during XCAT initialization.
Happy Holidays everyone!
 I'm a natural tester, even if it's nowhere in my job description. After all, how many people would stumble into the difference between 15GB free (in df -h, which rounds, UP in this case) and 148xxxxx bytes (just slightly less than 15GB, un-rounded) ? I did. But that's why I often run constrained; better I should find those things than you.
Well now, that took "a bit" longer than intended. While I had planned to knock these off/out quickly, life intervened. So here, in the immortal words of the priest trying to marry Princess Vespa (again) at the end of Spaceballs, is the short short version.
OpenStack believes in two categories of storage
- Emphemeral: in lifecycle terms, "part of" a compute instance. It is allocated and destroyed along with a compute instance, so while it can be read/write, the data written is not permanently available to all future authorized comers.
- Persistent: disk whose lifecycle is its own, independent of any compute instances. This is the way "old" operators like me tend to think of disks, since we're used to fussing around with the devices that house them.
There are a couple of different potential uses for disk (representative, not exhaustive, list):
- Boot disks
- Root filesystems
- User data
- Temp/scratch space
Some of those uses apply at the z/VM level (where should z/VM get space to allocate a boot disk for a new virtual server?), some at the virtual server/guest level (root filesystem), some at both (user data). Our Enabling manual's reference format doesn't differentiate between all these variations, although I'm nudging it in that direction as I get a better grip on the details myself (remember, I didn't invent either z/VM or OpenStack). When we released VM65721 in June, we also updated Enabling and that included a re-write of the Planning chapter on Disk Storage; it was a definite improvement, but I still found it confusing. In the course of preparing this blog entry, I ended up restructuring it again... rather than summarizing, here is the updated text as it stands at this moment.
xCAT requires disk storage to accomplish its functions, such as storing captured images. Its disks are
managed by the Logical Volume Manager (LVM); although the functions and LVM names vary based on
whether xCAT is configured as a CMA or non-CMA environment, the following apply to all
- A logical volume's size can be increased but not decreased. Trying to remove disks from the logical volume will result in lost data, and may make xCAT unusable.
- The logical volume contains user data. You should back up its contents regularly in case it is compromised for any reason.
- The amount of space required is largely dependent on the size of the disks that will be captured, and whether compression is used in the capture process. Uncompressed images are approximately the same size as the original disk, while compressed images are approximately 1/2 to 1/3 of the disk size (depending on how much of the disk is in use).
- 50 GB of space is recommended to start with if you take the default value for the xcat_free_space_threshold property. (For more information on this property, see “Settings for Nova” on page 31.)
- xCAT manages a storage pool named XCAT1. You provide a list of z/VM volume labels, which must not be assigned to the directory manager, as described in the section below. The xCAT boot process detects changes in the list and issues directory manager commands that add any new volumes to this storage pool
Storage is needed for OpenStack services to provide minidisks for the root directories of deployed virtual servers, as well as for storing code and user data, depending on the minidisks' configured roles. In addition, SCSI disk space is required only if the deployed systems will have persistent disks attached by OpenStack’s block storage service.
Running OpenStack services requires a 0101 minidisk for the CMA system disk, no matter what role the CMA has been assigned. This disk must be included in the xCAT virtual machine's directory entry; it can be managed by the directory manager in a disk pool. The disk can be either:
- An ECKD disk of 3338 cylinders, or
- A FBA/eDevice of 4806720 blocks.
Running OpenStack services in the controller role requires minidisks to store user data. CMA manages these disks with LVM, using the logical volume name cmo-data. The logical volume cmo-data contains all user data, such as all OpenStack configuration files and database files, all generated certificate keys, all xCAT image files, and all temporary files. The amount of space you need depends on how many images/files that you want to keep. Specify z/VM volume labels in the DMSSICMO COPY file property cmo_data_disk during the configuration process. (For more information on this property, see “DMSSICMO COPY File Properties” on page 17.)
Running OpenStack services in the controller or compute role requires minidisks for the root directory of deployed virtual servers. The directory manager allocates minidisks from a disk pool it manages; specify the volume pool name and type in the DMSSICMO COPY file property openstack_zvm_diskpool. (For more information on this property, see “DMSSICMO COPY File Properties” on page 17.) An existing disk pool other than XCAT1 must be used for the minidisk pool. The amount of space required is dependent upon the size of the root directories in the images being deployed or captured.
If persistent disks attached by OpenStack’s block storage service will be used by the deployed virtual servers, FCP disk pools must be defined to the SAN Volume Controller (SVC) for this use and their details specified in the controller ’s DMSSICMO COPY file. The amount of space required is dependent on the size and number of persistent disks used by the deployed virtual servers. For more information, see this page in the IBM developerWorks® community:
xCAT uses xcat-repo as its logical volume name. Its z/VM volume labels should be specified in the DMSSICNF COPY file property xcat_iso during the configuration process. For more on this property and on defining the LVM, see the “Defining the xCAT Image Repository for z/VM Images” section in the “Setting up and Configuring the Server Environment” chapter of z/VM: Systems Management Application Programming.
Modified on by JohnArwe
tl;dr version: Installing the z/VM's Cloud Management Appliance gives you a "manage from z using OpenStack APIs" cloud controller. It packages several components, including IBM Cloud Manager and IBM Platform Resource Scheduler, and gives you a simplified installation experience.
I'm going to describe things using the latest level of code available when I'm writing this post (Juno). The specifics flowing from that will be different for other releases, but the overall pattern should be very similar.
It's not as simple as we'd like yet, so expect the details to evolve over time.
What do you get?
I’ll describe these pieces in more detail later (some in later posts), but this is the big picture of the what … not the how. The “how” is more involved, as is often the case.
You obtain the z/VM Cloud Management Appliance by following the directions installed by the z/VM APAR [cma-apar]; you read that correctly - installing the APAR gives you directions to get the code, not the code. Since it uses an appliance model, the code gives you a box that's taped up fairly tight; as you install and then use it however, you'll see the sorts of things above. Hopefully you'll see enough to be useful, and not so many that you're overwhelmed. At a high level, after installing the z/VM APAR you need to allocate and format new disks for the executable code, get the code from Fix Central [fix-central], and install/configure it before you have a running cloud. There's also a TechDoc describing parts of the process in more detail.
Consult the Book of Armaments
^ [book-of-armaments] Visit the z/VM OpenStack cloud maintenance service page and then click on the release you want; the newest is always at the top. Don't miss the requirement to apply xCAT maintenance as well.Generally z/VM folks will not need to know the OpenStack release names, but sometimes they come in handy when talking to others in your organization that might know OpenStack, like the LDAP people [OpenStack-release-names]. This is a partial snapshot of the Juno-specific content on the z/VM service page:
When you install the Juno service (e.g. VM65676), use the normal z/VM service process; effectively you're creating the CMA42 FILE on z/VM's MAINT 400 disk. The APAR is not the code! [finger-moon].
CMA42 FILE is a text file whose basic outline is:
General Service Instructions
- Use these General Service Instructions only if migrating from an earlier release of the OpenStack support; if you're installing for the first time, skip the General Service Instructions entirely.
- Install any pre-requisites
- Migrate using the Icehouse pre-migration scripts
General Installation Instructions
- Decide if you're using 3390 (ECKD) or SCSI (FBA) disk storage.
- Format the disks and add them to the correct z/VM userids.
- Download the 2 files (probably to your laptop)
- Move the files to z/VM and install them
- Move the image file to z/VM
- Detour into the "Enabling z/VM for OpenStack" publication to configure, install, and run installation verification.
- Move the ICM installer file to z/VM (into xCAT's file system)
- install ICM
- Login and start running your cloud
Lobbest thou thy Holy Hand Grenade of Antioch
^ [book-of-armaments] I've done this a few times; OK, more than a few, if you count test systems, client systems, demonstration systems. Using that 20-20 hindsight, this is how I'd tell a friend to install CMA as a manage-from-z controller [tell-a-friend], first as a picture and then as a set of more detailed steps.
- Skim the hints/tips to see what others have missed
- Get all the z/VM work prep work done
- install pre-requisite service [cma-apar] for OpenStack and xCAT
- Detour into the "Enabling z/VM for OpenStack" publication [cma-apar]
- read the z/VM Configuration topic
- read the Planning topic, noting disk requirements
- read ahead in the installation instructions about disk requirements
- update xCAT’s directory entry
- make sure it says 8G of virtual memory
- if it has no 101 disk yet, note that
- do whatever is needed in z/VM for storage, including formatting any new volumes
- Stage all the OpenStack-specific bits
- Follow the CMA42 FILE instructions through unpacking the xCAT image, but stop before restoring it.
- Verify that the bits on your disks match what’s expected
- Detour into the "Enabling z/VM for OpenStack" publication
- Read the Configuration topics on z/VM, xCAT, and OpenStack. Skip over or just skim the details of each parameter for now. I’ll cover configuration in a series of future posts.
- Draw your desired-state networking diagram
- Assume that you are only building a single “all in one” cloud for now: one z/VM in the controller role. Once that is working, if you want to add other z/VMs as compute nodes, it is a small delta.
- Assume that you want to access the UIs from outside the system, so the xCAT MN at least needs a public IP address, OSA, and gateway.
- If it’s possible in your deployment environment to let OpenStack assign public IP addresses to instances it creates, do that; it simplifies debugging. If that’s not possible, give it a private IP address pool to use.
- Assume that you won’t be manipulating any storage through OpenStack (cinder). You can enable that later, incrementally, if you want to.
- Build your CNF/CMO files [note: I’ll cover this process in future posts]
- Read the details of each parameter at this point.
- Based on the assumptions above, you will leave many of them at their default value of “NONE”.
- Install the components
- Restore the xCAT VM image using the bits you already staged
- Run IVP (the installation verification process) per the "Enabling z/VM for OpenStack" publication, fix any problems: rinse, lather, repeat until you have zero warning messages
- Follow the CMA42 FILE instructions [note: ssh/scp syntax] to get the ICM installer (tarbin) file into the Linux file system running inside the xCAT virtual server
- Verify that the bits on your disks match what’s expected
- Run the ICM installer
- Login to ICM; per the "Enabling z/VM for OpenStack" publication section entitled "Final Configuration of the CMA via the IBM Cloud Manager..."
After that, let your priorities govern the “if”, “when”, and order of tasks. Some examples are:
- Configure ICM functions beyond those needed for the base CMA installation
- External LDAP integration
- Loosen some of the earlier assumptions
- Add (additional) compute nodes … other z/VM systems, x86, etc.
- Configure storage units for use by OpenStack, e.g. as use by an image repository
- Capture and/or deploy images
- Clients without existing processes for managing z/VM virtual servers or images can generally use the "Enabling z/VM for OpenStack" publication’s chapters on image capture and customization to manage their images entirely via ICM/OpenStack.
- Clients with existing processes for managing z/VM virtual servers or images might well find that OpenStack’s restrictions conflict with those processes; in this case it will generally be easier to manage images and virtual server creation in z/VM, and use ICM/OpenStack for other aspects of the virtual server’s life cycle. Typically, different users perform those sets of tasks. More on that in future posts.
- Enable the IBM Platform Resource Scheduler; by default, the CMA "box" contains the bits, but does not install them.
- [book-of-armaments] As I said in an earlier post, consequences of a youth misspent memorizing four (no, three!) Monty Python skits. In this case, from Monty Python and the Holy Grail.
- [cma-apar] currently, the latest version of the appliance is documented in APAR VM65676 which corresponds to ICM 18.104.22.168 and OpenStack Juno. The z/VM OpenStack cloud maintenance service page lists all available versions, along with links to the "Enabling z/VM for OpenStack" publication for each version.
- [finger-moon] thus paraphrasing the Zen koan, "the finger is not the moon."
- [fix-central] you can use this direct link for the March 13, 2015 (first) version of the Juno files. Generically, use the Fix Central link, select the "Find a product" tab, and start typing "IBM Cloud Manager with OpenStack"; after "ibm cl" auto-complete will give you a mercifully short list. The alternative drill-down route on the "Select a product" tab is longer, and described in the TechDoc.
- [OpenStack-release-names] OpenStack release names follow the same convention as recent Ubuntu Linux distributions, where the first letter of the code name sequentially increases with each new version.
- [tell-a-friend] A friend I like, that is.
Modified on by JohnArwe
This post is the first in a series I'm writing about getting an OpenStack-based zLinux cloud running on z/VM. I'm going to concentrate on the “manage from z” case, although a good deal of it applies to other topologies. Note that “manage from z” is capable of managing Linux systems running on Power or Intel, although at this point I have no plans to cover “manage to” on platforms other than z/VM. Plenty of materials already exist covering other platforms.
Did you know that your existing z/VM systems can run or control an OpenStack-based IaaS cloud?
You heard that right.
You can run an OpenStack-managed Linux cloud on your existing z/VM system.
Let that sink in.
z/VM 6.3 with the right service  is the entry point, and you'll need to enable SMAPI. If you're just kicking the tires, or want to run on z/VM only and without the ability to open support questions/problems, it's zero incremental cost over base z/VM (i.e. “free as in beer”). Running things with support, and/or on other platforms, requires licenses beyond the base .
Having recently come back to mainframes from IBM's Cloud and Smarter Infrastructure (formerly Tivoli) unit, I have something of a z/VM outsider's view. I used z/VM while working in z/OS development for testing, more than a decade ago, and I was exposed to all the cloud work (especially around standards and open source) while in CSI so I could already spell OpenStack (the technological base for de facto IaaS clouds). If you're someone who knows z/VM but not OpenStack, or vice versa, my goal is that these entries will still be understandable. I figure your primary goal is get an OpenStack-managed zLinux cloud up and running on z/VM, and anything new you learn about "the other side" is icing on the cake for you.
Mind you, having a well-developed sense of irony , whenever I talk about z/VM and Cloud my Evil Twin  thinks of the old Reese's ads :
You got your cloud on my [z/]VM!
You got your [z/]VM in my cloud!
Two great tastes that taste great together.
To be continued...
- Don't make the mistake of looking only at the table, once you expand the Juno section on the z/VM OpenStack maintenance page; the rest of the page has other content you'll need. For example, as the text surrounding the table says, you also need to be current on xCAT service, since xCAT is used as a component in the OpenStack support.
Read and interpret the licenses using your own counsel! IANAL.
A consequence of a youth misspent on Monty Python and the like
"Naughty, evil Zoot!" (Google it - from Monty Python and the Holy Grail); the notion of evil twin blogging I first came upon when Enrique surfaced in the blog of my first W3C Contact, MSM. I have yet to name mine.
Ads for Reese's Peanut Butter Cups
Time to look back,
... look around,
... and look forward.
I've been heavily involved in work with the W3C over the past 8+ years, as a chair, as a spec editor, and as a working group member. What they've built, and the open processes they use to build it, still impresses me.
On a more personal (set of) note(s), much of what I've spend the last few years laboring on has matured:
For me, this means taking a new job within IBM. I'll be involved with clouds on the mainframe: z/VM's OpenStack enablement for starters. This blog will continue, but as you might expect with a new day job its emphasis will evolve.
First, the answer; then, in HHGTTG style (42!), the question.
The question was: how do all these wonderful things I'm hearing about relate? How do I wrap my head around all this?
By way of background: Innovate is the annual conference that focuses on IBM's Rational brand - developer tooling. In the Exhibitor hall, there was a whole section of pedestals on IBM DevOps Services (until recently known as JazzHub); there was also a set of other pedestals in the Cloud section representing other capabilities that IBM offers: BlueMix, SoftLayer, Service Engage, and some others that simply don't relate to this view. The IBM Cloud Marketplace is the umbrella through which you can find them all, what's above and what's not (like BPaaS).
What the picture above says, and some additional details:
Clients choose how to consume what they need; there's no "one size fits all" answer. The good news is you get freedom of choice; the bad news is you have the responsibility of choosing (we'll help, if you want). Typically this choice is based on figuring out the subset of options that meet non-technical constraints, and then comparing lifecycle costs.
By the way: if you're happy where you are without any of this (aka remaining "on-premise"), that's fine.
If you want to try something out, cloud delivery options give you a way to kick the tires quickly and with minimal investment of your time or effort. If you prefer to deploy on-premise for production, that's fine too.
If you want to explore these options, the Big Question is: what do you want to pay someone else to do, in order to avoid doing it yourself? More on that in the next picture.
DevOps Services gives you a way - a delivery option - to construct applications using Rational tools delivered via the cloud.
Advantage: you're always on the latest level of development tooling code that you get from the cloud, since it's continuously delivered.
It includes a one-button "deploy into BlueMix" option. You can also deploy to other targets.
Your code still lives whereever it lives now, and you can use any tools (cloud or not) as a result. If you're starting something new it gives you an option to host the repository, but it will just as happily use an external repo like GitHub.
BlueMix gives you a way - a delivery option - to host running applications, for testing and for production.
You can avoid Dependency Hell by binding to other BlueMix services.
You can deploy your app as a BlueMix service for others to re-use - and to pay you for.
Service Engage actually is implemented as a BlueMix app - we're using it for production ourselves.
Service Engage gives you a way - a delivery option - to get systems management from the cloud.
Advantage: you're always on the latest level of systems management code that you get from the cloud, since it's continuously delivered. We also add offering on an on-going basis... in February we had 3, now it's 5, and it's only going to grow over time. These offerings correspond what you might know as Tivoli products - certainly more people at the peds recognized them that way; Tivoli changed its name to Cloud and Smarter Infrastructure last year, if you missed that.
If you want to try something out, cloud delivery options give you a way to kick the tires quickly and with minimal investment of your time or effort. Each offering includes live demo and free trial options.
Your existing applications and data still live whereever they live now.
I must have talked through that picture 20 times over the 3-ish ped-days I was there. After the pattern emerged, we drew the whole picture out on the SoftLayer ped's whiteboard, which made it easier to see/refer to. I guarantee you it was neither as neatly done as the version above, nor as complete as the text points above allow.
The next picture focuses on that Big Question - what do you want to pay someone else to do, in order to avoid doing it yourself? - just in case you're not familiar with which pieces are "in the box" at these levels.
Attribution: I lifted the second picture from a presentation the BlueMix folks did.