To live the Agile/DevOps lifestyle (being agile is more attitude than practices), one thing you need to do is drive out manual process steps wherever you find them in your project or team. [Well, almost every place -- I'd prefer we didn't automate away the team lunch.] And there will be times when you have to grit your teeth, stick your nose in a bunch of books and blogs and just make it happen. This is the story of such a time...
This quite excellent article describing modern Java development practices (http://www.jamesward.com/2014/12/03/java-doesnt-suck-youre-just-using-it-wrong
) opens with 10 Page Wikis to Setup Dev Environments Suck
(I very much agree with this and many other points in the piece). I recently worked with a team that had a 50+ page document -- not even a wiki yet -- and it was generally accepted that if you got through it in a week, and things worked, that you had done well. I was aghast, but being the new guy, I just ground my way through it vowing that some day this had to be automated.
The biggest part of the grind -- and the most error-prone -- was configuring WebSphere Application Server for the project. There were pages and pages with instructions and screen shots and tables of data to enter. And if you messed up part of it, you might not realize it right away. This actually happened to me. Months after completing this stage, I was working on a different part of the application and I was getting errors no one else was seeing. Hours of debugging later, I realized that I had transposed characters in one of the JNDI names during the setup (months earlier). It just wasn't until I worked in this particular part of the code that it was actually a visible problem.
Backing up in time a bit, when I was first handed this doc, I mildly protested: "You don't have any automation or scripts for this?" The answers seemed to fall between "Never had the time" and "No one knows how". I didn't know how, but I knew that it could be done -- I had worked with some folks that wrote WAS admin scripts for a living. I had done some work in Python before and the WAS admin scripting was done in Jython (Python for the JVM), so...
When we started working on the next generation of our product (not just the next version, but a brand new, nearly green-field, newly architected version) I knew the time was now. If this were ever to get done, it had to be part of our build-first, DevOps-oriented reinvention of ourselves. So I lobbied for this as a high priority item and even agreed to do it because it needed doing. No sensible, sustainable, Agile-reinvention was going to begin with a new version of the 50+ page Dev Environment Setup Guide -- not even if it was turned into just a 10 page wiki.
Few people are good at everything and I certainly had no experience with Jython or WAS administrative scripting (and am certainly no expert at Windows batch scripts). And, frankly, I didn't really want to do this (when would I need this again?). But somewhere between the concepts of Whole Team
and Collective Ownership
, someone had to do it. When I'm not good at something that matters, I tend to lean in. I own my ignorant suckiness and fight through it (with lots of help from other people's blogs and StackOverflow.com
). The rest of this blog is a description of what I built in hopes that someone else fighting a similar fight can cop a short-cut on me.
Get on with it already
TL;DR -- It worked. We took a multi-page, multi-day, error-prone process and replaced it with a page or so of setup instructions and a couple scripts that take maybe 15 minutes to run. We made the WAS profile a throw-away item that can be recreated in minutes, not something to be cherished and protected. That is the DevOps Way.
Enough story telling, time to talk real WAS automation.
What we were trying to do isn't all that fancy:
1. Create a couple JAAS Authentication entries to use with data sources
2. Create a JDBC driver for DB2
3. Create a few data sources used by various applications
4. Add a collection of SSL certificate entries
5. Create a collection of Work Managers
6. Create a collection of Cache Managers
7. Save changes (so that we can)
8. Test the data sources
9. Happy Dance.
What we weren't trying to do is make it bullet-proof, so you won't find scads of error handling in the code. For the most part, if the script fails, the save operation is at the end, so nothing gets saved. You can figure out when went wrong, fix it up and run it again. If that doesn't work, crush the profile and start over. This is our Minimum Viable Product and it proved sufficient.
I'm not going to explain WAS scripting beyond saying that you get to write Python (technically Jython) code that runs on the JVM and uses libraries provided by WebSphere to complete most tasks. The real work is done by starting the wsadmin tool (which sets up the Jython environment and pre-loads some libraries you'll need). The WAS docs do a reasonable job with the details and some searching will fill in the gaps. Most of the action is at the command line (all examples use Windows) so a text editor that understands Python syntax is helpful. I'm also not going to explain the details of how the scripts work -- if you've read this far you clearly have an interest and enough background to understand or work it out.
What I do need to explain is something that it took me waaaayyyyy too long to notice: The WAS Administrative Console is your friend and will help you write your scripts. That little link off to the right is always there and I was almost done doing this before I realized it. DOH!
If you're not sure how to do something with a script, do it in the UI (you don't even need to save it), the click that link. In the image above, I created test data source then clicked the link and the window came up showing the scripting command.
CAVEAT: Some of the things in the scripts are very specific to our environment. They aren't meant to be configurable to your environment. Make a copy, learn what you can, change what you need to. I have trimmed and changed some of the data for brevity and privacy.
Setting the Environment (literally)
To coordinate between the scripts involved, we needed some environment specific information. We don't need it permanently, so rather than create new permanent environment variables, the user establishes some temporary variables in setmywasvars.bat and runs it in a new command window. There's a little bit of error checking to make sure they didn't leave out something that will cause an issue later. All subsequent commands should be run in the same command window since they will rely on these variables. Depending on your WAS configuration, you may need to update the port number in wsadmin.properties and you'll definitely need to add the WAS Admin credentials.
Then the user creates a WAS profile using makemyprofile.bat. Since later steps will require the server to be running, the script will offer to start it. At this point, the user may need to update their WAS Node name in setmywasvars.bat if this was their first profile. I'm not aware of a reliable way to determine that value ahead of this.
The last step is to run configuremyserver.bat which will process all of the steps described above (except the Happy Dance, that's for you to do). After creating the JAAS Authentication entries, it has to save changes and restart the server to get them picked up before it can continue. After making all of the other changes, it has to save them before it can successfully test that the data sources work.
Some task details
There is a subfolder for sslcerts. When the script runs, it will pick up and add each certificate to the system. If you have certificates to process, copy them into that folder.
For the Work and Cache Managers, which don't change often, the details are captured in the script. This could be externalized (similar to the sslcerts) if you need more flexibility.
Some versions of Windows are not properly handled by the Jython scripting. If you run the automation and see something like “KeyError: WAS_PROFILE_NAME” or “Failed to get environment...”, then you are running one of those versions. This issue and the fix are described here: http://blog.oliver-mueller.com/-216
. The fix is to create a file named %WAS_ROOT%\optionalLibraries\jython\registry and put this one line in it: python.os=nt. Now Jython can properly detect you are on Windows and will be able to process environment variables and complete your setup.
The batch files
There are quite a few small batch files that do little things -- some meant to be used by the user, others are more internal and some to save me time when I get pinged about something not working (e.g., showmywasvars.bat exists solely so I can ask someone to run it and confirm their settings). See the GitHub file list (or the file's comments) for more description.