The developerWorks Connections platform will be sunset on December 31, 2019. On January 1, 2020, this blog will no longer be available. More details available on our FAQ.

z/OS Communications Server

From archive: August 2015

Test and Learn Part Two: Actually Testing

RaquelPrieto Tags: ibm-blog ‎ 6,102 Views

PTF test for PI38376: A TCP connection can use the wrong maximum segment size (MSS) on V2R1

Part Two

Warning: this part gets pretty technical.

Test Execution - Coming up with things to do using said stuff

What I like to do after understanding the basic idea of what needs to be tested is to just jump straight in and see how far I can get before hitting any roadblocks. Usually these roadblocks are more configuration related, but I often just get completely confused. So what I usually do is revisit the customer's steps for recreation and add my own notes from various observations. That way if I ever need to test something that focuses on these areas of CommServer, I'll have something to go back to instead of relying on only memory.

I know that I need access to a system running V2R1. I also need to know whether the PTF has been applied. There doesn't seem to be an explicit way to determine whether a certain PTF has been applied to one of our IPL drivers, so my only option is to check by the actual test - if I see the results of the fix, then the PTF must be applied. This limitation is frustrating from a testing perspective because I believe it's good practice to perform test with as few assumptions as possible. Likewise, there doesn't seem to be a direct way to recreate the scenario on a system without the PTF applied unless I employ a workaround which adds more time spent on configuration rather than test.

That workaround for this case is to recreate the scenario on a back-level system, so my journey begins on a couple of V1R13 second-level IDs. I use a netstat display to get an idea of what is already set up and what I can use. A netstat -h command (to display the home list) shows me a DRVIPA I can modify and use in this test:
Dynamic VIPA:
IpAddr/PrefixLen: 197.26.201.1/24
Status: Backup Origin: VIPABackup DistStat: Dest

    I also look at the routes defined for this DVIPA using a netstat -r:

       Destination Gateway Flags Refcnt Interface
       ----------- ------- ----- ------ ---------
       197.26.201.1/32 16.26.81.72 UGHO 0000000000 O3ETHW0
       197.26.201.1/32 194.26.80.72 UGHO 0000000000 LIQDC00
       197.26.201.1/32 199.26.80.72 UGHO 0000000000 IQDIOLNKC71A500C

   So it looks like the DVIPA 197.11.201.1 is a backup for this system, let's call it SysB. Now I'll use a sysplex display command to see who owns it:

   VIPA DYNAMIC DISPLAY FROM TCPSVT AT SysB
IPADDR: 197.26.201.1
ORIGIN: VIPABACKUP
TCPNAME MVSNAME STATUS RANK DIST
       -------- ------- ------ ---- ----
TCPSVT    SysA ACTIVE BOTH
TCPSVT    SysB             BACKUP 010 DEST
       TCPSVT    SysC             BACKUP 020 DEST


There are three systems in the sysplex who own this DRVIPA that I'm using for test. For the sake of isolation and control, I will force SysC to leave the sysplex for now. Alternatively I could simply delete the DRVIPA definition on SysC, but I decided to take a more destructive route for no particular reason.

SysA is the owner of this DRVIPA, so I'll check the TCP/IP profiles and DVIPA definitions of these systems.

Our chosen DRVIPA is defined, and the MOVEABLE IMMED parameter for the corresponding VIPABACKUP definition was not present but is enabled by default, so I will need to either modify the definition or create my own and specify the MOVEABLE WHENIDLE parameter. Additionally, the relative Knowledge Center documentation states that as a rule, the DYNAMICXCF parameter must be specified in the TCP/IP profile to preserve existing connections on a VIPA takeover. I decided to create my own DVIPA definition files:

   Primary distributor definitions:
       VIPADYNAMIC
VIPADEFINE MOVEABLE WHENIDLE 255.255.255.0 197.26.201.1
ENDVIPADYNAMIC

       VIPADYNAMIC
       VIPADIST DISTM BASEWLM SYSPLEXPORTS 197.26.201.1
           PORT 21 80 623 923 925
           927 928 929 931 932
           1821 52002 50030
       DESTIP ALL
       ENDVIPADYNAMIC

   Backup stack definition:
        VIPADYNAMIC
VIPABACKUP 10 MOVEABLE WHENIDLE 255.255.255.0 197.26.201.1
ENDVIPADYNAMIC

I'll spare the details of error recreation and move on to the good stuff: seeing the fix in action!

1. Define the DRVIPA to be used

Since our SVT systems run all day, it's likely that the host route for my DRVIPA has already been created. Fortunately I found some instructions to remove this route without having to re-IPL or recycle the stacks:
       a. vary obey a file to delete the the DRVIPA
       b. verify the route is no longer present with D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
       c. verify the DRVIPA itself is deleted with D TCPIP,,N,VIPADCFG,IPADDR=197.26.201.1
       d. vary obey a file to define the DRVIPA
       e. verify the system has successfully re-added the DRVIPA and routes with the same commands

2. Start TCPIP on a backup system without OMPROUTE to simulate delay

After running this step I noticed that OMPROUTE was starting automatically when starting the stack. Re-examining the TCP/IP profile gives me the explanation for this behavior: Autolog was defined to automatically start OMPROUTE when the stack was started, so I just need to comment this out.

Then I noticed that the TCP/IP stacks were not joining the sysplex when started while OMPROUTE was down. Turns out that this is a result of the GLOBALCONFIG DELAYJOIN profile parameter, which prevents the stack from joining the sysplex unless OMPROUTE is started. I need to comment this out as well.

And finally the stack was refusing to join the sysplex due to not finding dynamic routes over monitored interfaces, so I commented out GLOBALCONFIG SYSPLEXMONITOR MONINTERFACE and added GLOBALCONFIG SYSPLEXMONITOR NOMONINTERFACE.

Since the default options for the systems in our test environment are fairly different from the profile options I need to recreate the customer's configuration, I will copy the existing profile to my own custom profile and start the stack with an override parameter: S TCPSVT,HOST=MYPROF,REUSASID=YES

   Finally, I have everything configured correctly and got some displays:

   D TCPIP,,N,VIPADCFG,IPADDR=197.26.201.1
15.25.26 EZD0101I NETSTAT CS V2R1 TCPSVT 786
DYNAMIC VIPA INFORMATION:
VIPA BACKUP:
IPADDR/PREFIXLEN: 197.26.201.1/24
RANK: 010 MOVEABLE: WHENIDLE SRVMGR: NO FLG:

   D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
EZD0101I NETSTAT CS V2R1 TCPSVT 830
IPV4 DESTINATIONS
DESTINATION GATEWAY FLAGS REFCNT INTERFACE
197.26.201.1/32 0.0.0.0 UH 0000000000 VIPLC51AC901
METRIC: 00000000 MTU: 65535
MVS SPECIFIC CONFIGURED PARAMETERS:
MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES

3. Start and stop the primary distributor without OMPROUTE to force the DRVIPA takeover on the backup stack
For this I will start the stack on the primary distributor then use a sysplex command to deactivate the DVIPA: v tcpip,tcpsvt,sys,deact,dvipa=197.26.201.1

EZD1298I DYNAMIC VIPA 197.26.201.1 DELETED FROM TCPSVT
EZZ8303I VIPA 197.26.201.1 GIVEN TO TCPSVT ON SysA

   From the backup stack:
       D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
       15.27.35 EZD0101I NETSTAT CS V2R1 TCPSVT 865
       IPV4 DESTINATIONS
       DESTINATION GATEWAY FLAGS REFCNT INTERFACE
       0 OF 0 RECORDS DISPLAYED
       END OF THE REPORT

VIPA DYNAMIC DISPLAY FROM TCPSVT AT SysB
IPADDR: 197.26.201.1
ORIGIN: VIPABACKUP
TCPNAME MVSNAME STATUS RANK DIST
-------- -------- ------ ---- ----
TCPSVT SysA ACTIVE
TCPSVT SysB BACKUP 010

   And on the primary:
       EZD0101I NETSTAT CS V2R1 TCPSVT 205
       IPV4 DESTINATIONS
       DESTINATION GATEWAY FLAGS REFCNT INTERFACE
       197.26.201.1/32 0.0.0.0 UH 0000000000 VIPLC51AC901
           METRIC: 00000000 MTU: 65535
           MVS SPECIFIC CONFIGURED PARAMETERS:
           MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
           ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
           VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES

   Then stopping the primary distributor...

   EZZ8301I VIPA 197.26.201.1 TAKEN OVER FROM TCPSVT ON SysA

   D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
EZD0101I NETSTAT CS V2R1 TCPSVT 648
IPV4 DESTINATIONS
DESTINATION GATEWAY FLAGS REFCNT INTERFACE
197.26.201.1/32 0.0.0.0 UH 0000000000 VIPLC51AC901
METRIC: 00000000 MTU: 65535
MVS SPECIFIC CONFIGURED PARAMETERS:
MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES

4. Restart the primary distribute with OMPROUTE to takeback the DRVIPA: v tcpip,tcpsvt,sys,react,dvipa=197.26.201.1

5. Start OMPROUTE on the backup so OSPF host routes will be learned from the distributor

   D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
EZD0101I NETSTAT CS V2R1 TCPSVT 868
IPV4 DESTINATIONS
DESTINATION GATEWAY FLAGS REFCNT INTERFACE
197.26.201.1/32 16.26.81.72 UGHO 0000000000 O3ETHW0
METRIC: 00000171 MTU: 1500
MVS SPECIFIC CONFIGURED PARAMETERS:
MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES
197.26.201.1/32 199.26.80.72 UGHO 0000000000 IQDIOLNKC71A500C
METRIC: 00000171 MTU: 1500
MVS SPECIFIC CONFIGURED PARAMETERS:
MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES

6. If a connection is established from the backup to the DRVIPA on the distributor, a netstat display on this connection should show the MSS is not set to 536

I used an internal traffic simulation tool to establish a connection from the backup stack to the DRVIPA. First, I issued a netstat -V DETAIL -P 50030 to see the connection on the distributor, then used the unique port as a filter on a netstat -A display on the backup stack:

MVS TCP/IP NETSTAT CS V2R1 TCPIP Name: TCPSVT 10:55:43
       Dynamic VIPA Connection Routing Table:
       Dest: 197.26.201.1..50030
       Source: 16.26.81.12..1487
       DestXCF: 199.26.80.12
           PolicyRule: *NONE*
           PolicyAction: *NONE*
           Intf: IQDIOLNKC71A5048
           VipaRoute: No Gw: 0.0.0.0

Local Socket: 16.26.81.12..1487
Foreign Socket: 197.26.201.1..50030
BytesIn: 00000000000000650000
BytesOut: 00000000000000650000
SegmentsIn: 00000000000000000001
SegmentsOut: 00000000000000000002
StartDate: 08/26/2015 StartTime: 14:55:26
Last Touched: 14:56:26 State: Establsh
RcvNxt: 0399845097 SndNxt: 4265975520
ClientRcvNxt: 0399845097 ClientSndNxt: 4265975520
InitRcvSeqNum: 0399195096 InitSndSeqNum: 4265325519
CongestionWindow: 0000004344 SlowStartThreshold: 0000065535
IncomingWindowNum: 0399976165 OutgoingWindowNum: 4266041055
SndWl1: 0399195096 SndWl2: 4265325520
SndWnd: 0000065535 MaxSndWnd: 0000065535
SndUna: 4265975520 rtt_seq: 4265325519
MaximumSegmentSize: 0000001448 DSField: E0
...

Definitely not an MSS of 536.

7. Dump the TCP/IP address space to determine the MTU size that was set
After consulting with the more experienced ranks for this task, I got a little more experience with IPCSGO to view a dump in a slightly more manageable way than just a bunch of raw data. From there I was able to view the IP routing table with the IPCS primary options menu: CS/OS390 --> TCP/IP --> Configuration --> Route --> Ipv4, then entering the name of my TCP/IP stack. At the top of this table its address space ID within the TCP/IP address space is displayed which I then use to view the MTU value that was selected for this route. Likewise, I'll need the address of the actual route I want to examine. A find on my IP address gives me the entry and the address:

Take note of that ASID.

"Configuration" is the keyword I was looking for, since there are multiple routes for this particular destination.

But now I have to dive in to that bunch of hexadecimal raw data. Back to the main IPCS menu, the Browse option allows me to view the contents of the dump as hex characters. I have to search for the ASID within the address space that was dumped, so I change the Override Defaults Address space option with ASID(X'00CC') and hit enter. [Hint: this can also be done from the Defaults option in the IPCS primary option menu]

This next part was pretty intense, and it involved looking at the actual Communications Server code using another resource to determine at which pointer combined with which offset were the RTE and RTIOCTL eye catchers (on the right in the image). From there, I need to move to the RTOP, which is the eye catcher that indicates a set of RTE control blocks representing a route to a specific IP address. The RTOP is referenced at this address:

Putting a question mark beside that address and hitting enter takes me to that address. Then, counting from a pointer to a certain offset amount (as determined from the actual code) tells me what the MTU was set for this RTOP.

That guy.

0000FFFF is 65,535 in decimal. Looking back at the test information:

"The problem occurs when an implicit host route for the DRVIPA is generated with the default MTU 576 instead of 65535 on a backup system."

And the MTU for the route to 197.26.201.1 on the backup system is 65,535 instead of 576. Looks like it's working! From here I can look at all of the RTE control blocks under this particular RTOP to see what MTU values were set. The RTOP MTU should be the lowest value of all RTE control blocks, with the exception of the host route. I can move from one RTE to the next by putting a question mark by the pointer; once the "next pointer" is a bunch of zeroes, that's the last RTE under this RTOP:

No, I don't just magically know this. It's determined by looking at the offset specified in the code. It'd be cool if I did magically know this, though.

Of course, I want to be certain that the PTF is, in fact, applied. To do this I can compare the date displayed in the eye catcher of the module that was edited:

Magic and sorcery.

Browsing through that same address space and using the command "f ezbxfdvi nobreak" to search for the name of the module that was edited for this fix brought me to what I was looking for. Here I can see both the last date it was modified ("15.189" for 2015, day 189) and the internal PTF number which corresponds to my PTF record. This confirms that the PTF has been applied on our test systems and has fixed the issue.

And of course some final thoughts

People are helpful around here and I'm grateful for that, because even the best documentation could only get me so far. The level of customization that's possible just for the TCP/IP stack is immense, so learning one type of customer configuration by having to mimic it in a test environment feels like I'm only scratching the surface.

By testing a fix for an error that a customer found, it was interesting to gain a new perspective other than my normal set of activities revolving around testing pre-GA'd code. I now have notes explaining new concepts and had a chance to explore something new in the brain-frying world of z Systems Parallel Sysplex, with minimal brain frying.

Likewise, I'm glad to be part of something that values the customer. This error was very specific, but it may be something to investigate in our own test suites.

Modified on by SamReynolds

Planning for the expected - minimizing application unavailability for planned outages

mfitz Tags: ibm-blog ‎ 5,992 Views

Although being a highly unlikely occurrence, IT departments need to spend resources to ensure that their business critical data and applications can be successfully recovered in the event of an unplanned failure of their production site. Minimizing data loss is the highest priority but it can be at a tradeoff with application availability. Typically, following an unplanned outage, the entire data center can be restarted at the disaster recovery (DR) site but this can take several hours or longer before the applications are available.

What is sometimes overlooked is how to maintain application availability for a far more likely scenario, a planned outage for a maintenance activity. IT departments schedule maintenance windows in order to apply software fixes or perform application upgrades. Their goal is to minimize the number of maintenance windows required as well as ensure the duration of each window is as short as possible. Despite these efforts, these maintenance windows can still last for several hours and occur multiple times per year. Since recovering the data and applications on the DR site could take several hours, it makes little sense to attempt to utilize the DR site during planned maintenance activities, as the time it takes to switch to the DR site and back to the production site could be longer than the maintenance window itself. As a result, IT departments try to schedule these maintenance windows with the aim of minimizing the impact to their customers, usually on a weekend night.

What if there was a way to quickly switch access to business critical applications and their data from one site to another in a few minutes, rather than a few hours? With application unavailability for maintenance windows shrinking down to several minutes, these windows could be scheduled more frequently, ensuring systems and applications are always running with the most up-to-date fixes. So how can this be accomplished? By using a software data replication product to keep data sources used by the applications in sync across two sites, and IBM Multi-site Workload Lifeline to distribute connections for these applications, such a reduction in site switch times can be achieved.

IBM Multi-site Workload Lifeline, or Lifeline for short, provides the ability to perform a graceful switch of the applications and their data sources, called workloads by Lifeline, during planned outages. By using simple Lifeline commands, workload migration from one site to another can be easily performed, minimizing the down time for planned events such as scheduled maintenance activities. So what makes Lifeline different from existing disaster recovery solutions? Well first, Lifeline is not an all-or-nothing solution. Rather than initially plan for, and provide system resources for the planned recovery of all workloads in the production site, IT departments can focus on their most critical workload first, and gradually roll out the solution for additional workloads, as needed. A second differentiator is that Lifeline requires no application changes or changes to the clients accessing the applications and data. Following a planned outage, no manual changes in the network topology is necessary before the workload is able to be accessed on the alternate site.

As mentioned earlier, a key component to ensure a quick switch of applications and data to the alternate site is software data replication. Depending on the data source being used by the application, a different software replication product would be used to keep the data source in sync across the sites. For example, for applications utilizing DB2, IBM InfoSphere Data Replication for DB2 would be used to keep DB2 data in sync. Lifeline ensures connections for a workload are distributed to only one site at a time, to make certain that updates to the data source are occurring on only one site at any point in time.

Lifeline enables the graceful switch of a workload from one site to the other by:

First, preventing new connections for the workload to be distributed to either site while allowing existing connections to the production site a chance to complete their work,
Next, resetting any connections on the production site that have not completed their work. This guarantees that no additional updates to the workload's data source can occur on this site.
Finally, allowing new connections for the workload to be distributed to the alternate site.

In subsequent blogs, I'll cover more topics such as how Lifeline can also be used to quickly recover from unplanned outages as well as the different types of workloads that Lifeline can provide workload recovery for both planned and unplanned outages. In the meantime, you can learn more about Lifeline by going to the following link:

http://www.ibm.com/software/products/en/network-lifeline

Modified on by SamReynolds

Test and Learn Part One: Learning Things

RaquelPrieto Tags: customer test sysplex commserver ibm-blog ‎ 8,397 Views

PTF test for PI38376: A TCP connection can use the wrong maximum segment size (MSS) on V2R1

Part One

A test role here at Communications Server for z/OS involves more than just testing the latest and greatest code. Customers run into issues and fixes need to be made available for them, but not before they are internally tested. Today I describe my experience testing this PTF as a still relatively new member of the z/OS System test team. A lot of frustrations were had, but fortunately I still have all of my hair and had the chance to learn some new things.

Gathering Information - I know some stuff, maybe?

To start, I look at the PTF record and corresponding web pages in our internal source control tool to gather some initial information. I end up with a Notepad++ document full of haphazardly pasted notes from various resources to sift through and make some sense of. Fortunately there are a ton of details, which makes for a happy tester. I'll leave the majority of the nitty-gritty out and summarize the situation:

A distributed DVIPA (DRVIPA) is defined for at least two systems in a sysplex: one being the primary distributor and the other a backup.
If the backup stack is started before the primary distributor and takes over the DVIPA, an implicit (host) route from the backup to the primary distributor is created for that DRVIPA with an MTU size of 576 during the SYN stage of a TCP handshake. In the failing case the multipath routing algorithm is used, which chooses the smallest MTU value among all possible routes to the DRVIPA but ignores the default host route. Although OMPROUTE uses OSPF to advertise host routes with larger MTU sizes, the MTU for this particular route remains "stuck", resulting in an MSS (maximum segment size) of 536 for outbound TCP connection setup requests. No bueno.

"The problem occurs when an implicit host route for the DRVIPA is generated with the default MTU 576 instead of 65535 on a backup system. This is accomplished by starting the backup system first before the distributor."

To make matters a bit more convoluted, the conglomerate of notes inform me that I will not be able to view the incorrect MTU with a simple netstat route display. Instead, I'll have to dump the TCP/IP stack after recreating the scenario and scour through the raw memory. Staring at hexadecimal. Looking for something called an "RTE" in something else called an "RTOP."

Additionally, the customer's error description included steps to recreate the error:
   1. Define DRVIPA to be used
       -Backup definition must be defined without the MOVEABLE IMMED
   2. Start TCPIP on a backup system without OMPROUTE
   3. Start and stop the primary distributor without OMPROUTE to force the DRVIPA takeover on the backup stack
       -At this point the host route for the DRVIPA with the MTU of 576 is created on the backup stack
   4. Restart the primary distributor with OMPROUTE to takeback the DRVIPA
   5. Start OMPROUTE on the backup so OSPF host routes will be learned from the distributor
       -At this point the MTU value set at 576 will get "stuck"
   6. If a connection is established from the backup to the DRVIPA on the distributor, a netstat display on this connection will show the MSS set to 536
   7. Dump the TCP/IP address space
       -Examine the dump to find the MTU value of 576 from RTOP in RTE

So far, I know our environment has, at least, bits and pieces of this customer configuration. The SVT environment has DVIPAs defined with their corresponding VIPADISTRIBUTE and VIPABACKUP definitions. Since these DVIPA definitions were built to be highly customizable to suit a customer's needs, the amount of options and parameters possible combined with ensuring correct syntax can be overwhelming at times. For this reason my preferred method is to take from example - there are already so many different kinds of configuration files saved over the years in our test environment that there is likely one I can use as a template for this test.

As a tester, however, I could have saved a decent amount of time if all of the test information I needed were in a single location, instead of a number of separate records/web pages. I had to dig around to find useful pieces of information, and the first place that I looked (the PTF test record directly assigned to me) did not contain detailed error recreation instructions.

Research - I figure out some stuff

Terms and concepts

From the information I've gathered so far, I need to define some acronyms and understand some concepts not previously encountered.

RTOP: Google search didn't come up with anything, and neither did the two 'Terminology' bots on Sametime, so I went to the V2R1 Knowledge Center. A search gave me the RTOPTS, or run-time options for the language environment which doesn't seem related, so I asked a more experienced tester who wasn't familiar with the acronym either. Luckily I was eventually able to find someone who was familiar with it, and it turns out that RTOP is an identifier (an "eye catcher" as we call it) for a control block of a group of routes to a given IP address destination in a dump of the TCP/IP address. Internal stuff, so that's why it wasn't publicly searchable.
RTE: Another hopeless Google search but 'RTE', turns out, is related to a TCPIPCS ROUTE report in IPCS. All of that happens to be in the Knowledge Center. Looking as a sample TCPIPCS ROUTE report gives me more clues - it looks like RTE is just a shortened name for the "Route" field in the report. RTOP has something to do with this report, but I don't quite see where it fits in just yet. Eventually, the same person who explained what RTOP was explained that RTE is an eye catcher for the route control block in a memory dump; there can be multiple RTE entries for a single RTOP.
MOVEABLE IMMED parameter for VIPABACKUP definitions: sticking to Knowledge Center for this one, the MOVEABLE IMMEDIATE parameter refers to the behavior of a DVIPA in the case of stack takebacks. So, if the stack owned the DVIPA then went down, transfers ownership of that DVIPA to the backup stack defined, and then comes back up, the stack will regain control and all new connections for that DVIPA. In this test scenario, the customer does not have MOVEABLE IMMEDIATE defined, so I will need to ensure that it is also not defined on my test systems.
Implicit host route/routing, and how to spot it: I thought way too hard about this one and should have just asked someone right away because, as it turns out, it is simply a host route or a route to an IP address on the HOME list (signified by the H flag in a netstat display). It appears that the term "implicit" is older language exclusive to the mainframe crowd. That happens quite a bit around here.
Multipath, including where it's defined and how to tell that it is enabled: What I want to know is where and how multipath is defined on a stack or router, or at least how to tell that it is being used. Back at the Knowledge Center I came across the Routing section with an OSPF overview. Just as the name suggests, multipath allows for a routing table to contain multiple routes ("paths") to a destination. There is also an IPCONFIG MULTIPATH or NOMULTIPATH statement for the TCP/IP profile, which either enables or disables multipath routing for outbound traffic, respectively. When testing I will need to verify that the MULTIPATH statement is configured for the TCP/IP stack I will use; I can also confirm it is being used by looking at a routing table netstat display. If multiple routes exists for a single destination, multipath is in use, at least for inbound traffic.
How to examine a dump of the TCP/IP address space to find the IP routing table that should show the DRVIPA implicit host route with MTU 576: I know this part is specific to our environment and what we already have set up, but to find the MTU value buried in raw hex could be a formidable undertaking.

Test procedure

The combined PTF records already provide a lot of information that will save me a significant amount of time by outlining steps for recreation with a good amount of detail - high quality test relies on the presence of as much relevant information as possible. The challenge at this point, however, is determining how I will mimic the customer's configuration and recreation steps in our own shared test environment.
For this test I will need at least two systems in the same sysplex. One as the primary distributor of the DRVIPA and one as a backup. Likewise, I'll need a method to "stop the distributor" which can be done using a few different ways such as forcing the stack to the leave the sysplex, or using a VARY DEACTIVATE command on the DRVIPA itself, or stopping the stack entirely. I'll need to define this DRVIPA similar to the customer's configuration and be able to access and modify TCP/IP profiles. Finally I'll need to be able to stop and start OMPROUTE on either stack.

At this point I think it's time to bring up some systems and see all of this for myself.

Stay tuned for Part Two where I dive in and actually do some testing!

Modified on by SamReynolds

Summer 2015 SHARE Conference Wrap-up

SamReynolds ‎ 6,510 Views

The Summer 2015 SHARE Conference was a great educational event and a continued celebration of SHARE’s 60th anniversary! Five speakers from the Enterprise Networking Solutions organization presented 14 sessions on z/OS Communications Server and 6 on ISPF, including two-part hands-on labs for Configuration Assistant for z/OSMF, and the ISPF editor. We also participated in two panel sessions where our attendees brought in plenty of great questions for discussion.

Given the recent availability announcement of z/OS V2R2, there was clearly a focus on z/OS V2R2, including an overview of z/OS V2R2 Communications Server, a detailed review of the Shared Memory Communications for RDMA (SMC-R) capability (including the V2R2 enhancements such as RoCE adapter virtualization), and the ISPF enhancements in z/OS V2R2. Other topics discussed included Enterprise Extender, sysplex technologies, network security, z/OS CS performance, mail, OSA, and connecting CICS with TCP/IP. Attendance at our sessions (and across the board at SHARE in Orlando) was very good, and we would like to thank all of you who attended our sessions for the great feedback and dialogue.

For those that couldn't be at the conference last week, I will remind you that you can download most of the charts for the topics we presented by going to the following link:

https://share.confex.com/share/125/webprogram/start.html

Please plan to join us for the Winter 2016 SHARE Conference in San Antonio, Texas, February 28 - March 4, 2015.

z/OS CS at the Summer 2015 SHARE Conference in Orlando

SamReynolds ‎ 5,859 Views

The Summer 2015 SHARE Conference is in Orlando, Florida next week (August 9th - 14th). As always, there will be a good selection of content focused on z/OS Communications Server, including the following sessions from five of our team here in Research Triangle Park, NC:

z/OS V2R2 Communications Server Technical Update, Part 1 of 2 (Gus Kassimis and Sam Reynolds)
z/OS V2R2 Communications Server Technical Update, Part 2 of 2 (Gus Kassimis and Sam Reynolds)
z/OS Communications Server: Shared Memory Communications over RDMA (SMC-R) Protocol (Gus Kassimis)
z/OS Communications Server Performance: Updates and Recommendations (Dave Herr)
Sysplex and Network Technologies and Considerations (Gus Kassimis)
Getting the Most Out of Your OSA Adapter with z/OS Communications Server (Dave Herr)
Leveraging z/OS Communications Server Applications Transparent Transport Layer Security (AT-TLS) for a Lower Cost and More Rapid TLS Deployment (Lin Overby)
z/OS Communications Server Intrusion Detection Services (Lin Overby)
z/OSMF Configuration Assistant for z/OS Communications Server 2.1 Hands-on Lab, Part 1 of 2 (Todd Valler and Linda Harrison)
z/OSMF Configuration Assistant for z/OS Communications Server 2.1 Hands-on Lab, Part 2 of 2 (Todd Valler and Linda Harrison)
Enterprise Extender on z/OS Communications Server: SNA Hints and Tips (Sam Reynolds)
z/OS Communications Server Hints and Tips (Todd Valler)
Change is Coming: Motivation and Considerations for Migrating from SMTPD/Sendmail to CSSMTP (Todd Valler)
Connecting CICS with TCP/IP (Gus Kassimis and William Yates)

Also, there will be two panel sessions for open discussion of mainframe networking topics:

z/OS Communications Server Free-for-All (Dave Herr, Lin Overby, Sam Reynolds, and Todd Valler)
Ask the Experts: Networking, Network Management, Network Security (Lin Overby, Sam Reynolds, Dean Bulter, and Grant Gibson)

Lastly, I will be presenting the following ISPF topics:

ISPF Hidden Treasures and New Features, Part 1 of 2
ISPF Hidden Treasures and New Features, Part 2 of 2
ISPF Editor - Beyond the Basics Hands-on Lab, Part 1 of 2 (with Tom Conley and Liam Doherty)
ISPF Editor - Beyond the Basics Hands-on Lab, Part 2 of 2 (with Tom Conley and Liam Doherty)
PDSE Member Generations: Implementation and ISPF Exploitation (with Thomas Reed)

We hope to see you there! For those that can’t join us, I’ll be tweeting (IBM_Commserver on Twitter) and posting updates to Facebook (Facebook.com/IBMCommserver) throughout the week.

Feed for Blog Entries Feed for Blog Comments

Blogs

▼ Tags

▼ Similar Blogs

Application P...

Application I...

z/VSE Blog

Network and S...

System z Mana...

▼ Similar Ideation Blogs

IBM PureData-...

主机知刊

▼ Archive

▼ Blog Authors

z/OS Communications Server

Test and Learn Part Two: Actually Testing

PTF test for PI38376: A TCP connection can use the wrong maximum segment size (MSS) on V2R1

Part Two

Test Execution - Coming up with things to do using said stuff

And of course some final thoughts

Planning for the expected - minimizing application unavailability for planned outages

Test and Learn Part One: Learning Things

PTF test for PI38376: A TCP connection can use the wrong maximum segment size (MSS) on V2R1

Part One

Gathering Information - I know some stuff, maybe?

Research - I figure out some stuff

Terms and concepts

Test procedure

Stay tuned for Part Two where I dive in and actually do some testing!

Summer 2015 SHARE Conference Wrap-up

z/OS CS at the Summer 2015 SHARE Conference in Orlando