PTF test for PI38376: A TCP connection can use the wrong maximum segment size (MSS) on V2R1
Part Two
Warning: this part gets pretty technical.
Test Execution - Coming up with things to do using said stuff
What I like to do after understanding the basic idea of what needs to be tested is to just jump straight in and see how far I can get before hitting any roadblocks. Usually these roadblocks are more configuration related, but I often just get completely confused. So what I usually do is revisit the customer's steps for recreation and add my own notes from various observations. That way if I ever need to test something that focuses on these areas of CommServer, I'll have something to go back to instead of relying on only memory.
I know that I need access to a system running V2R1. I also need to know whether the PTF has been applied. There doesn't seem to be an explicit way to determine whether a certain PTF has been applied to one of our IPL drivers, so my only option is to check by the actual test - if I see the results of the fix, then the PTF must be applied. This limitation is frustrating from a testing perspective because I believe it's good practice to perform test with as few assumptions as possible. Likewise, there doesn't seem to be a direct way to recreate the scenario on a system without the PTF applied unless I employ a workaround which adds more time spent on configuration rather than test.
That workaround for this case is to recreate the scenario on a back-level system, so my journey begins on a couple of V1R13 second-level IDs. I use a netstat display to get an idea of what is already set up and what I can use. A netstat -h command (to display the home list) shows me a DRVIPA I can modify and use in this test:
Dynamic VIPA:
IpAddr/PrefixLen: 197.26.201.1/24
Status: Backup Origin: VIPABackup DistStat: Dest
I also look at the routes defined for this DVIPA using a netstat -r:
Destination Gateway Flags Refcnt Interface
----------- ------- ----- ------ ---------
197.26.201.1/32 16.26.81.72 UGHO 0000000000 O3ETHW0
197.26.201.1/32 194.26.80.72 UGHO 0000000000 LIQDC00
197.26.201.1/32 199.26.80.72 UGHO 0000000000 IQDIOLNKC71A500C
So it looks like the DVIPA 197.11.201.1 is a backup for this system, let's call it SysB. Now I'll use a sysplex display command to see who owns it:
VIPA DYNAMIC DISPLAY FROM TCPSVT AT SysB
IPADDR: 197.26.201.1
ORIGIN: VIPABACKUP
TCPNAME MVSNAME STATUS RANK DIST
-------- ------- ------ ---- ----
TCPSVT SysA ACTIVE BOTH
TCPSVT SysB BACKUP 010 DEST
TCPSVT SysC BACKUP 020 DEST
There are three systems in the sysplex who own this DRVIPA that I'm using for test. For the sake of isolation and control, I will force SysC to leave the sysplex for now. Alternatively I could simply delete the DRVIPA definition on SysC, but I decided to take a more destructive route for no particular reason.
SysA is the owner of this DRVIPA, so I'll check the TCP/IP profiles and DVIPA definitions of these systems.
Our chosen DRVIPA is defined, and the MOVEABLE IMMED parameter for the corresponding VIPABACKUP definition was not present but is enabled by default, so I will need to either modify the definition or create my own and specify the MOVEABLE WHENIDLE parameter. Additionally, the relative Knowledge Center documentation states that as a rule, the DYNAMICXCF parameter must be specified in the TCP/IP profile to preserve existing connections on a VIPA takeover. I decided to create my own DVIPA definition files:
Primary distributor definitions:
VIPADYNAMIC
VIPADEFINE MOVEABLE WHENIDLE 255.255.255.0 197.26.201.1
ENDVIPADYNAMIC
VIPADYNAMIC
VIPADIST DISTM BASEWLM SYSPLEXPORTS 197.26.201.1
PORT 21 80 623 923 925
927 928 929 931 932
1821 52002 50030
DESTIP ALL
ENDVIPADYNAMIC
Backup stack definition:
VIPADYNAMIC
VIPABACKUP 10 MOVEABLE WHENIDLE 255.255.255.0 197.26.201.1
ENDVIPADYNAMIC
I'll spare the details of error recreation and move on to the good stuff: seeing the fix in action!
1. Define the DRVIPA to be used
Since our SVT systems run all day, it's likely that the host route for my DRVIPA has already been created. Fortunately I found some instructions to remove this route without having to re-IPL or recycle the stacks:
a. vary obey a file to delete the the DRVIPA
b. verify the route is no longer present with D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
c. verify the DRVIPA itself is deleted with D TCPIP,,N,VIPADCFG,IPADDR=197.26.201.1
d. vary obey a file to define the DRVIPA
e. verify the system has successfully re-added the DRVIPA and routes with the same commands
2. Start TCPIP on a backup system without OMPROUTE to simulate delay
After running this step I noticed that OMPROUTE was starting automatically when starting the stack. Re-examining the TCP/IP profile gives me the explanation for this behavior: Autolog was defined to automatically start OMPROUTE when the stack was started, so I just need to comment this out.
Then I noticed that the TCP/IP stacks were not joining the sysplex when started while OMPROUTE was down. Turns out that this is a result of the GLOBALCONFIG DELAYJOIN profile parameter, which prevents the stack from joining the sysplex unless OMPROUTE is started. I need to comment this out as well.
And finally the stack was refusing to join the sysplex due to not finding dynamic routes over monitored interfaces, so I commented out GLOBALCONFIG SYSPLEXMONITOR MONINTERFACE and added GLOBALCONFIG SYSPLEXMONITOR NOMONINTERFACE.
Since the default options for the systems in our test environment are fairly different from the profile options I need to recreate the customer's configuration, I will copy the existing profile to my own custom profile and start the stack with an override parameter: S TCPSVT,HOST=MYPROF,REUSASID=YES
Finally, I have everything configured correctly and got some displays:
D TCPIP,,N,VIPADCFG,IPADDR=197.26.201.1
15.25.26 EZD0101I NETSTAT CS V2R1 TCPSVT 786
DYNAMIC VIPA INFORMATION:
VIPA BACKUP:
IPADDR/PREFIXLEN: 197.26.201.1/24
RANK: 010 MOVEABLE: WHENIDLE SRVMGR: NO FLG:
D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
EZD0101I NETSTAT CS V2R1 TCPSVT 830
IPV4 DESTINATIONS
DESTINATION GATEWAY FLAGS REFCNT INTERFACE
197.26.201.1/32 0.0.0.0 UH 0000000000 VIPLC51AC901
METRIC: 00000000 MTU: 65535
MVS SPECIFIC CONFIGURED PARAMETERS:
MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES
3. Start and stop the primary distributor without OMPROUTE to force the DRVIPA takeover on the backup stack
For this I will start the stack on the primary distributor then use a sysplex command to deactivate the DVIPA: v tcpip,tcpsvt,sys,deact,dvipa=197.26.201.1
EZD1298I DYNAMIC VIPA 197.26.201.1 DELETED FROM TCPSVT
EZZ8303I VIPA 197.26.201.1 GIVEN TO TCPSVT ON SysA
From the backup stack:
D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
15.27.35 EZD0101I NETSTAT CS V2R1 TCPSVT 865
IPV4 DESTINATIONS
DESTINATION GATEWAY FLAGS REFCNT INTERFACE
0 OF 0 RECORDS DISPLAYED
END OF THE REPORT
VIPA DYNAMIC DISPLAY FROM TCPSVT AT SysB
IPADDR: 197.26.201.1
ORIGIN: VIPABACKUP
TCPNAME MVSNAME STATUS RANK DIST
-------- -------- ------ ---- ----
TCPSVT SysA ACTIVE
TCPSVT SysB BACKUP 010
And on the primary:
EZD0101I NETSTAT CS V2R1 TCPSVT 205
IPV4 DESTINATIONS
DESTINATION GATEWAY FLAGS REFCNT INTERFACE
197.26.201.1/32 0.0.0.0 UH 0000000000 VIPLC51AC901
METRIC: 00000000 MTU: 65535
MVS SPECIFIC CONFIGURED PARAMETERS:
MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES
Then stopping the primary distributor...
EZZ8301I VIPA 197.26.201.1 TAKEN OVER FROM TCPSVT ON SysA
D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
EZD0101I NETSTAT CS V2R1 TCPSVT 648
IPV4 DESTINATIONS
DESTINATION GATEWAY FLAGS REFCNT INTERFACE
197.26.201.1/32 0.0.0.0 UH 0000000000 VIPLC51AC901
METRIC: 00000000 MTU: 65535
MVS SPECIFIC CONFIGURED PARAMETERS:
MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES
4. Restart the primary distribute with OMPROUTE to takeback the DRVIPA: v tcpip,tcpsvt,sys,react,dvipa=197.26.201.1
5. Start OMPROUTE on the backup so OSPF host routes will be learned from the distributor
D TCPIP,,N,ROUTE,DETAIL,IPADDR=197.26.201.1
EZD0101I NETSTAT CS V2R1 TCPSVT 868
IPV4 DESTINATIONS
DESTINATION GATEWAY FLAGS REFCNT INTERFACE
197.26.201.1/32 16.26.81.72 UGHO 0000000000 O3ETHW0
METRIC: 00000171 MTU: 1500
MVS SPECIFIC CONFIGURED PARAMETERS:
MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES
197.26.201.1/32 199.26.80.72 UGHO 0000000000 IQDIOLNKC71A500C
METRIC: 00000171 MTU: 1500
MVS SPECIFIC CONFIGURED PARAMETERS:
MAXRETRANSMITTIME: 120.000 MINRETRANSMITTIME: 0.500
ROUNDTRIPGAIN: 0.125 VARIANCEGAIN: 0.250
VARIANCEMULTIPLIER: 2.000 DELAYACKS: YES
6. If a connection is established from the backup to the DRVIPA on the distributor, a netstat display on this connection should show the MSS is not set to 536
I used an internal traffic simulation tool to establish a connection from the backup stack to the DRVIPA. First, I issued a netstat -V DETAIL -P 50030 to see the connection on the distributor, then used the unique port as a filter on a netstat -A display on the backup stack:
MVS TCP/IP NETSTAT CS V2R1 TCPIP Name: TCPSVT 10:55:43
Dynamic VIPA Connection Routing Table:
Dest: 197.26.201.1..50030
Source: 16.26.81.12..1487
DestXCF: 199.26.80.12
PolicyRule: *NONE*
PolicyAction: *NONE*
Intf: IQDIOLNKC71A5048
VipaRoute: No Gw: 0.0.0.0
Local Socket: 16.26.81.12..1487
Foreign Socket: 197.26.201.1..50030
BytesIn: 00000000000000650000
BytesOut: 00000000000000650000
SegmentsIn: 00000000000000000001
SegmentsOut: 00000000000000000002
StartDate: 08/26/2015 StartTime: 14:55:26
Last Touched: 14:56:26 State: Establsh
RcvNxt: 0399845097 SndNxt: 4265975520
ClientRcvNxt: 0399845097 ClientSndNxt: 4265975520
InitRcvSeqNum: 0399195096 InitSndSeqNum: 4265325519
CongestionWindow: 0000004344 SlowStartThreshold: 0000065535
IncomingWindowNum: 0399976165 OutgoingWindowNum: 4266041055
SndWl1: 0399195096 SndWl2: 4265325520
SndWnd: 0000065535 MaxSndWnd: 0000065535
SndUna: 4265975520 rtt_seq: 4265325519
MaximumSegmentSize: 0000001448 DSField: E0
...
Definitely not an MSS of 536.
7. Dump the TCP/IP address space to determine the MTU size that was set
After consulting with the more experienced ranks for this task, I got a little more experience with IPCSGO to view a dump in a slightly more manageable way than just a bunch of raw data. From there I was able to view the IP routing table with the IPCS primary options menu: CS/OS390 --> TCP/IP --> Configuration --> Route --> Ipv4, then entering the name of my TCP/IP stack. At the top of this table its address space ID within the TCP/IP address space is displayed which I then use to view the MTU value that was selected for this route. Likewise, I'll need the address of the actual route I want to examine. A find on my IP address gives me the entry and the address:
"Configuration" is the keyword I was looking for, since there are multiple routes for this particular destination.
But now I have to dive in to that bunch of hexadecimal raw data. Back to the main IPCS menu, the Browse option allows me to view the contents of the dump as hex characters. I have to search for the ASID within the address space that was dumped, so I change the Override Defaults Address space option with ASID(X'00CC') and hit enter. [Hint: this can also be done from the Defaults option in the IPCS primary option menu]
This next part was pretty intense, and it involved looking at the actual Communications Server code using another resource to determine at which pointer combined with which offset were the RTE and RTIOCTL eye catchers (on the right in the image). From there, I need to move to the RTOP, which is the eye catcher that indicates a set of RTE control blocks representing a route to a specific IP address. The RTOP is referenced at this address:
Putting a question mark beside that address and hitting enter takes me to that address. Then, counting from a pointer to a certain offset amount (as determined from the actual code) tells me what the MTU was set for this RTOP.
That guy.
0000FFFF is 65,535 in decimal. Looking back at the test information:
"The problem occurs when an implicit host route for the DRVIPA is generated with the default MTU 576 instead of 65535 on a backup system."
And the MTU for the route to 197.26.201.1 on the backup system is 65,535 instead of 576. Looks like it's working! From here I can look at all of the RTE control blocks under this particular RTOP to see what MTU values were set. The RTOP MTU should be the lowest value of all RTE control blocks, with the exception of the host route. I can move from one RTE to the next by putting a question mark by the pointer; once the "next pointer" is a bunch of zeroes, that's the last RTE under this RTOP:
No, I don't just magically know this. It's determined by looking at the offset specified in the code. It'd be cool if I did magically know this, though.
Of course, I want to be certain that the PTF is, in fact, applied. To do this I can compare the date displayed in the eye catcher of the module that was edited:
Magic and sorcery.
Browsing through that same address space and using the command "f ezbxfdvi nobreak" to search for the name of the module that was edited for this fix brought me to what I was looking for. Here I can see both the last date it was modified ("15.189" for 2015, day 189) and the internal PTF number which corresponds to my PTF record. This confirms that the PTF has been applied on our test systems and has fixed the issue.
And of course some final thoughts
People are helpful around here and I'm grateful for that, because even the best documentation could only get me so far. The level of customization that's possible just for the TCP/IP stack is immense, so learning one type of customer configuration by having to mimic it in a test environment feels like I'm only scratching the surface.
By testing a fix for an error that a customer found, it was interesting to gain a new perspective other than my normal set of activities revolving around testing pre-GA'd code. I now have notes explaining new concepts and had a chance to explore something new in the brain-frying world of z Systems Parallel Sysplex, with minimal brain frying.
Likewise, I'm glad to be part of something that values the customer. This error was very specific, but it may be something to investigate in our own test suites.