Editor's note: This article is part of a library of custom VuC functions developed by performance test engineers to handle special situations encountered in scripting applications. For an overview of the library and links to other custom functions, see "A Library of Custom VuC Functions."
The html_from_cmd_data utility parses the command data report for a test run to reconstruct all of the HTML pages and optionally other objects (such as images and JAR files) that are returned during playback of a VU HTTP test script, as a set of individual files. This allows direct comparison with the files recorded in the VU script (which can be generated with the companion utility,
html_from_vu
) in order to track down playback issues.
The utility was originally written to help track down the cause of an obscure problem that was making script playback fail, by comparing the HTML pages returned at playback time with those returned when the script was originally recorded. This approach allows the recording-time and playback-time versions of all HTML pages visited in a set of scripts to be compared quickly using a file comparison utility such as WinDiff or CSDiff. Those files that differ can be compared by eye using a browser, or the individual differences can be studied with the file comparison tool. This saves substantial amounts of time in tracking down playback issues.
Similarly, the pages returned by different virtual users can be compared quickly. This can be useful when trying to isolate the cause of intermittent load-related failures. The utility generates files for all virtual testers included in the command data report output. Simply comparing the files for a virtual tester that reported errors with one that reported no errors can help isolate issues very quickly.
However, detailed comparisons like this aren't always required -- very often, simply viewing the regenerated HTML for pages causing playback errors will give an immediate clue as to why the test is failing.
Note that it's also possible to capture the pages returned at playback time by inserting additional code into the VU script. Scott Barber discusses many of the reasons for wanting to access the returned pages, along with some clever techniques for doing so, in Part 5 of his excellent "Beyond Performance Testing" series. Had I read that article before I came across the obscure error that inspired this utility, I would probably have opted for doing it Scott's way. However, having written the utility, I generally prefer this approach because it doesn't require any modifications to the test script.
The utility doesn't only generate the HTML pages. Optionally, all objects referenced on the page (such as images and JAR files) and even the individual HTTP headers can be saved as separate files. This is less often useful than extracting the HTML, but in some circumstances can be extremely powerful when trying to verify that scripts have been behaving correctly, after the event.
One notable example of this was a test of a Web portal that delivered graphs generated
dynamically from a data warehouse. Following a test run, inspection of the test log or the command data report allowed verification that something had been returned when a graph was requested, but it was hard to say whether it was the correct result from this:
>>>> cmdid: [ProdGrap~340][340] src: [ProdGraphs.s] line [6648] GIF89ad`013101f10000ffffff0000000000ffff00002c00000000640131010002ff848fa9cbed 0fa39cb4da8bb3debcfb0f86e24896e689a6eacab6ee0bc7f24cd7f68de7facef7fe0f0c0a87c4 a2f188`L*`97cca6f3098d4aa7d4aaf58acd6ab7dcaef70b0e8bc7e4f281109fe063aa45107ccc 0c358a4df0d122221348081f5397db1f`F9i`b51767fd48fc20f8980a11041f132314`(~`107c8 c`@DJ_`1f5397db1f`F9i`b51767fd48020003`0B`02001141f0`# |
Turning that "mixed hex" representation of the image back into a GIF file made the verification much easier. Figure 1 is the actual GIF extracted from the above text.
Figure 1: Actual GIF extracted from its "mixed hex" representation
In this case, the ability to view the GIFs returned at playback time allowed visual confirmation that the scripts were requesting the correct graphs based on parameters driven from a datapool. The submission form in this case was quite complex, with fairly obscure hidden fields that needed to be correlated correctly in order to drive the test from the datapool. With the architecture of the portal under test, if the parameters defining the graph hadn't been correlated correctly, the same graph would have been requested repeatedly and the results would have been delivered from the portal's page cache instead of regenerated by returning to the data warehouse for the source data. This would have understated the load substantially.
Being able to demonstrate visually to the client that the test was returning the correct graphs significantly increased the client's confidence (and mine!) in the realism of the test.
A Perl runtime environment is required in order to run the utility, which is a Perl script. The utility was developed using ActivePerl version 5.8.1, but it should work with any relatively recent release of Perl (version 5 or above). ActivePerl is available for free download from the ActiveState site. A Perl runtime environment is also installed automatically as part of the installation of various other products, including some of the Rational suites. Alternatively, Perl is generally available on any Unix or Linux platform.
Note that in order for the command data report to contain the necessary detail, full logging must have been enabled for the test run (for at least some virtual testers).
The utility code is contained in the file html_from_cmd_data.pl, supplied in a ZIP file that can be downloaded. To install the utility, place the Perl script in a directory that's on the path so that it can be used without specifying a path name (for example, create a directory called c:\perlscripts and add that to the path).
html_from_cmd_data.pl [options] cmd_data_file
where cmd_data_file is the name of a file containing the command data report from which to extract result data. Note that the test must have been run with full logging enabled and the command data report must include results for all commands (not just failures). This is explained in more detail below.
The options are as follows:
-
-help-- Displays a summary list of commands. -
-ddir -- Specifies the root directory under which to create output files (defaults to the current working directory). -
-a-- Instructs the utility to regenerate all file types, not just HTML. -
-hdr-- Creates files for HTTP response headers. -
-no_skip_dups-- Instructs utility not to ignore multiple occurrences of the same command in one run of a script. (There's a minor bug in TestManager that sometimes generates additional, spurious entries for the same command in the command data report, causing the utility to get confused and leading to truncation of HTML pages. The default behavior of the utility is therefore to skip duplicate entries to avoid this issue. But if the VU script has been edited so that some commands are executed repeatedly during a single iteration of the script, the logic to avoid the TestManager bug has the effect of suppressing the second and subsequent occurrences of each page. This option is provided to override the default behavior and attempt to handle the repeated entries. If you have genuinely repeated execution of commands and happen to hit the bug in the command data report, you'll have to try both options and make do with what you get.)
Follow these steps to generate output files from a command data report:
- Before running the suite, enable full logging for the virtual testers required by selecting all "Log data control" checkboxes under the Logging tab of the TSS Environment Variables window.
- Following completion of the test run, click Reports > New > Command Data on the TestManager menu, then click the Change Options button and select "Include success and failure messages" and "Include error messages" to generate a command data report including success, failure, and error messages.
- Run the command data report and save it.
- Use Windows Explorer to locate the saved report in the repository and copy the file to a temporary directory. The report will be found as a file with a long, ugly name and the extension
.outin the log folder directory for the test run:"..\TestDatastore\TMS_Builds\Build 1.Build\Default.LogFolder\<Log Name>.Log"Note that it's important to copy the file like this, rather than using cut and paste from the report viewer window, to avoid changing how end-of-line characters are represented in the file.
- Start a command prompt and change directory to the location of the report file.
- Invoke the Perl script, specifying whatever options you require. For example, to generate all file types from the report file
Sample Report.out, you would enter the following command:C:\Temp\cmd_data_files>html_from_cmd_data.pl -a "Sample Report.out"
Details of the virtual testers and VU scripts processed are displayed:
User Group1[1] Script_A Script_B Script_C User Group1[2] Script_A Script_B Script_C User Group2[1] Script_D Script_E |
The generated files are stored in a directory hierarchy by virtual tester and test script. The filenames of the generated files are based on the command IDs from the test scripts. If a single user executes a script repeatedly, a sequence number is added to the filenames for uniqueness.
For example, the sample report file Sample Report.out would generate the following structure:
\Sample Report\
\UserGroup1[1]\
\ScriptA\
\ScriptB\
\ScriptC\
\UserGroup1[2]\
\ScriptA\
\ScriptB\
\ScriptC\
\UserGroup2[1]\
\ScriptD\
\ScriptE\ |
Comparing the files generated with this utility with the equivalent files extracted from the recorded VU script using
html_from_vu
is easiest with a good file comparison utility such as WinDiff or CSDiff, which are freely available for download. WinDiff is particularly useful as it allows comparison of entire directories and can be configured so that it only reports on files that exist in both locations (some files are generated from the command data report that are not present in the VU script, and vice versa).
The first step is a directory-level comparison showing a summary of all files present in the two directories, or sets of directories, to highlight the files that differ. You can compare the directories for the recording and playback of a single VU script one by one. Alternatively, you can compare in a single pass a set of directories corresponding to all scripts executed by one virtual tester, by placing copies of all VU scripts executed by the virtual tester in one directory before generating the HTML from recording time. Refer to the description of html_from_vu for details.
Figure 2 shows the summary of differences displayed by WinDiff for the issue that originally inspired this utility.
Figure 2: Summary of file differences displayed by WinDiff
Drilling down in turn on each of the files marked as different, we found that most of the differences were easily explained and weren't material. (Typically, these were differences either in variables such as session cookies or in fields such as current date and time.) The highlighted file had very significant differences, however, as shown by the red and yellow differences bar at the left of the screen in Figure 3.
Figure 3: Areas of significant difference in one of our files
In this particular case, simply opening the two versions of the HTML page with a browser didn't give any immediate clues -- the two pages looked similar at first glance and there were no obvious errors. The file generated at playback time is shown in Figure 4.
Figure 4: File generated at playback time
Detailed comparison showed that there was in fact a visible difference -- the presence of the Apply button on one page and not on the other, which provided a clue as to what was happening. Closer inspection of the differences with WinDiff showed that the recorded HTML (highlighted in red in Figure 3) contained large amounts of JavaScript code, whereas that from playback (highlighted in yellow) contained only a couple of lines of JavaScript, at the very start of the part of the file that differed. The line of JavaScript
document.cookie = "DiscovererViewerJavaScript=Enabled"
sets a cookie if the browser has JavaScript enabled. This is a trick used by the Oracle Discoverer environment to determine whether the browser is JavaScript capable. If it is, the next request from the browser will have the cookie set, and Discoverer will know that it can send an intelligent version of the HTML pages (including substantial amounts of JavaScript). If the browser isn't JavaScript capable, the cookie won't be set and Discoverer will send a very different version of the pages (including the Apply button shown above, since it's not possible to perform actions automatically on entry of field values without JavaScript).
Robot handles cookies automatically only if they're set via the HTTP header, not if they're set in JavaScript. This meant that Discoverer treated the virtual testers as if they weren't JavaScript enabled and sent entirely different pages from those expected, resulting in numerous playback errors.
When we had tracked this down, it was simple to fix, by the way. The VU script merely needed to set the cookie (with the VU set_cookie command) at the appropriate point.
As mentioned, the utility was created to assist with a particular performance-testing project and has been enhanced over the course of a few subsequent assignments. It hasn't been tested thoroughly over a wide range of Web sites, so there will undoubtedly be situations that it doesn't handle. If you find it useful but there are specific cases that it doesn't cope with, feel free to contact me with the details and I'll add them to the list of things that might get addressed one day. No promises. Alternatively, enhance it for me and let me have an improved copy back.
There are a few known limitations:
- VU scripts for chunk-encoded HTTP connections that were recorded with the network recording method generate spurious hex strings in the extracted HTML. This is due to a minor issue in TestManager -- it treats the byte counts for the HTTP chunks as part of the HTML.
- Some files may be generated with inappropriate file extensions. The script tries to work this out based on a combination of what was requested and the content type returned -- but this isn't straightforward and I've spent far too long on it already!
- The script may not work reliably if the command data report has been saved or edited from a Windows-based editor. For example, copying and pasting from the report viewer to WordPad in order to select the portion for one virtual tester probably won't work. DOS/Windows format new lines (CRLF) can break the script. Filter the virtual testers required when running the command data report and simply copy the file, as described above.
- In some circumstances, TestManager seems to get slightly confused when generating the command data report, resulting in responses being reported for the wrong command IDs. This in turn means that the HTML files generated by this utility have the wrong command ID in their names, so a direct comparison with the files generated from the original VU script won't match up some of the files correctly. This makes doing the comparisons just a little more difficult, but it's still easier than reading the raw command data report.
Richard Leeke is one of the founding partners of Equinox Ltd., a software architecture consulting company based in Wellington, New Zealand. Richard specializes in the areas of performance testing and the diagnosis and resolution of application, database, and infrastructure related performance issues. He's a fairly regular contributor (of both questions and answers) to various performance testing related forums and can be contacted by e-mail.
Comments (Undergoing maintenance)
Table of contents
Local resources
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
