Bob Sutor wrote:
I’m going to tackle the issue of quality and standards organizations in a future entry, but let me say thatI couldn't agree with Bob's points more. His post follows on with a set of proposed criteria by which a standard can be measured. The comments also have a number of highly insightful observations and recommendations for additional criteria.
- Standards organizations are not all equal in quality, though it doesn’t seem that everyone knows that.
- A given standards organization can produce two standards of wildly divergent quality.
- In my opinion, the key measurement of a standards organization isnot the quantity of standards produced by the quality of standardsproduced.
Within many of the standards organizations in which I participate, there has been a gradual, but steady progression towards producing higher quality specifications. At OASIS, there is a criteria for becoming an OASIS Standard that there be three statements from TC members that they were "using" the specification. The running joke was that the meaning of "use" could be that you were "using" the specification to raise a leg on your kitchen table so it didn't wobble. We've come a long way since those days. Unfortunately, the OASIS TC process hasn't improved this requirement (despite my constant rantings that they should:-). However, it is now typical that a TC will hold an interoperability event(s) to test multiple independent implementations of the specification(s) prior to advancing to OASIS Committee Specification status. It would be really nice if OASIS would make such interoperability testing a requirement, and change the requirement from a "Statement of Use" to something a bit more substantial, such as the requirements that the W3C imposes on specifications advancing to W3C Recommendation status.
Within the W3C there is a requirement for advancing a specification to W3C Recommendation, there must be (typically) a minimum of two or more interoperable implementations of each feature in the specification. Those features that the working group feels might not meet the requisite exit criteria for the CR phase (also known as the Call for Implementations phase), are marked as "at risk" and must be removed from the specification should they not achieve the stated exit criteria.
In the WS-I, the Board recently adopted a resolution that in order to advance to WS-I Final Material status, that a Profile had to have 5 independent interoperable implementations of the profile. Some have made a stink about this new requirement, but IMO, it is a "Good Thing(tm)" because for a Profile to improve interoperability, it needs to be broadly adopted/implemented.
But, I digress...
The addition of test cases/assertions, as suggested by Arnaud and others in comments to Bob's post, that can be used to measure conformance to the specification are, IMO a key to delivering a higher quiality standard. I'd like to see more of this across the standards landscape. This is something that I have been pushing for in WS-I Profiles, and something that will manifest itself when we publish the WS-I Basic Profile 2.0 draft (real soon now, I hope).