Topic
  • 12 replies
  • Latest Post - ‏2012-12-06T15:43:40Z by KeithCollyer
JonChard
JonChard
6 Posts

Pinned topic Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread [Event closed]

‏2012-11-29T14:52:34Z |
**Even though this event is over, you are welcome to browse this discussion**
 
 This thread is discussing agile systems engineering with respect to asset- versus process-driven environments, documentation issues and contract-driven environments.

Some key points that have been raised in the main thread include:

  • Some systems customers expect comprehensive documentation and do not expect to see an incomplete implementation.
  • Improving documentation processes can yield significant cost savings – Agile is a possible way to achieve this.
  • Agile is about less documentation and more working stuff.
  • How to deal with shared risks when deliverables move from documentation to working stuff?
  • ‘Working stuff’ is more honest than impressive documentation.
  • Customers or contract monitors must collaborate in the Agile process.
Updated on 2013-01-07T22:40:03Z at 2013-01-07T22:40:03Z by sjpeich
  • WalkerRoyce
    WalkerRoyce
    14 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-11-29T15:33:53Z  
    This seems to me to be the crux of successful improvement in outcomes.
     
    To transform successfully from conventional engineering governance to more agile economic governance requires a significant cultural transformation. This is best achieved through the pursuit of one simple change theme: Integration testing should precede unit testing. In practice, this theme is overly simplistic: Integration and unit testing actually proceed in parallel. However, to accelerate the transformation to increased agility, it is best to simplify and clarify that the highest priority is to achieve intermediate milestones of executable test cases of integrated functionality.

    Keeping this integration-first theme in mind will help you increase the agility of the larger project team, no matter what role you play in a software project:

    • Project managers. Lay out plans, resources, and measures that prioritize integration testing of key usage scenarios in multiple checkpoints for stakeholder steering.
    • System engineers. Analyze the system context to define first the integrated behaviors, qualities, and usage scenarios that represent the holistic value and whose elaboration will reduce the most uncertainty. 
    • Architects. Elaborate the architecture of the solution and the evaluation criteria for incremental demonstration of the most important behaviors and attributes, such as changeability, performance, integrity, security, usability, and reliability.
    • Testers. Identify testing infrastructure, data sets, sequences, harnesses, drivers, and test cases that permit automated regression testing and integration testing to proceed without reliance on completely tested units and components.
    • Designers/developers. Develop units, services, and components that are always executable and testable, evolving from initial versions that permit execution within their usage context (that is, satisfy their interface) in a trivial way, and then progress toward more complete components that meet quality expectations across their entire operational spectrum.
    [This is an excerpt from a paper that I have submitted to ICSE 2013: Agility at Scale, Economic Governance, Measured Improvement and Disciplined Delivery. Alan Brown and Scott Ambler were co-authors.]
  • KeithCollyer
    KeithCollyer
    14 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-11-29T17:01:17Z  
    This seems to me to be the crux of successful improvement in outcomes.
     
    To transform successfully from conventional engineering governance to more agile economic governance requires a significant cultural transformation. This is best achieved through the pursuit of one simple change theme: Integration testing should precede unit testing. In practice, this theme is overly simplistic: Integration and unit testing actually proceed in parallel. However, to accelerate the transformation to increased agility, it is best to simplify and clarify that the highest priority is to achieve intermediate milestones of executable test cases of integrated functionality.

    Keeping this integration-first theme in mind will help you increase the agility of the larger project team, no matter what role you play in a software project:

    • Project managers. Lay out plans, resources, and measures that prioritize integration testing of key usage scenarios in multiple checkpoints for stakeholder steering.
    • System engineers. Analyze the system context to define first the integrated behaviors, qualities, and usage scenarios that represent the holistic value and whose elaboration will reduce the most uncertainty. 
    • Architects. Elaborate the architecture of the solution and the evaluation criteria for incremental demonstration of the most important behaviors and attributes, such as changeability, performance, integrity, security, usability, and reliability.
    • Testers. Identify testing infrastructure, data sets, sequences, harnesses, drivers, and test cases that permit automated regression testing and integration testing to proceed without reliance on completely tested units and components.
    • Designers/developers. Develop units, services, and components that are always executable and testable, evolving from initial versions that permit execution within their usage context (that is, satisfy their interface) in a trivial way, and then progress toward more complete components that meet quality expectations across their entire operational spectrum.
    [This is an excerpt from a paper that I have submitted to ICSE 2013: Agility at Scale, Economic Governance, Measured Improvement and Disciplined Delivery. Alan Brown and Scott Ambler were co-authors.]
    Walker, I'm missing something here. I can fully understand how you can do integration first testing for software - in fact I have seen case studies of companies that dropped unit testing altogether as it simply wasn't cost effective (mind you, they also did rigorous inspections of designs and code). But I have real difficulty seeing how this can be applied to physical products. Unless you are using "testing" as a short-hand to include verification and validation,which can be done on work products (such as models, documents, etc.) other than final items. In any case, can you point at some material that explains how to apply this principle for physical systems.
     
    Thanks
    Updated on 2012-11-29T17:01:17Z at 2012-11-29T17:01:17Z by KeithCollyer
  • WalkerRoyce
    WalkerRoyce
    14 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-11-29T17:29:23Z  
    Walker, I'm missing something here. I can fully understand how you can do integration first testing for software - in fact I have seen case studies of companies that dropped unit testing altogether as it simply wasn't cost effective (mind you, they also did rigorous inspections of designs and code). But I have real difficulty seeing how this can be applied to physical products. Unless you are using "testing" as a short-hand to include verification and validation,which can be done on work products (such as models, documents, etc.) other than final items. In any case, can you point at some material that explains how to apply this principle for physical systems.
     
    Thanks
     You are not missing anything. I am talking about software integration. These same principles should be applied to the software in physical products where software is the dominant component. This is easy in some contexts, harder in others, depending on the availability of the hardware platform, hardware components, simulators, and other infrastructure support.
     
    The default implementation of the V model in most software intensive systems (such as OFPs, real-time control, command centers, simulators, etc.) tends to demand that unit/component/service tests be complete before initiating integration/system behavioral and performance testing. This postpones the resolution of the bigger uncertainties and "malignant" changes uncovered in integration tests, and instead over-invests in completeness and coverage testing in the components are ensuring that they are free of "benign" changes and defects. The predictable result of this misguided planning priority is that 40% or more of the life-cycle is expended in late scrap and rework.
     
  • BrianNolan
    BrianNolan
    9 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-11-29T18:25:41Z  
    Walker, I'm missing something here. I can fully understand how you can do integration first testing for software - in fact I have seen case studies of companies that dropped unit testing altogether as it simply wasn't cost effective (mind you, they also did rigorous inspections of designs and code). But I have real difficulty seeing how this can be applied to physical products. Unless you are using "testing" as a short-hand to include verification and validation,which can be done on work products (such as models, documents, etc.) other than final items. In any case, can you point at some material that explains how to apply this principle for physical systems.
     
    Thanks
     Keith, check out Mike Mott's (elko.mike) article: http://www.iafastro.net/iac/archive/browse/IAC-07/D1/3/7286/
     
    and his posts yesterday on main thread at 2:05 and 2:42 (I think).  Pulling integration forward in heavy physical environment involves lots of modeling, simulation, and emulation (although some of that can be reduced, as Mike notes).  I suppose this is V+V instead of testing, but the risk reduction and integration problems reduced are, I believe, significant.
  • SteveDiCamillo
    SteveDiCamillo
    15 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-11-29T20:58:00Z  
     Thanks! OK, that makes a lot of sense. In which case I am totally in agreement.
     
    However... If you use executable models (which can be of physical systems), you can, and probably should, start doing validation early in the game.
     
    Edited to add...
     
    And if you treat early validation as early integration testing then Walker's approach (generalized) still applies. In fact, even more so, as the earlier that you start the validation, the higher the level of the artefacts that you are working with. So, with this insight, we can generalize "integration test before unit test" to "validate early (validate often?)", which is a sound systems engineering principle. Alternatively, we can see this as specializing the SE principle to a particular as applied to software.
    Right - this is a modified V with continuous integration testing of virtual/abstract subsystems and components as you progress down the left side of the V.  This approach works well for some cases, but where there are complex SW controls for complex electromechanical products, this can be very difficult to accomplish.  The goal is to design quality into the product rather that test quality into the product.  It's interesting that this goal is very prominent in most safety critical development standards...and those same standards tend to rely heavily on a process that require extensive documentation and tons of auditable artifacts to show that this goal was met.
  • KeithCollyer
    KeithCollyer
    14 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-11-30T09:30:11Z  
     You are not missing anything. I am talking about software integration. These same principles should be applied to the software in physical products where software is the dominant component. This is easy in some contexts, harder in others, depending on the availability of the hardware platform, hardware components, simulators, and other infrastructure support.
     
    The default implementation of the V model in most software intensive systems (such as OFPs, real-time control, command centers, simulators, etc.) tends to demand that unit/component/service tests be complete before initiating integration/system behavioral and performance testing. This postpones the resolution of the bigger uncertainties and "malignant" changes uncovered in integration tests, and instead over-invests in completeness and coverage testing in the components are ensuring that they are free of "benign" changes and defects. The predictable result of this misguided planning priority is that 40% or more of the life-cycle is expended in late scrap and rework.
     
     Thanks! OK, that makes a lot of sense. In which case I am totally in agreement.
     
    However... If you use executable models (which can be of physical systems), you can, and probably should, start doing validation early in the game.
     
    Edited to add...
     
    And if you treat early validation as early integration testing then Walker's approach (generalized) still applies. In fact, even more so, as the earlier that you start the validation, the higher the level of the artefacts that you are working with. So, with this insight, we can generalize "integration test before unit test" to "validate early (validate often?)", which is a sound systems engineering principle. Alternatively, we can see this as specializing the SE principle to a particular as applied to software.
    Updated on 2012-11-30T09:30:11Z at 2012-11-30T09:30:11Z by KeithCollyer
  • Victoria Bunyard
    Victoria Bunyard
    13 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-12-05T09:34:47Z  
     However the contract driven model requiring comprehensive documentation is still prevalent particularly in industries with safety critical elements - so delivering working stuff is good, but there will always be a need for good documentation (note that comprehensive and good do not equate to heavy, or even ''on paper'' in this context)
     
    The statement:
     
    ''Improving documentation processes can yield significant cost savings – Agile is a possible way to achieve this. ''
     
    This is true - but it can only be achieved if the ''need for'' and ''quality of'' the documentation is a core, designed in, element of the agile processes adopted.
    Updated on 2012-12-05T09:34:47Z at 2012-12-05T09:34:47Z by Victoria Bunyard
  • GustavoGrillo
    GustavoGrillo
    2 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-12-05T16:22:27Z  
     However the contract driven model requiring comprehensive documentation is still prevalent particularly in industries with safety critical elements - so delivering working stuff is good, but there will always be a need for good documentation (note that comprehensive and good do not equate to heavy, or even ''on paper'' in this context)
     
    The statement:
     
    ''Improving documentation processes can yield significant cost savings – Agile is a possible way to achieve this. ''
     
    This is true - but it can only be achieved if the ''need for'' and ''quality of'' the documentation is a core, designed in, element of the agile processes adopted.
    I think the problem here is to know what do you need documentation for?
    Software requirements aren't always the best kind of documentation, but these words are often used interchangeably.
    I usually ask a lot of "whys" to governance needs because sometimes people are trying to fulfill  old interpretations of the needs and not the needs themselves.
    I don't know if improving documentation reduces costs, it can easily increase them if you document for the sake of displaying  beautiful documents on the the company management retreat. You need to know what you mean by 'improving': more detail? faster, yet not so detailed? aligned with the corporate architecture?
    This is where agile meets lean: you gotta do the least work (no matter if it is coding or documentation) to achieve the REAL goals of the business. We shouldn't document for the QA department, we should document for the customer.
  • Victoria Bunyard
    Victoria Bunyard
    13 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-12-05T17:44:01Z  
    I think the problem here is to know what do you need documentation for?
    Software requirements aren't always the best kind of documentation, but these words are often used interchangeably.
    I usually ask a lot of "whys" to governance needs because sometimes people are trying to fulfill  old interpretations of the needs and not the needs themselves.
    I don't know if improving documentation reduces costs, it can easily increase them if you document for the sake of displaying  beautiful documents on the the company management retreat. You need to know what you mean by 'improving': more detail? faster, yet not so detailed? aligned with the corporate architecture?
    This is where agile meets lean: you gotta do the least work (no matter if it is coding or documentation) to achieve the REAL goals of the business. We shouldn't document for the QA department, we should document for the customer.
     So three key questions:
     
    1 - Why are you documenting?
    2 - What do you really have to document?
    3 - Who will use it?
     
    With a related question - documented is not the same as document - do you really need a document?
  • KeithCollyer
    KeithCollyer
    14 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-12-06T11:57:32Z  
     So three key questions:
     
    1 - Why are you documenting?
    2 - What do you really have to document?
    3 - Who will use it?
     
    With a related question - documented is not the same as document - do you really need a document?
     I want a "Like" button!
  • GustavoGrillo
    GustavoGrillo
    2 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-12-06T11:59:16Z  
     So three key questions:
     
    1 - Why are you documenting?
    2 - What do you really have to document?
    3 - Who will use it?
     
    With a related question - documented is not the same as document - do you really need a document?
     Exactly, these three questions could save a lot of time, money and headaches. These questions could be broken into something like this:
    1-To which standards the documentation must really conform? Which legal issues do apply? What will audits request?
    2-User requirements? Technical architecture? High-level stakeholder's needs?
    3-End-users? Auditors? Developers? Sponsors?
    These are, of course, just a few examples.
  • KeithCollyer
    KeithCollyer
    14 Posts

    Re: Virtual Round Table - Agile systems engineering - Asset, governance, documentation and contractual issues breakout thread

    ‏2012-12-06T15:43:40Z  
    I think the problem here is to know what do you need documentation for?
    Software requirements aren't always the best kind of documentation, but these words are often used interchangeably.
    I usually ask a lot of "whys" to governance needs because sometimes people are trying to fulfill  old interpretations of the needs and not the needs themselves.
    I don't know if improving documentation reduces costs, it can easily increase them if you document for the sake of displaying  beautiful documents on the the company management retreat. You need to know what you mean by 'improving': more detail? faster, yet not so detailed? aligned with the corporate architecture?
    This is where agile meets lean: you gotta do the least work (no matter if it is coding or documentation) to achieve the REAL goals of the business. We shouldn't document for the QA department, we should document for the customer.
     We document for whoever would be the stakeholders for the documentation. Yes, the customer is one of these. The QA department? Nah, not so much - unless you are doing something that has to conform to standards, when their input can be invaluable. And that's the thing, really, Vicky's three questions, and you supplemental, are really good. We should not document for the sake of documenting, we document because to do so is meaningful ("document like you mean it!" could be a slogan). And, as Vicky points out (and I have said myself many times in the past), documented is not the same as in a document. The second key point about documenting something (after it being meaningful) is that it should be accessible. In many cases, that means a document, but not always. I have blogged on documenting here (for IBMers, it is also here)