A couple of weeks ago, I posed the following in my blog: what are the grand challenges of software and software engineering? What do we know we don't know?
I had about a dozen people rise to the question and a lively discussion among many of them unfolded via email. Following are some highlights of that correspondence.
Communication was a common theme. Tom Welsh asked "How do we bridge the communication gap between business (and government, etc.) people who want all the benefits of IT - NOW, cheaply, and without taking much trouble - and the technical experts who know (or can find out) how to build software that works?" Similarly, Arisio noted that "The greatest challenge of software development still is: COMMUNICATION. Communication between our users/clients and us, the developers; comunication between the developers in order to understand and implement and test the user's requirements."
A number of ex-Rationalites, existing IBMers, and others tossed about an issue that was first introduced by the venerable Joe (run, don't walk, to buy his book) Marasco who wrote that "it seems to me that after all these years we have not yet sorted out the 'people' versus 'process' conundrum in software development. Debate rages, curmudgeons opine, new processes spring forth, and we make very slow progress. True, our systems are more ambitious, and the bar is constantly raised, but our success rate lags that of other disciplines." Joe goes on to note three common responses to this conundrum: "1. Make every problem a 'small' problem, by devising systems in such a way that we compartmentalize the development of large systems into an array of smaller ones that are tractable by small teams. 2. Improve the quality of the people through better education, training, mentoring, team building, and other devices. Our emphasis here is to raise the level of the profession such that even the weaker players do not pose a threat to effective development. 3. Improve the quality of the processes we use. Here we accept human frailty and attempt to compensate for it by systematically following a discipline of checks and balances. These are not mutually exclusive, and some combination of all of them may be the right approach." He then asks "However, given that resources are always finite, where do we spend our money?" He then then notes that "It's not people vs. process. It's people and process. It's an investment trade-off decision, and we need better tools to make better decisions." Jim Archer picked up on Joe's comment about the quality of people (Joe noted that "The quality of the people is measured by knowledge, experience, work ethic, and the ability to function as a team.") and observed that "I'm afraid that this encapsulates the failure of most software hiring. Even IBM thinks you need these things, especially the last one [the ability to function as a team]. The thing that it does not capture is productivity." Jim went on to relay a story about one of Rational's first programmers who "was just out of graduate school (as were many of us), so experience wasn't that great an asset. He was nobody's idea of a team player, though his ability to largely ignore the team was more productive than some who insisted on destructive engagement. Oddly enough, his a priori knowledge was largely irrelevant. Howard learned what he needed to know when he saw the problem. He had a great work ethic, but so did a number of our least productive members. Howard had the ability to produce systems of code that did something useful and worked, and he produced them at very high speed. It is this ability to produce that software teams need. The secondary attributes are what we mostly hire for."
Dave Bernstein picked up a riff regarding the impact of focusing a development team on customer success and noted that "Focusing on customer success does not mean slavishly implementing whatever customers tell you they need, prioritizing customer requirements over architectural requirements, [or] optimizing for incremental revenue. To focus on customer success, an organization must continuously understand the context in which its customers operate, e.g. problem domain, business environment, organizational style, regulatory constraints, identify opportunities within this context to provide business-critical value, deliver business-critical value, [and] interpret and weigh feedback from customers, prospects, analysts, sales reps, investors, and other stakeholders. For this to be sustainable, the organization must maintain a flexible, well-designed architecture upon which solutions can be efficiently delivered. There is no simple set of rules for resolving the inevitable conflicts and tradeoffs: good judgment is continuously required. The role of 'focus on customer success' as an important component of of forging high performance teams stands in contrast to a less-desirable set of organizational success measures, e.g. span-of-control, featuers of code contributed, headcount, papers generated, personal retention, [and] title." Aahish Virkam reflected upon the successful projects that he had encountered and noted that they were "successful because they were written by a small set of productive developers that wrote code quickly and got the product to do what they intended the product to do in a short amount of time and with relatively less code-filling in the design as they went along. A lot of design in the hands of unproductive engineers only leads to a lot of bad code, enough bad code that the few good/productive programmers in the team are unable to work around the large volume and get things to work anyway. The best chance of success is for a project with only high level requirements and design with a small team of good people that can take the high level requirements/design and convert them to code."
Rich Hill noted that "The biggest paradigm shift in modern programming would have to be the ability to use visual representation of object in the development process. UML has given us a little taste into this process, but extending the modern basis of UML and actually being able to develop in a purely visual atmosphere could lead to astounding improvements in performance and could easily lead to new patterns that are easily recognizable."
Oskar Ojala offered that the next grand challenge is simply "How to keep significantly raising the level of abstraction in face of ever growing software complexity."
Finally, Bill Higgins asked the brilliant question "Will software developers ever learn their history so that they don't consistently repeat mistakes and bad practices documented 37 years ago?"
Thank you all for your comments (and the dialog continues).