Reintroducing "engineering" to software.

I've been noticing an interesting "convergence" going on in this corner of the universe:  "engineering" is being re-introduced to the idea of software.  It's fascinating how no sooner do I have the idea to write this blog (while entirely not connected to the Internet), then I re-connected to get an article that's entirely speaking of the same root issue.

Of course, I'm not saying that engineering had completely left the software universe, though, a strong argument can be made that for the last decade and then some, software has allowed engineering to escape from the premises.  In particular, architecture, analysis, systems thinking, design, hardware and other integration issues, and planned, deliberate, methodical testing have largely been allowed to merely "emerge" from the work that was completed.

Again, I'm not advocating for Big ____ Up-Front as a solution, what I'm pointing out is that many people who embrace agile methods incorporate engineering practices into how they organize and perform their work, but enough don't that it raises issues with agile scalability.

And here's where I *am* expecting to annoy some people...

Programming and Development are NOT the same.

Development is an engineering function.  Developers ought to be using engineering practices in what they do.  Just look at the word "development".  The connotation is that something is "grown" or "evolved".  The denotation of 'development' in the technical sense is that it is done deliberately, not by happenstance.

This idea is where I believe software, in general, not limited to agile practices have short-changed themselves.  Too often, activities that amount to nothing more than programming are called development when no actual engineering is happening.  In other words, programming is allowed to take place without any (or at best without enough) engineering, and therefore what's really happening is the building of something without any/enough forethought about the thing itself that is to be built.  Instead, what happens too often is all the focus is on "staying busy" (albeit on ostensibly priority work), but what is worked on is absent sufficient technical rhyme or reason.

Is this true of all agile development?  NO.  But, it is what happens in many organizations when they don't have sufficient technical leadership.  For what it's worth, many development projects don't need much engineering, and product development is sufficiently described in tasks defined by few people.  So the jump from development to programming is small and fast. 

However, there are projects (or tasks) of sufficient technical complexity that skipping the engineering and handling such projects/tasks as programming alone is where I believe a space is created for the unfair reputation for agile and its scalability, as well as some of the anti-process bias among agile proponents.  When I read (and sometimes contribute) to agile and non-agile software groups, I'm often struck by the same thought: where's this person coming from?  This is basic engineering!

But that's the matter, isn't it?  Programming isn't development without engineering and too many programmers aren't engineers (not should they be) but are being told to "develop" without given the time or resources or something to do the engineering.  And so what they're really being told to do is "program", not "develop".  Someone, somewhere doesn't see and/or understand that what many projects need are to be engineered.

I think what this points to is a persistent phenomenon plaguing software: it's not being taken seriously as an engineering discipline.  Sometimes by leadership in organizations where software is being worked on, sometimes by programmers and sometimes by customers.  I'm sure there's plenty of blame to spread around, and spreading the blame is both a waste of time and not the point at all.

Programming is to software as assembly is to construction.  Not everyone swinging the hammer needs to be the civil engineer nor the architect, and not everyone with a nail gun can be the foreman (and no, I'm not likening the skill set of programmers to those of construction workers, and no, I'm not saying construction workers aren't smart....geez).  There has to have been engineering taking place before software can be actually developed, and as evinced by the kinds of challenges I encounter regularly, enough software shops are going about their work absent acknowledgement or awareness or consideration for the engineering that has taken place or has yet to take place (or should have taken place but didn't[!]).

Process stuff generally finds its roots in engineering.  Especially process stuff as found in CMMI for Development.  Excepting processes that are over-engineered, are themselves lacking in engineering, or are odious even alone in a room, I'm beginning to piece together that resistance to processes in general, and CMMI in particular, is actually from a lack of engineering discipline in the software practice and not from anything intrinsic to process as a topic.

It's no wonder CMMI is so hard to use by so many, it assumes people are not only experts in process improvement, it also assumes everyone using it is an engineer.  Some people are nail-gun swingers, worried about getting enough done that day to avoid having to work on the weekend.  Meanwhile, someone else already worried about in what order to build piece the trusses together and someone before that worried about the right number of trusses and their thickness and someone before that worried about its shape.

It's becoming fairly clear that anyone fooling themselves into believing that agile advocates not doing an architecture at all, or a design at all, or other engineering activities at all are doing themselves a disservice.  In fact, I'd go so far as to say that once an architecture has been settled upon and once a design becomes clear, that agile practices can happen more freely and effectively.  More so, I'd assert that the future of agile "scalability" depends on these.

What I and a colleague are setting out to do over the next few months is help agile scale by re-introducing engineering to software, and while we can't fix the software universe, we hope to help agile out by giving it some engineering practices that software (as a whole) lacks -- not everywhere, just in too many corners -- but that we believe agile can really take and run with.

Let me know if you want to play with us.