Tuesday, April 08, 2014

Software Developmers: prototype of future knowledge workers?

‘‘Knowledge work is not easily defined in quantitative terms, . . . To make knowledge work productive will be the great management task of this century, just as to make manual work productive was the great management task of the last century.’’ Peter Drucker (1969)

In recent months I have repeatedly been reminded of an argument, or perhaps a hypothesis, I advanced in my 2008 book Changing Software Development. In the introduction chapter I suggested that software developers might be regarded as the prototype of future knowledge workers. I’ve republished “Prototype of Future Knowledge Workers” as a blog post yesterday.

I’ve been reminded of this in a number of casual conversations recently, like the one I had with Laurie Young (@wildfalcon), again when rereading some Peter Drucker and because of media pieces like one in the Financial Times (“Do it like a software developer” by Lisa Pollack) suggesting that companies with successful software development teams might have something to teach other companies, and also by the topics submitted to the Agile on the Beach conference.

Let me explain, firstly let me explain why I make this argument, then let me explain the significance - and with reference to Agile - and finally, for completeness let me quote from the book…

Software Developers - programmers, testers, and everyone else - are what academics would call knowledge workers. We work with our brains rather than our brawn. We generate value because of what we know and what that knowledge allows us to do.

There are lots of other knowledge workers, teachers, lawyers, doctors, and during the twentieth century the number of knowledge work professions and number of knowledge workers grew.

In many ways software developers are no different, but in one very important ways they are very different. Software developers are able to create new tools for machines which allow them to change the way they do knowledge work.

Think about some of the tools your common knowledge worker uses during the typical day:

  • E-mail: invented by Ray Tomlinson, a programmer
  • Spreadsheets: invented by programmers Dan Bricklin and Bob Frankston
  • The Web: invented by programmer Tim Berners-Lee
  • Wiki sites: invented by programmer Ward Cunningham

Even if Twitter, FaceBook, Video conferencing, Voice over IP and countless other knowledge sharing and collaboration technologies were not invented by programmers I will bet good money on one common factor: software developers, programmers, had early, if not first access to these technologies. That is very very significant.

Lets be clear:

  • Unlike other knowledge workers software developers have the ability to create new tools which can change their work.
  • Software developers have early, if not first, access to tools which benefit knowledge work.

One might also add that people working with technology welcome and embrace new tools and technologies (on the whole) so the barriers for technology change are lower.

This means those working in software technology - and particularly those developing software - have more experience and most history of using these tools, and consequently are the likely to be the most adapted to working with these tools.

Thus if you want to see how new technologies, new tools, will change knowledge work look at those who create the tools and those closest to them. This shouldn’t be a surprise, after all knowledge work is about managing and using information, and these technologists create the tools we use to store, retrieve, manage, manipulate, combine, and share information.

(And there is a difference between knowledge and information but I’m not going into that here, its in the book.)

By itself this argument is interesting and perhaps good for developer self-esteem. We could leave it there but…. it also begs a question:

If software developers are the prototype of future knowledge workers what does it suggest will happen next? Specifically, how can this insight help answer the challenge posed by Peter Drucker above?

As a step towards answering those two questions let us rephrase the question:

  • What are software developers doing today which might benefit other workers?
  • Or to put it in Drucker’s language: what can managers of non-software staff learn from software teams which will make their teams more productive?

One way would be to look at the tools software developers are using today which might benefit other groups.

Version control springs to mind but version control has existed for at least 30 years and hasn’t caught on with the masses. We could continue to examine the toolset, one of my favourites is Markdown. Or we could look at the way software developers are managed. This was the thrust of the FT piece which looked at personnel management (I refuse to say HR) practices.

But there is something else… Agile itself.

One of the reasons I believe Agile came about was that the tools we use changes. Cobol gave way to Java which gave way to Ruby. OS/360 gave way to VMS which gave way to Windows (or was it Unix?) which gave way to Unix derivatives. One might say paper gave way to the web and wiki’s, telephones gave way to VOIP, telegraphs gave way to Twitter.

Our tools changed which allowed - even forced? - our processes to change. As knowledge workers outside of IT embrace these these tools its possible that their working practices will change in a similar way. My favourite example is a presentation given by Kate Sullivan at Agile on the Beach two years ago in which she described how the legal department at Lonely Planet adopted Agile.

There are several Agile practices which area are easier to imagine outside of software than others. For example:

  • Visual boards: lots of places!
  • Standup meetings: example abound and usually predate Agile, e.g. bar staff, Japanese office workers, NATO commanders (in Bosnia and probably elsewhere)
  • Test driven working: consider many of the business examples from Lean Start-Up
  • Pair programming: airline pilots and surgeons have been doing this for a long time but Kate added Pair Lawyering
  • Iterations: although I’ve yet to get a concrete example most news organizations clearly operate in daily iterations

During the second half of last year I was involved with a client whose project was to write documentation. Specifications to be precision but not specifications for software - there was an element of that but that was not the aim. We used iterations, visual board and other techniques. We even used a walking skeleton. It worked. We delivered to schedule.

We in software have developed something good. Something that is spreading beyond software. Now I wonder, whats next?

Monday, April 07, 2014

The Prototype of Future Knowledge Workers

The following is an except from my 2008 book “Changing Software Development: Learning to be Agile”. I’ve been thinking about this suggestion a lot recently and have a blog post in the works. Hence I thought now would be a good time to share this, it also means I can reference this post in the one that comes next… and who know, I might even rustle up a few sales for the book!

The Prototype of Future Knowledge Workers

Highlighting IT workers as knowledge workers allows us to learn from the existing body of knowledge on the subject. IT workers are not alone; they are knowledge workers and there’s much to learn from other knowledge workers, and from research and literature about knowledge work in general. There’s no need for IT managers (and writers) to re-invent the wheel.

Yet, in another way, the existing literature, research and experience can’t help IT workers and their managers. This is because IT workers, and software developers in particular, are at the cutting edge of knowledge work. In many ways, they’re the prototype of the future knowledge worker; they’re pushing the boundaries of twenty-first century knowledge work.

This occurs because, to paraphrase Karl Marx, software developers control the means of production. Modern knowledge work is enabled by and dependent on information technology: e-mail for communication, web sites for distribution, databases for storage, word processors for writing reports, spreadsheets for analysis – the list is endless! These technologies are created by software developers and used by legions of knowledge workers worldwide. The key difference between software knowledge workers and the others is that other knowledge workers can only use the tools that exist. If a tool doesn’t exist, they can’t use it. Conversely, software developers have the means to create any tool they can imagine.

Consequently, it was a programmer, Ward Cunningham, who invented the Wiki. Programmers Dan Bricklin and Bob Frankston invented the electronic spreadsheet. Even earlier, it was another programmer, Ray Tomlinson, who invented inter-machine e-mail. This doesn’t mean that non-programmers can’t invent electronic tools. Others can invent tools, but for programmers the barriers between imagining a tool and creating the tool are far lower.

Lower barriers mean that programmers create many more tools than other types of worker. Some tools fail, while others are very specific to a specific problem, organization or task in hand, but when tools do work it is programmers who get to use them first. In addition, because IT people have had Internet access for far longer than any other group, the propensity to use it to find tools and share new tools is far greater. So tools such as Cunningham’s Wiki were in common use by software developers years before they were used by other knowledge workers.

Early Internet access has had other effects too: IT workers were early adopters of remote working, either as individual home workers or as members of remote development teams; IT people are far more likely to turn to the Web for assistance with problems and more likely to find it, because IT information has been stored on the Web since the very beginning.

The net effect of these factors and others means that software developers are often the first to adopt new tools and techniques in their knowledge work. They’re also the first to find problems with such tools and techniques. Consequently, these workers are at the cutting edge of twenty-first century knowledge work; they are the prototype for other knowledge workers. Other knowledge workers, and their managers, can learn from the way in which IT people work today, provided that we recognize these workers as knowledge workers.

From “Changing Software Development: Learning to be Agile” by Allan Kelly, 2008.

Thursday, March 27, 2014

Notes to students doing research about Agile

The dissertation/thesis writing season is approaching so I expect the recent set of questions from a student is the first of many. Actually they occur all year round but March to September is the busy period. So, to pre-empt any students bearing questions about Agile - and so I can just repeat my stock answers - here are some ideas you might want to consider first.

Lets start with the request itself.

I and many other people who are silly enough to write and blog about Agile software development get plenty of e-mail from students doing research on Agile. Fair enough, I like to help but if you don’t ask nicely I’m unlikely to respond.

  • I won’t be answering stock surveys that are sent from a mail program.
  • I’m much more likely to answer a personal request which indicates you actually know my name and even read some of what I have written.

I’m skeptical of most quantitive research (e.g. surveys) research on Agile simply because the question that I normally see are rubbish. Qualitative research is probably better but

  1. It is not as satisfying as quantitive, you can’t conclude “58% of teams use Agile” you come up with some wishy-washy case studies
  2. You actually have to go and talk to people, you can’t just compile a mailing list and feed it into Survey Monkey. That means getting out and finding people to talk to

If you must go down the quantitative route please do your homework, do some qualitative before you begin so you can ask good questions.

Now the number one mistake made by students who approach me is: Assuming there is such a thing as Agile.

That is to say assuming:

  • one is either Agile or not Agile,
  • secondly: one knows that one is, or is not Agile and
  • thirdly: one is prepared to openly and honestly state it.

Agile is not binary. Nor for that matter is Waterfall. While we are about it Agile and Waterfall are not necessarily binary alternatives. Have a look at my Agile Spectrum article: the thing that is called “Agile”, and the thing that is called “Waterfall” are simply points on a spectrum - probably the two extremes - and most teams are somewhere in-between.

Nor is there anyway (at the moment) to objectively determine where a team is on the spectrum. Well actually there are plenty of people who will do an assessment of one sort or another but you have to look at what criteria they are using. My suggestion is get away from Agile as a whole and look at Agile practices.

Likewise get away from the methods - Scrum, XP, Kanban, etc. If you must look at different methods look at them in the context of practices. Agile methods are sets of practices, sometimes with values and principles thrown in.

Practices are the easiest of these things to observe and measure. They are objective, the other bits are subjective.

If you really want to dig deep - and make work for yourselves - you might want to investigate team values - do teams actually value the things Agile claims to value? This is going to take some doing. You probably want to look at what actually happens, shadow a manager for a week and see if their decisions accord with the values.

You might do it quantitatively by composing one of those surveys that asks people to choose between answers, e.g. “Under pressure to ship more often which are you most likely to do: a) make your stories smaller, b) offer financial incentives to the team, c) reduce unit testing, d) increase unit testing”.

Now some suggestions for some research, Agile research questions:

  • Validate my spectrum: its a hypothesis, come up with some criteria and do a survey to see if you can determine the spread
  • Practices: what are the practices that teams actually use? Specifically, what are the practices they say they use and which ones do they actually use?
  • Which practices are the most popular?
  • What difference do the practices make?
  • How does Agile differ between product development companies and corporate IT? Add “solution providers” (e.g. Accenture, EDS, CapGemini and thousands of smaller companies) for extra marks.
  • Correlation of practices to approach: do teams which call themselves Agile actually use practices associated with Agile? And likewise for Waterfall. A couple of years ago a student at Loughborough University, Adam Williams, rose to my challenge. He tried this, his survey was small, 20-odd companies, and he found little correlation at all. In fact he found some teams who described themselves as Waterfall but used several Agile practices. I would love to see this study expanded.
  • What do Scrum Masters really do and how are they different from Project Managers?
  • What are the recruitment practices of Agile teams and companies?
  • Forget Agile, look at Waterfall, find teams who actively claim to do Waterfall and find out what happens, how do the unsuccessful ones differ from the successful? Are there any successful waterfall teams?
  • Examine some of the more controversial practices: TDD, pair programming, refactoring. Do a meta-analysis of the studies to date or do your own studies.
  • Extend Keith Briathwaite’s work the correlation between TDD and cyclomatic complexity.
  • TDD, benefits and the power law - I should blog about this but for now just take my word for it.

Finally the big one.

Real Agility is not about methods, practices or names, it is the state of being Agile. (Something that was meant to be in the title of my first book but was messed up.)

  • What is the state of Agile? - in theory and in reality
  • What advantage does the state of Agile confer?

So if any student out there wants to raise to the challenge on these questions please get in contact, I’d love to help and to see the final research.

Wednesday, March 19, 2014

#NoProjects becomes #BeyondProjects and online

On Monday night I delivered a talk at Skills Matter in London entitled “Beyond Projects: or the end of projects and what to do about it.” This was the latest evolution of the #NoProjects meme being advanced by Steve Smith, Joshua Arnold and myself.

Steve, Joshua and I have been thinking that #NoProjects is a bit negative and something like #BeyondProjects would be more positive but we didn’t reach a final decision. Having read some of the Tweets after Monday I’ve decided.

#NoProjects is now #BeyondProjects - that also sidesteps the problem of teenagers on Twitter using the first of those hashtags to denote the end of their coursework.

The talk was well received and I’ve got mails and conversations to follow up.

The #BeyondProjects slides are now online at Slideshare and Skills Matter have published a video of the talk.

If you want to know more about this see my blog last year "Beyond Projects, Beyond #NoProjects” see too Steve’s past blogs and Joshua’s blogs, particularly “The Problem with Projects.” What is really interesting is Steve, Joshua and I all started rethinking projects from different points of view.

My thinking originated with the cognitive dissonance that exists because successful software doesn’t end but projects do. I think projects don’t make a good model.

Steve comes from a continuous delivery point of view.

And Joshua emphasises risk and value more.

Gradually these three points are being woven into one story.


Monday, March 03, 2014

Xanpan update

As regular readers will know I have been working on a new book, or rather a new ebook called Xanpan. Indeed, since I am using the LeanPub system some people are now reading the book.

The content in this volume stabilised a few months ago but as always my English is a little eratic. The book needed a professional copy edit - indeed even better writers than I have their books professionally copy edited. This was done a few weeks ago but I’ve only now been able to complete all the other changes.

So today I am proud to announce a new version of Xanpan.

I would like to offer readers of this blog a buy the book at a discount, half the recommended price. If you follow this link to Xanpan on the LeanPub site or enter the code “BlogMarch2014” when buying the book the price will be reduced. These discounts should be good until the end of next month.

(I’m sorry, LeanPub works in US dollars so although most readers of this blog are in places which are not the USA, particularly the UK, I have no choice.)

Apart from the copy edit the most significant change is I’ve changed the subtitle from “Personal reflections on agile software development” to “Team centric software development.”

While this book still contains my reflections - and sense making - on the development process, and particularly agile, the feedback I had from several people suggested that Xanpan differs from other methods by its emphasis on the team.

I expect the team centric aspect to be more noticeable in a second volume I plan to write “Xanpan: Management heuristics”. And I have a thought for a third volume “Xanpan: Requirements” but not for a while yet.

The next task is to make this version of Xanpan available as a hard copy. That should happen in the next few weeks.

In the meantime, if you have any comments or feedback please let me know.

Sunday, February 23, 2014

Managers who don't TDD be unemployable before 2022.

In the furious discussion over my “Programmers without TDD will be unemployable” last month a couple of people asked: “How will this come to pass?” And a few others said: “Managers will never let this happen.” Let me answer both these comments in one go:
Managers are the missing link between today’s largely TDD-free code development and the 2022 world where TDD will be essential to get a job.
I know not few managers are perfect. And I accept that there are managers out there who might not want their developers doing Test Driven Development. But in my experience you don’t need to be a perfect manager to understand the logic of TDD, a quick look at the Microsoft research study should address that.

And in my experience it is developers rather than managers are the biggest block to practising TDD effectively. I can usually persuade a manager that their developers should do TDD. I can even (sometimes) - putting on my salesman hat - persuade them to spend money on training and/or coaching for developers to learn TDD.

What I find a much greater challenge is persuading developers that TDD might be beneficial to them. That TDD might help them find bugs quicker, or that their legacy code would benefit, or that it will help them separate business and UI logic. Just persuading programmers to code in a different way is hard.

And it is even harder to persuade them that having some help, going on a course or having some coaching, is a faster route to proficiency than reading a book and watching some screencasts.

I know TDD is not a panacea but I do think it is the best we’ve got right now. I know it doesn’t cure all known ills. And I know that in some cases the wrong solution, I know that other techniques - code reviews and functional languages for example - can also reduce bugs but…

Firstly too many developers question TDD without properly trying it, and they propose alternative remedies which also fail to get tried.

Secondly, as I said: managers are not perfect. You don’t need to be a perfect manager (indeed few, if any, are).

I predict that once Development Managers get the TDD-bug they will over do it. TDD will be mandated even when it is not applicable, or when another solution might be better. For a while TDD will be a management fad.

That also mean that the style and quality of TDD will be massively variable. Like Agile before it will be a buzz-word for the developer CV (resume) and more importantly, for the Manager CV.

In the same way that Managers will not employ Programmers who don’t know how to TDD - and are not enthusiastic about it - Managers will not get employed if they don’t expect their programmers to do TDD.
Managers who think TDD is a waste of time will be unemployable before 2022.
Even managers have managers. Even managers need to get new jobs from time to time.

I often meet programmers who tell me “I would like to do TDD but our managers won’t let us” but I have never met a manager who has ever said: “My programmers want to do TDD but I have forbidden it.” Managers, particularly the better ones, are open to hearing better ways.

Look at the blog comments after the “Programmers without TDD” post - on my blog, on DZone, JavaCodeGeeks and the other sites where it was syndicated. There are several programmers saying “Managers won’t let it happen” or “It doesn’t work” but I don’t recall even one manager saying “I’ve stopped my developers from doing TDD.”

Whenever a programmer tells me “I would like to do TDD but my manager won’t let me” I ask: “Have you actually asked the manager concerned? Have you discussed it?” invariably the answer is “No.” Programmers assume their managers’ will not like it.

The quality movement wasn’t always popular, people questioned Philip Crosby’s Quality is Free logic, it took time for the chip industry to change (The Anderson Bombshell) and it took over 20 years for Ford, GM, etc. to match Japanese (Toyota) standards but today you would not get very far if you suggested the Austin Allegro was a good car.

There is another force at work here: companies which don’t take quality seriously, or can’t change fast enough, will lose out to those who do. In the early 1970s British Leyland, later Austin Rover was the third biggest car producer in the world. The bits of the company that embraced quality - Mini, Jaguar and Land Rover - still exist, the rest doesn’t.

The manager connection also explains another question that was posed following my first blog. Several people said: “I can see how software companies get this, but inside corporate IT they will never buy this.”

I understand where these people are coming from, software companies will be the first to mandate TDD, most corporate IT departments will be slow on the uptake but they will come round because:
  • Corporate IT has long copied software firms but after a long delay. This pattern will play out with TDD too.
  • While some people work their entire careers in corporate IT there are also a lot of people who worked in software companies earlier in their career (such businesses often pay less than corporate IT.) By 2022 the people arriving from software companies into corporate IT will expect to do TDD. And the managers arriving in corporate IT will expect their programmers to do it too.
  • Consultancies like Accenture, IBM Global Services, Tata etc. who do outsourced work will pick up TDD from software companies (they see themselves as software companies). When people move from these companies to corporate IT groups they will take TDD with them, when software passes from the outsourcer to the client it will come with tests, and when corporate IT looks for best practice (O, how they love best practice!) they will find TDD.
  • Graduates from the better colleges and universities will also arrive expecting to do TDD.
If you followed the comments and Tweets surrounding the original post you will have seen several people say their company already mandated TDD for new recruits. Each one of those companies, to the best of my knowledge, has high performing teams which deliver software which contributes directly to company success. Why wouldn’t a manager want that?

Developers, programmers, may have started the TDD ball rolling but it is managers who will finish the job.

Sunday, February 16, 2014

Manual testing sinful?

One of the asides I made in “Programmers without TDD will be unemployable” which caused a bit of outrage in the testing community was my comment “Manual testing is a sin.” While I have been unfair to many testers, and somewhat simplistic, I still stand by the statement. Let me expand on why I still stand by the comment and why I am wrong.

It is all a question of context. Look at the full line the quote appeared in:
“Unit testing will be overwhelmingly automated. Manual testing is a sin. Manual unit testing doubly so.”

In the context of Unit Testing, as one who has undertaken formal and informal manual unit testing I believe unit testing can and should be fully automated. Informal unit testing is too ad hoc, prone to error, time consuming and difficult to replicate. Formal manual unit testing suffers all these problems and is hideously expensive.

In the context of unit testing I stand by “Manual testing is a sin.” I will also go further.

The vast majority of testing I see performed by professional testers - i.e. not unit testing - is manual, the vast majority of this testing could be automated. If you automate the testing the initial costs might go up - I say might because a good natural language test script is expensive to write itself - but the cost of executing the test will fall. More importantly the time it takes to run the test will fall dramatically.

The combination of low cost and fast turn execution means the tests become a very very effective feedback loop.

Failure to seize this amazing opportunity is something I consider sinful. It may be expensive but it is necessary. Even seizing 20% of this testing would be massively beneficial.

What I do not consider sinful is Exploratory Testing which almost by definition cannot be automated and therefore needs to be a manual process. Exploratory testing is about looking for the unusual, the unexpected, the thing you didn’t think of before, the thing you only think of because you now see it.

Automated exploratory testing might even be a contradiction in terms therefore manual exploratory testing is not sinful.

But, and this is a pretty big BUT: if the initial quality is low (e.g. unit testing is absent or poor) and the testing (e.g. system, integration, acceptance testing) that comes before exploratory testing is ineffective then exploratory testing will be effective at finding defects (because the earlier rounds didn’t find them) but it will be ineffective as exploratory testing.

The time and effort spent on "exploratory testing" is really being spent doing system testing. Real exploratory testing itself will not be possible when there are many defects because the exploration will continually be interrupted. 

Unfortunately the subtlety of this logic is lost on most organizations who just label everything that happens after writing actual production code “testing” and is made worse when testing is separated from the developers and, more importantly, those who understand what the system should do (the business, the BAs, the product managers, the users, etc.)

Worse still, the easy availability of cheap offshore testing capability means that rather than address the root cause of defects (low initial quality, system ineffective testing, misunderstanding of exploratory testing) means the whole thing can be hidden away cheaply.

There are probably some other examples of testing which cannot be automated and are not sinful. But I repeatedly find individuals and organizations who too ready to jump to “it cannot be automated” after very limited, or no, effort is spent on trying to automate it.

Similarly the fact that a professional tester cannot automate the testing of a piece of software system does not mean it cannot be automated. Automated testing - at least the setup - involves programming skills which many testers simply do not have, indeed it is wrong to assume they would have them.

Finally, there are a class of systems which as far as I know defy automated testing. Or rather defy automation at the moment. They defy automation because the creators of these systems have little or no interest in allowing automation. This set includes: SAP, Microsoft Sharepoint, most CRM systems such as Microsoft Dynamics, Murex (although I believe efforts are afoot here) and some other large packages.

I find it incredible that companies like SAP, Oracle and Microsoft can sell systems which cannot be effectively tested. But actually this is part of the sales-pitch. Because these systems are marketed and sold as “no programming required” to offer automated test tools would expose the lie. If you are involved with selecting one of these systems ask: how will we test it?

The inability to effectively test these systems is their Achilles heal, it is one of the reasons I don’t like these systems and it makes me wonder about whether organizations are right to use these systems.

Wednesday, February 12, 2014

Unemployable without TDD - a follow up

The reaction to my last entry - Programmers without TDD will be unemployable by 2022 - took me a little by surprised. I expected there would be a reaction, perhaps a big reaction, but I frequently expect some reaction and usually I get none. This time I got a reaction and the scale of the reaction continues to surprise me.

Particularly gratifying are all the comments on my blog, on DZone, JavaCodeGeeks, and probably some other sites that syndicate my blog but haven’t bothered to tell me. The fact that people were moved enough to write anything, supportive or not, means I caused people to think and join the discussion.

I’ve come back on many, certainly not all, of those comments directly but I won’t come back to every point. In this post I’d like to touch on a few of points which I think need expansion or clarification. Sorry if you think this is me being defensive. Stop reading if you think so, consider this a blog post for one reader only, me!

(This post, and a couple of others coming out of the TDD post does mean it will be a little while before I can get back to my software economics series.)

First, the many cynical comments (e.g. “This seems unlikely”) don’t surprise me. I frequently come across such cynicism in my work. Indeed, were such cynicism not widespread then I don’t think there would have been much interest in the prediction or any point in making it.

I am rarely sympathetic to arguments that it “won’t work for me” or “won’t work in my domain” because overwhelmingly these arguments are at best a (flawed?) logical argument, but they are usually made without trying it, they are conjecture, . When someone comes and says “We tried it... we really tried it... we invested in it... we stuck at it for a few months... and it made us worse” then I’ll accept it as so. Usually I find people who argue against TDD have never tried is or tried it superficially.

Second a point of clarification. I am happy to acknowledge that there are other ways of reducing a defect count. Code reviews spring to mind. There is a lot of research that shows that when done well they can be very very effective - and also justify the time spent.

Over dinner the other night Phil Nash also suggested highly typed functional languages - or was it functional languages and strong typing? I’m sure Phil is right in both points eitherway. Although if you ever saw the any of the Lisp/Scheme I wrote in college you would know bugs are entire possible in functional languages!

The point is: there are other techniques, probably some I don’t even know about. I single TDD out because a) I know about it and b) I think it has achieved critical mass and will be adopted more widely in future.

Because of alternative approached there will still exists jobs where TDD is not a prerequisite but they will be few and far between, in particular niches. There will be a few jobs you can get without past TDD experience but you will be expected to learn it. The same way you can occasionally get a job as a Java programmer even if you have never programmed Java. But if I was still developing, I won’t like to bet my career on finding such jobs.

Next, and most gratifying, the discussion on Twitter and in blog comments that followed the publication flushed out several companies who already operate a policy of only hiring developers who practice TDD. 7 Digital and Compare the Market stick in the memory because a) I’ve known them for a while and more importantly, b) they are hiring! Thanks guys, consider yourself plugged.

A few other people tweeted or commented to say they personally operated such a policy but didn’t give company names.

So let me float a hypothesis - based on the fact that I regard TDD as a better way of developing code than not doing it:

The divide between productivity and value-add of companies which do TDD style development and those which don’t is growing. The first group are generally among the best, most productive, highest value add, and the second group which don’t TDD are a long way behind. The first group is accelerating onto BDD, Continuous Delivery and other “new” ideas, while the second aren’t.
Sure there are companies which say they do this, and there are companies where application is not consistent - some people do and some people don’t. For such companies to join the high performing ones they need to do more and do it more consistently.

There are networks effects to doing TDD: the more people who do it in an organization, and the more consistently it is done then the greater the benefits. If one developer starts doing TDD their code will be better and everyone will benefit, but with each extra developer doing it the benefit grows greater per person - there are economies of scale here. And when everyone does it then the benefits are far more than the sum of the parts.

Yes this is conjecture, or if you prefer, call it intuition.

I don’t know what form it will take - I think it will be mixed up, some will be methodology, some will be more relaxed, and it will differ from place to place, team to team.

Next there was some discussion of how the transmission mechanism, how would TDD being good actually get it inside corporate IT departments? That is something I’ll come back to in another post, I have a view on that which might poor petrol on this particular fire.

People also threw stones at me on Twitter because my company sells TDD training. Lets get this straight: we offer such services because we believe in it, because I believe in it I wrote blog posts like I did.

It is insulting and extremely cynical to suggest that I write blog posts like I did purely to sell training. It would be nice if life was that simple, you would see more blog posts like that and I would be a lot better off.

Now I’m not saying you have to have training to learn TDD - although I do recommend it. I am also happy read Steve and Nats book, Growing Object Oriented Software - and yes, that is an Amazon associates link, is that a crime? I make about £10 a year from such links.

But ask yourself, if you were to buy such training who would you rather buy it from? Someone who sells it as another course or someone who passionately believes in it?

Before I finish, two other things about the reactions post surprised me. That so many people railed against my aside comment about IDEs. Of course IDEs will still exists and will still be useful. I just suggested that would be less because they were debugging environments and thus we might see changes in IDEs.

Also, very few people mentioned my comments about University courses. In many ways those comments were the driving force behind writing the post. TDD is a proven technique and Universities should be teaching it on programming courses, and they should be teaching code reviewing and probably some more techniques too.

Finally, as someone else pointed out my prediction is a safe bet. If by 2022 I am proved wrong nobody will remember my prediction. And if I’m proved right nobody will care when I say “told you so.” Besides, I expect people more famous than me will make similar predictions - both for and against - and they are far more likely to be remembered than me.

Tuesday, January 07, 2014

Programmers without TDD will be unemployable by 2022 (a prediction)

New year is traditionally the time of predictions, and several of the blogs I read have been engaging in predictions (e.g. Ian Sommerville “Software Engineerng looking forward 20 years.”). This is not a tradition I usually engage in myself but for once I’d like to make one. (I’ll get back to software economics next time, I need to make some conclusions.)

Actually, this is not a new prediction, it is a prediction I’ve been making verbally for a couple of years but I’ve never put it on the record so here goes:
By 2022 it will be not be possible to get a professional programming job if you do not practice TDD routinely.
I started making this prediction a couple of years ago when I said: “In ten years time”, sometimes when I’ve repeated the prediction I’ve stuck to 10-years, other times I’ve compensated and said 9-years or 8-years. I might be out slightly - if anything I think it will happen sooner rather than later, 2022 might be conservative.

By TDD I mean Test Driven Development - also called Test First (or Design Driven) Development. This might be Classic/Chicago-TDD, London-School-TDD or Dan North style Behaviour Driven Development. Broadly speaking the same skills and similar tools are involved although there are significant differences, i.e. if you don’t have the ability to do TDD you can’t do BDD, but there is more to BDD than to TDD.

The characteristics I am concerned with are:
  • Developer written automated unit test, e.g. if you write Java code you write unit tests in Java... or Ruby, or some other computer language
  • The automated unit tests are executed routinely, at least every day
This probably means refactoring, although as I’ve heard Jason Gorman point out: interest in refactoring training is far less than that in TDD training.

I’d like to think that TDD as standard - especially London School - also implies more delayed design decisions but I’m not sure this will follow through. In part that is because there is a cadre of “designers” (senior developers, older developers, often with the title “architect”) who are happy to talk, and possibly do, “design” but would not denigrate themselves to write code. Until we fix our career model big up front design is here to stay. (Another blog entry I must write one day...) 

I’m not making any predictions about the quality of the TDD undertaken. Like programming in general I expect the best will be truly excellent, while the bulk will be at best mediocre.

What I am claiming is:
  • It will not be acceptable to question TDD in an interview. It will be so accepted that anyone doesn’t know what TDD is, who can’t use TDD in an exercise or who claims “I don’t do TDD because its a waste of time” or “TDD is unproven” will not get the job. (I already know companies where this is the case, I expect it to be universal by 2022.)
  • Programmers will once again be expected to write unit tests for their work. (Before the home computer revolution I believe most professional programmers actually did this. My generation didn’t.)
  • Unit testing will be overwhelmingly automated. Manual testing is a sin. Manual unit testing doubly so.
And I believe, in general, software will be better (fewer bugs, more maintainable) as a result of these changes, and as a result programmer productivity will be generally higher (even if they write less code they will have fewer bugs to fix.)

Why do I feel confident in making this prediction?

 Exactly because of those last points: with any form of TDD in place the number of code bugs is reduced, maintainability is enhanced and productivity is increased. These are benefits both programmers and businesses want.

The timescale I suggest is purely intuition, this might happen before 2022 or it might happen after. I’m one of the worst people to ask because of my work I overwhelmingly see companies that don’t do this but would benefit from doing it - and if they listen to the advice they are paying me for they start doing it.

However I believe we are rapidly approaching “the tipping point”. Once TDD as standard reaches a certain critical mass it will become the norm, even those companies that don’t actively choose to do it will find that their programmers start doing it as simple professionalism.

A more interesting question to ask is: What does this mean? What are the implications?

Right now I think the industry is undergoing a major skills overhaul as all the programmers out there who don’t know how to do TDD learn how to do it. As TDD is a testable skill it is very easy to tell who has done it/can do it, and who just decided to “sex up” their CV/Resume. (This is unlike Agile in general where it is very difficult to tell who actually understand it and who has just read a book or two.)

In the next few years I think there will be plenty of work for those offering TDD training and coaching - I regularly get enquiries about C++ TDD, less so about other languages but TDD and TDD training is more widespread there. The work won’t dry up but it will change from being “Introduction to TDD” to “Improving TDD” and “Advanced TDD” style courses.

A bigger hit is going to be on Universities and other Colleges which claim to teach programming. Almost all the recent graduates I meet have not been taught TDD at all. If TDD has even been mentioned then they are ahead of the game. I do meet a few who have been taught to programme this way but they are few and far between.

Simply: if Colleges don’t teach TDD as part of programming courses their graduates aren’t going to employable, that will make the colleges less attractive to good students.

Unfortunately I also predict that it won’t be until colleges see their students can’t get jobs that colleges sit up and take notice.

If you are a potential student looking to study Computer Science/Software Engineering at College I recommend you ignore any college that does not teach programming with TDD. If you are a college looking to produce employable programmers from your IT course I recommend you embrace TDD as fast as possible - it will give you an advantage in recruiting students now, and give your students an advantage finding work.

(If you are a University or College that claims to run an “Agile” module then make sure teach TDD - yes, I’m thinking of one in particular, its kind of embarrassing, Ric.)

And if you are a University which still believes that your Computer Science students don’t really need to programme - because they are scientists, logisticians, mathematicians and shouldn’t be programming at all then make sure you write this in big red letters on your prospectus.

In business simply doing TDD, especially done well, will over time fix a lot of the day-to-day issues software companies and corporate IT have, the supply side will be improved. However unless companies address the supply side they won’t actually see much of this benefit, if anything things will get worse (read my software demand curve analysis or wait for the next posts on software economics.)

Finally, debuggers are going to be less important, good use of TDD removes most of the need for a debugger (thats where the time comes from), which means IDEs will be less important, which means the developers tool market is going to change.

(Postscript: sorry about the formatting problems with the first upload.)

Monday, December 23, 2013

Agile and the Demand Curve

In the last two posts I’ve taken two different views on what might happen to the Software Demand curve when we introduce Agile:

When we discussed how Agile effects the software supply curve things were pretty clear cut. Agile, in the long term boosts supply, and may even flatten the curve - making it more responsive to price changes, i.e. more elastic. Unfortunately things are not so clear cut with the demand curve.

Good news: There is a story Agile people - OK me! - sometimes tell - about what “should” happen with demand in an Agile environment.

Bad news: The is another story about demand in an Agile environment, one that Classicist (i.e. believer in “Waterfall” style up front requirements and planning) fear. And I’ve seen that too. It ain’t pretty.

But, the key point is: Few of the Agile tools work directly on the demand curve. After all, Agile is about software and the demand for software is derived from something else so why would they?

Let me say that again in other words because it is important: Nobody wants software for the sake of software, the demand for software is derived from some other demand, e.g. a retailer wants to sell online, that means they need software.

The current Agile tool set (e.g. User Stories, Acceptance Criteria, Specification by Example, etc.) which touches on demand - call it need, want, requirements, specification, whatever you want - does not address the underlying demand. These tools operate with the demand as presented.

Sure there can be feedback from the tools and Agile deliveries but feedback is often absent. Even when feedback is present those responsible for the underlying demand may not want to hear - or act - on the feedback. And in an environment where success is measured by criteria like “On Budget, On Schedule, On Features” acting on feedback may be impossible.

Which brings us back to the 3 Styles of Agile: Iterative, Incremental and Evolutionary (part 1) and 3 Styles of Agile part 2. (At some point in the future I will reprise those three styles in the light of this analysis.)

One of the key factors in determining whether the feedback can affect underlying demand is whether those responsible (e.g. senior managers) who largely originate the underlying demand are open to feedback. Which brings us to another story I need to tell, another time...

Where does this leave us then?

If the current Agile toolset cannot address the underlying demand we need new tools. We might choose to add those tools to the thing we call Agile or we might choose to put them in another category.

Personally I believe many of those tools do exist but they exist outside the Agile space. They are fellow travellers if you will. For example: Beyond Budgeting is clearly one of those tools. However, many of those tools have their own names, they are not Agile, they are what they are (e.g. Beyond Budgetting). They fit very nicely with Agile but they are not.

However that is my personal view. I also think that Agile is a marketing bang-wagon and those tools will be conscripted to the Agile course and in many cases put under the Agile umbrella.

Either way the problem we need to recognise is: addressing the software demand curve Agile is necessary but not sufficient by itself.