Unscrum me, please 2

This is part 2. You might want to read part one, if you did not already.

When does Agile work?

Despite my misgivings, it looks like there are some really successful agile teams out there.

1. The team shares a common vision and a common goal to deliver a good product.
2. The team has sufficient standing with management and customer representatives to push through important technical requirements.
3. The system has a lot of interactive features, is hard to specify in advance and constant input from the customer representative is needed.
4. The system falls into a category the team is experienced with or overarching design problems are rare and the team is able to deal with them.
5. The team is sufficiently experienced to design on-the-fly and see possible long-term design problems up front (while still designing on the fly).
6. Everybody in the team realizes the need to communicate and the team self-organizes neccesary communication. 
7. Every team member is comfortable with the idea of constantly working on small scale task items (2-3 days) and with (semi-) publicly and constantly breaking down their (often daily) work schedule to the team.
8. The product owner and team take measures to ensure cleanup work is planned into the task list.
9. The product owner and team take measures to ensure the team is always slightly underworked and thus reduce the stress created by the constant need to release and integrate.

It's just a job

(1). Yeah. But no. Numbers:

52,3% percent of Americans are unhappy at work. And 41% of people have a job that pays the bills, but does not really interest them. In Germany, 64% of workers just (want to) do their job. 24% are regarded to have internally quit. The highly motivated team that has an interest in the product they are building is very rare.

Regardless of the quarter that would rather quit immediately, I am not willing to insinuate that many of these people are qualified and willing to work. And even if the highly motivated developers look down on career guys and girls, the latter are probably the ones that will stay sane, healthy and  less likely to burn out.

It's interesting that some consultants tell people, that job happiness often stems from thinking "It's only a job not my life". Slightly contradictory? Of course.

Still, with these numbers, the chances to have a fully motivated team are below 1% in a team of five ((1.0 - 0.64)^5). It's not that the other ones aren't good or do not do their job, but they are not motivated to give up so much of themselves to get something done they are not really that interested in.

There are many stories floating around, how agile magically made teams more motivated and productive, but I think that is just because the change seems to have removed the few (unknown) blockers that made people unhappy. I figure, they usually were motivated to get things done before.

So now, the goal would be to find those blockers and demotivators. But that is hard and the general idea of agile seems to be to shift the responsibility to the team and also blame the team (or the organization) when things go wrong. This is actually quite a common theme in modern capitalism: Move responsibility either sideways (socialize) or down. The result is usually that the responsibility rests with people without power to decide or that everybody feels helpless (market dictatorship).

You need  a way to integrate the career developers that have just a job and the highly motivated people. Add to this the qualification delta between team members, their differing personalities and work values and you might want to really add back some external management.

Some Scrum advocates basically say, the Scrum way or the high way. They classify even a person that is (only) sceptic about agile as a passive resistor. If you do not believe in the one true god, you are a heretic that needs to be excised from this community! If it is put like this, resistant to those religious zealots should actually be required.

End of the Food Chain

We have a developer shortage. It might be a good idea to think why that is. The industry mostly does not treat the developers well. It will not be getting any better, qualified people are rare, even if you might think you can import more (cheaper).

(3) Short term projects, too-tight budgets, small-scale work, violation of work and quality values, you name it. Many are working on multiple projects, many of them short to mid term. There's not investment of people into a product, because there's no real long-term product. "Do stuff y for company x" is not product building, it's an (outsourced) service job.

(8) and (9) The reality: Many projects are fixed-price. Everybody wants estimates, but nobody can give proper estimates so we push the job down. Contracts need to be negotiated and fixed. Agile process then often leaves room for constant change requests from the customer. The result is a death march, which delivers value to the customer, but kills (literally) developers.

Agile demands that this needs to change and it seems to work for some companies. For many, including many that I know of, it won't.

I would like to iterate again the main contributors to work-related stress according to the (hopefully by now well-known) RISP study:

• constant overcommitment
• loss of control
• conflicting requirements and value

In waterfall projects, after the final scramble for a release, there is a phase of relative slowness. Agile removes that and replaces it with a carefully managed but continuous pace. It is assumed that the team is responsible for making that pace sustainable.

I can find exactly two studies that compares stress levels in agile and non-agile environments. One compares a single waterfall to a single Scrum project and finds that  Scrum developers are less stressed. The second is a survey with very mixed results. In no way is switching to Agile or Scrum in particular the silver bullet. Of course, the agile zealots will want to dissect what the projects without the productivity gain did wrong.

On the other hand, agile orientation seems to correlate very strongly with the satisfaction of certain psychological needs, most importantly co-worker and supervisor support. The need for autonomy, whose lack is --I repeat my self- a direct stressor is negatively influencing the agile orientation of a developer. Agile only works when you have a tight-knit team that supports each other, is supported by management  and where each team member is able to reduce or limit their own self-efficacy.

Interestingly enough, this also means that agile can go horribly wrong: If a team member has a high self-efficacy (good people usually do), they do not like to rely on the team so much (a tendency, but it's still there). This pushes their psychological needs, which reduces their agility. And now we are in the next section ...

MacDonaldization

Once upon a time there was still the idea floating around that (relatively) unqualified developers could do the dirty work when given a sufficiently though-out specification. That did not work out so well, did it?

Then we had the Brooks team, where you had one very good developer (the  surgeon) who does most of the critical work, while others support him and work at the periphery. Guess what? That's actually the mode many experienced developers like to work: They want to be responsible, they want to make choices, they want to create a product. If the problem gets to big, they try to select chunks to give to other people.

Unfortunately, there are two problems: The surgeon becomes the single point of failure and many modern software is so complex, that a single surgeon cannot handle it all. So you start with the idea of splitting the project submodules so that multiple teams can work on a project and then integrate their results.

Now you've got the risk of silos, where expertise is built up and not shared.

So you break down the silos and build "cross functional" teams and try to enforce information sharing. A nice (or not so nice) way of doing this is continuous integration. Continuous integration basically means "you need to get your stuff together with everything else very frequently".

Now, we make the increments as small as possible. Some people mandate integration multiple times a day, but it should happen at least daily. Guess what you align your task size with? Exactly: the integration increments.

So I now work in at most half-day sizes chunks, because I need the other half-day to communicate and to fix the integration problems (I'm exaggerating).  I am not allowed to sit and think on the overall design during a sprint, because we had the design meeting up front, detailing out every task. So I write tests up from for code that I do not even know if it will fit into the design. When I need a change later, I now have twice as much code to refactor (actual code plus tests) and I have to do it more frequently. All this under the constant pressure of not having to say "I did not get a lot further yesterday. No, I'm not blocked, the task just requires more thinking to get done right". I frequently try out mini-architectures to see if they fit the overall system. For some people, I cannot do that, because I have to deliver value in much smaller chunks. And there's the assumption: Everybody that is not working on small chunks, doing daily mainline integration and is constantly reporting his progress to the team in as small chunks as possible, is a bad person. He's not smart, he's a bad team player.

Any smart person that takes pride in their own achievement and values their privacy will resist constant observation. I would even go so far as so claim that this is a fundamental violation of their rights as an individual. Agile practices can be violating if enforced. And since the agile requires full commitment, the zealots try to missionize the resistor, making things worse. Some Scrum advocates even claim, that it's the Scrum way or the high way. They classify even a person that is (only) sceptic about agile as a passive resistor. If you do not believe in the one true god, you are a heretic that needs to be excised from this community!

Let's give back to the agile zealots, shall we? Because they called us siloed, pseudo-supercoder teamkillers that need to be excised from the team. And yes, they do. Not in as many words, but they still tell us to fuck off.

I think many over-zealous agile proponents seem to have been heavily burned at some point by some him-/herself over-zealous, overly-protective, non-communicative pseudo-supercoder, that wasn't all that great in the end. And now they project that trauma onto every project they are working on.

If you are not a very heavily integrated team worker, you are non-functional. You must cooperate, communicate, multitask (Not that all of these are on the burnout risk list). For some people, agile removes personal comfort zones without leaving them a way out. And it pushes the blame into them.

Agile zealots seem to have a sworn enemy: The highly skilled problem solver that sees communication as a necessary chore not as a his/her default operating mode. In this world, structuring a project so that multiple, highly skilled, independent people can work on it together without constantly looking over each others shoulders is frowned upon, it is deemed impossible. To them, these introverts seem to be disfunct, non-teamplayers and despicable.

There, I wrote it. Scrum is about hating some people's personalities. It's Processes over People.

Hell, let's make it even more provocative: Agile is about monochronic construction-line developers and managers rolling out their personal version of "works for them" discipline. Smart, but more introvert people that like to work on larger tasks in private are their enemy. They hat them, think them disfunction and broken.

People over process my ass. Have a bad day? The team needs to know, because your tasks are only  ever a few hours long! Why aren't there any more green tests on your list? Why did you do nothing today? Why is your task not finished? Agile frowns upon asking that question, but you already need a tight-knit team for those questions not to come up. And that team did not need agile in the first place.

Agile coach's solution is to kill the "resistant" team member off.

Just fuck the fact that many good people actually like to work on something in larger chunks and they can handle it. Problem is, of course, the system is not built this way. While tasks are reduced to small stories, proper modularization is not encouraged. You split of stuff so that the unit tests run, but cross cutting concerns seem to be never addressed. *sigh* You put (significantly) more stress on some people and call that method developer friendly. And the only answer to have for those that cannot cope is to offend them and tell them they are (mentally) broken.

People that I would like to call good agile proponents are more moderate. They think that this amount of aggressive teamwork is a skill that can be learned and that some people only take longer to learn that do others. But they also offer no solutions to good engineers that simply cannot do it. Unfortunately thus, agile puts some smart people into a corner without considering if that is not even more wasteful that any alternative.
 

The premise --in my opionion-- that agile makes up for is just often not realistic. And I strongly refuse to be a construction line developer. And I am not alone, that reduction to a mere code machine is on the top list of work related stress generators in IT.

And mind you: Without smart, independent thinkers that are able to do (some) things on their own, you create a mediocre product. Most of the team might feel good, but the end product is mediocre in the eyes of an experienced person. And when the "introverts" tell you, that it could have been so much better, you dismiss them as arrogant and elitist.

Maybe it really is about being able to reduce one's work values. That conflict also on the burnout generator list.

Hybrid might be a good idea

I think that you still need to fit both worlds: I know that I have spent months at the start of projects to get a broad design overview for a project. You really need the wireframes to get a concept and a broad idea of the commitment you can make, i.e. if the project seems actually doable in the suggested (and paid) timeframe. Thus

• Broadly plan ahead. You need to know where to stop, though. Do not overplan, but get a broad sense of dependencies. This also helps you to get decoupled modules, which --in turn-- you can use to keep your big thinkers and big tinkers happy. And: You know what experience also means? Experienced people on your team can spot potential pitfals early. Make use of them.
  
  You only need a broad plan on the lines of "we want to be at x by date y". The plan may change (significantly) over the project's process, but a broad roadmap is really a good idea.
  
• Always undercommit. If a planned work period (please stop calling these things sprints already) ends just on time, you put too much into it. You always need room for little tunings and cleanup. It is second nature for experienced developers to do cleanups on the fly (HEADS UP commits should be rare and communicated well, though). It is also important for the psychological needs that people have the feeling of getting done more that they should. Psychology is more important than you think.
  
• Leave room for detours and longer tasks. This also fits in line with the need to broadly plan ahead. Experienced people will know that they need to implement a certain feature or try out if something works. These side paths are very important to develop a good system. "I found a good way to solve the problem x as a basis for feature y" is very, very satisfying. People that are only working on the short term task queue are discouraged from these detours, because they do not have the feeling that they can catch up on their workload (by social pressure, see part 1). Don't make your people into idiots.

Properly executed Kanban seems to get it right. Just saying ...