Montag, 28. September 2015

Sustainable Work Environment

Hopefully, the German myth of the "Fachkräftemangel", the fairy tale of the shortage of skilled workers, engineers and IT specialist has now been buried for good. And I really hope so, since I probably cannot bear another study with random factors (7.14) multiplied onto the real numbers because of gaps in the data.

Note that a very similar thing is happening in the US with exactly the same tricks and exactly the same untrue statements with the same faking of statistics and studies. Actually, no, it's supposed to be even worse in the US and some experts actually claimed it cannot happen as badly over here. Well, they seem to be quite wrong. It is happening. Maybe the situation is not as bad as overseas, but it does happen: Corporations lobbying for cheap skilled work and having their worker training paid for by the government or shifted as a burden onto the worker to do in their free time.

Business as usual: Capitalize results, socialize costs. Liar, liar, pants on fire. So, I claim, and I am not the first one: At least in IT, we do not have a skilled worker shortage.

We have a shortage on sustainable work environments.

There are problems in any work environment, today.

A 2008 study said
  • only 15% of workers are motivated and feel obliged to their company.
  • 64% only work-to-rule, which probably hurts themselves more than their company
  • 24% have quit internally and also do the absolute minimum required to do their jobs.
Jeff Atwood wrote an article about programmer types in 2007, stating that 80% of programmers are the 9-to-5, leave me alone with work after work type that do not really care about their craft. He is quite arrogant about it, but in my opinion he is missing a very important point: If you start out as an 20%er (motivated, truly caring about your job), most development jobs nowadays seem to be such that they make you into an 80%er. It might also simply be a defense strategy to keep sane. When you realize, nobody seems to care but you, you simply stop trying to push the boulder up the mountain.

And if I am to judge, environment in many (read the numbers!) companies are essentially setup to make you into an 80%er (before eventually dropping you): Whenever I open my ears, I almost hear the very same complaint almost from every acquaintance that is also working in IT:

Experienced people, familiar with the technologies that their company is using and/or offering are heavily overtasked. It's usually in the range from 10% (for the lucky ones) up to 40% overload and this overload goes on for weeks and months.

If you are fluent in German, read this. If not, I'll summarize:
  • Almost half of all IT works claim they have too little off time. More than 50% percent claim a lack of work-life-balance.
  • Almost the same percentage claim that their work is not organized in such a way as to avoid and prevent stress.
  • Almost 30% state that if work requirements stay as they are, they clearly won't make it till retirement. An additional 40% think it more likely they cannot sustain their pace until retirement.
Read the last point one more time. Carefully and remember: Results sum up to 100%, there's no overlap:
70% percent of all IT workers consider it more likely than not that they will not be able to do their current jobs until retirement age.
Well, I consider that quite a good definition of unsustainable.

Want more numbers?
If you look at German law, these might be considered illegal work environments.

The Rhein-Ruhr-Institute did a working paper in 2009, listing some of the major problems:
  • constant overtasking. To achieve set goals, workers need to constantly "voluntarily" work long hours.
  • new requirements and change requests are added without adjustment of deadlines and budget restrictions.
  • unplanned tasks are piled on top of planned work without plan adjustment (documentation, status reports).
  • lack of control: possibility for creativeness is limited by change  requests, strict financial restrictions and tight deadlines.
  • project workers are held accountable for results, but have unsufficient resources to achieve goals. There is a conflict between assigned tasks and available resources.
  • limited control over work environment. Selection of suitable tools and  methods is not under the control of the specialized worker.
  • project workers are held accountable for results, but cannot influence priorisation, schedule or work organisation.
  • lack of gratification. Performed work is not appreciated and seems meaningless despite high workload.
  • team support is lacking. Teams change constantly, workers need to compete with each other, teamplay is implicitly discouraged.
  • lack of fairness. bad communication, suggestions to improve are ignored and workers are given the impression that they are foremost expense factors. Combine this with the lack of control over work environments.
  • loss of identification with work and company 
  • conflict between personal work ethics (do quality work) and job reality (meet unrealistic deadlines).
  • permanent pressure to keep up to date without time allocation or support (resulting in trying to keep up to date in free time)
Most strongly affected are project managers and team leaders, because they experience pressure from both sides. Support from the top seems curiously absent, something I would consider a total management failure.

Oh, and wait: Did I read that correctly? Outsourcing is a conceived a threat and creates a highly stressful environment on both sides ...

The studies also strongly relativize earlier studies that work in IT is usually creative, self-determined, and  autonomous. Either the work environment has changed significantly since these initial observations or the situation was bad in the first place.

A DIWA-IT study claims that the work environment has changed significantly for the worse and blames
  • tighter financial investments in IT projects
  • high "replaceability" of workers because of offshoring, economization, and internationalization.
  • loss of influence from workers, "industrialization", market orientation. Workers feel both overtasked and helpless to achieve their goals.
  • social cohesion in team is cancelled by management methods and constant pressure.
  • "system of constant probation", result oriented work, workers are trying to achieve goals they cannot achieve with their resources. Workers that do not achieve results are threatend with the loss of their job ("be more productive or be fired").
Especially the latter is a marvelous strategy: Everything seems to be managed with threats. Perceived economic pressure is pushed through directly to workers: Adapt or perish.

I only recently read an article by some cloud computing advocate. He/she basically said that corporate IT needs to offer services that are on par with off-the-shelf cloud offerings, because those are so much better and easier to use than what corporate IT offers. The threat then is: Give me this or corporate IT will die. Seriously, who are you? If such people are responsible for anything, they should be made personally responsible for any work-related illnesses of their subordinates and customers.
You want to know, why your employees are no longer motivated? Because you are brilliantly killing their motivation and harvest their energy, leaving them drained. You offer them no support and try to manage them with threats.

So what do you offer IT workers? Small scale work with limited scope but big responsibilities. Constant need to do long hours without even seeing the results or with no or only little appreciation for their work.

And when nobody cares about my work, I would try to stop caring about my work, too. Because when there's no appreciation, but only running against a wall, continuing to run against the wall is a recipe for disaster.

My theory is: All of this was working back in the day when the workload was not too high and people where motivated by their tasks, because they had both responsibility for larger systems and the possibility to influence decisions. The churning of "new" technologies was slower (I repeat: new technology does not necessarily mean innovation and new is not neccessarily better in IT) and knowledge was significantly more stable. People kept up with changes mostly in their free time or during work lull times. And now? Lull times are being or have been eliminated. Experts talk about increased work density. This requires more focus on day-to-day tasks during work time. The result is that workers
  • either stop honing their skills and only do their jobs. With the constant churn of incompatible no-progress, their net-worth is reduced simply by the adoption of "new" technologies.
  • or shift their knowledge acquisition into their free time. This causes them to get more likely to burn out or at least fatigue. Combine this with overload and lack of appreciation in their day jobs and the constant threat of outsourcing and the like for added benefit.
Also the complexity of some of the modern systems make it next to impossible to test them out with the resources available to a single person. You cannot learn to administer a network cluster with a few machines alone even if you use virtual machines. I do not have access to enterprise grade network equipment to fiddle with. I lack a few cloud servers to play with puppet, vagrant, and all the other nice tools. However, people still try and try and they feel helpless while doing so.

This theory is backed by the DIWA-IT working paper: IT workers shut off their own, health-related early warning system because of a loss of inner (identity) coherence. Most of them probably really liked their profession in the first place and wanted to get something done. But getting things done has become harder and harder with the (often needlessly) increased complexity of the IT landscape. Add constant churn (useless change), global competition and greatly reduced rewards and gratifications and you have a disaster recipe.

Because of the constant threats, workers compromise and these compromises more often than not are in conflict with personal vision and work ethics. So they not only compromise their personal values but are also forced to spend more and more time doing work they feel is insufficient to simply retain their livelihood because they experience a very real existential threat (loss of job, unemployability).

It is, in essence, a burnout mill, where you either go numb or fall of the cliff. And it does not get any better: The cited studies and working papers are from 2005, 2008, and 2011.


Sonntag, 6. September 2015

You broke my regular expressions

A quick one to get you to re-think some of the regular expressions you wrote:

What part of

    AAAA

does

    (AA|AAAA)

match?

If you thought AAAA, welcome to POSIX (and C, C++, boost) country, where regular expressions return the longest leftmost match over the whole expression.

If you thought AA, welcome to Perl, Python, Ruby, PCRE, Java, and the like, where alternates are matched left-to-right and only the first one matches.

The latter is called eager alternative matching, but I would rather like to call it short-circuiting. It seems to simplify matching of alternatives and reduce backtracking (and thus memory requirements), or in short: it's faster. It can also be annoying if the engine does not allow you to switch between the different interpretations, because sometimes you really want the real leftmost longest match. And none of the current engines allow that switch.

It's not only alternates, however: Matching A(A)?(AB)? against AAB matches AA instead of AAB with modern modern engines. Left longest match? Nope. Instead, the quantifiers are greedy even if the longest match rules is violated. You need to re-write the above as A(A)??(AB)? (note the double quotation marks) to get POSIX-like behavior.

That greedy matching modifiers even exists is an artifact of dropping leftmost longest semantics. Modern regular expression engines do not perform longest leftmost overall matching. If they claim otherwise, the documentation author is clueless.

So, some time in last 20 years your regular expressions slowly but surely has their semantics changed for the sake of more features and more speed. And it still leads to confusion.

What do we learn from this? Well ... this what happens when you --more or less-- silently change something that was accepted as the norm for quite a long time. It is really quite important to document such subtle but fundamental changes loudly, prominently, and clearly.

And yes, I want my leftmost longest match back at least as an option. Python's regex now has that option since version 2015.09.15. Nice to have and thank you Matthew.

Samstag, 5. September 2015

Rules for Explainers

I've been dawdling in programming forums for quite a while now and something really strikes me again and again: Some of the smartest girls and guys out there are really bad at explaining and teaching.

  1. Solving a problem is different from helping someone solve a problem.

    You will find quite a few answers to technical questions, where someone simply drops a correct solution that has nothing to do with the original post.

    Often enough they do not even bother to explain what the original poster did wrong. They just solve the problem and leave. There, I am much smarter than you, I solved your problem. Now go away.

    Don't do this. When you answer someone's question, you are not only a problem solver, you are a teacher. In that role, you are supposed to help the other learn something. And reverse engineering a working solution is not a very effective learning method, because it quite often does not  yield any insight on the underlying mechanics.

    It is the old feed him for a day-problem: The goal is to give the "student" enough information to help himself/herself the next time round. The oh-so-common howto-style of documentation is exactly the opposite: stray from the path of the howto and you get lost.

    Somebody without a deep understanding of a problem is very likely to  incorrectly reason about why your solution works and his/hers does not.
    So, when you find out somebody has a non-solution to a problem, try to at least a little thing about out how she/he actually arrived at the wrong solution. Empathy is not optional, here.
     
  2. Don't explain with your goggles on.

    Recently, somebody told me, alternate expressions of a regex are matched left to right, matching the first, ignoring the leftmost longest rule.

    That's correct. For Python. Almost everywhere else (POSIX, Ruby, ...), this is wrong, and so actually a really hard gotcha.

    Regular expressions traditionally applied the leftmost longest match rule over the whole match. Python and PCRE regular expressions are different, they short-circuit the first matching alternate, if a subsequent match could have been longer, it is ignored.

    The problem is, that the poster did state it as an overall fact, not bothering to explain that this is actually a very specific concept. He/she was using his Python goggles to look at the problem, not bothering or not being able to take the broader view. This usually gets across either as  arrogant or ignorant. It gets worse when you basically drop this as a brief comment on someone and then leave. Or in short: It's just rude.

    To be fair, in the described situation, the poster and I discussed this and came around to reconcile our view of the situation. This pushed up my respect for him quite a notch. Confrontation is only bad when there's no progress from it.
  3. Don't assume

    One of my favorite responses to one of my postings starts with "If you believe that ...". Everything that follows is sarcastic condescension, very thinly veiled. And when called out, the responder simply re-asserted his position without bothering to recheck his facts. Order No. 227, anyone? Three mistakes in one go: Assuming that I meant what he/she understood (I clearly didn't), immediately thinking that his/her opinion/intellect is superior to mine (I cannot comment on that), and non-reconsidering his/her position when I stood my ground (possibly because of arrogance).

    It happens a lot with nerds, especially younger ones. Hell, I think most of us are like this before age teaches us that we are not the smartest person on the planet. But for some the humbling down never happens. It's also kind of a self-reinforcing mechanism: Most people just walk away right after the first time you are rude at them and they never come back. This looks like a victory ("I was right"), but it is actually downright failure. The goal wasn't to be right, but to convince others. People walking away from you are not persuaded, they are disgusted. You see a pattern: Yep, again, assumption won out and re-assessment did not happen. You only think you are always right, because you never re-evaluate. Congratulations.

    Walking through life always totally assured that your interpretation of the world is correct gives you a lot of momentum and can take you far. It also means that it hurts all the more whenever you hit that waiting brick wall.

    Allow yourself the doubt, it makes you a better person and a better teacher. And give others the benefit of doubt.
To be continued...