Montag, 23. Oktober 2017

systemd: Please, No, Not Like This

I have become convinced, that I have spent too much time with the systemd debate already. I will try to refuse to do so any further.

I have also become convinced that systemd is a pretty mediocre implementation of some nice, but insufficiently thought-out ideas that nonetheless are pushed in highly aggressive manner. This aggressive push has already been clearly indicated as intentional, including the need to overcome expected resistance no matter what.

To be blunt: My personal verification step is over. I do not like systemd, its design, its implementation, how it is maintained, how (some of) its developers and proponents intentionally alienate whole groups of people, how even valid criticism of it is often declared irrelevant, and I have real trouble with the attitude of many of the people associated with its ecosystem. The whole thing annoys me more than it makes my work simpler, and the situation makes me feel like the things I value in open source and software engineering are being actively undermined.

If you are not able to accept that (and maybe why) I might feel this way, really, you are part of the problem. Because that inability and/or unwillingness to respect an opinion different from your own is a part of the problem.

I don't wish to be evangelized, nor do I wish do be convinced how nice systemd and its developers/maintainers are. That train has passed. I do understand that systemd tries to solve problems but I think the approach taken highly problematic both technically as well as politically and socially.

Thank you
There are several sides to my dislike, depending on who of "me" you ask:

Me, the software developer

That part of me is more than taken aback. I've read my share of system source code and I do not consider the parts I have read quality, maintainable code. Systemd developers don't document internal interfaces and they seem to see internal modularization as something that only lesser developers need to do. Yet even their external interface documentation is severely lacking, not properly documenting semantics and invariants. This also seems to hold with regard to implementing standardized behavior: Do what works in a narrow use case, ignore inconvenient details. As for their own designs: They seem to lean towards complexity and special cases without proper arguing their case. Use of orthogonal concepts, avoiding implicit "magic" automatisms (especially when they cannot be turned off) seem completely unheard off or actively eschewed. The result is that  --in many cases-- systemd features interact in undocumented and unexpected ways and  implementation details are unnecessarily exposed in interfaces, creating longevity and interface stability problems.

In many cases when I resort to chasing down a bug in systemd, I seem to find at least one occurrence of serious WTF. Let it be functions that expect an undocumented amount of buffer space passed in from their callers, a message send function explicitly silencing an error when the send buffer is full, shotgun parsing all over the place, asynchronous sync, blatant ignorance of (secure) language design, configuration formats and associated documentation that freely mix low level with high level directives into a mush with unexpected interactions requiring manual workarounds. Yet there are some pretty boastful claims (like "validates all input") that don't hold up to scrutiny or are even security regressions from previous behaviour. Such arrogant bluster combined with basic mistakes makes me very critical about any claim the authors make about their own system. In general, while systemd code looks superficially well written, its lack of sane interface design and modularization leaves me baffled.

Given these observations, I consider systemd to be a product of sloppy "quick and dirty" engineering, valuing addition of new features or at least the addition of their version of some concept over quality, interoperability, interchangeability, and stability. Systemd seems like the "lean startup" of Open Source. With the associated benefits (pace, features) and also with the associated problems (technical debt, interface design).

Systemd feels like a system built by people who work under the unshakeable belief that the very initial solution sketch they come up with is automatically the best solution. They often claim that there are no real alternatives to their own design and implementation, but their arguments mostly seem to remain shallow and not well-founded, a pattern I have dubbed "replacing competence by confidence". A credible discussion on why a complete, only partially compatible or incompatible re-implementation is necessary is mostly missing.

I often compare this to the Death character in Terry Pratchett's "Reaper Man" and how Death builds a tree swing. He starts with a limited understanding of the problem space and proceeds never taking a step backwards to reconsider a decision once committed to it. The obvious results are not what you want in a critical system component, since it requires the developers to pile workaround on top of workaround.

As an explanation, we get skewed comparisons that leave out basic facts or are based on information that sometimes is decades out of date, but the actual points critics make seem mostly ignored or "whataboutismed away".  More often than not, the "discussion style" reminds me of authoritarian propaganda: You are being informed of "facts" and not made part of an open discussion. Quite obviously, that toxic style triggers huge amounts of equally destructive pushback.

Add to that the tendency to water up promises that turn out to be too hard to keep. Examples: Reliably killing a process only works in certain (typical) cases, can be circumvented by default. Logging reliable metadata? Only works "most of the time", recently watered down because of the performance hit of metadata retrieval. Reliable system message bus? No, only best effort, croaks under load.  Use of PIDfile= has a privilege escalation bug, not implementing previously existing hardening PIDFile= implements relatively complex privilege checks, bypassing pre-existing use of simple user separation by the kernel. I have repeatedly found that such regressions/broken promises are mentioned seemingly only in passing, sometimes as a counter-argument on completely unrelated issues. This in turn causes me cognitive dissonance, since systemd's guarantees seem be often more like empty promises, to be broken at will. I find this both unprofessional as well has dishonest.

Yet, systemd developers have repeatedly claimed no wrongdoing on their part, and --obviously-- that everybody that has a different opinion is clearly uninformed, stupid, unfair, or has other, sinister motives. Of course, if you have a "dissenting opinion", they can always just threaten to refuse cooperation.

On the architectural side, the ideal of openness seems mostly gone. Instead, interoperability and openness are replaced by a hard push of a single, monocultured implementation with barely defined semantics that makes only very limited effort to stick to well-known-concepts. Even if there is no reason to diverge from existing syntax and semantics, systemd often simply ignores prior art.

Yet its own interfaces are frequently poorly defined and often mix multiple independent concerns into a single interface. Documentation language is often vague and specifications remain superficial and incomplete. Even more troublesome, systemd seems prone to introducing hardcoded automagic that cannot be turned off. The repeated bullheading of such features leaves me with the bad taste that systemd's use case analysises are done only with a very narrow focus, requiring others to work around its limitations.

As a conclusion, I think, with regard to open systems, systemd behaves like a bad citizen. It makes a lot of effort to cater to (large scale) administrators and packagers. But from an engineering perspective, I consider systemd a huge regression: It moves away from the concept of open, well-defined protocols, from the ideal of largely replaceable implementations of well-defined interfaces (conceptual composability) to a single, ever-changing implementation whose "expected behavior" is for large parts only provided by its (mostly undocumented) source code and that be changed any time at the convenience of the developers.

This is not free and open software, this is some people do what they see fit and to hell with everybody else that does not completely share their vision. "We are calling the shots. Shut up and do not bother us. We know better, anyway."

Me, the sysadmin

From that perspective, systemd is mostly a neat set of features under a single hood.

Mostly.

This is probably what caters to packagers and system administrators: systemd makes some things decidedly more easy than before. And might it seem really a breath of fresh air if you did not know any of the alternatives that existed before it. Nothing what systemd does is really new, but systemd encapsulates a lot of stuff behind a deceivingly simple interface.

Deceivingly.

It also has its share of unnecessary POLA violations. Documentation is scattered, incomplete, misleading and sometimes downright incorrect. And the docs do not even attempt to give you a mental model of the inner workings of the software. Their over-the-top, "this is the best thing since sliced bread" style of writing annoys me to the bone. Getting a good overview of systemd is hard and I have personally found that this is because that the developers don't bother to provide a coherent and consistent external interface in many cases. It feels like to provide a new feature, the developers have repeatedly duct taped new code to an existing system, never re-thinking their internal model (if such one exists) to provide a coherent and consistent interface. As is, systemd feels more often than not like a slim wrapper on top of kernel interfaces (without proper documentation on which kernel interfaces are wrapped), that combines multiple features but repeatedly fails to provide a coherent, composable model of itself. "Conglomeration of special cases with unplanned emergent interactions" is the best term I have come up with for this so far.

Me, the person

I find the blatant public narcissism, (sometimes not so) covert abuse and manipulation pouring out of systemd's main developer(s) and quite a few proponents horrifying. The use of thought-terminating clichés, wolf speech, proof by assertion, ipse dixit assertions, whataboutisms, shifting the goalpost, stonewalling, and similar "dirty" discussion tricks is particularly worrying. This list is extended by attempts of (emotional) blackmail "Be nice to me or I will ignore you" that should speak volumes about the personalities involved. Yet, somehow, these eminent signs of the dark triad do not give pause, but rather most people seem completely blind or in (unconscious) denial. And don't even get me started on the frequent use of manipulative, covert aggressive speech patterns.
I keep stumbling about the "pulseaudio introduction to Ubuntu" debacle, where somehow, all resulting problems are now blamed exclusively on Ubuntu. There is, however, another interesting timeline where Poettering attacked Ubuntu for being such slow adopters and repeatedly claimed to Ubuntu devs that pulseaudio was production ready. However, when the big fallout happened, blame landed purely at Ubuntu.
Of course, covert aggression is meant to achieve exactly that: It is a method to hide the disregard for the opinion of others, a manipulation tool to lull those around you (and sometimes even the victim) into the false sense that civility has been maintained. But in fact it is just a more devious method of being aggressive and antisocial. A way that seems completely unintelligible to a large set of people who do not understand even the limited subtlety of the process. I am sorry to tell you, but most destructive behavior does not announce its intent with fanfare. Instead it is surprisingly banal and thus maybe even more destructive. Only the really sucessful  abusers manage to make themselves appear (including to themselves) as only defending their own good intentions, because obviously, nobody really wants to be evil.

And, yes, quite a few "open source folks" do play this game (as do many others). That large parts of the community refuse to even consider the issue, is even more dysfunctional. White, middle aged, heterosexual men that a certain dev complains about? Yes, this is exactly their game. Mirror, mirror on the wall.
This is --of course-- not a problem limited exclusively to systemd supporters. Idiots happen everywhere, regardless of their convictions. Yet --again-- the  open disregard, the casual devaluation of whole parts of a community (e.g. "security circus") are a repeated pattern. All of these are strong indicators for pervasive destructive personality patterns. And --I might note-- tu quoque does not excuse bad behavior. You are not a nice person because others might be worse.
This also leads me to another, very much sociological issue that is by chance even worse than the primary culprits: the blind apologism or even unfounded aggression so pervasive on both sides of the debate.

I've seen this pattern before around typical abusers. They are surrounded by a multitude of what I call "asshole excusers" (more formally known as co-dependents) that keep on looking for more and more reasons why their idol is not has he/she seems and that everybody else is to blame. Groupthink is rampant.

To quote some of the "arguments" I had to read: "You may not complain, because you get this for free" (i.e. free software may not be held against requirements), "if you don't like it, write your own" (yeah, right), and "you have a choice" (there's no enterprise distribution without systemd and software dependencies on systemd are increasing) and other kill phrases are in frequent use. You may not question the emperor.

Freiheit ist immer nur Freiheit des anders Denkenden

(Freedom is always the freedom of dissenters.)

-- Rosa Luxemburg

What hits me are not the arrogance and aggression. Arrogant jerks are easy. What hits me are the repeated attempts to argue the abuse away and to silence criticism. Excusing the perpetrator is never a good sign (even if he/she is also a victim: tu quoque). Yet for many members of the community, introspection is forbidden. If you point out the flaws in the process you are a "vocal minority", "uninformed", a "whiner", a "snowflake". Sounds like Open Source Trumpland? It probably is ...
I cannot shake the feeling that the behaviors around systemd will serve as an excuse for the toxic parts of the community to feel even more empowered. Certain aggressive personalities already used the excuse "Linus curses at kernel developers, so it's okay to be an asshole" to rationalize their own aggression. With systemd, they get another excuse. Because when systemd proponents can dismiss any criticism to their own toxicity as deluded utterings, so can everybody else and claim it's okay to do so, too. And obviously, the "other side", will also dig into their trenches, because how are you going talk to people that disregard you and your opinion (without actually listening to it).
Make your own conclusions from that.
And as always, the whole situation would be hilarious, if it was not so overarchingly toxic while trying to maintain the illusion of progress. Pushing your agenda through aggressive means isn't progress, it's regression into a dark age of non-cooperation and despotism. Yes, it might get you to your goal faster, but it also burns all bridges.

Donnerstag, 19. Oktober 2017

The natural urge to debunk propaganda

If you are like me, you have obtained a natural urge to doubt and find flaws in the reasoning of propaganda and marketing. This natural skepticism often made out to be to be a European stereotype: Where the American sees an opportunity, the European sees hidden agendas. Or it's only a German stereotype.

Whatever it is, that knee-jerk reaction of rebuttal is usually triggered by a certain certain use of language. Among the trigger points are
  • hyperbole

    The classic car salesman trick. There's clearly something wrong if the only things you hear make something sound better than perfect.
  • high-handed dismissal of inconvenient details 

    As a rhetorical trick, this is often used in dismissing criticism as fake and unfounded. If so, it usually goes hand in hand with a non sequitur.

    "the developers addressed this". Really? Where? How? When? Also: appeal to (false) authority.
    "people took care not to" Really? In what way? [citation needed] Also: Argumentum ad populum when the previous Argument was from oneselves perspective.

    Combine with tricks on word semantics, if you want the casual reader to accept your statement, when it actually states a contradiction:

    "you can remove it by disabling it". Disable and remove are not the same operation. At this point, it should be mandatory to explain what the effect of disabling something is and if its is really disabled or just no longer effective (which --again-- are two separate things).

  • Claiming decisions or opinions as facts

    This is also called "Proof by Assertion". Even if your ego might think otherwise, claiming something does not make it true. Actually, believing such a thing rather is a tell-sign of psychopathy. And yes, I mean that.

    "the de-facto standard" (argumentum ad populum, maybe even silent majority, bandwagon fallacy, post hoc ergo propter hoc)
    "everybody knows that ..." (Argumentum ad Populum, Appeal to Ancient Wisdom)
    "x did this very badly" (Parade of Horribles, Red Herring, False Dilemma)

Where does this urge to refute these arguments come from?

Well, quite frankly, all of this stuff is manipulative and abusive behavior. You might have gotten used to it, because its so ubiquitous in commercials and political announcements and sometimes even in everyday interactions, but it is still abusive.

I like a phrase by George Simon for this kind of interaction: "covert aggression". Not the swaggering "I'll hit you if you don't do what I say", but the seemingly gentle "I know better than you and I know you don't know any better, so I don't take you seriously".

However, the latter is even more devious and destructive than the former. The former is at least honest. The latter is a devious way of avoiding a fair confrontation. Quite often, somewhere in their lives the manipulators have experienced repeatedly that it is hard to win a proper discussions, but that you can quite often get away with covert manipulation. Add to that a missing or disabled moral compass and you get a manipulative personality.

I'm not sure that the manipulators do this consciously. Quite often enough they seem mentally unequipped to even understand that they are not presenting logical arguments, but that they are pushing their agenda by manipulation. They seem to think that this is normal. Of course, that again can be a clear case of gaslighting: Making you doubt your own observations makes you easier to manipulate.

Covert manipulation is an voluntary (or negligent) attack on your ability to make an informed decision. And while you might not be able to pinpoint the exact methods (obviously, because it won't work otherwise), it still feels wrong. And that's where the urge to resist comes from: It is your subconsciousness telling you that your opposite is neither acting openly and nor in your best interest.

If you mention the problem, they will usually try to argue it away. At which point, the only reaction is to walk away and never come back.

Mittwoch, 18. Oktober 2017

A little story ...

I'd like to tell you a story. If you take it as an allegory, so be it.

Once upon a time there way a village. The village was relatively nice place, even if a little odd and rustic at times. But people liked to live there.
At one point in time, a guy from the village decided that the road leading to the village was a disgrace and that he wanted to improve it. So he just got started. Some people really thought the old road was good enough, but who where they to stop a young guy from improving things.

The young one worked hard for many days. He wasn't very experienced in road building, so he also made many mistakes, but he finally managed to improve the road. It was not the finest piece of workmanship but you could drive a lot faster on it than on the old gravel road that existed.

Everybody knew that on such a fine road, you could drive quite fast on so people did. Some recklessly so, but most of them moderately.

But one night, there was a grave accident. Just around a bend, a car ran into an unmoving obstacle. The  car was a wreck and the driver was dead. There was great clamor ... and there was even more clamour, when it was found out, what the obstacle was: Someone, in the middle of the night, had anchored a steel rod right in the middle of the driving lane  without telling anyone. 

Of course, everybody wanted to remove the rod as quickly as possible. They were starting to do just that, when the young man who built the road stepped forward. 

"Yeah, I did that. That curve is dangerous, so I added the steel rod so that you need to drive around it slowly. Don't you see? It's clearly the driver's fault for going too fast. The rod needs to stay where it is to prevent people from driving too fast. If you need them, you can just put up speed signs!"

People with this kind of attitude exist in reality. Just look a little.

Samstag, 14. Oktober 2017

The curse of the silent assumption

There seems to be a recent surge in the entitled never-do-wrong corner of the world. Or rather it seems that the "fuck you if you do not share my opinion" crowd has finally become mainstream.
This has brought me back to thinking about two little facts that I found to be true not only in the domains I learned them to be true, but also in many other areas:
  • Silent assumptions are the death of a complex system.
  • Interactions between seemingly simple things can unexpectedly be fatal.
I'd like to beg your pardon in advance, since my best examples for both of these things are from knowledge representation logics, probably because I spent quite a bit of my professional life so far in that area.

Silent assumptions are the death of all complex systems

If you have heard a class about formal (knowledge) representation systems, you probably heard about the "frame problem". 

Let's assume that you have a robot R and two boxes A and B.

If you introduce an action move(box, loc1, loc2) that can move a box from a location to another, you also need to establish certain facts about that action that are circumstantial to the primary outcome (box at loc2).

In particular:
  • after move(A, loc1, loc2), B remains at its previous location.
  • after move(B, loc1, loc2), A remains at its previous location.


If you introduce another box C, you also have to state the invariant for C, same for D, .... At this point, it should be easy to see, that this gets out of hand quickly. In fact, it gets out of hand so quickly, that probably the most common mistake when modelling a domain is to forget at least one frame axiom.

You do not even need a mutable universe, to realize the problem, though. For example, if we model the hierarchy

Animal
\- Cat
\- Dog

most people almost immediately assume that Cat and Dog are disjoint classes. And then they never encode that (silent) assumption, because it is just obvious.

This is coincidentally one of the most common mistakes from Computer Science students starting with conceptual modelling. They, nowadays, usually have at least some background in object oriented programming. And in any OO language I know of, classes are disjoint unless they have a common ancestor.

It is just too easy to make this into a silent assumption and transfer that assumption into other formalisms. And hence there are probably thousands of CatDog ontologies out there, where things that really cannot be the same thing are allowed to be.

Us humans are so very good at making these silent assumptions, that we tend to unconsciously skip over them. Obviously, when moving box A, B doesn't get moved. Unless A is the heavier of the two boxes, was initially on a seesaw, with B on the other side.

You get the drill.

Let's face it: You cannot avoid silent assumptions, because then you end up with what I call "world transformer" functions that have the whole universe in their parameter set and return a different, modified universe from that.

But here's a bit of well meant advice: Avoid making assumptions about the obviousness of silent assumptions. That stuff usually comes back to haunt you with a vengeance.


Interactions between seemingly simple things can unexpectedly be fatal.


This is also something I picked up the hard way from knowledge representation systems. If you have a System S and add feature A, the combination SA is totally fine. So is SB for some feature B. But if you make SAB, the whole thing falls apart and becomes undecidable.

Don't believe me? Let me quote two people quite a bit smarter than me [Branchman83]:

We illustrate how great care needs to be taken in the design of a representational facility, even when our intuitions about the language tell us that it is a simple one. As it turns out, even an apparently modest representation language can prove intractable.


Yeah. If you combine two innocent looking things, the result may no longer be that innocent.

Note how that combines with silent assumptions, where the interaction of two programs looks totally innocent to the casual reader, but if you add in the silent assumptions, you get a total, unmaintainable mess.

Or in good old Dijkstra's words [EWD340]:

The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.
Another piece of well-meant advice: When you build a system out of parts, very carefully control the interactions of those parts.

Also from personal experience: The less people actually know and the less self-esteem they have, the harder they stick to their silent assumptions.


[Branchman83]
The Tractability of Subsumption in Frame-Based Description Languages (1983)
Ronald J. Brachman, Hector J. Levesque
Proceedings of AAAI-84
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.464.7663

The Humble Programmer
Edsger W. Dijkstra
ACL Turing Lecture 1972

Montag, 31. Juli 2017

systemd pitfalls

This is a loose collection of various issues you might encounter when using/deploying systemd.

Disclaimer

Systemd is very much a moving target with a relatively spotty history of communicating both existing semantics as well as breaking changes. Even criticial security issues have been in the past fixed without notice, not requesting CVEs, even attacking people who suggested to do so.

Thus whatever you read here might already be out of date.

I try to keep this list --sometimes through repeated editing-- at least somewhat neutral. I might not always manage. Please accept my apologies.

Because there are so many of them: Missing or incomplete documentation is indicated by 📚. Note that this does not mean that an issue can be fixed by only updating the documentation.

The Journal

  • journald completely disregards RFC 5424, section 6.3, including no support for picking up structured log data nor forwarding its own structured data. It does forward existing structured syslog data by virtue of leaving log messages unaltered.
  • journald makes it impossible for syslog implementations to pick up trusted metadata via the kernel. Since it imposes itself between the syslog daemon and the logging service, all kernel-obtainable metadata is from the journal server. If you need that, you must interface with journald. (workaround module for rsyslog, which has trouble with corrupted journal files).
  • The journal's query API is essentially a reverse polish datalog query builder with fixed threefive level nesting and a fixed operator type at each level. The API mixes parsed with non-parsed operations instead of providing a  query language or criteria construction engine.
  • journald stores its file descriptors in PID1 when it is stopped via sd_pid_notify_with_fds(). As this is not implemented as a reliable transmission, journald restarts have a chance of losing all logging streams.
  • Journald automatically attempts to set nocow if /var/log/journal is on a btrfs filesystem 📚. Nocow also disables btrfs checksumming and thus potential data recovery from multiple block copies. This is not mitigated by the journal's limited checksumming. nocow is re-enabled when a journal file is put offline 📚.
  • 📚 The journal file format description is --to this date-- still incomplete. There is no mention of --for example-- sealing and LZ4 compression.
  • 📚 The journal still seems to strip white space from log messages before forwarding them to syslog. This means, that e.g. multiline log entries with whitespace indentation a continuation marker are mangled.
  • When /var/log/journal resides on a separate filesystem, journald might create the journal in (one of the) the parent filesystems and then mount /var{,/log{, /journal}} over that location, making the journal inaccessible during runtime. To fix this, you need to make journald wait for the mount point. Waiting for the directory using .path unit might not work, since it is journald that creates the directory with Storage=persistent.
  • journalctl "-r" does not combine well with "-n" and does the wrong thing.
  • Journald's timestamps are not necessarily when the event happens, but when the journal daemon processes its queue, so that means if it gets less CPU time for some reason or another, there will be a mismatch between the actual time of the event happening and the time appearing in the journal (thanks to @rt2800pci1).

Documentation and "Closed Design" Issues 

  • 📚 The journald query API is  documented by completely avoiding any of the well-known jargon (conjunctive query, predicate, variable, literal, atom) for database/datalog queries and instead uses custom idioms.

Careless Maintainership

  • StartLimitInterval= was silently moved from the [Service] to the [Unit] section. Compatibility is provided for StartLimitInterval=, but not for StartLimitIntervalSec=.📚 for such changes does not appear where it should.

POLA violations

  • $X and ${X} expand differently in unit files. The former does word splitting and quote stripping, while the latter does neither. This is different from basically everywhere else.
  • systemctl stop $service && systemctl start $service may not be the same as systemctl restart $service. PID1 only keeps file descriptors as long as it holds a reference to a unit (documented as "unit not fully stopped"), which is no longer the case after "stop", hence all stored file descriptors are lost.

Service Management

  • It seems systemd does two special things to process running in its DynamicUser facility, first it will try to reuse the same user id if the service remains unchanged by the means of generating a special hash, and second, if you leave IPC objects around by any means (shm, mq), it will pin that UID in memory (see dynamic-user.c and search for ipc, the clean-ipc function returns -1) and never use it. Under a special configuration, this will lead to UID starvation in that range if you restart the service too many times (or use a transient one), if the small UID range is itself not enough to cause this. Also, two services having the StateDirectory= means the one starting later will have the dir chowned to it and screw the former's permissions (thanks to @rt2800pci1).

 Design issues

  • Interaction with journald via PID1 is a deadlock candidate (PID1 blocked on writing to DBUS, dbus blocked on logging to journald, journald blocked on sd_notify), worked around by write-polling the notify socket and issuing a non-blocking write. Again, this might mean you loose all logging streams on restart if the system is busy. 
  • 📚 In general, interaction with the systemd bus by various components is a mix of blocking and non-blocking IO, including silent dropping of messages on EAGAIN. Messaging is either unreliable or contains more possible deadlock candidates.
  • Startup notification via sd_notify() is typically only supported from what systemd considers the main PID of a unit. The presence of the notify socket (and hence notification support) is indicated by an environment variable $NOTIFY_SOCKET. If you spawn notify-capabable subservices from your systemd unit, you need to unset $NOTIFY_SOCKET to prevent systemd warnings, since the subservice will try to notify system of its own startup.

Security Unconscious Design/Implementation 

  • The initial kill for KillMode=process and KillMode=mixed are sent from PID1 as root, ignoring hardening even in previous SysV implementations. Combined with PIDFile=, this allows an unprivileged service to kill an arbitrary process when the service is stopped.  As of #7816 there are additional checks for PID files from unprivileged services. The kill is still performed as root.
  • PID1 has a single-threaded message loop with only limited QoS. For example, a unit can cause a denial of service in PID1 by repeatedly sending file descriptors to PID1. This requires fdstore to be enabled explicitly, though.