Subject: Re: Security lessons from liblzma - libsystemd

From: Magnus Hagander <[email-masked]>
Date: Wed, 3 Apr 2024 23:19:54 +0200

On Wed, Apr 3, 2024 at 7:57 PM Andres Freund <[email-masked]> wrote:

> Hi,
>
> As most will know by now, the way xz debacle was able to make sshd
> vulnerable
> was through a dependency from sshd to libsystemd and then from libsystemd
> to
> liblzma. One lesson from this is that unnecessary dependencies can still
> increase risk.
>

Yeah, I think that's something to consider for every dependency added. I
think we're fairly often protected against "adding too many libraries"
because many libraries simply don't exist for all the platforms we want to
build on. But it's nevertheless something to think about each time.

It's worth noting that we have an optional dependency on libsystemd as well.
>
> Openssh has now integrated [1] a patch to remove the dependency on
> libsystemd
> for triggering service manager readyness notifications, by inlining the
> necessary function. That's not hard, the protocol is pretty simple.
>
> I suspect we should do the same. We're not even close to being a target as
> attractive as openssh, but still, it seems unnecessary.
>

+1.

When the code is this simple, we should definitely consider carrying it
ourselves. At least if we don't expect to need *other* functionality from
the same library in the future, which I doubt we will from libsystemd.

An argument could be made to instead just remove support, but I think it's
> quite valuable to have intra service dependencies that can rely on the
> server
> actually having started up.
>
>
If we remove support we're basically just asking most of our linux
packagers to add it back in, and they will add it back in the same way we
did it. I think we do everybody a disservice if we do that. It's useful
functionality.

//Magnus

---

From: Tom Lane <[email-masked]>
Date: Wed, 03 Apr 2024 17:58:55 -0400

I didn't read the patch, but if it's short and stable enough then this
seems like a good idea.  (If openssh and we are using such a patch,
that will probably be a big enough stake in the ground to prevent
somebody deciding to change the protocol ...)

>> server actually having started up.

> did it. I think we do everybody a disservice if we do that. It's useful

Yeah, that idea seems particularly silly in view of the desire
expressed earlier in this thread to reduce the number of patches
carried by packagers.  People packaging for systemd-using distros
will not consider that this functionality is optional.

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Wed, 3 Apr 2024 15:16:03 -0700

Hi,

On 2024-04-03 17:58:55 -0400, Tom Lane wrote:
> seems like a good idea.

It's basically just checking for an env var, opening the unix socket indicated
by that, writing a string to it and closing the socket again.

> (If openssh and we are using such a patch, that will probably be a big
> enough stake in the ground to prevent somebody deciding to change the
> protocol ...)

One version of the openssh patch to remove liblzma was submitted by one of the
core systemd devs, so I think they agree that it's a stable API.  The current
protocol supports adding more information by adding attributes, so it should
be extensible enough anyway.

> will not consider that this functionality is optional.

Yep.

Greetings,

Andres Freund

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 4 Apr 2024 01:10:20 +0200

On 03.04.24 23:19, Magnus Hagander wrote:
> ourselves. At least if we don't expect to need *other* functionality 
> from the same library in the future, which I doubt we will from libsystemd.

Well, I've long had it on my list to do some integration to log directly 
to the journal, so you can preserve metadata better.  I'm not sure right 
now whether this would use libsystemd, but it's not like there is 
absolutely no other systemd-related functionality that could be added.

Personally, I think this proposed change is trying to close a barndoor 
after a horse has bolted.  There are many more interesting and scary 
libraries in the dependency tree of "postgres", so just picking off one 
right now doesn't really accomplish anything.  The next release of 
libsystemd will drop all the compression libraries as hard dependencies, 
so the issue in that sense is gone anyway.  Also, fun fact: liblzma is 
also a dependency via libxml2.

---

From: Étienne BERSAC <[email-masked]>
Date: Mon, 08 Apr 2024 12:05:18 +0200

Hi,

> There are many more interesting and scary libraries in the dependency
> tree of "postgres", so just picking off one right now doesn't really
> accomplish anything.  The next release of libsystemd will drop all
> the compression libraries as hard dependencies, so the issue in that
> sense is gone anyway.  Also, fun fact: liblzma is also a dependency
> via libxml2.

Having an audit of all libraries linked to postgres and their level of
trust should help to point the next weak point. I'm pretty sure we have
several of these tiny libraries maintained by a lone out of time hacker
linked somewhere. What is the next xz ?

Regards,
Étienne 
-- 
DALIBO

---

From: Magnus Hagander <[email-masked]>
Date: Fri, 12 Apr 2024 16:46:15 +0200

> absolutely no other systemd-related functionality that could be added.

Ah interesting. I hadn't thought of that use-case.

> also a dependency via libxml2.

To be clear, I didn't mean to single out this one, just saying that it's
something we should keep in consideration in general when adding library
dependencies. Every new dependency, no matter how small, increases the
management and risks for it. And we should just be aware of that and weigh
them against each other.

As in we should *consider* it, that doesn't' mean we should necessarily
*do* it.

(And yes, there are many scary dependencies down the tree)

-- 
 Magnus Hagander
 Me: https://www.hagander.net/ <http://www.hagander.net/>
 Work: https://www.redpill-linpro.com/ <http://www.redpill-linpro.com/>

---

From: Andres Freund <[email-masked]>
Date: Fri, 12 Apr 2024 09:00:11 -0700

Hi,

On 2024-04-04 01:10:20 +0200, Peter Eisentraut wrote:
> Well, I've long had it on my list to do some integration to log directly to
> the journal, so you can preserve metadata better.  I'm not sure right now
> whether this would use libsystemd, but it's not like there is absolutely no
> other systemd-related functionality that could be added.

Interesting. I think that'd in all likelihood end up using libsystemd.

> Personally, I think this proposed change is trying to close a barndoor after
> a horse has bolted.  There are many more interesting and scary libraries in
> the dependency tree of "postgres", so just picking off one right now doesn't
> really accomplish anything.  The next release of libsystemd will drop all
> the compression libraries as hard dependencies, so the issue in that sense
> is gone anyway.  Also, fun fact: liblzma is also a dependency via libxml2.

I agree that doing this just because of future risk in liblzma is probably not
worth it. Despite soon not being an unconditional dependency of libsystemd
anymore, I'd guess that in a few months it's going to be one of the better
vetted libraries out there.  But I don't think that means it's not worth
reducing the dependencies that we have unconditionally loaded.

Having a dependency to a fairly large library (~900k), which could be avoided
with ~30 LOC, is just not worth it. Independent of liblzma. Not from a
performance POV, nor from a risk POV.

I'm actually fairly bothered by us linking to libxml2. It was effectively
unmaintained for most of the last decade, with just very occasional drive-by
commits. And it's not that there weren't significant bugs or such. Maintenance
has picked up some, but it's still not well maintained, I'd say.  If I wanted
to attack postgres, it's where I'd start.

My worry level, in decreasing order, about postmaster-level dependencies:
- libxml2 - effectively unmaintained, just about maintained today
- gssapi - heavily undermaintained from what I can see, lots of complicated code
- libldap - undermaintained, lots of creaky old code
- libpam - undermaintained, lots of creaky old code
- the rest

Greetings,

Andres Freund

---

From: Michael Paquier <[email-masked]>
Date: Tue, 16 Apr 2024 09:35:59 +0900

> to attack postgres, it's where I'd start.

Indeed, libxml2 worries me to, as much as out-of-core extensions.
There are a bunch of these out there, some of them not that
maintained, and they could face similar attacks.
--
Michael