Message-ID: <23337c51-7a48-d5a8-569d-ef3ce6fe235f@timescale.com>
Date: Thu, 7 Jul 2022 16:24:49 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.9.1
Subject: Re: Kerberos delegation support in libpq and postgres_fdw
Content-Language: en-US
To: Stephen Frost <sfrost@snowman.net>
Cc: pgsql-hackers <pgsql-hackers@lists.postgresql.org>,
 Magnus Hagander <magnus@hagander.net>,
 Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
 Tom Lane <tgl@sss.pgh.pa.us>, Robert Haas <robertmhaas@gmail.com>
References: 
 <CO1PR05MB8023CC2CB575E0FAAD7DF4F8A8E29@CO1PR05MB8023.namprd05.prod.outlook.com>
 <CO1PR05MB80235F71EE030594A275AF5DA8E49@CO1PR05MB8023.namprd05.prod.outlook.com>
 <19afb899-318d-c61f-2de4-0e7dcbf5e0fd@enterprisedb.com>
 <20220301012847.GQ10577@tamriel.snowman.net>
 <c71f08c30aed2aa19f28244d3a41d315ce9a91de.camel@vmware.com>
 <CAOuzzgogMK9Kj=urNg_iCaJHamLvM29Qc5d0jRDm6TucO0KpQA@mail.gmail.com>
 <b75459ab91879ff081c2d0ca1d3a2d3590b08de1.camel@vmware.com>
 <20220406192703.GY10577@tamriel.snowman.net>
 <20220408042126.GC10577@tamriel.snowman.net>
 <20220408122130.GD10577@tamriel.snowman.net>
From: Jacob Champion <jchampion@timescale.com>
In-Reply-To: <20220408122130.GD10577@tamriel.snowman.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/23337c51-7a48-d5a8-569d-ef3ce6fe235f%40timescale.com>
Precedence: bulk

On 4/8/22 05:21, Stephen Frost wrote:
> Added a few more tests and updated the documentation too.  Sadly, seems
> we've missed the deadline for v15 though for lack of feedback on these.
> Would really like to get some other folks commenting as these are new
> pg_hba and postgresql.conf options being added.

Sorry for the incredibly long delay; I lost track of this thread during
the email switch. I'm testing the patch with various corner cases to try
to figure out how it behaves, so this isn't a full review, but I wanted
to jump through some of the emails I missed and at least give you some
responses.

As an overall note, I think the patch progression, and adding more
explicit control over when credentials may be delegated, is very
positive, and +1 for the proposed libpq connection option elsewhere in
the thread.

On 4/7/22 21:21, Stephen Frost wrote:
>> That an admin might have a credential cache that's picked up and used
>> for connections from a regular user backend to another system strikes me
>> as an altogether concerning idea.  Even so, in such a case, the admin
>> would have had to set up the user mapping with 'password required =
>> false' or it wouldn't have worked for a non-superuser anyway, so I'm not
>> sure that I'm too worried about this case.
>
> To address this, I also added a new GUC which allows an administrator to
> control what the credential cache is set to for user-authenticated
> backends, with a default of MEMORY:, which should generally be safe and
> won't cause a user backend to pick up on a file-based credential cache
> which might exist on the server somewhere.  This gives the administrator
> the option to set it to more-or-less whatever they'd like though, so if
> they want to set it to a file-based credential cache, then they can do
> so (I did put some caveats about doing that into the documentation as I
> don't think it's generally a good idea to do...).

I'm not clear on how this handles the collision case. My concern was
with a case where you have more than one foreign table/server, and they
need to use separate credentials. It's not obvious to me how changing
the location of a (single, backend-global) cache mitigates that problem.

I'm also missing something about why password_required=false is
necessary (as opposed to simply setting a password in the USER MAPPING).
My current test case doesn't make use of password_required=false and it
appears to work just fine.

On 4/6/22 12:27, Stephen Frost wrote:
>> Another danger might be disclosure/compromise of middlebox secrets? Is
>> it possible for someone who has one half of the credentials to snoop on
>> a gssenc connection between the proxy Postgres and the backend
>> Postgres?
>
> A compromised middlebox would, of course, be an issue- for any kind of
> delegated credentials (which certainly goes for cleartext passwords
> being passed along, and that's currently the only thing we support..).
> One nice thing about GSSAPI is that the client and the server validate
> each other, so it wouldn't just be 'any' middle-box but would have to be
> one that was actually a trusted system in the infrastructure which has
> somehow been compromised and was still trusted.

I wasn't clear enough, sorry -- I mean that we have to prove that
defaulting allow_cred_delegation to true doesn't cause the compromise of
existing deployments.

As an example, right now I'm trying to characterize behavior with the
following pg_hba setup on the foreign server:

    hostgssenc all all ... password

So in other words we're using GSS as transport encryption only, not as
an authentication provider. On the middlebox, we create a FOREIGN
SERVER/TABLE that points to this, and set up a USER MAPPING (with no
USAGE rights) that contains the necessary password. (I'm using a
plaintext password to make it more obvious what the danger is, not
suggesting that this would be good practice.)

As far as I can tell, to make this work today, a server admin has to set
up a local credential cache with the keys for some one-off principal. It
doesn't have to be an admin principal, because the point is just to
provide transport protection for the password, so it's not really
particularly scary to make it available to user backends. But this new
proposed feature lets the client override that credential cache,
substituting their own credentials, for which they have all the Kerberos
symmetric key material.

So my question is this: does substituting my credentials for the admin's
credentials let me weaken or break the transport encryption on the
backend connection, and grab the password that I'm not supposed to have
access to as a front-end client?

I honestly don't know the answer; GSSAPI is a black box that defers to
Kerberos and there's a huge number of specs that I've been slowly making
my way through. But in my tests, if I turn on credential forwarding,
Wireshark is suddenly able to use the *client's* keys to decrypt pieces
of the TGS's conversations with the *middlebox*, including session keys,
and that doesn't make me feel very good about the strength of the crypto
when the middlebox starts talking to the backend foreign server.

Maybe there's some ephemeral exchange going on that makes it too hard to
attack in practice, or some other mitigations. But Wireshark doesn't
understand how to dissect the libpq gssenc exchange, and I don't know
the specs well enough yet, so I can't really prove it either way. Do you
know more about the underlying GSS exchange, or else which specs cover
the low-level details? I'm trying to avoid writing Wireshark dissector
code, but maybe that'd be useful either way...

--Jacob