pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedAmazon RDS auth tokens in .pgpass
10+ messages / 4 participants
[nested] [flat]
* Amazon RDS auth tokens in .pgpass
@ 2020-08-31 17:31 Nicholas Chammas <nicholas.chammas@gmail.com>
0 siblings, 1 reply; 10+ messages in thread
From: Nicholas Chammas @ 2020-08-31 17:31 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
I suspect there are some restrictions on the kind of data you can put in
.pgpass that are not documented sufficiently here:
https://www.postgresql.org/docs/12/libpq-pgpass.html
I am trying to connect to a Postgres database on Amazon RDS using IAM
authentication. This works by having IAM generate an authentication token
that you use as the password for the database user you're connecting as.
You can read more about this here:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.AWSCLI.Post...
Here's an example of what one of these auth tokens looks like (with some of
the sections redacted/altered from a real token):
```
some-host.us-east-1.rds.amazonaws.com:5432/?DBUser=someuser&Action=connect&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEHcaDD1hC2q3GGNNaftOvCLRAetaDArXOt6kpr1Ac83hzwtPxEojvZzARJN%2Ftys%2BkLnxsP6FmHmIMmMERWeGBiJmNcUyXWYY%2BSU9oduSSeAv%2BCpYy028Cep%2Bpyl1Km3B5axPAA2q0L4NWa41LQOayWF8F7%2FlB540%2B0aSkZ%2BIucM%2BLZXTcQl3Q0nJiIgu65lhuME4q3Mvst1ZEaZWfUegCWGaX0npajiKbNU2Ut3FH%2F6046RxlNwO4jg4vteTcPk%2BfgMjuAkf5gFR9EYMTziRx1dJGJn8VGFLTugGVITQI%2FEC1iuqKNb%2Fs%2FoFMisfzgF1nd7kxcZYYAmQtugBnDuJuIboYwYAXI2qn7HDbhgQ5v%2FwPWfZieWO&X-Amz-Credential=ABCDEFGHIJKLMNOPQRST%2F20200831%2Fus-east-1%2Frds-db%2Faws4_request&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-Date=20200831T163708Z&X-Amz-Signature=fb41c59dcaf4e013eda6885a4d456549366d2612ecf06307d2443b6a44461ab7
```
That whole thing is the password for `someuser`.
Now, I know that .pgpass works for me because I can use it to connect to
another user that has a regular-looking password like `some-password-123`.
And I know that using these auth tokens with `someuser` also works because
psql successfully connects to that user if I provide the auth token via the
`PGPASSWORD` environment variable.
It's only the combination of a) .pgpass and b) RDS auth token that doesn't
work. This leads me to believe something is going wrong on a code path
specific to .pgpass, or that there are some undocumented restrictions on
what can go in .pgpass.
The docs do instruct:
> If an entry needs to contain : or \, escape this character with \.
So I tried escaping the one `:` character in the auth token with a
backslash, but that didn't help. Thinking that perhaps there are other
characters that might need escaping, I also tried escaping every character
in the auth token -- i.e. `\s\o\m\e\-\h\o\s\t\.\u\s...` -- but that didn't
work either. psql fails to connect with:
```
psql: error: could not connect to server: FATAL: PAM authentication failed
for user "someuser"
```
Are the docs on .pgpass missing something? Is there perhaps a bug in how
.pgpass is being parsed?
Nick
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Amazon RDS auth tokens in .pgpass
@ 2020-08-31 18:04 Tom Lane <tgl@sss.pgh.pa.us>
parent: Nicholas Chammas <nicholas.chammas@gmail.com>
0 siblings, 2 replies; 10+ messages in thread
From: Tom Lane @ 2020-08-31 18:04 UTC (permalink / raw)
To: Nicholas Chammas <nicholas.chammas@gmail.com>; +Cc: pgsql-admin@lists.postgresql.org
Nicholas Chammas <nicholas.chammas@gmail.com> writes:
> I am trying to connect to a Postgres database on Amazon RDS using IAM
> authentication. This works by having IAM generate an authentication token
> that you use as the password for the database user you're connecting as.
> You can read more about this here:
> https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.AWSCLI.Post...
> Here's an example of what one of these auth tokens looks like (with some of
> the sections redacted/altered from a real token):
> ```
> some-host.us-east-1.rds.amazonaws.com:5432/?DBUser=someuser&Action=connect&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEHcaDD1hC2q3GGNNaftOvCLRAetaDArXOt6kpr1Ac83hzwtPxEojvZzARJN%2Ftys%2BkLnxsP6FmHmIMmMERWeGBiJmNcUyXWYY%2BSU9oduSSeAv%2BCpYy028Cep%2Bpyl1Km3B5axPAA2q0L4NWa41LQOayWF8F7%2FlB540%2B0aSkZ%2BIucM%2BLZXTcQl3Q0nJiIgu65lhuME4q3Mvst1ZEaZWfUegCWGaX0npajiKbNU2Ut3FH%2F6046RxlNwO4jg4vteTcPk%2BfgMjuAkf5gFR9EYMTziRx1dJGJn8VGFLTugGVITQI%2FEC1iuqKNb%2Fs%2FoFMisfzgF1nd7kxcZYYAmQtugBnDuJuIboYwYAXI2qn7HDbhgQ5v%2FwPWfZieWO&X-Amz-Credential=ABCDEFGHIJKLMNOPQRST%2F20200831%2Fus-east-1%2Frds-db%2Faws4_request&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-Date=20200831T163708Z&X-Amz-Signature=fb41c59dcaf4e013eda6885a4d456549366d2612ecf06307d2443b6a44461ab7
> ```
Ugh :-(
> It's only the combination of a) .pgpass and b) RDS auth token that doesn't
> work. This leads me to believe something is going wrong on a code path
> specific to .pgpass, or that there are some undocumented restrictions on
> what can go in .pgpass.
Looking at the source code, there's a hard-wired restriction that lines of
.pgpass can't be more than 320 characters long (well, NAMEDATALEN*5, but
very few builds don't have NAMEDATALEN=64). I see that somebody very
recently added code to make libpq print a warning for overlength lines,
but I wonder why they didn't just, um, remove the restriction. We had
not previously heard of a use-case for passwords with hundreds of
characters in them, but I guess we need to cope.
If you're in a position to rebuild libpq, could you check that changing
LINELEN in fe-connect.c to something large enough (like 1K) fixes your
problem? While that's clearly one issue, it'd be good to verify that
there's not another one lurking behind it.
regards, tom lane
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Amazon RDS auth tokens in .pgpass
@ 2020-08-31 18:16 Stephen Frost <sfrost@snowman.net>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 10+ messages in thread
From: Stephen Frost @ 2020-08-31 18:16 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Nicholas Chammas <nicholas.chammas@gmail.com>; pgsql-admin@lists.postgresql.org
Greetings,
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Nicholas Chammas <nicholas.chammas@gmail.com> writes:
> > It's only the combination of a) .pgpass and b) RDS auth token that doesn't
> > work. This leads me to believe something is going wrong on a code path
> > specific to .pgpass, or that there are some undocumented restrictions on
> > what can go in .pgpass.
>
> Looking at the source code, there's a hard-wired restriction that lines of
> .pgpass can't be more than 320 characters long (well, NAMEDATALEN*5, but
> very few builds don't have NAMEDATALEN=64). I see that somebody very
> recently added code to make libpq print a warning for overlength lines,
> but I wonder why they didn't just, um, remove the restriction. We had
> not previously heard of a use-case for passwords with hundreds of
> characters in them, but I guess we need to cope.
My memory might be faulty, but I've got some specific recollection of
people complaining about this before and the community response being
"who would ever need such very long things?!"
In particular, this seems relevant...
https://www.postgresql.org/message-id/29496.1539388920%40sss.pgh.pa.us
but I'm also inclined to add support for these longer values.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Amazon RDS auth tokens in .pgpass
@ 2020-08-31 18:21 Tom Lane <tgl@sss.pgh.pa.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 2 replies; 10+ messages in thread
From: Tom Lane @ 2020-08-31 18:21 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Nicholas Chammas <nicholas.chammas@gmail.com>; pgsql-admin@lists.postgresql.org
Stephen Frost <sfrost@snowman.net> writes:
> My memory might be faulty, but I've got some specific recollection of
> people complaining about this before and the community response being
> "who would ever need such very long things?!"
Yeah, so I guessed wrong ;-). Never too late to absorb new facts though.
That thread does point out that passwordFromFile() is far from the
only place that assumes passwords aren't going to be longer than
what would be sane to enter manually. I wonder whether we need to
worry about the other bottlenecks.
regards, tom lane
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Amazon RDS auth tokens in .pgpass
@ 2020-08-31 18:25 Stephen Frost <sfrost@snowman.net>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 10+ messages in thread
From: Stephen Frost @ 2020-08-31 18:25 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Nicholas Chammas <nicholas.chammas@gmail.com>; pgsql-admin@lists.postgresql.org
Greetings,
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > My memory might be faulty, but I've got some specific recollection of
> > people complaining about this before and the community response being
> > "who would ever need such very long things?!"
>
> Yeah, so I guessed wrong ;-). Never too late to absorb new facts though.
Sure. :)
> That thread does point out that passwordFromFile() is far from the
> only place that assumes passwords aren't going to be longer than
> what would be sane to enter manually. I wonder whether we need to
> worry about the other bottlenecks.
Well, as I said in that thread two years ago, seems like we should make
it work everywhere and be consistent between frontend and backend
regarding what's supported. Perhaps even clearly document what the
limit is too...
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 818B) download
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Amazon RDS auth tokens in .pgpass
@ 2020-08-31 18:26 Ron <ronljohnsonjr@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 10+ messages in thread
From: Ron @ 2020-08-31 18:26 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
On 8/31/20 1:21 PM, Tom Lane wrote:
> Stephen Frost <sfrost@snowman.net> writes:
>> My memory might be faulty, but I've got some specific recollection of
>> people complaining about this before and the community response being
>> "who would ever need such very long things?!"
> Yeah, so I guessed wrong ;-). Never too late to absorb new facts though.
>
> That thread does point out that passwordFromFile() is far from the
> only place that assumes passwords aren't going to be longer than
> what would be sane to enter manually. I wonder whether we need to
> worry about the other bottlenecks.
Would it be better to add a .pgtoken file which gets examined (and bypasses
those other length restrictions) if the user isn't in .pgpass? Or is that a
feature which would have to wait for v14?
--
Angular momentum makes the world go 'round.
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Amazon RDS auth tokens in .pgpass
@ 2020-08-31 18:37 Tom Lane <tgl@sss.pgh.pa.us>
parent: Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 10+ messages in thread
From: Tom Lane @ 2020-08-31 18:37 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Nicholas Chammas <nicholas.chammas@gmail.com>; pgsql-admin@lists.postgresql.org
Stephen Frost <sfrost@snowman.net> writes:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> That thread does point out that passwordFromFile() is far from the
>> only place that assumes passwords aren't going to be longer than
>> what would be sane to enter manually. I wonder whether we need to
>> worry about the other bottlenecks.
> Well, as I said in that thread two years ago, seems like we should make
> it work everywhere and be consistent between frontend and backend
> regarding what's supported. Perhaps even clearly document what the
> limit is too...
In the case of passwordFromFile(), the line doesn't only contain a
password. There's also a hostname that has no a-priori upper length,
and some other fields too; not to mention that if we have a convention
for comments then it's unfriendly to have an a-priori upper length for
comment lines. So I'm thinking that the correct thing to do in
passwordFromFile() is use an expansible buffer and fail only on OOM.
There remains, though, the question of whether any of the other limits
are problematic.
regards, tom lane
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Amazon RDS auth tokens in .pgpass
@ 2020-09-01 18:25 Tom Lane <tgl@sss.pgh.pa.us>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 10+ messages in thread
From: Tom Lane @ 2020-09-01 18:25 UTC (permalink / raw)
To: Nicholas Chammas <nicholas.chammas@gmail.com>; +Cc: pgsql-admin@lists.postgresql.org
Nicholas Chammas <nicholas.chammas@gmail.com> writes:
> On Mon, Aug 31, 2020 at 2:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Looking at the source code, there's a hard-wired restriction that lines of
>> .pgpass can't be more than 320 characters long (well, NAMEDATALEN*5, but
>> very few builds don't have NAMEDATALEN=64). I see that somebody very
>> recently added code to make libpq print a warning for overlength lines,
>> but I wonder why they didn't just, um, remove the restriction. We had
>> not previously heard of a use-case for passwords with hundreds of
>> characters in them, but I guess we need to cope.
> Just FYI, the auth tokens generated by Amazon RDS appear to be 796 bytes
> long.
Thanks. I've pushed a fix to remove libpq's undocumented restriction
on the length of a .pgpass line. It will be in November's releases.
regards, tom lane
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Amazon RDS auth tokens in .pgpass
@ 2020-09-01 22:18 Nicholas Chammas <nicholas.chammas@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 10+ messages in thread
From: Nicholas Chammas @ 2020-09-01 22:18 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin@lists.postgresql.org
Thanks Tom. It looks like this is the commit corresponding to your fix:
https://github.com/postgres/postgres/commit/b55b4dad99e99d5306744a4e8ef8021fa3a922e4
On Tue, Sep 1, 2020 at 2:25 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Nicholas Chammas <nicholas.chammas@gmail.com> writes:
> > On Mon, Aug 31, 2020 at 2:04 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> Looking at the source code, there's a hard-wired restriction that lines
> of
> >> .pgpass can't be more than 320 characters long (well, NAMEDATALEN*5, but
> >> very few builds don't have NAMEDATALEN=64). I see that somebody very
> >> recently added code to make libpq print a warning for overlength lines,
> >> but I wonder why they didn't just, um, remove the restriction. We had
> >> not previously heard of a use-case for passwords with hundreds of
> >> characters in them, but I guess we need to cope.
>
> > Just FYI, the auth tokens generated by Amazon RDS appear to be 796 bytes
> > long.
>
> Thanks. I've pushed a fix to remove libpq's undocumented restriction
> on the length of a .pgpass line. It will be in November's releases.
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2020-09-01 22:18 UTC | newest]
Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-08-31 17:31 Amazon RDS auth tokens in .pgpass Nicholas Chammas <nicholas.chammas@gmail.com>
2020-08-31 18:04 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-08-31 18:16 ` Stephen Frost <sfrost@snowman.net>
2020-08-31 18:21 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-08-31 18:25 ` Stephen Frost <sfrost@snowman.net>
2020-08-31 18:37 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-08-31 18:26 ` Ron <ronljohnsonjr@gmail.com>
2020-09-01 18:25 ` Tom Lane <tgl@sss.pgh.pa.us>
2020-09-01 22:18 ` Nicholas Chammas <nicholas.chammas@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox