pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedpsql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"
8+ messages / 3 participants
[nested] [flat]
* psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"
@ 2024-12-25 17:05 Markus KARG <markus@headcrashing.eu>
2024-12-25 17:39 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Tom Lane <tgl@sss.pgh.pa.us>
2024-12-25 17:52 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Heikki Linnakangas <hlinnaka@iki.fi>
0 siblings, 2 replies; 8+ messages in thread
From: Markus KARG @ 2024-12-25 17:05 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown
below. If this is a bug, I kindly ask to fix it. If this is intended
behavior, I kindly as for a link with instructions how to work around
the problem.
Server:
I am running the official PostgreSQL 17.2 Docker Container
(https://hub.docker.com/layers/library/postgres/17.2/images/sha256-c063081175f45f4a3a5ac03c234e060e67...)
proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container
https://hub.docker.com/layers/library/traefik/v3.2.3/images/sha256-06966a9ba1747ad724a490b8f27df1434...).
Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.
I have neither reconfigured TLS in any other way, nor have I manually
provided TLS certificates to neither client nor server.
Client:
Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am
asking psql to connect to my server. While psql 16.3 and earlier
versions successfully connect via the TLS proxy to the PostgreSQL
server, psql 16.4 and later versions fail doing so:
root@hetzner-2:~# docker run -it postgres:16.3 psql "host=headcrashing.eu port=5432
dbname=postgres user=postgres password=... sslmode=require"
psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian 17.2-1.pgdg120+1))
WARNING: psql major version 16, server major version 17.
Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
Type"help" for help.
postgres=# \q
root@hetzner-2:~# docker run -it postgres:16.4 psql "host=headcrashing.eu port=5432
dbname=postgres user=postgres password=... sslmode=require"
psql: error: connection to server at"headcrashing.eu" (49.13.53.107), port 5432 failed: SSL error: tlsv1 alert no application protocol
Public Test Environment
Feel free to connect to my personal PostgreSQL 17 instance running at
|postgres.headcrashing.eu:5432| (TLS required).
With kind regards
-Markus
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"
2024-12-25 17:05 psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
@ 2024-12-25 17:39 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-12-25 17:45 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
1 sibling, 1 reply; 8+ messages in thread
From: Tom Lane @ 2024-12-25 17:39 UTC (permalink / raw)
To: Markus KARG <markus@headcrashing.eu>; +Cc: pgsql-bugs@lists.postgresql.org
Markus KARG <markus@headcrashing.eu> writes:
> Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown
> below.
Have you tried the current release (16.6)?
regards, tom lane
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"
2024-12-25 17:05 psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
2024-12-25 17:39 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Tom Lane <tgl@sss.pgh.pa.us>
@ 2024-12-25 17:45 ` Markus KARG <markus@headcrashing.eu>
0 siblings, 0 replies; 8+ messages in thread
From: Markus KARG @ 2024-12-25 17:45 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-bugs@lists.postgresql.org
> Markus KARG<markus@headcrashing.eu> writes:
>> Summary: The TLS behavior of psql changed between 16.3 and 16.4 as shown
>> below.
> Have you tried the current release (16.6)?
>
> regards, tom lane
Yes, but psql 16.6 fails in the same way:
root@hetzner-2:~# docker run -it postgres:16.6 psql
"host=headcrashing.eu port=5432 dbname=postgres user=postgres
password=HeadcrashingPostgres2022 sslmode=require"
psql: error: connection to server at "headcrashing.eu" (49.13.53.107),
port 5432 failed: SSL error: tlsv1 alert no application protocol
-Markus
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"
2024-12-25 17:05 psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
@ 2024-12-25 17:52 ` Heikki Linnakangas <hlinnaka@iki.fi>
2024-12-25 18:22 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
1 sibling, 1 reply; 8+ messages in thread
From: Heikki Linnakangas @ 2024-12-25 17:52 UTC (permalink / raw)
To: Markus KARG <markus@headcrashing.eu>; pgsql-bugs@lists.postgresql.org
On 25/12/2024 19:05, Markus KARG wrote:
> I am running the official PostgreSQL 17.2 Docker Container (https://
> hub.docker.com/layers/library/postgres/17.2/images/sha256-
> c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6)
> proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container
> https://hub.docker.com/layers/library/traefik/v3.2.3/images/
> sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c). Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.
In v17, libpq requests the ALPN extension in the TLS handshake. Looks
like the proxy doesn't know about the "postgresql" ALPN protocol, and
rejects the connection.
I guess Traefik needs some configuration changes to tell it that the
"postgresql" protocol is expected. Or code changes.
> Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am
> asking psql to connect to my server. While psql 16.3 and earlier
> versions successfully connect via the TLS proxy to the PostgreSQL
> server, psql 16.4 and later versions fail doing so:
>
> root@hetzner-2:~# docker run -it postgres:16.3 psql "host=headcrashing.eu port=5432
> dbname=postgres user=postgres password=... sslmode=require"
> psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian 17.2-1.pgdg120+1))
> WARNING: psql major version 16, server major version 17.
> Some psql features might not work.
> SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256, compression: off)
> Type"help" for help.
>
> postgres=# \q
> root@hetzner-2:~# docker run -it postgres:16.4 psql "host=headcrashing.eu port=5432
> dbname=postgres user=postgres password=... sslmode=require"
> psql: error: connection to server at"headcrashing.eu" (49.13.53.107), port 5432 failed: SSL error: tlsv1 alert no application protocol
There were no changes between 16.3 and 16.4 to explain this. When I test
that with v16 client that I built from sources, I don't get that error.
The error message suggests that you're actually using libpq v17. And
indeed I get that error when connecting with v17 client. Perhaps the
postgres:16.4 docker image was built with v17 libpq?
--
Heikki Linnakangas
Neon (https://neon.tech)
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"
2024-12-25 17:05 psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
2024-12-25 17:52 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Heikki Linnakangas <hlinnaka@iki.fi>
@ 2024-12-25 18:22 ` Markus KARG <markus@headcrashing.eu>
2024-12-25 18:52 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
0 siblings, 1 reply; 8+ messages in thread
From: Markus KARG @ 2024-12-25 18:22 UTC (permalink / raw)
To: Heikki Linnakangas <hlinnaka@iki.fi>; pgsql-bugs@lists.postgresql.org
> On 25/12/2024 19:05, Markus KARG wrote:
>> I am running the official PostgreSQL 17.2 Docker Container (https://
>> hub.docker.com/layers/library/postgres/17.2/images/sha256-
>> c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6)
>> proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container
>> https://hub.docker.com/layers/library/traefik/v3.2.3/images/
>> sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c).
>> Traefik utilizes ACME with Let's Encrypt to produce the TLS certificate.
>
> In v17, libpq requests the ALPN extension in the TLS handshake. Looks
> like the proxy doesn't know about the "postgresql" ALPN protocol, and
> rejects the connection.
>
> I guess Traefik needs some configuration changes to tell it that the
> "postgresql" protocol is expected. Or code changes.
Traefik does NOT REJECT the connection (if it would, the error message
from psql would be different).
Traefik is "postgres-aware" already since 3.0.0, while I am running 3.2.3.
Note that psql v16.3 works fine but psql v16.4 is not, so a change
introduced by v17 this CANNOT be the cause of the current problem.
>> Using the official PostgresSQL Docker Container (16.3 vs 16.4+), I am
>> asking psql to connect to my server. While psql 16.3 and earlier
>> versions successfully connect via the TLS proxy to the PostgreSQL
>> server, psql 16.4 and later versions fail doing so:
>>
>> root@hetzner-2:~# docker run -it postgres:16.3 psql
>> "host=headcrashing.eu port=5432 dbname=postgres user=postgres
>> password=... sslmode=require"
>> psql (16.3 (Debian 16.3-1.pgdg120+1), server 17.2 (Debian
>> 17.2-1.pgdg120+1))
>> WARNING: psql major version 16, server major version 17.
>> Some psql features might not work.
>> SSL connection (protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256,
>> compression: off)
>> Type"help" for help.
>>
>> postgres=# \q
>> root@hetzner-2:~# docker run -it postgres:16.4 psql
>> "host=headcrashing.eu port=5432 dbname=postgres user=postgres
>> password=... sslmode=require"
>> psql: error: connection to server at"headcrashing.eu" (49.13.53.107),
>> port 5432 failed: SSL error: tlsv1 alert no application protocol
>
> There were no changes between 16.3 and 16.4 to explain this. When I
> test that with v16 client that I built from sources, I don't get that
> error.
>
> The error message suggests that you're actually using libpq v17. And
> indeed I get that error when connecting with v17 client. Perhaps the
> postgres:16.4 docker image was built with v17 libpq?
I am using the original, pre-built container images found on Docker Hub
and have NOT built them on my own. I am not a PostgresSQL committer
either. So I cannot answer your question.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"
2024-12-25 17:05 psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
2024-12-25 17:52 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Heikki Linnakangas <hlinnaka@iki.fi>
2024-12-25 18:22 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
@ 2024-12-25 18:52 ` Markus KARG <markus@headcrashing.eu>
2024-12-25 19:37 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 8+ messages in thread
From: Markus KARG @ 2024-12-25 18:52 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
>> On 25/12/2024 19:05, Markus KARG wrote:
>>> I am running the official PostgreSQL 17.2 Docker Container (https://
>>> hub.docker.com/layers/library/postgres/17.2/images/sha256-
>>> c063081175f45f4a3a5ac03c234e060e67618ebe75b49e2a7ffb79f8357bd1e6)
>>> proxied by a TLSv1.3 proxy (official Traefik 3.2.3 Docker Container
>>> https://hub.docker.com/layers/library/traefik/v3.2.3/images/
>>> sha256-06966a9ba1747ad724a490b8f27df1434c64e8eee5d681df03c4761c9653f62c).
>>> Traefik utilizes ACME with Let's Encrypt to produce the TLS
>>> certificate.
>>
>> In v17, libpq requests the ALPN extension in the TLS handshake. Looks
>> like the proxy doesn't know about the "postgresql" ALPN protocol, and
>> rejects the connection.
>>
>> I guess Traefik needs some configuration changes to tell it that the
>> "postgresql" protocol is expected. Or code changes.
Following this theory, sslnegotiation=postgres used with psql v17.x
should solve the problem, as it falls back to the same mechanism
supported in v16.3, correct? Unfortunately it fails with the same error
message:
root@hetzner-2:~# docker run -it postgres:17.2 psql
"host=headcrashing.eu port=5432 dbname=postgres user=postgres
password=... sslmode=require sslnegotiation=postgres"
psql: error: connection to server at "headcrashing.eu" (49.13.53.107),
port 5432 failed: SSL error: tlsv1 alert no application protocol
The same happens when using sslnegotication=direct.
Regarding your question: I have tried "sslnegotiation=postgres" and
"ssqlnegotiation=direct" with the original official PostgreSQL Docker
Container found on Docker Hub. Using v16.3 psql says it doesn't know the
option "sslnegotiation", and 16.4...17 say "tlsv1 alert no application
protocol". Does PROOFS your claim that the original images on Docker Hub
are using libpq v17 even for psql v16.4+!
-Markus
-Markus
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"
2024-12-25 17:05 psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
2024-12-25 17:52 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Heikki Linnakangas <hlinnaka@iki.fi>
2024-12-25 18:22 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
2024-12-25 18:52 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
@ 2024-12-25 19:37 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-12-25 22:47 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
0 siblings, 1 reply; 8+ messages in thread
From: Tom Lane @ 2024-12-25 19:37 UTC (permalink / raw)
To: Markus KARG <markus@headcrashing.eu>; +Cc: pgsql-bugs@lists.postgresql.org
Markus KARG <markus@headcrashing.eu> writes:
> Following this theory, sslnegotiation=postgres used with psql v17.x
> should solve the problem, as it falls back to the same mechanism
> supported in v16.3, correct?
No, sorry, it looks like we insert the ALPN extension into the SSL
request packet regardless of that. Maybe there should have been
a way to suppress that, but v17 libpq doesn't provide one.
> Regarding your question: I have tried "sslnegotiation=postgres" and
> "ssqlnegotiation=direct" with the original official PostgreSQL Docker
> Container found on Docker Hub.
Let's clarify something here: there is nothing "official" about either
that docker container or anything else you might find on DockerHub.
The Postgres community does not produce any such packaging. You'd
have to discussion the inclusion of v17 libpq with whoever did build
that container. I know that Debian thinks it's a good idea to use
latest libpq with older servers, and this builder might be following
their lead. (The timing would be about right, since v17.0 came out
about the same time as 16.4.) This example does show that there
are pitfalls in that policy.
regards, tom lane
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol"
2024-12-25 17:05 psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
2024-12-25 17:52 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Heikki Linnakangas <hlinnaka@iki.fi>
2024-12-25 18:22 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
2024-12-25 18:52 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
2024-12-25 19:37 ` Re: psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Tom Lane <tgl@sss.pgh.pa.us>
@ 2024-12-25 22:47 ` Markus KARG <markus@headcrashing.eu>
0 siblings, 0 replies; 8+ messages in thread
From: Markus KARG @ 2024-12-25 22:47 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-bugs@lists.postgresql.org
> Markus KARG<markus@headcrashing.eu> writes:
>> Following this theory, sslnegotiation=postgres used with psql v17.x
>> should solve the problem, as it falls back to the same mechanism
>> supported in v16.3, correct?
> No, sorry, it looks like we insert the ALPN extension into the SSL
> request packet regardless of that. Maybe there should have been
> a way to suppress that, but v17 libpq doesn't provide one.
Thank you, Tom, you saved my day! I have explicitly added the name
`postgresql` to the ALPN configuration on Traefik, and immediately psql
v16.4+ (including psql v17) worked like a charm!
As a reference for others, in `traefik-dynamic.yaml` add the follwing to
make it work (as long as you patch the `default` TLS options, take care
to keep all those protocols contained; alternative path a per-router tls
config instead to ONLY have postgresql in that list):
## Dynamic configuration
tls:
options:
default:
alpnProtocols:
- acme-tls/1
- http/1.1
- h2
- postgresql
>> Regarding your question: I have tried "sslnegotiation=postgres" and
>> "ssqlnegotiation=direct" with the original official PostgreSQL Docker
>> Container found on Docker Hub.
> Let's clarify something here: there is nothing "official" about either
> that docker container or anything else you might find on DockerHub.
> The Postgres community does not produce any such packaging. You'd
> have to discussion the inclusion of v17 libpq with whoever did build
> that container. I know that Debian thinks it's a good idea to use
> latest libpq with older servers, and this builder might be following
> their lead. (The timing would be about right, since v17.0 came out
> about the same time as 16.4.) This example does show that there
> are pitfalls in that policy.
Thanks a lot for this really essential information! In fact, the authors
of that official docker image (they really call it that way) call
themselves "The PostgreSQL Docker Community". This is totally
misleading, as one understands that to be "The Docker Community withing
the PostgreSQL Community", which it (apparently) is not. Instead, it
seems that it actually is "The PostgreSQL Community at Docker, Inc.",
people totally unrelated to the "real" PostgreSQL Community!
Thanks a lot, Tom, for your kind help. You provided the best Christmas
present this year! :-)
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2024-12-25 22:47 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-12-25 17:05 psql v16.3 successfully connects via TLSv1.3 proxy, but psql v16.4 says "tlsv1 alert no application protocol" Markus KARG <markus@headcrashing.eu>
2024-12-25 17:39 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-12-25 17:45 ` Markus KARG <markus@headcrashing.eu>
2024-12-25 17:52 ` Heikki Linnakangas <hlinnaka@iki.fi>
2024-12-25 18:22 ` Markus KARG <markus@headcrashing.eu>
2024-12-25 18:52 ` Markus KARG <markus@headcrashing.eu>
2024-12-25 19:37 ` Tom Lane <tgl@sss.pgh.pa.us>
2024-12-25 22:47 ` Markus KARG <markus@headcrashing.eu>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox