Subject: POC: Carefully exposing information without authentication

From: Greg Sabino Mullane <[email-masked]>
Date: Thu, 29 May 2025 10:32:57 -0400

Proposal: Allow a carefully curated selection of information to be shown
without authentication.

A common task for an HA system or a load balancer is to quickly determine
which of your Postgres clusters is the primary, and which are the replicas.
The canonical way to do this is to log in to each server with a valid
username and password, and then run pg_is_in_recovery().
That's a lot of work to determine if a server is a replica or not, and it
struck me that this true/false information about a running cluster is not
super-sensitive information. In other words, would it really be wrong if
there was a way to advertise that information without having to log in? I
toyed with the idea of Postgres maintaining some sort of signal file, but
then I realized that we already have a process, listening on a known port,
that has that information available to us.

Thus, this POC (proof of concept), which lets the postmaster scan for
incoming requests and quickly handle them *before* doing forking and
authenticating. We scan for a simple trigger string, and immediately return
the information to the client.

It also occured to me that since we are going to need to provide a
non-Postgres-protocol special trigger string, and we might as well do
something like "GET /info" to allow existing programs to treat Postgres as
a mini http server. To that end, we end
up with something like this:

$ psql -p 5432 -tc 'select pg_is_in_recovery()'
t

$ curl http://localhost:5432/foobar
curl: (52) Empty reply from server

$ curl http://localhost:5432/info
RECOVERY: 1

To accomplish this, we have boolean GUC flags (defaulting to false) that
control which information is exposed. For the example above, the
expose_recovery boolean has been set to true. If any of these GUCs are
true, we take a slight detour right after we accept() but before we
actually fork. We use recv with the MSG_PEEK flag to take a quick scan of
the incoming data, and use strncmp to see if it matches. If it does, we
send() some information and move on without forking. If it doesn't, we
simply move on as if we were never there, and proceed to the next step of
forking a new backend to start the authentication process.

There are three pieces of information that can be exposed with this patch.
There may be more in the future, but these are all simple, global, and not
(IMO) security leaks. The GUCs are expose_recovery, expose_sysid, and
expose_version. Each one adds a line to the output in a KEY: VALUE format
for the GET /info endpoint. The raw value is output for the direct
endpoints:

* GET /replica
* GET /sysid
* GET /version

The expose_recovery GUC uses RecoveryInProgress() to return a 1 or a 0.
This is returned by the GET /replica endpoint.

The expose_sysid GUC returns GetSystemIdentifier(). Since this can be
thought of as a fingerprint for the server, it's a nice way for external
programs to determine if the cluster is the same one it saw last time, or
for leader/replica matching.

The purpose of the expose_version GUC is to output PG_VERSION_NUM. This
will allow external tools - particularly security scanners - to know the
exact version of Postgres that is running. While some may consider this
privileged information, tools are already taking advantage of our debug
loophole to make an educated guess about the version. See my old post about
this:
https://www.endpointdev.com/blog/2010/05/finding-postgresql-version-without/
Note that this guess by security scanners is sometimes wrong, or only able
to cover a range of versions. Thus, we should give them the correct answer,
rather than providing a dubious one via some trickery.

Here's some example output with all three enabled:

$ psql -c 'alter system set expose_recovery=on'
$ psql -c 'alter system set expose_version=on'
$ psql -c 'alter system set expose_sysid=on'
$ psql -c 'select pg_reload_conf()'

$ curl http://localhost:5432/version
180000

$ curl http://localhost:5432/info
RECOVERY: 1
SYSID: 7504513530771111839
VERSION: 180000

But wait! We can do more. For the recovery, we don't even need a string
that spells out "RECOVERY:", we only need to know if the server is in
recovery or not. In short, a boolean. Patroni does this in its API with a
call of HEAD /replica. It returns a different HTTP code if the server is a
replica (200) or not a replica (503). We can do the same thing! What's
more, we can do it in a way that will allow existing calls to simply point
to the postgres server instead of a Patroni process, and get the same
result back, but faster. Here's an example of what that looks like:

## Calling Patroni
$ curl -s -w "%{http_code}" -o /dev/null -I http://localhost:8008/replica
200

## Calling Postgres directly
$ curl -s -w "%{http_code}" -o /dev/null -I http://localhost:5432/replica
200

Here's a simple Python program showing how easy it is to grab this
information:

import socket

try:
  with socket.create_connection(('localhost', 5432), timeout=1) as s:
    s.sendall(b'GET /sysid')
    print(s.recv(200).split(b'\r\n\r\n',1)[1].decode())
except Exception as e:
  print(f"Error: {e}")

That's the basic idea: proof of concept patch is attached. Additional
things to do:

* handling socket quirks (esp. Win32)
* docs (once details are hashed out)
* moving things around (everything is in one function right now for reading
ease)

Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support

---

From: Antonin Houska <[email-masked]>
Date: Fri, 30 May 2025 17:02:11 +0200

Greg Sabino Mullane <[email-masked]> wrote:

> Proposal: Allow a carefully curated selection of information to be shown without authentication.
> 
> A common task for an HA system or a load balancer is to quickly determine which of your Postgres clusters is the primary, and which are the
> replicas. The canonical way to do this is to log in to each server with a valid username and password, and then run pg_is_in_recovery().
> That's a lot of work to determine if a server is a replica or not, and it struck me that this true/false information about a running cluster is not
> super-sensitive information. In other words, would it really be wrong if there was a way to advertise that information without having to log in?
> I toyed with the idea of Postgres maintaining some sort of signal file, but then I realized that we already have a process, listening on a known
> port, that has that information available to us.
> 
> Thus, this POC (proof of concept), which lets the postmaster scan for incoming requests and quickly handle them *before* doing forking and
> authenticating. We scan for a simple trigger string, and immediately return the information to the client.

Why is it important not to fork?  My understanding is that pg_is_ready also
tries to start a regular connection, i.e. forks a new backend. I think this
functionality would fit into libpq. (I've got no strong opinion on the amount
of information to be revealed this way. In any case, a GUC to enable the
feature only if the DBA wants it makes sense.)

-- 
Antonin Houska
Web: https://www.cybertec-postgresql.com

---

From: Greg Sabino Mullane <[email-masked]>
Date: Fri, 30 May 2025 20:13:45 -0400

On Fri, May 30, 2025 at 11:02 AM Antonin Houska <[email-masked]> wrote:

> Why is it important not to fork?

Good question. Forking is expensive, and there is also a lot of
housekeeping associated with it that is simply not needed here. We want
this to be lightweight, and simple. No need to fork if we are just going to
do a few strncmp() calls and a send(). However, I'm not highly opposed to
fork-first, as I understand that we want to not slow down postmaster. My
testing showed a barely measurable impact, but I will defer to whatever
decision the elder Postgres gods decide on.

> My understanding is that pg_is_ready also tries to start a regular
> connection, i.e. forks a new backend.

Yep. I consider pg_isready a spiritual cousin to this feature, but it's not
something that can really do what this does.

Cheers,
Greg

---

From: Tom Lane <[email-masked]>
Date: Fri, 30 May 2025 21:34:47 -0400

Greg Sabino Mullane <[email-masked]> writes:
> do a few strncmp() calls and a send().

send() can block.  I think calling it in the postmaster is a
nonstarter.  For comparison, we make an effort to not do any
communication with incoming clients until after forking a child
to do the communication.  The one exception is if we have to
report fork failure --- but we don't make any strong guarantees
about that report succeeding.  (IIRC, we put the port into nonblock
mode and try only once.)  That's probably not a behavior you want
to adopt for non-edge-case usages.

Another point is that you'll recall that there's a lot of
interest in switching to a threaded model.  The argument that
"fork is too expensive" may not have a long shelf life.

I'm not taking a position on whether $SUBJECT is a good idea
in the first place.

			regards, tom lane

---

From: Greg Sabino Mullane <[email-masked]>
Date: Sat, 31 May 2025 00:48:13 -0400

On Fri, May 30, 2025 at 9:34 PM Tom Lane <[email-masked]> wrote:

> I think calling it in the postmaster is a nonstarter.

Thanks for the feedback. Please find attached version two, which moves the
code to the very start of BackendInitialize in
tcop/backend_startup.c. If we handle the request, we simply proc_exit and
avoid all the other backend startup stuff. So still a big win. I also made
a first rough pass at the documentation.

Cheers,
Greg

---

From: Greg Sabino Mullane <[email-masked]>
Date: Thu, 2 Oct 2025 22:23:22 -0400

Please find attached version 3, rebased for PG 19 and now featuring some
tests.

Cheers,
Greg

---

From: Greg Sabino Mullane <[email-masked]>
Date: Thu, 23 Oct 2025 05:57:19 -0400

Version 4 attached, rebased to account for new tests, plus a new
instra-test check to make sure LWP::UserAgent is available before running.

Cheers,
Greg

---

From: Greg Sabino Mullane <[email-masked]>
Date: Tue, 24 Mar 2026 21:59:19 -0400

Thank you for looking over this. New version attached.

On Tue, Feb 17, 2026 at 2:58 PM Andres Freund <[email-masked]> wrote:

> What about direct TLS connections?

Not handled.

How can a cluster coordinator trust unauthenticated plain text
> communication that can just be man-in-the-middled?
>

They cannot. But that's why this is only exposing non-critical information.
Right now the security scanners that are banging on port 5432 and scraping
the returned error lines are not worried about man-in-the-middle. :)
Obviously, if your threat model is people capturing and modifying
non-encrypted traffic to your Postgres server, you would not use this.

It's not obvious that it's a good idea to expose this on the same socket as
> normal client connections.  IMO you'd want to limit this to a smaller set
> of interfaces than normal client connections.
>

I'm not entirely clear what that smaller set would mean in practice.

> IIRC the socket is in blocking mode at this point (that's only changed in
> pq_init()), therefore this might actually block?  While it's unlikely, I
> don't see any guarantee that a single receive would actually get the whole
> message from the client either, so this seems like it might fail spuriously.
>

Yes, there are some very unlikely edge cases, but this is meant to be good
enough, not a perfectly bulletproof HTTP server. Clients should try again
on failures. Which if they do occur for this trivial amount of traffic
probably indicates much bigger problems.

If we were to do this, I'd recommend a single expose GUC that has the
> different values as a comma separated list, instead a growing list of GUCs.
>

Done - see attached for a new version which consolidates the bools into a
single comma-separated GUC called "expose_information". I also added some
docs, and changed the "replica" to return "REPLICA" instead of "RECOVERY".
I like the latter better, but replica lines up better with existing tools.

-- 
Cheers,
Greg