Subject: Possibility to disable `ALTER SYSTEM`

From: Gabriele Bartolini <[email-masked]>
Date: Thu, 7 Sep 2023 21:51:14 +0200

Hi everyone,

I would like to propose a patch that allows administrators to disable
`ALTER SYSTEM` via either a runt-time option to pass to the Postgres server
process at startup (e.g. `--disable-alter-system=true`, false by default)
or a new GUC (or even both), without changing the current default method of
the server.

The main reason is that this would help improve the “security by default”
posture of Postgres in a Kubernetes/Cloud Native environment - and, in
general, in any environment on VMs/bare metal behind a configuration
management system in which changes should only be made in a declarative way
and versioned like Ansible Tower, to cite one.

Below you find some background information and the longer story behind this
proposal.

Sticking to the Kubernetes use case, I am primarily speaking on behalf of
the CloudNativePG open source operator (cloudnative-pg.io, of which I am
one of the maintainers). However, I am sure that this option could benefit
any operator for Postgres - an operator is the most common and recommended
way to run a complex application like a PostgreSQL database management
system inside Kubernetes.

In this case, the state of a PostgreSQL cluster (for example its number of
replicas, configuration, storage, etc.) is defined in a Custom Resource
Definition in the form of configuration, typically YAML, and the operator
works with Kubernetes to ensure that, at any moment, the requested Postgres
cluster matches the observed one. This is a very basic example in
CloudNativePG:
https://cloudnative-pg.io/documentation/current/samples/cluster-example.yaml

As I was mentioning above, in a Cloud Native environment it is expected
that workloads are secure by default. Without going into much detail, many
decisions have been made in that direction by operators for Postgres,
including CloudNativePG. The goal of this proposal is to provide a way to
ensure that changes to the PostgreSQL configuration in a Kubernetes
controlled Postgres cluster are allowed only through the Kubernetes API.

Basically, if you want to change an option for PostgreSQL, you need to
change the desired state in the Kubernetes resource, then Kubernetes will
converge (through the operator). In simple words, it’s like empowering the
operator to impersonate the PostgreSQL superuser.

However, given that we cannot force this use case, there could be roles
with the login+superuser privileges connecting to the PostgreSQL instance
and potentially “interfering” with the requested state defined in the
configuration by imperatively running “ALTER SYSTEM” commands.

For example: CloudNativePG has a fixed value for some GUCs in order to
manage a full HA cluster, including SSL, log, some WAL and replication
settings. While the operator eventually reconciles those settings, even the
temporary change of those settings in a cluster might be harmful. Think for
example of a user that, through `ALTER SYSTEM`, tries to change WAL level
to minimal, or change the setting of the log (we require CSV), potentially
creating issues to the underlying instance and cluster (potentially leaving
it in an unrecoverable state in the case of other more invasive GUCS).

At the moment, a possible workaround is that `ALTER SYSTEM` can be blocked
by making the postgresql.auto.conf read only, but the returned message is
misleading and that’s certainly bad user experience (which is very
important in a cloud native environment):

```
postgres=# ALTER SYSTEM SET wal_level TO minimal;
ERROR:  could not open file "postgresql.auto.conf": Permission denied
```

For this reason, I would like to propose the option to be given to the
postgres process at startup, in order to be as less invasive as possible
(the operator could then start Postgres requesting `ALTER SYSTEM` to be
disabled). That’d be my preference at the moment, if possible.

Alternatively, or in addition, the introduction of a GUC to disable `ALTER
SYSTEM` altogether. This enables tuning this setting through configuration
at the Kubernetes level, only if the operators require it - without
damaging the rest of the users.

Before I start writing any lines of code and propose a patch, I would like
first to understand if there’s room for it.

Thanks for your attention and … looking forward to your feedback!

Ciao,
Gabriele
--
Gabriele Bartolini
Vice President, Cloud Native at EDB
enterprisedb.com

---

From: Joe Conway <[email-masked]>
Date: Thu, 7 Sep 2023 15:57:22 -0400

On 9/7/23 15:51, Gabriele Bartolini wrote:
> `ALTER SYSTEM` via either a runt-time option to pass to the Postgres 
> server process at startup (e.g. `--disable-alter-system=true`, false by 
> default) or a new GUC (or even both), without changing the current 
> default method of the server.

Without trying to debate the particulars, a huge +1 for the concept of 
allowing ALTER SYSTEM to be disabled. FWIW I would vote the same for 
control over COPY PROGRAM.

Not coincidentally both concepts were built into set_user: 
https://github.com/pgaudit/set_user

-- 
Joe Conway
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

---

From: Tom Lane <[email-masked]>
Date: Thu, 07 Sep 2023 16:27:14 -0400

> or a new GUC (or even both), without changing the current default method of

ALTER SYSTEM is already heavily restricted.  I don't think we need random
kluges added to the permissions system.  I especially don't believe in
kluges to the effect of "superuser doesn't have all permissions anymore".

If you nonetheless feel that that's a good idea for your use case,
you can implement the restriction with an event trigger or the like.

			regards, tom lane

---

From: Gabriele Bartolini <[email-masked]>
Date: Fri, 8 Sep 2023 13:31:16 +0200

Hi Tom,

On Thu, 7 Sept 2023 at 22:27, Tom Lane <[email-masked]> wrote:

> server
> ALTER SYSTEM is already heavily restricted.

Could you please help me better understand what you mean here?

> I don't think we need random kluges added to the permissions system.

If you allow me, why do you think disabling ALTER SYSTEM altogether is a
random kluge? Again, I'd like to better understand this position. I've
personally been in many conversations on the security side of things for
Postgres in Kubernetes environments, and this is a frequent concern by
users who request that changes to the Postgres system (not a database)
should only be done declaratively and prevented from within the system.

Thanks,
Gabriele

---

From: Isaac Morland <[email-masked]>
Date: Fri, 8 Sep 2023 10:11:30 -0400

> should only be done declaratively and prevented from within the system.

Alternate idea, not sure how good this is: Use existing OS security
features (regular permissions, or more modern features such as the
immutable attribute) to mark the postgresql.auto.conf file as not being
writeable. Then any attempt to ALTER SYSTEM should result in an error.

---

From: Álvaro Hernández <[email-masked]>
Date: Sat, 9 Sep 2023 01:24:02 +0200

On 7/9/23 21:51, Gabriele Bartolini wrote:
> server process at startup (e.g. `--disable-alter-system=true`, false 
> by default) or a new GUC (or even both), without changing the current 
> The main reason is that this would help improve the “security by 
> default” posture of Postgres in a Kubernetes/Cloud Native environment 
> - and, in general, in any environment on VMs/bare metal behind a 
> configuration management system in which changes should only be made 
> in a declarative way and versioned like Ansible Tower, to cite one.
>
> Below you find some background information and the longer story behind 
> this proposal.
>
> Sticking to the Kubernetes use case, I am primarily speaking on behalf 
> of the CloudNativePG open source operator (cloudnative-pg.io 
> <http://cloudnative-pg.io>, of which I am one of the maintainers). 
> However, I am sure that this option could benefit any operator for 
> Postgres - an operator is the most common and recommended way to run a 
> complex application like a PostgreSQL database management system 
> inside Kubernetes.
>
> In this case, the state of a PostgreSQL cluster (for example its 
> number of replicas, configuration, storage, etc.) is defined in a 
> Custom Resource Definition in the form of configuration, typically 
> YAML, and the operator works with Kubernetes to ensure that, at any 
> moment, the requested Postgres cluster matches the observed one. This 
> is a very basic example in CloudNativePG: 
> As I was mentioning above, in a Cloud Native environment it is 
> expected that workloads are secure by default. Without going into much 
> detail, many decisions have been made in that direction by operators 
> for Postgres, including CloudNativePG. The goal of this proposal is to 
> provide a way to ensure that changes to the PostgreSQL configuration 
> in a Kubernetes controlled Postgres cluster are allowed only through 
> the Kubernetes API.
>
> change the desired state in the Kubernetes resource, then Kubernetes 
> will converge (through the operator). In simple words, it’s like 
> empowering the operator to impersonate the PostgreSQL superuser.
>

     Coming from a similar background to Gabriele's, I support this 
proposal.

     In StackGres (https://stackgres.io) we also allow users to manage 
postgresql.conf's configuration declaratively. We have a CRD (Custom 
Resource Definition) that precisely defines and controls how a 
postgresql.conf configuration looks like (see 
https://stackgres.io/doc/latest/reference/crd/sgpgconfig/). This 
configuration, once created by the user, is strongly validated by 
StackGres (parameters are valid for the given major version, values are 
within the ranges and appropriate types) and then periodically applied 
to the database if there's any drift between that user-declared 
(desired) state and current system status.

     Similarly, we also have some parameters which the user is not 
allowed to change 
(https://gitlab.com/ongresinc/stackgres/-/blob/main/stackgres-k8s/src/operator/src/main/resources/postgresql-blocklist.properties). 
If the user is allowed to use ALTER SYSTEM and modify some of these 
parameters, significant breakage can happen in the cluster, as the 
operator may become "confused" and manual operation may be required, 
breaking many of the user's expectations of stability and how the system 
works and heals automatically.

     Sure, as mentioned elsewhere in the thread, a "malicious" user can 
still use other mechanisms such as COPY or untrusted PLs to still 
overwrite the configuration. But both attempts are obviously conscious 
attempts to break the system (and if so, it's all yours to break it). 
But ALTER SYSTEM may be an *unintended* way to break it, causing a bad 
user's experience. This may be defined more of a way to avoid users 
shooting themselves in the feet, inadvertedly.

     There's apparently some opposition to implementing this. But given 
that there's also interest in having it, I'd like to know what the 
negative effects of implementing such a startup configuration property 
would be, so that advantages can be compared with the disadvantages.

     All that being said, the behavior to prevent ALTER SYSTEM can also 
be easily implemented as an extension. Actually some colleagues wrote 
one with a similar scope 
(https://gitlab.com/ongresinc/extensions/noset), and I believe it could 
be the base for a similar extension focused on preventing ALTER SYSTEM.

     Regards,

     Álvaro

-- 

Alvaro Hernandez

-----------
OnGres

---

From: Gabriele Bartolini <[email-masked]>
Date: Tue, 30 Jan 2024 18:05:37 +0100

Hi,

I am sending an updated patch, and submitting this to the next commit fest,
as I still believe this could be very useful.

Thanks,
Gabriele