Subject: max_parallel_workers question

From: Jeff Davis <[email-masked]>
Date: Fri, 27 Sep 2019 17:07:39 -0700

The current docs for max_parallel_workers start out:

"Sets the maximum number of workers that the system can support for
parallel operations..."

In my interpretation, "the system" means the entire cluster, but the
max_parallel_workers setting is PGC_USERSET. That's a bit confusing,
because two different backends can have different settings for "the
maximum number ... the system can support".

max_parallel_workers is compared against the total number of parallel
workers in the system, which appears to be why the docs are worded that
way. But it's still confusing to me.

If the purpose is to make sure parallel queries don't take up all of
the worker processes, perhaps we should rename the setting
reserved_worker_processes, and make it PGC_SUPERUSER.

If the purpose is to control execution within a backend, perhaps we
should just compare it to the count of parallel processes that the
backend is already using.

If the purpose is just to be a more flexible version of
max_worker_processes, maybe we should change it to PGC_SIGHUP?

If it has multiple purposes, perhaps we should have multiple GUCs?

Regards,
	Jeff Davis

---

From: Robert Haas <[email-masked]>
Date: Sat, 28 Sep 2019 00:10:53 -0400

> maximum number ... the system can support".

Oops.

I intended it to mean "the entire cluster." Basically, how many
workers out of max_worker_processes are you willing to use for
parallel query, as opposed to other things. I agree that PGC_USERSET
doesn't make any sense.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Jeff Davis <[email-masked]>
Date: Sat, 28 Sep 2019 10:36:45 -0700

> parallel query, as opposed to other things. I agree that PGC_USERSET

In that case, PGC_SIGHUP seems most appropriate.

It also might make more sense to rename it to reserved_worker_processes
and invert the meaning. To me, that would be more clear that it's
designed to prevent parallel query from interfering with other uses of
worker processes.

Another option would be to make it two pools, one for parallel workers
and one for everything else, and each one would be controlled by a
PGC_POSTMASTER setting. But it seems like some thought went into trying
to share the pool of workers[1], so I assume there was a good reason
you wanted to do that.

Regards,
	Jeff Davis

[1] If I'm reading correctly, it uses both lock-free code and
intentional overflow.

---

From: Robert Haas <[email-masked]>
Date: Fri, 4 Oct 2019 08:33:30 -0400

> In that case, PGC_SIGHUP seems most appropriate.

Yeah.

> designed to prevent parallel query from interfering with other uses of

I don't think that would work as well.  Some day we might have another
class of worker processes with its own independent limit, and then
this terminology would get confusing.  It makes sense to say that you
can have up to 10 worker processes of which at most 4 can be used for
parallel query and at most 3 can be used for logical replication, but
it doesn't make nearly as much sense to say that you can have up to 10
worker processes of which 6 can't be used for parallel query and of
which 7 can't be used for logical application.  That leaves, uh, how
many?

> to share the pool of workers[1], so I assume there was a good reason

Here again, I imagine that in the future we might have various
different worker classes that need to share the total number of
workers, but not necessarily via a hard partition. For example, you
could sensible say that there are 3 purposes for workers and 10
workers, and no single purpose can consume more than 4 workers. Even
though 4 * 3 > 10, it's a completely reasonable configuration. The
early bird gets the juiciest worm, and the late bird doesn't starve to
death.  Even a more extreme configuration where you limit each purpose
to, say, 7 workers could be reasonable. Here there is a risk of
starvation, but you may know that in your environment it's not likely
to last for very long.