Subject: Posgresql 14 and CarbonBlack on RHEL8?

From: Ron Johnson <[email-masked]>
Date: Tue, 30 Apr 2024 21:21:14 -0400

(CarbonBlack is cross-platform AV software sold by VMware.)

Currently we're running PG 9.6.24 on RHEL 6.10 with CB (version unknown to
me) in production, and testing PG 14.11 on RHEL 8.9 with CB 2.15.2
(hopefully going into production next month).

Both old and new VMs are 32 CPU with 128GB RAM.
Nothing but PG, CB and itsm software runs on these systems.

When running stress tests on the systems (in prod, during the maintenance
window), 171K events/second are generated on the RHEL8 servers, and CB
needs (according to top(1)) 325% of CPU to handle that, and still dropping
92% of them.
The RHEL6 system doesn't bat an eye at running the exact same test (36 cron
jobs running psql executing SELECT statements).

The small RHEL8/PG14 non-prod systems show similar load when lots of SELECT
statements run.

Has anyone else seen this?  If so, how did you resolve it?

---

From: Tom Lane <[email-masked]>
Date: Tue, 30 Apr 2024 22:07:44 -0400

> jobs running psql executing SELECT statements).

Is JIT enabled on the newer system?  If so try turning it off, or else
raise the associated cost settings.  We've seen lots of reports of
workloads where, by default, the planner is too aggressive about
applying JIT.

			regards, tom lane

---

From: Ron Johnson <[email-masked]>
Date: Wed, 1 May 2024 04:44:16 -0400

> workloads where, by default, the planner is too aggressive about

A puzzling suggestion.  Why should it impact AV software?

At one point, I disabled JIT to test its impact on PG, performance was a
bit of a wash (some queries were a bit faster, some were a bit slower), but
I didn't monitor CB.

Just now, I did ALTER SYSTEM SET jit='off'; and re-ran the stress test.  No
impact to CarbonBlack.