pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Improve heuristics for compressing the KnownAssignedXids array. 6+ messages / 1 participants [nested] [flat]
* pgsql: Improve heuristics for compressing the KnownAssignedXids array. @ 2022-11-29 20:43 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 6+ messages in thread From: Tom Lane @ 2022-11-29 20:43 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Improve heuristics for compressing the KnownAssignedXids array. Previously, we'd compress only when the active range of array entries reached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids). If max_connections is large, the first term could result in not compressing for a long time, resulting in much wastage of cycles in hot-standby backends scanning the array to take snapshots. Get rid of that term, and just bound it to 2 * pArray->numKnownAssignedXids. That however creates the opposite risk, that we might spend too much effort compressing. Hence, consider compressing only once every 128 commit records. (This frequency was chosen by benchmarking. While we only tried one benchmark scenario, the results seem stable over a fairly wide range of frequencies.) Also, force compression when processing RecoveryInfo WAL records (which should be infrequent); the old code could perform compression then, but would do so only after the same array-range check as for the transaction-commit path. Also, opportunistically run compression if the startup process is about to wait for WAL, though not oftener than once a second. This should prevent cases where we waste lots of time by leaving the array not-compressed for long intervals due to low WAL traffic. Lastly, add a simple check to keep us from uselessly compressing when the array storage is already compact. Back-patch, as the performance problem is worse in pre-v14 branches than in HEAD. Simon Riggs and Michail Nikolaev, with help from Tom Lane and Andres Freund. Discussion: https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/8242752f9c104030085cb167e6e1dd5bed481360 Modified Files -------------- src/backend/access/transam/xlogrecovery.c | 6 ++ src/backend/storage/ipc/procarray.c | 133 +++++++++++++++++++++++------- src/include/storage/procarray.h | 1 + 3 files changed, 110 insertions(+), 30 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Improve heuristics for compressing the KnownAssignedXids array. @ 2022-11-29 20:43 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 6+ messages in thread From: Tom Lane @ 2022-11-29 20:43 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Improve heuristics for compressing the KnownAssignedXids array. Previously, we'd compress only when the active range of array entries reached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids). If max_connections is large, the first term could result in not compressing for a long time, resulting in much wastage of cycles in hot-standby backends scanning the array to take snapshots. Get rid of that term, and just bound it to 2 * pArray->numKnownAssignedXids. That however creates the opposite risk, that we might spend too much effort compressing. Hence, consider compressing only once every 128 commit records. (This frequency was chosen by benchmarking. While we only tried one benchmark scenario, the results seem stable over a fairly wide range of frequencies.) Also, force compression when processing RecoveryInfo WAL records (which should be infrequent); the old code could perform compression then, but would do so only after the same array-range check as for the transaction-commit path. Also, opportunistically run compression if the startup process is about to wait for WAL, though not oftener than once a second. This should prevent cases where we waste lots of time by leaving the array not-compressed for long intervals due to low WAL traffic. Lastly, add a simple check to keep us from uselessly compressing when the array storage is already compact. Back-patch, as the performance problem is worse in pre-v14 branches than in HEAD. Simon Riggs and Michail Nikolaev, with help from Tom Lane and Andres Freund. Discussion: https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/55fa993d7eacb3f3ef99015649d44606201767e5 Modified Files -------------- src/backend/access/transam/xlogrecovery.c | 6 ++ src/backend/storage/ipc/procarray.c | 133 +++++++++++++++++++++++------- src/include/storage/procarray.h | 1 + 3 files changed, 110 insertions(+), 30 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Improve heuristics for compressing the KnownAssignedXids array. @ 2022-11-29 20:43 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 6+ messages in thread From: Tom Lane @ 2022-11-29 20:43 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Improve heuristics for compressing the KnownAssignedXids array. Previously, we'd compress only when the active range of array entries reached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids). If max_connections is large, the first term could result in not compressing for a long time, resulting in much wastage of cycles in hot-standby backends scanning the array to take snapshots. Get rid of that term, and just bound it to 2 * pArray->numKnownAssignedXids. That however creates the opposite risk, that we might spend too much effort compressing. Hence, consider compressing only once every 128 commit records. (This frequency was chosen by benchmarking. While we only tried one benchmark scenario, the results seem stable over a fairly wide range of frequencies.) Also, force compression when processing RecoveryInfo WAL records (which should be infrequent); the old code could perform compression then, but would do so only after the same array-range check as for the transaction-commit path. Also, opportunistically run compression if the startup process is about to wait for WAL, though not oftener than once a second. This should prevent cases where we waste lots of time by leaving the array not-compressed for long intervals due to low WAL traffic. Lastly, add a simple check to keep us from uselessly compressing when the array storage is already compact. Back-patch, as the performance problem is worse in pre-v14 branches than in HEAD. Simon Riggs and Michail Nikolaev, with help from Tom Lane and Andres Freund. Discussion: https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/06998eab16b2165b5f27014a0843b75e377cc63e Modified Files -------------- src/backend/access/transam/xlog.c | 6 ++ src/backend/storage/ipc/procarray.c | 133 ++++++++++++++++++++++++++++-------- src/include/storage/procarray.h | 1 + 3 files changed, 110 insertions(+), 30 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Improve heuristics for compressing the KnownAssignedXids array. @ 2022-11-29 20:43 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 6+ messages in thread From: Tom Lane @ 2022-11-29 20:43 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Improve heuristics for compressing the KnownAssignedXids array. Previously, we'd compress only when the active range of array entries reached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids). If max_connections is large, the first term could result in not compressing for a long time, resulting in much wastage of cycles in hot-standby backends scanning the array to take snapshots. Get rid of that term, and just bound it to 2 * pArray->numKnownAssignedXids. That however creates the opposite risk, that we might spend too much effort compressing. Hence, consider compressing only once every 128 commit records. (This frequency was chosen by benchmarking. While we only tried one benchmark scenario, the results seem stable over a fairly wide range of frequencies.) Also, force compression when processing RecoveryInfo WAL records (which should be infrequent); the old code could perform compression then, but would do so only after the same array-range check as for the transaction-commit path. Also, opportunistically run compression if the startup process is about to wait for WAL, though not oftener than once a second. This should prevent cases where we waste lots of time by leaving the array not-compressed for long intervals due to low WAL traffic. Lastly, add a simple check to keep us from uselessly compressing when the array storage is already compact. Back-patch, as the performance problem is worse in pre-v14 branches than in HEAD. Simon Riggs and Michail Nikolaev, with help from Tom Lane and Andres Freund. Discussion: https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/6e8ad11521cccda7dad29f3e6750eb0b9be46527 Modified Files -------------- src/backend/access/transam/xlog.c | 6 ++ src/backend/storage/ipc/procarray.c | 133 ++++++++++++++++++++++++++++-------- src/include/storage/procarray.h | 1 + 3 files changed, 110 insertions(+), 30 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Improve heuristics for compressing the KnownAssignedXids array. @ 2022-11-29 20:43 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 6+ messages in thread From: Tom Lane @ 2022-11-29 20:43 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Improve heuristics for compressing the KnownAssignedXids array. Previously, we'd compress only when the active range of array entries reached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids). If max_connections is large, the first term could result in not compressing for a long time, resulting in much wastage of cycles in hot-standby backends scanning the array to take snapshots. Get rid of that term, and just bound it to 2 * pArray->numKnownAssignedXids. That however creates the opposite risk, that we might spend too much effort compressing. Hence, consider compressing only once every 128 commit records. (This frequency was chosen by benchmarking. While we only tried one benchmark scenario, the results seem stable over a fairly wide range of frequencies.) Also, force compression when processing RecoveryInfo WAL records (which should be infrequent); the old code could perform compression then, but would do so only after the same array-range check as for the transaction-commit path. Also, opportunistically run compression if the startup process is about to wait for WAL, though not oftener than once a second. This should prevent cases where we waste lots of time by leaving the array not-compressed for long intervals due to low WAL traffic. Lastly, add a simple check to keep us from uselessly compressing when the array storage is already compact. Back-patch, as the performance problem is worse in pre-v14 branches than in HEAD. Simon Riggs and Michail Nikolaev, with help from Tom Lane and Andres Freund. Discussion: https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/c4a153d778318afbc0d2cd92067f49b62673444f Modified Files -------------- src/backend/access/transam/xlog.c | 6 ++ src/backend/storage/ipc/procarray.c | 133 ++++++++++++++++++++++++++++-------- src/include/storage/procarray.h | 1 + 3 files changed, 110 insertions(+), 30 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgsql: Improve heuristics for compressing the KnownAssignedXids array. @ 2022-11-29 20:43 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 6+ messages in thread From: Tom Lane @ 2022-11-29 20:43 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Improve heuristics for compressing the KnownAssignedXids array. Previously, we'd compress only when the active range of array entries reached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids). If max_connections is large, the first term could result in not compressing for a long time, resulting in much wastage of cycles in hot-standby backends scanning the array to take snapshots. Get rid of that term, and just bound it to 2 * pArray->numKnownAssignedXids. That however creates the opposite risk, that we might spend too much effort compressing. Hence, consider compressing only once every 128 commit records. (This frequency was chosen by benchmarking. While we only tried one benchmark scenario, the results seem stable over a fairly wide range of frequencies.) Also, force compression when processing RecoveryInfo WAL records (which should be infrequent); the old code could perform compression then, but would do so only after the same array-range check as for the transaction-commit path. Also, opportunistically run compression if the startup process is about to wait for WAL, though not oftener than once a second. This should prevent cases where we waste lots of time by leaving the array not-compressed for long intervals due to low WAL traffic. Lastly, add a simple check to keep us from uselessly compressing when the array storage is already compact. Back-patch, as the performance problem is worse in pre-v14 branches than in HEAD. Simon Riggs and Michail Nikolaev, with help from Tom Lane and Andres Freund. Discussion: https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com Branch ------ REL_11_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/a6c9e1db2b6673100138f5b9f1ba31d55de7af9a Modified Files -------------- src/backend/access/transam/xlog.c | 6 ++ src/backend/storage/ipc/procarray.c | 133 ++++++++++++++++++++++++++++-------- src/include/storage/procarray.h | 1 + 3 files changed, 110 insertions(+), 30 deletions(-) ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2022-11-29 20:43 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-11-29 20:43 pgsql: Improve heuristics for compressing the KnownAssignedXids array. Tom Lane <tgl@sss.pgh.pa.us> 2022-11-29 20:43 pgsql: Improve heuristics for compressing the KnownAssignedXids array. Tom Lane <tgl@sss.pgh.pa.us> 2022-11-29 20:43 pgsql: Improve heuristics for compressing the KnownAssignedXids array. Tom Lane <tgl@sss.pgh.pa.us> 2022-11-29 20:43 pgsql: Improve heuristics for compressing the KnownAssignedXids array. Tom Lane <tgl@sss.pgh.pa.us> 2022-11-29 20:43 pgsql: Improve heuristics for compressing the KnownAssignedXids array. Tom Lane <tgl@sss.pgh.pa.us> 2022-11-29 20:43 pgsql: Improve heuristics for compressing the KnownAssignedXids array. Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox