public inbox for [email protected]help / color / mirror / Atom feed
Re: [HACKERS] Custom compression methods 14+ messages / 3 participants [nested] [flat]
* Re: [HACKERS] Custom compression methods @ 2018-06-19 14:57 Robert Haas <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Robert Haas @ 2018-06-19 14:57 UTC (permalink / raw) To: Konstantin Knizhnik <[email protected]>; +Cc: Alexander Korotkov <[email protected]>; Ildus Kurbangaliev <[email protected]>; Ildar Musin <[email protected]>; Tomas Vondra <[email protected]>; Alvaro Herrera <[email protected]>; Евгений Шишкин <[email protected]>; Andres Freund <[email protected]>; Oleg Bartunov <[email protected]>; Craig Ringer <[email protected]>; Peter Eisentraut <[email protected]>; pgsql-hackers; Chapman Flack <[email protected]> On Mon, Apr 23, 2018 at 12:34 PM, Konstantin Knizhnik <[email protected]> wrote: > May be. But in any cases, there are several direction where compression can > be used: > - custom compression algorithms > - libpq compression > - page level compression > ... > > and them should be somehow finally "married" with each other. I agree that we should try to avoid multiplying the number of compression-related APIs. Ideally there should be one API for registering a compression algorithms, and then there can be different methods of selecting that compression algorithm depending on the purpose for which it will be used. For instance, you could select a column compression format using some variant of ALTER TABLE ... ALTER COLUMN, but you would obviously use some other method to select the WAL compression format. However, it's a little unclear to me how we would actually make the idea of a single API work. For column compression, we need everything to be accessible through the catalogs. For something like WAL compression, we need it to be completely independent of the catalogs. Those things are opposites, so a single API can't have both properties. Maybe there can be some pieces shared, but as much as I'd like it to be otherwise, it doesn't seem possible to share it completely. I also agree with Ildus and Alexander that we cannot and should not try to solve every problem in one patch. Rather, we should just think ahead, so that we make as much of what goes into this patch reusable in the future as we can. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: New strategies for freezing, advancing relfrozenxid early @ 2023-01-05 18:19 Matthias van de Meent <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Matthias van de Meent @ 2023-01-05 18:19 UTC (permalink / raw) To: Peter Geoghegan <[email protected]>; +Cc: Jeff Davis <[email protected]>; John Naylor <[email protected]>; Andres Freund <[email protected]>; Justin Pryzby <[email protected]>; PostgreSQL Hackers <[email protected]> On Thu, 5 Jan 2023 at 02:21, I wrote: > > On Tue, 3 Jan 2023 at 21:30, Peter Geoghegan <[email protected]> wrote: > > > > Attached is v14. > > [PATCH v14 3/3] Finish removing aggressive mode VACUUM. > > I've not completed a review for this patch - I'll continue on that > tomorrow: This is that. > @@ -2152,10 +2109,98 @@ lazy_scan_noprune(LVRelState *vacrel, > [...] > + /* wait 10ms, then 20ms, then 30ms, then give up */ > [...] > + pg_usleep(1000L * 10L * i); Could this use something like autovacuum_cost_delay? I don't quite like the use of arbitrary hardcoded millisecond delays - it can slow a system down by a significant fraction, especially on high-contention systems, and this potential of 60ms delay per scanned page can limit the throughput of this new vacuum strategy to < 17 pages/second (<136kB/sec) for highly contended sections, which is not great. It is also not unlikely that in the time it was waiting, the page contents were updated significantly (concurrent prune, DELETEs committed), which could result in improved bounds. I think we should redo the dead items check if we waited, but failed to get a lock - any tuples removed now reduce work we'll have to do later. > +++ b/doc/src/sgml/ref/vacuum.sgml > [...] Pages where > + all tuples are known to be frozen are always skipped. "...are always skipped, unless the >DISABLE_PAGE_SKIPPING< option is used." > +++ b/doc/src/sgml/maintenance.sgml There are a lot of details being lost from the previous version of that document. Some of the details are obsolete (mentions of aggressive VACUUM and freezing behavior), but others are not (FrozenTransactionId in rows from a pre-9.4 system, the need for vacuum for prevention of issues surrounding XID wraparound). I also am not sure this is the best place to store most of these mentions, but I can't find a different place where these details on certain interesting parts of the system are documented, and plain removal of the information does not sit right with me. Specifically, I don't like the removal of the following information from our documentation: - Size of pg_xact and pg_commit_ts data in relation to autovacuum_freeze_max_age Although it is less likely with the new behaviour that we'll hit these limits due to more eager freezing of transactions, it is still important for users to have easy access to this information, and tuning this for storage size is not useless information. - The reason why VACUUM is essential to the long-term consistency of Postgres' MVCC system Informing the user about our use of 32-bit transaction IDs and that we update an epoch when this XID wraps around does not automatically make the user aware of the issues that surface around XID wraparound. Retaining the explainer for XID wraparound in the docs seems like a decent idea - it may be moved, but please don't delete it. - Special transaction IDs, their meaning and where they can occur I can't seem to find any other information in the docs section, and it is useful to have users understand that certain values are considered special: FrozenTransactionId and BootstrapTransactionId. Kind regards, Matthias van de Meent ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v2 08/17] Make opp freeze heuristic compatible with prune+freeze record @ 2024-01-07 21:11 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-01-07 21:11 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 58 +++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 9c709315192..e715fc29a83 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -241,6 +241,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, PruneState prstate; HeapTupleData tup; bool do_freeze; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; TransactionId frz_conflict_horizon = InvalidTransactionId; @@ -410,6 +414,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -467,11 +478,42 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = presult->all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -563,20 +605,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (presult->all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); -- 2.40.1 --rdqtp5puvxqotfdw Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0009-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v4 10/19] Make opp freeze heuristic compatible with prune+freeze record @ 2024-01-07 21:11 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-01-07 21:11 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 57 +++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 20907ba5408..9edf6bf72d7 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -246,6 +246,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, PruneState prstate; HeapTupleData tup; bool do_freeze; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; /* @@ -439,6 +443,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -483,11 +494,41 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = presult->all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -579,20 +620,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (presult->all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { /* -- 2.40.1 --tez7m2a73jtztiij Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0011-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v3 08/17] Make opp freeze heuristic compatible with prune+freeze record @ 2024-01-07 21:11 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-01-07 21:11 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 58 +++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index abf6bdb2d99..f164b7957ed 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -255,6 +255,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, PruneState prstate; HeapTupleData tup; bool do_freeze; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; TransactionId frz_conflict_horizon = InvalidTransactionId; @@ -424,6 +428,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -481,11 +492,42 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = presult->all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -577,20 +619,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (presult->all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); -- 2.40.1 --racicctn4wry6xe5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0009-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v4 10/19] Make opp freeze heuristic compatible with prune+freeze record @ 2024-01-07 21:11 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-01-07 21:11 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 57 +++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 20907ba5408..9edf6bf72d7 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -246,6 +246,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, PruneState prstate; HeapTupleData tup; bool do_freeze; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; /* @@ -439,6 +443,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -483,11 +494,41 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = presult->all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -579,20 +620,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (presult->all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { /* -- 2.40.1 --tez7m2a73jtztiij Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0011-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v3 08/17] Make opp freeze heuristic compatible with prune+freeze record @ 2024-01-07 21:11 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-01-07 21:11 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 58 +++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index abf6bdb2d99..f164b7957ed 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -255,6 +255,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, PruneState prstate; HeapTupleData tup; bool do_freeze; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; TransactionId frz_conflict_horizon = InvalidTransactionId; @@ -424,6 +428,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -481,11 +492,42 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = presult->all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -577,20 +619,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (presult->all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); -- 2.40.1 --racicctn4wry6xe5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0009-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v2 08/17] Make opp freeze heuristic compatible with prune+freeze record @ 2024-01-07 21:11 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-01-07 21:11 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 58 +++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 9c709315192..e715fc29a83 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -241,6 +241,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, PruneState prstate; HeapTupleData tup; bool do_freeze; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; TransactionId frz_conflict_horizon = InvalidTransactionId; @@ -410,6 +414,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -467,11 +478,42 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = presult->all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -563,20 +605,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (presult->all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); -- 2.40.1 --rdqtp5puvxqotfdw Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0009-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v3 08/17] Make opp freeze heuristic compatible with prune+freeze record @ 2024-01-07 21:11 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-01-07 21:11 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 58 +++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index abf6bdb2d99..f164b7957ed 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -255,6 +255,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, PruneState prstate; HeapTupleData tup; bool do_freeze; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; TransactionId frz_conflict_horizon = InvalidTransactionId; @@ -424,6 +428,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -481,11 +492,42 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = presult->all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -577,20 +619,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (presult->all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); -- 2.40.1 --racicctn4wry6xe5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0009-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v4 10/19] Make opp freeze heuristic compatible with prune+freeze record @ 2024-01-07 21:11 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-01-07 21:11 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 57 +++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 20907ba5408..9edf6bf72d7 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -246,6 +246,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, PruneState prstate; HeapTupleData tup; bool do_freeze; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; /* @@ -439,6 +443,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -483,11 +494,41 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = presult->all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -579,20 +620,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (presult->all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { /* -- 2.40.1 --tez7m2a73jtztiij Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0011-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v2 08/17] Make opp freeze heuristic compatible with prune+freeze record @ 2024-01-07 21:11 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-01-07 21:11 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 58 +++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 9c709315192..e715fc29a83 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -241,6 +241,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, PruneState prstate; HeapTupleData tup; bool do_freeze; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; TransactionId frz_conflict_horizon = InvalidTransactionId; @@ -410,6 +414,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -467,11 +478,42 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = presult->all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -563,20 +605,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (presult->all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz); -- 2.40.1 --rdqtp5puvxqotfdw Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0009-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v7 09/16] Make opp freeze heuristic compatible with prune+freeze record @ 2024-03-26 00:48 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-03-26 00:48 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 57 +++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index e009c7579dd..d38de9b063d 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -247,6 +247,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, TransactionId visibility_cutoff_xid; bool do_freeze; bool all_visible_except_removable; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; /* @@ -456,6 +460,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -500,11 +511,41 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -569,20 +610,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { TransactionId frz_conflict_horizon = InvalidTransactionId; -- 2.40.1 --ck6erxojvlx23byk Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0010-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v7 09/16] Make opp freeze heuristic compatible with prune+freeze record @ 2024-03-26 00:48 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-03-26 00:48 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 57 +++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index e009c7579dd..d38de9b063d 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -247,6 +247,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, TransactionId visibility_cutoff_xid; bool do_freeze; bool all_visible_except_removable; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; /* @@ -456,6 +460,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -500,11 +511,41 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -569,20 +610,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { TransactionId frz_conflict_horizon = InvalidTransactionId; -- 2.40.1 --ck6erxojvlx23byk Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0010-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH v9 09/21] Make opp freeze heuristic compatible with prune+freeze record @ 2024-03-26 00:48 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Melanie Plageman @ 2024-03-26 00:48 UTC (permalink / raw) Once the prune and freeze records are combined, we will no longer be able to use a test of whether or not pruning emitted an FPI to decide whether or not to opportunistically freeze a freezable page. While this heuristic should be improved, for now, approximate the previous logic by keeping track of whether or not a hint bit FPI was emitted during visibility checks (when checksums are on) and combine that with checking XLogCheckBufferNeedsBackup(). If we just finished deciding whether or not to prune and the current buffer seems to need an FPI after modification, it is likely that pruning would have emitted an FPI. --- src/backend/access/heap/pruneheap.c | 57 +++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 15 deletions(-) diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c index 312695f806c..f0decff35dc 100644 --- a/src/backend/access/heap/pruneheap.c +++ b/src/backend/access/heap/pruneheap.c @@ -247,6 +247,10 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, TransactionId visibility_cutoff_xid; bool do_freeze; bool all_visible_except_removable; + bool do_prune; + bool whole_page_freezable; + bool hint_bit_fpi; + bool prune_fpi = false; int64 fpi_before = pgWalUsage.wal_fpi; /* @@ -456,6 +460,13 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, } } + /* + * If checksums are enabled, heap_prune_satisfies_vacuum() may have caused + * an FPI to be emitted. Then reset fpi_before for no prune case. + */ + hint_bit_fpi = fpi_before != pgWalUsage.wal_fpi; + fpi_before = pgWalUsage.wal_fpi; + /* * For vacuum, if the whole page will become frozen, we consider * opportunistically freezing tuples. Dead tuples which will be removed by @@ -500,11 +511,41 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, if (off_loc) *off_loc = InvalidOffsetNumber; + do_prune = prstate.nredirected > 0 || + prstate.ndead > 0 || + prstate.nunused > 0; + + /* + * Only incur overhead of checking if we will do an FPI if we might use + * the information. + */ + if (do_prune && pagefrz) + prune_fpi = XLogCheckBufferNeedsBackup(buffer); + + /* Is the whole page freezable? And is there something to freeze */ + whole_page_freezable = all_visible_except_removable && + presult->all_frozen; + + /* + * Freeze the page when heap_prepare_freeze_tuple indicates that at least + * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also + * freeze when pruning generated an FPI, if doing so means that we set the + * page all-frozen afterwards (might not happen until final heap pass). + * XXX: Previously, we knew if pruning emitted an FPI by checking + * pgWalUsage.wal_fpi before and after pruning. Once the freeze and prune + * records are combined, this heuristic couldn't be used anymore. The + * opportunistic freeze heuristic must be improved; however, for now, try + * to approximate it. + */ + do_freeze = pagefrz && + (pagefrz->freeze_required || + (whole_page_freezable && presult->nfrozen > 0 && (prune_fpi || hint_bit_fpi))); + /* Any error while applying the changes is critical */ START_CRIT_SECTION(); /* Have we found any prunable items? */ - if (prstate.nredirected > 0 || prstate.ndead > 0 || prstate.nunused > 0) + if (do_prune) { /* * Apply the planned item changes, then repair page fragmentation, and @@ -569,20 +610,6 @@ heap_page_prune_and_freeze(Relation relation, Buffer buffer, /* Record number of newly-set-LP_DEAD items for caller */ presult->nnewlpdead = prstate.ndead; - /* - * Freeze the page when heap_prepare_freeze_tuple indicates that at least - * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also - * freeze when pruning generated an FPI, if doing so means that we set the - * page all-frozen afterwards (might not happen until final heap pass). - */ - if (pagefrz) - do_freeze = pagefrz->freeze_required || - (all_visible_except_removable && presult->all_frozen && - presult->nfrozen > 0 && - fpi_before != pgWalUsage.wal_fpi); - else - do_freeze = false; - if (do_freeze) { TransactionId frz_conflict_horizon = InvalidTransactionId; -- 2.40.1 --caj67xgx3lukmr5f Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0010-Separate-tuple-pre-freeze-checks-and-invoke-earli.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
end of thread, other threads:[~2024-03-26 00:48 UTC | newest] Thread overview: 14+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2018-06-19 14:57 Re: [HACKERS] Custom compression methods Robert Haas <[email protected]> 2023-01-05 18:19 Re: New strategies for freezing, advancing relfrozenxid early Matthias van de Meent <[email protected]> 2024-01-07 21:11 [PATCH v3 08/17] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-01-07 21:11 [PATCH v4 10/19] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-01-07 21:11 [PATCH v3 08/17] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-01-07 21:11 [PATCH v2 08/17] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-01-07 21:11 [PATCH v4 10/19] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-01-07 21:11 [PATCH v3 08/17] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-01-07 21:11 [PATCH v2 08/17] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-01-07 21:11 [PATCH v4 10/19] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-01-07 21:11 [PATCH v2 08/17] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-03-26 00:48 [PATCH v7 09/16] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-03-26 00:48 [PATCH v9 09/21] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]> 2024-03-26 00:48 [PATCH v7 09/16] Make opp freeze heuristic compatible with prune+freeze record Melanie Plageman <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox