public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v25 5/5] doc: Add Collation Versions section. 14+ messages / 7 participants [nested] [flat]
* [PATCH v25 5/5] doc: Add Collation Versions section. @ 2020-03-11 02:01 Thomas Munro <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Thomas Munro @ 2020-03-11 02:01 UTC (permalink / raw) Supply a brief introduction to collation version concepts. Author: Thomas Munro Reviewed-by: Julien Rouhaud Discussion: https://postgr.es/m/CAEepm%3D0uEQCpfq_%2BLYFBdArCe4Ot98t1aR4eYiYTe%3DyavQygiQ%40mail.gmail.com --- doc/src/sgml/charset.sgml | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/doc/src/sgml/charset.sgml b/doc/src/sgml/charset.sgml index 4b4563c5b9..c537bdfc28 100644 --- a/doc/src/sgml/charset.sgml +++ b/doc/src/sgml/charset.sgml @@ -948,6 +948,41 @@ CREATE COLLATION ignore_accents (provider = icu, locale = 'und-u-ks-level1-kc-tr </tip> </sect3> </sect2> + + <sect2 id="collation-versions"> + <title>Collation Versions</title> + + <para> + The ordering defined by a collation is not necessarily fixed over time. + If a collation changes for any reason, persistent data structures such as + b-trees that depend on a stable ordering of text might be corrupted. + <productname>PostgreSQL</productname> defends against this by recording + the current version of each referenced collation for any index that + depends on it in the + <link linkend="catalog-pg-depend"><structname>pg_depend</structname></link> + catalog, if the collation provider makes it available. If the provider + later begins to report a different version, a warning will be reported + when the index is accessed, until either the <xref linkend="sql-reindex"/> + or the <xref linkend="sql-alterindex"/> command is used to update the + version. + </para> + <para> + Version information is available for collations from the + <literal>icu</literal> provider on all operating systems. For the + <literal>libc</literal> provider, versions are currently only available + on systems using the GNU C library (most Linux systems). + </para> + + <note> + <para> + When using the GNU C library for collations, the C library's version + is used as a proxy for the collation version. Many Linux distributions + change collation definitions only when upgrading the C library, but this + approach is imperfect as maintainers are free to back-port newer + collation definitions to older C library releases. + </para> + </note> + </sect2> </sect1> <sect1 id="multibyte"> -- 2.20.1 --lrZ03NoBR/3+SXJZ-- ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-04-04 14:04 Heikki Linnakangas <[email protected]> 0 siblings, 1 reply; 14+ messages in thread From: Heikki Linnakangas @ 2025-04-04 14:04 UTC (permalink / raw) To: Дарья Шанина <[email protected]>; [email protected] On 04/04/2025 16:40, Дарья Шанина wrote: > Hello everyone! > I have a question. > > What would be better for the function autoprewarm_dump_now in case when > we need to allocate memory that exceeds 1 GB: Hmm, so if I counted right, sizeof(BlockInfoRecord) == 20 bytes, which means that you can fit about 409 GB worth of buffers in a 1 GB allocation. So autoprewarm will currently not work with shared_buffers > 409 GB. That's indeed quite unfortunate. > 1) allocate enough memory for the entire shared_buffer array > (1..NBuffers) using palloc_extended; That would be a pretty straightforward fix. > 2) allocate the maximum of currently possible memory (1 GB) using an > ordinary palloc. That'd put an upper limit on how much is prewarmed. It'd be a weird limitation. And prewarming matters the most with large shared_buffers. 3) Don't pre-allocate the array, write it out in a streaming fashion. Unfortunately the file format doesn't make that easy: the number of entries is at the beginning of the file. You could count the entries beforehand, but the buffers can change concurrently. You could write a placeholder first, and seek back to the beginning of the file to fill in the real number at the end. The problem with that is that the number of bytes needed for the count itself varies. I suppose we could write some spaces as placeholders to accommodate the max count. In apw_load_buffers(), we also load the file into (DSM) memory. There's no similar 1 GB limit in dsm_create(), but I think it's a bit unfortunate that the array needs to be allocated upfront upon loading. In short, ISTM the easy answer here is "use palloc_extended". But there's a lot of room for further optimizations. -- Heikki Linnakangas Neon (https://neon.tech) ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-04-04 16:17 Melanie Plageman <[email protected]> parent: Heikki Linnakangas <[email protected]> 0 siblings, 1 reply; 14+ messages in thread From: Melanie Plageman @ 2025-04-04 16:17 UTC (permalink / raw) To: Heikki Linnakangas <[email protected]>; +Cc: Дарья Шанина <[email protected]>; [email protected] On Fri, Apr 4, 2025 at 10:04 AM Heikki Linnakangas <[email protected]> wrote: > > In apw_load_buffers(), we also load the file into (DSM) memory. There's > no similar 1 GB limit in dsm_create(), but I think it's a bit > unfortunate that the array needs to be allocated upfront upon loading. Unrelated to this problem, but I wondered why autoprewarm doesn''t launch background workers for each database simultaneously instead of waiting for each one to finish a db before moving onto the next one. Is it simply to limit the number of bgworkers taking up resources? - Melanie ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-04-04 16:36 Robert Haas <[email protected]> parent: Melanie Plageman <[email protected]> 0 siblings, 1 reply; 14+ messages in thread From: Robert Haas @ 2025-04-04 16:36 UTC (permalink / raw) To: Melanie Plageman <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Дарья Шанина <[email protected]>; [email protected] On Fri, Apr 4, 2025 at 12:17 PM Melanie Plageman <[email protected]> wrote: > Unrelated to this problem, but I wondered why autoprewarm doesn''t > launch background workers for each database simultaneously instead of > waiting for each one to finish a db before moving onto the next one. > Is it simply to limit the number of bgworkers taking up resources? That's probably part of it, but also (1) a system that allowed for multiple workers would be somewhat more complex to implement and (2) I'm not sure how beneficial it would be. We go to some trouble to make the I/O as sequential as possible, and this would detract from that. I also don't know how long prewarming normally takes -- if it's fast enough already, then maybe this doesn't matter. But if somebody is having a problem with autoprewarm being slow and wants to implement a multi-worker system to make it faster, cool. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-05-29 13:16 Daria Shanina <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 1 reply; 14+ messages in thread From: Daria Shanina @ 2025-05-29 13:16 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] Hello! I have made a patch, now we can allocate more than 1 GB of memory for the autoprewarm_dump_now function. Best regards, Daria Shanina пт, 4 апр. 2025 г. в 19:36, Robert Haas <[email protected]>: > On Fri, Apr 4, 2025 at 12:17 PM Melanie Plageman > <[email protected]> wrote: > > Unrelated to this problem, but I wondered why autoprewarm doesn''t > > launch background workers for each database simultaneously instead of > > waiting for each one to finish a db before moving onto the next one. > > Is it simply to limit the number of bgworkers taking up resources? > > That's probably part of it, but also (1) a system that allowed for > multiple workers would be somewhat more complex to implement and (2) > I'm not sure how beneficial it would be. We go to some trouble to make > the I/O as sequential as possible, and this would detract from that. I > also don't know how long prewarming normally takes -- if it's fast > enough already, then maybe this doesn't matter. But if somebody is > having a problem with autoprewarm being slow and wants to implement a > multi-worker system to make it faster, cool. > > -- > Robert Haas > EDB: http://www.enterprisedb.com > Attachments: [text/x-patch] v1-0001-PGPRO-9971-Allocate-enough-memory-with-huge-share.patch (880B, ../../CAMp4U1c8S7oTsuNahvkFLR7OeSxnamtOHgey=rcMi=Y7cWg7ng@mail.gmail.com/3-v1-0001-PGPRO-9971-Allocate-enough-memory-with-huge-share.patch) download | inline diff: From 7b8af18231b539378ec4fc432186b551c08a7774 Mon Sep 17 00:00:00 2001 From: Darya Shanina <[email protected]> Date: Tue, 22 Apr 2025 11:27:01 +0300 Subject: [PATCH v1] [PGPRO-9971] Allocate enough memory with huge shared_buffers. Tags: commitfest_hotfix --- contrib/pg_prewarm/autoprewarm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/pg_prewarm/autoprewarm.c b/contrib/pg_prewarm/autoprewarm.c index d061731706a..ea5f7bc49c9 100644 --- a/contrib/pg_prewarm/autoprewarm.c +++ b/contrib/pg_prewarm/autoprewarm.c @@ -598,7 +598,7 @@ apw_dump_now(bool is_bgworker, bool dump_unlogged) } block_info_array = - (BlockInfoRecord *) palloc(sizeof(BlockInfoRecord) * NBuffers); + (BlockInfoRecord *) palloc_extended((sizeof(BlockInfoRecord) * NBuffers), MCXT_ALLOC_HUGE); for (num_blocks = 0, i = 0; i < NBuffers; i++) { -- 2.43.0 ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-05-29 13:21 Tom Lane <[email protected]> parent: Daria Shanina <[email protected]> 0 siblings, 2 replies; 14+ messages in thread From: Tom Lane @ 2025-05-29 13:21 UTC (permalink / raw) To: Daria Shanina <[email protected]>; +Cc: Robert Haas <[email protected]>; Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] Daria Shanina <[email protected]> writes: > I have made a patch, now we can allocate more than 1 GB of memory for the > autoprewarm_dump_now function. Is that solving a real-world problem? If it is, shouldn't we be looking for a different approach that doesn't require such a huge amount of memory? regards, tom lane ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-05-29 13:58 Robert Haas <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 0 replies; 14+ messages in thread From: Robert Haas @ 2025-05-29 13:58 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Daria Shanina <[email protected]>; Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] On Thu, May 29, 2025 at 9:21 AM Tom Lane <[email protected]> wrote: > Is that solving a real-world problem? If it is, shouldn't we be > looking for a different approach that doesn't require such a huge > amount of memory? Upthread, Heikki said that this function currently fails with shared_buffers>409GB. While I'm not opposed to a more efficient solution, it seems reasonable to me to suppose that if you've got shared_buffers>409GB, you're able to afford having this function use >1GB. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-05-30 10:07 Daria Shanina <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 1 reply; 14+ messages in thread From: Daria Shanina @ 2025-05-30 10:07 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Robert Haas <[email protected]>; Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] Some of our clients encountered a problem — they needed to allocate shared_buffers = 700 GB on a server with 1.5 TB RAM, and the error "invalid memory alloc request size 1835008000" occurred. That is, these are not mental exercises. Best regards, Daria Shanina чт, 29 мая 2025 г. в 16:21, Tom Lane <[email protected]>: > Daria Shanina <[email protected]> writes: > > I have made a patch, now we can allocate more than 1 GB of memory for the > > autoprewarm_dump_now function. > > Is that solving a real-world problem? If it is, shouldn't we be > looking for a different approach that doesn't require such a huge > amount of memory? > > regards, tom lane > -- С уважением, Шанина Дарья Александровна ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-06-03 17:17 Robert Haas <[email protected]> parent: Daria Shanina <[email protected]> 0 siblings, 2 replies; 14+ messages in thread From: Robert Haas @ 2025-06-03 17:17 UTC (permalink / raw) To: Daria Shanina <[email protected]>; +Cc: Tom Lane <[email protected]>; Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] On Fri, May 30, 2025 at 6:07 AM Daria Shanina <[email protected]> wrote: > Some of our clients encountered a problem — they needed to allocate shared_buffers = 700 GB on a server with 1.5 TB RAM, and the error "invalid memory alloc request size 1835008000" occurred. That is, these are not mental exercises. I think the proposed patch should be committed and back-patched, after fixing it so that it's pgindent-clean and adding a comment. Does anyone have strong objection to that? -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-06-03 17:24 Tom Lane <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 1 reply; 14+ messages in thread From: Tom Lane @ 2025-06-03 17:24 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Daria Shanina <[email protected]>; Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] Robert Haas <[email protected]> writes: > I think the proposed patch should be committed and back-patched, after > fixing it so that it's pgindent-clean and adding a comment. Does > anyone have strong objection to that? Not here. I do wonder if we can't find a more memory-efficient way, but I concur that any such change would likely not be back-patch material. regards, tom lane ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-06-03 17:56 Robert Haas <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Robert Haas @ 2025-06-03 17:56 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Daria Shanina <[email protected]>; Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] On Tue, Jun 3, 2025 at 1:24 PM Tom Lane <[email protected]> wrote: > Robert Haas <[email protected]> writes: > > I think the proposed patch should be committed and back-patched, after > > fixing it so that it's pgindent-clean and adding a comment. Does > > anyone have strong objection to that? > > Not here. I do wonder if we can't find a more memory-efficient way, > but I concur that any such change would likely not be back-patch > material. Cool. Regarding memory efficiency, I was just discussing radix trees this morning with Sawada-san and others. "lib/radixtree.h" seems unsuitable here because it supposes that each node should cover 1 byte of key material, which seems unsuitable for a 20-byte key, but I wonder if we could reuse "common/blkreftable.h" which I created for incremental backup but which seems like it could probably work here as well. Implementation sketch: The "limit block" concept would be irrelevant in this case, so you just wouldn't call BlockRefTableSetLimitBlock. Instead, you'd call BlockRefTableMarkBlockModified for each memory-resident block -- the name would be a misnomer, and we might want to do some renaming. blkreftable.c/h already has infrastructure to read and write the data structure from and to disk, so we wouldn't need to reinvent that. The reason I think this might work out well is that for moderately sparse block sets, the space usage is about 2 bytes per block, while for dense sets block sets, it converges to 1 bit per block. If you make the set of blocks super-duper sparse then it might use more memory than what we do today, but I'm not sure that's a very realistic scenario in practice -- you'd probably need to have a gigantic number of relations in the database and absolutely no locality of access. See the comment in blkreftable.c just above "#define BLOCKS_PER_CHUNK" for what the format actually is. -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-06-03 18:58 Andres Freund <[email protected]> parent: Robert Haas <[email protected]> 1 sibling, 1 reply; 14+ messages in thread From: Andres Freund @ 2025-06-03 18:58 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Daria Shanina <[email protected]>; Tom Lane <[email protected]>; Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] Hi, On 2025-06-03 13:17:43 -0400, Robert Haas wrote: > On Fri, May 30, 2025 at 6:07 AM Daria Shanina <[email protected]> wrote: > > Some of our clients encountered a problem — they needed to allocate shared_buffers = 700 GB on a server with 1.5 TB RAM, and the error "invalid memory alloc request size 1835008000" occurred. That is, these are not mental exercises. > > I think the proposed patch should be committed and back-patched, after > fixing it so that it's pgindent-clean and adding a comment. Does > anyone have strong objection to that? No, seems like a thing that pretty obviously should be fixed. Greetings, Andres Freund ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-06-06 13:18 Robert Haas <[email protected]> parent: Andres Freund <[email protected]> 0 siblings, 1 reply; 14+ messages in thread From: Robert Haas @ 2025-06-06 13:18 UTC (permalink / raw) To: Andres Freund <[email protected]>; +Cc: Daria Shanina <[email protected]>; Tom Lane <[email protected]>; Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] On Tue, Jun 3, 2025 at 2:58 PM Andres Freund <[email protected]> wrote: > > I think the proposed patch should be committed and back-patched, after > > fixing it so that it's pgindent-clean and adding a comment. Does > > anyone have strong objection to that? > > No, seems like a thing that pretty obviously should be fixed. Done. *quakes in fear of incoming buildfarm results* -- Robert Haas EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: autoprewarm_dump_now @ 2025-06-17 15:15 Daria Shanina <[email protected]> parent: Robert Haas <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Daria Shanina @ 2025-06-17 15:15 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Andres Freund <[email protected]>; Tom Lane <[email protected]>; Melanie Plageman <[email protected]>; Heikki Linnakangas <[email protected]>; [email protected] Hi, I launched pgindent and created a new patch. Here it is. *quakes in fear of incoming buildfarm results* I hope there will be good results at buildfarm. пт, 6 июн. 2025 г. в 16:19, Robert Haas <[email protected]>: > On Tue, Jun 3, 2025 at 2:58 PM Andres Freund <[email protected]> wrote: > > > I think the proposed patch should be committed and back-patched, after > > > fixing it so that it's pgindent-clean and adding a comment. Does > > > anyone have strong objection to that? > > > > No, seems like a thing that pretty obviously should be fixed. > > Done. > > *quakes in fear of incoming buildfarm results* > > -- > Robert Haas > EDB: http://www.enterprisedb.com > -- Best regards, Daria Shanina Attachments: [text/x-patch] v1-0001-PGPRO-9971-Allocate-enough-memory-with-huge-share_buffers.patch (928B, ../../CAMp4U1cAXDKCwV4Cj=c6E2kAUD=3s9LBpPWHtfhwiFahF16iCw@mail.gmail.com/3-v1-0001-PGPRO-9971-Allocate-enough-memory-with-huge-share_buffers.patch) download | inline diff: From a4776e4ae0c2ffe3061b64dec7ac60d730db12b0 Mon Sep 17 00:00:00 2001 From: Darya Shanina <[email protected]> Date: Tue, 22 Apr 2025 11:27:01 +0300 Subject: [PATCH v1] [PGPRO-9971] Allocate enough memory with huge shared_buffers. Tags: commitfest_hotfix --- contrib/pg_prewarm/autoprewarm.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/contrib/pg_prewarm/autoprewarm.c b/contrib/pg_prewarm/autoprewarm.c index d061731706a..4dbda58eeae 100644 --- a/contrib/pg_prewarm/autoprewarm.c +++ b/contrib/pg_prewarm/autoprewarm.c @@ -597,8 +597,9 @@ apw_dump_now(bool is_bgworker, bool dump_unlogged) return 0; } - block_info_array = - (BlockInfoRecord *) palloc(sizeof(BlockInfoRecord) * NBuffers); + block_info_array = (BlockInfoRecord *) + palloc_extended((sizeof(BlockInfoRecord) * NBuffers), + MCXT_ALLOC_HUGE); for (num_blocks = 0, i = 0; i < NBuffers; i++) { -- 2.43.0 ^ permalink raw reply [nested|flat] 14+ messages in thread
end of thread, other threads:[~2025-06-17 15:15 UTC | newest] Thread overview: 14+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-03-11 02:01 [PATCH v25 5/5] doc: Add Collation Versions section. Thomas Munro <[email protected]> 2025-04-04 14:04 Re: autoprewarm_dump_now Heikki Linnakangas <[email protected]> 2025-04-04 16:17 ` Re: autoprewarm_dump_now Melanie Plageman <[email protected]> 2025-04-04 16:36 ` Re: autoprewarm_dump_now Robert Haas <[email protected]> 2025-05-29 13:16 ` Re: autoprewarm_dump_now Daria Shanina <[email protected]> 2025-05-29 13:21 ` Re: autoprewarm_dump_now Tom Lane <[email protected]> 2025-05-29 13:58 ` Re: autoprewarm_dump_now Robert Haas <[email protected]> 2025-05-30 10:07 ` Re: autoprewarm_dump_now Daria Shanina <[email protected]> 2025-06-03 17:17 ` Re: autoprewarm_dump_now Robert Haas <[email protected]> 2025-06-03 17:24 ` Re: autoprewarm_dump_now Tom Lane <[email protected]> 2025-06-03 17:56 ` Re: autoprewarm_dump_now Robert Haas <[email protected]> 2025-06-03 18:58 ` Re: autoprewarm_dump_now Andres Freund <[email protected]> 2025-06-06 13:18 ` Re: autoprewarm_dump_now Robert Haas <[email protected]> 2025-06-17 15:15 ` Re: autoprewarm_dump_now Daria Shanina <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox