pg.ddx.io pgsql-committers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: pg_restore -jN does not equate "multiple jobs", so partly revert
8+ messages / 7 participants
[nested] [flat]
* pgsql: pg_restore -jN does not equate "multiple jobs", so partly revert
@ 2009-04-14 00:06 Alvaro Herrera <alvherre@postgresql.org>
2009-04-14 09:49 ` Re: pgsql: pg_restore -jN does not equate "multiple jobs", so partly revert Simon Riggs <simon@2ndQuadrant.com>
0 siblings, 1 reply; 8+ messages in thread
From: alvherre@postgresql.org @ 2009-04-14 00:06 UTC (permalink / raw)
To: pgsql-committers
Log Message:
-----------
pg_restore -jN does not equate "multiple jobs", so partly revert the
previous patch.
Per note from Tom.
Modified Files:
--------------
pgsql/src/bin/pg_dump:
pg_restore.c (r1.98 -> r1.99)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/bin/pg_dump/pg_restore.c?r1=1.98&r2=1.99)
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pgsql: pg_restore -jN does not equate "multiple jobs", so partly revert
2009-04-14 00:06 pgsql: pg_restore -jN does not equate "multiple jobs", so partly revert Alvaro Herrera <alvherre@postgresql.org>
@ 2009-04-14 09:49 ` Simon Riggs <simon@2ndQuadrant.com>
2009-04-14 11:33 ` Re: pgsql: pg_restore -jN does not equate "multiple jobs", so partly revert Andrew Dunstan <andrew@dunslane.net>
0 siblings, 1 reply; 8+ messages in thread
From: Simon Riggs @ 2009-04-14 09:49 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@postgresql.org>; Andrew Dunstan <adunstan@postgresql.org>; +Cc: pgsql-committers
On Tue, 2009-04-14 at 00:06 +0000, Alvaro Herrera wrote:
> Log Message:
> -----------
> pg_restore -jN does not equate "multiple jobs", so partly revert the
> previous patch.
I notice that the release notes talk about "parallel restore", yet the
word parallel is not used in the docs anywhere. The section on restoring
a dump makes no mention of this new facility, nor does the perf tips
section.
Also says that the option doesn't work with --single-transaction, so we
should say that you've added in TRUNCATE to make the COPY go faster (or
haven't you? I know we discussed it).
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pgsql: pg_restore -jN does not equate "multiple jobs", so partly revert
2009-04-14 00:06 pgsql: pg_restore -jN does not equate "multiple jobs", so partly revert Alvaro Herrera <alvherre@postgresql.org>
2009-04-14 09:49 ` Re: pgsql: pg_restore -jN does not equate "multiple jobs", so partly revert Simon Riggs <simon@2ndQuadrant.com>
@ 2009-04-14 11:33 ` Andrew Dunstan <andrew@dunslane.net>
0 siblings, 0 replies; 8+ messages in thread
From: Andrew Dunstan @ 2009-04-14 11:33 UTC (permalink / raw)
To: Simon Riggs <simon@2ndQuadrant.com>; +Cc: Alvaro Herrera <alvherre@postgresql.org>; Andrew Dunstan <adunstan@postgresql.org>; pgsql-committers
Simon Riggs wrote:
> Also says that the option doesn't work with --single-transaction, so we
> should say that you've added in TRUNCATE to make the COPY go faster (or
> haven't you? I know we discussed it).
>
>
Yes, it does use TRUNCATE.
cheers
andrew
^ permalink raw reply [nested|flat] 8+ messages in thread
* pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl
@ 2024-03-21 01:10 Masahiko Sawada <msawada@postgresql.org>
2024-03-25 22:16 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl David Rowley <dgrowleyml@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Masahiko Sawada @ 2024-03-21 01:10 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Add TIDStore, to store sets of TIDs (ItemPointerData) efficiently.
TIDStore is a data structure designed to efficiently store large sets
of TIDs. For TID storage, it employs a radix tree, where the key is
a block number, and the value is a bitmap representing offset
numbers. The TIDStore can be created on a DSA area and used by
multiple backend processes simultaneously.
There are potential future users such as tidbitmap.c, though it's very
likely the interface will need to evolve as we come to understand the
needs of different kinds of users. For example, we can support
updating the offset bitmap of existing values.
Currently, the TIDStore is not used for anything yet, aside from the
test code. But an upcoming patch will use it.
This includes a unit test module, in src/test/modules/test_tidstore.
Co-authored-by: John Naylor
Discussion: https://postgr.es/m/CAD21AoAfOZvmfR0j8VmZorZjL7RhTiQdVttNuC4W-Shdc2a-AA%40mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/30e144287a72529c9cd9fd6b07fe96eb8a1e270e
Modified Files
--------------
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 463 +++++++++++++++++++++
src/include/access/tidstore.h | 49 +++
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/.gitignore | 4 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 97 +++++
src/test/modules/test_tidstore/meson.build | 33 ++
.../modules/test_tidstore/sql/test_tidstore.sql | 65 +++
.../modules/test_tidstore/test_tidstore--1.0.sql | 27 ++
src/test/modules/test_tidstore/test_tidstore.c | 317 ++++++++++++++
.../modules/test_tidstore/test_tidstore.control | 4 +
src/tools/pgindent/typedefs.list | 5 +
15 files changed, 1091 insertions(+)
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl
2024-03-21 01:10 pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <msawada@postgresql.org>
@ 2024-03-25 22:16 ` David Rowley <dgrowleyml@gmail.com>
2024-03-26 02:35 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: David Rowley @ 2024-03-25 22:16 UTC (permalink / raw)
To: Masahiko Sawada <msawada@postgresql.org>; +Cc: pgsql-committers@lists.postgresql.org
On Thu, 21 Mar 2024 at 14:10, Masahiko Sawada <msawada@postgresql.org> wrote:
>
> Add TIDStore, to store sets of TIDs (ItemPointerData) efficiently.
> src/backend/access/common/tidstore.c | 463 +++++++++++++++++++++
I was looking at this code and I saw the following:
/* choose the maxBlockSize to be no larger than 1/16 of max_bytes */
while (16 * maxBlockSize > max_bytes * 1024L)
maxBlockSize >>= 1;
I saw the "* 1024L" and was confused by it. "max_bytes * 1024L"
converts the number of bytes into "millibytes" ([1]), which I don't
think is correct.
Either "max_bytes" is a bad name for this variable or the * 1024L
should be removed.
David
[1] https://www.convertunits.com/info/millibyte
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl
2024-03-21 01:10 pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <msawada@postgresql.org>
2024-03-25 22:16 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl David Rowley <dgrowleyml@gmail.com>
@ 2024-03-26 02:35 ` Masahiko Sawada <sawada.mshk@gmail.com>
2024-03-26 03:21 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl John Naylor <johncnaylorls@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Masahiko Sawada @ 2024-03-26 02:35 UTC (permalink / raw)
To: David Rowley <dgrowleyml@gmail.com>; +Cc: Masahiko Sawada <msawada@postgresql.org>; pgsql-committers@lists.postgresql.org
On Tue, Mar 26, 2024 at 7:16 AM David Rowley <dgrowleyml@gmail.com> wrote:
>
> On Thu, 21 Mar 2024 at 14:10, Masahiko Sawada <msawada@postgresql.org> wrote:
> >
> > Add TIDStore, to store sets of TIDs (ItemPointerData) efficiently.
>
> > src/backend/access/common/tidstore.c | 463 +++++++++++++++++++++
>
> I was looking at this code and I saw the following:
>
> /* choose the maxBlockSize to be no larger than 1/16 of max_bytes */
> while (16 * maxBlockSize > max_bytes * 1024L)
> maxBlockSize >>= 1;
>
> I saw the "* 1024L" and was confused by it. "max_bytes * 1024L"
> converts the number of bytes into "millibytes" ([1]), which I don't
> think is correct.
>
> Either "max_bytes" is a bad name for this variable or the * 1024L
> should be removed.
>
Right. We discussed it on the original thread too[1]. Since we're
going to change the create API of TidStore, I'm going to fix it along
with the API changes[2]. Even if we don't make the API change, I'll
fix it individually.
Regards,
[1] https://www.postgresql.org/message-id/CANWCAZZTE-14ofsucofTuhFsfuDGBNf%3DNZb22TMYT8bxA41oQQ%40mail.g...
[2] https://www.postgresql.org/message-id/CAD21AoAxXGf7FbuA9jvx%2BXUFrEJUz9JNOmkxF8SLppOOOvhBtg%40mail.g...
Please see v79-0003-Rethink-create-and-attach-APIs-of-shared-TidStor.patch.
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl
2024-03-21 01:10 pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <msawada@postgresql.org>
2024-03-25 22:16 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl David Rowley <dgrowleyml@gmail.com>
2024-03-26 02:35 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <sawada.mshk@gmail.com>
@ 2024-03-26 03:21 ` John Naylor <johncnaylorls@gmail.com>
2024-03-26 03:48 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: John Naylor @ 2024-03-26 03:21 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: David Rowley <dgrowleyml@gmail.com>; Masahiko Sawada <msawada@postgresql.org>; pgsql-committers@lists.postgresql.org
On Tue, Mar 26, 2024 at 9:36 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Mar 26, 2024 at 7:16 AM David Rowley <dgrowleyml@gmail.com> wrote:
> >
> > Either "max_bytes" is a bad name for this variable or the * 1024L
> > should be removed.
> >
>
> Right. We discussed it on the original thread too[1]. Since we're
> going to change the create API of TidStore, I'm going to fix it along
> with the API changes[2]. Even if we don't make the API change, I'll
> fix it individually.
v79 still has the offending multiplier, btw. Those two things are
unrelated anyway, so I don't see a reason to hold off making this
small fix.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl
2024-03-21 01:10 pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <msawada@postgresql.org>
2024-03-25 22:16 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl David Rowley <dgrowleyml@gmail.com>
2024-03-26 02:35 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <sawada.mshk@gmail.com>
2024-03-26 03:21 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl John Naylor <johncnaylorls@gmail.com>
@ 2024-03-26 03:48 ` Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 0 replies; 8+ messages in thread
From: Masahiko Sawada @ 2024-03-26 03:48 UTC (permalink / raw)
To: John Naylor <johncnaylorls@gmail.com>; +Cc: David Rowley <dgrowleyml@gmail.com>; Masahiko Sawada <msawada@postgresql.org>; pgsql-committers@lists.postgresql.org
On Tue, Mar 26, 2024 at 12:22 PM John Naylor <johncnaylorls@gmail.com> wrote:
>
> On Tue, Mar 26, 2024 at 9:36 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Tue, Mar 26, 2024 at 7:16 AM David Rowley <dgrowleyml@gmail.com> wrote:
> > >
> > > Either "max_bytes" is a bad name for this variable or the * 1024L
> > > should be removed.
> > >
> >
> > Right. We discussed it on the original thread too[1]. Since we're
> > going to change the create API of TidStore, I'm going to fix it along
> > with the API changes[2]. Even if we don't make the API change, I'll
> > fix it individually.
>
> v79 still has the offending multiplier, btw. Those two things are
> unrelated anyway, so I don't see a reason to hold off making this
> small fix.
Okay, I'll fix them shortly.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2024-03-26 03:48 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2009-04-14 00:06 pgsql: pg_restore -jN does not equate "multiple jobs", so partly revert Alvaro Herrera <alvherre@postgresql.org>
2009-04-14 09:49 ` Simon Riggs <simon@2ndQuadrant.com>
2009-04-14 11:33 ` Andrew Dunstan <andrew@dunslane.net>
2024-03-21 01:10 pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <msawada@postgresql.org>
2024-03-25 22:16 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl David Rowley <dgrowleyml@gmail.com>
2024-03-26 02:35 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <sawada.mshk@gmail.com>
2024-03-26 03:21 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl John Naylor <johncnaylorls@gmail.com>
2024-03-26 03:48 ` Re: pgsql: Add TIDStore, to store sets of TIDs (ItemPointerData) efficientl Masahiko Sawada <sawada.mshk@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox