pg.ddx.io pgsql-committers@postgresql.org mailing list archive help / color / mirror / Atom feed From: itagaki@pgfoundry.org
To: pgsql-committers@postgresql.org
Subject: pgbulkload - pgbulkload: Add parallel loader for pg_bulkload 3.0.
Date: Thu, 4 Jun 2009 03:26:11 +0000 (UTC)
Message-ID: <20090604032611.3664D17ADF12@pgfoundry.org> (raw )
Log Message:
-----------
Add parallel loader for pg_bulkload 3.0.
If LOADER=PARALLEL is specified, input file parse and data load are executed
in different backends. Parsed tuples are passed through a shared memory queue.
Loading would be improved at best twice as faster as loading with single process.
Modified Files:
--------------
pgbulkload/bin:
Makefile (r1.5 -> r1.6)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/Makefile?r1=1.5&r2=1.6 )
pg_bulkload.c (r1.9 -> r1.10)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/pg_bulkload.c?r1=1.9&r2=1.... )
postgresql (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/postgresql?r1=1.3&r2=1.4 )
pgbulkload/bin/expected:
init.out (r1.2 -> r1.3)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/expected/init.out?r1=1.2&r... )
pgbulkload/bin/sql:
init.sql (r1.2 -> r1.3)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/sql/init.sql?r1=1.2&r2=1.3 )
pgbulkload/include:
pg_btree.h (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/pg_btree.h?r1=1.3&r2=1... )
pg_bulkload.h (r1.7 -> r1.8)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/pg_bulkload.h?r1=1.7&r... )
reader.h (r1.2 -> r1.3)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/reader.h?r1=1.2&r2=1.3 )
writer.h (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/writer.h?r1=1.3&r2=1.4 )
pgbulkload/lib:
Makefile (r1.6 -> r1.7)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/Makefile?r1=1.6&r2=1.7 )
pg_btree.c (r1.8 -> r1.9)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_btree.c?r1=1.8&r2=1.9 )
pg_bulkload.c (r1.10 -> r1.11)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_bulkload.c?r1=1.10&r2=1... )
pg_bulkload.sql.in (r1.5 -> r1.6)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_bulkload.sql.in?r1=1.5&... )
pg_strutil.c (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_strutil.c?r1=1.3&r2=1.4 )
reader.c (r1.3 -> r1.4)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/reader.c?r1=1.3&r2=1.4 )
uninstall_pg_bulkload.sql (r1.4 -> r1.5)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/uninstall_pg_bulkload.sql?r1=1... )
Added Files:
-----------
pgbulkload/bin/expected:
load_parallel.out (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/expected/load_parallel.out?rev... )
pgbulkload/bin/sql:
load_parallel.sql (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/sql/load_parallel.sql?rev=1.1&... )
pgbulkload/lib:
parser_binary.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/parser_binary.c?rev=1.1&co... )
parser_csv.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/parser_csv.c?rev=1.1&conte... )
parser_tuple.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/parser_tuple.c?rev=1.1&con... )
source.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/source.c?rev=1.1&content-t... )
writer_buffered.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer_buffered.c?rev=1.1&... )
writer_direct.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer_direct.c?rev=1.1&co... )
writer_parallel.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer_parallel.c?rev=1.1&... )
pgbulkload/lib/pgut:
pgut-ipc.c (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pgut/pgut-ipc.c?rev=1.1&co... )
pgut-ipc.h (r1.1)
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pgut/pgut-ipc.h?rev=1.1&co... )
Removed Files:
-------------
pgbulkload/lib:
pg_heap_buffered.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_heap_buffered.c )
pg_heap_direct.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_heap_direct.c )
pg_input_csv.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_input_csv.c )
pg_input_fixed.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_input_fixed.c )
writer.c
(http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer.c )
view thread (2+ messages) latest in thread
Message-ID: <20090604032611.3664D17ADF12@pgfoundry.org>
Permalink: ../20090604032611.3664D17ADF12@pgfoundry.org/
Also on: postgresql.org/message-id/20090604032611.3664D17ADF12@pgfoundry.org
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-committers@postgresql.org
Cc: itagaki@pgfoundry.org
Subject: Re: pgbulkload - pgbulkload: Add parallel loader for pg_bulkload 3.0.
In-Reply-To: <20090604032611.3664D17ADF12@pgfoundry.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox