Received: from maia.hub.org (unknown [200.46.204.183]) by mail.postgresql.org (Postfix) with ESMTP id 823AA634A34 for ; Thu, 4 Jun 2009 00:26:14 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 46169-01 for ; Thu, 4 Jun 2009 00:26:13 -0300 (ADT) Received: from pgfoundry.org (pgfoundry.org [200.46.204.130]) by mail.postgresql.org (Postfix) with ESMTP id 59BEF6335B0 for ; Thu, 4 Jun 2009 00:26:13 -0300 (ADT) Received: from localhost (unknown [200.46.208.211]) by pgfoundry.org (Postfix) with ESMTP id 1DCC817ADF36 for ; Thu, 4 Jun 2009 03:26:12 +0000 (UTC) Received: from pgfoundry.org ([200.46.204.130]) by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024) with ESMTP id 70398-03 for ; Thu, 4 Jun 2009 00:26:10 -0300 (ADT) Received: by pgfoundry.org (Postfix, from userid 1357) id 3664D17ADF12; Thu, 4 Jun 2009 03:26:11 +0000 (UTC) MIME-Version: 1.0 To: pgsql-committers@postgresql.org Subject: pgbulkload - pgbulkload: Add parallel loader for pg_bulkload 3.0. X-Mailer: activitymail 1.26, http://search.cpan.org/dist/activitymail/ Content-Type: text/plain; charset=utf-8 Message-Id: <20090604032611.3664D17ADF12@pgfoundry.org> Date: Thu, 4 Jun 2009 03:26:11 +0000 (UTC) From: itagaki@pgfoundry.org (User Itagaki) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests=none X-Spam-Level: X-Archive-Number: 200906/39 X-Sequence-Number: 37243 Log Message: ----------- Add parallel loader for pg_bulkload 3.0. If LOADER=PARALLEL is specified, input file parse and data load are executed in different backends. Parsed tuples are passed through a shared memory queue. Loading would be improved at best twice as faster as loading with single process. Modified Files: -------------- pgbulkload/bin: Makefile (r1.5 -> r1.6) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/Makefile?r1=1.5&r2=1.6) pg_bulkload.c (r1.9 -> r1.10) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/pg_bulkload.c?r1=1.9&r2=1.10) postgresql (r1.3 -> r1.4) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/postgresql?r1=1.3&r2=1.4) pgbulkload/bin/expected: init.out (r1.2 -> r1.3) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/expected/init.out?r1=1.2&r2=1.3) pgbulkload/bin/sql: init.sql (r1.2 -> r1.3) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/sql/init.sql?r1=1.2&r2=1.3) pgbulkload/include: pg_btree.h (r1.3 -> r1.4) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/pg_btree.h?r1=1.3&r2=1.4) pg_bulkload.h (r1.7 -> r1.8) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/pg_bulkload.h?r1=1.7&r2=1.8) reader.h (r1.2 -> r1.3) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/reader.h?r1=1.2&r2=1.3) writer.h (r1.3 -> r1.4) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/include/writer.h?r1=1.3&r2=1.4) pgbulkload/lib: Makefile (r1.6 -> r1.7) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/Makefile?r1=1.6&r2=1.7) pg_btree.c (r1.8 -> r1.9) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_btree.c?r1=1.8&r2=1.9) pg_bulkload.c (r1.10 -> r1.11) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_bulkload.c?r1=1.10&r2=1.11) pg_bulkload.sql.in (r1.5 -> r1.6) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_bulkload.sql.in?r1=1.5&r2=1.6) pg_strutil.c (r1.3 -> r1.4) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_strutil.c?r1=1.3&r2=1.4) reader.c (r1.3 -> r1.4) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/reader.c?r1=1.3&r2=1.4) uninstall_pg_bulkload.sql (r1.4 -> r1.5) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/uninstall_pg_bulkload.sql?r1=1.4&r2=1.5) Added Files: ----------- pgbulkload/bin/expected: load_parallel.out (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/expected/load_parallel.out?rev=1.1&content-type=text/x-cvsweb-markup) pgbulkload/bin/sql: load_parallel.sql (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/bin/sql/load_parallel.sql?rev=1.1&content-type=text/x-cvsweb-markup) pgbulkload/lib: parser_binary.c (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/parser_binary.c?rev=1.1&content-type=text/x-cvsweb-markup) parser_csv.c (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/parser_csv.c?rev=1.1&content-type=text/x-cvsweb-markup) parser_tuple.c (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/parser_tuple.c?rev=1.1&content-type=text/x-cvsweb-markup) source.c (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/source.c?rev=1.1&content-type=text/x-cvsweb-markup) writer_buffered.c (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer_buffered.c?rev=1.1&content-type=text/x-cvsweb-markup) writer_direct.c (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer_direct.c?rev=1.1&content-type=text/x-cvsweb-markup) writer_parallel.c (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer_parallel.c?rev=1.1&content-type=text/x-cvsweb-markup) pgbulkload/lib/pgut: pgut-ipc.c (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pgut/pgut-ipc.c?rev=1.1&content-type=text/x-cvsweb-markup) pgut-ipc.h (r1.1) (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pgut/pgut-ipc.h?rev=1.1&content-type=text/x-cvsweb-markup) Removed Files: ------------- pgbulkload/lib: pg_heap_buffered.c (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_heap_buffered.c) pg_heap_direct.c (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_heap_direct.c) pg_input_csv.c (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_input_csv.c) pg_input_fixed.c (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/pg_input_fixed.c) writer.c (http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/pgbulkload/pgbulkload/lib/writer.c)