pg.ddx.io pgsql-patches@postgresql.org mailing list archive
help / color / mirror / Atom feed[Fwd: Re: [HACKERS] pg_dump additional options for performance]
54+ messages / 9 participants
[nested] [flat]
* [Fwd: Re: [HACKERS] pg_dump additional options for performance]
@ 2008-05-07 14:37 Simon Riggs <simon@2ndquadrant.com>
2008-06-26 22:42 ` Re: [Fwd: Re: [HACKERS] pg_dump additional options for performance] Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-05-07 14:37 UTC (permalink / raw)
To: pgsql-patches
Re-sending post as discussed with Bruce...
On Sun, 2008-03-23 at 12:45 -0300, Alvaro Herrera wrote:
> Bruce Momjian wrote:
> >
> > Added to TODO:
> >
> > o Allow pre/data/post files when dumping a single object, for
> > performance reasons
> >
> > http://archives.postgresql.org/pgsql-hackers/2008-02/msg00205.php
>
> "When dumping a single object"?? Do you mean database?
It would be for whatever set of objects are specified through the use of
databases, table include/exclude switches.
I've written a patch that implements these new switches on the commands
as shown
pg_dump --schema-pre-load
pg_dump --schema-post-load
pg_restore --schema-pre-load
pg_restore --schema-post-load
I have not implemented --schema-pre-file=xxx style because they don't
make any sense when using pg_restore in direct database connection mode.
On reflection I don't see any particular need to produce multiple files
as output, which just complicates an already horrendous user interface.
This is a minimal set of changes and includes nothing at all about
directories, parallelisation in the code etc..
This has the following use cases amongst others...
* dump everything to a file, then use pg_restore first --schema-pre-load
and then --data-only directly into the database, then pg_restore
--schema-post-load to a file so we can edit that file into multiple
pieces to allow index creation in parallel
* dump of database into multiple files by manually specifying which
tables go where, then reload in parallel using multiple psql sessions
The patch tests OK after some testing, though without a test suite that
probably isn't more than a few percentage points of all the possible
code paths. There are no docs for it, as yet.
---
Further thinking on this....
Some further refinement might replace --data-only and --schema-only with
--want-schema-pre
--want-data
--want-schema-post
--want-schema (same as --want-schema-pre --want-schema-post)
These could be used together e.g. --want-schema-pre --want-data
whereas the existing --data-only type switches cannot.
Which would be a straightforward and useful change to the enclosed
patch.
That way of doing things is hierarchically extensible to include further
subdivisions of the set of SQL commands produced, e.g. divide
--want-post-schema into objects required to support various inter-table
dependencies and those that don't such as additional indexes. I don't
personally think we need that though.
Comments?
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
Attachments:
pg_dump_prepost.v1.patch (text/x-patch, 29.9K) download
Index: src/bin/pg_dump/pg_backup.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup.h,v
retrieving revision 1.45
diff -c -r1.45 pg_backup.h
*** src/bin/pg_dump/pg_backup.h 25 Jan 2007 03:30:43 -0000 1.45
--- src/bin/pg_dump/pg_backup.h 20 Mar 2008 11:48:34 -0000
***************
*** 88,94 ****
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dataOnly;
int dropSchema;
char *filename;
int schemaOnly;
--- 88,94 ----
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dumpObjFlags; /* which objects types to dump */
int dropSchema;
char *filename;
int schemaOnly;
Index: src/bin/pg_dump/pg_backup_archiver.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.c,v
retrieving revision 1.152
diff -c -r1.152 pg_backup_archiver.c
*** src/bin/pg_dump/pg_backup_archiver.c 14 Jan 2008 19:27:41 -0000 1.152
--- src/bin/pg_dump/pg_backup_archiver.c 20 Mar 2008 11:48:34 -0000
***************
*** 56,62 ****
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static teReqs _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
--- 56,62 ----
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static int _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
***************
*** 129,135 ****
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! teReqs reqs;
OutputContext sav;
bool defnDumped;
--- 129,135 ----
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! int reqs;
OutputContext sav;
bool defnDumped;
***************
*** 175,193 ****
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then we set the dataOnly flag.
*
! * We could scan for wanted TABLE entries, but that is not the same as
! * dataOnly. At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!ropt->dataOnly)
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0)
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
--- 175,193 ----
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then say we only want DATA type objects.
*
! * We could scan for wanted TABLE entries, but that is not the same.
! * At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!WANT_DATA(ropt->dumpObjFlags))
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_PRE_SCHEMA(reqs) || WANT_POST_SCHEMA(reqs))
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
***************
*** 195,201 ****
}
if (impliedDataOnly)
{
! ropt->dataOnly = impliedDataOnly;
ahlog(AH, 1, "implied data-only restore\n");
}
}
--- 195,201 ----
}
if (impliedDataOnly)
{
! ropt->dumpObjFlags = REQ_DATA;
ahlog(AH, 1, "implied data-only restore\n");
}
}
***************
*** 236,242 ****
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
--- 236,242 ----
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_PRE_SCHEMA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
***************
*** 278,284 ****
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!ropt->dataOnly && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
--- 278,284 ----
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!WANT_DATA(ropt->dumpObjFlags) && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
***************
*** 286,292 ****
defnDumped = false;
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
--- 286,293 ----
defnDumped = false;
! if ((WANT_PRE_SCHEMA(reqs) && WANT_PRE_SCHEMA(ropt->dumpObjFlags)) ||
! (WANT_POST_SCHEMA(reqs) && WANT_POST_SCHEMA(ropt->dumpObjFlags))) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
***************
*** 331,337 ****
/*
* If we have a data component, then process it
*/
! if ((reqs & REQ_DATA) != 0)
{
/*
* hadDumper will be set if there is genuine data component for
--- 332,338 ----
/*
* If we have a data component, then process it
*/
! if (WANT_DATA(reqs))
{
/*
* hadDumper will be set if there is genuine data component for
***************
*** 343,349 ****
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && (reqs & REQ_DATA) != 0)
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
--- 344,350 ----
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && WANT_DATA(reqs))
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
***************
*** 415,421 ****
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
--- 416,422 ----
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_PRE_SCHEMA(reqs)) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
***************
*** 473,479 ****
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
--- 474,480 ----
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
***************
*** 499,505 ****
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
--- 500,506 ----
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
***************
*** 1321,1327 ****
return NULL;
}
! teReqs
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
--- 1322,1328 ----
return NULL;
}
! int
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
***************
*** 2026,2035 ****
te->defn);
}
! static teReqs
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! teReqs res = REQ_ALL;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
--- 2027,2036 ----
te->defn);
}
! static int
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! int res = ropt->dumpObjFlags;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
***************
*** 2109,2125 ****
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
- /* Mask it if we only want schema */
- if (ropt->schemaOnly)
- res = res & REQ_SCHEMA;
-
- /* Mask it we only want data */
- if (ropt->dataOnly)
- res = res & REQ_DATA;
-
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~REQ_SCHEMA;
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
--- 2110,2118 ----
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~(REQ_PRE_SCHEMA | REQ_POST_SCHEMA);
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
Index: src/bin/pg_dump/pg_backup_archiver.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.h,v
retrieving revision 1.76
diff -c -r1.76 pg_backup_archiver.h
*** src/bin/pg_dump/pg_backup_archiver.h 7 Nov 2007 12:24:24 -0000 1.76
--- src/bin/pg_dump/pg_backup_archiver.h 20 Mar 2008 11:48:34 -0000
***************
*** 158,169 ****
STAGE_FINALIZING
} ArchiverStage;
! typedef enum
! {
! REQ_SCHEMA = 1,
! REQ_DATA = 2,
! REQ_ALL = REQ_SCHEMA + REQ_DATA
! } teReqs;
typedef struct _archiveHandle
{
--- 158,173 ----
STAGE_FINALIZING
} ArchiverStage;
! #define REQ_PRE_SCHEMA (1 << 0)
! #define REQ_DATA (1 << 1)
! #define REQ_POST_SCHEMA (1 << 2)
! #define REQ_ALL (REQ_PRE_SCHEMA + REQ_DATA + REQ_POST_SCHEMA)
!
! #define WANT_PRE_SCHEMA(req) ((req & REQ_PRE_SCHEMA) == REQ_PRE_SCHEMA)
! #define WANT_DATA(req) ((req & REQ_DATA) == REQ_DATA)
! #define WANT_POST_SCHEMA(req) ((req & REQ_POST_SCHEMA) == REQ_POST_SCHEMA)
! #define WANT_ALL(req) ((req & REQ_ALL) == REQ_ALL)
!
typedef struct _archiveHandle
{
***************
*** 317,323 ****
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern teReqs TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
--- 321,327 ----
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern int TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
Index: src/bin/pg_dump/pg_dump.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_dump.c,v
retrieving revision 1.482
diff -c -r1.482 pg_dump.c
*** src/bin/pg_dump/pg_dump.c 30 Jan 2008 18:35:55 -0000 1.482
--- src/bin/pg_dump/pg_dump.c 20 Mar 2008 11:48:34 -0000
***************
*** 78,83 ****
--- 78,90 ----
bool dataOnly;
bool aclsSkip;
+ static int use_schemaPreLoadOnly;
+ static int use_schemaPostLoadOnly;
+
+ /* groups of objects: default is we dump all groups */
+
+ int dumpObjFlags;
+
/* subquery used to convert user ID (eg, datdba) to user name */
static const char *username_subquery;
***************
*** 214,219 ****
--- 221,227 ----
int numTables;
DumpableObject **dobjs;
int numObjs;
+ int num_object_groups = 0;
int i;
bool force_password = false;
int compressLevel = -1;
***************
*** 266,271 ****
--- 274,281 ----
*/
{"disable-dollar-quoting", no_argument, &disable_dollar_quoting, 1},
{"disable-triggers", no_argument, &disable_triggers, 1},
+ {"schema-pre-load", no_argument, &use_schemaPreLoadOnly, 1},
+ {"schema-post-load", no_argument, &use_schemaPostLoadOnly, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 456,467 ****
if (optind < argc)
dbname = argv[optind];
! if (dataOnly && schemaOnly)
{
! write_msg(NULL, "options -s/--schema-only and -a/--data-only cannot be used together\n");
exit(1);
}
if (dataOnly && outputClean)
{
write_msg(NULL, "options -c/--clean and -a/--data-only cannot be used together\n");
--- 466,506 ----
if (optind < argc)
dbname = argv[optind];
! /*
! * Look for conflicting options relating to object types
! */
! if (dataOnly)
! num_object_groups++;
! if (schemaOnly)
! num_object_groups++;
! if (use_schemaPreLoadOnly == 1)
! num_object_groups++;
! if (use_schemaPostLoadOnly == 1)
! num_object_groups++;
!
! if (num_object_groups > 1)
{
! write_msg(NULL, "multiple object groups cannot be used together\n");
exit(1);
}
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (use_schemaPreLoadOnly == 1)
+ dumpObjFlags = REQ_PRE_SCHEMA;
+
+ if (use_schemaPostLoadOnly == 1)
+ dumpObjFlags = REQ_POST_SCHEMA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_PRE_SCHEMA | REQ_POST_SCHEMA);
+
if (dataOnly && outputClean)
{
write_msg(NULL, "options -c/--clean and -a/--data-only cannot be used together\n");
***************
*** 626,632 ****
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && !schemaOnly)
outputBlobs = true;
/*
--- 665,671 ----
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && WANT_PRE_SCHEMA(dumpObjFlags))
outputBlobs = true;
/*
***************
*** 635,641 ****
*/
tblinfo = getSchemaData(&numTables);
! if (!schemaOnly)
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
--- 674,680 ----
*/
tblinfo = getSchemaData(&numTables);
! if (WANT_DATA(dumpObjFlags))
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
***************
*** 689,695 ****
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && !dataOnly)
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
--- 728,734 ----
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && WANT_DATA(dumpObjFlags))
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
***************
*** 710,716 ****
ropt->noOwner = outputNoOwner;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dataOnly = dataOnly;
if (compressLevel == -1)
ropt->compression = 0;
--- 749,755 ----
ropt->noOwner = outputNoOwner;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dumpObjFlags = dumpObjFlags;
if (compressLevel == -1)
ropt->compression = 0;
***************
*** 3299,3305 ****
continue;
/* Ignore indexes of tables not to be dumped */
! if (!tbinfo->dobj.dump)
continue;
if (g_verbose)
--- 3338,3344 ----
continue;
/* Ignore indexes of tables not to be dumped */
! if (!tbinfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
continue;
if (g_verbose)
***************
*** 5034,5040 ****
int ncomments;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with catalogId, using table */
--- 5073,5079 ----
int ncomments;
/* Comments are SCHEMA not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Search for comments associated with catalogId, using table */
***************
*** 5085,5091 ****
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with relation, using table */
--- 5124,5130 ----
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Search for comments associated with relation, using table */
***************
*** 5437,5443 ****
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || dataOnly)
return;
/* don't dump dummy namespace from pre-7.3 source */
--- 5476,5482 ----
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* don't dump dummy namespace from pre-7.3 source */
***************
*** 5486,5492 ****
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || dataOnly)
return;
/* Dump out in proper style */
--- 5525,5531 ----
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Dump out in proper style */
***************
*** 6131,6137 ****
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 6170,6176 ----
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 6178,6184 ****
if (!include_everything)
return false;
/* And they're schema not data */
! if (dataOnly)
return false;
return true;
}
--- 6217,6223 ----
if (!include_everything)
return false;
/* And they're schema not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return false;
return true;
}
***************
*** 6199,6205 ****
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (dataOnly)
return;
/*
--- 6238,6244 ----
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 6439,6445 ****
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 6478,6484 ----
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 6800,6806 ****
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (dataOnly)
return;
if (OidIsValid(cast->castfunc))
--- 6839,6845 ----
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
if (OidIsValid(cast->castfunc))
***************
*** 6950,6956 ****
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || dataOnly)
return;
/*
--- 6989,6995 ----
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 7334,7340 ****
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || dataOnly)
return;
/*
--- 7373,7379 ----
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 7622,7628 ****
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || dataOnly)
return;
/*
--- 7661,7667 ----
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 7871,7877 ****
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 7910,7916 ----
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8025,8031 ****
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8064,8070 ----
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8228,8234 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8267,8273 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8297,8303 ****
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8336,8342 ----
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8382,8388 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8421,8427 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8448,8454 ****
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8487,8493 ----
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8584,8590 ****
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (dataOnly || aclsSkip)
return;
sql = createPQExpBuffer();
--- 8623,8629 ----
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (!WANT_PRE_SCHEMA(dumpObjFlags) || aclsSkip)
return;
sql = createPQExpBuffer();
***************
*** 8621,8627 ****
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (!dataOnly)
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
--- 8660,8666 ----
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (WANT_PRE_SCHEMA(dumpObjFlags))
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
***************
*** 8928,8934 ****
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || dataOnly)
return;
/* Don't print inherited defaults, either */
--- 8967,8973 ----
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Don't print inherited defaults, either */
***************
*** 9013,9019 ****
PQExpBuffer q;
PQExpBuffer delq;
! if (dataOnly)
return;
q = createPQExpBuffer();
--- 9052,9058 ----
PQExpBuffer q;
PQExpBuffer delq;
! if (!WANT_POST_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9082,9088 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 9121,9127 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9452,9458 ****
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (!dataOnly)
{
resetPQExpBuffer(delqry);
--- 9491,9497 ----
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (WANT_PRE_SCHEMA(dumpObjFlags))
{
resetPQExpBuffer(delqry);
***************
*** 9545,9551 ****
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (!schemaOnly)
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
--- 9584,9590 ----
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (WANT_PRE_SCHEMA(dumpObjFlags))
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
***************
*** 9578,9584 ****
const char *p;
int findx;
! if (dataOnly)
return;
query = createPQExpBuffer();
--- 9617,9623 ----
const char *p;
int findx;
! if (!WANT_POST_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 9779,9785 ****
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || dataOnly)
return;
/*
--- 9818,9824 ----
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
return;
/*
Index: src/bin/pg_dump/pg_restore.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_restore.c,v
retrieving revision 1.85
diff -c -r1.85 pg_restore.c
*** src/bin/pg_dump/pg_restore.c 11 Dec 2007 19:01:06 -0000 1.85
--- src/bin/pg_dump/pg_restore.c 20 Mar 2008 11:48:34 -0000
***************
*** 77,82 ****
--- 77,91 ----
static int use_setsessauth = 0;
static int disable_triggers = 0;
static int no_data_for_failed_tables = 0;
+ bool dataOnly = false;
+ bool schemaOnly = false;
+
+ static int use_schemaPreLoadOnly;
+ static int use_schemaPostLoadOnly;
+ int num_object_groups = 0;
+
+ int dumpObjFlags;
+
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
***************
*** 113,118 ****
--- 122,129 ----
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{"disable-triggers", no_argument, &disable_triggers, 1},
{"no-data-for-failed-tables", no_argument, &no_data_for_failed_tables, 1},
+ {"schema-pre-load", no_argument, &use_schemaPreLoadOnly, 1},
+ {"schema-post-load", no_argument, &use_schemaPostLoadOnly, 1},
{NULL, 0, NULL, 0}
};
***************
*** 143,149 ****
switch (c)
{
case 'a': /* Dump data only */
! opts->dataOnly = 1;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
--- 154,160 ----
switch (c)
{
case 'a': /* Dump data only */
! dataOnly = true;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
***************
*** 211,217 ****
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! opts->schemaOnly = 1;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
--- 222,228 ----
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! schemaOnly = true;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
***************
*** 289,294 ****
--- 300,340 ----
opts->useDB = 1;
}
+ /*
+ * Look for conflicting options relating to object types
+ */
+ if (dataOnly)
+ num_object_groups++;
+ if (schemaOnly)
+ num_object_groups++;
+ if (use_schemaPreLoadOnly == 1)
+ num_object_groups++;
+ if (use_schemaPostLoadOnly == 1)
+ num_object_groups++;
+
+ if (num_object_groups > 1)
+ {
+ write_msg(NULL, "multiple object groups cannot be used together\n");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (use_schemaPreLoadOnly == 1)
+ dumpObjFlags = REQ_PRE_SCHEMA;
+
+ if (use_schemaPostLoadOnly == 1)
+ dumpObjFlags = REQ_POST_SCHEMA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_PRE_SCHEMA | REQ_POST_SCHEMA);
+
opts->disable_triggers = disable_triggers;
opts->use_setsessauth = use_setsessauth;
opts->noDataForFailedTables = no_data_for_failed_tables;
* Re: [Fwd: Re: [HACKERS] pg_dump additional options for performance]
2008-05-07 14:37 [Fwd: Re: [HACKERS] pg_dump additional options for performance] Simon Riggs <simon@2ndquadrant.com>
@ 2008-06-26 22:42 ` Bruce Momjian <bruce@momjian.us>
0 siblings, 0 replies; 54+ messages in thread
From: Bruce Momjian @ 2008-06-26 22:42 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: pgsql-patches
Added to July patch queue. Thanks.
---------------------------------------------------------------------------
Simon Riggs wrote:
> Re-sending post as discussed with Bruce...
>
> On Sun, 2008-03-23 at 12:45 -0300, Alvaro Herrera wrote:
> > Bruce Momjian wrote:
> > >
> > > Added to TODO:
> > >
> > > o Allow pre/data/post files when dumping a single object, for
> > > performance reasons
> > >
> > > http://archives.postgresql.org/pgsql-hackers/2008-02/msg00205.php
> >
> > "When dumping a single object"?? Do you mean database?
>
> It would be for whatever set of objects are specified through the use of
> databases, table include/exclude switches.
>
> I've written a patch that implements these new switches on the commands
> as shown
>
> pg_dump --schema-pre-load
> pg_dump --schema-post-load
>
> pg_restore --schema-pre-load
> pg_restore --schema-post-load
>
> I have not implemented --schema-pre-file=xxx style because they don't
> make any sense when using pg_restore in direct database connection mode.
> On reflection I don't see any particular need to produce multiple files
> as output, which just complicates an already horrendous user interface.
>
> This is a minimal set of changes and includes nothing at all about
> directories, parallelisation in the code etc..
>
> This has the following use cases amongst others...
>
> * dump everything to a file, then use pg_restore first --schema-pre-load
> and then --data-only directly into the database, then pg_restore
> --schema-post-load to a file so we can edit that file into multiple
> pieces to allow index creation in parallel
>
> * dump of database into multiple files by manually specifying which
> tables go where, then reload in parallel using multiple psql sessions
>
>
> The patch tests OK after some testing, though without a test suite that
> probably isn't more than a few percentage points of all the possible
> code paths. There are no docs for it, as yet.
>
> ---
>
> Further thinking on this....
>
> Some further refinement might replace --data-only and --schema-only with
> --want-schema-pre
> --want-data
> --want-schema-post
> --want-schema (same as --want-schema-pre --want-schema-post)
>
> These could be used together e.g. --want-schema-pre --want-data
> whereas the existing --data-only type switches cannot.
>
> Which would be a straightforward and useful change to the enclosed
> patch.
>
> That way of doing things is hierarchically extensible to include further
> subdivisions of the set of SQL commands produced, e.g. divide
> --want-post-schema into objects required to support various inter-table
> dependencies and those that don't such as additional indexes. I don't
> personally think we need that though.
>
>
> Comments?
>
> --
> Simon Riggs
> 2ndQuadrant http://www.2ndQuadrant.com
[ Attachment, skipping... ]
>
> --
> Sent via pgsql-patches mailing list (pgsql-patches@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-patches
--
Bruce Momjian <bruce@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
^ permalink raw reply [nested|flat] 54+ messages in thread
* pg_dump additional options for performance
@ 2008-07-20 03:07 Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-20 03:07 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: pgsql-patches
Simon,
I agree with adding these options in general, since I find myself
frustrated by having to vi huge dumps to change simple schema things.
A couple of comments on the patch though:
- Conflicting option handling
I think we are doing our users a disservice by putting it on them to
figure out exactly what:
multiple object groups cannot be used together
means to them. You and I may understand what an "object group" is,
and why there can be only one, but it's a great deal less clear than
the prior message of
options -s/--schema-only and -a/--data-only cannot be used together
My suggestion would be to either list out the specific options which
can't be used together, as was done previously, or add a bit of (I
realize, boring) code and actually tell the user which of the
conflicting options were used.
- Documentation
When writing the documentation I would stress that "pre-schema" and
"post-schema" be defined in terms of PostgreSQL objects and why they
are pre vs. post.
- Technically, the patch needs to be updated slightly since another
pg_dump-related patch was committed recently which also added
options and thus causes a conflict.
Beyond those minor points, the patch looks good to me.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-20 04:47 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-07-20 04:47 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-patches
On Sat, 2008-07-19 at 23:07 -0400, Stephen Frost wrote:
> Simon,
>
> I agree with adding these options in general, since I find myself
> frustrated by having to vi huge dumps to change simple schema things.
> A couple of comments on the patch though:
>
> - Conflicting option handling
> I think we are doing our users a disservice by putting it on them to
> figure out exactly what:
> multiple object groups cannot be used together
> means to them. You and I may understand what an "object group" is,
> and why there can be only one, but it's a great deal less clear than
> the prior message of
> options -s/--schema-only and -a/--data-only cannot be used together
> My suggestion would be to either list out the specific options which
> can't be used together, as was done previously, or add a bit of (I
> realize, boring) code and actually tell the user which of the
> conflicting options were used.
>
> - Documentation
> When writing the documentation I would stress that "pre-schema" and
> "post-schema" be defined in terms of PostgreSQL objects and why they
> are pre vs. post.
>
> - Technically, the patch needs to be updated slightly since another
> pg_dump-related patch was committed recently which also added
> options and thus causes a conflict.
>
> Beyond those minor points, the patch looks good to me.
Thanks for the review. I'll make the changes you suggest.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-20 06:37 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-07-20 06:37 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-patches
On Sun, 2008-07-20 at 05:47 +0100, Simon Riggs wrote:
> On Sat, 2008-07-19 at 23:07 -0400, Stephen Frost wrote:
> > Simon,
> >
> > I agree with adding these options in general, since I find myself
> > frustrated by having to vi huge dumps to change simple schema things.
> > A couple of comments on the patch though:
> >
> > - Conflicting option handling
> > I think we are doing our users a disservice by putting it on them to
> > figure out exactly what:
> > multiple object groups cannot be used together
> > means to them. You and I may understand what an "object group" is,
> > and why there can be only one, but it's a great deal less clear than
> > the prior message of
> > options -s/--schema-only and -a/--data-only cannot be used together
> > My suggestion would be to either list out the specific options which
> > can't be used together, as was done previously, or add a bit of (I
> > realize, boring) code and actually tell the user which of the
> > conflicting options were used.
> >
> > - Documentation
> > When writing the documentation I would stress that "pre-schema" and
> > "post-schema" be defined in terms of PostgreSQL objects and why they
> > are pre vs. post.
> >
> > - Technically, the patch needs to be updated slightly since another
> > pg_dump-related patch was committed recently which also added
> > options and thus causes a conflict.
> >
> > Beyond those minor points, the patch looks good to me.
>
> Thanks for the review. I'll make the changes you suggest.
Patch updated to head, plus changes/docs requested.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Attachments:
pg_dump_prepost.v3.patch (text/x-patch, 36.7K) download
Index: doc/src/sgml/ref/pg_dump.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_dump.sgml,v
retrieving revision 1.102
diff -c -r1.102 pg_dump.sgml
*** doc/src/sgml/ref/pg_dump.sgml 13 Apr 2008 03:49:21 -0000 1.102
--- doc/src/sgml/ref/pg_dump.sgml 20 Jul 2008 06:33:30 -0000
***************
*** 133,139 ****
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
--- 133,140 ----
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> or <option>--schema-pre-load</> or
! <option>--schema-post-load</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
***************
*** 443,448 ****
--- 444,471 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-pre-load</option></term>
+ <listitem>
+ <para>
+ Dump only the object definitions (schema) required to load data. Dumps
+ exactly what <option>--schema-only</> would dump, but only those
+ statements before the data load.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--schema-post-load</option></term>
+ <listitem>
+ <para>
+ Dump only the object definitions (schema) required after data has been
+ loaded. Dumps exactly what <option>--schema-only</> would dump, but
+ only those statements after the data load.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 774,779 ****
--- 797,830 ----
</para>
<para>
+ The output of pg_dump can be notionally divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Pre-Schema - objects required before data loading, such as
+ <command>CREATE TABLE</command>.
+ This part can be requested using <option>--schema-pre-load</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Post-Schema - objects required after data loading, such as
+ <command>ALTER TABLE</command> and <command>CREATE INDEX</command>.
+ This part can be requested using <option>--schema-post-load</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
Because <application>pg_dump</application> is used to transfer data
to newer versions of <productname>PostgreSQL</>, the output of
<application>pg_dump</application> can be loaded into
Index: doc/src/sgml/ref/pg_restore.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_restore.sgml,v
retrieving revision 1.75
diff -c -r1.75 pg_restore.sgml
*** doc/src/sgml/ref/pg_restore.sgml 13 Apr 2008 03:49:21 -0000 1.75
--- doc/src/sgml/ref/pg_restore.sgml 20 Jul 2008 06:33:18 -0000
***************
*** 321,326 ****
--- 321,350 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-post-load</option></term>
+ <listitem>
+ <para>
+ Dump only the object definitions (schema) required after data has been
+ loaded. Dumps exactly what <option>--schema-only</> would dump, but
+ only those statements after the data load.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>-S <replaceable class="parameter">username</replaceable></option></term>
+ <term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
+ <listitem>
+ <para>
+ Specify the superuser user name to use when disabling triggers.
+ This is only relevant if <option>--disable-triggers</> is used.
+ (Usually, it's better to leave this out, and instead start the
+ resulting script as superuser.)
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 572,577 ****
--- 596,629 ----
</para>
<para>
+ The actions of pg_restore can be notionally divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Pre-Schema - objects required before data loading, such as
+ <command>CREATE TABLE</command>.
+ This part can be requested using <option>--schema-pre-load</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Post-Schema - objects required after data loading, such as
+ <command>ALTER TABLE</command> and <command>CREATE INDEX</command>.
+ This part can be requested using <option>--schema-post-load</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
The limitations of <application>pg_restore</application> are detailed below.
<itemizedlist>
Index: src/bin/pg_dump/pg_backup.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup.h,v
retrieving revision 1.47
diff -c -r1.47 pg_backup.h
*** src/bin/pg_dump/pg_backup.h 13 Apr 2008 03:49:21 -0000 1.47
--- src/bin/pg_dump/pg_backup.h 20 Jul 2008 05:19:34 -0000
***************
*** 89,95 ****
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dataOnly;
int dropSchema;
char *filename;
int schemaOnly;
--- 89,95 ----
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dumpObjFlags; /* which objects types to dump */
int dropSchema;
char *filename;
int schemaOnly;
Index: src/bin/pg_dump/pg_backup_archiver.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.c,v
retrieving revision 1.157
diff -c -r1.157 pg_backup_archiver.c
*** src/bin/pg_dump/pg_backup_archiver.c 4 May 2008 08:32:21 -0000 1.157
--- src/bin/pg_dump/pg_backup_archiver.c 20 Jul 2008 05:19:34 -0000
***************
*** 56,62 ****
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static teReqs _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
--- 56,62 ----
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static int _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
***************
*** 129,135 ****
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! teReqs reqs;
OutputContext sav;
bool defnDumped;
--- 129,135 ----
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! int reqs;
OutputContext sav;
bool defnDumped;
***************
*** 175,193 ****
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then we set the dataOnly flag.
*
! * We could scan for wanted TABLE entries, but that is not the same as
! * dataOnly. At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!ropt->dataOnly)
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0)
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
--- 175,193 ----
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then say we only want DATA type objects.
*
! * We could scan for wanted TABLE entries, but that is not the same.
! * At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!WANT_DATA(ropt->dumpObjFlags))
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_PRE_SCHEMA(reqs) || WANT_POST_SCHEMA(reqs))
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
***************
*** 195,201 ****
}
if (impliedDataOnly)
{
! ropt->dataOnly = impliedDataOnly;
ahlog(AH, 1, "implied data-only restore\n");
}
}
--- 195,201 ----
}
if (impliedDataOnly)
{
! ropt->dumpObjFlags = REQ_DATA;
ahlog(AH, 1, "implied data-only restore\n");
}
}
***************
*** 236,242 ****
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
--- 236,242 ----
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_PRE_SCHEMA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
***************
*** 278,284 ****
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!ropt->dataOnly && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
--- 278,284 ----
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!WANT_DATA(ropt->dumpObjFlags) && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
***************
*** 286,292 ****
defnDumped = false;
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
--- 286,293 ----
defnDumped = false;
! if ((WANT_PRE_SCHEMA(reqs) && WANT_PRE_SCHEMA(ropt->dumpObjFlags)) ||
! (WANT_POST_SCHEMA(reqs) && WANT_POST_SCHEMA(ropt->dumpObjFlags))) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
***************
*** 331,337 ****
/*
* If we have a data component, then process it
*/
! if ((reqs & REQ_DATA) != 0)
{
/*
* hadDumper will be set if there is genuine data component for
--- 332,338 ----
/*
* If we have a data component, then process it
*/
! if (WANT_DATA(reqs))
{
/*
* hadDumper will be set if there is genuine data component for
***************
*** 343,349 ****
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && (reqs & REQ_DATA) != 0)
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
--- 344,350 ----
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && WANT_DATA(reqs))
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
***************
*** 415,421 ****
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
--- 416,422 ----
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_PRE_SCHEMA(reqs)) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
***************
*** 473,479 ****
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
--- 474,480 ----
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
***************
*** 499,505 ****
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
--- 500,506 ----
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
***************
*** 1321,1327 ****
return NULL;
}
! teReqs
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
--- 1322,1328 ----
return NULL;
}
! int
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
***************
*** 2026,2035 ****
te->defn);
}
! static teReqs
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! teReqs res = REQ_ALL;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
--- 2027,2036 ----
te->defn);
}
! static int
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! int res = ropt->dumpObjFlags;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
***************
*** 2109,2125 ****
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
- /* Mask it if we only want schema */
- if (ropt->schemaOnly)
- res = res & REQ_SCHEMA;
-
- /* Mask it we only want data */
- if (ropt->dataOnly)
- res = res & REQ_DATA;
-
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~REQ_SCHEMA;
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
--- 2110,2118 ----
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~(REQ_PRE_SCHEMA | REQ_POST_SCHEMA);
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
Index: src/bin/pg_dump/pg_backup_archiver.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.h,v
retrieving revision 1.76
diff -c -r1.76 pg_backup_archiver.h
*** src/bin/pg_dump/pg_backup_archiver.h 7 Nov 2007 12:24:24 -0000 1.76
--- src/bin/pg_dump/pg_backup_archiver.h 20 Jul 2008 05:19:34 -0000
***************
*** 158,169 ****
STAGE_FINALIZING
} ArchiverStage;
! typedef enum
! {
! REQ_SCHEMA = 1,
! REQ_DATA = 2,
! REQ_ALL = REQ_SCHEMA + REQ_DATA
! } teReqs;
typedef struct _archiveHandle
{
--- 158,173 ----
STAGE_FINALIZING
} ArchiverStage;
! #define REQ_PRE_SCHEMA (1 << 0)
! #define REQ_DATA (1 << 1)
! #define REQ_POST_SCHEMA (1 << 2)
! #define REQ_ALL (REQ_PRE_SCHEMA + REQ_DATA + REQ_POST_SCHEMA)
!
! #define WANT_PRE_SCHEMA(req) ((req & REQ_PRE_SCHEMA) == REQ_PRE_SCHEMA)
! #define WANT_DATA(req) ((req & REQ_DATA) == REQ_DATA)
! #define WANT_POST_SCHEMA(req) ((req & REQ_POST_SCHEMA) == REQ_POST_SCHEMA)
! #define WANT_ALL(req) ((req & REQ_ALL) == REQ_ALL)
!
typedef struct _archiveHandle
{
***************
*** 317,323 ****
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern teReqs TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
--- 321,327 ----
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern int TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
Index: src/bin/pg_dump/pg_dump.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_dump.c,v
retrieving revision 1.496
diff -c -r1.496 pg_dump.c
*** src/bin/pg_dump/pg_dump.c 18 Jul 2008 03:32:52 -0000 1.496
--- src/bin/pg_dump/pg_dump.c 20 Jul 2008 05:57:58 -0000
***************
*** 72,77 ****
--- 72,81 ----
bool dataOnly;
bool aclsSkip;
+ /* groups of objects: default is we dump all groups */
+
+ int dumpObjFlags;
+
/* subquery used to convert user ID (eg, datdba) to user name */
static const char *username_subquery;
***************
*** 224,231 ****
RestoreOptions *ropt;
static int disable_triggers = 0;
! static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
--- 228,237 ----
RestoreOptions *ropt;
static int disable_triggers = 0;
! static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ static int use_schemaPreLoadOnly;
+ static int use_schemaPostLoadOnly;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
***************
*** 265,270 ****
--- 271,278 ----
{"disable-dollar-quoting", no_argument, &disable_dollar_quoting, 1},
{"disable-triggers", no_argument, &disable_triggers, 1},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-pre-load", no_argument, &use_schemaPreLoadOnly, 1},
+ {"schema-post-load", no_argument, &use_schemaPostLoadOnly, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 456,467 ****
if (optind < argc)
dbname = argv[optind];
! if (dataOnly && schemaOnly)
{
! write_msg(NULL, "options -s/--schema-only and -a/--data-only cannot be used together\n");
exit(1);
}
if (dataOnly && outputClean)
{
write_msg(NULL, "options -c/--clean and -a/--data-only cannot be used together\n");
--- 464,504 ----
if (optind < argc)
dbname = argv[optind];
! /*
! * Look for conflicting options relating to object groupings
! */
! if (schemaOnly && dataOnly)
! {
! write_msg(NULL, "options %s and %s cannot be used together\n",
! "-s/--schema-only", "-a/--data-only");
! exit(1);
! }
! else if ((schemaOnly || dataOnly) &&
! (use_schemaPreLoadOnly == 1 || use_schemaPostLoadOnly == 1))
{
! write_msg(NULL, "options %s and %s cannot be used together\n",
! schemaOnly ? "-s/--schema-only" : "-a/--data-only",
! use_schemaPostLoadOnly == 1 ? "--schema-post-load" : "--schema-pre-load ");
exit(1);
}
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (use_schemaPreLoadOnly == 1)
+ dumpObjFlags = REQ_PRE_SCHEMA;
+
+ if (use_schemaPostLoadOnly == 1)
+ dumpObjFlags = REQ_POST_SCHEMA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_PRE_SCHEMA | REQ_POST_SCHEMA);
+
if (dataOnly && outputClean)
{
write_msg(NULL, "options -c/--clean and -a/--data-only cannot be used together\n");
***************
*** 638,644 ****
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && !schemaOnly)
outputBlobs = true;
/*
--- 675,681 ----
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && WANT_PRE_SCHEMA(dumpObjFlags))
outputBlobs = true;
/*
***************
*** 650,656 ****
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (!schemaOnly)
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
--- 687,693 ----
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (WANT_DATA(dumpObjFlags))
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
***************
*** 704,710 ****
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && !dataOnly)
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
--- 741,747 ----
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && WANT_DATA(dumpObjFlags))
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
***************
*** 726,732 ****
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dataOnly = dataOnly;
if (compressLevel == -1)
ropt->compression = 0;
--- 763,769 ----
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dumpObjFlags = dumpObjFlags;
if (compressLevel == -1)
ropt->compression = 0;
***************
*** 3389,3395 ****
continue;
/* Ignore indexes of tables not to be dumped */
! if (!tbinfo->dobj.dump)
continue;
if (g_verbose)
--- 3426,3432 ----
continue;
/* Ignore indexes of tables not to be dumped */
! if (!tbinfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
continue;
if (g_verbose)
***************
*** 5140,5146 ****
int ncomments;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with catalogId, using table */
--- 5177,5183 ----
int ncomments;
/* Comments are SCHEMA not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Search for comments associated with catalogId, using table */
***************
*** 5191,5197 ****
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with relation, using table */
--- 5228,5234 ----
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Search for comments associated with relation, using table */
***************
*** 5543,5549 ****
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || dataOnly)
return;
/* don't dump dummy namespace from pre-7.3 source */
--- 5580,5586 ----
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* don't dump dummy namespace from pre-7.3 source */
***************
*** 5592,5598 ****
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || dataOnly)
return;
/* Dump out in proper style */
--- 5629,5635 ----
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Dump out in proper style */
***************
*** 6237,6243 ****
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 6274,6280 ----
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 6284,6290 ****
if (!include_everything)
return false;
/* And they're schema not data */
! if (dataOnly)
return false;
return true;
}
--- 6321,6327 ----
if (!include_everything)
return false;
/* And they're schema not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return false;
return true;
}
***************
*** 6305,6311 ****
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (dataOnly)
return;
/*
--- 6342,6348 ----
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 6565,6571 ****
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 6602,6608 ----
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 6960,6966 ****
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (dataOnly)
return;
if (OidIsValid(cast->castfunc))
--- 6997,7003 ----
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
if (OidIsValid(cast->castfunc))
***************
*** 7110,7116 ****
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || dataOnly)
return;
/*
--- 7147,7153 ----
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 7494,7500 ****
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || dataOnly)
return;
/*
--- 7531,7537 ----
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 7802,7808 ****
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || dataOnly)
return;
/*
--- 7839,7845 ----
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 8071,8077 ****
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8108,8114 ----
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8225,8231 ****
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8262,8268 ----
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8428,8434 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8465,8471 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8497,8503 ****
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8534,8540 ----
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8582,8588 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8619,8625 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8648,8654 ****
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8685,8691 ----
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8784,8790 ****
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (dataOnly || aclsSkip)
return;
sql = createPQExpBuffer();
--- 8821,8827 ----
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (!WANT_PRE_SCHEMA(dumpObjFlags) || aclsSkip)
return;
sql = createPQExpBuffer();
***************
*** 8821,8827 ****
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (!dataOnly)
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
--- 8858,8864 ----
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (WANT_PRE_SCHEMA(dumpObjFlags))
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
***************
*** 9128,9134 ****
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || dataOnly)
return;
/* Don't print inherited defaults, either */
--- 9165,9171 ----
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Don't print inherited defaults, either */
***************
*** 9213,9219 ****
PQExpBuffer q;
PQExpBuffer delq;
! if (dataOnly)
return;
q = createPQExpBuffer();
--- 9250,9256 ----
PQExpBuffer q;
PQExpBuffer delq;
! if (!WANT_POST_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9282,9288 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 9319,9325 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9675,9681 ****
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (!dataOnly)
{
resetPQExpBuffer(delqry);
--- 9712,9718 ----
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (WANT_PRE_SCHEMA(dumpObjFlags))
{
resetPQExpBuffer(delqry);
***************
*** 9778,9784 ****
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (!schemaOnly)
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
--- 9815,9821 ----
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (WANT_PRE_SCHEMA(dumpObjFlags))
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
***************
*** 9811,9817 ****
const char *p;
int findx;
! if (dataOnly)
return;
query = createPQExpBuffer();
--- 9848,9854 ----
const char *p;
int findx;
! if (!WANT_POST_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 10019,10025 ****
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || dataOnly)
return;
/*
--- 10056,10062 ----
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
return;
/*
Index: src/bin/pg_dump/pg_restore.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_restore.c,v
retrieving revision 1.88
diff -c -r1.88 pg_restore.c
*** src/bin/pg_dump/pg_restore.c 13 Apr 2008 03:49:22 -0000 1.88
--- src/bin/pg_dump/pg_restore.c 20 Jul 2008 05:57:51 -0000
***************
*** 78,83 ****
--- 78,90 ----
static int no_data_for_failed_tables = 0;
static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ bool dataOnly = false;
+ bool schemaOnly = false;
+
+ static int use_schemaPreLoadOnly;
+ static int use_schemaPostLoadOnly;
+
+ int dumpObjFlags;
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
***************
*** 114,119 ****
--- 121,128 ----
{"disable-triggers", no_argument, &disable_triggers, 1},
{"no-data-for-failed-tables", no_argument, &no_data_for_failed_tables, 1},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-pre-load", no_argument, &use_schemaPreLoadOnly, 1},
+ {"schema-post-load", no_argument, &use_schemaPostLoadOnly, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 145,151 ****
switch (c)
{
case 'a': /* Dump data only */
! opts->dataOnly = 1;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
--- 154,160 ----
switch (c)
{
case 'a': /* Dump data only */
! dataOnly = true;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
***************
*** 213,219 ****
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! opts->schemaOnly = 1;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
--- 222,228 ----
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! schemaOnly = true;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
***************
*** 295,300 ****
--- 304,344 ----
opts->useDB = 1;
}
+ /*
+ * Look for conflicting options relating to object groupings
+ */
+ if (schemaOnly && dataOnly)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "-s/--schema-only", "-a/--data-only");
+ exit(1);
+ }
+ else if ((schemaOnly || dataOnly) &&
+ (use_schemaPreLoadOnly == 1 || use_schemaPostLoadOnly == 1))
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ schemaOnly ? "-s/--schema-only" : "-a/--data-only",
+ use_schemaPostLoadOnly == 1 ? "--schema-post-load" : "--schema-pre-load ");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (use_schemaPreLoadOnly == 1)
+ dumpObjFlags = REQ_PRE_SCHEMA;
+
+ if (use_schemaPostLoadOnly == 1)
+ dumpObjFlags = REQ_POST_SCHEMA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_PRE_SCHEMA | REQ_POST_SCHEMA);
+
opts->disable_triggers = disable_triggers;
opts->noDataForFailedTables = no_data_for_failed_tables;
opts->noTablespace = outputNoTablespaces;
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-20 21:43 ` Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-20 21:43 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: pgsql-patches
Simon,
* Simon Riggs (simon@2ndquadrant.com) wrote:
> On Sun, 2008-07-20 at 05:47 +0100, Simon Riggs wrote:
> > On Sat, 2008-07-19 at 23:07 -0400, Stephen Frost wrote:
[...]
> > > - Conflicting option handling
Thanks for putting in the extra code to explicitly indicate which
conflicting options were used.
> > > - Documentation
> > > When writing the documentation I would stress that "pre-schema" and
> > > "post-schema" be defined in terms of PostgreSQL objects and why they
> > > are pre vs. post.
Perhaps this is up for some debate, but I find the documentation added
for these options to be lacking the definitions I was looking for, and
the explanation of why they are what they are. I'm also not sure I
agree with the "Pre-Schema" and "Post-Schema" nomenclature as it doesn't
really fit with the option names or what they do. Would you consider:
<term><option>--schema-pre-load</option></term>
<listitem>
<para>
Pre-Data Load - Minimum amount of the schema required before data
loading can begin. This consists mainly of creating the tables
using the <command>CREATE TABLE</command>.
This part can be requested using <option>--schema-pre-load</>.
</para>
</listitem>
<term><option>--schema-post-load</option></term>
<listitem>
<para>
Post-Data Load - The rest of the schema definition, including keys,
indexes, etc. By putting keys and indexes after the data has been
loaded the whole process of restoring data is much faster. This is
because it is faster to build indexes and check keys in bulk than
piecemeal as the data is loaded.
This part can be requested using <option>--schema-post-load</>.
</para>
</listitem>
Even this doesn't cover everything though- it's too focused on tables
and data loading. Where do functions go? What about types?
A couple of additional points:
- The command-line help hasn't been updated. Clearly, that also needs
to be done to consider the documentation aspect complete.
- There appears to be a bit of mistakenly included additions. The
patch to pg_restore.sgml attempts to add in documentation for
--superuser. I'm guessing that was unintentional, and looks like
just a mistaken extra copy&paste.
> > > - Technically, the patch needs to be updated slightly since another
> > > pg_dump-related patch was committed recently which also added
> > > options and thus causes a conflict.
I think this might have just happened again, funny enough. It's
something that a committer could perhaps fix, but if you're reworking
the patch anyway...
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-20 21:59 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-07-20 21:59 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-patches
On Sun, 2008-07-20 at 17:43 -0400, Stephen Frost wrote:
> Perhaps this is up for some debate, but I find the documentation added
> for these options to be lacking the definitions I was looking for, and
> the explanation of why they are what they are. I'm also not sure I
> agree with the "Pre-Schema" and "Post-Schema" nomenclature as it doesn't
> really fit with the option names or what they do. Would you consider:
Will reword.
> Even this doesn't cover everything though- it's too focused on tables
> and data loading. Where do functions go? What about types?
Yes, it is focused on tables and data loading. What about
functions/types? No relevance here.
> - The command-line help hasn't been updated. Clearly, that also needs
> to be done to consider the documentation aspect complete.
>
> - There appears to be a bit of mistakenly included additions. The
> patch to pg_restore.sgml attempts to add in documentation for
> --superuser. I'm guessing that was unintentional, and looks like
> just a mistaken extra copy&paste.
Thanks, will do.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-21 01:18 ` Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 06:11 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 2 replies; 54+ messages in thread
From: Stephen Frost @ 2008-07-21 01:18 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: pgsql-patches
* Simon Riggs (simon@2ndquadrant.com) wrote:
> On Sun, 2008-07-20 at 17:43 -0400, Stephen Frost wrote:
> > Even this doesn't cover everything though- it's too focused on tables
> > and data loading. Where do functions go? What about types?
>
> Yes, it is focused on tables and data loading. What about
> functions/types? No relevance here.
I don't see how they're not relevant, it's not like they're being
excluded and in fact they show up in the pre-load output. Heck, even if
they *were* excluded, that should be made clear in the documentation
(either be an explicit include list, or saying they're excluded).
Part of what's driving this is making sure we have a plan for future
objects and where they'll go. Perhaps it would be enough to just say
"pre-load is everything in the schema, except things which are faster
done in bulk (eg: indexes, keys)". I don't think it's right to say
pre-load is "only object definitions required to load data" when it
includes functions and ACLs though.
Hopefully my suggestion and these comments will get us to a happy
middle-ground.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-21 01:49 ` daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
1 sibling, 1 reply; 54+ messages in thread
From: daveg @ 2008-07-21 01:49 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
On Sun, Jul 20, 2008 at 09:18:29PM -0400, Stephen Frost wrote:
> * Simon Riggs (simon@2ndquadrant.com) wrote:
> > On Sun, 2008-07-20 at 17:43 -0400, Stephen Frost wrote:
> > > Even this doesn't cover everything though- it's too focused on tables
> > > and data loading. Where do functions go? What about types?
> >
> > Yes, it is focused on tables and data loading. What about
> > functions/types? No relevance here.
>
> I don't see how they're not relevant, it's not like they're being
> excluded and in fact they show up in the pre-load output. Heck, even if
> they *were* excluded, that should be made clear in the documentation
> (either be an explicit include list, or saying they're excluded).
>
> Part of what's driving this is making sure we have a plan for future
> objects and where they'll go. Perhaps it would be enough to just say
> "pre-load is everything in the schema, except things which are faster
> done in bulk (eg: indexes, keys)". I don't think it's right to say
> pre-load is "only object definitions required to load data" when it
> includes functions and ACLs though.
>
> Hopefully my suggestion and these comments will get us to a happy
> middle-ground.
One observation, indexes should be built right after the table data
is loaded for each table, this way, the index build gets a hot cache
for the table data instead of having to re-read it later as we do now.
-dg
--
David Gould daveg@sonic.net 510 536 1443 510 282 0869
If simplicity worked, the world would be overrun with insects.
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
@ 2008-07-21 01:58 ` Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-21 01:58 UTC (permalink / raw)
To: daveg <daveg@sonic.net>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
* daveg (daveg@sonic.net) wrote:
> One observation, indexes should be built right after the table data
> is loaded for each table, this way, the index build gets a hot cache
> for the table data instead of having to re-read it later as we do now.
That's not how pg_dump has traditionally worked, and the point of this
patch is to add options to easily segregate the main pieces of the
existing pg_dump output (main schema definition, data dump, key/index
building). You suggestion brings up an interesting point that should
pg_dump's traditional output structure change the "--schema-post-load"
set of objects wouldn't be as clear to newcomers since the load and the
indexes would be interleaved in the regular output.
I'd be curious about the performance impact this has on an actual load
too. It would probably be more valuable on smaller loads where it would
have less of an impact anyway than on loads larger than the cache size.
Still, not an issue for this patch, imv.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-21 03:34 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:54 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
0 siblings, 2 replies; 54+ messages in thread
From: Tom Lane @ 2008-07-21 03:34 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: daveg <daveg@sonic.net>; Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
Stephen Frost <sfrost@snowman.net> writes:
> * daveg (daveg@sonic.net) wrote:
>> One observation, indexes should be built right after the table data
>> is loaded for each table, this way, the index build gets a hot cache
>> for the table data instead of having to re-read it later as we do now.
> That's not how pg_dump has traditionally worked, and the point of this
> patch is to add options to easily segregate the main pieces of the
> existing pg_dump output (main schema definition, data dump, key/index
> building). You suggestion brings up an interesting point that should
> pg_dump's traditional output structure change the "--schema-post-load"
> set of objects wouldn't be as clear to newcomers since the load and the
> indexes would be interleaved in the regular output.
Yeah. Also, that is pushing into an entirely different line of
development, which is to enable multithreaded pg_restore. The patch
at hand is necessarily incompatible with that type of operation, and
wouldn't be used together with it.
As far as the documentation/definition aspect goes, I think it should
just say the parts are
* stuff needed before you can load the data
* the data
* stuff needed after loading the data
and not try to be any more specific than that. There are corner cases
that will turn any simple breakdown into a lie, and I doubt that it's
worth trying to explain them all. (Take a close look at the dependency
loop breaking logic in pg_dump if you doubt this.)
I hadn't realized that Simon was using "pre-schema" and "post-schema"
to name the first and third parts. I'd agree that this is confusing
nomenclature: it looks like it's trying to say that the data is the
schema, and the schema is not! How about "pre-data and "post-data"?
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-21 06:17 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-21 06:47 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
1 sibling, 2 replies; 54+ messages in thread
From: Simon Riggs @ 2008-07-21 06:17 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Stephen Frost <sfrost@snowman.net>; daveg <daveg@sonic.net>; pgsql-patches
On Sun, 2008-07-20 at 23:34 -0400, Tom Lane wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * daveg (daveg@sonic.net) wrote:
> >> One observation, indexes should be built right after the table data
> >> is loaded for each table, this way, the index build gets a hot cache
> >> for the table data instead of having to re-read it later as we do now.
>
> > That's not how pg_dump has traditionally worked, and the point of this
> > patch is to add options to easily segregate the main pieces of the
> > existing pg_dump output (main schema definition, data dump, key/index
> > building). You suggestion brings up an interesting point that should
> > pg_dump's traditional output structure change the "--schema-post-load"
> > set of objects wouldn't be as clear to newcomers since the load and the
> > indexes would be interleaved in the regular output.
Stephen: Agreed.
> Yeah. Also, that is pushing into an entirely different line of
> development, which is to enable multithreaded pg_restore. The patch
> at hand is necessarily incompatible with that type of operation, and
> wouldn't be used together with it.
>
> As far as the documentation/definition aspect goes, I think it should
> just say the parts are
> * stuff needed before you can load the data
> * the data
> * stuff needed after loading the data
> and not try to be any more specific than that. There are corner cases
> that will turn any simple breakdown into a lie, and I doubt that it's
> worth trying to explain them all. (Take a close look at the dependency
> loop breaking logic in pg_dump if you doubt this.)
Tom: Agreed.
> I hadn't realized that Simon was using "pre-schema" and "post-schema"
> to name the first and third parts. I'd agree that this is confusing
> nomenclature: it looks like it's trying to say that the data is the
> schema, and the schema is not! How about "pre-data and "post-data"?
OK by me. Any other takers?
I also suggested having three options
--want-pre-schema
--want-data
--want-post-schema
so we could ask for any or all parts in the one dump. --data-only and
--schema-only are negative options so don't allow this.
(I don't like those names either, just thinking about capabilities)
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-21 06:47 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 15:19 ` Re: pg_dump additional options for performance Andrew Dunstan <andrew@dunslane.net>
1 sibling, 1 reply; 54+ messages in thread
From: Tom Lane @ 2008-07-21 06:47 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; daveg <daveg@sonic.net>; pgsql-patches
Simon Riggs <simon@2ndquadrant.com> writes:
> I also suggested having three options
> --want-pre-schema
> --want-data
> --want-post-schema
> so we could ask for any or all parts in the one dump. --data-only and
> --schema-only are negative options so don't allow this.
> (I don't like those names either, just thinking about capabilities)
Maybe invert the logic?
--omit-pre-data
--omit-data
--omit-post-data
Not wedded to these either, just tossing out an idea...
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 06:47 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-21 15:19 ` Andrew Dunstan <andrew@dunslane.net>
2008-07-21 15:38 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 54+ messages in thread
From: Andrew Dunstan @ 2008-07-21 15:19 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; daveg <daveg@sonic.net>; pgsql-patches
Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
>
>> I also suggested having three options
>> --want-pre-schema
>> --want-data
>> --want-post-schema
>> so we could ask for any or all parts in the one dump. --data-only and
>> --schema-only are negative options so don't allow this.
>> (I don't like those names either, just thinking about capabilities)
>>
>
> Maybe invert the logic?
>
> --omit-pre-data
> --omit-data
> --omit-post-data
>
> Not wedded to these either, just tossing out an idea...
>
>
>
Please, no. Negative logic seems likely to cause endless confusion.
I'd even be happier with --schema-part-1 and --schema-part-2 if we can't
find some more expressive way of designating them.
cheers
andrew
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 06:47 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 15:19 ` Re: pg_dump additional options for performance Andrew Dunstan <andrew@dunslane.net>
@ 2008-07-21 15:38 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 54+ messages in thread
From: Tom Lane @ 2008-07-21 15:38 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; daveg <daveg@sonic.net>; pgsql-patches
Andrew Dunstan <andrew@dunslane.net> writes:
> Tom Lane wrote:
>> Maybe invert the logic?
>> --omit-pre-data
>> --omit-data
>> --omit-post-data
> Please, no. Negative logic seems likely to cause endless confusion.
I think it might actually be less confusing, because with this approach,
each switch has an identifiable default (no) and setting it doesn't
cause side-effects on settings of other switches. The interactions of
the switches as Simon presents 'em seem less than obvious.
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-21 11:56 ` Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
1 sibling, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-21 11:56 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; daveg <daveg@sonic.net>; pgsql-patches
Simon,
* Simon Riggs (simon@2ndquadrant.com) wrote:
> > I hadn't realized that Simon was using "pre-schema" and "post-schema"
> > to name the first and third parts. I'd agree that this is confusing
> > nomenclature: it looks like it's trying to say that the data is the
> > schema, and the schema is not! How about "pre-data and "post-data"?
>
> OK by me. Any other takers?
Having the command-line options be "--schema-pre-data" and
"--schema-post-data" is fine with me. Leaving them the way they are is
also fine by me. It's the documentation (back to pg_dump.sgml,
~774/~797) that starts talking about Pre-Schema and Post-Schema.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-23 16:40 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-07-23 16:40 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; daveg <daveg@sonic.net>; pgsql-patches@postgresql.org, Andrew Dunstan <andrew@dunslane.net>
On Mon, 2008-07-21 at 07:56 -0400, Stephen Frost wrote:
> Simon,
>
> * Simon Riggs (simon@2ndquadrant.com) wrote:
> > > I hadn't realized that Simon was using "pre-schema" and "post-schema"
> > > to name the first and third parts. I'd agree that this is confusing
> > > nomenclature: it looks like it's trying to say that the data is the
> > > schema, and the schema is not! How about "pre-data and "post-data"?
> >
> > OK by me. Any other takers?
>
> Having the command-line options be "--schema-pre-data" and
> "--schema-post-data" is fine with me. Leaving them the way they are is
> also fine by me. It's the documentation (back to pg_dump.sgml,
> ~774/~797) that starts talking about Pre-Schema and Post-Schema.
OK, Mr.Reviewer, sir:
* patch redone using --schema-before-data and --schema-after-data
* docs rewritten using short clear descriptions using only the words
"before" and "after", like in the option names
* all variable names changed
* retested
So prefixes "pre" and "post" no longer appear anywhere. No latin derived
phrases, just good ol' Anglo-Saxon words.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Attachments:
pg_dump_beforeafter.v4.patch (text/x-patch, 36.6K) download
Index: doc/src/sgml/ref/pg_dump.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_dump.sgml,v
retrieving revision 1.103
diff -c -r1.103 pg_dump.sgml
*** doc/src/sgml/ref/pg_dump.sgml 20 Jul 2008 18:43:30 -0000 1.103
--- doc/src/sgml/ref/pg_dump.sgml 23 Jul 2008 15:26:29 -0000
***************
*** 133,139 ****
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
--- 133,140 ----
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> or <option>--schema-before-data</> or
! <option>--schema-after-data</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
***************
*** 426,431 ****
--- 427,452 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-before-data</option></term>
+ <listitem>
+ <para>
+ Dump object definitions (schema) that occur before table data,
+ using the order produced by a full dump.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--schema-after-data</option></term>
+ <listitem>
+ <para>
+ Dump object definitions (schema) that occur after table data,
+ using the order produced by a full dump.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 790,795 ****
--- 811,844 ----
</para>
<para>
+ The output of <application>pg_dump</application> can be divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Before Data - objects output before data, which includes
+ <command>CREATE TABLE</command> statements and others.
+ This part can be requested using <option>--schema-before-data</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ After Data - objects output after data, which includes
+ <command>CREATE INDEX</command> statements and others.
+ This part can be requested using <option>--schema-after-data</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
Because <application>pg_dump</application> is used to transfer data
to newer versions of <productname>PostgreSQL</>, the output of
<application>pg_dump</application> can be loaded into
Index: doc/src/sgml/ref/pg_restore.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_restore.sgml,v
retrieving revision 1.75
diff -c -r1.75 pg_restore.sgml
*** doc/src/sgml/ref/pg_restore.sgml 13 Apr 2008 03:49:21 -0000 1.75
--- doc/src/sgml/ref/pg_restore.sgml 23 Jul 2008 16:00:29 -0000
***************
*** 321,326 ****
--- 321,346 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-before-data</option></term>
+ <listitem>
+ <para>
+ Restore object definitions (schema) that occur before table data,
+ using the order produced by a full restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--schema-after-data</option></term>
+ <listitem>
+ <para>
+ Restore object definitions (schema) that occur after table data,
+ using the order produced by a full restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 572,577 ****
--- 592,626 ----
</para>
<para>
+ The actions of <application>pg_restore</application> can be
+ divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Before Data - objects output before data, which includes
+ <command>CREATE TABLE</command> statements and others.
+ This part can be requested using <option>--schema-before-data</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ After Data - objects output after data, which includes
+ <command>CREATE INDEX</command> statements and others.
+ This part can be requested using <option>--schema-after-data</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
The limitations of <application>pg_restore</application> are detailed below.
<itemizedlist>
Index: src/bin/pg_dump/pg_backup.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup.h,v
retrieving revision 1.47
diff -c -r1.47 pg_backup.h
*** src/bin/pg_dump/pg_backup.h 13 Apr 2008 03:49:21 -0000 1.47
--- src/bin/pg_dump/pg_backup.h 23 Jul 2008 14:33:32 -0000
***************
*** 89,95 ****
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dataOnly;
int dropSchema;
char *filename;
int schemaOnly;
--- 89,95 ----
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dumpObjFlags; /* which objects types to dump */
int dropSchema;
char *filename;
int schemaOnly;
Index: src/bin/pg_dump/pg_backup_archiver.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.c,v
retrieving revision 1.157
diff -c -r1.157 pg_backup_archiver.c
*** src/bin/pg_dump/pg_backup_archiver.c 4 May 2008 08:32:21 -0000 1.157
--- src/bin/pg_dump/pg_backup_archiver.c 23 Jul 2008 16:12:47 -0000
***************
*** 56,62 ****
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static teReqs _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
--- 56,62 ----
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static int _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
***************
*** 129,135 ****
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! teReqs reqs;
OutputContext sav;
bool defnDumped;
--- 129,135 ----
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! int reqs;
OutputContext sav;
bool defnDumped;
***************
*** 175,193 ****
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then we set the dataOnly flag.
*
! * We could scan for wanted TABLE entries, but that is not the same as
! * dataOnly. At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!ropt->dataOnly)
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0)
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
--- 175,193 ----
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then say we only want DATA type objects.
*
! * We could scan for wanted TABLE entries, but that is not the same.
! * At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!WANT_DATA(ropt->dumpObjFlags))
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_PRE_SCHEMA(reqs) || WANT_POST_SCHEMA(reqs))
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
***************
*** 195,201 ****
}
if (impliedDataOnly)
{
! ropt->dataOnly = impliedDataOnly;
ahlog(AH, 1, "implied data-only restore\n");
}
}
--- 195,201 ----
}
if (impliedDataOnly)
{
! ropt->dumpObjFlags = REQ_DATA;
ahlog(AH, 1, "implied data-only restore\n");
}
}
***************
*** 236,242 ****
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
--- 236,242 ----
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA_BEFORE_DATA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
***************
*** 278,284 ****
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!ropt->dataOnly && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
--- 278,284 ----
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!WANT_DATA(ropt->dumpObjFlags) && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
***************
*** 286,292 ****
defnDumped = false;
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
--- 286,293 ----
defnDumped = false;
! if ((WANT_PRE_SCHEMA(reqs) && WANT_PRE_SCHEMA(ropt->dumpObjFlags)) ||
! (WANT_POST_SCHEMA(reqs) && WANT_POST_SCHEMA(ropt->dumpObjFlags))) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
***************
*** 331,337 ****
/*
* If we have a data component, then process it
*/
! if ((reqs & REQ_DATA) != 0)
{
/*
* hadDumper will be set if there is genuine data component for
--- 332,338 ----
/*
* If we have a data component, then process it
*/
! if (WANT_DATA(reqs))
{
/*
* hadDumper will be set if there is genuine data component for
***************
*** 343,349 ****
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && (reqs & REQ_DATA) != 0)
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
--- 344,350 ----
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && WANT_DATA(reqs))
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
***************
*** 415,421 ****
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
--- 416,422 ----
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_PRE_SCHEMA(reqs)) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
***************
*** 473,479 ****
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
--- 474,480 ----
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
***************
*** 499,505 ****
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
--- 500,506 ----
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
***************
*** 1321,1327 ****
return NULL;
}
! teReqs
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
--- 1322,1328 ----
return NULL;
}
! int
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
***************
*** 2026,2035 ****
te->defn);
}
! static teReqs
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! teReqs res = REQ_ALL;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
--- 2027,2036 ----
te->defn);
}
! static int
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! int res = ropt->dumpObjFlags;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
***************
*** 2109,2125 ****
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
- /* Mask it if we only want schema */
- if (ropt->schemaOnly)
- res = res & REQ_SCHEMA;
-
- /* Mask it we only want data */
- if (ropt->dataOnly)
- res = res & REQ_DATA;
-
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~REQ_SCHEMA;
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
--- 2110,2118 ----
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~(REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
Index: src/bin/pg_dump/pg_backup_archiver.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.h,v
retrieving revision 1.76
diff -c -r1.76 pg_backup_archiver.h
*** src/bin/pg_dump/pg_backup_archiver.h 7 Nov 2007 12:24:24 -0000 1.76
--- src/bin/pg_dump/pg_backup_archiver.h 23 Jul 2008 16:13:23 -0000
***************
*** 158,169 ****
STAGE_FINALIZING
} ArchiverStage;
! typedef enum
! {
! REQ_SCHEMA = 1,
! REQ_DATA = 2,
! REQ_ALL = REQ_SCHEMA + REQ_DATA
! } teReqs;
typedef struct _archiveHandle
{
--- 158,173 ----
STAGE_FINALIZING
} ArchiverStage;
! #define REQ_SCHEMA_BEFORE_DATA (1 << 0)
! #define REQ_DATA (1 << 1)
! #define REQ_SCHEMA_AFTER_DATA (1 << 2)
! #define REQ_ALL (REQ_SCHEMA_BEFORE_DATA + REQ_DATA + REQ_SCHEMA_AFTER_DATA)
!
! #define WANT_PRE_SCHEMA(req) ((req & REQ_SCHEMA_BEFORE_DATA) == REQ_SCHEMA_BEFORE_DATA)
! #define WANT_DATA(req) ((req & REQ_DATA) == REQ_DATA)
! #define WANT_POST_SCHEMA(req) ((req & REQ_SCHEMA_AFTER_DATA) == REQ_SCHEMA_AFTER_DATA)
! #define WANT_ALL(req) ((req & REQ_ALL) == REQ_ALL)
!
typedef struct _archiveHandle
{
***************
*** 317,323 ****
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern teReqs TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
--- 321,327 ----
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern int TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
Index: src/bin/pg_dump/pg_dump.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_dump.c,v
retrieving revision 1.497
diff -c -r1.497 pg_dump.c
*** src/bin/pg_dump/pg_dump.c 20 Jul 2008 18:43:30 -0000 1.497
--- src/bin/pg_dump/pg_dump.c 23 Jul 2008 16:33:08 -0000
***************
*** 73,78 ****
--- 73,82 ----
bool aclsSkip;
const char *lockWaitTimeout;
+ /* groups of objects: default is we dump all groups */
+
+ int dumpObjFlags;
+
/* subquery used to convert user ID (eg, datdba) to user name */
static const char *username_subquery;
***************
*** 225,232 ****
RestoreOptions *ropt;
static int disable_triggers = 0;
! static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
--- 229,238 ----
RestoreOptions *ropt;
static int disable_triggers = 0;
! static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ static int use_schemaBeforeData;
+ static int use_schemaAfterData;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
***************
*** 267,272 ****
--- 273,280 ----
{"disable-triggers", no_argument, &disable_triggers, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-before-data", no_argument, &use_schemaBeforeData, 1},
+ {"schema-after-data", no_argument, &use_schemaAfterData, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 464,474 ****
if (optind < argc)
dbname = argv[optind];
! if (dataOnly && schemaOnly)
{
! write_msg(NULL, "options -s/--schema-only and -a/--data-only cannot be used together\n");
exit(1);
}
if (dataOnly && outputClean)
{
--- 472,517 ----
if (optind < argc)
dbname = argv[optind];
! /*
! * Look for conflicting options relating to object groupings
! */
! if (schemaOnly && dataOnly)
! {
! write_msg(NULL, "options %s and %s cannot be used together\n",
! "-s/--schema-only", "-a/--data-only");
! exit(1);
! }
! else if ((schemaOnly || dataOnly) &&
! (use_schemaBeforeData == 1 || use_schemaAfterData == 1))
{
! write_msg(NULL, "options %s and %s cannot be used together\n",
! schemaOnly ? "-s/--schema-only" : "-a/--data-only",
! use_schemaBeforeData == 1 ? "--schema-before-data" : "--schema-after-data");
exit(1);
}
+ else if (use_schemaBeforeData == 1 && use_schemaAfterData == 1)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "--schema-before-data", "--schema-after-data");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (use_schemaBeforeData == 1)
+ dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
+
+ if (use_schemaAfterData == 1)
+ dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
if (dataOnly && outputClean)
{
***************
*** 646,652 ****
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && !schemaOnly)
outputBlobs = true;
/*
--- 689,695 ----
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && WANT_PRE_SCHEMA(dumpObjFlags))
outputBlobs = true;
/*
***************
*** 658,664 ****
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (!schemaOnly)
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
--- 701,707 ----
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (WANT_DATA(dumpObjFlags))
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
***************
*** 712,718 ****
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && !dataOnly)
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
--- 755,761 ----
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && WANT_DATA(dumpObjFlags))
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
***************
*** 734,740 ****
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dataOnly = dataOnly;
if (compressLevel == -1)
ropt->compression = 0;
--- 777,783 ----
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dumpObjFlags = dumpObjFlags;
if (compressLevel == -1)
ropt->compression = 0;
***************
*** 3414,3420 ****
continue;
/* Ignore indexes of tables not to be dumped */
! if (!tbinfo->dobj.dump)
continue;
if (g_verbose)
--- 3457,3463 ----
continue;
/* Ignore indexes of tables not to be dumped */
! if (!tbinfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
continue;
if (g_verbose)
***************
*** 5165,5171 ****
int ncomments;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with catalogId, using table */
--- 5208,5214 ----
int ncomments;
/* Comments are SCHEMA not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Search for comments associated with catalogId, using table */
***************
*** 5216,5222 ****
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with relation, using table */
--- 5259,5265 ----
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Search for comments associated with relation, using table */
***************
*** 5568,5574 ****
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || dataOnly)
return;
/* don't dump dummy namespace from pre-7.3 source */
--- 5611,5617 ----
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* don't dump dummy namespace from pre-7.3 source */
***************
*** 5617,5623 ****
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || dataOnly)
return;
/* Dump out in proper style */
--- 5660,5666 ----
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Dump out in proper style */
***************
*** 6262,6268 ****
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 6305,6311 ----
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 6309,6315 ****
if (!include_everything)
return false;
/* And they're schema not data */
! if (dataOnly)
return false;
return true;
}
--- 6352,6358 ----
if (!include_everything)
return false;
/* And they're schema not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return false;
return true;
}
***************
*** 6330,6336 ****
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (dataOnly)
return;
/*
--- 6373,6379 ----
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 6590,6596 ****
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 6633,6639 ----
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 6985,6991 ****
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (dataOnly)
return;
if (OidIsValid(cast->castfunc))
--- 7028,7034 ----
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
if (OidIsValid(cast->castfunc))
***************
*** 7135,7141 ****
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || dataOnly)
return;
/*
--- 7178,7184 ----
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 7519,7525 ****
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || dataOnly)
return;
/*
--- 7562,7568 ----
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 7827,7833 ****
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || dataOnly)
return;
/*
--- 7870,7876 ----
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 8096,8102 ****
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8139,8145 ----
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8250,8256 ****
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8293,8299 ----
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8453,8459 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8496,8502 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8522,8528 ****
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8565,8571 ----
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8607,8613 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8650,8656 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8673,8679 ****
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8716,8722 ----
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8809,8815 ****
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (dataOnly || aclsSkip)
return;
sql = createPQExpBuffer();
--- 8852,8858 ----
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (!WANT_PRE_SCHEMA(dumpObjFlags) || aclsSkip)
return;
sql = createPQExpBuffer();
***************
*** 8846,8852 ****
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (!dataOnly)
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
--- 8889,8895 ----
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (WANT_PRE_SCHEMA(dumpObjFlags))
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
***************
*** 9153,9159 ****
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || dataOnly)
return;
/* Don't print inherited defaults, either */
--- 9196,9202 ----
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Don't print inherited defaults, either */
***************
*** 9238,9244 ****
PQExpBuffer q;
PQExpBuffer delq;
! if (dataOnly)
return;
q = createPQExpBuffer();
--- 9281,9287 ----
PQExpBuffer q;
PQExpBuffer delq;
! if (!WANT_POST_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9307,9313 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 9350,9356 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9700,9706 ****
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (!dataOnly)
{
resetPQExpBuffer(delqry);
--- 9743,9749 ----
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (WANT_PRE_SCHEMA(dumpObjFlags))
{
resetPQExpBuffer(delqry);
***************
*** 9803,9809 ****
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (!schemaOnly)
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
--- 9846,9852 ----
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (WANT_PRE_SCHEMA(dumpObjFlags))
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
***************
*** 9836,9842 ****
const char *p;
int findx;
! if (dataOnly)
return;
query = createPQExpBuffer();
--- 9879,9885 ----
const char *p;
int findx;
! if (!WANT_POST_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 10044,10050 ****
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || dataOnly)
return;
/*
--- 10087,10093 ----
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
return;
/*
Index: src/bin/pg_dump/pg_restore.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_restore.c,v
retrieving revision 1.88
diff -c -r1.88 pg_restore.c
*** src/bin/pg_dump/pg_restore.c 13 Apr 2008 03:49:22 -0000 1.88
--- src/bin/pg_dump/pg_restore.c 23 Jul 2008 16:24:48 -0000
***************
*** 78,83 ****
--- 78,90 ----
static int no_data_for_failed_tables = 0;
static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ bool dataOnly = false;
+ bool schemaOnly = false;
+
+ static int use_schemaBeforeData;
+ static int use_schemaAfterData;
+
+ int dumpObjFlags;
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
***************
*** 114,119 ****
--- 121,128 ----
{"disable-triggers", no_argument, &disable_triggers, 1},
{"no-data-for-failed-tables", no_argument, &no_data_for_failed_tables, 1},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-before-data", no_argument, &use_schemaBeforeData, 1},
+ {"schema-after-data", no_argument, &use_schemaAfterData, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 145,151 ****
switch (c)
{
case 'a': /* Dump data only */
! opts->dataOnly = 1;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
--- 154,160 ----
switch (c)
{
case 'a': /* Dump data only */
! dataOnly = true;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
***************
*** 213,219 ****
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! opts->schemaOnly = 1;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
--- 222,228 ----
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! schemaOnly = true;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
***************
*** 295,300 ****
--- 304,350 ----
opts->useDB = 1;
}
+ /*
+ * Look for conflicting options relating to object groupings
+ */
+ if (schemaOnly && dataOnly)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "-s/--schema-only", "-a/--data-only");
+ exit(1);
+ }
+ else if ((schemaOnly || dataOnly) &&
+ (use_schemaBeforeData == 1 || use_schemaAfterData == 1))
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ schemaOnly ? "-s/--schema-only" : "-a/--data-only",
+ use_schemaBeforeData == 1 ? "--schema-before-data" : "--schema-after-data");
+ exit(1);
+ }
+ else if (use_schemaBeforeData == 1 && use_schemaAfterData == 1)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "--schema-before-data", "--schema-after-data");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (use_schemaBeforeData == 1)
+ dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
+
+ if (use_schemaAfterData == 1)
+ dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
+
opts->disable_triggers = disable_triggers;
opts->noDataForFailedTables = no_data_for_failed_tables;
opts->noTablespace = outputNoTablespaces;
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-23 17:09 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-07-23 17:09 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; daveg <daveg@sonic.net>; pgsql-patches@postgresql.org, Andrew Dunstan <andrew@dunslane.net>
On Wed, 2008-07-23 at 17:40 +0100, Simon Riggs wrote:
> On Mon, 2008-07-21 at 07:56 -0400, Stephen Frost wrote:
> > Simon,
> >
> > * Simon Riggs (simon@2ndquadrant.com) wrote:
> > > > I hadn't realized that Simon was using "pre-schema" and "post-schema"
> > > > to name the first and third parts. I'd agree that this is confusing
> > > > nomenclature: it looks like it's trying to say that the data is the
> > > > schema, and the schema is not! How about "pre-data and "post-data"?
> > >
> > > OK by me. Any other takers?
> >
> > Having the command-line options be "--schema-pre-data" and
> > "--schema-post-data" is fine with me. Leaving them the way they are is
> > also fine by me. It's the documentation (back to pg_dump.sgml,
> > ~774/~797) that starts talking about Pre-Schema and Post-Schema.
>
> OK, Mr.Reviewer, sir:
>
> * patch redone using --schema-before-data and --schema-after-data
>
> * docs rewritten using short clear descriptions using only the words
> "before" and "after", like in the option names
>
> * all variable names changed
>
> * retested
>
> So prefixes "pre" and "post" no longer appear anywhere. No latin derived
> phrases, just good ol' Anglo-Saxon words.
...and with command line help also.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Attachments:
pg_dump_beforeafter.v5.patch (text/x-patch, 37.9K) download
Index: doc/src/sgml/ref/pg_dump.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_dump.sgml,v
retrieving revision 1.103
diff -c -r1.103 pg_dump.sgml
*** doc/src/sgml/ref/pg_dump.sgml 20 Jul 2008 18:43:30 -0000 1.103
--- doc/src/sgml/ref/pg_dump.sgml 23 Jul 2008 16:55:05 -0000
***************
*** 133,139 ****
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
--- 133,140 ----
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> or <option>--schema-before-data</> or
! <option>--schema-after-data</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
***************
*** 426,431 ****
--- 427,452 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-before-data</option></term>
+ <listitem>
+ <para>
+ Dump object definitions (schema) that occur before table data,
+ using the order produced by a full dump.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--schema-after-data</option></term>
+ <listitem>
+ <para>
+ Dump object definitions (schema) that occur after table data,
+ using the order produced by a full dump.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 790,795 ****
--- 811,844 ----
</para>
<para>
+ The output of <application>pg_dump</application> can be divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Before Data - objects output before data, which includes
+ <command>CREATE TABLE</command> statements and others.
+ This part can be requested using <option>--schema-before-data</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ After Data - objects output after data, which includes
+ <command>CREATE INDEX</command> statements and others.
+ This part can be requested using <option>--schema-after-data</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
Because <application>pg_dump</application> is used to transfer data
to newer versions of <productname>PostgreSQL</>, the output of
<application>pg_dump</application> can be loaded into
Index: doc/src/sgml/ref/pg_restore.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_restore.sgml,v
retrieving revision 1.75
diff -c -r1.75 pg_restore.sgml
*** doc/src/sgml/ref/pg_restore.sgml 13 Apr 2008 03:49:21 -0000 1.75
--- doc/src/sgml/ref/pg_restore.sgml 23 Jul 2008 16:55:05 -0000
***************
*** 321,326 ****
--- 321,346 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-before-data</option></term>
+ <listitem>
+ <para>
+ Restore object definitions (schema) that occur before table data,
+ using the order produced by a full restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--schema-after-data</option></term>
+ <listitem>
+ <para>
+ Restore object definitions (schema) that occur after table data,
+ using the order produced by a full restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 572,577 ****
--- 592,626 ----
</para>
<para>
+ The actions of <application>pg_restore</application> can be
+ divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Before Data - objects output before data, which includes
+ <command>CREATE TABLE</command> statements and others.
+ This part can be requested using <option>--schema-before-data</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ After Data - objects output after data, which includes
+ <command>CREATE INDEX</command> statements and others.
+ This part can be requested using <option>--schema-after-data</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
The limitations of <application>pg_restore</application> are detailed below.
<itemizedlist>
Index: src/bin/pg_dump/pg_backup.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup.h,v
retrieving revision 1.47
diff -c -r1.47 pg_backup.h
*** src/bin/pg_dump/pg_backup.h 13 Apr 2008 03:49:21 -0000 1.47
--- src/bin/pg_dump/pg_backup.h 23 Jul 2008 16:55:05 -0000
***************
*** 89,95 ****
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dataOnly;
int dropSchema;
char *filename;
int schemaOnly;
--- 89,95 ----
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dumpObjFlags; /* which objects types to dump */
int dropSchema;
char *filename;
int schemaOnly;
Index: src/bin/pg_dump/pg_backup_archiver.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.c,v
retrieving revision 1.157
diff -c -r1.157 pg_backup_archiver.c
*** src/bin/pg_dump/pg_backup_archiver.c 4 May 2008 08:32:21 -0000 1.157
--- src/bin/pg_dump/pg_backup_archiver.c 23 Jul 2008 16:55:05 -0000
***************
*** 56,62 ****
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static teReqs _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
--- 56,62 ----
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static int _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
***************
*** 129,135 ****
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! teReqs reqs;
OutputContext sav;
bool defnDumped;
--- 129,135 ----
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! int reqs;
OutputContext sav;
bool defnDumped;
***************
*** 175,193 ****
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then we set the dataOnly flag.
*
! * We could scan for wanted TABLE entries, but that is not the same as
! * dataOnly. At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!ropt->dataOnly)
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0)
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
--- 175,193 ----
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then say we only want DATA type objects.
*
! * We could scan for wanted TABLE entries, but that is not the same.
! * At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!WANT_DATA(ropt->dumpObjFlags))
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_PRE_SCHEMA(reqs) || WANT_POST_SCHEMA(reqs))
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
***************
*** 195,201 ****
}
if (impliedDataOnly)
{
! ropt->dataOnly = impliedDataOnly;
ahlog(AH, 1, "implied data-only restore\n");
}
}
--- 195,201 ----
}
if (impliedDataOnly)
{
! ropt->dumpObjFlags = REQ_DATA;
ahlog(AH, 1, "implied data-only restore\n");
}
}
***************
*** 236,242 ****
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
--- 236,242 ----
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA_BEFORE_DATA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
***************
*** 278,284 ****
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!ropt->dataOnly && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
--- 278,284 ----
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!WANT_DATA(ropt->dumpObjFlags) && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
***************
*** 286,292 ****
defnDumped = false;
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
--- 286,293 ----
defnDumped = false;
! if ((WANT_PRE_SCHEMA(reqs) && WANT_PRE_SCHEMA(ropt->dumpObjFlags)) ||
! (WANT_POST_SCHEMA(reqs) && WANT_POST_SCHEMA(ropt->dumpObjFlags))) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
***************
*** 331,337 ****
/*
* If we have a data component, then process it
*/
! if ((reqs & REQ_DATA) != 0)
{
/*
* hadDumper will be set if there is genuine data component for
--- 332,338 ----
/*
* If we have a data component, then process it
*/
! if (WANT_DATA(reqs))
{
/*
* hadDumper will be set if there is genuine data component for
***************
*** 343,349 ****
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && (reqs & REQ_DATA) != 0)
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
--- 344,350 ----
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && WANT_DATA(reqs))
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
***************
*** 415,421 ****
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
--- 416,422 ----
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_PRE_SCHEMA(reqs)) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
***************
*** 473,479 ****
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
--- 474,480 ----
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
***************
*** 499,505 ****
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
--- 500,506 ----
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
***************
*** 1321,1327 ****
return NULL;
}
! teReqs
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
--- 1322,1328 ----
return NULL;
}
! int
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
***************
*** 2026,2035 ****
te->defn);
}
! static teReqs
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! teReqs res = REQ_ALL;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
--- 2027,2036 ----
te->defn);
}
! static int
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! int res = ropt->dumpObjFlags;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
***************
*** 2109,2125 ****
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
- /* Mask it if we only want schema */
- if (ropt->schemaOnly)
- res = res & REQ_SCHEMA;
-
- /* Mask it we only want data */
- if (ropt->dataOnly)
- res = res & REQ_DATA;
-
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~REQ_SCHEMA;
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
--- 2110,2118 ----
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~(REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
Index: src/bin/pg_dump/pg_backup_archiver.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.h,v
retrieving revision 1.76
diff -c -r1.76 pg_backup_archiver.h
*** src/bin/pg_dump/pg_backup_archiver.h 7 Nov 2007 12:24:24 -0000 1.76
--- src/bin/pg_dump/pg_backup_archiver.h 23 Jul 2008 16:55:05 -0000
***************
*** 158,169 ****
STAGE_FINALIZING
} ArchiverStage;
! typedef enum
! {
! REQ_SCHEMA = 1,
! REQ_DATA = 2,
! REQ_ALL = REQ_SCHEMA + REQ_DATA
! } teReqs;
typedef struct _archiveHandle
{
--- 158,173 ----
STAGE_FINALIZING
} ArchiverStage;
! #define REQ_SCHEMA_BEFORE_DATA (1 << 0)
! #define REQ_DATA (1 << 1)
! #define REQ_SCHEMA_AFTER_DATA (1 << 2)
! #define REQ_ALL (REQ_SCHEMA_BEFORE_DATA + REQ_DATA + REQ_SCHEMA_AFTER_DATA)
!
! #define WANT_PRE_SCHEMA(req) ((req & REQ_SCHEMA_BEFORE_DATA) == REQ_SCHEMA_BEFORE_DATA)
! #define WANT_DATA(req) ((req & REQ_DATA) == REQ_DATA)
! #define WANT_POST_SCHEMA(req) ((req & REQ_SCHEMA_AFTER_DATA) == REQ_SCHEMA_AFTER_DATA)
! #define WANT_ALL(req) ((req & REQ_ALL) == REQ_ALL)
!
typedef struct _archiveHandle
{
***************
*** 317,323 ****
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern teReqs TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
--- 321,327 ----
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern int TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
Index: src/bin/pg_dump/pg_dump.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_dump.c,v
retrieving revision 1.497
diff -c -r1.497 pg_dump.c
*** src/bin/pg_dump/pg_dump.c 20 Jul 2008 18:43:30 -0000 1.497
--- src/bin/pg_dump/pg_dump.c 23 Jul 2008 17:04:24 -0000
***************
*** 73,78 ****
--- 73,82 ----
bool aclsSkip;
const char *lockWaitTimeout;
+ /* groups of objects: default is we dump all groups */
+
+ int dumpObjFlags;
+
/* subquery used to convert user ID (eg, datdba) to user name */
static const char *username_subquery;
***************
*** 225,232 ****
RestoreOptions *ropt;
static int disable_triggers = 0;
! static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
--- 229,238 ----
RestoreOptions *ropt;
static int disable_triggers = 0;
! static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ static int use_schemaBeforeData;
+ static int use_schemaAfterData;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
***************
*** 267,272 ****
--- 273,280 ----
{"disable-triggers", no_argument, &disable_triggers, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-before-data", no_argument, &use_schemaBeforeData, 1},
+ {"schema-after-data", no_argument, &use_schemaAfterData, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 464,474 ****
if (optind < argc)
dbname = argv[optind];
! if (dataOnly && schemaOnly)
{
! write_msg(NULL, "options -s/--schema-only and -a/--data-only cannot be used together\n");
exit(1);
}
if (dataOnly && outputClean)
{
--- 472,517 ----
if (optind < argc)
dbname = argv[optind];
! /*
! * Look for conflicting options relating to object groupings
! */
! if (schemaOnly && dataOnly)
! {
! write_msg(NULL, "options %s and %s cannot be used together\n",
! "-s/--schema-only", "-a/--data-only");
! exit(1);
! }
! else if ((schemaOnly || dataOnly) &&
! (use_schemaBeforeData == 1 || use_schemaAfterData == 1))
{
! write_msg(NULL, "options %s and %s cannot be used together\n",
! schemaOnly ? "-s/--schema-only" : "-a/--data-only",
! use_schemaBeforeData == 1 ? "--schema-before-data" : "--schema-after-data");
exit(1);
}
+ else if (use_schemaBeforeData == 1 && use_schemaAfterData == 1)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "--schema-before-data", "--schema-after-data");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (use_schemaBeforeData == 1)
+ dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
+
+ if (use_schemaAfterData == 1)
+ dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
if (dataOnly && outputClean)
{
***************
*** 646,652 ****
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && !schemaOnly)
outputBlobs = true;
/*
--- 689,695 ----
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && WANT_PRE_SCHEMA(dumpObjFlags))
outputBlobs = true;
/*
***************
*** 658,664 ****
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (!schemaOnly)
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
--- 701,707 ----
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (WANT_DATA(dumpObjFlags))
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
***************
*** 712,718 ****
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && !dataOnly)
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
--- 755,761 ----
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && WANT_DATA(dumpObjFlags))
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
***************
*** 734,740 ****
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dataOnly = dataOnly;
if (compressLevel == -1)
ropt->compression = 0;
--- 777,783 ----
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dumpObjFlags = dumpObjFlags;
if (compressLevel == -1)
ropt->compression = 0;
***************
*** 792,797 ****
--- 835,842 ----
printf(_(" --disable-dollar-quoting disable dollar quoting, use SQL standard quoting\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
+ printf(_(" --schema-before-data dump only the part of schema before table data\n"));
+ printf(_(" --schema-after-data dump only the part of schema after table data\n"));
printf(_(" --use-set-session-authorization\n"
" use SESSION AUTHORIZATION commands instead of\n"
" ALTER OWNER commands to set ownership\n"));
***************
*** 3414,3420 ****
continue;
/* Ignore indexes of tables not to be dumped */
! if (!tbinfo->dobj.dump)
continue;
if (g_verbose)
--- 3459,3465 ----
continue;
/* Ignore indexes of tables not to be dumped */
! if (!tbinfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
continue;
if (g_verbose)
***************
*** 5165,5171 ****
int ncomments;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with catalogId, using table */
--- 5210,5216 ----
int ncomments;
/* Comments are SCHEMA not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Search for comments associated with catalogId, using table */
***************
*** 5216,5222 ****
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with relation, using table */
--- 5261,5267 ----
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Search for comments associated with relation, using table */
***************
*** 5568,5574 ****
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || dataOnly)
return;
/* don't dump dummy namespace from pre-7.3 source */
--- 5613,5619 ----
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* don't dump dummy namespace from pre-7.3 source */
***************
*** 5617,5623 ****
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || dataOnly)
return;
/* Dump out in proper style */
--- 5662,5668 ----
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Dump out in proper style */
***************
*** 6262,6268 ****
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 6307,6313 ----
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 6309,6315 ****
if (!include_everything)
return false;
/* And they're schema not data */
! if (dataOnly)
return false;
return true;
}
--- 6354,6360 ----
if (!include_everything)
return false;
/* And they're schema not data */
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return false;
return true;
}
***************
*** 6330,6336 ****
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (dataOnly)
return;
/*
--- 6375,6381 ----
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 6590,6596 ****
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 6635,6641 ----
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 6985,6991 ****
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (dataOnly)
return;
if (OidIsValid(cast->castfunc))
--- 7030,7036 ----
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (!WANT_PRE_SCHEMA(dumpObjFlags))
return;
if (OidIsValid(cast->castfunc))
***************
*** 7135,7141 ****
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || dataOnly)
return;
/*
--- 7180,7186 ----
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 7519,7525 ****
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || dataOnly)
return;
/*
--- 7564,7570 ----
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 7827,7833 ****
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || dataOnly)
return;
/*
--- 7872,7878 ----
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/*
***************
*** 8096,8102 ****
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8141,8147 ----
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8250,8256 ****
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8295,8301 ----
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8453,8459 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8498,8504 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8522,8528 ****
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8567,8573 ----
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8607,8613 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8652,8658 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8673,8679 ****
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8718,8724 ----
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8809,8815 ****
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (dataOnly || aclsSkip)
return;
sql = createPQExpBuffer();
--- 8854,8860 ----
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (!WANT_PRE_SCHEMA(dumpObjFlags) || aclsSkip)
return;
sql = createPQExpBuffer();
***************
*** 8846,8852 ****
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (!dataOnly)
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
--- 8891,8897 ----
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (WANT_PRE_SCHEMA(dumpObjFlags))
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
***************
*** 9153,9159 ****
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || dataOnly)
return;
/* Don't print inherited defaults, either */
--- 9198,9204 ----
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || !WANT_PRE_SCHEMA(dumpObjFlags))
return;
/* Don't print inherited defaults, either */
***************
*** 9238,9244 ****
PQExpBuffer q;
PQExpBuffer delq;
! if (dataOnly)
return;
q = createPQExpBuffer();
--- 9283,9289 ----
PQExpBuffer q;
PQExpBuffer delq;
! if (!WANT_POST_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9307,9313 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 9352,9358 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9700,9706 ****
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (!dataOnly)
{
resetPQExpBuffer(delqry);
--- 9745,9751 ----
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (WANT_PRE_SCHEMA(dumpObjFlags))
{
resetPQExpBuffer(delqry);
***************
*** 9803,9809 ****
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (!schemaOnly)
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
--- 9848,9854 ----
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (WANT_PRE_SCHEMA(dumpObjFlags))
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
***************
*** 9836,9842 ****
const char *p;
int findx;
! if (dataOnly)
return;
query = createPQExpBuffer();
--- 9881,9887 ----
const char *p;
int findx;
! if (!WANT_POST_SCHEMA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 10044,10050 ****
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || dataOnly)
return;
/*
--- 10089,10095 ----
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
return;
/*
Index: src/bin/pg_dump/pg_restore.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_restore.c,v
retrieving revision 1.88
diff -c -r1.88 pg_restore.c
*** src/bin/pg_dump/pg_restore.c 13 Apr 2008 03:49:22 -0000 1.88
--- src/bin/pg_dump/pg_restore.c 23 Jul 2008 17:06:59 -0000
***************
*** 78,83 ****
--- 78,90 ----
static int no_data_for_failed_tables = 0;
static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ bool dataOnly = false;
+ bool schemaOnly = false;
+
+ static int use_schemaBeforeData;
+ static int use_schemaAfterData;
+
+ int dumpObjFlags;
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
***************
*** 114,119 ****
--- 121,128 ----
{"disable-triggers", no_argument, &disable_triggers, 1},
{"no-data-for-failed-tables", no_argument, &no_data_for_failed_tables, 1},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-before-data", no_argument, &use_schemaBeforeData, 1},
+ {"schema-after-data", no_argument, &use_schemaAfterData, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 145,151 ****
switch (c)
{
case 'a': /* Dump data only */
! opts->dataOnly = 1;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
--- 154,160 ----
switch (c)
{
case 'a': /* Dump data only */
! dataOnly = true;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
***************
*** 213,219 ****
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! opts->schemaOnly = 1;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
--- 222,228 ----
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! schemaOnly = true;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
***************
*** 295,300 ****
--- 304,350 ----
opts->useDB = 1;
}
+ /*
+ * Look for conflicting options relating to object groupings
+ */
+ if (schemaOnly && dataOnly)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "-s/--schema-only", "-a/--data-only");
+ exit(1);
+ }
+ else if ((schemaOnly || dataOnly) &&
+ (use_schemaBeforeData == 1 || use_schemaAfterData == 1))
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ schemaOnly ? "-s/--schema-only" : "-a/--data-only",
+ use_schemaBeforeData == 1 ? "--schema-before-data" : "--schema-after-data");
+ exit(1);
+ }
+ else if (use_schemaBeforeData == 1 && use_schemaAfterData == 1)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "--schema-before-data", "--schema-after-data");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (use_schemaBeforeData == 1)
+ dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
+
+ if (use_schemaAfterData == 1)
+ dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
+
opts->disable_triggers = disable_triggers;
opts->noDataForFailedTables = no_data_for_failed_tables;
opts->noTablespace = outputNoTablespaces;
***************
*** 405,410 ****
--- 455,462 ----
" do not restore data of tables that could not be\n"
" created\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
+ printf(_(" --schema-before-data dump only the part of schema before table data\n"));
+ printf(_(" --schema-after-data dump only the part of schema after table data\n"));
printf(_(" --use-set-session-authorization\n"
" use SESSION AUTHORIZATION commands instead of\n"
" OWNER TO commands\n"));
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-24 03:20 ` Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-24 03:20 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: pgsql-patches
Simon,
* Simon Riggs (simon@2ndquadrant.com) wrote:
> ...and with command line help also.
The documentation and whatnot looks good to me now. There are a couple
of other issues I found while looking through and testing the patch
though-
inline patch 1: src/bin/pg_dump/pg_dump.c (+3/-1 lines) download inline-1.patch
Index: src/bin/pg_dump/pg_dump.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_dump.c,v
retrieving revision 1.497
diff -c -r1.497 pg_dump.c
*** src/bin/pg_dump/pg_dump.c 20 Jul 2008 18:43:30 -0000 1.497
--- src/bin/pg_dump/pg_dump.c 23 Jul 2008 17:04:24 -0000
***************
*** 225,232 ****
RestoreOptions *ropt;
static int disable_triggers = 0;
! static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
--- 229,238 ----
RestoreOptions *ropt;
static int disable_triggers = 0;
! static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ static int use_schemaBeforeData;
+ static int use_schemaAfterData;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
***************
inline patch 2: src/bin/pg_dump/pg_restore.c (+12/-0 lines) download inline-2.patch
Index: src/bin/pg_dump/pg_restore.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_restore.c,v
retrieving revision 1.88
diff -c -r1.88 pg_restore.c
*** src/bin/pg_dump/pg_restore.c 13 Apr 2008 03:49:22 -0000 1.88
--- src/bin/pg_dump/pg_restore.c 23 Jul 2008 17:06:59 -0000
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (use_schemaBeforeData == 1)
+ dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
+
+ if (use_schemaAfterData == 1)
+ dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
+
***************
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-24 07:37 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-07-24 07:37 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-patches
On Wed, 2008-07-23 at 23:20 -0400, Stephen Frost wrote:
> * Simon Riggs (simon@2ndquadrant.com) wrote:
> > ...and with command line help also.
>
> The documentation and whatnot looks good to me now. There are a couple
> of other issues I found while looking through and testing the patch
> though-
Thanks for a good review.
> Index: src/bin/pg_dump/pg_dump.c
> ===================================================================
> RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_dump.c,v
> retrieving revision 1.497
> diff -c -r1.497 pg_dump.c
> *** src/bin/pg_dump/pg_dump.c 20 Jul 2008 18:43:30 -0000 1.497
> --- src/bin/pg_dump/pg_dump.c 23 Jul 2008 17:04:24 -0000
> ***************
> *** 225,232 ****
> RestoreOptions *ropt;
>
> static int disable_triggers = 0;
> ! static int outputNoTablespaces = 0;
> static int use_setsessauth = 0;
>
> static struct option long_options[] = {
> {"data-only", no_argument, NULL, 'a'},
> --- 229,238 ----
> RestoreOptions *ropt;
>
> static int disable_triggers = 0;
> ! static int outputNoTablespaces = 0;
> static int use_setsessauth = 0;
> + static int use_schemaBeforeData;
> + static int use_schemaAfterData;
>
> static struct option long_options[] = {
> {"data-only", no_argument, NULL, 'a'},
> ***************
>
> This hunk appears to have a bit of gratuitous whitespace change, not a
> big deal tho.
OK
> ***************
> *** 464,474 ****
> [...]
> + if (dataOnly)
> + dumpObjFlags = REQ_DATA;
> +
> + if (use_schemaBeforeData == 1)
> + dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
> +
> + if (use_schemaAfterData == 1)
> + dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
> +
> + if (schemaOnly)
> + dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
> ***************
>
> It wouldn't kill to be consistant between testing for '== 1' and just
> checking for non-zero. Again, not really a big deal, and I wouldn't
> mention these if there weren't other issues.
OK
> ***************
> *** 646,652 ****
> * Dumping blobs is now default unless we saw an inclusion switch or -s
> * ... but even if we did see one of these, -b turns it back on.
> */
> ! if (include_everything && !schemaOnly)
> outputBlobs = true;
>
> /*
> --- 689,695 ----
> * Dumping blobs is now default unless we saw an inclusion switch or -s
> * ... but even if we did see one of these, -b turns it back on.
> */
> ! if (include_everything && WANT_PRE_SCHEMA(dumpObjFlags))
> outputBlobs = true;
>
> /*
> ***************
>
> Shouldn't this change be to "WANT_DATA(dumpObjFlags)"? That's what most
> of the '!schemaOnly' get translated to. Otherwise I think you would be
> getting blobs when you've asked for just schema-before-data, which
> doesn't seem like it'd make much sense.
Yes, fixed
> ***************
> *** 712,718 ****
> dumpStdStrings(g_fout);
>
> /* The database item is always next, unless we don't want it at all */
> ! if (include_everything && !dataOnly)
> dumpDatabase(g_fout);
>
> /* Now the rearrangeable objects. */
> --- 755,761 ----
> dumpStdStrings(g_fout);
>
> /* The database item is always next, unless we don't want it at all */
> ! if (include_everything && WANT_DATA(dumpObjFlags))
> dumpDatabase(g_fout);
>
> /* Now the rearrangeable objects. */
> ***************
>
> Shouldn't this be 'WANT_PRE_SCHEMA(dumpObjFlags)'?
Yes, fixed
> ***************
> *** 3414,3420 ****
> continue;
>
> /* Ignore indexes of tables not to be dumped */
> ! if (!tbinfo->dobj.dump)
> continue;
>
> if (g_verbose)
> --- 3459,3465 ----
> continue;
>
> /* Ignore indexes of tables not to be dumped */
> ! if (!tbinfo->dobj.dump || !WANT_POST_SCHEMA(dumpObjFlags))
> continue;
>
> if (g_verbose)
> ***************
>
> I didn't test this, but it strikes me as an unnecessary addition? If
> anything, wouldn't this check make more sense being done right after
> dropping into getIndexes()? No sense going through the loop just for
> fun.. Technically, it's a behavioral change for --data-only since it
> used to gather index information anyway, but it's a good optimization if
> done in the right place.
Agreed. I've just removed this. Patch not about optimising logic.
> Also around here, there doesn't appear to be any checking in
> dumpEnumType(), which strikes me as odd. Wouldn't that deserve a
>
> if (!WANT_PRE_SCHEMA(dumpObjFlags))
> return;
>
> check? If not even some kind of equivilant ->dobj.dump check..
Agreed. That appears to be a bug in pg_dump, since this would currently
dump enums if --data-only was specified, which in my understanding would
be wrong.
Have included this:
/* Skip if not to be dumped */
if (!tinfo->dobj.dump || !WANT_BEFORE_SCHEMA(dumpObjFlags))
return;
> ***************
> *** 9803,9809 ****
> tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
> }
>
> ! if (!schemaOnly)
> {
> resetPQExpBuffer(query);
> appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
> --- 9848,9854 ----
> tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
> }
>
> ! if (WANT_PRE_SCHEMA(dumpObjFlags))
> {
> resetPQExpBuffer(query);
> appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
> ***************
>
> This is a mistaken logic invert, which results in setval's not being
> dumped at all when pulling out each piece seperately. It should be:
>
> if (WANT_DATA(dumpObjFlags))
>
> so that setval's are correctly included on the --data-only piece. As
> --data-only previously existed, this would be a regression too.
OK, fixed.
> Index: src/bin/pg_dump/pg_restore.c
> ===================================================================
> RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_restore.c,v
> retrieving revision 1.88
> diff -c -r1.88 pg_restore.c
> *** src/bin/pg_dump/pg_restore.c 13 Apr 2008 03:49:22 -0000 1.88
> --- src/bin/pg_dump/pg_restore.c 23 Jul 2008 17:06:59 -0000
> + if (dataOnly)
> + dumpObjFlags = REQ_DATA;
> +
> + if (use_schemaBeforeData == 1)
> + dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
> +
> + if (use_schemaAfterData == 1)
> + dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
> +
> + if (schemaOnly)
> + dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
> +
> ***************
>
> Ditto previous comment on this, but in pg_restore.c.
OK
>
> ***************
> *** 405,410 ****
> --- 455,462 ----
> " do not restore data of tables that could not be\n"
> " created\n"));
> printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
> + printf(_(" --schema-before-data dump only the part of schema before table data\n"));
> + printf(_(" --schema-after-data dump only the part of schema after table data\n"));
> printf(_(" --use-set-session-authorization\n"
> " use SESSION AUTHORIZATION commands instead of\n"
> " OWNER TO commands\n"));
> ***************
>
> Forgot to mention this on pg_dump.c, but in both pg_dump and pg_restore,
> and I hate to be the bearer of bad news, but your command-line
> documentation doesn't line up properly in the output. You shouldn't be
> using tabs there but instead should use spaces as the other help text
> does, so everything lines up nicely.
OK
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Attachments:
pg_dump_beforeafter.v6.patch (text/x-patch, 68.8K) download
Index: doc/src/sgml/ref/pg_dump.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_dump.sgml,v
retrieving revision 1.103
diff -c -r1.103 pg_dump.sgml
*** doc/src/sgml/ref/pg_dump.sgml 20 Jul 2008 18:43:30 -0000 1.103
--- doc/src/sgml/ref/pg_dump.sgml 24 Jul 2008 07:30:19 -0000
***************
*** 133,139 ****
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
--- 133,140 ----
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> or <option>--schema-before-data</> or
! <option>--schema-after-data</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
***************
*** 426,431 ****
--- 427,452 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-before-data</option></term>
+ <listitem>
+ <para>
+ Dump object definitions (schema) that occur before table data,
+ using the order produced by a full dump.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--schema-after-data</option></term>
+ <listitem>
+ <para>
+ Dump object definitions (schema) that occur after table data,
+ using the order produced by a full dump.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 790,795 ****
--- 811,844 ----
</para>
<para>
+ The output of <application>pg_dump</application> can be divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Before Data - objects output before data, which includes
+ <command>CREATE TABLE</command> statements and others.
+ This part can be requested using <option>--schema-before-data</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ After Data - objects output after data, which includes
+ <command>CREATE INDEX</command> statements and others.
+ This part can be requested using <option>--schema-after-data</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
Because <application>pg_dump</application> is used to transfer data
to newer versions of <productname>PostgreSQL</>, the output of
<application>pg_dump</application> can be loaded into
Index: doc/src/sgml/ref/pg_restore.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_restore.sgml,v
retrieving revision 1.75
diff -c -r1.75 pg_restore.sgml
*** doc/src/sgml/ref/pg_restore.sgml 13 Apr 2008 03:49:21 -0000 1.75
--- doc/src/sgml/ref/pg_restore.sgml 24 Jul 2008 07:30:19 -0000
***************
*** 321,326 ****
--- 321,346 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-before-data</option></term>
+ <listitem>
+ <para>
+ Restore object definitions (schema) that occur before table data,
+ using the order produced by a full restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--schema-after-data</option></term>
+ <listitem>
+ <para>
+ Restore object definitions (schema) that occur after table data,
+ using the order produced by a full restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 572,577 ****
--- 592,626 ----
</para>
<para>
+ The actions of <application>pg_restore</application> can be
+ divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Before Data - objects output before data, which includes
+ <command>CREATE TABLE</command> statements and others.
+ This part can be requested using <option>--schema-before-data</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ After Data - objects output after data, which includes
+ <command>CREATE INDEX</command> statements and others.
+ This part can be requested using <option>--schema-after-data</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
The limitations of <application>pg_restore</application> are detailed below.
<itemizedlist>
Index: src/bin/pg_dump/pg_backup.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup.h,v
retrieving revision 1.47
diff -c -r1.47 pg_backup.h
*** src/bin/pg_dump/pg_backup.h 13 Apr 2008 03:49:21 -0000 1.47
--- src/bin/pg_dump/pg_backup.h 24 Jul 2008 07:30:19 -0000
***************
*** 89,95 ****
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dataOnly;
int dropSchema;
char *filename;
int schemaOnly;
--- 89,95 ----
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dumpObjFlags; /* which objects types to dump */
int dropSchema;
char *filename;
int schemaOnly;
Index: src/bin/pg_dump/pg_backup_archiver.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.c,v
retrieving revision 1.157
diff -c -r1.157 pg_backup_archiver.c
*** src/bin/pg_dump/pg_backup_archiver.c 4 May 2008 08:32:21 -0000 1.157
--- src/bin/pg_dump/pg_backup_archiver.c 24 Jul 2008 07:30:19 -0000
***************
*** 56,62 ****
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static teReqs _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
--- 56,62 ----
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static int _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
***************
*** 129,135 ****
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! teReqs reqs;
OutputContext sav;
bool defnDumped;
--- 129,135 ----
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! int reqs;
OutputContext sav;
bool defnDumped;
***************
*** 175,193 ****
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then we set the dataOnly flag.
*
! * We could scan for wanted TABLE entries, but that is not the same as
! * dataOnly. At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!ropt->dataOnly)
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0)
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
--- 175,193 ----
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then say we only want DATA type objects.
*
! * We could scan for wanted TABLE entries, but that is not the same.
! * At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!WANT_DATA(ropt->dumpObjFlags))
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_SCHEMA_BEFORE_DATA(reqs) || WANT_SCHEMA_AFTER_DATA(reqs))
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
***************
*** 195,201 ****
}
if (impliedDataOnly)
{
! ropt->dataOnly = impliedDataOnly;
ahlog(AH, 1, "implied data-only restore\n");
}
}
--- 195,201 ----
}
if (impliedDataOnly)
{
! ropt->dumpObjFlags = REQ_DATA;
ahlog(AH, 1, "implied data-only restore\n");
}
}
***************
*** 236,242 ****
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
--- 236,242 ----
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA_BEFORE_DATA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
***************
*** 278,284 ****
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!ropt->dataOnly && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
--- 278,284 ----
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!WANT_DATA(ropt->dumpObjFlags) && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
***************
*** 286,292 ****
defnDumped = false;
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
--- 286,293 ----
defnDumped = false;
! if ((WANT_SCHEMA_BEFORE_DATA(reqs) && WANT_SCHEMA_BEFORE_DATA(ropt->dumpObjFlags)) ||
! (WANT_SCHEMA_AFTER_DATA(reqs) && WANT_SCHEMA_AFTER_DATA(ropt->dumpObjFlags))) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
***************
*** 331,337 ****
/*
* If we have a data component, then process it
*/
! if ((reqs & REQ_DATA) != 0)
{
/*
* hadDumper will be set if there is genuine data component for
--- 332,338 ----
/*
* If we have a data component, then process it
*/
! if (WANT_DATA(reqs))
{
/*
* hadDumper will be set if there is genuine data component for
***************
*** 343,349 ****
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && (reqs & REQ_DATA) != 0)
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
--- 344,350 ----
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && WANT_DATA(reqs))
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
***************
*** 415,421 ****
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
--- 416,422 ----
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_SCHEMA_BEFORE_DATA(reqs)) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
***************
*** 473,479 ****
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
--- 474,480 ----
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
***************
*** 499,505 ****
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
--- 500,506 ----
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
***************
*** 1321,1327 ****
return NULL;
}
! teReqs
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
--- 1322,1328 ----
return NULL;
}
! int
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
***************
*** 2026,2035 ****
te->defn);
}
! static teReqs
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! teReqs res = REQ_ALL;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
--- 2027,2036 ----
te->defn);
}
! static int
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! int res = ropt->dumpObjFlags;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
***************
*** 2109,2125 ****
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
- /* Mask it if we only want schema */
- if (ropt->schemaOnly)
- res = res & REQ_SCHEMA;
-
- /* Mask it we only want data */
- if (ropt->dataOnly)
- res = res & REQ_DATA;
-
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~REQ_SCHEMA;
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
--- 2110,2118 ----
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~(REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
Index: src/bin/pg_dump/pg_backup_archiver.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.h,v
retrieving revision 1.76
diff -c -r1.76 pg_backup_archiver.h
*** src/bin/pg_dump/pg_backup_archiver.h 7 Nov 2007 12:24:24 -0000 1.76
--- src/bin/pg_dump/pg_backup_archiver.h 24 Jul 2008 07:30:19 -0000
***************
*** 158,169 ****
STAGE_FINALIZING
} ArchiverStage;
! typedef enum
! {
! REQ_SCHEMA = 1,
! REQ_DATA = 2,
! REQ_ALL = REQ_SCHEMA + REQ_DATA
! } teReqs;
typedef struct _archiveHandle
{
--- 158,173 ----
STAGE_FINALIZING
} ArchiverStage;
! #define REQ_SCHEMA_BEFORE_DATA (1 << 0)
! #define REQ_DATA (1 << 1)
! #define REQ_SCHEMA_AFTER_DATA (1 << 2)
! #define REQ_ALL (REQ_SCHEMA_BEFORE_DATA + REQ_DATA + REQ_SCHEMA_AFTER_DATA)
!
! #define WANT_SCHEMA_BEFORE_DATA(req) ((req & REQ_SCHEMA_BEFORE_DATA) == REQ_SCHEMA_BEFORE_DATA)
! #define WANT_DATA(req) ((req & REQ_DATA) == REQ_DATA)
! #define WANT_SCHEMA_AFTER_DATA(req) ((req & REQ_SCHEMA_AFTER_DATA) == REQ_SCHEMA_AFTER_DATA)
! #define WANT_ALL(req) ((req & REQ_ALL) == REQ_ALL)
!
typedef struct _archiveHandle
{
***************
*** 317,323 ****
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern teReqs TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
--- 321,327 ----
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern int TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
Index: src/bin/pg_dump/pg_dump.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_dump.c,v
retrieving revision 1.497
diff -c -r1.497 pg_dump.c
*** src/bin/pg_dump/pg_dump.c 20 Jul 2008 18:43:30 -0000 1.497
--- src/bin/pg_dump/pg_dump.c 24 Jul 2008 07:35:28 -0000
***************
*** 73,78 ****
--- 73,82 ----
bool aclsSkip;
const char *lockWaitTimeout;
+ /* groups of objects: default is we dump all groups */
+
+ int dumpObjFlags;
+
/* subquery used to convert user ID (eg, datdba) to user name */
static const char *username_subquery;
***************
*** 227,232 ****
--- 231,238 ----
static int disable_triggers = 0;
static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ static int schemaBeforeData;
+ static int schemaAfterData;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
***************
*** 267,272 ****
--- 273,280 ----
{"disable-triggers", no_argument, &disable_triggers, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-before-data", no_argument, &schemaBeforeData, 1},
+ {"schema-after-data", no_argument, &schemaAfterData, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 420,425 ****
--- 428,437 ----
disable_triggers = 1;
else if (strcmp(optarg, "no-tablespaces") == 0)
outputNoTablespaces = 1;
+ else if (strcmp(optarg, "schema-before-data") == 0)
+ schemaBeforeData = 1;
+ else if (strcmp(optarg, "schema-after-data") == 0)
+ schemaAfterData = 1;
else if (strcmp(optarg, "use-set-session-authorization") == 0)
use_setsessauth = 1;
else
***************
*** 464,474 ****
if (optind < argc)
dbname = argv[optind];
! if (dataOnly && schemaOnly)
{
! write_msg(NULL, "options -s/--schema-only and -a/--data-only cannot be used together\n");
exit(1);
}
if (dataOnly && outputClean)
{
--- 476,521 ----
if (optind < argc)
dbname = argv[optind];
! /*
! * Look for conflicting options relating to object groupings
! */
! if (schemaOnly && dataOnly)
! {
! write_msg(NULL, "options %s and %s cannot be used together\n",
! "-s/--schema-only", "-a/--data-only");
! exit(1);
! }
! else if ((schemaOnly || dataOnly) &&
! (schemaBeforeData || schemaAfterData))
{
! write_msg(NULL, "options %s and %s cannot be used together\n",
! schemaOnly ? "-s/--schema-only" : "-a/--data-only",
! schemaBeforeData ? "--schema-before-data" : "--schema-after-data");
exit(1);
}
+ else if (schemaBeforeData && schemaAfterData)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "--schema-before-data", "--schema-after-data");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (schemaBeforeData)
+ dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
+
+ if (schemaAfterData)
+ dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
if (dataOnly && outputClean)
{
***************
*** 646,652 ****
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && !schemaOnly)
outputBlobs = true;
/*
--- 693,699 ----
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && WANT_DATA(dumpObjFlags))
outputBlobs = true;
/*
***************
*** 658,664 ****
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (!schemaOnly)
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
--- 705,711 ----
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (WANT_DATA(dumpObjFlags))
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
***************
*** 712,718 ****
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && !dataOnly)
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
--- 759,765 ----
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
***************
*** 734,740 ****
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dataOnly = dataOnly;
if (compressLevel == -1)
ropt->compression = 0;
--- 781,787 ----
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dumpObjFlags = dumpObjFlags;
if (compressLevel == -1)
ropt->compression = 0;
***************
*** 792,797 ****
--- 839,846 ----
printf(_(" --disable-dollar-quoting disable dollar quoting, use SQL standard quoting\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
+ printf(_(" --schema-before-data dump only the part of schema before table data\n"));
+ printf(_(" --schema-after-data dump only the part of schema after table data\n"));
printf(_(" --use-set-session-authorization\n"
" use SESSION AUTHORIZATION commands instead of\n"
" ALTER OWNER commands to set ownership\n"));
***************
*** 5165,5171 ****
int ncomments;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with catalogId, using table */
--- 5214,5220 ----
int ncomments;
/* Comments are SCHEMA not data */
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* Search for comments associated with catalogId, using table */
***************
*** 5216,5222 ****
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with relation, using table */
--- 5265,5271 ----
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* Search for comments associated with relation, using table */
***************
*** 5568,5574 ****
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || dataOnly)
return;
/* don't dump dummy namespace from pre-7.3 source */
--- 5617,5623 ----
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* don't dump dummy namespace from pre-7.3 source */
***************
*** 5617,5623 ****
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || dataOnly)
return;
/* Dump out in proper style */
--- 5666,5672 ----
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* Dump out in proper style */
***************
*** 5646,5651 ****
--- 5695,5704 ----
i;
char *label;
+ /* Skip if not to be dumped */
+ if (!tinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
+ return;
+
/* Set proper schema search path so regproc references list correctly */
selectSourceSchema(tinfo->dobj.namespace->dobj.name);
***************
*** 6262,6268 ****
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 6315,6321 ----
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 6309,6315 ****
if (!include_everything)
return false;
/* And they're schema not data */
! if (dataOnly)
return false;
return true;
}
--- 6362,6368 ----
if (!include_everything)
return false;
/* And they're schema not data */
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return false;
return true;
}
***************
*** 6330,6336 ****
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (dataOnly)
return;
/*
--- 6383,6389 ----
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/*
***************
*** 6590,6596 ****
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 6643,6649 ----
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 6985,6991 ****
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (dataOnly)
return;
if (OidIsValid(cast->castfunc))
--- 7038,7044 ----
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
if (OidIsValid(cast->castfunc))
***************
*** 7135,7141 ****
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || dataOnly)
return;
/*
--- 7188,7194 ----
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/*
***************
*** 7519,7525 ****
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || dataOnly)
return;
/*
--- 7572,7578 ----
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/*
***************
*** 7827,7833 ****
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || dataOnly)
return;
/*
--- 7880,7886 ----
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/*
***************
*** 8096,8102 ****
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8149,8155 ----
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8250,8256 ****
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8303,8309 ----
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8453,8459 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8506,8512 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8522,8528 ****
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8575,8581 ----
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8607,8613 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8660,8666 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8673,8679 ****
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8726,8732 ----
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8809,8815 ****
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (dataOnly || aclsSkip)
return;
sql = createPQExpBuffer();
--- 8862,8868 ----
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags) || aclsSkip)
return;
sql = createPQExpBuffer();
***************
*** 8846,8852 ****
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (!dataOnly)
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
--- 8899,8905 ----
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
***************
*** 9153,9159 ****
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || dataOnly)
return;
/* Don't print inherited defaults, either */
--- 9206,9212 ----
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* Don't print inherited defaults, either */
***************
*** 9238,9244 ****
PQExpBuffer q;
PQExpBuffer delq;
! if (dataOnly)
return;
q = createPQExpBuffer();
--- 9291,9297 ----
PQExpBuffer q;
PQExpBuffer delq;
! if (!WANT_SCHEMA_AFTER_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9307,9313 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 9360,9366 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || !WANT_SCHEMA_AFTER_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9700,9706 ****
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (!dataOnly)
{
resetPQExpBuffer(delqry);
--- 9753,9759 ----
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
{
resetPQExpBuffer(delqry);
***************
*** 9803,9809 ****
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (!schemaOnly)
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
--- 9856,9862 ----
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (WANT_DATA(dumpObjFlags))
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
***************
*** 9836,9842 ****
const char *p;
int findx;
! if (dataOnly)
return;
query = createPQExpBuffer();
--- 9889,9895 ----
const char *p;
int findx;
! if (!WANT_SCHEMA_AFTER_DATA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 10044,10050 ****
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || dataOnly)
return;
/*
--- 10097,10103 ----
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || !WANT_SCHEMA_AFTER_DATA(dumpObjFlags))
return;
/*
Index: src/bin/pg_dump/pg_restore.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_restore.c,v
retrieving revision 1.88
diff -c -r1.88 pg_restore.c
*** src/bin/pg_dump/pg_restore.c 13 Apr 2008 03:49:22 -0000 1.88
--- src/bin/pg_dump/pg_restore.c 24 Jul 2008 07:30:19 -0000
***************
*** 78,83 ****
--- 78,90 ----
static int no_data_for_failed_tables = 0;
static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ bool dataOnly = false;
+ bool schemaOnly = false;
+
+ static int schemaBeforeData;
+ static int schemaAfterData;
+
+ int dumpObjFlags;
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
***************
*** 114,119 ****
--- 121,128 ----
{"disable-triggers", no_argument, &disable_triggers, 1},
{"no-data-for-failed-tables", no_argument, &no_data_for_failed_tables, 1},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-before-data", no_argument, &schemaBeforeData, 1},
+ {"schema-after-data", no_argument, &schemaAfterData, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 145,151 ****
switch (c)
{
case 'a': /* Dump data only */
! opts->dataOnly = 1;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
--- 154,160 ----
switch (c)
{
case 'a': /* Dump data only */
! dataOnly = true;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
***************
*** 213,219 ****
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! opts->schemaOnly = 1;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
--- 222,228 ----
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! schemaOnly = true;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
***************
*** 249,254 ****
--- 258,267 ----
no_data_for_failed_tables = 1;
else if (strcmp(optarg, "no-tablespaces") == 0)
outputNoTablespaces = 1;
+ else if (strcmp(optarg, "schema-before-data") == 0)
+ schemaBeforeData = 1;
+ else if (strcmp(optarg, "schema-after-data") == 0)
+ schemaAfterData = 1;
else if (strcmp(optarg, "use-set-session-authorization") == 0)
use_setsessauth = 1;
else
***************
*** 295,300 ****
--- 308,354 ----
opts->useDB = 1;
}
+ /*
+ * Look for conflicting options relating to object groupings
+ */
+ if (schemaOnly && dataOnly)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "-s/--schema-only", "-a/--data-only");
+ exit(1);
+ }
+ else if ((schemaOnly || dataOnly) &&
+ (schemaBeforeData || schemaAfterData))
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ schemaOnly ? "-s/--schema-only" : "-a/--data-only",
+ schemaBeforeData ? "--schema-before-data" : "--schema-after-data");
+ exit(1);
+ }
+ else if (schemaBeforeData && schemaAfterData)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "--schema-before-data", "--schema-after-data");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (schemaBeforeData)
+ dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
+
+ if (schemaAfterData)
+ dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
+
opts->disable_triggers = disable_triggers;
opts->noDataForFailedTables = no_data_for_failed_tables;
opts->noTablespace = outputNoTablespaces;
***************
*** 405,410 ****
--- 459,466 ----
" do not restore data of tables that could not be\n"
" created\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
+ printf(_(" --schema-before-data dump only the part of schema before table data\n"));
+ printf(_(" --schema-after-data dump only the part of schema after table data\n"));
printf(_(" --use-set-session-authorization\n"
" use SESSION AUTHORIZATION commands instead of\n"
" OWNER TO commands\n"));
Index: src/interfaces/ecpg/preproc/keywords.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/interfaces/ecpg/preproc/keywords.c,v
retrieving revision 1.85
diff -c -r1.85 keywords.c
*** src/interfaces/ecpg/preproc/keywords.c 1 Jan 2008 19:45:59 -0000 1.85
--- src/interfaces/ecpg/preproc/keywords.c 18 Jul 2008 08:50:27 -0000
***************
*** 1,430 ****
/*-------------------------------------------------------------------------
*
* keywords.c
! * lexical token lookup for reserved words in PostgreSQL
*
* Portions Copyright (c) 1996-2008, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
*
* IDENTIFICATION
! * $PostgreSQL: pgsql/src/interfaces/ecpg/preproc/keywords.c,v 1.84 2007/11/15 21:14:45 momjian Exp $
*
*-------------------------------------------------------------------------
*/
- #include "postgres_fe.h"
-
- #include <ctype.h>
! #include "extern.h"
! #include "preproc.h"
! /* compile both keyword lists in one file because they are always scanned together */
! #include "ecpg_keywords.c"
/*
! * List of (keyword-name, keyword-token-value) pairs.
! *
! * !!WARNING!!: This list must be sorted, because binary
! * search is used to locate entries.
*/
! static const ScanKeyword ScanPGSQLKeywords[] = {
! /* name, value */
! {"abort", ABORT_P},
! {"absolute", ABSOLUTE_P},
! {"access", ACCESS},
! {"action", ACTION},
! {"add", ADD_P},
! {"admin", ADMIN},
! {"after", AFTER},
! {"aggregate", AGGREGATE},
! {"all", ALL},
! {"also", ALSO},
! {"alter", ALTER},
! {"always", ALWAYS},
! {"analyse", ANALYSE}, /* British spelling */
! {"analyze", ANALYZE},
! {"and", AND},
! {"any", ANY},
! {"array", ARRAY},
! {"as", AS},
! {"asc", ASC},
! {"assertion", ASSERTION},
! {"assignment", ASSIGNMENT},
! {"asymmetric", ASYMMETRIC},
! {"at", AT},
! {"authorization", AUTHORIZATION},
! {"backward", BACKWARD},
! {"before", BEFORE},
! {"begin", BEGIN_P},
! {"between", BETWEEN},
! {"bigint", BIGINT},
! {"binary", BINARY},
! {"bit", BIT},
! {"boolean", BOOLEAN_P},
! {"both", BOTH},
! {"by", BY},
! {"cache", CACHE},
! {"called", CALLED},
! {"cascade", CASCADE},
! {"cascaded", CASCADED},
! {"case", CASE},
! {"cast", CAST},
! {"chain", CHAIN},
! {"char", CHAR_P},
! {"character", CHARACTER},
! {"characteristics", CHARACTERISTICS},
! {"check", CHECK},
! {"checkpoint", CHECKPOINT},
! {"class", CLASS},
! {"close", CLOSE},
! {"cluster", CLUSTER},
! {"coalesce", COALESCE},
! {"collate", COLLATE},
! {"column", COLUMN},
! {"comment", COMMENT},
! {"commit", COMMIT},
! {"committed", COMMITTED},
! {"concurrently", CONCURRENTLY},
! {"configuration", CONFIGURATION},
! {"connection", CONNECTION},
! {"constraint", CONSTRAINT},
! {"constraints", CONSTRAINTS},
! {"content", CONTENT_P},
! {"conversion", CONVERSION_P},
! {"copy", COPY},
! {"cost", COST},
! {"create", CREATE},
! {"createdb", CREATEDB},
! {"createrole", CREATEROLE},
! {"createuser", CREATEUSER},
! {"cross", CROSS},
! {"csv", CSV},
! {"current", CURRENT_P},
! {"current_date", CURRENT_DATE},
! {"current_role", CURRENT_ROLE},
! {"current_time", CURRENT_TIME},
! {"current_timestamp", CURRENT_TIMESTAMP},
! {"cursor", CURSOR},
! {"cycle", CYCLE},
! {"database", DATABASE},
! {"day", DAY_P},
! {"deallocate", DEALLOCATE},
! {"dec", DEC},
! {"decimal", DECIMAL_P},
! {"declare", DECLARE},
! {"default", DEFAULT},
! {"defaults", DEFAULTS},
! {"deferrable", DEFERRABLE},
! {"deferred", DEFERRED},
! {"definer", DEFINER},
! {"delete", DELETE_P},
! {"delimiter", DELIMITER},
! {"delimiters", DELIMITERS},
! {"desc", DESC},
! {"dictionary", DICTIONARY},
! {"disable", DISABLE_P},
! {"discard", DISCARD},
! {"distinct", DISTINCT},
! {"do", DO},
! {"document", DOCUMENT_P},
! {"domain", DOMAIN_P},
! {"double", DOUBLE_P},
! {"drop", DROP},
! {"each", EACH},
! {"else", ELSE},
! {"enable", ENABLE_P},
! {"encoding", ENCODING},
! {"encrypted", ENCRYPTED},
! {"end", END_P},
! {"enum", ENUM_P},
! {"escape", ESCAPE},
! {"except", EXCEPT},
! {"excluding", EXCLUDING},
! {"exclusive", EXCLUSIVE},
! {"execute", EXECUTE},
! {"exists", EXISTS},
! {"explain", EXPLAIN},
! {"external", EXTERNAL},
! {"extract", EXTRACT},
! {"false", FALSE_P},
! {"family", FAMILY},
! {"fetch", FETCH},
! {"first", FIRST_P},
! {"float", FLOAT_P},
! {"for", FOR},
! {"force", FORCE},
! {"foreign", FOREIGN},
! {"forward", FORWARD},
! {"freeze", FREEZE},
! {"from", FROM},
! {"full", FULL},
! {"function", FUNCTION},
! {"get", GET},
! {"global", GLOBAL},
! {"grant", GRANT},
! {"granted", GRANTED},
! {"greatest", GREATEST},
! {"group", GROUP_P},
! {"handler", HANDLER},
! {"having", HAVING},
! {"header", HEADER_P},
! {"hold", HOLD},
! {"hour", HOUR_P},
! {"if", IF_P},
! {"ilike", ILIKE},
! {"immediate", IMMEDIATE},
! {"immutable", IMMUTABLE},
! {"implicit", IMPLICIT_P},
! {"in", IN_P},
! {"including", INCLUDING},
! {"increment", INCREMENT},
! {"index", INDEX},
! {"indexes", INDEXES},
! {"inherit", INHERIT},
! {"inherits", INHERITS},
! {"initially", INITIALLY},
! {"inner", INNER_P},
! {"inout", INOUT},
! {"input", INPUT_P},
! {"insensitive", INSENSITIVE},
! {"insert", INSERT},
! {"instead", INSTEAD},
! {"int", INT_P},
! {"integer", INTEGER},
! {"intersect", INTERSECT},
! {"interval", INTERVAL},
! {"into", INTO},
! {"invoker", INVOKER},
! {"is", IS},
! {"isnull", ISNULL},
! {"isolation", ISOLATION},
! {"join", JOIN},
! {"key", KEY},
! {"lancompiler", LANCOMPILER},
! {"language", LANGUAGE},
! {"large", LARGE_P},
! {"last", LAST_P},
! {"leading", LEADING},
! {"least", LEAST},
! {"left", LEFT},
! {"level", LEVEL},
! {"like", LIKE},
! {"limit", LIMIT},
! {"listen", LISTEN},
! {"load", LOAD},
! {"local", LOCAL},
! {"location", LOCATION},
! {"lock", LOCK_P},
! {"login", LOGIN_P},
! {"mapping", MAPPING},
! {"match", MATCH},
! {"maxvalue", MAXVALUE},
! {"minute", MINUTE_P},
! {"minvalue", MINVALUE},
! {"mode", MODE},
! {"month", MONTH_P},
! {"move", MOVE},
! {"name", NAME_P},
! {"names", NAMES},
! {"national", NATIONAL},
! {"natural", NATURAL},
! {"nchar", NCHAR},
! {"new", NEW},
! {"next", NEXT},
! {"no", NO},
! {"nocreatedb", NOCREATEDB},
! {"nocreaterole", NOCREATEROLE},
! {"nocreateuser", NOCREATEUSER},
! {"noinherit", NOINHERIT},
! {"nologin", NOLOGIN_P},
! {"none", NONE},
! {"nosuperuser", NOSUPERUSER},
! {"not", NOT},
! {"nothing", NOTHING},
! {"notify", NOTIFY},
! {"notnull", NOTNULL},
! {"nowait", NOWAIT},
! {"null", NULL_P},
! {"nullif", NULLIF},
! {"nulls", NULLS_P},
! {"numeric", NUMERIC},
! {"object", OBJECT_P},
! {"of", OF},
! {"off", OFF},
! {"offset", OFFSET},
! {"oids", OIDS},
! {"old", OLD},
! {"on", ON},
! {"only", ONLY},
! {"operator", OPERATOR},
! {"option", OPTION},
! {"or", OR},
! {"order", ORDER},
! {"out", OUT_P},
! {"outer", OUTER_P},
! {"overlaps", OVERLAPS},
! {"owned", OWNED},
! {"owner", OWNER},
! {"parser", PARSER},
! {"partial", PARTIAL},
! {"password", PASSWORD},
! {"placing", PLACING},
! {"plans", PLANS},
! {"position", POSITION},
! {"precision", PRECISION},
! {"prepare", PREPARE},
! {"prepared", PREPARED},
! {"preserve", PRESERVE},
! {"primary", PRIMARY},
! {"prior", PRIOR},
! {"privileges", PRIVILEGES},
! {"procedural", PROCEDURAL},
! {"procedure", PROCEDURE},
! {"quote", QUOTE},
! {"read", READ},
! {"real", REAL},
! {"reassign", REASSIGN},
! {"recheck", RECHECK},
! {"references", REFERENCES},
! {"reindex", REINDEX},
! {"relative", RELATIVE_P},
! {"release", RELEASE},
! {"rename", RENAME},
! {"repeatable", REPEATABLE},
! {"replace", REPLACE},
! {"replica", REPLICA},
! {"reset", RESET},
! {"restart", RESTART},
! {"restrict", RESTRICT},
! {"returning", RETURNING},
! {"returns", RETURNS},
! {"revoke", REVOKE},
! {"right", RIGHT},
! {"role", ROLE},
! {"rollback", ROLLBACK},
! {"row", ROW},
! {"rows", ROWS},
! {"rule", RULE},
! {"savepoint", SAVEPOINT},
! {"schema", SCHEMA},
! {"scroll", SCROLL},
! {"search", SEARCH},
! {"second", SECOND_P},
! {"security", SECURITY},
! {"select", SELECT},
! {"sequence", SEQUENCE},
! {"serializable", SERIALIZABLE},
! {"session", SESSION},
! {"session_user", SESSION_USER},
! {"set", SET},
! {"setof", SETOF},
! {"share", SHARE},
! {"show", SHOW},
! {"similar", SIMILAR},
! {"simple", SIMPLE},
! {"smallint", SMALLINT},
! {"some", SOME},
! {"stable", STABLE},
! {"standalone", STANDALONE_P},
! {"start", START},
! {"statement", STATEMENT},
! {"statistics", STATISTICS},
! {"stdin", STDIN},
! {"stdout", STDOUT},
! {"storage", STORAGE},
! {"strict", STRICT_P},
! {"strip", STRIP_P},
! {"substring", SUBSTRING},
! {"superuser", SUPERUSER_P},
! {"symmetric", SYMMETRIC},
! {"sysid", SYSID},
! {"system", SYSTEM_P},
! {"table", TABLE},
! {"tablespace", TABLESPACE},
! {"temp", TEMP},
! {"template", TEMPLATE},
! {"temporary", TEMPORARY},
! {"text", TEXT_P},
! {"then", THEN},
! {"time", TIME},
! {"timestamp", TIMESTAMP},
! {"to", TO},
! {"trailing", TRAILING},
! {"transaction", TRANSACTION},
! {"treat", TREAT},
! {"trigger", TRIGGER},
! {"trim", TRIM},
! {"true", TRUE_P},
! {"truncate", TRUNCATE},
! {"trusted", TRUSTED},
! {"type", TYPE_P},
! {"uncommitted", UNCOMMITTED},
! {"unencrypted", UNENCRYPTED},
! {"union", UNION},
! {"unique", UNIQUE},
! {"unknown", UNKNOWN},
! {"unlisten", UNLISTEN},
! {"until", UNTIL},
! {"update", UPDATE},
! {"user", USER},
! {"using", USING},
! {"vacuum", VACUUM},
! {"valid", VALID},
! {"validator", VALIDATOR},
! {"value", VALUE_P},
! {"values", VALUES},
! {"varchar", VARCHAR},
! {"varying", VARYING},
! {"verbose", VERBOSE},
! {"version", VERSION_P},
! {"view", VIEW},
! {"volatile", VOLATILE},
! {"when", WHEN},
! {"where", WHERE},
! {"whitespace", WHITESPACE_P},
! {"with", WITH},
! {"without", WITHOUT},
! {"work", WORK},
! {"write", WRITE},
! {"xml", XML_P},
! {"xmlattributes", XMLATTRIBUTES},
! {"xmlconcat", XMLCONCAT},
! {"xmlelement", XMLELEMENT},
! {"xmlforest", XMLFOREST},
! {"xmlparse", XMLPARSE},
! {"xmlpi", XMLPI},
! {"xmlroot", XMLROOT},
! {"xmlserialize", XMLSERIALIZE},
! {"year", YEAR_P},
! {"yes", YES_P},
! {"zone", ZONE},
! };
/*
! * Now do a binary search using plain strcmp() comparison.
*/
! const ScanKeyword *
! DoLookup(char *word, const ScanKeyword *low, const ScanKeyword *high)
! {
! while (low <= high)
! {
! const ScanKeyword *middle;
! int difference;
! middle = low + (high - low) / 2;
! difference = strcmp(middle->name, word);
! if (difference == 0)
! return middle;
! else if (difference < 0)
! low = middle + 1;
! else
! high = middle - 1;
! }
! return NULL;
! }
/*
* ScanKeywordLookup - see if a given word is a keyword
--- 1,436 ----
/*-------------------------------------------------------------------------
*
* keywords.c
! * lexical token lookup for key words in PostgreSQL
! *
! * NB: This file is also used by pg_dump.
! *
*
* Portions Copyright (c) 1996-2008, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
*
* IDENTIFICATION
! * $PostgreSQL: pgsql/src/backend/parser/keywords.c,v 1.198 2008/07/03 20:58:46 tgl Exp $
*
*-------------------------------------------------------------------------
*/
! /* Use c.h so that this file can be built in either frontend or backend */
! #include "c.h"
! #include <ctype.h>
/*
! * This macro definition overrides the YYSTYPE union definition in parse.h.
! * We don't need that struct in this file, and including the real definition
! * would require sucking in some backend-only include files.
*/
! #define YYSTYPE int
+ #include "parser/keywords.h"
+ #ifndef ECPG_COMPILE
+ #include "parser/parse.h"
+ #else
+ #include "preproc.h"
+ #endif
/*
! * List of keyword (name, token-value, category) entries.
! *
! * !!WARNING!!: This list must be sorted by ASCII name, because binary
! * search is used to locate entries.
*/
! const ScanKeyword ScanKeywords[] = {
! /* name, value, category */
! {"abort", ABORT_P, UNRESERVED_KEYWORD},
! {"absolute", ABSOLUTE_P, UNRESERVED_KEYWORD},
! {"access", ACCESS, UNRESERVED_KEYWORD},
! {"action", ACTION, UNRESERVED_KEYWORD},
! {"add", ADD_P, UNRESERVED_KEYWORD},
! {"admin", ADMIN, UNRESERVED_KEYWORD},
! {"after", AFTER, UNRESERVED_KEYWORD},
! {"aggregate", AGGREGATE, UNRESERVED_KEYWORD},
! {"all", ALL, RESERVED_KEYWORD},
! {"also", ALSO, UNRESERVED_KEYWORD},
! {"alter", ALTER, UNRESERVED_KEYWORD},
! {"always", ALWAYS, UNRESERVED_KEYWORD},
! {"analyse", ANALYSE, RESERVED_KEYWORD}, /* British spelling */
! {"analyze", ANALYZE, RESERVED_KEYWORD},
! {"and", AND, RESERVED_KEYWORD},
! {"any", ANY, RESERVED_KEYWORD},
! {"array", ARRAY, RESERVED_KEYWORD},
! {"as", AS, RESERVED_KEYWORD},
! {"asc", ASC, RESERVED_KEYWORD},
! {"assertion", ASSERTION, UNRESERVED_KEYWORD},
! {"assignment", ASSIGNMENT, UNRESERVED_KEYWORD},
! {"asymmetric", ASYMMETRIC, RESERVED_KEYWORD},
! {"at", AT, UNRESERVED_KEYWORD},
! {"authorization", AUTHORIZATION, TYPE_FUNC_NAME_KEYWORD},
! {"backward", BACKWARD, UNRESERVED_KEYWORD},
! {"before", BEFORE, UNRESERVED_KEYWORD},
! {"begin", BEGIN_P, UNRESERVED_KEYWORD},
! {"between", BETWEEN, TYPE_FUNC_NAME_KEYWORD},
! {"bigint", BIGINT, COL_NAME_KEYWORD},
! {"binary", BINARY, TYPE_FUNC_NAME_KEYWORD},
! {"bit", BIT, COL_NAME_KEYWORD},
! {"boolean", BOOLEAN_P, COL_NAME_KEYWORD},
! {"both", BOTH, RESERVED_KEYWORD},
! {"by", BY, UNRESERVED_KEYWORD},
! {"cache", CACHE, UNRESERVED_KEYWORD},
! {"called", CALLED, UNRESERVED_KEYWORD},
! {"cascade", CASCADE, UNRESERVED_KEYWORD},
! {"cascaded", CASCADED, UNRESERVED_KEYWORD},
! {"case", CASE, RESERVED_KEYWORD},
! {"cast", CAST, RESERVED_KEYWORD},
! {"chain", CHAIN, UNRESERVED_KEYWORD},
! {"char", CHAR_P, COL_NAME_KEYWORD},
! {"character", CHARACTER, COL_NAME_KEYWORD},
! {"characteristics", CHARACTERISTICS, UNRESERVED_KEYWORD},
! {"check", CHECK, RESERVED_KEYWORD},
! {"checkpoint", CHECKPOINT, UNRESERVED_KEYWORD},
! {"class", CLASS, UNRESERVED_KEYWORD},
! {"close", CLOSE, UNRESERVED_KEYWORD},
! {"cluster", CLUSTER, UNRESERVED_KEYWORD},
! {"coalesce", COALESCE, COL_NAME_KEYWORD},
! {"collate", COLLATE, RESERVED_KEYWORD},
! {"column", COLUMN, RESERVED_KEYWORD},
! {"comment", COMMENT, UNRESERVED_KEYWORD},
! {"commit", COMMIT, UNRESERVED_KEYWORD},
! {"committed", COMMITTED, UNRESERVED_KEYWORD},
! {"concurrently", CONCURRENTLY, UNRESERVED_KEYWORD},
! {"configuration", CONFIGURATION, UNRESERVED_KEYWORD},
! {"connection", CONNECTION, UNRESERVED_KEYWORD},
! {"constraint", CONSTRAINT, RESERVED_KEYWORD},
! {"constraints", CONSTRAINTS, UNRESERVED_KEYWORD},
! {"content", CONTENT_P, UNRESERVED_KEYWORD},
! {"continue", CONTINUE_P, UNRESERVED_KEYWORD},
! {"conversion", CONVERSION_P, UNRESERVED_KEYWORD},
! {"copy", COPY, UNRESERVED_KEYWORD},
! {"cost", COST, UNRESERVED_KEYWORD},
! {"create", CREATE, RESERVED_KEYWORD},
! {"createdb", CREATEDB, UNRESERVED_KEYWORD},
! {"createrole", CREATEROLE, UNRESERVED_KEYWORD},
! {"createuser", CREATEUSER, UNRESERVED_KEYWORD},
! {"cross", CROSS, TYPE_FUNC_NAME_KEYWORD},
! {"csv", CSV, UNRESERVED_KEYWORD},
! {"current", CURRENT_P, UNRESERVED_KEYWORD},
! {"current_date", CURRENT_DATE, RESERVED_KEYWORD},
! {"current_role", CURRENT_ROLE, RESERVED_KEYWORD},
! {"current_time", CURRENT_TIME, RESERVED_KEYWORD},
! {"current_timestamp", CURRENT_TIMESTAMP, RESERVED_KEYWORD},
! {"current_user", CURRENT_USER, RESERVED_KEYWORD},
! {"cursor", CURSOR, UNRESERVED_KEYWORD},
! {"cycle", CYCLE, UNRESERVED_KEYWORD},
! {"database", DATABASE, UNRESERVED_KEYWORD},
! {"day", DAY_P, UNRESERVED_KEYWORD},
! {"deallocate", DEALLOCATE, UNRESERVED_KEYWORD},
! {"dec", DEC, COL_NAME_KEYWORD},
! {"decimal", DECIMAL_P, COL_NAME_KEYWORD},
! {"declare", DECLARE, UNRESERVED_KEYWORD},
! {"default", DEFAULT, RESERVED_KEYWORD},
! {"defaults", DEFAULTS, UNRESERVED_KEYWORD},
! {"deferrable", DEFERRABLE, RESERVED_KEYWORD},
! {"deferred", DEFERRED, UNRESERVED_KEYWORD},
! {"definer", DEFINER, UNRESERVED_KEYWORD},
! {"delete", DELETE_P, UNRESERVED_KEYWORD},
! {"delimiter", DELIMITER, UNRESERVED_KEYWORD},
! {"delimiters", DELIMITERS, UNRESERVED_KEYWORD},
! {"desc", DESC, RESERVED_KEYWORD},
! {"dictionary", DICTIONARY, UNRESERVED_KEYWORD},
! {"disable", DISABLE_P, UNRESERVED_KEYWORD},
! {"discard", DISCARD, UNRESERVED_KEYWORD},
! {"distinct", DISTINCT, RESERVED_KEYWORD},
! {"do", DO, RESERVED_KEYWORD},
! {"document", DOCUMENT_P, UNRESERVED_KEYWORD},
! {"domain", DOMAIN_P, UNRESERVED_KEYWORD},
! {"double", DOUBLE_P, UNRESERVED_KEYWORD},
! {"drop", DROP, UNRESERVED_KEYWORD},
! {"each", EACH, UNRESERVED_KEYWORD},
! {"else", ELSE, RESERVED_KEYWORD},
! {"enable", ENABLE_P, UNRESERVED_KEYWORD},
! {"encoding", ENCODING, UNRESERVED_KEYWORD},
! {"encrypted", ENCRYPTED, UNRESERVED_KEYWORD},
! {"end", END_P, RESERVED_KEYWORD},
! {"enum", ENUM_P, UNRESERVED_KEYWORD},
! {"escape", ESCAPE, UNRESERVED_KEYWORD},
! {"except", EXCEPT, RESERVED_KEYWORD},
! {"excluding", EXCLUDING, UNRESERVED_KEYWORD},
! {"exclusive", EXCLUSIVE, UNRESERVED_KEYWORD},
! {"execute", EXECUTE, UNRESERVED_KEYWORD},
! {"exists", EXISTS, COL_NAME_KEYWORD},
! {"explain", EXPLAIN, UNRESERVED_KEYWORD},
! {"external", EXTERNAL, UNRESERVED_KEYWORD},
! {"extract", EXTRACT, COL_NAME_KEYWORD},
! {"false", FALSE_P, RESERVED_KEYWORD},
! {"family", FAMILY, UNRESERVED_KEYWORD},
! {"fetch", FETCH, UNRESERVED_KEYWORD},
! {"first", FIRST_P, UNRESERVED_KEYWORD},
! {"float", FLOAT_P, COL_NAME_KEYWORD},
! {"for", FOR, RESERVED_KEYWORD},
! {"force", FORCE, UNRESERVED_KEYWORD},
! {"foreign", FOREIGN, RESERVED_KEYWORD},
! {"forward", FORWARD, UNRESERVED_KEYWORD},
! {"freeze", FREEZE, TYPE_FUNC_NAME_KEYWORD},
! {"from", FROM, RESERVED_KEYWORD},
! {"full", FULL, TYPE_FUNC_NAME_KEYWORD},
! {"function", FUNCTION, UNRESERVED_KEYWORD},
! {"global", GLOBAL, UNRESERVED_KEYWORD},
! {"grant", GRANT, RESERVED_KEYWORD},
! {"granted", GRANTED, UNRESERVED_KEYWORD},
! {"greatest", GREATEST, COL_NAME_KEYWORD},
! {"group", GROUP_P, RESERVED_KEYWORD},
! {"handler", HANDLER, UNRESERVED_KEYWORD},
! {"having", HAVING, RESERVED_KEYWORD},
! {"header", HEADER_P, UNRESERVED_KEYWORD},
! {"hold", HOLD, UNRESERVED_KEYWORD},
! {"hour", HOUR_P, UNRESERVED_KEYWORD},
! {"identity", IDENTITY_P, UNRESERVED_KEYWORD},
! {"if", IF_P, UNRESERVED_KEYWORD},
! {"ilike", ILIKE, TYPE_FUNC_NAME_KEYWORD},
! {"immediate", IMMEDIATE, UNRESERVED_KEYWORD},
! {"immutable", IMMUTABLE, UNRESERVED_KEYWORD},
! {"implicit", IMPLICIT_P, UNRESERVED_KEYWORD},
! {"in", IN_P, RESERVED_KEYWORD},
! {"including", INCLUDING, UNRESERVED_KEYWORD},
! {"increment", INCREMENT, UNRESERVED_KEYWORD},
! {"index", INDEX, UNRESERVED_KEYWORD},
! {"indexes", INDEXES, UNRESERVED_KEYWORD},
! {"inherit", INHERIT, UNRESERVED_KEYWORD},
! {"inherits", INHERITS, UNRESERVED_KEYWORD},
! {"initially", INITIALLY, RESERVED_KEYWORD},
! {"inner", INNER_P, TYPE_FUNC_NAME_KEYWORD},
! {"inout", INOUT, COL_NAME_KEYWORD},
! {"input", INPUT_P, UNRESERVED_KEYWORD},
! {"insensitive", INSENSITIVE, UNRESERVED_KEYWORD},
! {"insert", INSERT, UNRESERVED_KEYWORD},
! {"instead", INSTEAD, UNRESERVED_KEYWORD},
! {"int", INT_P, COL_NAME_KEYWORD},
! {"integer", INTEGER, COL_NAME_KEYWORD},
! {"intersect", INTERSECT, RESERVED_KEYWORD},
! {"interval", INTERVAL, COL_NAME_KEYWORD},
! {"into", INTO, RESERVED_KEYWORD},
! {"invoker", INVOKER, UNRESERVED_KEYWORD},
! {"is", IS, TYPE_FUNC_NAME_KEYWORD},
! {"isnull", ISNULL, TYPE_FUNC_NAME_KEYWORD},
! {"isolation", ISOLATION, UNRESERVED_KEYWORD},
! {"join", JOIN, TYPE_FUNC_NAME_KEYWORD},
! {"key", KEY, UNRESERVED_KEYWORD},
! {"lancompiler", LANCOMPILER, UNRESERVED_KEYWORD},
! {"language", LANGUAGE, UNRESERVED_KEYWORD},
! {"large", LARGE_P, UNRESERVED_KEYWORD},
! {"last", LAST_P, UNRESERVED_KEYWORD},
! {"leading", LEADING, RESERVED_KEYWORD},
! {"least", LEAST, COL_NAME_KEYWORD},
! {"left", LEFT, TYPE_FUNC_NAME_KEYWORD},
! {"level", LEVEL, UNRESERVED_KEYWORD},
! {"like", LIKE, TYPE_FUNC_NAME_KEYWORD},
! {"limit", LIMIT, RESERVED_KEYWORD},
! {"listen", LISTEN, UNRESERVED_KEYWORD},
! {"load", LOAD, UNRESERVED_KEYWORD},
! {"local", LOCAL, UNRESERVED_KEYWORD},
! {"localtime", LOCALTIME, RESERVED_KEYWORD},
! {"localtimestamp", LOCALTIMESTAMP, RESERVED_KEYWORD},
! {"location", LOCATION, UNRESERVED_KEYWORD},
! {"lock", LOCK_P, UNRESERVED_KEYWORD},
! {"login", LOGIN_P, UNRESERVED_KEYWORD},
! {"mapping", MAPPING, UNRESERVED_KEYWORD},
! {"match", MATCH, UNRESERVED_KEYWORD},
! {"maxvalue", MAXVALUE, UNRESERVED_KEYWORD},
! {"minute", MINUTE_P, UNRESERVED_KEYWORD},
! {"minvalue", MINVALUE, UNRESERVED_KEYWORD},
! {"mode", MODE, UNRESERVED_KEYWORD},
! {"month", MONTH_P, UNRESERVED_KEYWORD},
! {"move", MOVE, UNRESERVED_KEYWORD},
! {"name", NAME_P, UNRESERVED_KEYWORD},
! {"names", NAMES, UNRESERVED_KEYWORD},
! {"national", NATIONAL, COL_NAME_KEYWORD},
! {"natural", NATURAL, TYPE_FUNC_NAME_KEYWORD},
! {"nchar", NCHAR, COL_NAME_KEYWORD},
! {"new", NEW, RESERVED_KEYWORD},
! {"next", NEXT, UNRESERVED_KEYWORD},
! {"no", NO, UNRESERVED_KEYWORD},
! {"nocreatedb", NOCREATEDB, UNRESERVED_KEYWORD},
! {"nocreaterole", NOCREATEROLE, UNRESERVED_KEYWORD},
! {"nocreateuser", NOCREATEUSER, UNRESERVED_KEYWORD},
! {"noinherit", NOINHERIT, UNRESERVED_KEYWORD},
! {"nologin", NOLOGIN_P, UNRESERVED_KEYWORD},
! {"none", NONE, COL_NAME_KEYWORD},
! {"nosuperuser", NOSUPERUSER, UNRESERVED_KEYWORD},
! {"not", NOT, RESERVED_KEYWORD},
! {"nothing", NOTHING, UNRESERVED_KEYWORD},
! {"notify", NOTIFY, UNRESERVED_KEYWORD},
! {"notnull", NOTNULL, TYPE_FUNC_NAME_KEYWORD},
! {"nowait", NOWAIT, UNRESERVED_KEYWORD},
! {"null", NULL_P, RESERVED_KEYWORD},
! {"nullif", NULLIF, COL_NAME_KEYWORD},
! {"nulls", NULLS_P, UNRESERVED_KEYWORD},
! {"numeric", NUMERIC, COL_NAME_KEYWORD},
! {"object", OBJECT_P, UNRESERVED_KEYWORD},
! {"of", OF, UNRESERVED_KEYWORD},
! {"off", OFF, RESERVED_KEYWORD},
! {"offset", OFFSET, RESERVED_KEYWORD},
! {"oids", OIDS, UNRESERVED_KEYWORD},
! {"old", OLD, RESERVED_KEYWORD},
! {"on", ON, RESERVED_KEYWORD},
! {"only", ONLY, RESERVED_KEYWORD},
! {"operator", OPERATOR, UNRESERVED_KEYWORD},
! {"option", OPTION, UNRESERVED_KEYWORD},
! {"or", OR, RESERVED_KEYWORD},
! {"order", ORDER, RESERVED_KEYWORD},
! {"out", OUT_P, COL_NAME_KEYWORD},
! {"outer", OUTER_P, TYPE_FUNC_NAME_KEYWORD},
! {"overlaps", OVERLAPS, TYPE_FUNC_NAME_KEYWORD},
! {"overlay", OVERLAY, COL_NAME_KEYWORD},
! {"owned", OWNED, UNRESERVED_KEYWORD},
! {"owner", OWNER, UNRESERVED_KEYWORD},
! {"parser", PARSER, UNRESERVED_KEYWORD},
! {"partial", PARTIAL, UNRESERVED_KEYWORD},
! {"password", PASSWORD, UNRESERVED_KEYWORD},
! {"placing", PLACING, RESERVED_KEYWORD},
! {"plans", PLANS, UNRESERVED_KEYWORD},
! {"position", POSITION, COL_NAME_KEYWORD},
! {"precision", PRECISION, COL_NAME_KEYWORD},
! {"prepare", PREPARE, UNRESERVED_KEYWORD},
! {"prepared", PREPARED, UNRESERVED_KEYWORD},
! {"preserve", PRESERVE, UNRESERVED_KEYWORD},
! {"primary", PRIMARY, RESERVED_KEYWORD},
! {"prior", PRIOR, UNRESERVED_KEYWORD},
! {"privileges", PRIVILEGES, UNRESERVED_KEYWORD},
! {"procedural", PROCEDURAL, UNRESERVED_KEYWORD},
! {"procedure", PROCEDURE, UNRESERVED_KEYWORD},
! {"quote", QUOTE, UNRESERVED_KEYWORD},
! {"read", READ, UNRESERVED_KEYWORD},
! {"real", REAL, COL_NAME_KEYWORD},
! {"reassign", REASSIGN, UNRESERVED_KEYWORD},
! {"recheck", RECHECK, UNRESERVED_KEYWORD},
! {"references", REFERENCES, RESERVED_KEYWORD},
! {"reindex", REINDEX, UNRESERVED_KEYWORD},
! {"relative", RELATIVE_P, UNRESERVED_KEYWORD},
! {"release", RELEASE, UNRESERVED_KEYWORD},
! {"rename", RENAME, UNRESERVED_KEYWORD},
! {"repeatable", REPEATABLE, UNRESERVED_KEYWORD},
! {"replace", REPLACE, UNRESERVED_KEYWORD},
! {"replica", REPLICA, UNRESERVED_KEYWORD},
! {"reset", RESET, UNRESERVED_KEYWORD},
! {"restart", RESTART, UNRESERVED_KEYWORD},
! {"restrict", RESTRICT, UNRESERVED_KEYWORD},
! {"returning", RETURNING, RESERVED_KEYWORD},
! {"returns", RETURNS, UNRESERVED_KEYWORD},
! {"revoke", REVOKE, UNRESERVED_KEYWORD},
! {"right", RIGHT, TYPE_FUNC_NAME_KEYWORD},
! {"role", ROLE, UNRESERVED_KEYWORD},
! {"rollback", ROLLBACK, UNRESERVED_KEYWORD},
! {"row", ROW, COL_NAME_KEYWORD},
! {"rows", ROWS, UNRESERVED_KEYWORD},
! {"rule", RULE, UNRESERVED_KEYWORD},
! {"savepoint", SAVEPOINT, UNRESERVED_KEYWORD},
! {"schema", SCHEMA, UNRESERVED_KEYWORD},
! {"scroll", SCROLL, UNRESERVED_KEYWORD},
! {"search", SEARCH, UNRESERVED_KEYWORD},
! {"second", SECOND_P, UNRESERVED_KEYWORD},
! {"security", SECURITY, UNRESERVED_KEYWORD},
! {"select", SELECT, RESERVED_KEYWORD},
! {"sequence", SEQUENCE, UNRESERVED_KEYWORD},
! {"serializable", SERIALIZABLE, UNRESERVED_KEYWORD},
! {"session", SESSION, UNRESERVED_KEYWORD},
! {"session_user", SESSION_USER, RESERVED_KEYWORD},
! {"set", SET, UNRESERVED_KEYWORD},
! {"setof", SETOF, COL_NAME_KEYWORD},
! {"share", SHARE, UNRESERVED_KEYWORD},
! {"show", SHOW, UNRESERVED_KEYWORD},
! {"similar", SIMILAR, TYPE_FUNC_NAME_KEYWORD},
! {"simple", SIMPLE, UNRESERVED_KEYWORD},
! {"smallint", SMALLINT, COL_NAME_KEYWORD},
! {"some", SOME, RESERVED_KEYWORD},
! {"stable", STABLE, UNRESERVED_KEYWORD},
! {"standalone", STANDALONE_P, UNRESERVED_KEYWORD},
! {"start", START, UNRESERVED_KEYWORD},
! {"statement", STATEMENT, UNRESERVED_KEYWORD},
! {"statistics", STATISTICS, UNRESERVED_KEYWORD},
! {"stdin", STDIN, UNRESERVED_KEYWORD},
! {"stdout", STDOUT, UNRESERVED_KEYWORD},
! {"storage", STORAGE, UNRESERVED_KEYWORD},
! {"strict", STRICT_P, UNRESERVED_KEYWORD},
! {"strip", STRIP_P, UNRESERVED_KEYWORD},
! {"substring", SUBSTRING, COL_NAME_KEYWORD},
! {"superuser", SUPERUSER_P, UNRESERVED_KEYWORD},
! {"symmetric", SYMMETRIC, RESERVED_KEYWORD},
! {"sysid", SYSID, UNRESERVED_KEYWORD},
! {"system", SYSTEM_P, UNRESERVED_KEYWORD},
! {"table", TABLE, RESERVED_KEYWORD},
! {"tablespace", TABLESPACE, UNRESERVED_KEYWORD},
! {"temp", TEMP, UNRESERVED_KEYWORD},
! {"template", TEMPLATE, UNRESERVED_KEYWORD},
! {"temporary", TEMPORARY, UNRESERVED_KEYWORD},
! {"text", TEXT_P, UNRESERVED_KEYWORD},
! {"then", THEN, RESERVED_KEYWORD},
! {"time", TIME, COL_NAME_KEYWORD},
! {"timestamp", TIMESTAMP, COL_NAME_KEYWORD},
! {"to", TO, RESERVED_KEYWORD},
! {"trailing", TRAILING, RESERVED_KEYWORD},
! {"transaction", TRANSACTION, UNRESERVED_KEYWORD},
! {"treat", TREAT, COL_NAME_KEYWORD},
! {"trigger", TRIGGER, UNRESERVED_KEYWORD},
! {"trim", TRIM, COL_NAME_KEYWORD},
! {"true", TRUE_P, RESERVED_KEYWORD},
! {"truncate", TRUNCATE, UNRESERVED_KEYWORD},
! {"trusted", TRUSTED, UNRESERVED_KEYWORD},
! {"type", TYPE_P, UNRESERVED_KEYWORD},
! {"uncommitted", UNCOMMITTED, UNRESERVED_KEYWORD},
! {"unencrypted", UNENCRYPTED, UNRESERVED_KEYWORD},
! {"union", UNION, RESERVED_KEYWORD},
! {"unique", UNIQUE, RESERVED_KEYWORD},
! {"unknown", UNKNOWN, UNRESERVED_KEYWORD},
! {"unlisten", UNLISTEN, UNRESERVED_KEYWORD},
! {"until", UNTIL, UNRESERVED_KEYWORD},
! {"update", UPDATE, UNRESERVED_KEYWORD},
! {"user", USER, RESERVED_KEYWORD},
! {"using", USING, RESERVED_KEYWORD},
! {"vacuum", VACUUM, UNRESERVED_KEYWORD},
! {"valid", VALID, UNRESERVED_KEYWORD},
! {"validator", VALIDATOR, UNRESERVED_KEYWORD},
! {"value", VALUE_P, UNRESERVED_KEYWORD},
! {"values", VALUES, COL_NAME_KEYWORD},
! {"varchar", VARCHAR, COL_NAME_KEYWORD},
! {"variadic", VARIADIC, RESERVED_KEYWORD},
! {"varying", VARYING, UNRESERVED_KEYWORD},
! {"verbose", VERBOSE, TYPE_FUNC_NAME_KEYWORD},
! {"version", VERSION_P, UNRESERVED_KEYWORD},
! {"view", VIEW, UNRESERVED_KEYWORD},
! {"volatile", VOLATILE, UNRESERVED_KEYWORD},
! {"when", WHEN, RESERVED_KEYWORD},
! {"where", WHERE, RESERVED_KEYWORD},
! {"whitespace", WHITESPACE_P, UNRESERVED_KEYWORD},
! /*
! * XXX we mark WITH as reserved to force it to be quoted in dumps, even
! * though it is currently unreserved according to gram.y. This is because
! * we expect we'll have to make it reserved to implement SQL WITH clauses.
! * If that patch manages to do without reserving WITH, adjust this entry
! * at that time; in any case this should be back in sync with gram.y after
! * WITH clauses are implemented.
! */
! {"with", WITH, RESERVED_KEYWORD},
! {"without", WITHOUT, UNRESERVED_KEYWORD},
! {"work", WORK, UNRESERVED_KEYWORD},
! {"write", WRITE, UNRESERVED_KEYWORD},
! {"xml", XML_P, UNRESERVED_KEYWORD},
! {"xmlattributes", XMLATTRIBUTES, COL_NAME_KEYWORD},
! {"xmlconcat", XMLCONCAT, COL_NAME_KEYWORD},
! {"xmlelement", XMLELEMENT, COL_NAME_KEYWORD},
! {"xmlforest", XMLFOREST, COL_NAME_KEYWORD},
! {"xmlparse", XMLPARSE, COL_NAME_KEYWORD},
! {"xmlpi", XMLPI, COL_NAME_KEYWORD},
! {"xmlroot", XMLROOT, COL_NAME_KEYWORD},
! {"xmlserialize", XMLSERIALIZE, COL_NAME_KEYWORD},
! {"year", YEAR_P, UNRESERVED_KEYWORD},
! {"yes", YES_P, UNRESERVED_KEYWORD},
! {"zone", ZONE, UNRESERVED_KEYWORD},
! };
! /* End of ScanKeywords, for use elsewhere */
! const ScanKeyword *LastScanKeyword = endof(ScanKeywords);
/*
* ScanKeywordLookup - see if a given word is a keyword
***************
*** 439,450 ****
* receive a different case-normalization mapping.
*/
const ScanKeyword *
! ScanKeywordLookup(char *text)
{
int len,
i;
char word[NAMEDATALEN];
! const ScanKeyword *res;
len = strlen(text);
/* We assume all keywords are shorter than NAMEDATALEN. */
--- 445,457 ----
* receive a different case-normalization mapping.
*/
const ScanKeyword *
! ScanKeywordLookup(const char *text)
{
int len,
i;
char word[NAMEDATALEN];
! const ScanKeyword *low;
! const ScanKeyword *high;
len = strlen(text);
/* We assume all keywords are shorter than NAMEDATALEN. */
***************
*** 468,476 ****
/*
* Now do a binary search using plain strcmp() comparison.
*/
! res = DoLookup(word, &ScanPGSQLKeywords[0], endof(ScanPGSQLKeywords) - 1);
! if (res)
! return res;
! return DoLookup(word, &ScanECPGKeywords[0], endof(ScanECPGKeywords) - 1);
}
--- 475,496 ----
/*
* Now do a binary search using plain strcmp() comparison.
*/
! low = &ScanKeywords[0];
! high = endof(ScanKeywords) - 1;
! while (low <= high)
! {
! const ScanKeyword *middle;
! int difference;
! middle = low + (high - low) / 2;
! difference = strcmp(middle->name, word);
! if (difference == 0)
! return middle;
! else if (difference < 0)
! low = middle + 1;
! else
! high = middle - 1;
! }
!
! return NULL;
}
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-24 07:54 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 54+ messages in thread
From: Tom Lane @ 2008-07-24 07:54 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-patches
Simon Riggs <simon@2ndquadrant.com> writes:
> [80k patch]
Surely there is a whole lot of unintended noise in this patch?
I certainly don't believe that you meant to change keywords.c
for instance.
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-24 08:56 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-07-24 08:56 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-patches
On Thu, 2008-07-24 at 03:54 -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > [80k patch]
>
> Surely there is a whole lot of unintended noise in this patch?
> I certainly don't believe that you meant to change keywords.c
> for instance.
Removed, thanks.
Unrelated to this patch, it seems I have some issues with my repository,
judging by this and another unrelated issue reported by Martin Zaun.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
Attachments:
pg_dump_beforeafter.v6.patch (text/x-patch, 38.7K) download
Index: doc/src/sgml/ref/pg_dump.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_dump.sgml,v
retrieving revision 1.103
diff -c -r1.103 pg_dump.sgml
*** doc/src/sgml/ref/pg_dump.sgml 20 Jul 2008 18:43:30 -0000 1.103
--- doc/src/sgml/ref/pg_dump.sgml 24 Jul 2008 07:30:19 -0000
***************
*** 133,139 ****
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
--- 133,140 ----
<para>
Include large objects in the dump. This is the default behavior
except when <option>--schema</>, <option>--table</>, or
! <option>--schema-only</> or <option>--schema-before-data</> or
! <option>--schema-after-data</> is specified, so the <option>-b</>
switch is only useful to add large objects to selective dumps.
</para>
</listitem>
***************
*** 426,431 ****
--- 427,452 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-before-data</option></term>
+ <listitem>
+ <para>
+ Dump object definitions (schema) that occur before table data,
+ using the order produced by a full dump.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--schema-after-data</option></term>
+ <listitem>
+ <para>
+ Dump object definitions (schema) that occur after table data,
+ using the order produced by a full dump.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 790,795 ****
--- 811,844 ----
</para>
<para>
+ The output of <application>pg_dump</application> can be divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Before Data - objects output before data, which includes
+ <command>CREATE TABLE</command> statements and others.
+ This part can be requested using <option>--schema-before-data</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ After Data - objects output after data, which includes
+ <command>CREATE INDEX</command> statements and others.
+ This part can be requested using <option>--schema-after-data</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
Because <application>pg_dump</application> is used to transfer data
to newer versions of <productname>PostgreSQL</>, the output of
<application>pg_dump</application> can be loaded into
Index: doc/src/sgml/ref/pg_restore.sgml
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/doc/src/sgml/ref/pg_restore.sgml,v
retrieving revision 1.75
diff -c -r1.75 pg_restore.sgml
*** doc/src/sgml/ref/pg_restore.sgml 13 Apr 2008 03:49:21 -0000 1.75
--- doc/src/sgml/ref/pg_restore.sgml 24 Jul 2008 07:30:19 -0000
***************
*** 321,326 ****
--- 321,346 ----
</varlistentry>
<varlistentry>
+ <term><option>--schema-before-data</option></term>
+ <listitem>
+ <para>
+ Restore object definitions (schema) that occur before table data,
+ using the order produced by a full restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><option>--schema-after-data</option></term>
+ <listitem>
+ <para>
+ Restore object definitions (schema) that occur after table data,
+ using the order produced by a full restore.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-S <replaceable class="parameter">username</replaceable></option></term>
<term><option>--superuser=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
***************
*** 572,577 ****
--- 592,626 ----
</para>
<para>
+ The actions of <application>pg_restore</application> can be
+ divided into three parts:
+ <itemizedlist>
+ <listitem>
+ <para>
+ Before Data - objects output before data, which includes
+ <command>CREATE TABLE</command> statements and others.
+ This part can be requested using <option>--schema-before-data</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ Table Data - data can be requested using <option>--data-only</>.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ After Data - objects output after data, which includes
+ <command>CREATE INDEX</command> statements and others.
+ This part can be requested using <option>--schema-after-data</>.
+ </para>
+ </listitem>
+ </itemizedlist>
+ This allows us to work more easily with large data dump files when
+ there is some need to edit commands or resequence their execution for
+ performance.
+ </para>
+
+ <para>
The limitations of <application>pg_restore</application> are detailed below.
<itemizedlist>
Index: src/bin/pg_dump/pg_backup.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup.h,v
retrieving revision 1.47
diff -c -r1.47 pg_backup.h
*** src/bin/pg_dump/pg_backup.h 13 Apr 2008 03:49:21 -0000 1.47
--- src/bin/pg_dump/pg_backup.h 24 Jul 2008 07:30:19 -0000
***************
*** 89,95 ****
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dataOnly;
int dropSchema;
char *filename;
int schemaOnly;
--- 89,95 ----
int use_setsessauth;/* Use SET SESSION AUTHORIZATION commands
* instead of OWNER TO */
char *superuser; /* Username to use as superuser */
! int dumpObjFlags; /* which objects types to dump */
int dropSchema;
char *filename;
int schemaOnly;
Index: src/bin/pg_dump/pg_backup_archiver.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.c,v
retrieving revision 1.157
diff -c -r1.157 pg_backup_archiver.c
*** src/bin/pg_dump/pg_backup_archiver.c 4 May 2008 08:32:21 -0000 1.157
--- src/bin/pg_dump/pg_backup_archiver.c 24 Jul 2008 07:30:19 -0000
***************
*** 56,62 ****
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static teReqs _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
--- 56,62 ----
static void _selectTablespace(ArchiveHandle *AH, const char *tablespace);
static void processEncodingEntry(ArchiveHandle *AH, TocEntry *te);
static void processStdStringsEntry(ArchiveHandle *AH, TocEntry *te);
! static int _tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls);
static void _disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static void _enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt);
static TocEntry *getTocEntryByDumpId(ArchiveHandle *AH, DumpId id);
***************
*** 129,135 ****
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! teReqs reqs;
OutputContext sav;
bool defnDumped;
--- 129,135 ----
{
ArchiveHandle *AH = (ArchiveHandle *) AHX;
TocEntry *te;
! int reqs;
OutputContext sav;
bool defnDumped;
***************
*** 175,193 ****
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then we set the dataOnly flag.
*
! * We could scan for wanted TABLE entries, but that is not the same as
! * dataOnly. At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!ropt->dataOnly)
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0)
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
--- 175,193 ----
* Work out if we have an implied data-only restore. This can happen if
* the dump was data only or if the user has used a toc list to exclude
* all of the schema data. All we do is look for schema entries - if none
! * are found then say we only want DATA type objects.
*
! * We could scan for wanted TABLE entries, but that is not the same.
! * At this stage, it seems unnecessary (6-Mar-2001).
*/
! if (!WANT_DATA(ropt->dumpObjFlags))
{
int impliedDataOnly = 1;
for (te = AH->toc->next; te != AH->toc; te = te->next)
{
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_SCHEMA_BEFORE_DATA(reqs) || WANT_SCHEMA_AFTER_DATA(reqs))
{ /* It's schema, and it's wanted */
impliedDataOnly = 0;
break;
***************
*** 195,201 ****
}
if (impliedDataOnly)
{
! ropt->dataOnly = impliedDataOnly;
ahlog(AH, 1, "implied data-only restore\n");
}
}
--- 195,201 ----
}
if (impliedDataOnly)
{
! ropt->dumpObjFlags = REQ_DATA;
ahlog(AH, 1, "implied data-only restore\n");
}
}
***************
*** 236,242 ****
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
--- 236,242 ----
AH->currentTE = te;
reqs = _tocEntryRequired(te, ropt, false /* needn't drop ACLs */ );
! if (((reqs & REQ_SCHEMA_BEFORE_DATA) != 0) && te->dropStmt)
{
/* We want the schema */
ahlog(AH, 1, "dropping %s %s\n", te->desc, te->tag);
***************
*** 278,284 ****
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!ropt->dataOnly && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
--- 278,284 ----
/* Dump any relevant dump warnings to stderr */
if (!ropt->suppressDumpWarnings && strcmp(te->desc, "WARNING") == 0)
{
! if (!WANT_DATA(ropt->dumpObjFlags) && te->defn != NULL && strlen(te->defn) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->defn);
else if (te->copyStmt != NULL && strlen(te->copyStmt) != 0)
write_msg(modulename, "warning from original dump file: %s\n", te->copyStmt);
***************
*** 286,292 ****
defnDumped = false;
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
--- 286,293 ----
defnDumped = false;
! if ((WANT_SCHEMA_BEFORE_DATA(reqs) && WANT_SCHEMA_BEFORE_DATA(ropt->dumpObjFlags)) ||
! (WANT_SCHEMA_AFTER_DATA(reqs) && WANT_SCHEMA_AFTER_DATA(ropt->dumpObjFlags))) /* We want the schema */
{
ahlog(AH, 1, "creating %s %s\n", te->desc, te->tag);
***************
*** 331,337 ****
/*
* If we have a data component, then process it
*/
! if ((reqs & REQ_DATA) != 0)
{
/*
* hadDumper will be set if there is genuine data component for
--- 332,338 ----
/*
* If we have a data component, then process it
*/
! if (WANT_DATA(reqs))
{
/*
* hadDumper will be set if there is genuine data component for
***************
*** 343,349 ****
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && (reqs & REQ_DATA) != 0)
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
--- 344,350 ----
/*
* If we can output the data, then restore it.
*/
! if (AH->PrintTocDataPtr !=NULL && WANT_DATA(reqs))
{
#ifndef HAVE_LIBZ
if (AH->compression != 0)
***************
*** 415,421 ****
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if ((reqs & REQ_SCHEMA) != 0) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
--- 416,422 ----
/* Work out what, if anything, we want from this entry */
reqs = _tocEntryRequired(te, ropt, true);
! if (WANT_SCHEMA_BEFORE_DATA(reqs)) /* We want the schema */
{
ahlog(AH, 1, "setting owner and privileges for %s %s\n",
te->desc, te->tag);
***************
*** 473,479 ****
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
--- 474,480 ----
_disableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "disabling triggers for %s\n", te->tag);
***************
*** 499,505 ****
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!ropt->dataOnly || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
--- 500,506 ----
_enableTriggersIfNecessary(ArchiveHandle *AH, TocEntry *te, RestoreOptions *ropt)
{
/* This hack is only needed in a data-only restore */
! if (!WANT_DATA(ropt->dumpObjFlags) || !ropt->disable_triggers)
return;
ahlog(AH, 1, "enabling triggers for %s\n", te->tag);
***************
*** 1321,1327 ****
return NULL;
}
! teReqs
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
--- 1322,1328 ----
return NULL;
}
! int
TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt)
{
TocEntry *te = getTocEntryByDumpId(AH, id);
***************
*** 2026,2035 ****
te->defn);
}
! static teReqs
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! teReqs res = REQ_ALL;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
--- 2027,2036 ----
te->defn);
}
! static int
_tocEntryRequired(TocEntry *te, RestoreOptions *ropt, bool include_acls)
{
! int res = ropt->dumpObjFlags;
/* ENCODING and STDSTRINGS items are dumped specially, so always reject */
if (strcmp(te->desc, "ENCODING") == 0 ||
***************
*** 2109,2125 ****
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
- /* Mask it if we only want schema */
- if (ropt->schemaOnly)
- res = res & REQ_SCHEMA;
-
- /* Mask it we only want data */
- if (ropt->dataOnly)
- res = res & REQ_DATA;
-
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~REQ_SCHEMA;
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
--- 2110,2118 ----
if ((strcmp(te->desc, "<Init>") == 0) && (strcmp(te->tag, "Max OID") == 0))
return 0;
/* Mask it if we don't have a schema contribution */
if (!te->defn || strlen(te->defn) == 0)
! res = res & ~(REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
/* Finally, if there's a per-ID filter, limit based on that as well */
if (ropt->idWanted && !ropt->idWanted[te->dumpId - 1])
Index: src/bin/pg_dump/pg_backup_archiver.h
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_backup_archiver.h,v
retrieving revision 1.76
diff -c -r1.76 pg_backup_archiver.h
*** src/bin/pg_dump/pg_backup_archiver.h 7 Nov 2007 12:24:24 -0000 1.76
--- src/bin/pg_dump/pg_backup_archiver.h 24 Jul 2008 07:30:19 -0000
***************
*** 158,169 ****
STAGE_FINALIZING
} ArchiverStage;
! typedef enum
! {
! REQ_SCHEMA = 1,
! REQ_DATA = 2,
! REQ_ALL = REQ_SCHEMA + REQ_DATA
! } teReqs;
typedef struct _archiveHandle
{
--- 158,173 ----
STAGE_FINALIZING
} ArchiverStage;
! #define REQ_SCHEMA_BEFORE_DATA (1 << 0)
! #define REQ_DATA (1 << 1)
! #define REQ_SCHEMA_AFTER_DATA (1 << 2)
! #define REQ_ALL (REQ_SCHEMA_BEFORE_DATA + REQ_DATA + REQ_SCHEMA_AFTER_DATA)
!
! #define WANT_SCHEMA_BEFORE_DATA(req) ((req & REQ_SCHEMA_BEFORE_DATA) == REQ_SCHEMA_BEFORE_DATA)
! #define WANT_DATA(req) ((req & REQ_DATA) == REQ_DATA)
! #define WANT_SCHEMA_AFTER_DATA(req) ((req & REQ_SCHEMA_AFTER_DATA) == REQ_SCHEMA_AFTER_DATA)
! #define WANT_ALL(req) ((req & REQ_ALL) == REQ_ALL)
!
typedef struct _archiveHandle
{
***************
*** 317,323 ****
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern teReqs TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
--- 321,327 ----
extern void ReadToc(ArchiveHandle *AH);
extern void WriteDataChunks(ArchiveHandle *AH);
! extern int TocIDRequired(ArchiveHandle *AH, DumpId id, RestoreOptions *ropt);
extern bool checkSeek(FILE *fp);
#define appendStringLiteralAHX(buf,str,AH) \
Index: src/bin/pg_dump/pg_dump.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_dump.c,v
retrieving revision 1.497
diff -c -r1.497 pg_dump.c
*** src/bin/pg_dump/pg_dump.c 20 Jul 2008 18:43:30 -0000 1.497
--- src/bin/pg_dump/pg_dump.c 24 Jul 2008 07:35:28 -0000
***************
*** 73,78 ****
--- 73,82 ----
bool aclsSkip;
const char *lockWaitTimeout;
+ /* groups of objects: default is we dump all groups */
+
+ int dumpObjFlags;
+
/* subquery used to convert user ID (eg, datdba) to user name */
static const char *username_subquery;
***************
*** 227,232 ****
--- 231,238 ----
static int disable_triggers = 0;
static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ static int schemaBeforeData;
+ static int schemaAfterData;
static struct option long_options[] = {
{"data-only", no_argument, NULL, 'a'},
***************
*** 267,272 ****
--- 273,280 ----
{"disable-triggers", no_argument, &disable_triggers, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-before-data", no_argument, &schemaBeforeData, 1},
+ {"schema-after-data", no_argument, &schemaAfterData, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 420,425 ****
--- 428,437 ----
disable_triggers = 1;
else if (strcmp(optarg, "no-tablespaces") == 0)
outputNoTablespaces = 1;
+ else if (strcmp(optarg, "schema-before-data") == 0)
+ schemaBeforeData = 1;
+ else if (strcmp(optarg, "schema-after-data") == 0)
+ schemaAfterData = 1;
else if (strcmp(optarg, "use-set-session-authorization") == 0)
use_setsessauth = 1;
else
***************
*** 464,474 ****
if (optind < argc)
dbname = argv[optind];
! if (dataOnly && schemaOnly)
{
! write_msg(NULL, "options -s/--schema-only and -a/--data-only cannot be used together\n");
exit(1);
}
if (dataOnly && outputClean)
{
--- 476,521 ----
if (optind < argc)
dbname = argv[optind];
! /*
! * Look for conflicting options relating to object groupings
! */
! if (schemaOnly && dataOnly)
! {
! write_msg(NULL, "options %s and %s cannot be used together\n",
! "-s/--schema-only", "-a/--data-only");
! exit(1);
! }
! else if ((schemaOnly || dataOnly) &&
! (schemaBeforeData || schemaAfterData))
{
! write_msg(NULL, "options %s and %s cannot be used together\n",
! schemaOnly ? "-s/--schema-only" : "-a/--data-only",
! schemaBeforeData ? "--schema-before-data" : "--schema-after-data");
exit(1);
}
+ else if (schemaBeforeData && schemaAfterData)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "--schema-before-data", "--schema-after-data");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (schemaBeforeData)
+ dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
+
+ if (schemaAfterData)
+ dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
if (dataOnly && outputClean)
{
***************
*** 646,652 ****
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && !schemaOnly)
outputBlobs = true;
/*
--- 693,699 ----
* Dumping blobs is now default unless we saw an inclusion switch or -s
* ... but even if we did see one of these, -b turns it back on.
*/
! if (include_everything && WANT_DATA(dumpObjFlags))
outputBlobs = true;
/*
***************
*** 658,664 ****
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (!schemaOnly)
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
--- 705,711 ----
if (g_fout->remoteVersion < 80400)
guessConstraintInheritance(tblinfo, numTables);
! if (WANT_DATA(dumpObjFlags))
getTableData(tblinfo, numTables, oids);
if (outputBlobs && hasBlobs(g_fout))
***************
*** 712,718 ****
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && !dataOnly)
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
--- 759,765 ----
dumpStdStrings(g_fout);
/* The database item is always next, unless we don't want it at all */
! if (include_everything && WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
dumpDatabase(g_fout);
/* Now the rearrangeable objects. */
***************
*** 734,740 ****
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dataOnly = dataOnly;
if (compressLevel == -1)
ropt->compression = 0;
--- 781,787 ----
ropt->noTablespace = outputNoTablespaces;
ropt->disable_triggers = disable_triggers;
ropt->use_setsessauth = use_setsessauth;
! ropt->dumpObjFlags = dumpObjFlags;
if (compressLevel == -1)
ropt->compression = 0;
***************
*** 792,797 ****
--- 839,846 ----
printf(_(" --disable-dollar-quoting disable dollar quoting, use SQL standard quoting\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
+ printf(_(" --schema-before-data dump only the part of schema before table data\n"));
+ printf(_(" --schema-after-data dump only the part of schema after table data\n"));
printf(_(" --use-set-session-authorization\n"
" use SESSION AUTHORIZATION commands instead of\n"
" ALTER OWNER commands to set ownership\n"));
***************
*** 5165,5171 ****
int ncomments;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with catalogId, using table */
--- 5214,5220 ----
int ncomments;
/* Comments are SCHEMA not data */
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* Search for comments associated with catalogId, using table */
***************
*** 5216,5222 ****
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (dataOnly)
return;
/* Search for comments associated with relation, using table */
--- 5265,5271 ----
PQExpBuffer target;
/* Comments are SCHEMA not data */
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* Search for comments associated with relation, using table */
***************
*** 5568,5574 ****
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || dataOnly)
return;
/* don't dump dummy namespace from pre-7.3 source */
--- 5617,5623 ----
char *qnspname;
/* Skip if not to be dumped */
! if (!nspinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* don't dump dummy namespace from pre-7.3 source */
***************
*** 5617,5623 ****
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || dataOnly)
return;
/* Dump out in proper style */
--- 5666,5672 ----
dumpType(Archive *fout, TypeInfo *tinfo)
{
/* Skip if not to be dumped */
! if (!tinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* Dump out in proper style */
***************
*** 5646,5651 ****
--- 5695,5704 ----
i;
char *label;
+ /* Skip if not to be dumped */
+ if (!tinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
+ return;
+
/* Set proper schema search path so regproc references list correctly */
selectSourceSchema(tinfo->dobj.namespace->dobj.name);
***************
*** 6262,6268 ****
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 6315,6321 ----
PQExpBuffer q;
/* Skip if not to be dumped */
! if (!stinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 6309,6315 ****
if (!include_everything)
return false;
/* And they're schema not data */
! if (dataOnly)
return false;
return true;
}
--- 6362,6368 ----
if (!include_everything)
return false;
/* And they're schema not data */
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return false;
return true;
}
***************
*** 6330,6336 ****
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (dataOnly)
return;
/*
--- 6383,6389 ----
FuncInfo *funcInfo;
FuncInfo *validatorInfo = NULL;
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/*
***************
*** 6590,6596 ****
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 6643,6649 ----
int i;
/* Skip if not to be dumped */
! if (!finfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 6985,6991 ****
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (dataOnly)
return;
if (OidIsValid(cast->castfunc))
--- 7038,7044 ----
TypeInfo *sourceInfo;
TypeInfo *targetInfo;
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
if (OidIsValid(cast->castfunc))
***************
*** 7135,7141 ****
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || dataOnly)
return;
/*
--- 7188,7194 ----
char *oprcanhash;
/* Skip if not to be dumped */
! if (!oprinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/*
***************
*** 7519,7525 ****
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || dataOnly)
return;
/*
--- 7572,7578 ----
int i;
/* Skip if not to be dumped */
! if (!opcinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/*
***************
*** 7827,7833 ****
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || dataOnly)
return;
/*
--- 7880,7886 ----
int i;
/* Skip if not to be dumped */
! if (!opfinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/*
***************
*** 8096,8102 ****
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8149,8155 ----
bool condefault;
/* Skip if not to be dumped */
! if (!convinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8250,8256 ****
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || dataOnly)
return;
query = createPQExpBuffer();
--- 8303,8309 ----
bool convertok;
/* Skip if not to be dumped */
! if (!agginfo->aggfn.dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 8453,8459 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8506,8512 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!prsinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8522,8528 ****
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8575,8581 ----
char *tmplname;
/* Skip if not to be dumped */
! if (!dictinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8607,8613 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8660,8666 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!tmplinfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8673,8679 ****
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 8726,8732 ----
int i_dictname;
/* Skip if not to be dumped */
! if (!cfginfo->dobj.dump || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 8809,8815 ****
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (dataOnly || aclsSkip)
return;
sql = createPQExpBuffer();
--- 8862,8868 ----
PQExpBuffer sql;
/* Do nothing if ACL dump is not enabled */
! if (!WANT_SCHEMA_BEFORE_DATA(dumpObjFlags) || aclsSkip)
return;
sql = createPQExpBuffer();
***************
*** 8846,8852 ****
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (!dataOnly)
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
--- 8899,8905 ----
{
if (tbinfo->relkind == RELKIND_SEQUENCE)
dumpSequence(fout, tbinfo);
! else if (WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
dumpTableSchema(fout, tbinfo);
/* Handle the ACL here */
***************
*** 9153,9159 ****
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || dataOnly)
return;
/* Don't print inherited defaults, either */
--- 9206,9212 ----
PQExpBuffer delq;
/* Only print it if "separate" mode is selected */
! if (!tbinfo->dobj.dump || !adinfo->separate || !WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
return;
/* Don't print inherited defaults, either */
***************
*** 9238,9244 ****
PQExpBuffer q;
PQExpBuffer delq;
! if (dataOnly)
return;
q = createPQExpBuffer();
--- 9291,9297 ----
PQExpBuffer q;
PQExpBuffer delq;
! if (!WANT_SCHEMA_AFTER_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9307,9313 ****
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || dataOnly)
return;
q = createPQExpBuffer();
--- 9360,9366 ----
PQExpBuffer delq;
/* Skip if not to be dumped */
! if (!coninfo->dobj.dump || !WANT_SCHEMA_AFTER_DATA(dumpObjFlags))
return;
q = createPQExpBuffer();
***************
*** 9700,9706 ****
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (!dataOnly)
{
resetPQExpBuffer(delqry);
--- 9753,9759 ----
*
* Add a 'SETVAL(seq, last_val, iscalled)' as part of a "data" dump.
*/
! if (WANT_SCHEMA_BEFORE_DATA(dumpObjFlags))
{
resetPQExpBuffer(delqry);
***************
*** 9803,9809 ****
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (!schemaOnly)
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
--- 9856,9862 ----
tbinfo->dobj.catId, 0, tbinfo->dobj.dumpId);
}
! if (WANT_DATA(dumpObjFlags))
{
resetPQExpBuffer(query);
appendPQExpBuffer(query, "SELECT pg_catalog.setval(");
***************
*** 9836,9842 ****
const char *p;
int findx;
! if (dataOnly)
return;
query = createPQExpBuffer();
--- 9889,9895 ----
const char *p;
int findx;
! if (!WANT_SCHEMA_AFTER_DATA(dumpObjFlags))
return;
query = createPQExpBuffer();
***************
*** 10044,10050 ****
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || dataOnly)
return;
/*
--- 10097,10103 ----
PGresult *res;
/* Skip if not to be dumped */
! if (!rinfo->dobj.dump || !WANT_SCHEMA_AFTER_DATA(dumpObjFlags))
return;
/*
Index: src/bin/pg_dump/pg_restore.c
===================================================================
RCS file: /home/sriggs/pg/REPOSITORY/pgsql/src/bin/pg_dump/pg_restore.c,v
retrieving revision 1.88
diff -c -r1.88 pg_restore.c
*** src/bin/pg_dump/pg_restore.c 13 Apr 2008 03:49:22 -0000 1.88
--- src/bin/pg_dump/pg_restore.c 24 Jul 2008 07:30:19 -0000
***************
*** 78,83 ****
--- 78,90 ----
static int no_data_for_failed_tables = 0;
static int outputNoTablespaces = 0;
static int use_setsessauth = 0;
+ bool dataOnly = false;
+ bool schemaOnly = false;
+
+ static int schemaBeforeData;
+ static int schemaAfterData;
+
+ int dumpObjFlags;
struct option cmdopts[] = {
{"clean", 0, NULL, 'c'},
***************
*** 114,119 ****
--- 121,128 ----
{"disable-triggers", no_argument, &disable_triggers, 1},
{"no-data-for-failed-tables", no_argument, &no_data_for_failed_tables, 1},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
+ {"schema-before-data", no_argument, &schemaBeforeData, 1},
+ {"schema-after-data", no_argument, &schemaAfterData, 1},
{"use-set-session-authorization", no_argument, &use_setsessauth, 1},
{NULL, 0, NULL, 0}
***************
*** 145,151 ****
switch (c)
{
case 'a': /* Dump data only */
! opts->dataOnly = 1;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
--- 154,160 ----
switch (c)
{
case 'a': /* Dump data only */
! dataOnly = true;
break;
case 'c': /* clean (i.e., drop) schema prior to create */
opts->dropSchema = 1;
***************
*** 213,219 ****
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! opts->schemaOnly = 1;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
--- 222,228 ----
opts->triggerNames = strdup(optarg);
break;
case 's': /* dump schema only */
! schemaOnly = true;
break;
case 'S': /* Superuser username */
if (strlen(optarg) != 0)
***************
*** 249,254 ****
--- 258,267 ----
no_data_for_failed_tables = 1;
else if (strcmp(optarg, "no-tablespaces") == 0)
outputNoTablespaces = 1;
+ else if (strcmp(optarg, "schema-before-data") == 0)
+ schemaBeforeData = 1;
+ else if (strcmp(optarg, "schema-after-data") == 0)
+ schemaAfterData = 1;
else if (strcmp(optarg, "use-set-session-authorization") == 0)
use_setsessauth = 1;
else
***************
*** 295,300 ****
--- 308,354 ----
opts->useDB = 1;
}
+ /*
+ * Look for conflicting options relating to object groupings
+ */
+ if (schemaOnly && dataOnly)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "-s/--schema-only", "-a/--data-only");
+ exit(1);
+ }
+ else if ((schemaOnly || dataOnly) &&
+ (schemaBeforeData || schemaAfterData))
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ schemaOnly ? "-s/--schema-only" : "-a/--data-only",
+ schemaBeforeData ? "--schema-before-data" : "--schema-after-data");
+ exit(1);
+ }
+ else if (schemaBeforeData && schemaAfterData)
+ {
+ write_msg(NULL, "options %s and %s cannot be used together\n",
+ "--schema-before-data", "--schema-after-data");
+ exit(1);
+ }
+
+ /*
+ * Decide which of the object groups we will dump
+ */
+ dumpObjFlags = REQ_ALL;
+
+ if (dataOnly)
+ dumpObjFlags = REQ_DATA;
+
+ if (schemaBeforeData)
+ dumpObjFlags = REQ_SCHEMA_BEFORE_DATA;
+
+ if (schemaAfterData)
+ dumpObjFlags = REQ_SCHEMA_AFTER_DATA;
+
+ if (schemaOnly)
+ dumpObjFlags = (REQ_SCHEMA_BEFORE_DATA | REQ_SCHEMA_AFTER_DATA);
+
opts->disable_triggers = disable_triggers;
opts->noDataForFailedTables = no_data_for_failed_tables;
opts->noTablespace = outputNoTablespaces;
***************
*** 405,410 ****
--- 459,466 ----
" do not restore data of tables that could not be\n"
" created\n"));
printf(_(" --no-tablespaces do not dump tablespace assignments\n"));
+ printf(_(" --schema-before-data dump only the part of schema before table data\n"));
+ printf(_(" --schema-after-data dump only the part of schema after table data\n"));
printf(_(" --use-set-session-authorization\n"
" use SESSION AUTHORIZATION commands instead of\n"
" OWNER TO commands\n"));
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-25 23:16 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 02:39 ` Re: pg_dump additional options for performance Greg Sabino Mullane <greg@turnstep.com>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 2 replies; 54+ messages in thread
From: Tom Lane @ 2008-07-25 23:16 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-patches
Simon Riggs <simon@2ndquadrant.com> writes:
> [ pg_dump_beforeafter.v6.patch ]
I looked over this patch a bit. I have a proposal for a slightly
different way of defining the new switches:
* --schema-before-data, --data-only, and --schema-after-data can be
specified in any combination to obtain any subset of the full dump.
If none are specified (which would in itself be a useless combination)
then the default is to dump all three sections, just as if all three
were specified.
* --schema-only is defined as equivalent to specifying both
--schema-before-data and --schema-after-data.
The patch as submitted enforces what seem largely arbitrary restrictions
on combining these switches. It made some sense before to treat
specifying both --schema-only and --data-only as an error, but it's not
clear to me why you shouldn't be able to write both --schema-before-data
and --schema-after-data, especially when there's a switch right beside
them that appears to be equivalent to that combination. So let's just
allow all the combinations.
The attached updated patch implements and documents this behavior,
and gets rid of the special linkage between --disable-triggers and
--data-only as previously discussed.
Unfortunately there's still a lot of work to do, and I don't feel like
doing it so I'm bouncing this patch back for further work.
The key problem is that pg_restore is broken: it emits nearly the same
output for --schema-before-data and --schema-after-data, because it
doesn't have any way to distinguish which objects in a full dump file
belong where. This is because the filtering logic was put in the wrong
place, namely in the ArchiveEntry creation routines in pg_dump.c, when
where it really needs to happen is while scanning the TocEntry list in
RestoreArchive(). (Note: it is perhaps worth keeping the pg_dump.c
filters so as to avoid doing extra server queries for objects that we
aren't going to dump anyway, but the core logic has to be in
RestoreArchive.)
Another issue is that the rules for deciding which objects are "before
data" and which are "after data" are wrong. In particular ACLs are after
data not before data, which is relatively easy to fix. Not so easy to fix
is that COMMENTs might be either before or after data depending on what
kind of object they are attached to.
(BTW, what about BLOB COMMENTS? They definitely can't be "before data".
ISTM you could make a case for them being "after data", if you think that
comments are always schema. But there is also a case for considering
them as data, because the objects they are attached to are data. I kind
of like the latter approach because it would create an invariant that
comments appear in the same dump section as the object commented on.
Thoughts?)
Implementing the filtering by examining the type of a TocEntry in
RestoreArchive is a bit of a PITA, but it's probably possible. The
main bad thing about that is the need to make an explicit list of every
type of TocEntry that exists now or ever has been emitted by any past
version of pg_dump. The design concept was that the type tags are
mainly documentation, and while we've had to bend that in places (mostly
for backward-compatibility reasons) this would be the first place we'd
have to throw it overboard completely.
And there's yet another issue here, which is that it's not entirely clear
that the type of an object uniquely determines whether it's before or
after data. This might be an emergent property of the object sorting
rules, but there is certainly not anything positively guaranteeing that
the dependency-driven topological sort will produce such a result, and
especially not that that will always be true in the future. So the
approach seems a bit fragile.
We could perhaps get rid of that problem, as well as the need to implement
object-type-determination logic, if we were to make RestoreArchive define
the groupings according to position in the TocEntry list: everything
before the first TABLE DATA or BLOB (and BLOB COMMENT?) entry is "before"
data, everything after the last one is "after" data, everything in between
is data. Then we only need to identify object types that are considered
"data", which we already have a rule for (whether hadDumper is true).
This is pretty attractive until you stop to consider the possibility
that there aren't any data entries in an archive (ie, it was made with
--schema-only): then there's no way to identify the boundary points.
We could solve that problem by inserting a "dummy data" TOC entry where
the data would have appeared, but this will only work in new archive
files. With an implementation like this, pg_restore with
--schema-before-data or --schema-after-data won't behave very nicely on a
pre-8.4 --schema-only archive file. (Presumably it would act as though
all the objects were "before" data.) Is that a small enough corner case
to live with in order to gain implementation simplicity and robustness?
BTW, another incomplete item is that pg_dumpall should probably be taught
to accept and pass down --schema-before-data and --schema-after-data
switches.
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-26 02:39 ` Greg Sabino Mullane <greg@turnstep.com>
1 sibling, 0 replies; 54+ messages in thread
From: Greg Sabino Mullane @ 2008-07-26 02:39 UTC (permalink / raw)
To: pgsql-patches
-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160
Tom Lane wrote:
> * --schema-before-data, --data-only, and --schema-after-data can be
I thought you were arguing for some better names at one point? Those seem
very confusing to me, especially "--schema-after-data". I know it means
"the parts of the schema that come after the data" but it could
also be read as other ways, including "put the schema after the data" - which
makes no sense, but the name is not exactly intuitive either. "Pre" and "Post"
at least are slightly better, IMO. How about --pre-data-schema
and --post-data-schema? Or --pre-data-section and --post-data-section?
Or (my favorites) --pre-data-commands and --post-data-commands? As the
existing docs say, "commands" are what we are generating.
> them as data, because the objects they are attached to are data. I kind
> of like the latter approach because it would create an invariant that
> comments appear in the same dump section as the object commented on.
> Thoughts?)
+1 on putting them next to the object commented on.
> And there's yet another issue here, which is that it's not entirely clear
> that the type of an object uniquely determines whether it's before or
> after data.
Wouldn't that be a problem with current dumps as well then?
> We could solve that problem by inserting a "dummy data" TOC entry where
> the data would have appeared, but this will only work in new archive
> files. With an implementation like this, pg_restore with
> --schema-before-data or --schema-after-data won't behave very nicely on a
> pre-8.4 --schema-only archive file. (Presumably it would act as though
> all the objects were "before" data.) Is that a small enough corner case
> to live with in order to gain implementation simplicity and robustness?
I'm not comfortable with corner cases for pg_restore backwards compatibility.
What exactly would happen (worse case) in that scenario?
- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200807252235
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----
iEYEAREDAAYFAkiKjgMACgkQvJuQZxSWSsiRMACg7c/VDo9hTTjukkFFvLYI31mL
BqkAn3FfepllvVnIwX+efA5cLPlVbDd0
=V/Sv
-----END PGP SIGNATURE-----
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-26 09:05 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-26 15:11 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 2 replies; 54+ messages in thread
From: Simon Riggs @ 2008-07-26 09:05 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-patches
On Fri, 2008-07-25 at 19:16 -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > [ pg_dump_beforeafter.v6.patch ]
> Unfortunately there's still a lot of work to do, and I don't feel like
> doing it so I'm bouncing this patch back for further work.
Fair enough. Thanks for the review.
> The key problem is that pg_restore is broken: it emits nearly the same
> output for --schema-before-data and --schema-after-data, because it
> doesn't have any way to distinguish which objects in a full dump file
> belong where. This is because the filtering logic was put in the wrong
> place, namely in the ArchiveEntry creation routines in pg_dump.c, when
> where it really needs to happen is while scanning the TocEntry list in
> RestoreArchive(). (Note: it is perhaps worth keeping the pg_dump.c
> filters so as to avoid doing extra server queries for objects that we
> aren't going to dump anyway, but the core logic has to be in
> RestoreArchive.)
My feeling is that this would take the patch off-track.
The key capability here is being able to split the dump into multiple
pieces. The equivalent capability on restore is *not* required, because
once the dump has been split the restore never needs to be. It might
seem that the patch should be symmetrical with respect to pg_dump and
pg_restore, but I see no use case for the pg_restore case.
The title of this email confirms that as original intention.
> I looked over this patch a bit. I have a proposal for a slightly
> different way of defining the new switches:
>
> * --schema-before-data, --data-only, and --schema-after-data can be
> specified in any combination to obtain any subset of the full dump.
> If none are specified (which would in itself be a useless combination)
> then the default is to dump all three sections, just as if all three
> were specified.
>
> * --schema-only is defined as equivalent to specifying both
> --schema-before-data and --schema-after-data.
>
> The patch as submitted enforces what seem largely arbitrary restrictions
> on combining these switches. It made some sense before to treat
> specifying both --schema-only and --data-only as an error, but it's not
> clear to me why you shouldn't be able to write both --schema-before-data
> and --schema-after-data, especially when there's a switch right beside
> them that appears to be equivalent to that combination. So let's just
> allow all the combinations.
I had it both ways at various points in development. I'm happy with what
you propose.
> The attached updated patch implements and documents this behavior,
> and gets rid of the special linkage between --disable-triggers and
> --data-only as previously discussed.
OK
> Another issue is that the rules for deciding which objects are "before
> data" and which are "after data" are wrong. In particular ACLs are after
> data not before data, which is relatively easy to fix.
OK
> Not so easy to fix
> is that COMMENTs might be either before or after data depending on what
> kind of object they are attached to.
Is there anything to fix? Comments are added by calls to dumpComment,
which are always made in conjunction with the dump of an object. So if
you dump the object you dump the comment. As long as objects are
correctly split out then comments will be also.
> (BTW, what about BLOB COMMENTS? They definitely can't be "before data".
> ISTM you could make a case for them being "after data", if you think that
> comments are always schema. But there is also a case for considering
> them as data, because the objects they are attached to are data. I kind
> of like the latter approach because it would create an invariant that
> comments appear in the same dump section as the object commented on.
> Thoughts?)
Yes, data. I'll look at this.
> Implementing the filtering by examining the type of a TocEntry in
> RestoreArchive is a bit of a PITA, but it's probably possible. The
> main bad thing about that is the need to make an explicit list of every
> type of TocEntry that exists now or ever has been emitted by any past
> version of pg_dump. The design concept was that the type tags are
> mainly documentation, and while we've had to bend that in places (mostly
> for backward-compatibility reasons) this would be the first place we'd
> have to throw it overboard completely.
>
> And there's yet another issue here, which is that it's not entirely clear
> that the type of an object uniquely determines whether it's before or
> after data. This might be an emergent property of the object sorting
> rules, but there is certainly not anything positively guaranteeing that
> the dependency-driven topological sort will produce such a result, and
> especially not that that will always be true in the future. So the
> approach seems a bit fragile.
Don't understand that. Objects are sorted in well-defined order,
specified in pg_dump_sort.c. Essentially we are saying that (according
to current numbering)
--schema-before-data priority 1-8
--data-only priority 9-11
--schema-after-data priority 12+
So the sort is explicitly defined, not implicit. I can add comments to
ensure that people changing the priority of objects across those
boundaries would be causing problems.
> We could perhaps get rid of that problem, as well as the need to implement
> object-type-determination logic, if we were to make RestoreArchive define
> the groupings according to position in the TocEntry list: everything
> before the first TABLE DATA or BLOB (and BLOB COMMENT?) entry is "before"
> data, everything after the last one is "after" data, everything in between
> is data. Then we only need to identify object types that are considered
> "data", which we already have a rule for (whether hadDumper is true).
> This is pretty attractive until you stop to consider the possibility
> that there aren't any data entries in an archive (ie, it was made with
> --schema-only): then there's no way to identify the boundary points.
>
> We could solve that problem by inserting a "dummy data" TOC entry where
> the data would have appeared, but this will only work in new archive
> files. With an implementation like this, pg_restore with
> --schema-before-data or --schema-after-data won't behave very nicely on a
> pre-8.4 --schema-only archive file. (Presumably it would act as though
> all the objects were "before" data.) Is that a small enough corner case
> to live with in order to gain implementation simplicity and robustness?
All of the above makes me certain I want to remove these options from
pg_restore.
> BTW, another incomplete item is that pg_dumpall should probably be taught
> to accept and pass down --schema-before-data and --schema-after-data
> switches.
OK
I'm conscious that the major work proposed will take weeks to complete
and we don't know what other problems it will cause (but I'm pretty
certain it will cause some). With regard to the use case, I see little
or no benefit from either of us doing that and regret I won't be able to
complete that.
Can we prune down to the base use case to avoid this overhead? i.e. have
these options on pg_dump only?
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-26 15:11 ` Stephen Frost <sfrost@snowman.net>
2008-07-26 16:24 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-26 15:11 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-patches
* Simon Riggs (simon@2ndquadrant.com) wrote:
> The key capability here is being able to split the dump into multiple
> pieces. The equivalent capability on restore is *not* required, because
> once the dump has been split the restore never needs to be. It might
> seem that the patch should be symmetrical with respect to pg_dump and
> pg_restore, but I see no use case for the pg_restore case.
I'm inclined to agree with this. It might have been nice to provide a
way to split out already-created dumps, but I suspect that people who
need that probably have already figured out a way to do it (I know I
have..). We should probably ensure that pg_restore doesn't *break* when
fed a partial dump.
> > The patch as submitted enforces what seem largely arbitrary restrictions
> > on combining these switches.
>
> I had it both ways at various points in development. I'm happy with what
> you propose.
I agree with removing the restrictions. I don't see much in the way of
use cases, but it reduces code and doesn't cause problems.
> > Another issue is that the rules for deciding which objects are "before
> > data" and which are "after data" are wrong. In particular ACLs are after
> > data not before data, which is relatively easy to fix.
>
> OK
This was partially why I was complaining about having documentation, and
a policy for that matter, which goes into more detail about why X is before
or after the data. I agree that they're after today, but I don't see
any particular reason why they should be one or the other. If we
adopted a policy like I proposed (--schema-post-data is essentially that
which uses the data and is faster done in bulk) then ACLs would be
before, and I tend to feel like it makes more sense that way.
> > Not so easy to fix
> > is that COMMENTs might be either before or after data depending on what
> > kind of object they are attached to.
>
> Is there anything to fix? Comments are added by calls to dumpComment,
> which are always made in conjunction with the dump of an object. So if
> you dump the object you dump the comment. As long as objects are
> correctly split out then comments will be also.
I agree with this, and it follows for BLOB comments- in any case, they
go with the object being dumped at the time of that object getting
dumped. Comments make sense as an extention of the object, not as a
seperate set of objects to be explicitly placed before or after the
data.
> All of the above makes me certain I want to remove these options from
> pg_restore.
I'm in agreement with this.
> > BTW, another incomplete item is that pg_dumpall should probably be taught
> > to accept and pass down --schema-before-data and --schema-after-data
> > switches.
>
> OK
I could go either way on this.
> Can we prune down to the base use case to avoid this overhead? i.e. have
> these options on pg_dump only?
Makes sense to me.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 15:11 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-26 16:24 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 54+ messages in thread
From: Tom Lane @ 2008-07-26 16:24 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
Stephen Frost <sfrost@snowman.net> writes:
>>> Another issue is that the rules for deciding which objects are "before
>>> data" and which are "after data" are wrong. In particular ACLs are after
>>> data not before data, which is relatively easy to fix.
>>
>> OK
> This was partially why I was complaining about having documentation, and
> a policy for that matter, which goes into more detail about why X is before
> or after the data. I agree that they're after today, but I don't see
> any particular reason why they should be one or the other.
If a table's ACL revokes owner insert privilege, and was placed before
the data load steps, those steps would fail. We are relying on the
default table privileges until we are done doing everything we need to
do to the tables (and perhaps other objects, I'm not sure if there are
any other comparable problems).
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-26 16:20 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 16:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
1 sibling, 2 replies; 54+ messages in thread
From: Tom Lane @ 2008-07-26 16:20 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-patches
Simon Riggs <simon@2ndquadrant.com> writes:
> On Fri, 2008-07-25 at 19:16 -0400, Tom Lane wrote:
>> The key problem is that pg_restore is broken:
> The key capability here is being able to split the dump into multiple
> pieces. The equivalent capability on restore is *not* required, because
> once the dump has been split the restore never needs to be.
This argument is nonsense. The typical usage of this capability, IMHO,
will be
pg_dump -Fc >whole.dump
pg_restore --schema-before-data whole.dump >before.sql
pg_restore --data-only whole.dump >data.sql
pg_restore --schema-after-data whole.dump >after.sql
followed by editing the schema pieces and then loading. One reason
is that this gives you a consistent dump, whereas three successive
pg_dump runs could never guarantee any such thing. Another reason
is that you may well not know when you prepare the dump that you
will need split output, because the requirement to edit the dump
is likely to be realized only when you go to load it.
In any case, why did you put the switches into pg_restore.c if you
thought it wasn't useful for pg_restore to handle them?
>> Not so easy to fix
>> is that COMMENTs might be either before or after data depending on what
>> kind of object they are attached to.
> Is there anything to fix?
Well, yeah. If you attach a comment to an after-data object and test
--schema-after-data, you'll notice the comment is lost.
>> And there's yet another issue here, which is that it's not entirely clear
>> that the type of an object uniquely determines whether it's before or
>> after data.
> Don't understand that. Objects are sorted in well-defined order,
> specified in pg_dump_sort.
After which we do a topological sort that enforces dependency ordering.
The question to worry about is whether there can ever be a dependency
from a normally-"before" object to a normally-"after" object, which
would cause the dependency sort to move the latter in front of the
former (in one way or another). I'm not saying that any such case can
occur today, but I don't think it's an impossibility for it to arise in
future. I don't want this relatively minor feature to be putting limits
on what kinds of dependencies the system can have.
> I'm conscious that the major work proposed will take weeks to complete
I don't think that what I am proposing is that complicated; I would
anticipate it requiring somewhere on the order of two dozen lines of
code. I was thinking of doing a preliminary loop through the TocEntry
list to identify the ordinal numbers of the first and last data items,
and then the main loop could compare a counter to those numbers to
decide which of the three sections it was in. Plus you'd need another
ArchiveEntry call someplace to prepare the "dummy data" item if one was
needed.
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-26 16:43 ` Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-26 16:43 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > On Fri, 2008-07-25 at 19:16 -0400, Tom Lane wrote:
> >> The key problem is that pg_restore is broken:
>
> > The key capability here is being able to split the dump into multiple
> > pieces. The equivalent capability on restore is *not* required, because
> > once the dump has been split the restore never needs to be.
>
> This argument is nonsense. The typical usage of this capability, IMHO,
> will be
>
> pg_dump -Fc >whole.dump
> pg_restore --schema-before-data whole.dump >before.sql
> pg_restore --data-only whole.dump >data.sql
> pg_restore --schema-after-data whole.dump >after.sql
>
> followed by editing the schema pieces and then loading.
I dislike, and doubt that I'd use, this approach. At the end of the
day, it ends up processing the same (very large amount of data) multiple
times. We have >60G dump files sometimes, and there's no way I'm going
to dump that into a single file first if I can avoid it. What we end up
doing today is --schema-only followed by vi'ing it and splitting it up
by hand, etc, then doing a seperate --data-only dump.
> One reason
> is that this gives you a consistent dump, whereas three successive
> pg_dump runs could never guarantee any such thing.
While this is technically true, in most cases people have control over
the schema bits and would likely be able to ensure that the schema
doesn't change during the time. At that point it's only the data, which
is still done in a transactional way.
> Another reason is that you may well not know when you prepare the
> dump that you will need split output, because the requirement to edit
> the dump is likely to be realized only when you go to load it.
This is a good point. My gut reaction is that, at least in my usage, it
would be more about "if it's larger than a gig, I might as well split it
out, just in case I need to touch something". Honestly, it's rare that
I don't have to make *some* change. Often that's the point of dumping
it out.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-26 17:43 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 18:03 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-28 00:44 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
0 siblings, 2 replies; 54+ messages in thread
From: Tom Lane @ 2008-07-26 17:43 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
Stephen Frost <sfrost@snowman.net> writes:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> This argument is nonsense. The typical usage of this capability, IMHO,
>> will be
>>
>> pg_dump -Fc >whole.dump
>> pg_restore --schema-before-data whole.dump >before.sql
>> pg_restore --data-only whole.dump >data.sql
>> pg_restore --schema-after-data whole.dump >after.sql
>>
>> followed by editing the schema pieces and then loading.
> I dislike, and doubt that I'd use, this approach. At the end of the
> day, it ends up processing the same (very large amount of data) multiple
> times.
Well, that's easily avoided: just replace the third step by restoring
directly to the target database.
pg_restore --schema-before-data whole.dump >before.sql
edit before.sql
pg_restore --schema-after-data whole.dump >after.sql
edit after.sql
psql -f before.sql target_db
pg_restore --data-only -d target_db whole.dump
psql -f after.sql target_db
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-26 18:03 ` Joshua D. Drake <jd@commandprompt.com>
2008-07-27 09:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
1 sibling, 1 reply; 54+ messages in thread
From: Joshua D. Drake @ 2008-07-26 18:03 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Stephen Frost <sfrost@snowman.net>; Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
On Sat, 2008-07-26 at 13:43 -0400, Tom Lane wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > I dislike, and doubt that I'd use, this approach. At the end of the
> > day, it ends up processing the same (very large amount of data) multiple
> > times.
>
> Well, that's easily avoided: just replace the third step by restoring
> directly to the target database.
>
> pg_restore --schema-before-data whole.dump >before.sql
> edit before.sql
> pg_restore --schema-after-data whole.dump >after.sql
> edit after.sql
> psql -f before.sql target_db
> pg_restore --data-only -d target_db whole.dump
> psql -f after.sql target_db
It seems to me we continue to hack a solution without a clear idea of
the problems involved. There are a number of what I would consider
significant issues with the backup / restore facilities as a whole with
PostgreSQL.
1. We use text based backups, even with custom format. We need a fast
binary representation as well.
2. We have no concurrency which means, anyone with any database over 50G
has unacceptable restore times.
3. We have to continue develop hacks to define custom utilization. Why
am I passing pre-data anything? It should be automatic. For example:
pg_backup (not dump, we aren't dumping. Dumping is usually associated
with some sort of crash or fould human behavoir. We are backing up).
pg_backup -U <user> -D database -F -f mybackup.sqlc
If I were to extract <mybackup.sqlc> I would get:
mybackup.datatypes
mybackup.tables
mybackup.data
mybackup.primary_keys
mybackup.indexes
mybackup.constraints
mybackup.grants
All would be the SQL representation.
Further I could do this:
pg_restore -U <user> -D <database> --data-types -f mybackup.sqlc
Which would restore just the SQL representation of the data types.
Or:
pg_restore -U <user> -D <database> --tables -f mybackup.sqlc
Which would restore *only* the tables. Yes it would error if I didn't
also specify --data-types.
Further we need to have concurrency capability. Once we have restored
datatypes and tables, there is zero reason not to launch connections on
data (and then primary keys and indexes) so:
pg_restore -U <user> -D <database> -C 4 --full -f mybackup.sqlc
Which would launch four connections to the database, and perform a full
restore per mybackup.sqlc.
Oh and pg_dumpall? It should have been removed right around the release
of 7.2, pg_dump -A please.
Anyway, I leave other peeps to flame me into oblivion.
Sincerely,
Joshua D. Drake
--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 18:03 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
@ 2008-07-27 09:37 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-27 16:57 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-28 01:32 ` Re: [PATCHES] pg_dump additional options for performance daveg <daveg@sonic.net>
0 siblings, 2 replies; 54+ messages in thread
From: Simon Riggs @ 2008-07-27 09:37 UTC (permalink / raw)
To: Joshua D. Drake <jd@commandprompt.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Stephen Frost <sfrost@snowman.net>; pgsql-patches
On Sat, 2008-07-26 at 11:03 -0700, Joshua D. Drake wrote:
> 2. We have no concurrency which means, anyone with any database over 50G
> has unacceptable restore times.
Agreed.
Also the core reason for wanting -w
> 3. We have to continue develop hacks to define custom utilization. Why
> am I passing pre-data anything? It should be automatic. For example:
>
> pg_backup (not dump, we aren't dumping. Dumping is usually associated
> with some sort of crash or fould human behavoir. We are backing up).
> pg_backup -U <user> -D database -F -f mybackup.sqlc
>
> If I were to extract <mybackup.sqlc> I would get:
>
> mybackup.datatypes
> mybackup.tables
> mybackup.data
> mybackup.primary_keys
> mybackup.indexes
> mybackup.constraints
> mybackup.grants
Sounds good.
Doesn't help with the main element of dump time: one table at a time to
one output file. We need a way to dump multiple tables concurrently,
ending in multiple files/filesystems.
> Oh and pg_dumpall? It should have been removed right around the release
> of 7.2, pg_dump -A please.
Good idea
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 18:03 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-27 09:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-27 16:57 ` Joshua D. Drake <jd@commandprompt.com>
2008-07-27 23:42 ` Re: pg_dump additional options for performance Andrew Dunstan <andrew@dunslane.net>
1 sibling, 1 reply; 54+ messages in thread
From: Joshua D. Drake @ 2008-07-27 16:57 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Stephen Frost <sfrost@snowman.net>; pgsql-patches
Simon Riggs wrote:
> On Sat, 2008-07-26 at 11:03 -0700, Joshua D. Drake wrote:
>
>> 2. We have no concurrency which means, anyone with any database over 50G
>> has unacceptable restore times.
>
> Agreed.
> Sounds good.
>
> Doesn't help with the main element of dump time: one table at a time to
> one output file. We need a way to dump multiple tables concurrently,
> ending in multiple files/filesystems.
Agreed but that is a problem I understand with a solution I don't. I am
all eyes on a way to fix that. One thought I had and please, be gentle
in response was some sort of async transaction capability. I know that
libpq has the ability to send async queries. Is it possible to do this:
send async(copy table to foo)
send async(copy table to bar)
send async(copy table to baz)
Where all three copies are happening in the background?
Sincerely,
Joshua D. Drake
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 18:03 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-27 09:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-27 16:57 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
@ 2008-07-27 23:42 ` Andrew Dunstan <andrew@dunslane.net>
2008-07-28 00:24 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
0 siblings, 1 reply; 54+ messages in thread
From: Andrew Dunstan @ 2008-07-27 23:42 UTC (permalink / raw)
To: Joshua D. Drake <jd@commandprompt.com>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; Stephen Frost <sfrost@snowman.net>; pgsql-patches
Joshua D. Drake wrote:
>
> Agreed but that is a problem I understand with a solution I don't. I
> am all eyes on a way to fix that. One thought I had and please, be
> gentle in response was some sort of async transaction capability. I
> know that libpq has the ability to send async queries. Is it possible
> to do this:
>
> send async(copy table to foo)
> send async(copy table to bar)
> send async(copy table to baz)
>
> Where all three copies are happening in the background?
>
>
IIRC, libpq doesn't let you have more than one async query active at one
time.
cheers
andrew
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 18:03 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-27 09:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-27 16:57 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-27 23:42 ` Re: pg_dump additional options for performance Andrew Dunstan <andrew@dunslane.net>
@ 2008-07-28 00:24 ` Joshua D. Drake <jd@commandprompt.com>
0 siblings, 0 replies; 54+ messages in thread
From: Joshua D. Drake @ 2008-07-28 00:24 UTC (permalink / raw)
To: Andrew Dunstan <andrew@dunslane.net>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Tom Lane <tgl@sss.pgh.pa.us>; Stephen Frost <sfrost@snowman.net>; pgsql-patches
Andrew Dunstan wrote:
>
>
> Joshua D. Drake wrote:
>>
>> Agreed but that is a problem I understand with a solution I don't. I
>> am all eyes on a way to fix that. One thought I had and please, be
>> gentle in response was some sort of async transaction capability. I
>> know that libpq has the ability to send async queries. Is it possible
>> to do this:
>>
>> send async(copy table to foo)
>> send async(copy table to bar)
>> send async(copy table to baz)
>>
>> Where all three copies are happening in the background?
>>
>>
>
> IIRC, libpq doesn't let you have more than one async query active at one
> time.
Now that I think on it harder, this isn't even a libpq problem (although
its involved), we need the postmaster do be able to do a background
async query. Which is (I am guessing) why libpq can only do one at a time.
Sincerely,
Joshua D. Drake
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: [PATCHES] pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 18:03 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-27 09:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-28 01:32 ` daveg <daveg@sonic.net>
1 sibling, 0 replies; 54+ messages in thread
From: daveg @ 2008-07-28 01:32 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; pgsql-hackers@postgresql.org; +Cc: Joshua D. Drake <jd@commandprompt.com>; Tom Lane <tgl@sss.pgh.pa.us>; Stephen Frost <sfrost@snowman.net>
On Sun, Jul 27, 2008 at 10:37:34AM +0100, Simon Riggs wrote:
>
> On Sat, 2008-07-26 at 11:03 -0700, Joshua D. Drake wrote:
>
> > 2. We have no concurrency which means, anyone with any database over 50G
> > has unacceptable restore times.
>
> Agreed.
>
> Also the core reason for wanting -w
>
> > 3. We have to continue develop hacks to define custom utilization. Why
> > am I passing pre-data anything? It should be automatic. For example:
[adding hackers for discussion]
On Sat, Jul 26, 2008 at 01:56:14PM -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > I want to dump tables separately for performance reasons. There are
> > documented tests showing 100% gains using this method. There is no gain
> > adding this to pg_restore. There is a gain to be had - parallelising
> > index creation, but this patch doesn't provide parallelisation.
>
> Right, but the parallelization is going to happen sometime, and it is
> going to happen in the context of pg_restore. So I think it's pretty
> silly to argue that no one will ever want this feature to work in
> pg_restore.
>
> To extend the example I just gave to Stephen, I think a fairly probable
> scenario is where you only need to tweak some "before" object
> definitions, and then you could do
>
> pg_restore --schema-before-data whole.dump >before.sql
> edit before.sql
> psql -f before.sql target_db
> pg_restore --data-only --schema-after-data -d target_db whole.dump
>
> which (given a parallelizing pg_restore) would do all the time-consuming
> steps in a fully parallelized fashion.
A few thoughts about pg_restore performance:
To take advantage of non-logged copy, the table create and data load should
be in the same transaction.
To take advantage of file and buffer cache, it would be better to create
indexes immediately after loading table data. Many tables will be small
enough to fit in cache on and this will avoid re-reading them for index
builds. This is more advantagious with more indexes on one table. There
may also be some filesytem placement benefits to building the indexes for
a table immediately after loading the data.
Creating constraints immediately after loading data also would benefit from
warm buffer and file caches. Doing this this is complicated by the need
for indexes and data in the referenced tables to exist first.
It seems that a high performance restore will want to procede in a different
order than the current sort order or that proposed by the before/data/after
patch.
- The simplest unit of work for parallelism may be the table and its
"decorations", eg indexes and relational constraints.
- Sort tables by foreign key dependency so that referenced tables are
loaded before referencing tables.
- Do table creation and data load together in one transaction to use
non-logged copy. Index builds, and constraint creation should follow
immediately, either as part of the same transaction, or possibly
parallelized themselves.
Table creation, data load, index builds, and constraint creation could
be packaged up as the unit of work to be done in a subprocess which either
completes or fails as a unit. The worker process would be called with
connection info, a file pointer to the data, and the DDL for the table.
pg_restore would keep a work queue of tables to be restored in FK dependency
order and also do the other schema operations such as functions and types.
-dg
--
David Gould daveg@sonic.net 510 536 1443 510 282 0869
If simplicity worked, the world would be overrun with insects.
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-28 00:44 ` Stephen Frost <sfrost@snowman.net>
2008-07-28 02:40 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
1 sibling, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-28 00:44 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > I dislike, and doubt that I'd use, this approach. At the end of the
> > day, it ends up processing the same (very large amount of data) multiple
> > times.
>
> Well, that's easily avoided: just replace the third step by restoring
> directly to the target database.
>
> pg_restore --schema-before-data whole.dump >before.sql
> edit before.sql
> pg_restore --schema-after-data whole.dump >after.sql
> edit after.sql
> psql -f before.sql target_db
> pg_restore --data-only -d target_db whole.dump
> psql -f after.sql target_db
This would depend on the dump being in the custom format, though I
suppose that ends up being true for any usage of these options. I've
never really been a fan of the custom format, in large part because it
doesn't really buy you all that much and makes changing things more
difficult (by having to extract out what you want to change, and then
omit it from the restore).
I can see some advantage to having the entire dump contained in a single
file and still being able to pull out pieces based on before/after.
Should we get a binary format which is much faster, I could see myself
being more likely to use pg_restore. Same for parallelization or, in my
fantasies, the ability to copy schema, tables, indexes, etc, in 'raw' PG
format between servers. Worse than having to vi an insanely large file,
or split it up to be able to modify the pieces you want, is having to
rebuild indexes, especially GIST ones. That's another topic though.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-28 00:44 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-28 02:40 ` Joshua D. Drake <jd@commandprompt.com>
2008-07-28 02:50 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 54+ messages in thread
From: Joshua D. Drake @ 2008-07-28 02:40 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
Stephen Frost wrote:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> Stephen Frost <sfrost@snowman.net> writes:
>>> I dislike, and doubt that I'd use, this approach. At the end of the
>>> day, it ends up processing the same (very large amount of data) multiple
>>> times.
> This would depend on the dump being in the custom format, though I
> suppose that ends up being true for any usage of these options. I've
> never really been a fan of the custom format, in large part because it
> doesn't really buy you all that much and makes changing things more
> difficult (by having to extract out what you want to change, and then
> omit it from the restore).
Custom format rocks for partial set restores from a whole dump. See the
TOC option :)
Joshua D. Drake
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-28 00:44 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-28 02:40 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
@ 2008-07-28 02:50 ` Stephen Frost <sfrost@snowman.net>
0 siblings, 0 replies; 54+ messages in thread
From: Stephen Frost @ 2008-07-28 02:50 UTC (permalink / raw)
To: Joshua D. Drake <jd@commandprompt.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
* Joshua D. Drake (jd@commandprompt.com) wrote:
> Custom format rocks for partial set restores from a whole dump. See the
> TOC option :)
I imagine it does, but that's very rarely what I need. Most of the time
we're dumping out a schema to load it into a seperate schema (usually on
another host). Sometimes that can be done by simply vi'ing the file to
change the search_path and whatnot, though more often we end up pipe'ing
the whole thing through sed. Since we don't allow regular users to do
much, and you have to 'set role postgres;' to do anything as superuser,
we also often end up adding 'set role postgres;' to the top of the .sql
files.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-26 16:56 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-26 17:56 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-07-26 16:56 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-patches
On Sat, 2008-07-26 at 12:20 -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > On Fri, 2008-07-25 at 19:16 -0400, Tom Lane wrote:
> >> The key problem is that pg_restore is broken:
>
> > The key capability here is being able to split the dump into multiple
> > pieces. The equivalent capability on restore is *not* required, because
> > once the dump has been split the restore never needs to be.
>
> This argument is nonsense.
> The typical usage of this capability, IMHO, will be
Arghh! That's not my stated use case!?#*!
I want to dump tables separately for performance reasons. There are
documented tests showing 100% gains using this method. There is no gain
adding this to pg_restore. There is a gain to be had - parallelising
index creation, but this patch doesn't provide parallelisation.
Anyway, clearly time for me to stop and have a break.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-26 17:56 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-07-27 01:33 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-27 09:31 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-28 00:52 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-08-01 18:02 ` Re: pg_dump additional options for performance chris <cbbrowne@ca.afilias.info>
0 siblings, 4 replies; 54+ messages in thread
From: Tom Lane @ 2008-07-26 17:56 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-patches
Simon Riggs <simon@2ndquadrant.com> writes:
> I want to dump tables separately for performance reasons. There are
> documented tests showing 100% gains using this method. There is no gain
> adding this to pg_restore. There is a gain to be had - parallelising
> index creation, but this patch doesn't provide parallelisation.
Right, but the parallelization is going to happen sometime, and it is
going to happen in the context of pg_restore. So I think it's pretty
silly to argue that no one will ever want this feature to work in
pg_restore.
To extend the example I just gave to Stephen, I think a fairly probable
scenario is where you only need to tweak some "before" object
definitions, and then you could do
pg_restore --schema-before-data whole.dump >before.sql
edit before.sql
psql -f before.sql target_db
pg_restore --data-only --schema-after-data -d target_db whole.dump
which (given a parallelizing pg_restore) would do all the time-consuming
steps in a fully parallelized fashion.
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 17:56 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-27 01:33 ` daveg <daveg@sonic.net>
3 siblings, 0 replies; 54+ messages in thread
From: daveg @ 2008-07-27 01:33 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Simon Riggs <simon@2ndquadrant.com>; Stephen Frost <sfrost@snowman.net>; pgsql-patches
On Sat, Jul 26, 2008 at 01:56:14PM -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > I want to dump tables separately for performance reasons. There are
> > documented tests showing 100% gains using this method. There is no gain
> > adding this to pg_restore. There is a gain to be had - parallelising
> > index creation, but this patch doesn't provide parallelisation.
>
> Right, but the parallelization is going to happen sometime, and it is
> going to happen in the context of pg_restore. So I think it's pretty
> silly to argue that no one will ever want this feature to work in
> pg_restore.
>
> To extend the example I just gave to Stephen, I think a fairly probable
> scenario is where you only need to tweak some "before" object
> definitions, and then you could do
>
> pg_restore --schema-before-data whole.dump >before.sql
> edit before.sql
> psql -f before.sql target_db
> pg_restore --data-only --schema-after-data -d target_db whole.dump
>
> which (given a parallelizing pg_restore) would do all the time-consuming
> steps in a fully parallelized fashion.
A few thoughts about pg_restore performance:
To take advantage of non-logged copy, the create and load should be in
the same transaction.
To take advantage of file and buffer cache, it would be be good to do
indexes immediately after table data. Many tables will be small enough
to fit in cache and this will avoid re-reading them for index builds. This
effect becomes stronger with more indexes on one table. There may also be
some filesytem placement benefit to building the indexes for a table
immediately after loading the data.
The buffer fan file cache advantage also applies to constraint creation,
but this is complicated by the need for indexes and data in the referenced
tables.
It seems that a high performance restore will want to proced in a different
order than the current sort order or that proposed by the before/data/after
patch.
- The simplest unit of work for parallelism may be the table and its
"decorations", eg indexes and relational constraints.
- Sort tables by foreign key dependency so that referenced tables are
loaded before referencing tables.
- Do table creation and data load together in one transaction to use
non-logged copy. Index builds, and constraint creation should follow
immediately, either as part of the same transaction, or possibly
parallelized themselves.
Table creation, data load, index builds, and constraint creation could
be packaged up as the unit of work to be done in a subprocess which either
completes or fails as a unit. The worker process would be called with
connection info, a file pointer to the data, and the DDL for the table.
pg_restore would keep a work queue of tables to be restored in FK dependency
order and also do the other schema operations such as functions and types.
-dg
--
David Gould daveg@sonic.net 510 536 1443 510 282 0869
If simplicity worked, the world would be overrun with insects.
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 17:56 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-27 09:31 ` Simon Riggs <simon@2ndquadrant.com>
3 siblings, 0 replies; 54+ messages in thread
From: Simon Riggs @ 2008-07-27 09:31 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Stephen Frost <sfrost@snowman.net>; pgsql-patches
On Sat, 2008-07-26 at 13:56 -0400, Tom Lane wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> > I want to dump tables separately for performance reasons. There are
> > documented tests showing 100% gains using this method. There is no gain
> > adding this to pg_restore. There is a gain to be had - parallelising
> > index creation, but this patch doesn't provide parallelisation.
>
> Right, but the parallelization is going to happen sometime, and it is
> going to happen in the context of pg_restore.
I honestly think there is less benefit that way than if we consider
things more as a whole:
To do data dump quickly we need to dump different tables to different
disks simultaneously. By its very nature, that cannot end with just a
single file. So the starting point for any restore must be potentially
more than one file.
There are two ways of dumping: either multi-thread pg_dump, or allow
multiple pg_dumps to work together. Second option much less work, same
result. (Either way we also need a way for multiple concurrent sessions
to share a snapshot.)
When restoring, we can then just use multiple pg_restore sessions to
restore the individual data files. Or again we can write a
multi-threaded pg_restore to do the same thing - why would I bother
doing that when I already can? It gains us nothing.
Parallelising the index creation seems best done using concurrent psql.
We've agreed some mods to psql to put multi-sessions in there. If we do
that right, then we can make pg_restore generate a psql script with
multi-session commands scattered appropriately throughout.
Parallel pg_restore is a lot of work for a narrow use case. Concurrent
psql provides a much wider set of use cases.
So fully parallelising dump/restore can be achieved by
* splitting dump into pieces (this patch)
* allowing sessions to share a common snapshot
* concurrent psql
* changes to pg_restore/psql/pg_dump to allow commands to be inserted
which will use concurrent psql features
If we do things this way then we have some useful tools that can be used
in a range of use cases, not just restore.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 17:56 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-28 00:52 ` Stephen Frost <sfrost@snowman.net>
3 siblings, 0 replies; 54+ messages in thread
From: Stephen Frost @ 2008-07-28 00:52 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Right, but the parallelization is going to happen sometime, and it is
> going to happen in the context of pg_restore. So I think it's pretty
> silly to argue that no one will ever want this feature to work in
> pg_restore.
I think you've about convinced me on this, and it annoys me. ;) Worse
is that it sounds like this might cause the options to not make it in
for 8.4, which would be quite frustrating.
> To extend the example I just gave to Stephen, I think a fairly probable
> scenario is where you only need to tweak some "before" object
> definitions, and then you could do
>
> pg_restore --schema-before-data whole.dump >before.sql
> edit before.sql
> psql -f before.sql target_db
> pg_restore --data-only --schema-after-data -d target_db whole.dump
>
> which (given a parallelizing pg_restore) would do all the time-consuming
> steps in a fully parallelized fashion.
Alright, this has been mulling around in the back of my head a bit and
has now finally surfaced- I like having the whole dump contained in a
single file, but I hate having what ends up being "out-dated" or "wrong"
or "not what was loaded" in the dump file. Doesn't seem likely to be
possible, but it'd be neat to be able to modify objects in the dump
file.
Also, something which often happens to me is that I need to change the
search_path or the role at the top of a .sql from pg_dump before
restoring it. Seems like using the custom format would make that
difficult without some pipe/cat/sed magic. Parallelization would make
using that kind of magic more difficult too, I would guess. Might be
something to think about.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 17:56 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-08-01 18:02 ` chris <cbbrowne@ca.afilias.info>
2008-08-02 15:24 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
3 siblings, 1 reply; 54+ messages in thread
From: chris @ 2008-08-01 18:02 UTC (permalink / raw)
To: pgsql-patches
tgl@sss.pgh.pa.us (Tom Lane) writes:
> Simon Riggs <simon@2ndquadrant.com> writes:
>> I want to dump tables separately for performance reasons. There are
>> documented tests showing 100% gains using this method. There is no gain
>> adding this to pg_restore. There is a gain to be had - parallelising
>> index creation, but this patch doesn't provide parallelisation.
>
> Right, but the parallelization is going to happen sometime, and it is
> going to happen in the context of pg_restore. So I think it's pretty
> silly to argue that no one will ever want this feature to work in
> pg_restore.
"Never" is a long time, agreed.
> To extend the example I just gave to Stephen, I think a fairly probable
> scenario is where you only need to tweak some "before" object
> definitions, and then you could do
>
> pg_restore --schema-before-data whole.dump >before.sql
> edit before.sql
> psql -f before.sql target_db
> pg_restore --data-only --schema-after-data -d target_db whole.dump
>
> which (given a parallelizing pg_restore) would do all the time-consuming
> steps in a fully parallelized fashion.
Do we need to wait until a fully-parallelizing pg_restore is
implemented before adding this functionality to pg_dump?
The particular extension I'm interested in from pg_dump, here, is the
ability to dump multiple tables concurrently. I've got disk arrays
with enough I/O bandwidth that this form of parallelization does
provide a performance benefit.
The result of that will be that *many* files are generated, and I
don't imagine we want to change pg_restore to try to make it read from
multiple files concurrently.
Further, it's actually not obvious that we *necessarily* care about
parallelizing loading data. The thing that happens every day is
backups. I care rather a lot about optimizing that; we do backups
each and every day, and optimizations to that process will accrue
benefits each and every day.
In contrast, restoring databases does not take place every day. When
it happens, yes, there's considerable value to making *that* go as
quickly as possible, but I'm quite willing to consider optimizing that
to be separate from optimizing backups.
I daresay I haven't used pg_restore any time recently, either. Any
time we have thought about using it, we've concluded that the
perceivable benefits were actually more of a mirage.
--
select 'cbbrowne' || '@' || 'linuxfinances.info';
http://cbbrowne.com/info/lsf.html
Rules of the Evil Overlord #145. "My dungeon cell decor will not
feature exposed pipes. While they add to the gloomy atmosphere, they
are good conductors of vibrations and a lot of prisoners know Morse
code." <http://www.eviloverlord.com/;
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 17:56 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-08-01 18:02 ` Re: pg_dump additional options for performance chris <cbbrowne@ca.afilias.info>
@ 2008-08-02 15:24 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 54+ messages in thread
From: Tom Lane @ 2008-08-02 15:24 UTC (permalink / raw)
To: chris <cbbrowne@ca.afilias.info>; +Cc: pgsql-patches
chris <cbbrowne@ca.afilias.info> writes:
> Do we need to wait until a fully-parallelizing pg_restore is
> implemented before adding this functionality to pg_dump?
They're independent problems ... and I would venture that parallel
dump is harder.
> Further, it's actually not obvious that we *necessarily* care about
> parallelizing loading data. The thing that happens every day is
> backups.
Maybe so, but I would say that routine backups shouldn't be designed
to eat 100% of your disk bandwidth anyway --- they'd be more like
background tasks.
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-21 11:54 ` Stephen Frost <sfrost@snowman.net>
2008-07-21 20:38 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-21 11:54 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: daveg <daveg@sonic.net>; Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
Tom,
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> As far as the documentation/definition aspect goes, I think it should
> just say the parts are
> * stuff needed before you can load the data
> * the data
> * stuff needed after loading the data
> and not try to be any more specific than that. There are corner cases
> that will turn any simple breakdown into a lie, and I doubt that it's
> worth trying to explain them all. (Take a close look at the dependency
> loop breaking logic in pg_dump if you doubt this.)
Even that is a lie though, which I guess is what my problem is. It's
really "everything for the schema, except stuff that is better done in
bulk", I believe. Also, I'm a bit concerned about people who would
argue that you need PKs and FKs before you can load the data. Probably
couldn't be avoided tho.
> I hadn't realized that Simon was using "pre-schema" and "post-schema"
> to name the first and third parts. I'd agree that this is confusing
> nomenclature: it looks like it's trying to say that the data is the
> schema, and the schema is not! How about "pre-data and "post-data"?
Argh. The command-line options follow the 'data'/'load' line
(--schema-pre-load and --schema-post-load), and so I think those are
fine. The problem was that in the documentation he switched to saying
they were "Pre-Schema" and "Post-Schema", which could lead to confusion.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 11:54 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-21 20:38 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 21:00 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
0 siblings, 1 reply; 54+ messages in thread
From: Tom Lane @ 2008-07-21 20:38 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: daveg <daveg@sonic.net>; Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
Stephen Frost <sfrost@snowman.net> writes:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> As far as the documentation/definition aspect goes, I think it should
>> just say the parts are
>> * stuff needed before you can load the data
>> * the data
>> * stuff needed after loading the data
> Even that is a lie though, which I guess is what my problem is.
True; the stuff done after is done that way at least in part for
performance reasons rather than because it has to be done that way.
(I think it's not only performance issues, though --- for circular
FKs you pretty much have to load the data first.)
>> I hadn't realized that Simon was using "pre-schema" and "post-schema"
>> to name the first and third parts. I'd agree that this is confusing
>> nomenclature: it looks like it's trying to say that the data is the
>> schema, and the schema is not! How about "pre-data and "post-data"?
> Argh. The command-line options follow the 'data'/'load' line
> (--schema-pre-load and --schema-post-load), and so I think those are
> fine. The problem was that in the documentation he switched to saying
> they were "Pre-Schema" and "Post-Schema", which could lead to confusion.
Ah, I see. No objection to those switch names, at least assuming we
want to stick to positive-logic switches. What did you think of the
negative-logic suggestion (--omit-xxx)?
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 11:54 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 20:38 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-21 21:00 ` Stephen Frost <sfrost@snowman.net>
2008-07-21 23:19 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-21 21:00 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: daveg <daveg@sonic.net>; Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
Tom, et al,
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Ah, I see. No objection to those switch names, at least assuming we
> want to stick to positive-logic switches. What did you think of the
> negative-logic suggestion (--omit-xxx)?
My preference is for positive-logic switches in general. The place
where I would use this patch would lend itself to being more options if
--omit-xxxx were used. I expect that would hold true for most people.
It would be:
--omit-data --omit-post-load
--omit-pre-load --omit-post-load
--omit-pre-load --omit-data
vs.
--schema-pre-load
--data-only
--schema-post-load
Point being that I'd be dumping these into seperate files where I could
more easily manipulate the pre-load or post-load files. I'd still want
pre/post load to be seperate though since this would be used in cases
where there's alot of data (hence the reason for the split) and putting
pre and post together and running them before data would slow things
down quite a bit.
Are there use cases for just --omit-post-load or --omit-pre-load?
Probably, but I just don't see any situation where I'd use them like
that.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 11:54 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 20:38 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 21:00 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-21 23:19 ` Tom Lane <tgl@sss.pgh.pa.us>
2008-07-22 06:35 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 54+ messages in thread
From: Tom Lane @ 2008-07-21 23:19 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: daveg <daveg@sonic.net>; Simon Riggs <simon@2ndquadrant.com>; pgsql-patches
Stephen Frost <sfrost@snowman.net> writes:
> Are there use cases for just --omit-post-load or --omit-pre-load?
Probably not many. The thing that's bothering me is the
action-at-a-distance property of the positive-logic switches.
How are we going to explain this?
"By default, --schema-pre-load, --data-only, --schema-post-load
are all ON. But if you turn one of them ON (never mind that
it was already ON by default), that changes the defaults for
the other two to OFF. Then you have to turn them ON (never
mind that the default for them is ON) if you want two out of
the three categories."
You have to bend your mind into a pretzel to wrap it around this
behavior. Yeah, it might be convenient once you understand it,
but how long will it take for the savings in typing to repay the
time to understand it and the mistakes along the way?
regards, tom lane
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 11:54 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 20:38 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 21:00 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 23:19 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
@ 2008-07-22 06:35 ` Simon Riggs <simon@2ndquadrant.com>
0 siblings, 0 replies; 54+ messages in thread
From: Simon Riggs @ 2008-07-22 06:35 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Stephen Frost <sfrost@snowman.net>; daveg <daveg@sonic.net>; pgsql-patches
On Mon, 2008-07-21 at 19:19 -0400, Tom Lane wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > Are there use cases for just --omit-post-load or --omit-pre-load?
>
> Probably not many. The thing that's bothering me is the
> action-at-a-distance property of the positive-logic switches.
> How are we going to explain this?
>
> "By default, --schema-pre-load, --data-only, --schema-post-load
> are all ON. But if you turn one of them ON (never mind that
> it was already ON by default), that changes the defaults for
> the other two to OFF. Then you have to turn them ON (never
> mind that the default for them is ON) if you want two out of
> the three categories."
While I accept your argument a certain amount, --schema-only and
--data-only already behave in the manner you describe. Whether we pick
include or exclude or both, it will make more sense than these existing
options, regrettably.
With regard to the logic, Insert and COPY also behave this way: if you
mention *any* columns then you only get the ones you mention. We manage
to describe that also. An Insert statement would be very confusing if
you had to list the columns you don't want.
So the --omit options seem OK if you assume we'll never add further
options or include additional SQL in the dump. But that seems an
unreliable prop, so I am inclined towards the inclusive approach.
> You have to bend your mind into a pretzel to wrap it around this
> behavior.
Perhaps my mind was already toroidally challenged? :-}
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-21 06:11 ` Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:46 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
1 sibling, 1 reply; 54+ messages in thread
From: Simon Riggs @ 2008-07-21 06:11 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-patches
On Sun, 2008-07-20 at 21:18 -0400, Stephen Frost wrote:
> * Simon Riggs (simon@2ndquadrant.com) wrote:
> > On Sun, 2008-07-20 at 17:43 -0400, Stephen Frost wrote:
> > > Even this doesn't cover everything though- it's too focused on tables
> > > and data loading. Where do functions go? What about types?
> >
> > Yes, it is focused on tables and data loading. What about
> > functions/types? No relevance here.
>
> I don't see how they're not relevant, it's not like they're being
> excluded and in fact they show up in the pre-load output. Heck, even if
> they *were* excluded, that should be made clear in the documentation
> (either be an explicit include list, or saying they're excluded).
>
> Part of what's driving this is making sure we have a plan for future
> objects and where they'll go. Perhaps it would be enough to just say
> "pre-load is everything in the schema, except things which are faster
> done in bulk (eg: indexes, keys)". I don't think it's right to say
> pre-load is "only object definitions required to load data" when it
> includes functions and ACLs though.
>
> Hopefully my suggestion and these comments will get us to a happy
> middle-ground.
I don't really understand what you're saying.
The options split the dump into 3 parts that's all: before the load, the
load and after the load.
--schema-pre-load says
"Dumps exactly what <option>--schema-only</> would dump, but only those
statements before the data load."
What is it you are suggesting? I'm unclear.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 06:11 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
@ 2008-07-21 11:46 ` Stephen Frost <sfrost@snowman.net>
2008-07-21 19:26 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
0 siblings, 1 reply; 54+ messages in thread
From: Stephen Frost @ 2008-07-21 11:46 UTC (permalink / raw)
To: Simon Riggs <simon@2ndquadrant.com>; +Cc: pgsql-patches
* Simon Riggs (simon@2ndquadrant.com) wrote:
> The options split the dump into 3 parts that's all: before the load, the
> load and after the load.
>
> --schema-pre-load says
> "Dumps exactly what <option>--schema-only</> would dump, but only those
> statements before the data load."
>
> What is it you are suggesting? I'm unclear.
That part is fine, the problem is that elsewhere in the documentation
(patch line starting ~774 before, ~797 after, to the pg_dump.sgml) you
change it to be "objects required before data loading", which isn't the
same.
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 196B) download
^ permalink raw reply [nested|flat] 54+ messages in thread
* Re: pg_dump additional options for performance
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 06:11 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:46 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
@ 2008-07-21 19:26 ` Simon Riggs <simon@2ndquadrant.com>
0 siblings, 0 replies; 54+ messages in thread
From: Simon Riggs @ 2008-07-21 19:26 UTC (permalink / raw)
To: Stephen Frost <sfrost@snowman.net>; +Cc: pgsql-patches
On Mon, 2008-07-21 at 07:46 -0400, Stephen Frost wrote:
> * Simon Riggs (simon@2ndquadrant.com) wrote:
> > The options split the dump into 3 parts that's all: before the load, the
> > load and after the load.
> >
> > --schema-pre-load says
> > "Dumps exactly what <option>--schema-only</> would dump, but only those
> > statements before the data load."
> >
> > What is it you are suggesting? I'm unclear.
>
> That part is fine, the problem is that elsewhere in the documentation
> (patch line starting ~774 before, ~797 after, to the pg_dump.sgml) you
> change it to be "objects required before data loading", which isn't the
> same.
OK, gotcha now - will change that. I thought you might mean something
about changing the output itself.
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
^ permalink raw reply [nested|flat] 54+ messages in thread
end of thread, other threads:[~2008-08-02 15:24 UTC | newest]
Thread overview: 54+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-05-07 14:37 [Fwd: Re: [HACKERS] pg_dump additional options for performance] Simon Riggs <simon@2ndquadrant.com>
2008-06-26 22:42 ` Bruce Momjian <bruce@momjian.us>
2008-07-20 03:07 pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 04:47 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 06:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-20 21:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-20 21:59 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 01:18 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 01:49 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-21 01:58 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 03:34 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 06:17 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 06:47 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 15:19 ` Re: pg_dump additional options for performance Andrew Dunstan <andrew@dunslane.net>
2008-07-21 15:38 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 11:56 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-23 16:40 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-23 17:09 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 03:20 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-24 07:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-24 07:54 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-24 08:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-25 23:16 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 02:39 ` Re: pg_dump additional options for performance Greg Sabino Mullane <greg@turnstep.com>
2008-07-26 09:05 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 15:11 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 16:24 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:20 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 16:43 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 17:43 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-26 18:03 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-27 09:37 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-27 16:57 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-27 23:42 ` Re: pg_dump additional options for performance Andrew Dunstan <andrew@dunslane.net>
2008-07-28 00:24 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-28 01:32 ` Re: [PATCHES] pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-28 00:44 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-28 02:40 ` Re: pg_dump additional options for performance Joshua D. Drake <jd@commandprompt.com>
2008-07-28 02:50 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-26 16:56 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-26 17:56 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-27 01:33 ` Re: pg_dump additional options for performance daveg <daveg@sonic.net>
2008-07-27 09:31 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-28 00:52 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-08-01 18:02 ` Re: pg_dump additional options for performance chris <cbbrowne@ca.afilias.info>
2008-08-02 15:24 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 11:54 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 20:38 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-21 21:00 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 23:19 ` Re: pg_dump additional options for performance Tom Lane <tgl@sss.pgh.pa.us>
2008-07-22 06:35 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 06:11 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
2008-07-21 11:46 ` Re: pg_dump additional options for performance Stephen Frost <sfrost@snowman.net>
2008-07-21 19:26 ` Re: pg_dump additional options for performance Simon Riggs <simon@2ndquadrant.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox