From operations_bradley@servillian.ca Tue Jan 18 00:17:41 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 4AABC1337B43 for ; Mon, 17 Jan 2011 20:17:41 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 55065-03 for ; Tue, 18 Jan 2011 00:17:34 +0000 (UTC) X-Greylist: delayed 00:09:47.364789 by SQLgrey-1.7.6 Received: from mail.shawcable.com (shawmail.shawcable.com [64.59.128.220]) by mail.postgresql.org (Postfix) with ESMTP id E5EDF13364DE for ; Mon, 17 Jan 2011 20:17:32 -0400 (AST) Received: from bpd2mi5no-svcs.prod.shawcable.com ([10.0.184.160]) by bpd2mo2no-ssvc.prod.shawcable.com with ESMTP; 17 Jan 2011 17:07:44 -0700 X-Cloudmark-SP-Filtered: true X-Cloudmark-SP-Result: v=1.1 cv=t+ZAJ7mrPn0ghCTQRrYVa3vbw5sbuRhC9KBpKnc45f0= c=1 sm=1 a=BLceEmwcHowA:10 a=rFBdEGy9vS/m0A3AhxL8nA==:17 a=YzUr2nKEWDLvlaGn-LoA:9 a=9mmBr3PAjAkofoIw4SYA:7 a=uTi-UYhoUti2V83mmYcfFhM3wnQA:4 a=CjuIK1q_8ugA:10 a=yMhMjlubAAAA:8 a=SSmOFEACAAAA:8 a=TZxhmd7RFYRxfCsiG3EA:9 a=CdVOxXWoEKgw8PNpHh4A:7 a=VsDAQTFtK2tNka_VzXUKu8phZsUA:4 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 X-IronPort-AV: E=Sophos;i="4.60,335,1291618800"; d="scan'208,217";a="247800736" Received: from unknown (HELO bpd2mi5no-cmts.prod.shawcable.com) ([192.168.183.160]) by bpd2mi5no-cmts.prod.shawcable.com with ESMTP; 17 Jan 2011 17:07:44 -0700 X-reinject: true Received: from unknown (HELO BradleyHolbrook) ([24.67.39.150]) by bpd2mi5no-dmz.prod.shawcable.com with ESMTP; 17 Jan 2011 17:07:43 -0700 From: "Bradley Holbrook" To: Subject: Postgres Backup Utility Date: Mon, 17 Jan 2011 16:07:43 -0800 Message-ID: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0362_01CBB660.AC7F7160" X-Mailer: Microsoft Outlook 14.0 Thread-Index: Acu2o3byZfx/c2ZtRkmfs+fFuxiwfw== Content-Language: en-ca X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.899 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201101/101 X-Sequence-Number: 36541 This is a multipart message in MIME format. ------=_NextPart_000_0362_01CBB660.AC7F7160 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello! First day on the new mailing list as I have need of some expert's advice. I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on lookup tables, but not content tables). Any help would be appreciated! Brad ------=_NextPart_000_0362_01CBB660.AC7F7160 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello!

 

First = day on the new mailing list as I have need of some expert’s = advice.

 

I need = to be able to quickly apply the structure updates from a development = database to a testing database, and do selective data updates (like on = lookup tables, but not content tables).

 

Any help = would be appreciated!

 

Brad

<= /body> ------=_NextPart_000_0362_01CBB660.AC7F7160-- From frenchm@cromwell.co.uk Tue Jan 18 13:52:15 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id B76FD133655C for ; Tue, 18 Jan 2011 09:52:15 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 51978-01 for ; Tue, 18 Jan 2011 13:52:06 +0000 (UTC) X-Greylist: delayed 00:05:16.221771 by SQLgrey-1.7.6 Received: from ariel.cromwell.co.uk (mx0.cromwell.co.uk [213.249.208.130]) by mail.postgresql.org (Postfix) with ESMTP id 446C71337B43 for ; Tue, 18 Jan 2011 09:52:05 -0400 (AST) Received: from roundway.Cromwell-tools.co.uk (roundway [194.216.28.40]) by ariel.cromwell.co.uk (8.12.8/8.12.8) with ESMTP id p0IDl8MO021637; Tue, 18 Jan 2011 13:47:09 GMT X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CBB716.21D844BD" Subject: Re: Postgres Backup Utility Date: Tue, 18 Jan 2011 13:46:37 -0000 Message-ID: <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [ADMIN] Postgres Backup Utility Thread-Index: Acu2o3byZfx/c2ZtRkmfs+fFuxiwfwAcmVlA References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> From: "French, Martin" To: "Bradley Holbrook" , X-Cromwell-Group-MailScanner-Information: Cromwell Group MailScanner X-Cromwell-Group-MailScanner: Clean X-Cromwell-Group-MailScanner-MCPCheck: MCP-Clean, MCP-Checker (score=0, required 10) X-Cromwell-Group-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (not cached, score=-2.36, required 2, autolearn=not spam, BAYES_00 -3.00, HTML_ATTR_UNIQUE 0.64, HTML_MESSAGE 0.00) X-Cromwell-Group-MailScanner-From: frenchm@cromwell.co.uk X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.909 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201101/113 X-Sequence-Number: 36553 This is a multi-part message in MIME format. ------_=_NextPart_001_01CBB716.21D844BD Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I'm assuming that this needs to be tightly controlled and as such a replication tool is out of the question? =20 In that case; The first thing to pop into my head here would be to use either use shell scripting, or to use the pg API and write a c program to handle it. =20 I remember doing something very similar with Oracle a few years back. =20 Cheers =20 Martin =20 =20 From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Bradley Holbrook Sent: 18 January 2011 00:08 To: pgsql-admin@postgresql.org Subject: [ADMIN] Postgres Backup Utility =20 Hello! =20 First day on the new mailing list as I have need of some expert's advice. =20 I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on lookup tables, but not content tables). =20 Any help would be appreciated! =20 Brad ___________________________________________________=20 =20=20 This email is intended for the named recipient. The information contained= =20 in it is confidential. You should not copy it for any purposes, nor=20 disclose its contents to any other party. If you received this email=20 in error, please notify the sender immediately via email, and delete it from your computer.=20 =20=20 Any views or opinions presented are solely those of the author and do not= =20 necessarily represent those of the company.=20 =20=20 PCI Compliancy: Please note, we do not send or wish to receive banking, cre= dit or debit card information by email or any other form of communication.=20= =20=20=20=20=20=20=20=20 Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive Wigston, Leicester LE18 1AT. Tel 0116 2888000 Registered in England and Wales, Reg No 00986161 VAT GB 115 5713 87 900 __________________________________________________ ------_=_NextPart_001_01CBB716.21D844BD Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I’m assuming that = this needs to be tightly controlled and as such a replication tool is out of the question= ?

 =

In that case; The first = thing to pop into my head here would be to use either use shell scripting, or to use= the pg API and write a c program to handle it.

 =

I remember doing somethi= ng very similar with Oracle a few years back.

 =

Cheers=

 =

Martin=

 =

 =

From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] = On Behalf Of Bradley Holbrook
Sent: 18 January 2011 00:08
To: pgsql-admin@postgresql.org
Subject: [ADMIN] Postgres Backup Utility

 

Hello!

 

First day on the new mailing list as I have need of some expert’s advice.

 

I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on look= up tables, but not content tables).

 

Any help would be appreciated!

 

Brad


___________________________________________________=20
=20=20
This email is intended for the named recipient. The information conta= ined=20
in it is confidential. You should not copy it for any purposes, nor= =20
disclose its contents to any other party. If you received this email= =20
in error, please notify the sender immediately via email, and delete
it from your computer.=20
=20=20
Any views or opinions presented are solely those of the author and do= not=20
necessarily represent those of the company.=20
=20=20
PCI Compliancy: Please note, we do not send or wish to receive bankin= g,
credit or debit card information by email or any other form of=20=20= =20=20=20=20=20=20=20=20
communication.=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive
Wigston, Leicester LE18 1AT. Tel 0116 2888000
Registered in England and Wales, Reg No 00986161
VAT GB 115 5713 87 900
__________________________________________________ ------_=_NextPart_001_01CBB716.21D844BD-- From operations_bradley@servillian.ca Wed Jan 19 00:29:19 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 4F687133798E for ; Tue, 18 Jan 2011 20:29:19 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 08920-01 for ; Wed, 19 Jan 2011 00:29:09 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail.shawcable.com (shawmail.shawcable.com [64.59.128.220]) by mail.postgresql.org (Postfix) with ESMTP id AE872133655C for ; Tue, 18 Jan 2011 20:29:07 -0400 (AST) Received: from bpd2mi4no-svcs.prod.shawcable.com ([10.0.184.123]) by bpd2mo1no-ssvc.prod.shawcable.com with ESMTP; 18 Jan 2011 09:57:07 -0700 X-Cloudmark-SP-Filtered: true X-Cloudmark-SP-Result: v=1.1 cv=W1EbViUdRLrB3xZls61b9cpNy9JNgdRUyrcMJe3sx/g= c=1 sm=1 a=fKHZdZmfl1IA:10 a=BLceEmwcHowA:10 a=rFBdEGy9vS/m0A3AhxL8nA==:17 a=GbhIXFikAAAA:8 a=epTmVMiNAAAA:8 a=iJ9aCO6GRrQD_T7OIDoA:9 a=bWlImOfUAsvqMw5by_4A:7 a=BwJzj4TyeP_aHFJ68zcvW418VA4A:4 a=CjuIK1q_8ugA:10 a=YsdO3FDBEMUA:10 a=YObXiYvaVLwA:10 a=lDsZwgibEb_jubUg:21 a=YS9T69ZPB3Ib9Cjc:21 a=yMhMjlubAAAA:8 a=SSmOFEACAAAA:8 a=Y2ELxnj51GYLGX921J4A:9 a=ieKT8QZfNYUopV0-3eYA:7 a=jVQJNVL32nVWZR0UlqjL6GwpHaYA:4 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 X-IronPort-AV: E=Sophos;i="4.60,339,1291618800"; d="scan'208,217";a="302587431" Received: from unknown (HELO bpd2mi4no-cmts.prod.shawcable.com) ([192.168.183.123]) by bpd2mi4no-cmts.prod.shawcable.com with ESMTP; 18 Jan 2011 09:57:07 -0700 X-reinject: true Received: from unknown (HELO BradleyHolbrook) ([24.67.39.150]) by bpd2mi4no-dmz.prod.shawcable.com with ESMTP; 18 Jan 2011 09:57:07 -0700 From: "Bradley Holbrook" To: "'French, Martin'" Cc: References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> In-Reply-To: <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> Subject: Re: Postgres Backup Utility Date: Tue, 18 Jan 2011 08:57:07 -0800 Message-ID: <004e01cbb730$bdb21930$39164b90$@servillian.ca> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_004F_01CBB6ED.AF8ED930" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHHkvKLrosR4aSyAm2SjCI6Zq933AO9xqP7k8FNV+A= Content-Language: en-ca X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.899 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201101/114 X-Sequence-Number: 36554 This is a multipart message in MIME format. ------=_NextPart_000_004F_01CBB6ED.AF8ED930 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Well, I can't just go dropping and recreating tables. it needs to create the correct alter statements if existing tables and or functions already exist. Secondly, when I'm finished changing the structure, I need to be able to select the list of tables that will have content updates. Using a script might be more work maintaining then it's worth. I have a backup utility that can do the job, but 3 tedious steps per schema, that only work about 10% of the time (and no batching options so that I can create a list of actions and run the list). From: French, Martin [mailto:frenchm@cromwell.co.uk] Sent: January-18-11 5:47 AM To: Bradley Holbrook; pgsql-admin@postgresql.org Subject: RE: [ADMIN] Postgres Backup Utility I'm assuming that this needs to be tightly controlled and as such a replication tool is out of the question? In that case; The first thing to pop into my head here would be to use either use shell scripting, or to use the pg API and write a c program to handle it. I remember doing something very similar with Oracle a few years back. Cheers Martin From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Bradley Holbrook Sent: 18 January 2011 00:08 To: pgsql-admin@postgresql.org Subject: [ADMIN] Postgres Backup Utility Hello! First day on the new mailing list as I have need of some expert's advice. I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on lookup tables, but not content tables). Any help would be appreciated! Brad ___________________________________________________ This email is intended for the named recipient. The information contained in it is confidential. You should not copy it for any purposes, nor disclose its contents to any other party. If you received this email in error, please notify the sender immediately via email, and delete it from your computer. Any views or opinions presented are solely those of the author and do not necessarily represent those of the company. PCI Compliancy: Please note, we do not send or wish to receive banking, credit or debit card information by email or any other form of communication. Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive Wigston, Leicester LE18 1AT. Tel 0116 2888000 Registered in England and Wales, Reg No 00986161 VAT GB 115 5713 87 900 __________________________________________________ ------=_NextPart_000_004F_01CBB6ED.AF8ED930 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Well, I can’t just go = dropping and recreating tables… it needs to create the correct = alter statements if existing tables and or functions already = exist.

 

Secondly, when I’m = finished changing the structure, I need to be able to select the list of = tables that will have content updates.

 

Using a = script might be more work maintaining then it’s worth. I have a = backup utility that can do the job, but 3 tedious steps per schema, that = only work about 10% of the time (and no batching options so that I can = create a list of actions and run the list).

 

 

=

From:= French, = Martin [mailto:frenchm@cromwell.co.uk]
Sent: January-18-11 = 5:47 AM
To: Bradley Holbrook; = pgsql-admin@postgresql.org
Subject: RE: [ADMIN] Postgres = Backup Utility

 

I’m assuming that this needs = to be tightly controlled and as such a replication tool is out of the = question?

 

In that = case; The first thing to pop into my head here would be to use either = use shell scripting, or to use the pg API and write a c program to = handle it.

 

I remember = doing something very similar with Oracle a few years = back.

 

Cheers

 

Martin

 

 

From:= = pgsql-admin-owner@postgresql.org = [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Bradley = Holbrook
Sent: 18 January 2011 00:08
To: = pgsql-admin@postgresql.org
Subject: [ADMIN] Postgres Backup = Utility

 

Hello!

 

First = day on the new mailing list as I have need of some expert’s = advice.

 

I need = to be able to quickly apply the structure updates from a development = database to a testing database, and do selective data updates (like on = lookup tables, but not content tables).

 

Any help = would be appreciated!

 

Brad


___________________________________________________ =

This email is intended for the named recipient. The information = contained
in it is confidential. You should not copy it for any = purposes, nor
disclose its contents to any other party. If you = received this email
in error, please notify the sender immediately = via email, and delete
it from your computer.

Any views or = opinions presented are solely those of the author and do not =
necessarily represent those of the company.

PCI Compliancy: = Please note, we do not send or wish to receive banking,
credit or = debit card information by email or any other form of
communication. =

Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive =
Wigston, Leicester LE18 1AT. Tel 0116 2888000
Registered in = England and Wales, Reg No 00986161
VAT GB 115 5713 87 900 =
__________________________________________________ =

------=_NextPart_000_004F_01CBB6ED.AF8ED930-- From frenchm@cromwell.co.uk Wed Jan 19 07:13:01 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id E8D59133655C for ; Wed, 19 Jan 2011 03:13:01 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 08902-07 for ; Wed, 19 Jan 2011 07:12:51 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from ariel.cromwell.co.uk (mx0.cromwell.co.uk [213.249.208.130]) by mail.postgresql.org (Postfix) with ESMTP id E661813364DE for ; Wed, 19 Jan 2011 03:12:50 -0400 (AST) Received: from roundway.Cromwell-tools.co.uk (roundway [194.216.28.40]) by ariel.cromwell.co.uk (8.12.8/8.12.8) with ESMTP id p0J7DDMO032750; Wed, 19 Jan 2011 07:13:13 GMT X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CBB7A8.44292720" Subject: Re: Postgres Backup Utility Date: Wed, 19 Jan 2011 07:12:43 -0000 Message-ID: <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [ADMIN] Postgres Backup Utility Thread-Index: AQHHkvKLrosR4aSyAm2SjCI6Zq933AO9xqP7k8FNV+CAAOl0MA== References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> From: "French, Martin" To: "Bradley Holbrook" Cc: X-Cromwell-Group-MailScanner-Information: Cromwell Group MailScanner X-Cromwell-Group-MailScanner: Clean X-Cromwell-Group-MailScanner-MCPCheck: MCP-Clean, MCP-Checker (score=0, required 10) X-Cromwell-Group-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (not cached, score=-2.36, required 2, autolearn=not spam, BAYES_00 -3.00, HTML_ATTR_UNIQUE 0.64, HTML_MESSAGE 0.00) X-Cromwell-Group-MailScanner-From: frenchm@cromwell.co.uk X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.909 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201101/115 X-Sequence-Number: 36555 This is a multi-part message in MIME format. ------_=_NextPart_001_01CBB7A8.44292720 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Ok, you say that you cannot drop and recreate, so you need to do this via alter statements only? That's obviously going to complicate matters, as a straight dump, drop, recreate, restore would be the fastest and by far simplest method. =20 So, Ideally, you'll need to do a table def comparison over the two databases, and generate the necessary sql to amend the tables in test accordingly?=20 =20 Querying the pg_catalog/information_schema over the two db's should give you the table ddl from which you can diff, and then generate the alter statements from the results.=20 =20 Cheers=20 =20 Martin =20 From: Bradley Holbrook [mailto:operations_bradley@servillian.ca]=20 Sent: 18 January 2011 16:57 To: French, Martin Cc: pgsql-admin@postgresql.org Subject: RE: [ADMIN] Postgres Backup Utility =20 Well, I can't just go dropping and recreating tables... it needs to create the correct alter statements if existing tables and or functions already exist. =20 Secondly, when I'm finished changing the structure, I need to be able to select the list of tables that will have content updates. =20 Using a script might be more work maintaining then it's worth. I have a backup utility that can do the job, but 3 tedious steps per schema, that only work about 10% of the time (and no batching options so that I can create a list of actions and run the list). =20 =20 From: French, Martin [mailto:frenchm@cromwell.co.uk]=20 Sent: January-18-11 5:47 AM To: Bradley Holbrook; pgsql-admin@postgresql.org Subject: RE: [ADMIN] Postgres Backup Utility =20 I'm assuming that this needs to be tightly controlled and as such a replication tool is out of the question? =20 In that case; The first thing to pop into my head here would be to use either use shell scripting, or to use the pg API and write a c program to handle it. =20 I remember doing something very similar with Oracle a few years back. =20 Cheers =20 Martin =20 =20 From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Bradley Holbrook Sent: 18 January 2011 00:08 To: pgsql-admin@postgresql.org Subject: [ADMIN] Postgres Backup Utility =20 Hello! =20 First day on the new mailing list as I have need of some expert's advice. =20 I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on lookup tables, but not content tables). =20 Any help would be appreciated! =20 Brad ___________________________________________________=20 This email is intended for the named recipient. The information contained=20 in it is confidential. You should not copy it for any purposes, nor=20 disclose its contents to any other party. If you received this email=20 in error, please notify the sender immediately via email, and delete=20 it from your computer.=20 Any views or opinions presented are solely those of the author and do not=20 necessarily represent those of the company.=20 PCI Compliancy: Please note, we do not send or wish to receive banking,=20 credit or debit card information by email or any other form of=20 communication.=20 Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive=20 Wigston, Leicester LE18 1AT. Tel 0116 2888000=20 Registered in England and Wales, Reg No 00986161=20 VAT GB 115 5713 87 900=20 __________________________________________________=20 ___________________________________________________=20 =20=20 This email is intended for the named recipient. The information contained= =20 in it is confidential. You should not copy it for any purposes, nor=20 disclose its contents to any other party. If you received this email=20 in error, please notify the sender immediately via email, and delete it from your computer.=20 =20=20 Any views or opinions presented are solely those of the author and do not= =20 necessarily represent those of the company.=20 =20=20 PCI Compliancy: Please note, we do not send or wish to receive banking, cre= dit or debit card information by email or any other form of communication.=20= =20=20=20=20=20=20=20=20 Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive Wigston, Leicester LE18 1AT. Tel 0116 2888000 Registered in England and Wales, Reg No 00986161 VAT GB 115 5713 87 900 __________________________________________________ ------_=_NextPart_001_01CBB7A8.44292720 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Ok, you say that you can= not drop and recreate, so you need to do this via alter statements only? That’s obviously going to complicate matters, as a straight dump, drop, recreate, restore would be the fastest and by far simplest method.<= /p>

 =

So, Ideally, you’l= l need to do a table def comparison over the two databases, and generate the neces= sary sql to amend the tables in test accordingly?

 =

Querying the pg_catalog/= information_schema over the two db’s should give you the table ddl from which you can di= ff, and then generate the alter statements from the results. =

 =

Cheers

 =

Martin=

 =

From: Bradley Holbr= ook [mailto:operations_bradley@servillian.ca]
Sent: 18 January 2011 16:57
To: French, Martin
Cc: pgsql-admin@postgresql.org
Subject: RE: [ADMIN] Postgres Backup Utility

 

Well, I can’t just go dropping and recreating tables… it needs to cre= ate the correct alter statements if existing tables and or functions already ex= ist.

 

Secondly, when I’m finished changing the structure, I need to be able to select= the list of tables that will have content updates.

 

Using a script might be more work maintaining then it’s worth. I have a bac= kup utility that can do the job, but 3 tedious steps per schema, that only work about 10% of the time (and no batching options so that I can create a list = of actions and run the list).

 

 

From: French, Martin [mailto:frenchm@cromwell.co.uk]
Sent: January-18-11 5:47 AM
To: Bradley Holbrook; pgsql-admin@postgresql.org
Subject: RE: [ADMIN] Postgres Backup Utility

 

I’m assuming that = this needs to be tightly controlled and as such a replication tool is out of the question?

 =

In that case; The first = thing to pop into my head here would be to use either use shell scripting, or to use= the pg API and write a c program to handle it.

 =

I remember doing somethi= ng very similar with Oracle a few years back.

 =

Cheers=

 =

Martin=

 =

 =

From: pgsql-admin-o= wner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Bradley Holbr= ook
Sent: 18 January 2011 00:08
To: pgsql-admin@postgresql.org
Subject: [ADMIN] Postgres Backup Utility

 

Hello!

 

First day on the new mailing list as I have need of some expert’s advice.

 

I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on look= up tables, but not content tables).

 

Any help would be appreciated!

 

Brad


___________________________________________________

This email is intended for the named recipient. The information contained <= br> in it is confidential. You should not copy it for any purposes, nor
disclose its contents to any other party. If you received this email
in error, please notify the sender immediately via email, and delete
it from your computer.

Any views or opinions presented are solely those of the author and do not <= br> necessarily represent those of the company.

PCI Compliancy: Please note, we do not send or wish to receive banking,
credit or debit card information by email or any other form of
communication.

Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive
Wigston, Leicester LE18 1AT. Tel 0116 2888000
Registered in England and Wales, Reg No 00986161
VAT GB 115 5713 87 900
__________________________________________________


___________________________________________________=20
=20=20
This email is intended for the named recipient. The information conta= ined=20
in it is confidential. You should not copy it for any purposes, nor= =20
disclose its contents to any other party. If you received this email= =20
in error, please notify the sender immediately via email, and delete
it from your computer.=20
=20=20
Any views or opinions presented are solely those of the author and do= not=20
necessarily represent those of the company.=20
=20=20
PCI Compliancy: Please note, we do not send or wish to receive bankin= g,
credit or debit card information by email or any other form of=20=20= =20=20=20=20=20=20=20=20
communication.=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20
=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20
Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive
Wigston, Leicester LE18 1AT. Tel 0116 2888000
Registered in England and Wales, Reg No 00986161
VAT GB 115 5713 87 900
__________________________________________________ ------_=_NextPart_001_01CBB7A8.44292720-- From bob_lunney@yahoo.com Wed Jan 19 19:17:44 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 1FA001337C3A for ; Wed, 19 Jan 2011 15:17:44 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 13655-02-8 for ; Wed, 19 Jan 2011 19:17:35 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from web39704.mail.mud.yahoo.com (web39704.mail.mud.yahoo.com [209.191.106.50]) by mail.postgresql.org (Postfix) with SMTP id 145201337B69 for ; Wed, 19 Jan 2011 15:13:57 -0400 (AST) Received: (qmail 14400 invoked by uid 60001); 19 Jan 2011 19:13:56 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1295464436; bh=QWdB2N/4mSsbIYvkdBe6GYlYJqOXybLXKkB1ByDB76c=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=ETj+5564aNImE6iX6i7BwRVicUZKLeo7OaesmJFh0nv45qLieCQ3RfN6vu/Hld3d4QbSC/XUNO4DRQufVKHOXr5jhJx/vozG6V2zaZ6BLht9MulW33ONAAQsTY4YVzCDdptQ+E3tJRLEo8RVJJ9vwiGyGAU89/8z8EYh6xWx7xM= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=P/ZjvU1Zaf9JykDCl5nzDRhQcrnop0CkYTdPxgD4niDT04OdgDGO+ha5isrd3m4qeO83QHPclP2BtWCq3F2YZ6IDm1B62X8IZfmSqlwOC+SVTrVjG815hzqqoh98P/f40sXXxwDhY622hqzH2bkO6d8bk/lSq8P2/6LXiJZ5OTA=; Message-ID: <536560.14223.qm@web39704.mail.mud.yahoo.com> X-YMail-OSG: JKwLImYVM1nGp4jMlPITQ3bdkXPMAxt8ZeT3f0QWnb_laX7 DqE8_k3Vc.lVbSGYsV6djoi6ODFm8dVaCpS9nqatFWsPkOJfvKSHgWCv3WFm UR7Or1A8fZFDmA00lxPc28dAHejcj.0Bom9Y6lAb_9xbCqDddwl257UwFeDs 2XV4VtZ4N9CaWeG_FFKwcIQ.9j6yG9bZbvHNzOH6p6vCgT5BXh8Xh42RXKp9 GJdMfGRJ8SNNfzGBobPtkoL1n.399QraF4MhNeojPPwdXORAh00GkVqwmQkD GYDtxugygPdPdGTBezLWKHeEt2toXWwxBP9giiDoyKVZfQQu09GJrhSw- Received: from [64.126.29.122] by web39704.mail.mud.yahoo.com via HTTP; Wed, 19 Jan 2011 11:13:56 PST X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.107.285259 Date: Wed, 19 Jan 2011 11:13:56 -0800 (PST) From: Bob Lunney Subject: Re: Postgres Backup Utility To: Bradley Holbrook Cc: pgsql-admin@postgresql.org In-Reply-To: <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-2061493445-1295464436=:14223" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.907 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RFC_ABUSE_POST=0.001, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201101/116 X-Sequence-Number: 36556 --0-2061493445-1295464436=:14223 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Brad, Google for "SQL Power Architect", download it, and try a schema comparison.= =C2=A0That might get you a ways down to road to what you want. Bob Lunney --- On Wed, 1/19/11, French, Martin wrote: From: French, Martin Subject: Re: [ADMIN] Postgres Backup Utility To: "Bradley Holbrook" Cc: pgsql-admin@postgresql.org Date: Wednesday, January 19, 2011, 2:12 AM =0A=0A=0A =0A =0A=0A=0A=0A=0A=0A=0A =0A=0A=0A=0AOk, you say that you cannot= drop=0Aand recreate, so you need to do this via alter statements only? Tha= t=E2=80=99s=0Aobviously going to complicate matters, as a straight dump, dr= op, recreate,=0Arestore would be the fastest and by far simplest method. = =0A=0A =C2=A0 =0A=0ASo, Ideally, you=E2=80=99ll need=0Ato do a table def co= mparison over the two databases, and generate the necessary=0Asql to amend = the tables in test accordingly? =0A=0A =C2=A0 =0A=0AQuerying the pg_catalo= g/information_schema=0Aover the two db=E2=80=99s should give you the table = ddl from which you can diff,=0Aand then generate the alter statements from = the results. =0A=0A =C2=A0 =0A=0ACheers =0A=0A =C2=A0 =0A=0AMartin =0A=0A= =C2=A0 =0A=0A=0A=0A=0A=0AFrom: Bradley Holbrook=0A[mailto:operations_bradl= ey@servillian.ca]=20 =0ASent: 18 January 2011 16:57 =0ATo: French, Martin =0ACc: pgsql-admin@postgresql.org =0ASubject: RE: [ADMIN] Postgres Backup Utility =0A=0A=0A=0A=0A=0A =C2=A0 = =0A=0AWell,=0AI can=E2=80=99t just go dropping and recreating tables=E2=80= =A6 it needs to create=0Athe correct alter statements if existing tables an= d or functions already exist. =0A=0A =C2=A0 =0A=0ASecondly,=0Awhen I=E2=80= =99m finished changing the structure, I need to be able to select the=0Alis= t of tables that will have content updates. =0A=0A =C2=A0 =0A=0AUsing=0Aa s= cript might be more work maintaining then it=E2=80=99s worth. I have a back= up=0Autility that can do the job, but 3 tedious steps per schema, that only= work=0Aabout 10% of the time (and no batching options so that I can create= a list of=0Aactions and run the list). =0A=0A =C2=A0 =0A=0A =C2=A0 =0A=0A= =0A=0A=0A=0AFrom: French, Martin=0A[mailto:frenchm@cromwell.co.uk]=20 =0ASent: January-18-11 5:47 AM =0ATo: Bradley Holbrook; pgsql-admin@postgresql.org =0ASubject: RE: [ADMIN] Postgres Backup Utility =0A=0A=0A=0A=0A=0A =C2=A0 = =0A=0AI=E2=80=99m assuming that this=0Aneeds to be tightly controlled and a= s such a replication tool is out of the=0Aquestion? =0A=0A =C2=A0 =0A=0AIn = that case; The first thing to=0Apop into my head here would be to use eithe= r use shell scripting, or to use the=0Apg API and write a c program to hand= le it. =0A=0A =C2=A0 =0A=0AI remember doing something very=0Asimilar with O= racle a few years back. =0A=0A =C2=A0 =0A=0ACheers =0A=0A =C2=A0 =0A=0AMart= in =0A=0A =C2=A0 =0A=0A =C2=A0 =0A=0A=0A=0A=0A=0AFrom: pgsql-admin-owner@po= stgresql.org=0A[mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Bradl= ey Holbrook =0ASent: 18 January 2011 00:08 =0ATo: pgsql-admin@postgresql.org =0ASubject: [ADMIN] Postgres Backup Utility =0A=0A=0A=0A=0A=0A =C2=A0 =0A= =0AHello! =0A=0A =C2=A0 =0A=0AFirst=0Aday on the new mailing list as I have= need of some expert=E2=80=99s advice. =0A=0A =C2=A0 =0A=0AI=0Aneed to be a= ble to quickly apply the structure updates from a development=0Adatabase to= a testing database, and do selective data updates (like on lookup=0Atables= , but not content tables). =0A=0A =C2=A0 =0A=0AAny=0Ahelp would be apprecia= ted! =0A=0A =C2=A0 =0A=0ABrad =0A=0A =0A___________________________________________________=20 =0A =0AThis email is intended for the named recipient. The information containe= d=20 =0Ain it is confidential. You should not copy it for any purposes, nor=20 =0Adisclose its contents to any other party. If you received this email=20 =0Ain error, please notify the sender immediately via email, and delete=20 =0Ait from your computer.=20 =0A =0AAny views or opinions presented are solely those of the author and do no= t=20 =0Anecessarily represent those of the company.=20 =0A =0APCI Compliancy: Please note, we do not send or wish to receive banking,= =20 =0Acredit or debit card information by email or any other form of=20 =0Acommunication.=20 =0A =0ACromwell Tools Limited, PO Box 14, 65 Chartwell Drive=20 =0AWigston, Leicester LE18 1AT. Tel 0116 2888000=20 =0ARegistered in England and Wales, Reg No 00986161=20 =0AVAT GB 115 5713 87 900=20 =0A__________________________________________________ =0A=0A=0A=0A =0A=0A ___________________________________________________ =0A =0A This email is intended for the named recipient. The information contained = =0A in it is confidential. You should not copy it for any purposes, nor =0A disclose its contents to any other party. If you received this email =0A in error, please notify the sender immediately via email, and delete=0A it from your computer. =0A =0A Any views or opinions presented are solely those of the author and do not = =0A necessarily represent those of the company. =0A =0A PCI Compliancy: Please note, we do not send or wish to receive banking,=0A credit or debit card information by email or any other form of =0A communication. =0A =0A Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive=0A Wigston, Leicester LE18 1AT. Tel 0116 2888000=0A Registered in England and Wales, Reg No 00986161=0A VAT GB 115 5713 87 900=0A __________________________________________________=0A=0A=0A=0A=0A --0-2061493445-1295464436=:14223 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Brad,

Google for "SQL Powe= r Architect", download it, and try a schema comparison.  That might ge= t you a ways down to road to what you want.

Bob Lu= nney

--- On Wed, 1/19/11, French, Martin <frenchm@cromwell.= co.uk> wrote:

From: French, Mart= in <frenchm@cromwell.co.uk>
Subject: Re: [ADMIN] Postgres Backup U= tility
To: "Bradley Holbrook" <operations_bradley@servillian.ca>Cc: pgsql-admin@postgresql.org
Date: Wednesday, January 19, 2011, 2:12= AM

=0A=0A=0A =0A =0A=0A=0A=0A=0A =0A=0A
=0A=0A

Ok, you say that you cannot drop=0Aand recreate, so you= need to do this via alter statements only? That=E2=80=99s=0Aobviously goin= g to complicate matters, as a straight dump, drop, recreate,=0Arestore woul= d be the fastest and by far simplest method.

=0A=0A

 

= =0A=0A

So= , Ideally, you=E2=80=99ll need=0Ato do a table def comparison over the two = databases, and generate the necessary=0Asql to amend the tables in test acc= ordingly?

=0A=0A

 

=0A=0A

Querying the pg_catalog/information_sc= hema=0Aover the two db=E2=80=99s should give you the table ddl from which y= ou can diff,=0Aand then generate the alter statements from the results.

=0A=0A

 

=0A=0A

Cheers

=0A=0A

 

=0A=0A

Martin =0A=0A

=  

=0A=0A
=0A=0A
=0A=0A

From: Bradley Holbrook=0A[mailto:operations_bradley@servilli= an.ca]
=0ASent: 18 January 2011 16:57
=0ATo: French, M= artin
=0ACc: pgsql-admin@postgresql.org
=0ASubject: RE:= [ADMIN] Postgres Backup Utility

=0A=0A
=0A=0A
=0A=0A<= p class=3D"yiv1343573220MsoNormal">  

=0A=0A

Well,=0AI can=E2=80=99t just go dropping and recreating tables=E2=80=A6 it= needs to create=0Athe correct alter statements if existing tables and or f= unctions already exist.

=0A=0A

 

=0A=0A

Secondly,=0Awhen I=E2=80=99m finished = changing the structure, I need to be able to select the=0Alist of tables th= at will have content updates.

=0A=0A

&nbs= p;

=0A=0A

Using=0Aa script might be more w= ork maintaining then it=E2=80=99s worth. I have a backup=0Autility that can= do the job, but 3 tedious steps per schema, that only work=0Aabout 10% of = the time (and no batching options so that I can create a list of=0Aactions = and run the list).

=0A=0A

  =0A=0A

 

=0A=0A
=0A=0A
=0A=0A

From: French, Martin=0A[mail= to:frenchm@cromwell.co.uk]
=0ASent: January-18-11 5:47 AM
=0A= To: Bradley Holbrook; pgsql-admin@postgresql.org
=0ASubject: RE: [ADMIN] Postgres Backup Utility

=0A=0A
=0A=0A
= =0A=0A

 

=0A=0A

I=E2=80=99m assuming that this=0Aneeds to be tightly controlled and as= such a replication tool is out of the=0Aquestion?

=0A=0A

 <= /p> =0A=0A

In that case; The first thing to=0Apop into my head here would be to use = either use shell scripting, or to use the=0Apg API and write a c program to= handle it.

=0A=0A

 

=0A=0A

I remember doing something very=0Asim= ilar with Oracle a few years back.

=0A=0A

 

=0A=0A

Cheers<= /p> =0A=0A

 

=0A=0A

Martin

=0A=0A

 

=0A=0A

 

=0A= =0A
=0A=0A
=0A=0A

From:= p= gsql-admin-owner@postgresql.org=0A[mailto:pgsql-admin-owner@postgresql.org]= On Behalf Of Bradley Holbrook
=0ASent: 18 January 2011 00= :08
=0ATo: pgsql-admin@postgresql.org
=0ASubject: [ADMI= N] Postgres Backup Utility

=0A=0A
=0A=0A
=0A=0A

 

=0A=0A

Hello= !

=0A=0A

 

=0A=0A

First=0Aday on the new mailing list as I have need of some e= xpert=E2=80=99s advice.

=0A=0A

 

=0A=0A

I=0Aneed to be able to quickly apply t= he structure updates from a development=0Adatabase to a testing database, a= nd do selective data updates (like on lookup=0Atables, but not content tabl= es).

=0A=0A

 

=0A=0A

Any=0Ahelp would be appreciated!

=0A=0A

 

=0A=0A

Brad=

=0A=0A


=0A_______________= ____________________________________
=0A
=0AThis email is intended f= or the named recipient. The information contained
=0Ain it is confident= ial. You should not copy it for any purposes, nor
=0Adisclose its conte= nts to any other party. If you received this email
=0Ain error, please = notify the sender immediately via email, and delete
=0Ait from your com= puter.
=0A
=0AAny views or opinions presented are solely those of th= e author and do not
=0Anecessarily represent those of the company.
= =0A
=0APCI Compliancy: Please note, we do not send or wish to receive ba= nking,
=0Acredit or debit card information by email or any other form o= f
=0Acommunication.
=0A
=0ACromwell Tools Limited, PO Box 14, 65= Chartwell Drive
=0AWigston, Leicester LE18 1AT. Tel 0116 2888000
= =0ARegistered in England and Wales, Reg No 00986161
=0AVAT GB 115 5713 = 87 900
=0A__________________________________________________
=0A=0A

=0A=0A =0A=0A
____________________________________________= _______ =0A
=0A
This email is intended for the named recipient. The= information contained =0A
in it is confidential. You should not copy i= t for any purposes, nor =0A
disclose its contents to any other party. I= f you received this email =0A
in error, please notify the sender immedia= tely via email, and delete=0A
it from your computer. =0A
=0A
Any= views or opinions presented are solely those of the author and do not =0A<= br>necessarily represent those of the company. =0A
=0A
PCI Complian= cy: Please note, we do not send or wish to receive banking,=0A
credit or= debit card information by email or any other form of =0A
commu= nication. =0A
= =0ACromwell Tools Limited, PO Box 14, 65 Chartwell Drive=0A
Wigston, Leice= ster LE18 1AT. Tel 0116 2888000=0A
Registered in England and Wales, Reg = No 00986161=0A
VAT GB 115 5713 87 900=0A
____________________________= ______________________=0A=0A

= =0A=0A --0-2061493445-1295464436=:14223-- From scott.marlowe@gmail.com Thu Jan 20 04:41:07 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 9053F1337B69 for ; Thu, 20 Jan 2011 00:41:07 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 92186-04 for ; Thu, 20 Jan 2011 04:41:00 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mail.postgresql.org (Postfix) with ESMTP id AC87D1337B39 for ; Thu, 20 Jan 2011 00:40:59 -0400 (AST) Received: by wwf26 with SMTP id 26so184987wwf.19 for ; Wed, 19 Jan 2011 20:40:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=m8YJNczjJJciN/kOnYB189OWuaXm040Xtn76ZkIlMkA=; b=hMsXjtmCbW40LvnkVAQ5mFOE2+o0hjLlX8tVTCVemaHH0Cjm56LVPy1UYPBIqcbHZH ke7R5+/hTOu7M0WRtFR+eBg8H4xZumuzwpQONYg/onYa/F3MJo8fYC8d/0Dycws28PDc 28APTB73m/r/SJAHeD15ybXAr2uCOvbVEdO+4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=TAyYsnxNRSIqUF1jxU2utcHQTF7F8R0kLl/mjC+vEL4YaBl8vWNMsVWO960VJEPoHp +RFz0cRkF03eOThJBxhIyFBM9Dp8eAhp+zKR96Wk/XPJsmKo220uXeRgQ+Pjwmp10oRr BPFph9KHEHAEUXc0UE6+A39QRXIRutmy4KXkk= MIME-Version: 1.0 Received: by 10.227.141.131 with SMTP id m3mr1743210wbu.125.1295498458171; Wed, 19 Jan 2011 20:40:58 -0800 (PST) Received: by 10.227.180.79 with HTTP; Wed, 19 Jan 2011 20:40:58 -0800 (PST) In-Reply-To: <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> Date: Wed, 19 Jan 2011 21:40:58 -0700 Message-ID: Subject: Re: Postgres Backup Utility From: Scott Marlowe To: "French, Martin" Cc: Bradley Holbrook , pgsql-admin@postgresql.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201101/119 X-Sequence-Number: 36559 On Wed, Jan 19, 2011 at 12:12 AM, French, Martin w= rote: > Ok, you say that you cannot drop and recreate, so you need to do this via > alter statements only? That=92s obviously going to complicate matters, as= a > straight dump, drop, recreate, restore would be the fastest and by far > simplest method. > > > > So, Ideally, you=92ll need to do a table def comparison over the two > databases, and generate the necessary sql to amend the tables in test > accordingly? Sorry but this is exactly backwards from good procedure. What you do it have your developers checkin the SQL code they used to alter the tables / add rows in the test database, and you apply that to testing / QA / staging / production. Any other way is a recipe for both disaster and DBA burnout. To make it easier, you can always turn on ddl logging on the developer's databases and then just trawl the logs for what they did. Still easier than trying to compare schemas and figure out what's different and how to write the SQL to make it happen. From frenchm@cromwell.co.uk Thu Jan 20 12:01:37 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id DA93C1337B69 for ; Thu, 20 Jan 2011 08:01:37 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 07159-07 for ; Thu, 20 Jan 2011 12:01:30 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from ariel.cromwell.co.uk (mx0.cromwell.co.uk [213.249.208.130]) by mail.postgresql.org (Postfix) with ESMTP id E0F1A1337B64 for ; Thu, 20 Jan 2011 08:01:29 -0400 (AST) Received: from roundway.Cromwell-tools.co.uk (roundway [194.216.28.40]) by ariel.cromwell.co.uk (8.12.8/8.12.8) with ESMTP id p0KC1CMU027125; Thu, 20 Jan 2011 12:01:30 GMT X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: Re: Postgres Backup Utility Date: Thu, 20 Jan 2011 12:00:47 -0000 Message-ID: <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [ADMIN] Postgres Backup Utility Thread-Index: Acu4XQ6G6RFDOE/FTCCexwO5kmPo1wAO5k/Q References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca><81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk><004e01cbb730$bdb21930$39164b90$@servillian.ca><81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> From: "French, Martin" To: "Scott Marlowe" Cc: "Bradley Holbrook" , X-Cromwell-Group-MailScanner-Information: Cromwell Group MailScanner X-Cromwell-Group-MailScanner: Clean X-Cromwell-Group-MailScanner-MCPCheck: MCP-Clean, MCP-Checker (score=0, required 10) X-Cromwell-Group-MailScanner-SpamCheck: not spam (whitelisted), SpamAssassin (not cached, score=-3, required 2, autolearn=not spam, BAYES_00 -3.00) X-Cromwell-Group-MailScanner-From: frenchm@cromwell.co.uk X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.91 tagged_above=-10 required=5 tests=BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201101/120 X-Sequence-Number: 36560 Having been a C/C++ developer many years before being a DBA, and having written ITIL software; How is migrating structure from a Development database to a test database whilst maintaining test data backwards? Besides, the OP was asking how to diff to databases and create ddl, not asking for us to comment on why he's doing it. Personally, I'd rather not go trawling through what can only be described as hundreds of thousands of lines of PostgreSQL log to find THE RIGHT DDL statements. To me that's surely a recipe for disaster, and certainly not a role I would expect to complete as a DBA. Yes the Developers should've kept the DDL for the changes, however; I know from experience that this isn't always the case, and have found it necessary to "Diff" databases on a regular basis for one reason or another.=20 Each time I've done something like this I have resulted to coding shell scripts, or C programs that are bespoke to accomplish this, as I've yet to find a tool that does this to my satisfaction levels. -----Original Message----- From: Scott Marlowe [mailto:scott.marlowe@gmail.com]=20 Sent: 20 January 2011 04:41 To: French, Martin Cc: Bradley Holbrook; pgsql-admin@postgresql.org Subject: Re: [ADMIN] Postgres Backup Utility On Wed, Jan 19, 2011 at 12:12 AM, French, Martin wrote: > Ok, you say that you cannot drop and recreate, so you need to do this via > alter statements only? That's obviously going to complicate matters, as a > straight dump, drop, recreate, restore would be the fastest and by far > simplest method. > > > > So, Ideally, you'll need to do a table def comparison over the two > databases, and generate the necessary sql to amend the tables in test > accordingly? Sorry but this is exactly backwards from good procedure. What you do it have your developers checkin the SQL code they used to alter the tables / add rows in the test database, and you apply that to testing / QA / staging / production. Any other way is a recipe for both disaster and DBA burnout. To make it easier, you can always turn on ddl logging on the developer's databases and then just trawl the logs for what they did. Still easier than trying to compare schemas and figure out what's different and how to write the SQL to make it happen. ___________________________________________________=20 =20=20 This email is intended for the named recipient. The information contained= =20 in it is confidential. You should not copy it for any purposes, nor=20 disclose its contents to any other party. If you received this email=20 in error, please notify the sender immediately via email, and delete it from your computer.=20 =20=20 Any views or opinions presented are solely those of the author and do not= =20 necessarily represent those of the company.=20 =20=20 PCI Compliancy: Please note, we do not send or wish to receive banking, cre= dit or debit card information by email or any other form of communication.=20= =20=20=20=20=20=20=20=20 Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive Wigston, Leicester LE18 1AT. Tel 0116 2888000 Registered in England and Wales, Reg No 00986161 VAT GB 115 5713 87 900 __________________________________________________ From scott.marlowe@gmail.com Thu Jan 20 15:40:33 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 8A0471337B34 for ; Thu, 20 Jan 2011 11:40:33 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 97918-05 for ; Thu, 20 Jan 2011 15:40:26 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wy0-f174.google.com (mail-wy0-f174.google.com [74.125.82.174]) by mail.postgresql.org (Postfix) with ESMTP id EA3901337B7E for ; Thu, 20 Jan 2011 11:40:25 -0400 (AST) Received: by wyb28 with SMTP id 28so747730wyb.19 for ; Thu, 20 Jan 2011 07:40:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=OuTgPkFypgBOVyl3J9TNGZ0ayv2VaDQvEjwOm9nI2GU=; b=HfRBOwR7YbnpDP1F5Qz5jykmDDWKpaCRrpIaMrdCyStkWb1y49HZPQos8wLrW4x/RQ 7YfkHvWZolalHizVSuS1Df8oUAW80NfRoyr3i38St/LCCPls+VepE1CK4a5rj17Nr8Dw iQEHUwwLT4wqV4gk1hKBmWDBm9CaG196JgVQE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=xgj8md+vfI7yscu/mZO3NlAjxoMLBjzM+5DSNs0cE/kGJNDRfULrkkoZXLjIaCJKUx NYAuy7NVk8fjL6H+XBGa6WB+PPL9xjKw+r4JJpoMczAJ8/GaXNkFpZxV2CMrxesWpih0 lNriPTlBEBmwG31uMbJydL4NdSIkvCIVmvrSc= MIME-Version: 1.0 Received: by 10.227.132.143 with SMTP id b15mr2514230wbt.9.1295538025637; Thu, 20 Jan 2011 07:40:25 -0800 (PST) Received: by 10.227.180.79 with HTTP; Thu, 20 Jan 2011 07:40:25 -0800 (PST) In-Reply-To: <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> Date: Thu, 20 Jan 2011 08:40:25 -0700 Message-ID: Subject: Re: Postgres Backup Utility From: Scott Marlowe To: "French, Martin" Cc: Bradley Holbrook , pgsql-admin@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201101/121 X-Sequence-Number: 36561 On Thu, Jan 20, 2011 at 5:00 AM, French, Martin wrote: > Having been a C/C++ developer many years before being a DBA, and having > written ITIL software; How is migrating structure from a Development > database to a test database whilst maintaining test data backwards? It's not. doing by running diffs on ddl / schema is. This is what the OP asked for: "I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on lookup tables, but not content tables)." > > Besides, the OP was asking how to diff to databases and create ddl, not > asking for us to comment on why he's doing it. Personally, I'd rather Re-read what he asked for. He did not ask how to run diffs of schema, he asked how to update it with changes from a test database. The best way to do this is to check in changes to the database and lookup tables so the DBA can then apply them when needed to the various other databases. But hey, feel free to answer a question the OP didn't ask with an answer he doesn't need. From scott.marlowe@gmail.com Thu Jan 20 15:46:14 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id F29011337BCA for ; Thu, 20 Jan 2011 11:46:14 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 00307-06-4 for ; Thu, 20 Jan 2011 15:46:08 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mail.postgresql.org (Postfix) with ESMTP id D346C1337BD6 for ; Thu, 20 Jan 2011 11:46:04 -0400 (AST) Received: by wwi17 with SMTP id 17so1626951wwi.1 for ; Thu, 20 Jan 2011 07:46:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=qiAh16I7ThPdryCokaqwfFygXRBnrrSR6SCsq56qhzE=; b=rPMkesodByGDicUcbF52BCk487WpMsAvSLVpE2Gchlqjv4Fsp+yW7e+GVzMzt31Uxu 9appXARAZ9TG+xIbbpgQ62VPZVgUrOg2JYhxjcVidXvF62E97Y+cCijPOfOzNmXgQT/H NLYZUq6ERA3kdBK0K0/V3eq8zNwovsy1bsOHw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=l2rOeabFF8R8xCg3qLCInEDG9YJGIgzc7EDdtKoI7zgK0fQWsNRDjnk8WK9X2IB9l0 Woa7k6eE1J4674pS3nc4WWvNIBKCn8+49q1Cn0X10SUlmaog00AR3Aa15I6eIeWIxKCJ Z2tQ7eJKhDWRVPa7yFN5101GOxTXgYfte4rsU= MIME-Version: 1.0 Received: by 10.227.180.134 with SMTP id bu6mr2454934wbb.183.1295538364391; Thu, 20 Jan 2011 07:46:04 -0800 (PST) Received: by 10.227.180.79 with HTTP; Thu, 20 Jan 2011 07:46:04 -0800 (PST) In-Reply-To: <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> Date: Thu, 20 Jan 2011 08:46:04 -0700 Message-ID: Subject: Re: Postgres Backup Utility From: Scott Marlowe To: "French, Martin" Cc: Bradley Holbrook , pgsql-admin@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201101/122 X-Sequence-Number: 36562 On Thu, Jan 20, 2011 at 5:00 AM, French, Martin wrote: > Personally, I'd rather > not go trawling through what can only be described as hundreds of > thousands of lines of PostgreSQL log to find THE RIGHT DDL statements. Oh that's easy. Grep out the statements that start with alter etc, ptu them all in a big email, send them to all the developers and tell them to let you know which ones need applying to QA / Staging / etc. Can't get your DDL to the DBA? Your code won't work. You break the build because of that, you get to fix it. From scott.marlowe@gmail.com Thu Jan 20 16:21:21 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 8B8C613364DE for ; Thu, 20 Jan 2011 12:21:21 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 99860-07 for ; Thu, 20 Jan 2011 16:21:14 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wy0-f174.google.com (mail-wy0-f174.google.com [74.125.82.174]) by mail.postgresql.org (Postfix) with ESMTP id 9789A1337B84 for ; Thu, 20 Jan 2011 12:21:13 -0400 (AST) Received: by wyb28 with SMTP id 28so788003wyb.19 for ; Thu, 20 Jan 2011 08:21:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=m7vRbILOAqSzSZ0a5ArNF8LHRxrvW4rnOMJFbGXfoEQ=; b=WrVU7XTB/4+0iAPo4KcOja0NXGyd3UcJYZ5cVRHXbpVSAV9brnaGibgpU1rivDarhK iNwf/LKhpb7QeY9zo9wbKl+YCI8RnEDYx9YH9MVyNAEs4RGt74doDJ39MWtfHRwzHUDk MXyEkmZRDn0BjfvaITERd0gHhHqN8oTz7qVU8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=w62LAuZi336C2cJGFKAM8FyiK3GrF1u1TRNCxuHAT9hZN0u+Mtu+IagqYVFZo2RD9H iEUMQZuVbQt5Gysfph1r6ZdE1A3f8YjFTpbnb0WgUUZ5zm1ZHCwCO0j9ULj/rp2NcLfx brsmn+wZEg/wetKN5cZYm+2En2nHRKjkaxRjY= MIME-Version: 1.0 Received: by 10.227.157.73 with SMTP id a9mr2561182wbx.154.1295540470013; Thu, 20 Jan 2011 08:21:10 -0800 (PST) Received: by 10.227.180.79 with HTTP; Thu, 20 Jan 2011 08:21:09 -0800 (PST) In-Reply-To: References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> Date: Thu, 20 Jan 2011 09:21:09 -0700 Message-ID: Subject: Re: Postgres Backup Utility From: Scott Marlowe To: "French, Martin" Cc: Bradley Holbrook , pgsql-admin@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201101/123 X-Sequence-Number: 36563 On Thu, Jan 20, 2011 at 8:46 AM, Scott Marlowe wr= ote: > On Thu, Jan 20, 2011 at 5:00 AM, French, Martin = wrote: > >> =A0Personally, I'd rather >> not go trawling through what can only be described as hundreds of >> thousands of lines of PostgreSQL log to find THE RIGHT DDL statements. > > Oh that's easy. =A0Grep out the statements that start with alter etc, > ptu them all in a big email, send them to all the developers and tell > them to let you know which ones need applying to QA / Staging / etc. > Can't get your DDL to the DBA? =A0Your code won't work. =A0You break the > build because of that, you get to fix it. Note that I also mentioned turning on logging JUST ddl statements. Assumiing you're not logging EVERYTHING in development (and why would you) you should only have a few dozen lines of log to grep, and little or none of it will be things other than DDL. --=20 To understand recursion, one must first understand recursion. From operations_bradley@servillian.ca Thu Jan 20 17:42:15 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id D63DB1337B34 for ; Thu, 20 Jan 2011 13:42:15 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 87072-07 for ; Thu, 20 Jan 2011 17:42:08 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail.shawcable.com (shawmail.shawcable.com [64.59.128.220]) by mail.postgresql.org (Postfix) with ESMTP id 2843213364DE for ; Thu, 20 Jan 2011 13:42:07 -0400 (AST) Received: from bpd2mi2no-svcs.prod.shawcable.com ([10.0.184.121]) by bpd2mo2no-ssvc.prod.shawcable.com with ESMTP; 20 Jan 2011 10:42:06 -0700 X-Cloudmark-SP-Filtered: true X-Cloudmark-SP-Result: v=1.1 cv=l0hTUOwXBBZC3xuFbE1ochJnP84ph5EBJB8ZSEz2E7A= c=1 sm=1 a=fKHZdZmfl1IA:10 a=BLceEmwcHowA:10 a=kj9zAlcOel0A:10 a=rFBdEGy9vS/m0A3AhxL8nA==:17 a=pGLkceISAAAA:8 a=epTmVMiNAAAA:8 a=GbhIXFikAAAA:8 a=C9ZUX6UaOPOE5U8F8JQA:9 a=sOC0X2tqlKw-NScL1lQA:7 a=Re-rJKmNGC4m7dJmJtoi4r88znMA:4 a=CjuIK1q_8ugA:10 a=MSl-tDqOz04A:10 a=YObXiYvaVLwA:10 a=YsdO3FDBEMUA:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 X-IronPort-AV: E=Sophos;i="4.60,352,1291618800"; d="scan'208";a="394136498" Received: from unknown (HELO bpd2mi2no-cmts.prod.shawcable.com) ([192.168.183.121]) by bpd2mi2no-cmts.prod.shawcable.com with ESMTP; 20 Jan 2011 10:42:05 -0700 X-reinject: true Received: from unknown (HELO BradleyHolbrook) ([24.67.39.150]) by bpd2mi2no-dmz.prod.shawcable.com with ESMTP; 20 Jan 2011 10:42:05 -0700 From: "Bradley Holbrook" To: "'Scott Marlowe'" , "'French, Martin'" Cc: References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> In-Reply-To: Subject: Re: Postgres Backup Utility Date: Thu, 20 Jan 2011 09:42:02 -0800 Message-ID: <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHHkvKLrosR4aSyAm2SjCI6Zq933AO9xqP7Akk6VIIDAixhoAH4cdVIAfRS2jAC6GVMfpNjcQpw Content-Language: en-ca X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201101/126 X-Sequence-Number: 36566 Thanks Scott... a couple comments. Our developers never decide what goes to where... they just happily plumb away on the development db until we're ready to take our product to testing (at regular intervals), once QA is passed, we wish to apply these to live. We have several diff tools and sync tools, but they take forever (especially the ones that only go one schema at a time). The DDL Logging sounds like a sufficient solution, can it be configured to only record create and alter commands (or create or replace commands on functions or updates on sequences, etc)? I'd likely write a script to have this emailed to me at the end of every day. I'm going to google DDL logging (never heard of it), but any good resources off the top of your head? Martin French is right though, ask your developers to write down all their SQL struct changes and they look at you funny... and being a developer myself I'd look at me funny. If you forget just once you're screwed into a day sifting through tables and code. -----Original Message----- From: Scott Marlowe [mailto:scott.marlowe@gmail.com] Sent: January-20-11 7:40 AM To: French, Martin Cc: Bradley Holbrook; pgsql-admin@postgresql.org Subject: Re: [ADMIN] Postgres Backup Utility On Thu, Jan 20, 2011 at 5:00 AM, French, Martin wrote: > Having been a C/C++ developer many years before being a DBA, and > having written ITIL software; How is migrating structure from a > Development database to a test database whilst maintaining test data backwards? It's not. doing by running diffs on ddl / schema is. This is what the OP asked for: "I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on lookup tables, but not content tables)." > > Besides, the OP was asking how to diff to databases and create ddl, > not asking for us to comment on why he's doing it. Personally, I'd > rather Re-read what he asked for. He did not ask how to run diffs of schema, he asked how to update it with changes from a test database. The best way to do this is to check in changes to the database and lookup tables so the DBA can then apply them when needed to the various other databases. But hey, feel free to answer a question the OP didn't ask with an answer he doesn't need. From scott.marlowe@gmail.com Thu Jan 20 17:51:17 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 3C44E1337B34 for ; Thu, 20 Jan 2011 13:51:17 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 55917-06 for ; Thu, 20 Jan 2011 17:51:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mail.postgresql.org (Postfix) with ESMTP id 0829713364DE for ; Thu, 20 Jan 2011 13:51:08 -0400 (AST) Received: by wwf26 with SMTP id 26so892224wwf.19 for ; Thu, 20 Jan 2011 09:51:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=yb0AC/k3O4Kib6pTelL5W3ZrWeMbtaVA+yHQK/ynJL8=; b=YsA66wlO9jsJ6zEqeEq8xM97jsGIYOyoRmxjttZ+O8bvTJ+eRIoTxAKemrQh/mV9+5 mwjl4WBnXwRWhZUQ9bWYpBwbY69/OoyavXUUvW7ISjXG9nvePnWYdl3anAgBdqX0qPOu wLL0nSLlfyapaZxffwetEpCL2S2QKIFw5Om88= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=AvKFFpDeiGTNvMw8X0kEoqfwrbxvbfp/luWbREYiVv+xkWpvc+OU1khnmg4DN5+3ae oWqDwifl5v7j2Z24eVh/8N93eiF0X5JBylRUcAQFe8mUQSE09jlfyXUue6Ui1Ea4ZE5H 9v1LTXHeuvRHyMu8uYpy58Cwx+98D3TIODhN0= MIME-Version: 1.0 Received: by 10.227.180.134 with SMTP id bu6mr2633235wbb.183.1295545867570; Thu, 20 Jan 2011 09:51:07 -0800 (PST) Received: by 10.227.180.79 with HTTP; Thu, 20 Jan 2011 09:51:06 -0800 (PST) In-Reply-To: <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> Date: Thu, 20 Jan 2011 10:51:06 -0700 Message-ID: Subject: Re: Postgres Backup Utility From: Scott Marlowe To: Bradley Holbrook Cc: "French, Martin" , pgsql-admin@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201101/127 X-Sequence-Number: 36567 On Thu, Jan 20, 2011 at 10:42 AM, Bradley Holbrook wrote: > Thanks Scott... a couple comments. > > Our developers never decide what goes to where... they just happily plumb > away on the development db until we're ready to take our product to testing > (at regular intervals), once QA is passed, we wish to apply these to live. > We have several diff tools and sync tools, but they take forever (especially > the ones that only go one schema at a time). > > The DDL Logging sounds like a sufficient solution, can it be configured to > only record create and alter commands (or create or replace commands on > functions or updates on sequences, etc)? I'd likely write a script to have > this emailed to me at the end of every day. I'm going to google DDL logging > (never heard of it), but any good resources off the top of your head? It's basically logging anything that changes the structure of the database. It would be easy enough to grep out what you do and don't want later. > Martin French is right though, ask your developers to write down all their > SQL struct changes and they look at you funny... and being a developer > myself I'd look at me funny. If you forget just once you're screwed into a > day sifting through tables and code. I've worked in three different shops now as a dev-dba and sysadmin, and in all three, all DDL changes had to be committed and / or handed over to the DBAs. period. Look funny all they want, they either give up the DDL or their code doesn't get pushed off dev servers onto anything else. At the very least they should be able to tell you which tables changed to go with which code changes, or you're not sure what code you can and can't push. I get both of your point on this, but it's a discipline issue that needs sorting out with the developers if you want to have reproduceable ddl changes in all your systems that match the code changes. From scott.marlowe@gmail.com Thu Jan 20 17:52:45 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 2F5081337B64 for ; Thu, 20 Jan 2011 13:52:45 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 26457-03 for ; Thu, 20 Jan 2011 17:52:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wy0-f174.google.com (mail-wy0-f174.google.com [74.125.82.174]) by mail.postgresql.org (Postfix) with ESMTP id 5B3D91337B34 for ; Thu, 20 Jan 2011 13:52:37 -0400 (AST) Received: by wyb28 with SMTP id 28so878072wyb.19 for ; Thu, 20 Jan 2011 09:52:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=kx2E/4PzYeze07IF1s8qg1Xoo7Hq630LAi0AH3LQ2go=; b=ujWDVlVuix9/inMoYVVKSO1RRmJ66aeoVC2papLkU5cBwUrCl0wykk8RO4yU48FBxr omJgvA24ZMXX1jD6xY4QebxWm2Qeq9D647CM/5i7DVDB3jVw27aLDLZBmEXl4lm9tPSG sg6DjUwvrXeuoKp08NgP8HG7vfJV+xkRob5sc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Wh6ojvIga0RkkNGgcNGl1WJdmWYWKOFlHoh2L1LQ3MD2hwdAZ3l2gdqf1dDK5YssgO EK/nQRefMqKJ5xp3kYlzXoZHPpaqEsIyxpSQyk/omEXxm9Om2yGVbt3nub7NKL+CNqZ0 J+73x960qKyCyvvzMkLbIuLYZ42AoTjUbDCpI= MIME-Version: 1.0 Received: by 10.227.141.131 with SMTP id m3mr2685052wbu.125.1295545956150; Thu, 20 Jan 2011 09:52:36 -0800 (PST) Received: by 10.227.180.79 with HTTP; Thu, 20 Jan 2011 09:52:36 -0800 (PST) In-Reply-To: References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> Date: Thu, 20 Jan 2011 10:52:36 -0700 Message-ID: Subject: Re: Postgres Backup Utility From: Scott Marlowe To: Bradley Holbrook Cc: "French, Martin" , pgsql-admin@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201101/128 X-Sequence-Number: 36568 Followup, note that you can set the log_statement='ddl' for an entire pg cluster, for a single database, or for a single user, if that helps. logging ddl does not log dml, or data changes, just structural changes. From scott.marlowe@gmail.com Thu Jan 20 18:49:39 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id BEA3013364DE for ; Thu, 20 Jan 2011 14:49:39 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 33357-04 for ; Thu, 20 Jan 2011 18:49:32 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mail.postgresql.org (Postfix) with ESMTP id B40861337B34 for ; Thu, 20 Jan 2011 14:49:31 -0400 (AST) Received: by wwf26 with SMTP id 26so949974wwf.19 for ; Thu, 20 Jan 2011 10:49:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=9xwP4jCjP+R/icGfQGU0O0KYgeX4pn1wcmxiPXWpa3s=; b=KS4ODGzg8KILWcwH1SGBminWgRQZQZQwolHds/pRQcWragIpQ6LaXB5zAfLf5hxrBs UakGH1y+GYiKC5mwlwtikP17DJW1utWb52WgcB0o+8xLWjGGuaIrGl/9IQg4WCdlUx+o PQ9RTgjtozkENKMiGabFJT0fYQocmeyeI5yXg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=N7wRtzKhGTj1OGEb4obz3HuavmJdtBL4Qhrr7fjnWXnv3JyGi9NDEVeuMcqCq0L8bu xDDeRiBIB697C1/inVK0STfxKMOTO5ZRq32r8k+Mmm+QUSHNVcTT8Jk5nuE5rQKTRsN5 wVmpvEbEBOxR7513Iy2bZBwiDXz9bqwaplHK0= MIME-Version: 1.0 Received: by 10.227.156.207 with SMTP id y15mr2798382wbw.38.1295549370244; Thu, 20 Jan 2011 10:49:30 -0800 (PST) Received: by 10.227.180.79 with HTTP; Thu, 20 Jan 2011 10:49:26 -0800 (PST) In-Reply-To: References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> Date: Thu, 20 Jan 2011 11:49:26 -0700 Message-ID: Subject: Re: Postgres Backup Utility From: Scott Marlowe To: Bradley Holbrook Cc: "French, Martin" , pgsql-admin@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201101/130 X-Sequence-Number: 36570 On Thu, Jan 20, 2011 at 10:52 AM, Scott Marlowe w= rote: > Followup, note that you can set the log_statement=3D'ddl' for an entire > pg cluster, for a single database, or for a single user, if that > helps. =A0logging ddl does not log dml, or data changes, just structural > changes. also also wik, you can use the log_line_prefix to make it so that you can grep out what you need easier too. #log_line_prefix =3D '' # special values: # %u =3D user name # %d =3D database name # %r =3D remote host and port # %h =3D remote host # %p =3D process ID # %t =3D timestamp without millis= econds # %m =3D timestamp with milliseco= nds # %i =3D command tag # %c =3D session ID # %l =3D session line number # %s =3D session start timestamp # %v =3D virtual transaction ID # %x =3D transaction ID (0 if non= e) # %q =3D stop here in non-session # processes # %% =3D '%' # e.g. '<%u%%%d> ' From ineyman@perceptron.com Thu Jan 20 21:19:08 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 363A11337BDA for ; Thu, 20 Jan 2011 17:19:08 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 85434-02-9 for ; Thu, 20 Jan 2011 21:19:00 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from EXCHANGE.perceptron.com (exchange.perceptron.com [68.22.13.203]) by mail.postgresql.org (Postfix) with ESMTP id 67B811337C6F for ; Thu, 20 Jan 2011 17:18:00 -0400 (AST) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: Re: Postgres Backup Utility Date: Thu, 20 Jan 2011 16:17:58 -0500 Message-ID: In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Postgres Backup Utility Thread-Index: Acu454MrnyvOp9c3RFu7ZLOlJx8U0g== References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca><81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk><004e01cbb730$bdb21930$39164b90$@servillian.ca><81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk><81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk><01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> From: "Igor Neyman" To: "Scott Marlowe" , "Bradley Holbrook" Cc: "French, Martin" , X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.91 tagged_above=-10 required=5 tests=BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201101/131 X-Sequence-Number: 36571 =20 > -----Original Message----- > From: Scott Marlowe [mailto:scott.marlowe@gmail.com]=20 > Sent: Thursday, January 20, 2011 12:51 PM > To: Bradley Holbrook > Cc: French, Martin; pgsql-admin@postgresql.org > Subject: Re: Postgres Backup Utility >=20 > On Thu, Jan 20, 2011 at 10:42 AM, Bradley Holbrook=20 > wrote: > > Thanks Scott... a couple comments. > > > > Our developers never decide what goes to where... they just happily=20 > > plumb away on the development db until we're ready to take=20 > our product=20 > > to testing (at regular intervals), once QA is passed, we=20 > wish to apply these to live. > > We have several diff tools and sync tools, but they take forever=20 > > (especially the ones that only go one schema at a time). > > > > The DDL Logging sounds like a sufficient solution, can it be=20 > > configured to only record create and alter commands (or create or=20 > > replace commands on functions or updates on sequences, etc)? I'd=20 > > likely write a script to have this emailed to me at the end=20 > of every=20 > > day. I'm going to google DDL logging (never heard of it),=20 > but any good resources off the top of your head? >=20 > It's basically logging anything that changes the structure of=20 > the database. It would be easy enough to grep out what you=20 > do and don't want later. >=20 > > Martin French is right though, ask your developers to write=20 > down all=20 > > their SQL struct changes and they look at you funny... and being a=20 > > developer myself I'd look at me funny. If you forget just=20 > once you're=20 > > screwed into a day sifting through tables and code. >=20 > I've worked in three different shops now as a dev-dba and=20 > sysadmin, and in all three, all DDL changes had to be=20 > committed and / or handed over to the DBAs. period. Look=20 > funny all they want, they either give up the DDL or their=20 > code doesn't get pushed off dev servers onto anything else. =20 > At the very least they should be able to tell you which=20 > tables changed to go with which code changes, or you're not=20 > sure what code you can and can't push. I get both of your=20 > point on this, but it's a discipline issue that needs sorting=20 > out with the developers if you want to have reproduceable ddl=20 > changes in all your systems that match the code changes. >=20 Completely agree with Scott. Only want to add that in this kind of development environment: development/test/production -=20 Source code versioning software is absolute necessity (there are many: CSV, SourceSafe, Perforce, ... - pick your choice). Regards, Igor Neyman From craig_james@emolecules.com Thu Jan 20 22:05:26 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id BF37F1337B93 for ; Thu, 20 Jan 2011 18:05:26 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 96870-08-2 for ; Thu, 20 Jan 2011 22:05:20 +0000 (UTC) X-Greylist: delayed 00:06:10.880401 by SQLgrey-1.7.6 Received: from mail-bw0-f46.google.com (mail-bw0-f46.google.com [209.85.214.46]) by mail.postgresql.org (Postfix) with ESMTP id 653B61337B94 for ; Thu, 20 Jan 2011 18:05:18 -0400 (AST) Received: by bwz15 with SMTP id 15so994671bwz.19 for ; Thu, 20 Jan 2011 14:05:17 -0800 (PST) Received: by 10.204.57.204 with SMTP id d12mr2423522bkh.69.1295560743956; Thu, 20 Jan 2011 13:59:03 -0800 (PST) Received: from Craig-Jamess-MacBook-Pro.local (wsip-70-167-124-126.sd.sd.cox.net [70.167.124.126]) by mx.google.com with ESMTPS id j11sm4224665bka.0.2011.01.20.13.59.01 (version=SSLv3 cipher=RC4-MD5); Thu, 20 Jan 2011 13:59:02 -0800 (PST) Message-ID: <4D38B023.20008@emolecules.com> Date: Thu, 20 Jan 2011 13:58:59 -0800 From: Craig James User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: pgsql-admin@postgresql.org Subject: Re: Postgres Backup Utility References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca><81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk><004e01cbb730$bdb21930$39164b90$@servillian.ca><81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk><81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk><01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.6 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201101/132 X-Sequence-Number: 36572 >> On Thu, Jan 20, 2011 at 10:42 AM, Bradley Holbrook >> wrote: >>> Thanks Scott... a couple comments. >>> >>> Our developers never decide what goes to where... they just happily >>> plumb away on the development db until we're ready to take >> our product >>> to testing (at regular intervals), once QA is passed, we >> wish to apply these to live. >>> We have several diff tools and sync tools, but they take forever >>> (especially the ones that only go one schema at a time). >>> >>> The DDL Logging sounds like a sufficient solution, can it be >>> configured to only record create and alter commands (or create or >>> replace commands on functions or updates on sequences, etc)? I'd >>> likely write a script to have this emailed to me at the end >> of every >>> day. I'm going to google DDL logging (never heard of it), >> but any good resources off the top of your head? >> >> It's basically logging anything that changes the structure of >> the database. It would be easy enough to grep out what you >> do and don't want later. >> >>> Martin French is right though, ask your developers to write >> down all >>> their SQL struct changes and they look at you funny... and being a >>> developer myself I'd look at me funny. If you forget just >> once you're >>> screwed into a day sifting through tables and code. >> >> I've worked in three different shops now as a dev-dba and >> sysadmin, and in all three, all DDL changes had to be >> committed and / or handed over to the DBAs. period. Look >> funny all they want, they either give up the DDL or their >> code doesn't get pushed off dev servers onto anything else. >> At the very least they should be able to tell you which >> tables changed to go with which code changes, or you're not >> sure what code you can and can't push. I get both of your >> point on this, but it's a discipline issue that needs sorting >> out with the developers if you want to have reproduceable ddl >> changes in all your systems that match the code changes. >> > > Completely agree with Scott. Me too. The idea that a developer can change the schema without telling anyone is laughable. The idea that someone else has to reverse engineer schema changes is ludicrous. We have a strict procedure that every schema change has to be accompanied by a script that applies the changes. It has to be checked into Subversion along with all the other code. Make a new rule: Developers have to document schema changes. Institute a zero-tolerance policy for omissions. If a developer can create the DDL to make the change, surely it's not too much trouble to spend another minute adding it to a file and checking that file in. Craig From desoi@pgedit.com Fri Jan 21 01:08:54 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 772B61337990 for ; Thu, 20 Jan 2011 21:08:54 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 53736-07 for ; Fri, 21 Jan 2011 01:08:47 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from a2s36.a2hosting.com (a2s36.a2hosting.com [216.119.134.130]) by mail.postgresql.org (Postfix) with ESMTP id 4DAFA13364DE for ; Thu, 20 Jan 2011 21:08:47 -0400 (AST) Received: from adsl-70-99-30.tys.bellsouth.net ([98.70.99.30] helo=[10.0.0.18]) by a2s36.a2hosting.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) (envelope-from ) id 1Pg5UZ-0002y1-DT; Thu, 20 Jan 2011 20:08:43 -0500 Subject: Re: Postgres Backup Utility Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: John DeSoi In-Reply-To: <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> Date: Thu, 20 Jan 2011 20:08:42 -0500 Cc: "'Scott Marlowe'" , "'French, Martin'" , Content-Transfer-Encoding: quoted-printable Message-Id: References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> To: Bradley Holbrook X-Mailer: Apple Mail (2.1082) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - a2s36.a2hosting.com X-AntiAbuse: Original Domain - postgresql.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - pgedit.com X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9 X-Spam-Level: X-Archive-Number: 201101/134 X-Sequence-Number: 36574 On Jan 20, 2011, at 12:42 PM, Bradley Holbrook wrote: > Our developers never decide what goes to where... they just happily = plumb > away on the development db until we're ready to take our product to = testing > (at regular intervals), once QA is passed, we wish to apply these to = live. > We have several diff tools and sync tools, but they take forever = (especially > the ones that only go one schema at a time). Using a diff tool, how would you know if a column was dropped and = another one added versus simply renaming a column? Won't that kind of = difference matter when you want to apply the changes to your production = database? John DeSoi, Ph.D. From scott.marlowe@gmail.com Fri Jan 21 01:44:52 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 823BA1337990 for ; Thu, 20 Jan 2011 21:44:52 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 98542-02 for ; Fri, 21 Jan 2011 01:44:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wy0-f174.google.com (mail-wy0-f174.google.com [74.125.82.174]) by mail.postgresql.org (Postfix) with ESMTP id 904DC13364DE for ; Thu, 20 Jan 2011 21:44:44 -0400 (AST) Received: by wyb28 with SMTP id 28so1311626wyb.19 for ; Thu, 20 Jan 2011 17:44:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=jsIv1PBjXZgN/7KuzV17SyzfwPp7cwTQ+3byoARdwt0=; b=ZNqY2J3H7YbJIU/mrRAN5hAcXSJK0l2rd5DAqgtdTTwxRnX85G+o2/dufDq+AH2U4R +kFVUt3+m3qQZuSpm/4spuV7y39NftJyHsvkfAHfG+KLmmQjAAHcmwa2/OwHH+U7yZwi cSmyVLivRn7M1TdOhm6R1UJZ5LHFQ5PyHS6ok= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=GeXQjkiKUN7yf8QkD2ML7nwcmmbPDPw6jsSRpeNwJGebPK5ANr3fO1MxDb4jicnDSm zbhJVxRcwkHTPmPDUV4a62nH3LJngcMWN5fVE+DWOQEF7xVK25idq+UTfbzkwnmOsb01 1fgVUK480KiCuO0Wjh9iYxgrXpe2xGyKOL/jI= MIME-Version: 1.0 Received: by 10.227.156.207 with SMTP id y15mr39529wbw.38.1295574284042; Thu, 20 Jan 2011 17:44:44 -0800 (PST) Received: by 10.227.180.79 with HTTP; Thu, 20 Jan 2011 17:44:43 -0800 (PST) In-Reply-To: References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> Date: Thu, 20 Jan 2011 18:44:43 -0700 Message-ID: Subject: Re: Postgres Backup Utility From: Scott Marlowe To: John DeSoi Cc: Bradley Holbrook , "French, Martin" , pgsql-admin@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201101/135 X-Sequence-Number: 36575 On Thu, Jan 20, 2011 at 6:08 PM, John DeSoi wrote: > > On Jan 20, 2011, at 12:42 PM, Bradley Holbrook wrote: > >> Our developers never decide what goes to where... they just happily plumb >> away on the development db until we're ready to take our product to testing >> (at regular intervals), once QA is passed, we wish to apply these to live. >> We have several diff tools and sync tools, but they take forever (especially >> the ones that only go one schema at a time). > > Using a diff tool, how would you know if a column was dropped and another one added versus simply renaming a column? Won't that kind of difference matter when you want to apply the changes to your production database? Yikes! That could be bad. Drop column xyz. Hope that data wasn't important. To see if the column was renamed or something you'd have to examine its rows to see if they matched a previous version. I think that at the least I'd trawl the logs with ddl logging turned on to get changes. But seriously, I'd just lay down the law. You want changes in the dbs outside development, you give me, the DBA, your DDL statements. No statements, no code push. From scott_ribe@elevated-dev.com Sat Jan 22 18:22:55 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id B03DA1337B4F for ; Sat, 22 Jan 2011 14:22:55 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 76756-10 for ; Sat, 22 Jan 2011 18:22:48 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail8.sea5.speakeasy.net (mail8.sea5.speakeasy.net [69.17.117.53]) by mail.postgresql.org (Postfix) with ESMTP id 310D01337B47 for ; Sat, 22 Jan 2011 14:22:47 -0400 (AST) Received: (qmail 9563 invoked from network); 22 Jan 2011 18:22:46 -0000 Received: from 25-76-42-72.skybeam.com (HELO [192.168.2.16]) (scott_ribe@killerbytes.com@[72.42.76.25]) (envelope-sender ) by mail8.sea5.speakeasy.net (qmail-ldap-1.03) with AES128-SHA encrypted SMTP for ; 22 Jan 2011 18:22:46 -0000 Subject: Re: Postgres Backup Utility Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Scott Ribe In-Reply-To: <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> Date: Sat, 22 Jan 2011 11:22:45 -0700 Cc: "'Scott Marlowe'" , "'French, Martin'" , Content-Transfer-Encoding: quoted-printable Message-Id: References: <036101cbb6a3$baa178e0$2fe46aa0$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640812A@roundway.Cromwell-tools.co.uk> <004e01cbb730$bdb21930$39164b90$@servillian.ca> <81976671721DF04B9DCA6ECD87941A402640863E@roundway.Cromwell-tools.co.uk> <81976671721DF04B9DCA6ECD87941A40264B2BB5@roundway.Cromwell-tools.co.uk> <01af01cbb8c9$58eaa0e0$0abfe2a0$@servillian.ca> To: Bradley Holbrook X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9 X-Spam-Level: X-Archive-Number: 201101/150 X-Sequence-Number: 36590 On Jan 20, 2011, at 10:42 AM, Bradley Holbrook wrote: > Martin French is right though, ask your developers to write down all = their > SQL struct changes and they look at you funny... and being a developer > myself I'd look at me funny. Well, it's what I do and it's trivial. Just don't type DDL directly into = a psql session, edit it in your favorite text editor, in a document = called updates-pending.sql, and copy & paste into psql. Sometimes a tiny = bit of *process* can actually help productivity ;-) --=20 Scott Ribe scott_ribe@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice