X-Original-To: pgsql-sql-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 5D7D43A4300 for ; Wed, 10 Nov 2004 19:32:57 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 76461-10 for ; Wed, 10 Nov 2004 19:32:53 +0000 (GMT) Received: from tigger.fuhr.org (tigger.fuhr.org [63.214.45.158]) by svr1.postgresql.org (Postfix) with ESMTP id B25733A4187 for ; Wed, 10 Nov 2004 19:32:53 +0000 (GMT) Received: from winnie.fuhr.org (winnie.fuhr.org [10.1.0.1]) by tigger.fuhr.org (8.13.1/8.13.1) with ESMTP id iAAJWo8l007409 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Wed, 10 Nov 2004 12:32:53 -0700 (MST) (envelope-from mfuhr@winnie.fuhr.org) Received: from winnie.fuhr.org (localhost [127.0.0.1]) by winnie.fuhr.org (8.13.1/8.13.1) with ESMTP id iAAJWovO018372 for ; Wed, 10 Nov 2004 12:32:50 -0700 (MST) (envelope-from mfuhr@winnie.fuhr.org) Received: (from mfuhr@localhost) by winnie.fuhr.org (8.13.1/8.13.1/Submit) id iAAJWoHB018371 for pgsql-sql@postgresql.org; Wed, 10 Nov 2004 12:32:50 -0700 (MST) (envelope-from mfuhr) Date: Wed, 10 Nov 2004 12:32:50 -0700 From: Michael Fuhr To: pgsql-sql@postgresql.org Subject: Re: Comparing two (largish) tables on different servers Message-ID: <20041110193250.GA18174@winnie.fuhr.org> References: <71E37EF6B7DCC1499CEA0316A256832801D4BAB0@loki.wc.globexplorer.net> <20041110091821.GD25775@sam.samason.me.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20041110091821.GD25775@sam.samason.me.uk> X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200411/116 X-Sequence-Number: 19435 On Wed, Nov 10, 2004 at 09:18:21AM +0000, Sam Mason wrote: > > mkfifo db1 > psql -h "db1" -t -q -c "$query" > db1 > mkfifo db2 > psql -h "db2" -t -q -c "$query" > db2 > diff -u -0 db1 db2 This should work for small data sets, but the OP said the tables were about 5G. Unless you use a cursor, psql will fetch the entire result before writing anything. Also, some implementations of diff might read all of the data from one file before reading much from the other file, especially if the files have differences. Hope you have lots of memory.... -- Michael Fuhr http://www.fuhr.org/~mfuhr/