X-Original-To: pgsql-general-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [64.117.224.130]) by svr1.postgresql.org (Postfix) with ESMTP id B0F4BD1B914 for ; Wed, 27 Aug 2003 01:38:12 -0300 (ADT) Received: from svr1.postgresql.org ([64.117.224.193]) by localhost (neptune.hub.org [64.117.224.130]) (amavisd-new, port 10024) with ESMTP id 71635-06 for ; Wed, 27 Aug 2003 01:37:41 -0300 (ADT) Received: from phaze.fireserve.net (phaze.fireserve.net [207.109.249.2]) by svr1.postgresql.org (Postfix) with ESMTP id 88819D1B8BF for ; Wed, 27 Aug 2003 01:37:36 -0300 (ADT) Received: from fireserve.net (dh0032.moore-s.fireserve.net [207.109.249.161]) by phaze.fireserve.net (8.11.6p2/linuxconf) with ESMTP id h7R4Ymt26977; Tue, 26 Aug 2003 21:34:48 -0700 Message-ID: <3F4C34B7.3050307@fireserve.net> Date: Tue, 26 Aug 2003 21:33:59 -0700 From: Dennis Gearon User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 X-Accept-Language: en-us, en, ru MIME-Version: 1.0 To: Jan Wieck Cc: Chris Travers , Tom Lane , Alvaro Herrera , Ron Johnson , pgsql-general@postgresql.org Subject: Re: Replication Ideas References: <3F483EBD.3020108@travelamericas.com> <1061705587.3733.452.camel@haggis> <3F4A420E.6090604@travelamericas.com> <20030825182441.GB9597@dcc.uchile.cl> <3F4A5724.3060002@travelamericas.com> <11627.1061845759@sss.pgh.pa.us> <3F4A8A7D.30308@travelamericas.com> <3F4C0CAE.3030901@Yahoo.com> In-Reply-To: <3F4C0CAE.3030901@Yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-MailScanner: Found to be clean X-MailScanner-SpamCheck: not spam, SpamAssassin (Disabled due to 20 consecutive timeouts) X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200308/1434 X-Sequence-Number: 47850 Jan Wieck wrote: > WARNING: This is getting long ... > > Postgres-R is a very interesting and inspiring idea. And I've been > kicking that concept around for a while now. What I don't like about > it is that it requires fundamental changes in the lock mechanism and > that it is based on the assumption of very low lock conflict. > > > In Postgres-R a committing transaction sends it's workset (WS - a list > of all updates done in this transaction) to the group communication > system (GC). The GC guarantees total order, meaning that all nodes > will receive all WSs in the same order, no matter how they have been > sent. > > If a node receives back it's own WS before any error occured, it goes > ahead and finalizes the commit. If it receives a foreign WS, it has to > apply the whole WS and commit it before it can process anything else. > If now a local transaction, in progress or while waiting for it's WS > to come back, holds a lock that is required to process such remote WS, > the local transaction needs to be aborted to unlock it's resources ... > it lost the total order race. > > > Postgres-R requires that all remote WSs are applied and committed > before a local transaction can commit. Otherwise it couldn't correctly > detect a lock conflict. So there will not be any read ahead. And since > the total order really counts here, it cannot apply any two remote WSs > in parallel, a race condition could possibly exist and a later WS in > the total order runs faster and locks up a previous one, so we have to > squeeze all remote WSs through one single replication work process. > And all the locally parallel executed transactions that wait for their > WSs to come back have to wait until that poor little worker is done > with the whole pile. Bye bye concurrency. And I don't know how the GC > will deal with the backlog either. Could well choke on it. > > I do not see how this will scale well in a multi-SMP-system cluster. > At least the serialization of WSs will become a horror if there is > significant lock contention like in a standard TPC-C on the district > row containing the order number counter. I don't know for sure, but I > suspect that with this kind of bottleneck, Postgres-R will have to > rollback more than 50% of it's transactions when there are more than 4 > nodes under heavy load (like in a benchmark run). That will suck ... > > > But ... initially I said that it is an inspiring concept ... soooo ... > > I am currently hacking around with some C+PL/TclU+Spread constructs > that might form a rude kind of prototype creature. > > My changes to the Postgres-R concept are that there will be as many > replicating slave processes as there are in summary masters out in the > cluster ... yes, it will try to utilize all the CPU's in the cluster! > For failover reliability, A committing transaction will hold before > finalizing the commit and send it's "I'm ready" to the GC. Every > replicator that reaches the same state send's "I'm ready" too. Spread > guarantees in SAFE_MESS mode that messages are delivered to all nodes > in a group or that at least LEAVE/DISCONNECT messages are deliverd > before. So if a node receives more than 50% of "I'm ready", there > would be a very small gap where multiple nodes have to fail in the > same split second so that the majority of nodes does NOT commit. A > node that reported "I'm ready" but lost more than 50% of the cluster > before committing has to rollback and rejoin or wait for operator > intervention. > > Now the idea is to split up the communication into GC distribution > groups per transaction. So working master backends and associated > replication backends will join/leave a unique group for every > transaction in the cluster. This way, the per process communication is > reduced to the required minimum. > > > As said, I am hacking on some code ... > > > Jan > > Chris Travers wrote: > >> Tom Lane wrote: >> >>> Chris Travers writes: >>> >>> >>>> Yes I have. Postgres-r is not a high-availability solution which is >>>> capable of transparent failover, >>>> >>> >>> >>> What makes you say that? My understanding is it's supposed to survive >>> loss of individual servers. >>> >>> regards, tom lane >>> >>> >>> >>> >> My mistake. I must have gotten them confused with another >> (asynchronous) replication project. >> >> Best Wishes, >> Chris Travers >> >> >> ---------------------------(end of broadcast)--------------------------- >> TIP 9: the planner will ignore your desire to choose an index scan if >> your >> joining column's datatypes do not match > > > As my british friends would say, "Bully for you",and I applaud you playing, struggling, learning from this for our sakes. Jeez, all I think about is me,huh?