Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 9672C1337BA3 for ; Wed, 30 Mar 2011 21:48:42 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 90264-05 for ; Thu, 31 Mar 2011 00:48:33 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from web39708.mail.mud.yahoo.com (web39708.mail.mud.yahoo.com [209.191.106.54]) by mail.postgresql.org (Postfix) with SMTP id A219C1337999 for ; Wed, 30 Mar 2011 21:48:31 -0300 (ADT) Received: (qmail 82523 invoked by uid 60001); 31 Mar 2011 00:48:30 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1301532510; bh=WsvbQpisso4dSTVEEzSk9/UGACd3/41jdAoNcQes7kI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=Oz7ORy/ixjLqnHiWfBOYYD++IWz0qxVCZzEQMG8IR4Nj8uFx9utxqi/An9S2cQlQ2NXgFvHb5k88bEXJ48VkRRIYksDdRWxNXoV1hbknjeGnirEpJ4SqEbRj+EaXbn/Y/wLg/llBNs6t1CSlQKJUqMBOiAg2nTwZ3nxU/k61B4U= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=TtjwEDJbcRjrdsrhPllNAmDfiUa5UGrIYzF15vBYHb1HCReR8N3Rkgun5Pky5jOOdwc2KZ9D0Db8LOyK+JFbeuApZdhVeaHGm37FRNKo0modnevle3ZjcxqJ6Q4+QZj2ycULkYbHjn8VM/KqpB0P/YoPWN4aQLrKe+tU9B6qxzo=; Message-ID: <769995.72751.qm@web39708.mail.mud.yahoo.com> X-YMail-OSG: L15_xg0VM1nl1wNJsTZ7CjHguI9gO1NhkBkd0kDSVZACAqX fuzmW8LchYrjg_hkg9Ca3k61At9DyjuV7dcpw_h8Owo3jtmYeWSYi3XXQ2.5 JyuDqUPIcHgauj29DJOePkRCEoCk4mx1UgZ3jYXTQ4TfaI2owbOO3qDxRtFR BrVNiHxXNOYaPGX4vhO4_6rXi0Ew3uO0lT.lr.wkYmUvRJw5lAObVdBjB2UG 9.raQOqO_gvA3mlTZgBmRNjyfOBOH7uR3_lH9Xh._MQwSHDwtUqQ_om5X3u0 pjcIIMD7r4Yu6EqdoeBJmbDjf7CzPkeuf91UDai2qI6VXMA7JZMn2RaAWigh acfs5dAC2aYmdOmxRuubIC29TnTcZQqmP7tg06V_imivK.ArS8TiUh5LHirk QwZuSFvAB3W7DlV5W Received: from [69.76.130.83] by web39708.mail.mud.yahoo.com via HTTP; Wed, 30 Mar 2011 17:48:30 PDT X-Mailer: YahooMailClassic/12.0.2 YahooMailWebService/0.8.109.295617 Date: Wed, 30 Mar 2011 17:48:30 -0700 (PDT) From: Bob Lunney Subject: Re: COPY with high # of clients, partitioned table locking issues? To: "pgsql-performance@postgresql.org" , John WStrange In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.909 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201103/373 X-Sequence-Number: 43052 John,=0A=0ASorry to hear you're struggling with such underpowered hardware.= ;-) A little more information would be helpful, though:=0A=0A1. What ve= rsion of PG are you running?=0A2. What are the constraints on the child ta= bles?=0A3. How many rows does each copy insert?=0A4. Are these wrapped in= transactions?=0A5. are the child tables created at the same time the copi= es are taking place? In the same transaction?=0A6. Are the indexes in pla= ce on the child table(s) when the copies are running? Do they have to be t= o validate the data?=0A7. What are the configuration settings for the data= base? (Just the ones changed from the default, please.)=0A8. Which file s= ystem are you running for the database files? Mount options?=0A9. Are the= WAL files on the same file system?=0A=0A=0ABob Lunney=0A=0A--- On Wed, 3/3= 0/11, Strange, John W wrote:=0A=0A> From: Str= ange, John W =0A> Subject: [PERFORM] COPY with= high # of clients, partitioned table locking issues?=0A> To: "pgsql-perfor= mance@postgresql.org" =0A> Date: Wednesda= y, March 30, 2011, 4:56 PM=0A> Just some information on our setup:=0A> =0A>= - HP DL585 G6 =0A> - 4 x AMD Opteron 8435 (24 cores)=0A> - 256GB RAM=0A> -= 2 FusionIO 640GB PCI-SSD (RAID0)=0A> - dual 10GB ethernet.=0A> =0A> - we h= ave several tables that we store calculated values=0A> in.=0A> - these are = inserted by a compute farm that calculates the=0A> results and stores them = into a partitioned schema (schema=0A> listed below)=0A> - whenever we do a = lot of inserts we seem to get exclusive=0A> locks.=0A> =0A> Is there someth= ing we can do to improve the performance=0A> around locking when doing a lo= t of parallel inserts with=0A> COPY into?=A0 We are not IO bound, what happ= ens is that=0A> the copies start to slow down and continue to come in and= =0A> cause the client to swap, we had hit over 800+ COPYS were in=0A> a wai= ting state, which forced us to start paging heavily=0A> creating an issue.= =A0 If we can figure out the locking=0A> issue the copys should clear faste= r requiring less memory in=0A> use.=0A> =0A> [ 2011-03-30 15:54:55.886 EDT = ] 14405 [local]=0A> asgprod:4d938288.3845 LOG:=A0 process 14405 still waiti= ng=0A> for ExclusiveLock on extension of relation 470273 of=0A> database 16= 384 after 5001.894 ms=0A> [ 2011-03-30 15:54:55.886 EDT ] 14405 [local]=0A>= asgprod:4d938288.3845 CONTEXT:=A0 COPY=0A> reportvalues_part_1931, line 1:= "660250=A0 =A0 =A0=0A> 41977959=A0 =A0 =A0 =A0=0A> 11917=A0=A0=A0584573.43= 642105709"=0A> [ 2011-03-30 15:54:55.886 EDT ] 14405 [local]=0A> asgprod:4d= 938288.3845 STATEMENT:=A0 COPY=0A> reportvalues_part_1931 FROM stdin USING = DELIMITERS '=A0=0A> =A0 =A0=A0=A0'=0A> [ 2011-03-30 15:54:56.015 EDT ] 7294= [local]=0A> asgprod:4d938939.1c7e LOG:=A0 process 7294 still waiting=0A> f= or ExclusiveLock on extension of relation 470606 of=0A> database 16384 afte= r 5062.968 ms=0A> [ 2011-03-30 15:54:56.015 EDT ] 7294 [local]=0A> asgprod:= 4d938939.1c7e CONTEXT:=A0 COPY=0A> reportvalues_part_1932, line 158: "66072= 9=A0=0A> =A0=A0=A041998839=A0 =A0 =A0 =A0=0A> 887=A0 =A0=A0=A045000.0"=0A> = [ 2011-03-30 15:54:56.015 EDT ] 7294 [local]=0A> asgprod:4d938939.1c7e STAT= EMENT:=A0 COPY=0A> reportvalues_part_1932 FROM stdin USING DELIMITERS '=A0= =0A> =A0 =A0 =A0 '=0A> [ 2011-03-30 15:54:56.077 EDT ] 25781 [local]=0A> as= gprod:4d938556.64b5 LOG:=A0 process 25781 still waiting=0A> for ExclusiveLo= ck on extension of relation 470606 of=0A> database 16384 after 5124.463 ms= =0A> =0A> relation=A0 =A0 =A0 |=A0 =A0 16384=0A> |=A0=A0=A0470606 |=A0 =A0 = =A0 |=A0=0A> =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0 |=A0 =A0 =A0 =A0 =A0= =A0=0A> =A0=A0=A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 = =A0 =A0 =A0 | 93/677526=A0 =A0 =A0=0A> =A0 =A0 | 14354 | RowExclusiveLock= =A0 =A0 =A0=0A> =A0=A0=A0| t=0A> relation=A0 =A0 =A0 |=A0 =A0 16384=0A> |= =A0=A0=A0470606 |=A0 =A0 =A0 |=A0=0A> =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A>= =A0 |=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 = =A0 =A0=A0=A0|=A0=0A> =A0 =A0 =A0 =A0 | 1047/4=A0 =A0 =A0=0A> =A0 =A0 =A0= =A0=A0| 27451 |=0A> RowExclusiveLock=A0 =A0 =A0 =A0=A0=A0| t=0A> relation= =A0 =A0 =A0 |=A0 =A0 16384=0A> |=A0=A0=A0470606 |=A0 =A0 =A0 |=A0=0A> =A0 = =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0 |=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0=A0= =A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 =A0 =A0 =A0 | 7= 24/58891=A0 =A0 =A0=0A> =A0 =A0 | 20721 | RowExclusiveLock=A0 =A0 =A0=0A> = =A0=A0=A0| t=0A> transactionid |=A0 =A0 =A0 =A0 =A0 |=A0=0A> =A0 =A0 =A0 = =A0 |=A0 =A0 =A0 |=A0=0A> =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0 |=A0 = =A0 =A0 94673393 |=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 = =A0 =A0 =A0 | 110/502566=A0 =A0 =A0=0A> =A0=A0=A0|=A0 1506 | ExclusiveLock= =A0 =A0=0A> =A0 =A0 =A0 =A0 | t=0A> virtualxid=A0 =A0 |=A0 =A0 =A0 =A0=0A>= =A0 |=A0 =A0 =A0 =A0 =A0 |=A0 =A0=0A> =A0 |=A0 =A0 =A0=A0=A0| 975/92=A0=0A= > =A0=A0=A0|=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=0A> =A0=A0=A0= |=A0 =A0 =A0=A0=A0|=A0=0A> =A0 =A0 =A0 =A0 | 975/92=A0 =A0 =A0=0A> =A0 =A0 = =A0=A0=A0| 25751 |=0A> ExclusiveLock=A0 =A0 =A0 =A0 =A0 =A0 | t=0A> extend= =A0 =A0 =A0 =A0 |=A0 =A0 16384=0A> |=A0=A0=A0470606 |=A0 =A0 =A0 |=A0=0A> = =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0 |=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0= =A0=A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 =A0 =A0 =A0 = | 672/102043=A0 =A0 =A0=0A> =A0=A0=A0| 20669 | ExclusiveLock=A0 =A0=0A> =A0= =A0 =A0 =A0 | f=0A> extend=A0 =A0 =A0 =A0 |=A0 =A0 16384=0A> |=A0=A0=A047= 0606 |=A0 =A0 =A0 |=A0=0A> =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0 |=A0 = =A0 =A0 =A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0= =A0|=A0=0A> =A0 =A0 =A0 =A0 | 1178/10=A0 =A0 =A0=0A> =A0 =A0 =A0 |=A0 6074 = | ExclusiveLock=A0=0A> =A0 =A0 =A0 =A0 =A0 | f=0A> virtualxid=A0 =A0 |=A0 = =A0 =A0 =A0=0A> =A0 |=A0 =A0 =A0 =A0 =A0 |=A0 =A0=0A> =A0 |=A0 =A0 =A0=A0= =A0| 37/889225=A0=0A> |=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=0A= > =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 =A0 =A0 =A0 | 37/889225=A0 =A0 = =A0=0A> =A0 =A0 |=A0 4623 | ExclusiveLock=A0 =A0=0A> =A0 =A0 =A0 =A0 | t=0A= > relation=A0 =A0 =A0 |=A0 =A0 16384=0A> |=A0=A0=A0405725 |=A0 =A0 =A0 |= =A0=0A> =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0 |=A0 =A0 =A0 =A0 =A0 =A0= =0A> =A0=A0=A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 =A0 = =A0 =A0 | 39/822056=A0 =A0 =A0=0A> =A0 =A0 | 32502 | AccessShareLock=A0 =A0= =A0=0A> =A0 =A0 | t=0A> transactionid |=A0 =A0 =A0 =A0 =A0 |=A0=0A> =A0 = =A0 =A0 =A0 |=A0 =A0 =A0 |=A0=0A> =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0= |=A0 =A0 =A0 94673831 |=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A>= =A0 =A0 =A0 =A0 | 917/278=A0 =A0 =A0=0A> =A0 =A0 =A0 | 23134 | ExclusiveLo= ck=A0 =A0=0A> =A0 =A0 =A0 =A0 | t=0A> relation=A0 =A0 =A0 |=A0 =A0 16384= =0A> |=A0=A0=A0470609 |=A0 =A0 =A0 |=A0=0A> =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 = =A0=0A> =A0 |=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=0A> =A0=A0= =A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 =A0 =A0 =A0 | 537/157021=A0 =A0 =A0=0A> = =A0=A0=A0| 11863 | RowExclusiveLock=A0 =A0=0A> =A0 =A0=A0=A0| t=0A> relati= on=A0 =A0 =A0 |=A0 =A0 16384=0A> |=A0=A0=A0470609 |=A0 =A0 =A0 |=A0=0A> =A0= =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0 |=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0=A0= =A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 =A0 =A0 =A0 | 5= 32/91114=A0 =A0 =A0=0A> =A0 =A0 |=A0 7282 | RowExclusiveLock=A0 =A0=0A> =A0= =A0=A0=A0| t=0A> virtualxid=A0 =A0 |=A0 =A0 =A0 =A0=0A> =A0 |=A0 =A0 =A0 = =A0 =A0 |=A0 =A0=0A> =A0 |=A0 =A0 =A0=A0=A0| 920/8=A0 =A0=0A> =A0 |=A0 =A0 = =A0 =A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0=A0|= =A0=0A> =A0 =A0 =A0 =A0 | 920/8=A0 =A0 =A0=0A> =A0 =A0 =A0 =A0 | 23137 | Ex= clusiveLock=A0=0A> =A0 =A0 =A0 =A0 =A0 | t=0A> relation=A0 =A0 =A0 |=A0 = =A0 16384=0A> |=A0=A0=A0425555 |=A0 =A0 =A0 |=A0=0A> =A0 =A0=A0=A0|=A0 =A0 = =A0 =A0 =A0=0A> =A0 |=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=0A> = =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 =A0 =A0 =A0 | 39/822056=A0 =A0 =A0= =0A> =A0 =A0 | 32502 | AccessShareLock=A0 =A0 =A0=0A> =A0 =A0 | t=0A> rela= tion=A0 =A0 =A0 |=A0 =A0 16384=0A> |=A0=A0=A0470606 |=A0 =A0 =A0 |=A0=0A> = =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0 |=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0= =A0=A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0=A0|=A0=0A> =A0 =A0 =A0 =A0 = | 915/10=A0 =A0 =A0=0A> =A0 =A0 =A0=A0=A0| 22619 |=0A> RowExclusiveLock=A0 = =A0 =A0 =A0=A0=A0| t=0A> relation=A0 =A0 =A0 |=A0 =A0 16384=0A> |=A0=A0=A0= 470606 |=A0 =A0 =A0 |=A0=0A> =A0 =A0=A0=A0|=A0 =A0 =A0 =A0 =A0=0A> =A0 |=A0= =A0 =A0 =A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=0A> =A0=A0=A0|=A0 =A0 =A0=A0= =A0|=A0=0A> =A0 =A0 =A0 =A0 | 344/387563=A0 =A0 =A0=0A> =A0=A0=A0| 30343 | = RowExclusiveLock=A0 =A0=0A> =A0 =A0=A0=A0| tNumber of child tables: 406 (Us= e=0A> \d+ to list them.)=0A> =0A> =0A> riskresults=3D# \d reportvalues_part= _1932;=0A> =A0 =A0 Table "public.reportvalues_part_1932"=0A> =A0 =A0 Column= =A0 =A0 |=A0 =A0=0A> =A0=A0=A0Type=A0 =A0 =A0=A0=A0|=0A> Modifiers=0A> ----= ----------+------------------+-----------=0A> reportid=A0 =A0=A0=A0| integ= er=A0 =A0=0A> =A0 =A0 =A0 | not null=0A> scenarioid=A0=A0=A0| integer=A0 = =A0 =A0=0A> =A0 =A0 | not null=0A> instrumentid | integer=A0 =A0 =A0 =A0 = =A0 |=0A> not null=0A> value=A0 =A0 =A0 =A0 | double precision |=0A> Index= es:=0A> =A0 =A0 "reportvalues_part_1932_pkey" PRIMARY KEY,=0A> btree (repor= tid, scenarioid, instrumentid)=0A> Inherits: reportvalues_part=0A> =0A> ris= kresults=3D# \d reportvalues_part;=0A> =A0 =A0 =A0 Table "public.reportvalu= es_part"=0A> =A0 =A0 Column=A0 =A0 |=A0 =A0=0A> =A0=A0=A0Type=A0 =A0 =A0=A0= =A0|=0A> Modifiers=0A> --------------+------------------+-----------=0A> r= eportid=A0 =A0=A0=A0| integer=A0 =A0=0A> =A0 =A0 =A0 | not null=0A> scenar= ioid=A0=A0=A0| integer=A0 =A0 =A0=0A> =A0 =A0 | not null=0A> instrumentid = | integer=A0 =A0 =A0 =A0 =A0 |=0A> not null=0A> value=A0 =A0 =A0 =A0 | dou= ble precision |=0A> Indexes:=0A> =A0 =A0 "reportvalues_part_pkey" PRIMARY K= EY, btree=0A> (reportid, scenarioid, instrumentid)=0A> Number of child tabl= es: 406 (Use \d+ to list them.)=0A> =0A> This communication is for informat= ional purposes only. It=0A> is not=0A> intended as an offer or solicitation= for the purchase or=0A> sale of=0A> any financial instrument or as an offi= cial confirmation of=0A> any=0A> transaction. All market prices, data and o= ther information=0A> are not=0A> warranted as to completeness or accuracy a= nd are subject to=0A> change=0A> without notice. Any comments or statements= made herein do=0A> not=0A> necessarily reflect those of JPMorgan Chase & C= o., its=0A> subsidiaries=0A> and affiliates.=0A> =0A> This transmission may= contain information that is=0A> privileged,=0A> confidential, legally priv= ileged, and/or exempt from=0A> disclosure=0A> under applicable law. If you = are not the intended=0A> recipient, you=0A> are hereby notified that any di= sclosure, copying,=0A> distribution, or=0A> use of the information containe= d herein (including any=0A> reliance=0A> thereon) is STRICTLY PROHIBITED. A= lthough this transmission=0A> and any=0A> attachments are believed to be fr= ee of any virus or other=0A> defect=0A> that might affect any computer syst= em into which it is=0A> received and=0A> opened, it is the responsibility o= f the recipient to ensure=0A> that it=0A> is virus free and no responsibili= ty is accepted by JPMorgan=0A> Chase &=0A> Co., its subsidiaries and affili= ates, as applicable, for=0A> any loss=0A> or damage arising in any way from= its use. If you received=0A> this=0A> transmission in error, please immedi= ately contact the=0A> sender and=0A> destroy the material in its entirety, = whether in electronic=0A> or hard=0A> copy format. Thank you.=0A> =0A> Plea= se refer to http://www.jpmorgan.com/pages/disclosures=0A> for=0A> disclosur= es relating to European legal entities.=0A> =0A> -- =0A> Sent via pgsql-per= formance mailing list (pgsql-performance@postgresql.org)=0A> To make change= s to your subscription:=0A> http://www.postgresql.org/mailpref/pgsql-perfor= mance=0A> =0A=0A=0A