From morgan.ramsay@gmail.com Mon Sep 15 00:19:18 2014 Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1XTK0j-0006dZ-T2 for pgsql-admin@arkaria.postgresql.org; Mon, 15 Sep 2014 00:19:18 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1XTK0i-0002Xb-VV for pgsql-admin@arkaria.postgresql.org; Mon, 15 Sep 2014 00:19:17 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1XTK0h-0002XS-Im for pgsql-admin@postgresql.org; Mon, 15 Sep 2014 00:19:15 +0000 Received: from mail-pa0-x22e.google.com ([2607:f8b0:400e:c03::22e]) by makus.postgresql.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1XTK0Y-0000WA-C9 for pgsql-admin@postgresql.org; Mon, 15 Sep 2014 00:19:13 +0000 Received: by mail-pa0-f46.google.com with SMTP id kq14so5192960pab.5 for ; Sun, 14 Sep 2014 17:19:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:subject:date:message-id:mime-version:content-type :thread-index:content-language; bh=Yenj6BbX4ftXvwHKjEps9Kdzp6JJU3EUc89rXCK6VEE=; b=uYc+1CxZAZb0hAEkr/AMM0C2WydDgR5hy0cmWE/a20PUznh+0v4GfBkEB4c0XaV+PH Y2aIP7qLOXKxuQ+aCkTt/q2AA6ghV//UzP5wCEklggw7m4GTJ1B0M3tn7rPZhd/D1hx2 zaVHYW2tm6o3DMJ3WpiMU/6G0BlHllYI9ageBqvYuZRLuSxoTdP2Ixl9NYsMS9Y3HXTs OYN3QnLV6vl9I5ynlziO9uFgYlZS3zjOZ0Is9I+gGU5Ph8aoXONWPNJboI7b5yfCW7uG iWbxrnIKvnYx6aQqqMlG29kzsELhvP3SOS0S3EfSgbLk6CBJBcarvab8nqsDhR2EtOMs QpEA== X-Received: by 10.69.19.202 with SMTP id gw10mr33939538pbd.106.1410740344671; Sun, 14 Sep 2014 17:19:04 -0700 (PDT) Received: from Heretic (ip72-197-173-101.sd.sd.cox.net. [72.197.173.101]) by mx.google.com with ESMTPSA id yp10sm10091317pac.18.2014.09.14.17.19.02 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 14 Sep 2014 17:19:03 -0700 (PDT) From: "Morgan Ramsay" To: Subject: Out of memory running 560 MB query Date: Sun, 14 Sep 2014 17:18:51 -0700 Message-ID: <1a9b601cfd07a$a16c71a0$e44554e0$@gmail.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_1A9B7_01CFD03F.F50DE7C0" X-Mailer: Microsoft Outlook 14.0 Thread-Index: Ac/Qegahm+uaZuuXT+aGNj4UZs7pZw== Content-Language: en-us X-Antivirus: avast! (VPS 140914-1, 09/14/2014), Outbound message X-Antivirus-Status: Clean X-Pg-Spam-Score: 0.7 (/) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org This is a multipart message in MIME format. ------=_NextPart_000_1A9B7_01CFD03F.F50DE7C0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit I have 560 MB worth of INSERT statements in a .sql file. I'm trying to run that query but I'm getting an out of memory error. Any idea how to successfully run a query that large? What's the size limit of a query? -Morgan ------=_NextPart_000_1A9B7_01CFD03F.F50DE7C0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I have 560 = MB worth of INSERT statements in a .sql file. I’m trying to run = that query but I’m getting an out of memory = error.

 

Any idea how to successfully run a query that large? = What’s the size limit of a query?

 

—Morgan

------=_NextPart_000_1A9B7_01CFD03F.F50DE7C0-- From rosser.schwarz@gmail.com Mon Sep 15 00:28:22 2014 Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1XTK9W-000730-Fg for pgsql-admin@arkaria.postgresql.org; Mon, 15 Sep 2014 00:28:22 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1XTK9V-0004bl-VD for pgsql-admin@arkaria.postgresql.org; Mon, 15 Sep 2014 00:28:22 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1XTK9U-0004be-Sl for pgsql-admin@postgresql.org; Mon, 15 Sep 2014 00:28:20 +0000 Received: from mail-ie0-x234.google.com ([2607:f8b0:4001:c03::234]) by magus.postgresql.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1XTK9N-0004Ku-5J for pgsql-admin@postgresql.org; Mon, 15 Sep 2014 00:28:19 +0000 Received: by mail-ie0-f180.google.com with SMTP id rd18so3599716iec.25 for ; Sun, 14 Sep 2014 17:28:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZXC6CrAtNZ4wEI5UTBj/adLuaXIcED8J/c8yUcOqJJM=; b=QPG8OwOGttnP+HEK8dNFNNXqaBV0HxRPRiIoRo4z6hi4b45lDxd3SOCRc2sivcukja o1jdSpB3YLaM/c/ldSzDbaSVZMreT/p2hYPag68kDl/SS3uE5RWGAMLHH05hHGClpeYY 7kzktvQ8+jPqK8PqUGQ9xavYqrddv9uC3WVWFd+ksn2VrOoBco9IHmfYdARfnGD9m/65 TgtX1AhHsqx79vLNhHWQy3L3kx97LCsi0lvwj5NXssE49qUdU9j3vYT5O9/j43MgOGz+ ifCj2H8zXdgTBGsx30IE62uUe2s8lx7vODOb4mUkoHQKW8W1H4g93661ylPyVCaNZ90V 50qQ== MIME-Version: 1.0 X-Received: by 10.50.6.77 with SMTP id y13mr13618840igy.21.1410740889950; Sun, 14 Sep 2014 17:28:09 -0700 (PDT) Received: by 10.50.33.3 with HTTP; Sun, 14 Sep 2014 17:28:09 -0700 (PDT) In-Reply-To: <1a9b601cfd07a$a16c71a0$e44554e0$@gmail.com> References: <1a9b601cfd07a$a16c71a0$e44554e0$@gmail.com> Date: Sun, 14 Sep 2014 17:28:09 -0700 Message-ID: Subject: Re: Out of memory running 560 MB query From: Rosser Schwarz To: Morgan Ramsay Cc: PgSQL ADMIN Content-Type: multipart/alternative; boundary=047d7bdca0c69da1f805030fb585 X-Pg-Spam-Score: -2.0 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org --047d7bdca0c69da1f805030fb585 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Sun, Sep 14, 2014 at 5:18 PM, Morgan Ramsay wrote: > Any idea how to successfully run a query that large? What=E2=80=99s the s= ize limit > of a query? > Without seeing your actual query (or a syntactically similar one, if there are private details you can't disclose), it's difficult to say. In general, though, a couple of thoughts come to mind: Can you run it as a single COPY statement, instead of however many individual INSERTS? Failing that, can you use the multi-row VALUES syntax for your INSERT? As an aside, this type of question is probably more suited for the pgsql-sql list, rather than -admin. rls --=20 :wq --047d7bdca0c69da1f805030fb585 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On Sun, Sep 14, 2014 at 5:18 PM, Morgan Ramsay <morgan.ramsay@gmail.com> wrote:

Any idea how to success= fully run a query that large? What=E2=80=99s the size limit of a query?

=

Without seeing your actual query (or a syntactically similar one, if the= re are private details you can't disclose), it's difficult to say. = =C2=A0In general, though, a couple of thoughts come to mind:

=
Can you run it as a single COPY statement, instead of however many in= dividual INSERTS? =C2=A0Failing that, can you use the multi-row VALUES synt= ax for your INSERT?

As an aside, this type of question i= s probably more suited for the pgsql-sql list, rather than -admin.

rls

--
:wq --047d7bdca0c69da1f805030fb585-- From morgan.ramsay@gmail.com Mon Sep 15 00:54:06 2014 Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1XTKYQ-00082O-8B for pgsql-admin@arkaria.postgresql.org; Mon, 15 Sep 2014 00:54:06 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1XTKYP-00033Q-4u for pgsql-admin@arkaria.postgresql.org; Mon, 15 Sep 2014 00:54:05 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1XTKYN-00033H-T9 for pgsql-admin@postgresql.org; Mon, 15 Sep 2014 00:54:04 +0000 Received: from mail-pd0-x236.google.com ([2607:f8b0:400e:c02::236]) by magus.postgresql.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1XTKYJ-0004ng-7j for pgsql-admin@postgresql.org; Mon, 15 Sep 2014 00:54:02 +0000 Received: by mail-pd0-f182.google.com with SMTP id w10so5153260pde.41 for ; Sun, 14 Sep 2014 17:53:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:references:in-reply-to:subject:date:message-id :mime-version:content-type:thread-index:content-language; bh=7QyhPBgKYTUd3vtJIgjSlfYplFuhpcXa1+uGETKETxQ=; b=qbmSmHpul39xslRGEeM2bV6UiXRf5anHqWCX+4Z92g4mr8dVt0ywr2CZ29yMUWW1R1 5bifGOjSC+ThfhXtyK3EewI8mMltbADWxqK3iE9+On9es4u6vicDoFeQgyAnHzbNxpPO lPU8tRKPin/xoKunkTsk+mmnp3j1IirI1qpNDm9P569kfPq54eZMDX5ZWN5wy0rZyDUf xUE/VApMFo3490qk2Lr9koHBu3UxzT6r4/z1F61T/0Otsfmdo1UzZrIIrA1TVYQCLO1t U59CXySsPQQw6n15iaNY8QxH779QI0fbv3qzji8pXrqlSxQvwgtT7/QDn47bHbuacrks 5Wlw== X-Received: by 10.70.89.165 with SMTP id bp5mr38115pdb.142.1410742435254; Sun, 14 Sep 2014 17:53:55 -0700 (PDT) Received: from Heretic (ip72-197-173-101.sd.sd.cox.net. [72.197.173.101]) by mx.google.com with ESMTPSA id xu4sm10110176pab.30.2014.09.14.17.53.52 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 14 Sep 2014 17:53:53 -0700 (PDT) From: "Morgan Ramsay" To: "'Rosser Schwarz'" Cc: "'PgSQL ADMIN'" References: <1a9b601cfd07a$a16c71a0$e44554e0$@gmail.com> In-Reply-To: Subject: Re: Out of memory running 560 MB query Date: Sun, 14 Sep 2014 17:53:39 -0700 Message-ID: <000001cfd07f$7e577ac0$7b067040$@gmail.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0001_01CFD044.D1F93F00" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHyoDfDIMwHaIn9ozECB0XQXJ9dXgEW1zpTm7MFw6A= Content-Language: en-us X-Antivirus: avast! (VPS 140914-1, 09/14/2014), Outbound message X-Antivirus-Status: Clean X-Pg-Spam-Score: -1.7 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org This is a multipart message in MIME format. ------=_NextPart_000_0001_01CFD044.D1F93F00 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sorry, I was running the query on what appeared to be a cached copy of = the old table=E2=80=94or something. =20 After a reboot, everything seems to be working. The admin client = wasn=E2=80=99t responding but now all of my data is there. =20 With regard to the list, I wasn=E2=80=99t sure whether there was a .conf = setting that limited the size of queries or whether there was a query = size limit. =20 =E2=80=94Morgan =20 =20 From: Rosser Schwarz [mailto:rosser.schwarz@gmail.com]=20 Sent: Sunday, September 14, 2014 5:28 PM To: Morgan Ramsay Cc: PgSQL ADMIN Subject: Re: [ADMIN] Out of memory running 560 MB query =20 On Sun, Sep 14, 2014 at 5:18 PM, Morgan Ramsay = wrote: Any idea how to successfully run a query that large? What=E2=80=99s the = size limit of a query? =20 Without seeing your actual query (or a syntactically similar one, if = there are private details you can't disclose), it's difficult to say. = In general, though, a couple of thoughts come to mind: =20 Can you run it as a single COPY statement, instead of however many = individual INSERTS? Failing that, can you use the multi-row VALUES = syntax for your INSERT? =20 As an aside, this type of question is probably more suited for the = pgsql-sql list, rather than -admin. =20 rls =20 --=20 :wq=20 ------=_NextPart_000_0001_01CFD044.D1F93F00 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable

Sorry, I = was running the query on what appeared to be a cached copy of the old = table=E2=80=94or something.

 =

After a = reboot, everything seems to be working. The admin client wasn=E2=80=99t = responding but now all of my data is there.

 =

With = regard to the list, I wasn=E2=80=99t sure whether there was a .conf = setting that limited the size of queries or whether there was a query = size limit.

 =

=E2=80=94Mo= rgan

 =

 =

From:= = Rosser Schwarz [mailto:rosser.schwarz@gmail.com]
Sent: = Sunday, September 14, 2014 5:28 PM
To: Morgan = Ramsay
Cc: PgSQL ADMIN
Subject: Re: [ADMIN] Out of = memory running 560 MB query

 

On = Sun, Sep 14, 2014 at 5:18 PM, Morgan Ramsay <morgan.ramsay@gmail.com> = wrote:

Any idea = how to successfully run a query that large? What=E2=80=99s the size = limit of a query?

 

Without seeing your actual query (or a syntactically = similar one, if there are private details you can't disclose), it's = difficult to say.  In general, though, a couple of thoughts come to = mind:

 

Can = you run it as a single COPY statement, instead of however many = individual INSERTS?  Failing that, can you use the multi-row VALUES = syntax for your INSERT?

 

As an aside, this type of question is probably more = suited for the pgsql-sql list, rather than = -admin.

 

rls

 

-- =
:wq

------=_NextPart_000_0001_01CFD044.D1F93F00-- From rajeshmadiwale65@gmail.com Mon Sep 15 04:48:02 2014 Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1XTOCn-000115-Q3 for pgsql-admin@arkaria.postgresql.org; Mon, 15 Sep 2014 04:48:02 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1XTOCn-00071a-6K for pgsql-admin@arkaria.postgresql.org; Mon, 15 Sep 2014 04:48:01 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1XTOCl-00071R-2n for pgsql-admin@postgresql.org; Mon, 15 Sep 2014 04:47:59 +0000 Received: from mail-we0-x233.google.com ([2a00:1450:400c:c03::233]) by magus.postgresql.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1XTOCh-0000cN-Dh for pgsql-admin@postgresql.org; Mon, 15 Sep 2014 04:47:57 +0000 Received: by mail-we0-f179.google.com with SMTP id u56so3309152wes.24 for ; Sun, 14 Sep 2014 21:47:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=KY7jMWc4sDoVPLyPivpuj7wuAAhAvsELd0Nw7URWPMg=; b=UhjbuHEhTgh3XrAGi1uNtX5oge1KnUYUmROQnXYks1CsZbafo/vDNea/CQQOO34fIf acNLPNF/29+0KXA0Owq8MtxdYt1BP13CJ1eDH8Zxpng6mr7Q2/INer9709LCF+IPHRme QHE0Fr2bPMIHLS9KINaXJ4A2tvdmXKlph6/BwnFlPAwQF3s9avnzf7Wm9D9jQri02bt2 SGbmZSGvb7Y2Jro/i/53rGdYzd3pSJ3Ae6OTHn1tCzxxXBlSJmU1FRqUhYXP/GEuSqsF Ro8nmIN9Zb8wOaTl3kykHyJLlq54Gmwz57JArSIIiXfq13Keu6RDS61OfsiskZBQrwYs ielQ== MIME-Version: 1.0 X-Received: by 10.180.99.34 with SMTP id en2mr20925141wib.34.1410756473730; Sun, 14 Sep 2014 21:47:53 -0700 (PDT) Received: by 10.180.206.235 with HTTP; Sun, 14 Sep 2014 21:47:53 -0700 (PDT) In-Reply-To: <000001cfd07f$7e577ac0$7b067040$@gmail.com> References: <1a9b601cfd07a$a16c71a0$e44554e0$@gmail.com> <000001cfd07f$7e577ac0$7b067040$@gmail.com> Date: Mon, 15 Sep 2014 10:17:53 +0530 Message-ID: Subject: Re: Out of memory running 560 MB query From: Rajesh Madiwale To: Morgan Ramsay Cc: Rosser Schwarz , PgSQL ADMIN Content-Type: multipart/alternative; boundary=f46d0442847e7b3b040503135663 X-Pg-Spam-Score: -0.7 (/) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org --f46d0442847e7b3b040503135663 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Mon, Sep 15, 2014 at 6:23 AM, Morgan Ramsay wrote: > Sorry, I was running the query on what appeared to be a cached copy of th= e > old table=E2=80=94or something. > > > > After a reboot, everything seems to be working. The admin client wasn=E2= =80=99t > responding but now all of my data is there. > > > > With regard to the list, I wasn=E2=80=99t sure whether there was a .conf = setting > that limited the size of queries or whether there was a query size limit. > > Hi, I suspect this issue occured due to no proper configuration of parameter( shared_buffer and maintenance_work_mem) in postgresql.conf file. For greater insert/copy operation shared_buffer and maintenance_work_mem parameters should be configured to resolve the issue. Regards, Rajesh Madiwale, Database Administrator, Shreeyansh Technologies. > > > =E2=80=94Morgan > > > > > > *From:* Rosser Schwarz [mailto:rosser.schwarz@gmail.com] > *Sent:* Sunday, September 14, 2014 5:28 PM > *To:* Morgan Ramsay > *Cc:* PgSQL ADMIN > *Subject:* Re: [ADMIN] Out of memory running 560 MB query > > > > On Sun, Sep 14, 2014 at 5:18 PM, Morgan Ramsay > wrote: > > Any idea how to successfully run a query that large? What=E2=80=99s the s= ize limit > of a query? > > > > Without seeing your actual query (or a syntactically similar one, if ther= e > are private details you can't disclose), it's difficult to say. In > general, though, a couple of thoughts come to mind: > > > > Can you run it as a single COPY statement, instead of however many > individual INSERTS? Failing that, can you use the multi-row VALUES synta= x > for your INSERT? > > > > As an aside, this type of question is probably more suited for the > pgsql-sql list, rather than -admin. > > > > rls > > > > -- > :wq > --f46d0442847e7b3b040503135663 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

=C2=A0

=E2=80=94Morgan

=C2=A0

=C2=A0

From: Rosser Schwarz [mailto:rosser.schwarz@gmail.com] =
Sent: Sunday, September 14, 2014 5:28 PM
To: Morgan Ra= msay
Cc: PgSQL ADMIN
Subject: Re: [ADMIN] Out of memory= running 560 MB query

=C2=A0

O= n Sun, Sep 14, 2014 at 5:18 PM, Morgan Ramsay <morgan.ramsay@gmail.com> wrote:<= u>

Any idea how to successfully run a query that large? What=E2=80=99s the= size limit of a query?

=C2=A0

Without seeing your actual query (or a syntactically similar one, if ther= e are private details you can't disclose), it's difficult to say. = =C2=A0In general, though, a couple of thoughts come to mind:<= /p>

=C2=A0

Can you run it as a single COPY statement, instead of howeve= r many individual INSERTS? =C2=A0Failing that, can you use the multi-row VA= LUES syntax for your INSERT?

<= u>=C2=A0

As an aside, this = type of question is probably more suited for the pgsql-sql list, rather tha= n -admin.

=C2=A0<= u>

rls

=C2=A0

--
:wq


--f46d0442847e7b3b040503135663--