Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1WvAzq-0002js-Ne for pgsql-performance@arkaria.postgresql.org; Thu, 12 Jun 2014 19:49:14 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1WvAzq-0001Kw-1S for pgsql-performance@arkaria.postgresql.org; Thu, 12 Jun 2014 19:49:14 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1WvAzp-0001Kk-5v for pgsql-performance@postgresql.org; Thu, 12 Jun 2014 19:49:13 +0000 Received: from post.officenet.no ([195.225.13.103]) by magus.postgresql.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1WvAzm-00059D-Jk for pgsql-performance@postgresql.org; Thu, 12 Jun 2014 19:49:11 +0000 Received: from [10.47.1.10] (helo=tc7-on) by post.officenet.no with esmtp (Exim 4.76) (envelope-from ) id 1WvAzj-0001u6-W3 for pgsql-performance@postgresql.org; Thu, 12 Jun 2014 21:49:10 +0200 Received: from localhost ([127.0.0.1] helo=tc7-on) by tc7-on with esmtp (Exim 4.76) (envelope-from ) id 1WvAzQ-0006Nb-MZ for pgsql-performance@postgresql.org; Thu, 12 Jun 2014 21:48:48 +0200 Date: Thu, 12 Jun 2014 21:48:48 +0200 (CEST) From: Andreas Joseph Krogh To: pgsql-performance@postgresql.org Message-ID: In-Reply-To: <24297.1402585086@sss.pgh.pa.us> Subject: Re: OFFSET/LIMIT - Disparate Performance w/ Go application MIME-Version: 1.0 X-Mailer: Visena Mail 1.9.0-SNAPSHOT X-Spam-Score: -0.3 X-Spam-Report: SpamAssasin (score=-0.3, required 5.0 ALL_TRUSTED=-1, HTML_IMAGE_ONLY_28=0.726, HTML_MESSAGE=0.001) X-Pg-Spam-Score: -0.5 (/) Content-Type: multipart/related; boundary="----=_Part_686_1791755185.1402602528472" List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org ------=_Part_685_1623081397.1402602528472 Content-Type: multipart/related; boundary="----=_Part_686_1791755185.1402602528472" ------=_Part_686_1791755185.1402602528472 Content-Type: multipart/alternative; boundary="----=_Part_687_1952989846.1402602528489" ------=_Part_687_1952989846.1402602528489 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable P=C3=A5 torsdag 12. juni 2014 kl. 16:58:06, skrev Tom Lane >: Matt Silverlock writ= es: > Hi all. This might be tricky in so much as there=EF=BF=BD=EF=BF=BD=EF=BF= =BDs a few moving parts=20 (when isn=EF=BF=BD=EF=BF=BD=EF=BF=BDt there?), but I=EF=BF=BD=EF=BF=BD=EF= =BF=BDve tried to test the postgres side as much as=20 possible. > Trying to work out a potential database bottleneck with a HTTP applicati= on=20 (written in Go): > Pages that render HTML templates but don=EF=BF=BD=EF=BF=BD=EF=BF=BDt per= form DB queries can hit=20 ~36k+ req/s > Pages that perform a SELECT on a single row net about ~6.6k req/s:=20 db.Get(l, "SELECT * FROM listings WHERE id =3D $1 AND expiry_date >=20 current_date", l.Id) > Pages that SELECT multiple rows with OFFSET and LIMIT conditions struggl= e=20 to top 1.3k req/s You don't show us exactly what you're doing with OFFSET/LIMIT, but I'm going to guess that you're using it to paginate large query results. That's basically always going to suck: Postgres has no way to implement OFFSET except to generate and then throw away that number of initial rows. If you do the same query over again N times with different OFFSETs, it's going to cost you N times as much as the base query would. =C2=A0 Are ther= e any=20 plans to make PG implement OFFSET more efficiently, so it doesn't have to "= read=20 and throw away"? =C2=A0 I used SQL Server back in 2011 in a project and see= m to=20 remember they implemented offset pretty fast. Paging in a resultset of mill= ions=20 was much faster than in PG. =C2=A0 -- Andreas Jospeh Krogh CTO / Partner - = Visena AS=20 Mobile: +47 909 56 963 andreas@visena.com =20 www.visena.com =C2=A0 ------=_Part_687_1952989846.1402602528489 Content-Type: text/html;charset=UTF-8 Content-Transfer-Encoding: quoted-printable
P=C3=A5 torsdag 12. juni 2014 kl. 16:58:06, skrev Tom Lane <tgl@sss.pgh.pa.us>:
Mat= t Silverlock <matt@eatsleeprepeat.net> writes:
> Hi all. This might be tricky in so much as there=EF=BF=BD=EF=BF=BD=EF= =BF=BDs a few moving parts (when isn=EF=BF=BD=EF=BF=BD=EF=BF=BDt there?), b= ut I=EF=BF=BD=EF=BF=BD=EF=BF=BDve tried to test the postgres side as much a= s possible.
> Trying to work out a potential database bottleneck with a HTTP applica= tion (written in Go):
> Pages that render HTML templates but don=EF=BF=BD=EF=BF=BD=EF=BF=BDt p= erform DB queries can hit ~36k+ req/s
> Pages that perform a SELECT on a single row net about ~6.6k req/s: db.= Get(l, "SELECT * FROM listings WHERE id =3D $1 AND expiry_date > cu= rrent_date", l.Id)
> Pages that SELECT multiple rows with OFFSET and LIMIT conditions strug= gle to top 1.3k req/s

You don't show us exactly what you're doing with OFFSET/LIMIT, but I'm
going to guess that you're using it to paginate large query results.
That's basically always going to suck: Postgres has no way to implement
OFFSET except to generate and then throw away that number of initial rows.<= br> If you do the same query over again N times with different OFFSETs, it's going to cost you N times as much as the base query would.
=C2=A0
Are there any plans to make PG implement OFFSET more efficiently, so i= t doesn't have to "read and throw away"?
=C2=A0
I used SQL Server back in 2011 in a project and seem to remember they = implemented offset pretty fast. Paging in a resultset of millions was much = faster than in PG.
=C2=A0
--
Andrea= s Jospeh Krogh
CTO / Partner<= /span> - Visena AS
Mobile: +47 90= 9 56 963
=3D""
=C2=A0
------=_Part_687_1952989846.1402602528489-- ------=_Part_686_1791755185.1402602528472 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-ID: iVBORw0KGgoAAAANSUhEUgAAAIUAAAAYCAYAAADUIj6hAAAABHNCSVQICAgIfAhkiAAABzBJREFU aEPtmNFxHDcMhmVP3i1VECpvnjzkVIHWFfhcgVcVRKrAUgWRK/C6Al8H3lTgy0PGbzFdQc4VJP/H ADs43q6kROeJNbOYgQACIAgCWJKng4MZ5gxUGXh0U0b+SIuF9G+EWXj2Q15vbrE/NPtk9uub7Gfd t5mBx1NhqSEo8HshjbEUtlO2yIM9tsx5dZP9rPt2MzDZFAqZE4LGcFjdsg3saQaHt7fY70X949On aS+OZidDBkavD33157L4xaw2os+Mp/DAC10l2XhOCeStj0W5ajrJL8WfCi80Xgf9vVlrhg9yRONe /f7xI2vNsIcM7JwU9o6oGyJrLT8JOA1aX9sKP4wl94bAniukEbo/n7YPmuTETzIab4Y9ZWCrKexd 8M58lxPCvvD6aihfvexbEQoPYO8NQRO0JnddGO6FJYaVsBde7cXj7KRk4LsqDxQzCYeGsJNgaXZe +JXkyGgWINq3Gp+bHNILz+wEYs5qH1eJrouNrpDXtk4O6x1IvtD40GWyJYYdkF0jIbYZlN16xygI gl9smbMDsmFdfAKb23xGB8H/mv3tOJfArs1kukk79DEPdQ5u0g1vCvvqKXIW8mZYW+F3Tg4r8HvZ kQCCLydK8EFMQCe5N8RgL9mRG0SqQFuNvdE6beSs0qPDBuCdg0+gvClso8SbTO4ki7mQzQqBrcMH QPwReg2wW8umEe/+L8T/LEzBGF9nXjzZ46s+ITHvhegW8LL39xm6App7KYL/GE+vcYnFbJiP/0aY hdiCnRC7jaj7eiK2ETInC5PRF6LYkSN0+HabF75WuT6syCyI0YkVGGMvUC33AmfZTDUEj8u6IVhu EhRUJ2U2g6UlugyNb03Hl9obHwlxJbcRzcYjO4QPjVfGFTQav4vrmp7cpMp2qbHnBxWJbisbho2Q XI6C1sIHDUFhH4Hij4UbIev66eANeiwbkA+LBsO36zAHzoXU7AhbqLAXEiOYTXcSdO8VS9L4wN8U BIYhBd6oSUgYMijOkecRuTdQa/YiBXhbXFcnCnI2SrfeBG9NydrLYMhGHa5qB9pQIxlzAE6Okjzx JO5afGe6kmhBicWKQNKuTZ5EW+MjQY8v/9rQ0bhJSJyNGUe/JH1t8h1iMbdSPAvxHYin6YmN9YBX QmTYZZNh14vHhhhal5vtcIrJjmvszPRJdExH3MWHvykoBEc9CoCGWJisOLOGoCORs1FvIMbYA8z3 kwM59oemy6LlWrLxFLmWgiQAfEGd8S+NssbK+EhyGLxUkhhiy717waBqHOJYSEacwBejkFNhjJNj v/gANOdQxPfciP/JVBCK2cOIcg3RRJ+CPrLPNVhhN6F38VKMF3XLlIJrjU5C8gMFxvKDvOcPc4rV NjCn7KM0BV+161V8viSCuJZ8SITGJGEh7IUUlxOFMYUH1kJOCN4Wh+I5pqCuK01k40kSNtnKiKIl UfxAgW5sU5JlS05rtt5YFJF1SWoqHv6BRgQcA4/bdb9WRjmMk3jyUMAbIoyJq9e4cVmgzKt9j5gN J/aYDhk+hhjExwaPcz5PObA5Zd9bvz5UzCTZuZDidu5AchpiKSwPR+TV1bCWqBQ9nCjJ5neivC82 Nr4LeS2j1gw5LUqwBuhGQQU5UwHeSslXkwIynz2U2A2yKDgG7OffwLA3TpGRpk0Tzpj3ZEJXi/GR a6GN0e0NtppChePdcAz1FTS+FN8Kpxoiykk+J8fC5tenjbu9kSqpHLu9jBrhUuhNwVGbpyZrzrl0 HPVD8SXj5EOOD/eDC64VjvYBZLuUbIVAfBN1t/C/SU+cAGtdGo+fVnzycUX5wmn6eCKPmWYJG2E/ ppTsuRCbvUD9fwquksG5GqLVKhzDQ3GrEyLKSXhsiK3T5j9EyxffCFOYi2wUlPyFFDQAhehESDjQ GoX0ho0oj8RPou6TxHJdcT0NTcWkO0AnG7+uXsnHqcas/72wvWF+mSf7N/Watp9kTXoluzeS0fB9 9CcZ/hvhSZTfh99pispZOXL9KrGrARkNUMu9ITbScZWs7xOYNt9pwxSZtYDsX/GE3xTkrXgwAsXm fud08FiTeC+m2/o7ppo+PTS/NBK5ARpDG5YHr++DpmVfreYdiX8mnp+DzPEG9WbqJON0JBenZofs sxBA1gj5NXGvfJu/Qh7HwQh/VDUEyUxCit5hH94QfKkEdu+GwK8BX0hvCF+D67xhjmXQCXMwJCZ+ olK0A1EKRCE4snsh42w8Mv/Zhxw9iD7Cjo7CyQC/KyF6oBfShL4PYgG+CHsYKyZxvxZSZBAgjhIz YDz+AbfD37GtbaoSKzgGt+lKfI/GZtay6vG4VXTpPsh+IcQhOk9I7WYeP5AM3HZ9+DaSGIp9HIuu huC4pCGGx+YD2fcc5tfIAA0h/Et4/jX8zz4fWAasIf4UbR9Y6HO4d8jAnd4U0Y8ageuCB+c+H5R3 CHU2mTMwZ+B/y8DfSMBLLOYXVuEAAAAASUVORK5CYII= ------=_Part_686_1791755185.1402602528472-- ------=_Part_685_1623081397.1402602528472--