X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [64.117.225.181]) by svr1.postgresql.org (Postfix) with ESMTP id 21A61CC084 for ; Mon, 14 Jul 2003 16:52:57 +0000 (GMT) Received: from svr1.postgresql.org ([64.117.224.193]) by localhost (svr5.postgresql.org [64.117.225.181]) (amavisd-new, port 10024) with ESMTP id 36254-01 for ; Mon, 14 Jul 2003 16:53:21 +0000 (GMT) Received: from martin.sysdetect.com (martin.sysdetect.com [65.209.102.1]) by svr1.postgresql.org (Postfix) with ESMTP id 90497CC0C1 for ; Mon, 14 Jul 2003 13:52:54 -0300 (ADT) Received: (from mail@localhost) by martin.sysdetect.com (8.11.4/8.11.3) id h6EGrLx12268 for ; Mon, 14 Jul 2003 16:53:21 GMT Received: from winwood.sysdetect.com(172.16.1.1) via SMTP by mail.sysdetect.com, id smtpds14224; Mon Jul 14 16:53:19 2003 Received: from sysd.com (thriller.sysdetect.com [172.16.1.33]) by winwood.sysdetect.com (8.11.6/8.11.6) with ESMTP id h6EGrI921571 for ; Mon, 14 Jul 2003 12:53:19 -0400 Message-ID: <3F12DFFE.3090207@sysd.com> Date: Mon, 14 Jul 2003 12:53:18 -0400 From: Rich Cullingford User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3) Gecko/20030314 X-Accept-Language: en-us, en MIME-Version: 1.0 To: pgsql-performance@postgresql.org Subject: Java Out-of-memory errors on attempts to read tables with millions of rows Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200307/222 X-Sequence-Number: 2515 Greetings, We have several tables (in a PG 7.3.3 database on RH Linux 7.3) with 2M+ rows (each row 300-400 bytes in length) that we SELECT into a JDBC ResultSet for display to the user. We expected that the driver would not actually transmit data from the database until the application began issuing getXXX() calls. (IIRC, this is the way the Oracle driver works, and we had created a buffering mechanism to use it.) Instead, the driver appears to be attempting to create the whole rowset in Java memory before returning, and the application runs out of memory. (Java has been configured to use up to 1.5G on the machine this occurs on.) Now the SELECT is preceded by a COUNT of the rows that the same query would return, so perhaps that's what's causing the problem. But the question is, is this the way a ResultSet is supposed to work? Are there any configuration options available that modify this behavior? Are there commercial implementations of PG JDBC that don't have this problem? (Shame on me, but I have to ask. :) Any help will be greatly appreciated! Rich Cullingford rculling@sysd.com