X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id A50A23A46A9 for ; Thu, 18 Nov 2004 16:43:25 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 50398-08 for ; Thu, 18 Nov 2004 16:43:23 +0000 (GMT) Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by svr1.postgresql.org (Postfix) with ESMTP id DC3143A46DC for ; Thu, 18 Nov 2004 16:43:19 +0000 (GMT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id iAIGhC65017520; Thu, 18 Nov 2004 11:43:12 -0500 (EST) To: "David Parker" Cc: pgsql-hackers@postgreSQL.org, Jan Wieck , "Matthew T. O'Connor" Subject: Timing of pgstats updates In-reply-to: <07FDEE0ED7455A48AC42AC2070EDFF7C26BAC3@corpsrv2.tazznetworks.com> References: <07FDEE0ED7455A48AC42AC2070EDFF7C26BAC3@corpsrv2.tazznetworks.com> Comments: In-reply-to "David Parker" message dated "Thu, 18 Nov 2004 11:12:12 -0500" Date: Thu, 18 Nov 2004 11:43:12 -0500 Message-ID: <17519.1100796192@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200411/639 X-Sequence-Number: 61211 "David Parker" writes: > What I think is happening with the missing pg_statistic entries: > The install of our application involves a lot of data importing (via > JDBC) in one large transaction, which can take up to 30 minutes. (I > realize I left out this key piece of info in my original post...) > The pg_autovacuum logic is relying on data from pg_stat_all_tables to > make the decision about running analyze. As far as I can tell, the data > in this view gets updated outside of the transaction, because I saw the > numbers growing while I was importing. I saw pg_autovacuum log messages > for running analyze on several tables, but no statistics data showed up > for these, I assume because the actual data in the table wasn't yet > visible to pg_autovacuum because the import transaction had not finished > yet. > When the import finished, not all of the tables affected by the import > were re-visited because they had not bumped up over the threshold again, > even though the analyze run for those tables had not generated any stats > because of the still-open transaction. Bingo. The per-table activity stats are sent to the collector whenever the backend waits for a client command. Given a moderately long transaction block doing updates, it's not hard at all to imagine that autovacuum would kick off vacuum and/or analyze while the updating transaction is still in progress. The resulting operation is of course a waste of time. It'd be trivial to adjust postgres.c so that per-table stats are only transmitted when we exit the transaction (basically move the pgstat_report_tabstat call down a couple lines so it's not called if IsTransactionOrTransactionBlock). This seems like a good change to me. Does anyone not like it? regards, tom lane