Received: from makus.postgresql.org (makus.postgresql.org [98.129.198.125])
	by mail.postgresql.org (Postfix) with ESMTP id 8F53D1CF17EB
	for <pgsql-performance@postgresql.org>;
	Tue, 13 Mar 2012 03:30:19 -0300 (ADT)
Received: from [90.184.12.24] (helo=shrek.krogh.cc)
	by makus.postgresql.org with esmtp (Exim 4.72)
	(envelope-from <jesper@krogh.cc>) id 1S7LFR-0001tX-Uk
	for pgsql-performance@postgresql.org; Tue, 13 Mar 2012 06:30:19 +0000
Received: from localhost (localhost [127.0.0.1])
	by shrek.krogh.cc (Postfix) with ESMTP id 3883C1AEAF
	for <pgsql-performance@postgresql.org>;
	Tue, 13 Mar 2012 07:30:03 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at shrek.krogh.cc
Received: from shrek.krogh.cc ([127.0.0.1])
	by localhost (shrek.krogh.cc [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id gybL-pg8AF-M for <pgsql-performance@postgresql.org>;
	Tue, 13 Mar 2012 07:29:58 +0100 (CET)
Received: from [90.184.13.151] (unknown [90.184.13.151])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	(Authenticated sender: jesper@krogh.cc)
	by shrek.krogh.cc (Postfix) with ESMTPSA id 34DAB1957D
	for <pgsql-performance@postgresql.org>;
	Tue, 13 Mar 2012 07:29:58 +0100 (CET)
Message-ID: <4F5EE965.8010706@krogh.cc>
Date: Tue, 13 Mar 2012 07:29:57 +0100
From: Jesper Krogh <jesper@krogh.cc>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: pgsql-performance@postgresql.org
Subject: Re: Gin index insert performance issue
References: <4F5EDE67.3040908@gmail.com>
In-Reply-To: <4F5EDE67.3040908@gmail.com>
Content-Type: multipart/alternative;
	boundary="------------040709090005080403090609"
X-Host-Lookup-Failed: Reverse DNS lookup failed for 90.184.12.24 (failed)
X-Pg-Spam-Score: -1.1 (-)
X-Archive-Number: 201203/103
X-Sequence-Number: 46553

This is a multi-part message in MIME format.
--------------040709090005080403090609
Content-Type: text/plain; charset=gb18030; format=flowed
Content-Transfer-Encoding: 7bit

On 13/03/12 06:43, Rural Hunter wrote:
>  I tried to increase work_mem but the inserts hang more time each time
>  with less frequency. So it makes almost no difference for the total
>  hanging time. Frequent vacuum is not a choice since the hang happens
>  very 3-5 mins. is there any improvement I can make with pg for such
>  data volumn(still increasing) or it's time to turn to other full text
>  search solution such as lucene etc?

We're using gin for fts-search, current index-size is up to 149GB and yes
the update process is quite tough on the disk-io-subsystem.

What you're experiencing is filling of the fastupdate queue, thats being
flushed. Setting wok_mem higher causes the system to stall for longer
period less frequent and has a side cost on queries that need to go through
the pending list (that is bigger) in addition to the index-search. To me
it seems like all other writing/updating processes are being stalled when
the pending list is flushed, but I am not sure about the specifice here.

Our solution is to turn "fastupdate" off for our gin-indices.
http://www.postgresql.org/docs/9.0/static/sql-createindex.html
Can also be set with ALTER TABLE ALTER INDEX

I would have preferred a "backend local" batch-update process so it
could batch up everything from its own transaction instead of interferring
with other transactions.

I would say, that we came from Xapian and the PG-index is a way better
fit for our application. The benefits of having the fts index next to 
all the
other data saves a significant amount of development time in the application
both in terms of development and maintaince. (simpler, easier and more 
manageble).

-- 
Jesper


--------------040709090005080403090609
Content-Type: text/html; charset=gb18030
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <meta content="text/html; charset=gb18030" http-equiv="Content-Type">
  </head>
  <body bgcolor="#ffffff" text="#000000">
    On 13/03/12 06:43, Rural Hunter wrote:<br>
    <span style="white-space: pre;">&gt; I tried to increase work_mem
      but the inserts hang more time each time<br>
      &gt; with less frequency. So it makes almost no difference for the
      total<br>
      &gt; hanging time. Frequent vacuum is not a choice since the hang
      happens<br>
      &gt; very 3-5 mins. is there any improvement I can make with pg
      for such<br>
      &gt; data volumn(still increasing) or it's time to turn to other
      full text<br>
      &gt; search solution such as lucene etc?</span><br>
    <br>
    We're using gin for fts-search, current index-size is up to 149GB
    and yes<br>
    the update process is quite tough on the disk-io-subsystem. <br>
    <br>
    What you're experiencing is filling of the fastupdate queue, thats
    being <br>
    flushed. Setting wok_mem higher causes the system to stall for
    longer<br>
    period less frequent and has a side cost on queries that need to go
    through<br>
    the pending list (that is bigger) in addition to the index-search.
    To me<br>
    it seems like all other writing/updating processes are being stalled
    when <br>
    the pending list is flushed, but I am not sure about the specifice
    here. <br>
    <br>
    Our solution is to turn "fastupdate" off for our gin-indices. <br>
    <a class="moz-txt-link-freetext" href="http://www.postgresql.org/docs/9.0/static/sql-createindex.html">http://www.postgresql.org/docs/9.0/static/sql-createindex.html</a><br>
    Can also be set with ALTER TABLE ALTER INDEX<br>
    <br>
    I would have preferred a "backend local" batch-update process so it
    <br>
    could batch up everything from its own transaction instead of
    interferring<br>
    with other transactions. <br>
    <br>
    I would say, that we came from Xapian and the PG-index is a way
    better<br>
    fit for our application. The benefits of having the fts index next
    to all the<br>
    other data saves a significant amount of development time in the
    application<br>
    both in terms of development and maintaince. (simpler, easier and
    more manageble). <br>
    <br>
    -- <br>
    Jesper<br>
    <br>
  </body>
</html>

--------------040709090005080403090609--