References: <1536339842382-0.post@n3.nabble.com> <8736uk6eh5.fsf@gmail.com>
 <1536385261054-0.post@n3.nabble.com>
User-agent: mu4e 0.9.18; emacs 26.1
From: Tim Cross <theophilusx@gmail.com>
To: padusuma <venkata.adusumalli@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Performance of INSERT into temporary tables using psqlODBC driver
In-reply-to: <1536385261054-0.post@n3.nabble.com>
Date: Mon, 10 Sep 2018 08:26:06 +1000
Message-ID: <87zhwq48ap.fsf@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk


padusuma <venkata.adusumalli@gmail.com> writes:

>>We are inserting large numbers (millions) of rows into a postgres
>>database from a Javascript application and found using the COPY command
>>was much, much faster than doing regular inserts (even with multi-insert
>>commit). If you can do this using the driver you are using, that will
>>give you the largest performance boost.
>
> The data to be inserted into temporary tables is obtained from one or more
> queries run earlier and the data is available as a vector of strings. If I
> need to use COPY FROM command, then the application would need to create a
> file with the data to be inserted and the file needs to be readable by the
> user running database server process, which may not be always possible
> unless the application is running on the same host. I think this approach
> may not be feasible for our application.
>

OK, that does make a difference. If your data is already in the
database, COPY is not going to help you much.

> I have increased the value for /temp_buffers/ server parameter from the
> default 8 MB to 128 MB. However, this change did not affect the INSERT time
> for temporary tables.

It isn't clear why you create vectors of strings rather than just select
into or something similar.

There are no 'quick fixes' which can be applied without real analysis of
the system. However, based on the limited information available, you may
want to consider -

- Increase work_mem to reduce use of temp files. Need it to be 2 to 3
  times largest temp file (but use common sense)

- Tweak wal checkpoint parameters to prevent wal checkpoints occurring
  too frequently. Note that there is a play off here between frequency
  of checkpoints and boot time after a crash. Fewer wal checkpoints will
  usually improve performance, but recovery time is longer.

- Verify your inserts into temporary tables is the bottleneck and not
  the select from existing data (explain plan etc and adjust indexes
  accordingly).

How effectively you can increase insert times will depend on what the
memory and cpu profile of the system is. More memory, less use of temp
files, faster system, so spend a bit of time to make sure your system is
configured to squeeze as much out of that RAM as you can!

--
Tim Cross