From: Tim Cross <theophilusx@gmail.com>
To: padusuma <venkata.adusumalli@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Performance of INSERT into temporary tables using psqlODBC driver
Date: Mon, 10 Sep 2018 08:26:06 +1000
Message-ID: <87zhwq48ap.fsf@gmail.com> (raw)
In-Reply-To: <1536385261054-0.post@n3.nabble.com>
References: <1536339842382-0.post@n3.nabble.com>
<8736uk6eh5.fsf@gmail.com>
<1536385261054-0.post@n3.nabble.com>
padusuma <venkata.adusumalli@gmail.com> writes:
>>We are inserting large numbers (millions) of rows into a postgres
>>database from a Javascript application and found using the COPY command
>>was much, much faster than doing regular inserts (even with multi-insert
>>commit). If you can do this using the driver you are using, that will
>>give you the largest performance boost.
>
> The data to be inserted into temporary tables is obtained from one or more
> queries run earlier and the data is available as a vector of strings. If I
> need to use COPY FROM command, then the application would need to create a
> file with the data to be inserted and the file needs to be readable by the
> user running database server process, which may not be always possible
> unless the application is running on the same host. I think this approach
> may not be feasible for our application.
>
OK, that does make a difference. If your data is already in the
database, COPY is not going to help you much.
> I have increased the value for /temp_buffers/ server parameter from the
> default 8 MB to 128 MB. However, this change did not affect the INSERT time
> for temporary tables.
It isn't clear why you create vectors of strings rather than just select
into or something similar.
There are no 'quick fixes' which can be applied without real analysis of
the system. However, based on the limited information available, you may
want to consider -
- Increase work_mem to reduce use of temp files. Need it to be 2 to 3
times largest temp file (but use common sense)
- Tweak wal checkpoint parameters to prevent wal checkpoints occurring
too frequently. Note that there is a play off here between frequency
of checkpoints and boot time after a crash. Fewer wal checkpoints will
usually improve performance, but recovery time is longer.
- Verify your inserts into temporary tables is the bottleneck and not
the select from existing data (explain plan etc and adjust indexes
accordingly).
How effectively you can increase insert times will depend on what the
memory and cpu profile of the system is. More memory, less use of temp
files, faster system, so spend a bit of time to make sure your system is
configured to squeeze as much out of that RAM as you can!
--
Tim Cross
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: theophilusx@gmail.com, venkata.adusumalli@gmail.com
Subject: Re: Performance of INSERT into temporary tables using psqlODBC driver
In-Reply-To: <87zhwq48ap.fsf@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox