public inbox for [email protected]  

help / color / mirror / Atom feed

From: Lukáš Sobotka <[email protected]>
To: Tom Lane <[email protected]>
Cc: [email protected]
Subject: Re: Using COPY command in pqlib
Date: Fri, 19 Jul 2019 12:34:19 +0200
Message-ID: <CAJif3kJdz6yx+31DSnD1eq_cNF0tccNBG2-FfSo4Y7rQv3L3QQ@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <CAJif3k+Q_nksEkS3SwpaButxxTcOkjgG91_sLLcc=986-2VZAg@mail.gmail.com>
	<[email protected]>

Hi Tom,

I see we are not using copy in correct way.

Many thanks for your time.
Lukas

pá 19. 7. 2019 v 0:26 odesílatel Tom Lane <[email protected]> napsal:

> =?UTF-8?B?THVrw6HFoSBTb2JvdGth?= <[email protected]> writes:
> > I would like to ask you on correct way how to use pqlib for copying data.
> > We tried to do it according documentation but some problems occurred.
>
> > We have a multi-threaded application, all threads hold own non-blocking
> > connection into the remote database. Each thread is collecting data and
> > when a buffer is filled they copy data to remote DB as quick as possible.
> > Copying can run several times per minute. The postgresql related part of
> > thread main loop is briefly shown in attached file main.c.
>
> > Most of the time copying works well but big delay sometimes occurred. I
> > captured pcap (see https://ibb.co/2MhmSTZ) and the delay is between
> getting
> > PGRES_COPY_IN from PQresultStatus function and pushing data. But I am
> sure
> > that commands PQputCopyData and PQputCopyEnd are called immediately after
> > client gets PQexec.
>
> > What am I missing? Is there some other command which we should use?
>
> Well, you didn't show us any of the relevant parts of your code.
> There is extra stuff you need to deal with if you're running the
> connection in nonblock mode.
>
> --- what are you doing when PQputCopyData returns zero?
>
> The documentation says you should wait for write-ready and try again.
> (You could just try again immediately, I suppose, but that would make it
> a busy-wait loop which doesn't seem like a great idea.)
>
> --- what are you doing after PQputCopyEnd?
>
> The documentation says
>
>     The result is 1 if the termination message was sent; or in nonblocking
>     mode, this may only indicate that the termination message was
>     successfully queued. (In nonblocking mode, to be certain that the data
>     has been sent, you should next wait for write-ready and call PQflush,
>     repeating until it returns zero.) Zero indicates that the function
>     could not queue the termination message because of full buffers; this
>     will only happen in nonblocking mode. (In this case, wait for
>     write-ready and try the PQputCopyEnd call again.) If a hard error
>     occurs, -1 is returned; you can use PQerrorMessage to retrieve
>     details.
>
>                         regards, tom lane
>

view thread (3+ messages)

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected]
  Subject: Re: Using COPY command in pqlib
  In-Reply-To: <CAJif3kJdz6yx+31DSnD1eq_cNF0tccNBG2-FfSo4Y7rQv3L3QQ@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox