public inbox for [email protected]  

help / color / mirror / Atom feed

From: Vladimir Ryabtsev <[email protected]>
To: Adrian Klaver <[email protected]>
Cc: [email protected]
Subject: Re: Memory
Date: Sat, 21 Dec 2024 13:52:40 -0800
Message-ID: <CAMqTPqktrwkxn_11wPXfJRWoe=KgLTXo7j9U3Xxj9UBPYaKf-Q@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <CAMqTPqn8y8Y+uDY0FPvX6ghD1DftLyz2nD6n6HhGOg-gHP4JdA@mail.gmail.com>
	<[email protected]>

Now I see, the doc is already great.
Thanks for pointing out Adrian.


On Sat, Dec 21, 2024 at 8:34 AM Adrian Klaver <[email protected]>
wrote:

> On 12/21/24 02:45, Vladimir Ryabtsev wrote:
> > Hi community,
> >
> > I am reading a big dataset using code similar to this:
> >
> > query = '''
> > SELECT timestamp, data_source, tag, agg_value
> > FROM my_table
> > '''I
> > batch_size = 10_000_000
> >
> > with psycopg.connect(cs, cursor_factory=psycopg.ClientrCursor) as conn:
>
> FYI, ClientCursor.
>
>
> >
> > I looked the documentation, but did not find specifics related to
> > performance differences between Server and Client cursors.
> >
> > I am fine with ServerCursor, but I need to ask, is it by design that
> > with ClientCursor the result set is copied into memory despite
> > fetchmany() limit? ClientCursor is the default class, so may be worth
> > documenting the difference (sorry, if I missed that).
>
> Client side cursor
>
>
> https://www.psycopg.org/psycopg3/docs/advanced/cursors.html#client-side-cursors
>
> "In such querying pattern, after a cursor sends a query to the server
> (usually calling execute()), the server replies transferring to the
> client the whole set of results requested, which is stored in the state
> of the same cursor and from where it can be read from Python code (using
> methods such as fetchone() and siblings)."
>
>
> https://www.psycopg.org/psycopg3/docs/api/cursors.html#psycopg.Cursor.fetchmany
>
> "fetchmany(size: int = 0) → list[+Row]
>
>      Return the next size records from the current recordset.
>
>      size default to self.arraysize if not specified.
>
>      Return type:
>
>          Sequence[Row], with Row defined by row_factory
>
> "
>
>
> Server side cursor
>
>
> https://www.psycopg.org/psycopg3/docs/advanced/cursors.html#server-side-cursors
>
> "PostgreSQL has its own concept of cursor too (sometimes also called
> portal). When a database cursor is created, the query is not necessarily
> completely processed: the server might be able to produce results only
> as they are needed. Only the results requested are transmitted to the
> client: if the query result is very large but the client only needs the
> first few records it is possible to transmit only them.
>
> The downside is that the server needs to keep track of the partially
> processed results, so it uses more memory and resources on the server."
>
> >
> > Thank you.
> >
>
> --
> Adrian Klaver
> [email protected]
>
>

view thread (3+ messages)

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected]
  Subject: Re: Memory
  In-Reply-To: <CAMqTPqktrwkxn_11wPXfJRWoe=KgLTXo7j9U3Xxj9UBPYaKf-Q@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox