public inbox for [email protected]  

help / color / mirror / Atom feed

From: Josef Šimánek <[email protected]>
To: sud <[email protected]>
Cc: pgsql-general <[email protected]>
Subject: Re: Load a csv or a avro?
Date: Fri, 5 Jul 2024 12:02:50 +0200
Message-ID: <CAFp7QwoNgw_3rGsJF1nK5Mzceg2_+UL0-9pJ2WRCw04O71S4zA@mail.gmail.com> (raw)
In-Reply-To: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>
References: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>

pá 5. 7. 2024 v 11:08 odesílatel sud <[email protected]> napsal:
>
> Hello all,
>
> Its postgres database. We have option of getting files in csv and/or in avro format messages from another system to load it into our postgres database. The volume will be 300million messages per day across many files in batches.
>
> My question was, which format should we chose in regards to faster data loading performance ? and if any other aspects to it also should be considered apart from just loading performance?

We are able to load ~300 million rows per one day using CSV and COPY
functions (https://www.postgresql.org/docs/current/libpq-copy.html#LIBPQ-COPY-SEND).

view thread (3+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: Load a csv or a avro?
  In-Reply-To: <CAFp7QwoNgw_3rGsJF1nK5Mzceg2_+UL0-9pJ2WRCw04O71S4zA@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox