public inbox for [email protected]  

help / color / mirror / Atom feed

From: Kashif Zeeshan <[email protected]>
To: sud <[email protected]>
Cc: pgsql-general <[email protected]>
Subject: Re: Load a csv or a avro?
Date: Fri, 5 Jul 2024 14:57:37 +0500
Message-ID: <CAAPsdhd=JfvVaNMnJ1eUAuBts3664Q0r8JE-OBijvEuTQO4P3w@mail.gmail.com> (raw)
In-Reply-To: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>
References: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>

Hi

There are different data formats available, following are few points for
there performance implications

1. CSV : It's easy to use and widely supported but it can be slower due to
parsing overload.
2. Binary : Its faster to load but not human understandable.

Hope this helps.

Regards
Kashif Zeeshan

On Fri, Jul 5, 2024 at 2:08 PM sud <[email protected]> wrote:

> Hello all,
>
> Its postgres database. We have option of getting files in csv and/or in
> avro format messages from another system to load it into our postgres
> database. The volume will be 300million messages per day across many files
> in batches.
>
> My question was, which format should we chose in regards to faster data
> loading performance ? and if any other aspects to it also should be
> considered apart from just loading performance?
>

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: Load a csv or a avro?
  In-Reply-To: <CAAPsdhd=JfvVaNMnJ1eUAuBts3664Q0r8JE-OBijvEuTQO4P3w@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox