MIME-Version: 1.0
References: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>
In-Reply-To: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>
From: Kashif Zeeshan <kashi.zeeshan@gmail.com>
Date: Fri, 5 Jul 2024 14:57:37 +0500
Message-ID: <CAAPsdhd=JfvVaNMnJ1eUAuBts3664Q0r8JE-OBijvEuTQO4P3w@mail.gmail.com>
Subject: Re: Load a csv or a avro?
To: sud <suds1434@gmail.com>
Cc: pgsql-general <pgsql-general@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000952aeb061c7d18e5"
Archived-At: <https://www.postgresql.org/message-id/CAAPsdhd%3DJfvVaNMnJ1eUAuBts3664Q0r8JE-OBijvEuTQO4P3w%40mail.gmail.com>
Precedence: bulk

--000000000000952aeb061c7d18e5
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi

There are different data formats available, following are few points for
there performance implications

1. CSV : It's easy to use and widely supported but it can be slower due to
parsing overload.
2. Binary : Its faster to load but not human understandable.

Hope this helps.

Regards
Kashif Zeeshan

On Fri, Jul 5, 2024 at 2:08=E2=80=AFPM sud <suds1434@gmail.com> wrote:

> Hello all,
>
> Its postgres database. We have option of getting files in csv and/or in
> avro format messages from another system to load it into our postgres
> database. The volume will be 300million messages per day across many file=
s
> in batches.
>
> My question was, which format should we chose in regards to faster data
> loading performance ? and if any other aspects to it also should be
> considered apart from just loading performance?
>

--000000000000952aeb061c7d18e5
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi<div><br></div><div>There are different data formats ava=
ilable, following are few points for there performance implications</div><d=
iv><br></div><div>1. CSV : It&#39;s easy to use and widely=C2=A0supported b=
ut it can be slower due to parsing overload.</div><div>2. Binary : Its fast=
er=C2=A0to load but not human understandable.</div><div><br></div><div>Hope=
 this=C2=A0helps.</div><div><br></div><div>Regards</div><div>Kashif Zeeshan=
</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_=
attr">On Fri, Jul 5, 2024 at 2:08=E2=80=AFPM sud &lt;<a href=3D"mailto:suds=
1434@gmail.com">suds1434@gmail.com</a>&gt; wrote:<br></div><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid r=
gb(204,204,204);padding-left:1ex"><div dir=3D"auto"><p dir=3D"ltr">Hello al=
l,=C2=A0</p>
<p dir=3D"ltr">Its postgres database. We have option of getting files in cs=
v and/or in avro format messages from another system to load it into our po=
stgres database. The volume will be 300million messages per day across many=
 files in batches. </p>
<p dir=3D"ltr"><span style=3D"font-size:1em">My question was, which format =
should we chose in regards to faster data loading performance ? and if any =
other aspects to it also should be considered apart from just loading perfo=
rmance?</span></p></div>
</blockquote></div>

--000000000000952aeb061c7d18e5--