MIME-Version: 1.0
References: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>
 <11d5753c-578e-41a2-af17-6de956f03058@aklaver.com>
In-Reply-To: <11d5753c-578e-41a2-af17-6de956f03058@aklaver.com>
From: sud <suds1434@gmail.com>
Date: Sun, 7 Jul 2024 01:39:48 +0530
Message-ID: <CAD=mzVXSeb3dG-=aHfpOo2oP7Od7CmnhdVc2+pDCRdSysBa-6A@mail.gmail.com>
Subject: Re: Load a csv or a avro?
To: Adrian Klaver <adrian.klaver@aklaver.com>
Cc: pgsql-general <pgsql-general@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000c2a018061c99c314"
Archived-At: <https://www.postgresql.org/message-id/CAD%3DmzVXSeb3dG-%3DaHfpOo2oP7Od7CmnhdVc2%2BpDCRdSysBa-6A%40mail.gmail.com>
Precedence: bulk

--000000000000c2a018061c99c314
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Jul 5, 2024 at 8:24=E2=80=AFPM Adrian Klaver <adrian.klaver@aklaver=
.com>
wrote:

> On 7/5/24 02:08, sud wrote:
> > Hello all,
> >
> > Its postgres database. We have option of getting files in csv and/or in
> > avro format messages from another system to load it into our postgres
> > database. The volume will be 300million messages per day across many
> > files in batches.
>
> Are dumping the entire contents of each file or are you pulling a
> portion of the data out?
>
>
>
Yes, all the fields in the file have to be loaded to the columns in the
tables in postgres. But how will that matter here for deciding if we should
ask the data in .csv or .avro format from the outside system to load into
the postgres database in row and column format? Again my understanding was
that irrespective of anything , the .csv file load will always faster as
because the data is already stored in row and column format as compared to
the .avro file in which the parser has to perform additional job to make it
row and column format or map it to the columns of the database table. Is my
understanding correct here?

--000000000000c2a018061c99c314
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">On Fri, Jul 5, 2024 at 8:24=E2=80=AFPM Ad=
rian Klaver &lt;<a href=3D"mailto:adrian.klaver@aklaver.com">adrian.klaver@=
aklaver.com</a>&gt; wrote:<br></div><div class=3D"gmail_quote"><blockquote =
class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px sol=
id rgb(204,204,204);padding-left:1ex">On 7/5/24 02:08, sud wrote:<br>
&gt; Hello all,<br>
&gt; <br>
&gt; Its postgres database. We have option of getting files in csv and/or i=
n <br>
&gt; avro format messages from another system to load it into our postgres =
<br>
&gt; database. The volume will be 300million messages per day across many <=
br>
&gt; files in batches.<br>
<br>
Are dumping the entire contents of each file or are you pulling a <br>
portion of the data out?<br><br>
<br></blockquote><div><br></div><div>Yes, all the fields in the file have t=
o be loaded to the columns in the tables in postgres. But how will that mat=
ter here for deciding if we should ask the=C2=A0data in .csv or .avro forma=
t from the outside system to load into the postgres database in=C2=A0row an=
d column format? Again my understanding=C2=A0was that irrespective=C2=A0of =
anything , the .csv file load will always faster as because the data is alr=
eady=C2=A0stored in row and column format as compared=C2=A0to the .avro fil=
e in which the parser has to perform additional=C2=A0job to make it row and=
 column format or map it to the columns of the database table. Is my unders=
tanding=C2=A0correct here?</div></div></div>

--000000000000c2a018061c99c314--