MIME-Version: 1.0
References: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>
 <CAFp7QwoNgw_3rGsJF1nK5Mzceg2_+UL0-9pJ2WRCw04O71S4zA@mail.gmail.com>
In-Reply-To: <CAFp7QwoNgw_3rGsJF1nK5Mzceg2_+UL0-9pJ2WRCw04O71S4zA@mail.gmail.com>
From: Muhammad Ikram <mmikram@gmail.com>
Date: Fri, 5 Jul 2024 15:15:28 +0500
Message-ID: <CAGeimVo_EOJO+BViDnLoxEdFoFKQkeHU=gniEQ9e2GbjAUvUHg@mail.gmail.com>
Subject: Re: Load a csv or a avro?
To: =?UTF-8?B?Sm9zZWYgxaBpbcOhbmVr?= <josef.simanek@gmail.com>
Cc: sud <suds1434@gmail.com>, pgsql-general <pgsql-general@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="00000000000008e4c0061c7d5971"
Archived-At: <https://www.postgresql.org/message-id/CAGeimVo_EOJO%2BBViDnLoxEdFoFKQkeHU%3DgniEQ9e2GbjAUvUHg%40mail.gmail.com>
Precedence: bulk

--00000000000008e4c0061c7d5971
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi,

Performance Considerations
    Avro files are smaller due to compression so needing less I/O time.
whereas CSV files are simpler but larger in size so read/write will need
more time.
    COPY command works very well with CSV files whereas ETL process is
required for handling Avro.

Regards,
Muhammad Ikram


On Fri, Jul 5, 2024 at 3:03=E2=80=AFPM Josef =C5=A0im=C3=A1nek <josef.siman=
ek@gmail.com>
wrote:

> p=C3=A1 5. 7. 2024 v 11:08 odes=C3=ADlatel sud <suds1434@gmail.com> napsa=
l:
> >
> > Hello all,
> >
> > Its postgres database. We have option of getting files in csv and/or in
> avro format messages from another system to load it into our postgres
> database. The volume will be 300million messages per day across many file=
s
> in batches.
> >
> > My question was, which format should we chose in regards to faster data
> loading performance ? and if any other aspects to it also should be
> considered apart from just loading performance?
>
> We are able to load ~300 million rows per one day using CSV and COPY
> functions (
> https://www.postgresql.org/docs/current/libpq-copy.html#LIBPQ-COPY-SEND).
>
>
>

--=20
Muhammad Ikram

--00000000000008e4c0061c7d5971
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div><h3>Performance Considerations</h3>=
<div>=C2=A0 =C2=A0 Avro files are smaller due to compression so needing les=
s I/O time. whereas CSV files are simpler but larger in size so read/write =
will need more time.</div><div>=C2=A0 =C2=A0 COPY command works very well w=
ith CSV files whereas ETL process is required for handling Avro.</div><div>=
<br></div><div>Regards,</div><div>Muhammad Ikram</div><div><br></div><h3></=
h3></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gma=
il_attr">On Fri, Jul 5, 2024 at 3:03=E2=80=AFPM Josef =C5=A0im=C3=A1nek &lt=
;<a href=3D"mailto:josef.simanek@gmail.com">josef.simanek@gmail.com</a>&gt;=
 wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px =
0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">p=C3=A1 =
5. 7. 2024 v 11:08 odes=C3=ADlatel sud &lt;<a href=3D"mailto:suds1434@gmail=
.com" target=3D"_blank">suds1434@gmail.com</a>&gt; napsal:<br>
&gt;<br>
&gt; Hello all,<br>
&gt;<br>
&gt; Its postgres database. We have option of getting files in csv and/or i=
n avro format messages from another system to load it into our postgres dat=
abase. The volume will be 300million messages per day across many files in =
batches.<br>
&gt;<br>
&gt; My question was, which format should we chose in regards to faster dat=
a loading performance ? and if any other aspects to it also should be consi=
dered apart from just loading performance?<br>
<br>
We are able to load ~300 million rows per one day using CSV and COPY<br>
functions (<a href=3D"https://www.postgresql.org/docs/current/libpq-copy.ht=
ml#LIBPQ-COPY-SEND" rel=3D"noreferrer" target=3D"_blank">https://www.postgr=
esql.org/docs/current/libpq-copy.html#LIBPQ-COPY-SEND</a>).<br>
<br>
<br>
</blockquote></div><br clear=3D"all"><div><br></div><span class=3D"gmail_si=
gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><d=
iv dir=3D"ltr"><div>Muhammad Ikram<br><br></div></div></div>

--00000000000008e4c0061c7d5971--