MIME-Version: 1.0
References: 
 <CAEphvJPSh1Cd+NA14Q3e-1B9JH6Ebm1oLgLx93HkkTZa=EBA_g@mail.gmail.com>
 <0137F599-5558-4B33-9533-94D9E28B7CC2@icloud.com>
In-Reply-To: <0137F599-5558-4B33-9533-94D9E28B7CC2@icloud.com>
From: Anex Hul <anexsql2014@gmail.com>
Date: Mon, 16 Dec 2024 08:05:12 -0600
Message-ID: 
 <CAEphvJPsxa5fsU0vFqdAz547URuho3HMZ=g1Sw75U09OOL=pDg@mail.gmail.com>
Subject: Re: Performance
To: Rui DeSousa <rui.desousa@icloud.com>
Cc: pgsql-admin@lists.postgresql.org
Content-Type: multipart/alternative; boundary="000000000000f16ba4062963ab28"
Archived-At: 
 <https://www.postgresql.org/message-id/CAEphvJPsxa5fsU0vFqdAz547URuho3HMZ%3Dg1Sw75U09OOL%3DpDg%40mail.gmail.com>
Precedence: bulk

--000000000000f16ba4062963ab28
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thank you all for your response.

Show quoted text
1. What version of PG is it?  ("SELECT VERSION();" should tell you.)

PG Version 16

2. Are you truncating the table after each test run, or deleting all
records, or appending?

created new schema for each run.

3. Is the blob data stored in BYTEA column data, or are you using the
(discouraged) "Large Objects"?

Blob storage

4. How are you loading the blob data?

used the Import data using a COPY statement, followed this doc

https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-u=
se-pg-azure-storage?tabs=3Dportal-01%2Cportal-02%2Cportal-03%2Cportal-04%2C=
portal-05

On Sun, Dec 15, 2024, 10:22=E2=80=AFPM Rui DeSousa <rui.desousa@icloud.com>=
 wrote:

>
>
> > On Dec 15, 2024, at 8:22 PM, Anex Hul <anexsql2014@gmail.com> wrote:
> >
> > Hello everyone,
> >
> > Testing 100 million records data import from Azure blob storage to Azur=
e
> postgresql. I did run the test 5 times and the time it took keep increasi=
ng
> for each run.
> > Is there know justification for this linear increment of the time it
> took for same size of data?
>
> Check you I/O quotas; you might have hit quota limits and being throttled=
.

--000000000000f16ba4062963ab28
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">Thank you all for your response.</p>
<p dir=3D"ltr">Show quoted text<br>
1. What version of PG is it?=C2=A0 (&quot;SELECT VERSION();&quot; should te=
ll you.)<br></p>
<p dir=3D"ltr">PG Version 16<br></p>
<p dir=3D"ltr">2. Are you truncating the table after each test run, or dele=
ting all records, or appending?</p>
<p dir=3D"ltr">created new schema for each run.<br></p>
<p dir=3D"ltr">3. Is the blob data stored in BYTEA column data, or are you =
using the (discouraged) &quot;Large Objects&quot;?</p>
<p dir=3D"ltr">Blob storage </p>
<p dir=3D"ltr">4. How are you loading the blob data?<br></p>
<p dir=3D"ltr">used the Import data using a COPY statement, followed this d=
oc</p>
<p dir=3D"ltr"><a href=3D"https://learn.microsoft.com/en-us/azure/postgresq=
l/flexible-server/how-to-use-pg-azure-storage?tabs=3Dportal-01%2Cportal-02%=
2Cportal-03%2Cportal-04%2Cportal-05">https://learn.microsoft.com/en-us/azur=
e/postgresql/flexible-server/how-to-use-pg-azure-storage?tabs=3Dportal-01%2=
Cportal-02%2Cportal-03%2Cportal-04%2Cportal-05</a> </p>
<br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" class=
=3D"gmail_attr">On Sun, Dec 15, 2024, 10:22=E2=80=AFPM Rui DeSousa &lt;<a h=
ref=3D"mailto:rui.desousa@icloud.com">rui.desousa@icloud.com</a>&gt; wrote:=
<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord=
er-left:1px #ccc solid;padding-left:1ex"><br>
<br>
&gt; On Dec 15, 2024, at 8:22 PM, Anex Hul &lt;<a href=3D"mailto:anexsql201=
4@gmail.com" target=3D"_blank" rel=3D"noreferrer">anexsql2014@gmail.com</a>=
&gt; wrote:<br>
&gt; <br>
&gt; Hello everyone,<br>
&gt; <br>
&gt; Testing 100 million records data import from Azure blob storage to Azu=
re postgresql. I did run the test 5 times and the time it took keep increas=
ing for each run. <br>
&gt; Is there know justification for this linear increment of the time it t=
ook for same size of data?<br>
<br>
Check you I/O quotas; you might have hit quota limits and being throttled.<=
/blockquote></div>

--000000000000f16ba4062963ab28--