public inbox for [email protected]
help / color / mirror / Atom feedPerformance
5+ messages / 3 participants
[nested] [flat]
* Performance
@ 2024-12-16 01:22 Anex Hul <[email protected]>
2024-12-16 02:14 ` Re: Performance Ron Johnson <[email protected]>
2024-12-16 04:22 ` Re: Performance Rui DeSousa <[email protected]>
0 siblings, 2 replies; 5+ messages in thread
From: Anex Hul @ 2024-12-16 01:22 UTC (permalink / raw)
To: [email protected]
Hello everyone,
Testing 100 million records data import from Azure blob storage to Azure
postgresql. I did run the test 5 times and the time it took keep increasing
for each run.
Is there know justification for this linear increment of the time it took
for same size of data?
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Performance
2024-12-16 01:22 Performance Anex Hul <[email protected]>
@ 2024-12-16 02:14 ` Ron Johnson <[email protected]>
1 sibling, 0 replies; 5+ messages in thread
From: Ron Johnson @ 2024-12-16 02:14 UTC (permalink / raw)
To: Pgsql-admin <[email protected]>
On Sun, Dec 15, 2024 at 8:22 PM Anex Hul <[email protected]> wrote:
> Hello everyone,
>
> Testing 100 million records data import from Azure blob storage to Azure
> postgresql. I did run the test 5 times and the time it took keep increasing
> for each run.
> Is there know justification for this linear increment of the time it took
> for same size of data?
>
1. What version of PG is it? ("SELECT VERSION();" should tell you.)
2. Are you truncating the table after each test run, or deleting all
records, or appending?
3. Is the blob data stored in BYTEA column data, or are you using the
(discouraged) "Large Objects"?
4. How are you loading the blob data?
--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Performance
2024-12-16 01:22 Performance Anex Hul <[email protected]>
@ 2024-12-16 04:22 ` Rui DeSousa <[email protected]>
2024-12-16 14:05 ` Re: Performance Anex Hul <[email protected]>
1 sibling, 1 reply; 5+ messages in thread
From: Rui DeSousa @ 2024-12-16 04:22 UTC (permalink / raw)
To: Anex Hul <[email protected]>; +Cc: [email protected]
> On Dec 15, 2024, at 8:22 PM, Anex Hul <[email protected]> wrote:
>
> Hello everyone,
>
> Testing 100 million records data import from Azure blob storage to Azure postgresql. I did run the test 5 times and the time it took keep increasing for each run.
> Is there know justification for this linear increment of the time it took for same size of data?
Check you I/O quotas; you might have hit quota limits and being throttled.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Performance
2024-12-16 01:22 Performance Anex Hul <[email protected]>
2024-12-16 04:22 ` Re: Performance Rui DeSousa <[email protected]>
@ 2024-12-16 14:05 ` Anex Hul <[email protected]>
2024-12-16 14:20 ` Re: Performance Ron Johnson <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Anex Hul @ 2024-12-16 14:05 UTC (permalink / raw)
To: Rui DeSousa <[email protected]>; +Cc: [email protected]
Thank you all for your response.
Show quoted text
1. What version of PG is it? ("SELECT VERSION();" should tell you.)
PG Version 16
2. Are you truncating the table after each test run, or deleting all
records, or appending?
created new schema for each run.
3. Is the blob data stored in BYTEA column data, or are you using the
(discouraged) "Large Objects"?
Blob storage
4. How are you loading the blob data?
used the Import data using a COPY statement, followed this doc
https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-use-pg-azure-storage?tabs=...
On Sun, Dec 15, 2024, 10:22 PM Rui DeSousa <[email protected]> wrote:
>
>
> > On Dec 15, 2024, at 8:22 PM, Anex Hul <[email protected]> wrote:
> >
> > Hello everyone,
> >
> > Testing 100 million records data import from Azure blob storage to Azure
> postgresql. I did run the test 5 times and the time it took keep increasing
> for each run.
> > Is there know justification for this linear increment of the time it
> took for same size of data?
>
> Check you I/O quotas; you might have hit quota limits and being throttled.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Performance
2024-12-16 01:22 Performance Anex Hul <[email protected]>
2024-12-16 04:22 ` Re: Performance Rui DeSousa <[email protected]>
2024-12-16 14:05 ` Re: Performance Anex Hul <[email protected]>
@ 2024-12-16 14:20 ` Ron Johnson <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Ron Johnson @ 2024-12-16 14:20 UTC (permalink / raw)
To: Pgsql-admin <[email protected]>
On Mon, Dec 16, 2024 at 9:05 AM Anex Hul <[email protected]> wrote:
[snip]
> 2. Are you truncating the table after each test run, or deleting all
> records, or appending?
>
> created new schema for each run.
>
> 3. Is the blob data stored in BYTEA column data, or are you using the
> (discouraged) "Large Objects"?
>
> Blob storage
>
Postgresql does not know what "Blob storage" means.
> 4. How are you loading the blob data?
>
> used the Import data using a COPY statement, followed this doc
>
>
> https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-use-pg-azure-storage?tabs=...
>
If you're using a Microsoft extension, then you'd better ask Microsoft.
--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2024-12-16 14:20 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-12-16 01:22 Performance Anex Hul <[email protected]>
2024-12-16 02:14 ` Ron Johnson <[email protected]>
2024-12-16 04:22 ` Rui DeSousa <[email protected]>
2024-12-16 14:05 ` Anex Hul <[email protected]>
2024-12-16 14:20 ` Ron Johnson <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox