public inbox for [email protected]help / color / mirror / Atom feed
Performance 5+ messages / 3 participants [nested] [flat]
* Performance @ 2024-12-16 01:22 Anex Hul <[email protected]> 0 siblings, 2 replies; 5+ messages in thread From: Anex Hul @ 2024-12-16 01:22 UTC (permalink / raw) To: [email protected] Hello everyone, Testing 100 million records data import from Azure blob storage to Azure postgresql. I did run the test 5 times and the time it took keep increasing for each run. Is there know justification for this linear increment of the time it took for same size of data? ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Performance @ 2024-12-16 02:14 Ron Johnson <[email protected]> parent: Anex Hul <[email protected]> 1 sibling, 0 replies; 5+ messages in thread From: Ron Johnson @ 2024-12-16 02:14 UTC (permalink / raw) To: Pgsql-admin <[email protected]> On Sun, Dec 15, 2024 at 8:22 PM Anex Hul <[email protected]> wrote: > Hello everyone, > > Testing 100 million records data import from Azure blob storage to Azure > postgresql. I did run the test 5 times and the time it took keep increasing > for each run. > Is there know justification for this linear increment of the time it took > for same size of data? > 1. What version of PG is it? ("SELECT VERSION();" should tell you.) 2. Are you truncating the table after each test run, or deleting all records, or appending? 3. Is the blob data stored in BYTEA column data, or are you using the (discouraged) "Large Objects"? 4. How are you loading the blob data? -- Death to <Redacted>, and butter sauce. Don't boil me, I'm still alive. <Redacted> lobster! ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Performance @ 2024-12-16 04:22 Rui DeSousa <[email protected]> parent: Anex Hul <[email protected]> 1 sibling, 1 reply; 5+ messages in thread From: Rui DeSousa @ 2024-12-16 04:22 UTC (permalink / raw) To: Anex Hul <[email protected]>; +Cc: [email protected] > On Dec 15, 2024, at 8:22 PM, Anex Hul <[email protected]> wrote: > > Hello everyone, > > Testing 100 million records data import from Azure blob storage to Azure postgresql. I did run the test 5 times and the time it took keep increasing for each run. > Is there know justification for this linear increment of the time it took for same size of data? Check you I/O quotas; you might have hit quota limits and being throttled. ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Performance @ 2024-12-16 14:05 Anex Hul <[email protected]> parent: Rui DeSousa <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Anex Hul @ 2024-12-16 14:05 UTC (permalink / raw) To: Rui DeSousa <[email protected]>; +Cc: [email protected] Thank you all for your response. Show quoted text 1. What version of PG is it? ("SELECT VERSION();" should tell you.) PG Version 16 2. Are you truncating the table after each test run, or deleting all records, or appending? created new schema for each run. 3. Is the blob data stored in BYTEA column data, or are you using the (discouraged) "Large Objects"? Blob storage 4. How are you loading the blob data? used the Import data using a COPY statement, followed this doc https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-use-pg-azure-storage?tabs=... On Sun, Dec 15, 2024, 10:22 PM Rui DeSousa <[email protected]> wrote: > > > > On Dec 15, 2024, at 8:22 PM, Anex Hul <[email protected]> wrote: > > > > Hello everyone, > > > > Testing 100 million records data import from Azure blob storage to Azure > postgresql. I did run the test 5 times and the time it took keep increasing > for each run. > > Is there know justification for this linear increment of the time it > took for same size of data? > > Check you I/O quotas; you might have hit quota limits and being throttled. ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Performance @ 2024-12-16 14:20 Ron Johnson <[email protected]> parent: Anex Hul <[email protected]> 0 siblings, 0 replies; 5+ messages in thread From: Ron Johnson @ 2024-12-16 14:20 UTC (permalink / raw) To: Pgsql-admin <[email protected]> On Mon, Dec 16, 2024 at 9:05 AM Anex Hul <[email protected]> wrote: [snip] > 2. Are you truncating the table after each test run, or deleting all > records, or appending? > > created new schema for each run. > > 3. Is the blob data stored in BYTEA column data, or are you using the > (discouraged) "Large Objects"? > > Blob storage > Postgresql does not know what "Blob storage" means. > 4. How are you loading the blob data? > > used the Import data using a COPY statement, followed this doc > > > https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-use-pg-azure-storage?tabs=... > If you're using a Microsoft extension, then you'd better ask Microsoft. -- Death to <Redacted>, and butter sauce. Don't boil me, I'm still alive. <Redacted> lobster! ^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2024-12-16 14:20 UTC | newest] Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-12-16 01:22 Performance Anex Hul <[email protected]> 2024-12-16 02:14 ` Ron Johnson <[email protected]> 2024-12-16 04:22 ` Rui DeSousa <[email protected]> 2024-12-16 14:05 ` Anex Hul <[email protected]> 2024-12-16 14:20 ` Ron Johnson <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox