public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 5/6] Add copy progress reporting regression tests. 5+ messages / 4 participants [nested] [flat]
* [PATCH 5/6] Add copy progress reporting regression tests. @ 2021-02-09 12:06 Josef Šimánek <[email protected]> 0 siblings, 0 replies; 5+ messages in thread From: Josef Šimánek @ 2021-02-09 12:06 UTC (permalink / raw) This tests some basic features of copy progress reporting. Sadly, we can only request one snapshot of progress_reporting each transaction / statement, so we can't (easily) get intermediate results without each result requiring a seperate statement being run. Author: Josef Šimánek, Matthias van de Meent --- src/test/regress/input/copy.source | 57 +++++++++++++++++++++++++++++ src/test/regress/output/copy.source | 44 ++++++++++++++++++++++ 2 files changed, 101 insertions(+) diff --git a/src/test/regress/input/copy.source b/src/test/regress/input/copy.source index a1d529ad36..ddde33e7cc 100644 --- a/src/test/regress/input/copy.source +++ b/src/test/regress/input/copy.source @@ -201,3 +201,60 @@ select * from parted_copytest where b = 1; select * from parted_copytest where b = 2; drop table parted_copytest; + +-- +-- progress reporting +-- + +-- setup +-- reuse employer datatype, that has a small sized data set + +create table progress_reporting ( + name text, + age int4, + location point, + salary int4, + manager name +); + +create function notice_after_progress_reporting() returns trigger AS +$$ +declare report record; +begin + -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to + -- variance in system process ids, and concurrency in runs of tests. + -- Additionally, due to the usage of this test in pg_regress, the 'datid' + -- also is not consistent between runs. + select into report (to_jsonb(r) - '{pid,relid,datid}'::text[]) as value + from pg_stat_progress_copy r + where pid = pg_backend_pid(); + + raise info 'progress: %', report.value::text; + + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +create trigger check_after_progress_reporting + after insert on progress_reporting + for each statement + execute function notice_after_progress_reporting(); + +-- reporting of STDIO imports, and correct bytes-processed/tuples-processed reporting + +copy progress_reporting from stdin; +sharon 25 (15,12) 1000 sam +sam 30 (10,5) 2000 bill +bill 20 (11,10) 1000 sharon +\. + +-- reporting of FILE imports, and correct reporting of tuples-excluded + +copy progress_reporting from '@abs_srcdir@/data/emp.data' + where (salary < 2000); + +-- cleanup progress_reporting + +drop trigger check_after_progress_reporting on progress_reporting; +drop function notice_after_progress_reporting(); +drop table progress_reporting; diff --git a/src/test/regress/output/copy.source b/src/test/regress/output/copy.source index 938d3551da..60f4206aa1 100644 --- a/src/test/regress/output/copy.source +++ b/src/test/regress/output/copy.source @@ -165,3 +165,47 @@ select * from parted_copytest where b = 2; (1 row) drop table parted_copytest; +-- +-- progress reporting +-- +-- setup +-- reuse employer datatype, that has a small sized data set +create table progress_reporting ( + name text, + age int4, + location point, + salary int4, + manager name +); +create function notice_after_progress_reporting() returns trigger AS +$$ +declare report record; +begin + -- We cannot expect 'pid' nor 'relid' to be consistent over runs due to + -- variance in system process ids, and concurrency in runs of tests. + -- Additionally, due to the usage of this test in pg_regress, the 'datid' + -- also is not consistent between runs. + select into report (to_jsonb(r) - '{pid,relid,datid}'::text[]) as value + from pg_stat_progress_copy r + where pid = pg_backend_pid(); + + raise info 'progress: %', report.value::text; + + RETURN NEW; +END; +$$ LANGUAGE plpgsql; +create trigger check_after_progress_reporting + after insert on progress_reporting + for each statement + execute function notice_after_progress_reporting(); +-- reporting of STDIO imports, and correct bytes-processed/tuples-processed reporting +copy progress_reporting from stdin; +INFO: progress: {"command": "COPY FROM", "datname": "regression", "io_target": "STDIO", "bytes_total": 0, "bytes_processed": 79, "tuples_excluded": 0, "tuples_processed": 3} +-- reporting of FILE imports, and correct reporting of tuples-excluded +copy progress_reporting from '@abs_srcdir@/data/emp.data' + where (salary < 2000); +INFO: progress: {"command": "COPY FROM", "datname": "regression", "io_target": "FILE", "bytes_total": 79, "bytes_processed": 79, "tuples_excluded": 1, "tuples_processed": 2} +-- cleanup progress_reporting +drop trigger check_after_progress_reporting on progress_reporting; +drop function notice_after_progress_reporting(); +drop table progress_reporting; -- 2.26.2 --------------DA8D60BF026F7ACE69A9AEE5 Content-Type: text/x-patch; charset=UTF-8; name="v7-0006-0003-review.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0006-0003-review.patch" ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Cygwin cleanup @ 2023-01-03 12:24 vignesh C <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: vignesh C @ 2023-01-03 12:24 UTC (permalink / raw) To: Justin Pryzby <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]> On Wed, 9 Nov 2022 at 06:34, Justin Pryzby <[email protected]> wrote: > > On Thu, Oct 20, 2022 at 10:40:40PM -0500, Justin Pryzby wrote: > > On Thu, Aug 04, 2022 at 04:16:06PM +1200, Thomas Munro wrote: > > > On Thu, Aug 4, 2022 at 3:38 PM Justin Pryzby <[email protected]> wrote: > > > > [train wreck] > > > > > > Oh my, so I'm getting the impression we might actually be totally > > > unstable on Cygwin. Which surprises me because ... wait a minute ... > > > lorikeet isn't even running most of the tests. So... we don't really > > > know the degree to which any of this works at all? > > > > Right. > > > > Maybe it's of limited interest, but .. > > > > This updates the patch to build and test with meson. > > Which first requires patching some meson.builds. > > I guess that's needed for some current BF members, too. > > Unfortunately, ccache+PCH causes gcc to crash :( > > Resending with the 'only-if' line commented (doh). > And some fixes to 001 as Andres pointed out by on other thread. Is there still some work pending for this thread as Andres had committed some part, if so, can you post an updated patch for the same. Regards, Vignesh ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Cygwin cleanup @ 2023-01-03 14:25 Justin Pryzby <[email protected]> parent: vignesh C <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Justin Pryzby @ 2023-01-03 14:25 UTC (permalink / raw) To: vignesh C <[email protected]>; +Cc: Thomas Munro <[email protected]>; [email protected]; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]> On Tue, Jan 03, 2023 at 05:54:56PM +0530, vignesh C wrote: > > On Thu, Oct 20, 2022 at 10:40:40PM -0500, Justin Pryzby wrote: > > > On Thu, Aug 04, 2022 at 04:16:06PM +1200, Thomas Munro wrote: > > > > On Thu, Aug 4, 2022 at 3:38 PM Justin Pryzby <[email protected]> wrote: > > > > > [train wreck] > > > > > > > > Oh my, so I'm getting the impression we might actually be totally > > > > unstable on Cygwin. Which surprises me because ... wait a minute ... > > > > lorikeet isn't even running most of the tests. So... we don't really > > > > know the degree to which any of this works at all? > > > > > > Right. > > > > > > Maybe it's of limited interest, but .. > > > > > > This updates the patch to build and test with meson. > > > Which first requires patching some meson.builds. > > > I guess that's needed for some current BF members, too. > > > Unfortunately, ccache+PCH causes gcc to crash :( > > > > Resending with the 'only-if' line commented (doh). > > And some fixes to 001 as Andres pointed out by on other thread. > > Is there still some work pending for this thread as Andres had > committed some part, if so, can you post an updated patch for the > same. Thomas, what's your opinion ? ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Cygwin cleanup @ 2023-01-05 12:22 Thomas Munro <[email protected]> parent: Justin Pryzby <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Thomas Munro @ 2023-01-05 12:22 UTC (permalink / raw) To: Justin Pryzby <[email protected]>; +Cc: vignesh C <[email protected]>; [email protected]; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]> On Wed, Jan 4, 2023 at 3:25 AM Justin Pryzby <[email protected]> wrote: > On Tue, Jan 03, 2023 at 05:54:56PM +0530, vignesh C wrote: > > Is there still some work pending for this thread as Andres had > > committed some part, if so, can you post an updated patch for the > > same. > > Thomas, what's your opinion ? One observation is that your CI patch *nearly* succeeds, even if hacked to turn on the full TAP tests, if applied on top of the WaitEventSet-for-postmaster patch: https://cirrus-ci.com/task/4533371804581888 No cigar though, it still failed a few times for me in the subscription tests with EAGAIN, when accessing semaphores: semctl(24576010, 14, SETVAL, 0) failed: Resource temporarily unavailable That isn't an error I expect from semctl(), but from some cursory research it seems like that system call is actually talking to the cygserver process over a pipe (?) to implement SysV semaphores. Maybe it couldn't keep up, but doesn't like to block? Perhaps we could try to tune that server, but let's try the POSIX kind of semaphores instead. From a quick peek at the source, they are implemented some other way on direct native NT voodoo, no cygserver involved. https://cirrus-ci.com/task/5142810819559424 [still running at time of writing] Gotta run, but I'll check again in the morning to see if that does better... ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Cygwin cleanup @ 2023-01-05 21:39 Thomas Munro <[email protected]> parent: Thomas Munro <[email protected]> 0 siblings, 0 replies; 5+ messages in thread From: Thomas Munro @ 2023-01-05 21:39 UTC (permalink / raw) To: Justin Pryzby <[email protected]>; +Cc: vignesh C <[email protected]>; [email protected]; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; Andres Freund <[email protected]> On Fri, Jan 6, 2023 at 1:22 AM Thomas Munro <[email protected]> wrote: > https://cirrus-ci.com/task/5142810819559424 [still running at time of writing] > > Gotta run, but I'll check again in the morning to see if that does better... Yes! Two successful runs with all TAP tests so far. So it looks like we can probably stop lorikeet's spurious failures, by happy coincidence due to other work, and we could seriously consider committing this optional CI test for it, much like we have the optional MSYS build. Any interest in producing a tidied up version of the patch, Justin? Or I can, but I'll go and work on other things first. I pushed a change to switch the semaphore implementation. I haven't personally seen that failure mode on lorikeet, but I would guess that's because (1) it's only running a tiny subset of the tests, (2) it crashes for the other reason with higher likelihood, and/or (3) it's not using much concurrency yet because the build farm doesn't use meson yet. ^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2023-01-05 21:39 UTC | newest] Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-02-09 12:06 [PATCH 5/6] Add copy progress reporting regression tests. Josef Šimánek <[email protected]> 2023-01-03 12:24 Re: Cygwin cleanup vignesh C <[email protected]> 2023-01-03 14:25 ` Re: Cygwin cleanup Justin Pryzby <[email protected]> 2023-01-05 12:22 ` Re: Cygwin cleanup Thomas Munro <[email protected]> 2023-01-05 21:39 ` Re: Cygwin cleanup Thomas Munro <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox