public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 08/19] vcregress: add alltaptests 9+ messages / 5 participants [nested] [flat]
* [PATCH 08/19] vcregress: add alltaptests @ 2022-02-25 23:00 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 9+ messages in thread From: Justin Pryzby @ 2022-02-25 23:00 UTC (permalink / raw) https://www.postgresql.org/message-id/[email protected] https://www.postgresql.org/message-id/flat/[email protected] In passing, document taptest PROVE_FLAGS ci-os-only: windows See also: d835dd6685246f0737ca42ab68242210681bb220 13d856e177e69083f543d6383eeda9e12ce3c55c fed6df486dca1b9e53d3f560031b9a236c99f4bb --- .cirrus.yml | 10 ++------- src/tools/msvc/vcregress.pl | 43 +++++++++++++++++++++++++++++++++---- 2 files changed, 41 insertions(+), 12 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index f2bb05266cc..ea1f493923a 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -437,14 +437,8 @@ task: test_ssl_script: | set with_ssl=openssl %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/ssl/ - test_subscription_script: | - %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/subscription/ - test_authentication_script: | - %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/authentication/ - test_recovery_script: | - %T_C% perl src/tools/msvc/vcregress.pl recoverycheck - test_bin_script: | - %T_C% perl src/tools/msvc/vcregress.pl bincheck + test_tap_script: | + %T_C% perl src/tools/msvc/vcregress.pl alltaptests test_ecpg_script: | rem tries to build additional stuff vcvarsall x64 diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl index 54c31b0e14f..1b1387acac0 100644 --- a/src/tools/msvc/vcregress.pl +++ b/src/tools/msvc/vcregress.pl @@ -51,7 +51,7 @@ if (-e "src/tools/msvc/buildenv.pl") my $what = shift || ""; if ($what =~ - /^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck|taptest)$/i + /^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck|taptest|alltaptests)$/i ) { $what = uc $what; @@ -109,6 +109,7 @@ my %command = ( BINCHECK => \&bincheck, RECOVERYCHECK => \&recoverycheck, UPGRADECHECK => \&upgradecheck, # no-op + ALLTAPTESTS => \&alltaptests, TAPTEST => \&taptest,); my $proc = $command{$what}; @@ -291,6 +292,9 @@ sub tap_check $ENV{PG_REGRESS} = "$topdir/$Config/pg_regress/pg_regress"; $ENV{REGRESS_SHLIB} = "$topdir/src/test/regress/regress.dll"; + print "============================================================\n"; + print "Checking $dir: @args\n"; + rmtree('tmp_check'); system(@args); my $status = $? >> 8; @@ -305,8 +309,7 @@ sub bincheck my $mstat = 0; - # Find out all the existing TAP tests by looking for t/ directories - # in the tree. + # Find the TAP tests by looking for t/ directories my @bin_dirs = glob("$topdir/src/bin/*"); # Process each test @@ -321,6 +324,36 @@ sub bincheck return; } +sub alltaptests +{ + InstallTemp(); + + my $mstat = 0; + + # Find out all the existing TAP tests by looking for t/ directories + # in the tree. + my @tap_dirs = (); + my @top_dir = ($topdir); + File::Find::find( + { wanted => sub { + /^t\z/s + && $File::Find::name !~ /\/(kerberos|ldap|ssl|ssl_passphrase_callback)\// # opt-in + && push(@tap_dirs, $File::Find::name); + } + }, + @top_dir); + + # Process each test + foreach my $test_path (@tap_dirs) + { + my $dir = dirname($test_path); + my $status = tap_check($dir); + $mstat ||= $status; + } + exit $mstat if $mstat; + return; +} + sub taptest { my $dir = shift; @@ -671,6 +704,7 @@ sub usage print STDERR "Usage: vcregress.pl <mode> [<arg>]\n\n", "Options for <mode>:\n", + " alltaptests run all tap tests (except kerberos, ldap, ssl, ssl_passphrase_callback)\n", " bincheck run tests of utilities in src/bin/\n", " check deploy instance and run regression tests on it\n", " contribcheck run tests of modules in contrib/\n", @@ -688,6 +722,7 @@ sub usage "\nOptions for <arg>: (used by contribcheck and modulescheck)\n", " install also run tests which require a new instance\n", "\nOption for <arg>: for taptest\n", - " TEST_DIR (required) directory where tests reside\n"; + " TEST_DIR (required) directory where tests reside\n", + " PROVE_FLAGS flags to pass to prove\n"; exit(1); } -- 2.17.1 --Sw7tCqrGA+HQ0/zt Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0009-tmp-run-tap-tests-first.patch" ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded @ 2023-06-14 22:23 James Addison <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: James Addison @ 2023-06-14 22:23 UTC (permalink / raw) To: Konstantin Knizhnik <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Hannu Krosing <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik <[email protected]> wrote: > > > > On 12.06.2023 3:23 PM, Pavel Borisov wrote: > > Is the following true or not? > > > > 1. If we switch processes to threads but leave the amount of session > > local variables unchanged, there would be hardly any performance gain. > > 2. If we move some backend's local variables into shared memory then > > the performance gain would be very near to what we get with threads > > having equal amount of session-local variables. > > > > In other words, the overall goal in principle is to gain from less > > memory copying wherever it doesn't add the burden of locks for > > concurrent variables access? > > > > Regards, > > Pavel Borisov, > > Supabase > > > > > IMHO both statements are not true. > Switching to threads will cause less context switch overhead (because > all threads are sharing the same memory space and so preserve TLB. > How big will be this advantage? In my prototype I got ~10%. But may be > it is possible to fin workloads when it is larger. Hi Konstantin - do you have code/links that you can share for the prototype and benchmarks used to gather those results? ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded @ 2023-06-15 07:12 Konstantin Knizhnik <[email protected]> parent: James Addison <[email protected]> 0 siblings, 1 reply; 9+ messages in thread From: Konstantin Knizhnik @ 2023-06-15 07:12 UTC (permalink / raw) To: James Addison <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Hannu Krosing <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers On 15.06.2023 1:23 AM, James Addison wrote: > On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik<[email protected]> wrote: >> >> >> On 12.06.2023 3:23 PM, Pavel Borisov wrote: >>> Is the following true or not? >>> >>> 1. If we switch processes to threads but leave the amount of session >>> local variables unchanged, there would be hardly any performance gain. >>> 2. If we move some backend's local variables into shared memory then >>> the performance gain would be very near to what we get with threads >>> having equal amount of session-local variables. >>> >>> In other words, the overall goal in principle is to gain from less >>> memory copying wherever it doesn't add the burden of locks for >>> concurrent variables access? >>> >>> Regards, >>> Pavel Borisov, >>> Supabase >>> >>> >> IMHO both statements are not true. >> Switching to threads will cause less context switch overhead (because >> all threads are sharing the same memory space and so preserve TLB. >> How big will be this advantage? In my prototype I got ~10%. But may be >> it is possible to fin workloads when it is larger. > Hi Konstantin - do you have code/links that you can share for the > prototype and benchmarks used to gather those results? Sorry, I have already shared the link: https://github.com/postgrespro/postgresql.pthreads/ As you can see last commit was 6 years ago when I stopped work on this project. Why? I already tried to explain it: - benefits from switching to threads were not so large. May be I just failed to fid proper workload, but is was more or less expected result, because most of the code was not changed - it uses the same sync primitives, the same local catalog/relation caches,.. To take all advantage of multithreadig model it is necessary to rewrite many components, especially related with interprocess communication. But maintaining such fork of Postgres and synchronize it with mainstream requires too much efforts and I was not able to do it myself. There are three different but related directions of improving current Postgres: 1. Replacing processes with threads 2. Builtin connection pooler 3. Lightweight backends (shared catalog/relation/prepared statements caches) The motivation for such changes are also similar: 1. Increase Postgres scalability 2. Reduce memory consumption 3. Make Postgres better fir cloud and serverless requirements I am not sure now which one should be addressed first or them can be done together. Replacing static variables with thread-local is the first and may be the easiest step. It requires more or less mechanical changes. More challenging thing is replacing private per-backend data structures with shared ones (caches, file descriptors,...) ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded @ 2023-06-15 08:41 James Addison <[email protected]> parent: Konstantin Knizhnik <[email protected]> 0 siblings, 2 replies; 9+ messages in thread From: James Addison @ 2023-06-15 08:41 UTC (permalink / raw) To: Konstantin Knizhnik <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Hannu Krosing <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers On Thu, 15 Jun 2023 at 08:12, Konstantin Knizhnik <[email protected]> wrote: > > > > On 15.06.2023 1:23 AM, James Addison wrote: > > On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik <[email protected]> wrote: > > > On 12.06.2023 3:23 PM, Pavel Borisov wrote: > > Is the following true or not? > > 1. If we switch processes to threads but leave the amount of session > local variables unchanged, there would be hardly any performance gain. > 2. If we move some backend's local variables into shared memory then > the performance gain would be very near to what we get with threads > having equal amount of session-local variables. > > In other words, the overall goal in principle is to gain from less > memory copying wherever it doesn't add the burden of locks for > concurrent variables access? > > Regards, > Pavel Borisov, > Supabase > > > IMHO both statements are not true. > Switching to threads will cause less context switch overhead (because > all threads are sharing the same memory space and so preserve TLB. > How big will be this advantage? In my prototype I got ~10%. But may be > it is possible to fin workloads when it is larger. > > Hi Konstantin - do you have code/links that you can share for the > prototype and benchmarks used to gather those results? > > > > Sorry, I have already shared the link: > https://github.com/postgrespro/postgresql.pthreads/ Nope, my mistake for not locating the existing link - thank you. Is there a reason that parser-related files (flex/bison) are added as part of the changeset? (I'm trying to narrow it down to only the changes necessary for the functionality. so far it looks mostly fairly minimal, which is good. the adjustments to progname are another thing that look a bit unusual/maybe unnecessary for the feature) > As you can see last commit was 6 years ago when I stopped work on this project. > Why? I already tried to explain it: > - benefits from switching to threads were not so large. May be I just failed to fid proper workload, but is was more or less expected result, > because most of the code was not changed - it uses the same sync primitives, the same local catalog/relation caches,.. > To take all advantage of multithreadig model it is necessary to rewrite many components, especially related with interprocess communication. > But maintaining such fork of Postgres and synchronize it with mainstream requires too much efforts and I was not able to do it myself. I get the feeling that there are probably certain query types or patterns where a significant, order-of-magnitude speedup is possible with threads - but yep, I haven't seen those described in detail yet on the mailing list (but as hinted by my not noticing the github link previously, maybe I'm not following the list closely enough). What workloads did you try with your version of the project? > There are three different but related directions of improving current Postgres: > 1. Replacing processes with threads > 2. Builtin connection pooler > 3. Lightweight backends (shared catalog/relation/prepared statements caches) > > The motivation for such changes are also similar: > 1. Increase Postgres scalability > 2. Reduce memory consumption > 3. Make Postgres better fir cloud and serverless requirements > > I am not sure now which one should be addressed first or them can be done together. > > Replacing static variables with thread-local is the first and may be the easiest step. > It requires more or less mechanical changes. More challenging thing is replacing private per-backend data structures > with shared ones (caches, file descriptors,...) Thank you. Personally I think that motivation two (reducing memory consumption) -- as long as it can be done without detrimentally affecting functionality or correctness, and without making the code harder to develop/understand -- could provide benefits for all three of the motivating cases (and, in fact, for non-cloud/serverful use cases too). This is making me wonder about other performance/scalability areas that might not have been considered due to focus on the details of the existing codebase, but I'll save that for another thread and will try to learn more first. ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded @ 2023-06-15 09:04 Hannu Krosing <[email protected]> parent: James Addison <[email protected]> 1 sibling, 2 replies; 9+ messages in thread From: Hannu Krosing @ 2023-06-15 09:04 UTC (permalink / raw) To: James Addison <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers On Thu, Jun 15, 2023 at 10:41 AM James Addison <[email protected]> wrote: > > This is making me wonder about other performance/scalability areas > that might not have been considered due to focus on the details of the > existing codebase, but I'll save that for another thread and will try > to learn more first. A gradual move to more shared structures seems to be a way forward It should get us all the benefits of threading minus the need for TLB reloading and (in some cases) reduction of per-process virtual memory mapping tables. In any case we would need to implement all the locking and parallelism management of these shared structures that are not there in the current process architecture. So a fair bit of work but also a clearly defined benefits of 1) reduced memory usage 2) no need to rebuild caches for each new connection 3) no need to track PREPARE statements inside connection poolers. There can be extra complexity when different connections use the same prepared statement name (say "PREP001") for different queries. For this wel likely will need a good cooperation with connection pooler where it passes some kind of client connection id along at the transaction start ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded @ 2023-06-15 09:07 Hannu Krosing <[email protected]> parent: Hannu Krosing <[email protected]> 1 sibling, 1 reply; 9+ messages in thread From: Hannu Krosing @ 2023-06-15 09:07 UTC (permalink / raw) To: James Addison <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers One more unexpected benefit of having shared caches would be easing access to other databases. If the system caches are there for all databases anyway, then it becomes much easier to make queries using objects from multiple databases. Note that this does not strictly need threads, just shared caches. On Thu, Jun 15, 2023 at 11:04 AM Hannu Krosing <[email protected]> wrote: > > On Thu, Jun 15, 2023 at 10:41 AM James Addison <[email protected]> wrote: > > > > This is making me wonder about other performance/scalability areas > > that might not have been considered due to focus on the details of the > > existing codebase, but I'll save that for another thread and will try > > to learn more first. > > A gradual move to more shared structures seems to be a way forward > > It should get us all the benefits of threading minus the need for TLB > reloading and (in some cases) reduction of per-process virtual memory > mapping tables. > > In any case we would need to implement all the locking and parallelism > management of these shared structures that are not there in the > current process architecture. > > So a fair bit of work but also a clearly defined benefits of > 1) reduced memory usage > 2) no need to rebuild caches for each new connection > 3) no need to track PREPARE statements inside connection poolers. > > There can be extra complexity when different connections use the same > prepared statement name (say "PREP001") for different queries. > For this wel likely will need a good cooperation with connection > pooler where it passes some kind of client connection id along at the > transaction start ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded @ 2023-06-15 19:36 Konstantin Knizhnik <[email protected]> parent: James Addison <[email protected]> 1 sibling, 0 replies; 9+ messages in thread From: Konstantin Knizhnik @ 2023-06-15 19:36 UTC (permalink / raw) To: James Addison <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Hannu Krosing <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers On 15.06.2023 11:41 AM, James Addison wrote: > On Thu, 15 Jun 2023 at 08:12, Konstantin Knizhnik <[email protected]> wrote: >> >> >> On 15.06.2023 1:23 AM, James Addison wrote: >> >> On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik <[email protected]> wrote: >> >> >> On 12.06.2023 3:23 PM, Pavel Borisov wrote: >> >> Is the following true or not? >> >> 1. If we switch processes to threads but leave the amount of session >> local variables unchanged, there would be hardly any performance gain. >> 2. If we move some backend's local variables into shared memory then >> the performance gain would be very near to what we get with threads >> having equal amount of session-local variables. >> >> In other words, the overall goal in principle is to gain from less >> memory copying wherever it doesn't add the burden of locks for >> concurrent variables access? >> >> Regards, >> Pavel Borisov, >> Supabase >> >> >> IMHO both statements are not true. >> Switching to threads will cause less context switch overhead (because >> all threads are sharing the same memory space and so preserve TLB. >> How big will be this advantage? In my prototype I got ~10%. But may be >> it is possible to fin workloads when it is larger. >> >> Hi Konstantin - do you have code/links that you can share for the >> prototype and benchmarks used to gather those results? >> >> >> >> Sorry, I have already shared the link: >> https://github.com/postgrespro/postgresql.pthreads/ > Nope, my mistake for not locating the existing link - thank you. > > Is there a reason that parser-related files (flex/bison) are added as > part of the changeset? (I'm trying to narrow it down to only the > changes necessary for the functionality. so far it looks mostly > fairly minimal, which is good. the adjustments to progname are > another thing that look a bit unusual/maybe unnecessary for the > feature) Sorry, absolutely no reason - just my fault. >> As you can see last commit was 6 years ago when I stopped work on this project. >> Why? I already tried to explain it: >> - benefits from switching to threads were not so large. May be I just failed to fid proper workload, but is was more or less expected result, >> because most of the code was not changed - it uses the same sync primitives, the same local catalog/relation caches,.. >> To take all advantage of multithreadig model it is necessary to rewrite many components, especially related with interprocess communication. >> But maintaining such fork of Postgres and synchronize it with mainstream requires too much efforts and I was not able to do it myself. > I get the feeling that there are probably certain query types or > patterns where a significant, order-of-magnitude speedup is possible > with threads - but yep, I haven't seen those described in detail yet > on the mailing list (but as hinted by my not noticing the github link > previously, maybe I'm not following the list closely enough). > > What workloads did you try with your version of the project? I do not remember now precisely (6 years passed). But definitely I tried pgbench, especially read-only pgbench (to be more CPU rather than disk bounded) ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded @ 2023-06-15 19:49 Konstantin Knizhnik <[email protected]> parent: Hannu Krosing <[email protected]> 1 sibling, 0 replies; 9+ messages in thread From: Konstantin Knizhnik @ 2023-06-15 19:49 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; James Addison <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers On 15.06.2023 12:04 PM, Hannu Krosing wrote: > So a fair bit of work but also a clearly defined benefits of > 1) reduced memory usage > 2) no need to rebuild caches for each new connection > 3) no need to track PREPARE statements inside connection poolers. Shared plan cache (not only prepared statements cache) also opens way to more sophisticated query optimizations. Right now we are not performing some optimization (like constant expression folding) just because them increase time of processing normal queries. This is why queries generated by ORMs or wizards, which can contain a lot of dumb stuff, are not well simplified by Postgres. With MS-Sql it is quite frequent that query execution time is much smaller than query optimization time. Having shared plan cache allows us to spend more time in optimization without risk to degrade performance. ^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded @ 2023-07-28 18:10 Matthias van de Meent <[email protected]> parent: Hannu Krosing <[email protected]> 0 siblings, 0 replies; 9+ messages in thread From: Matthias van de Meent @ 2023-07-28 18:10 UTC (permalink / raw) To: Hannu Krosing <[email protected]>; +Cc: James Addison <[email protected]>; Konstantin Knizhnik <[email protected]>; Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers On Thu, 15 Jun 2023 at 11:07, Hannu Krosing <[email protected]> wrote: > > One more unexpected benefit of having shared caches would be easing > access to other databases. > > If the system caches are there for all databases anyway, then it > becomes much easier to make queries using objects from multiple > databases. We have several optimizations in our visibility code that allow us to remove dead tuples from this database when another database still has a connection that has an old snapshot in which the deleting transaction of this database has not yet committed. This is allowed because we can say with confidence that other database's connections will never be able to see this database's tables. If we were to allow cross-database data access, that would require cross-database snapshot visibility checks, and that would severely hinder these optimizations. As an example, it would increase the work we need to do for snapshots: For the snapshot data of tables that aren't shared catalogs, we only need to consider our own database's backends for visibility. With cross-database visibility, we would need to consider all active backends for all snapshots, and this can be significantly more work. Kind regards, Matthias van de Meent Neon (https://neon.tech/) ^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2023-07-28 18:10 UTC | newest] Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-02-25 23:00 [PATCH 08/19] vcregress: add alltaptests Justin Pryzby <[email protected]> 2023-06-14 22:23 Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]> 2023-06-15 07:12 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]> 2023-06-15 08:41 ` Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]> 2023-06-15 09:04 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]> 2023-06-15 09:07 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]> 2023-07-28 18:10 ` Re: Let's make PostgreSQL multi-threaded Matthias van de Meent <[email protected]> 2023-06-15 19:49 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]> 2023-06-15 19:36 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox