public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 08/19] vcregress: add alltaptests
9+ messages / 5 participants
[nested] [flat]
* [PATCH 08/19] vcregress: add alltaptests
@ 2022-02-25 23:00 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 9+ messages in thread
From: Justin Pryzby @ 2022-02-25 23:00 UTC (permalink / raw)
https://www.postgresql.org/message-id/[email protected]
https://www.postgresql.org/message-id/flat/[email protected]
In passing, document taptest PROVE_FLAGS
ci-os-only: windows
See also:
d835dd6685246f0737ca42ab68242210681bb220
13d856e177e69083f543d6383eeda9e12ce3c55c
fed6df486dca1b9e53d3f560031b9a236c99f4bb
---
.cirrus.yml | 10 ++-------
src/tools/msvc/vcregress.pl | 43 +++++++++++++++++++++++++++++++++----
2 files changed, 41 insertions(+), 12 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index f2bb05266cc..ea1f493923a 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -437,14 +437,8 @@ task:
test_ssl_script: |
set with_ssl=openssl
%T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/ssl/
- test_subscription_script: |
- %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/subscription/
- test_authentication_script: |
- %T_C% perl src/tools/msvc/vcregress.pl taptest ./src/test/authentication/
- test_recovery_script: |
- %T_C% perl src/tools/msvc/vcregress.pl recoverycheck
- test_bin_script: |
- %T_C% perl src/tools/msvc/vcregress.pl bincheck
+ test_tap_script: |
+ %T_C% perl src/tools/msvc/vcregress.pl alltaptests
test_ecpg_script: |
rem tries to build additional stuff
vcvarsall x64
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 54c31b0e14f..1b1387acac0 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -51,7 +51,7 @@ if (-e "src/tools/msvc/buildenv.pl")
my $what = shift || "";
if ($what =~
- /^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck|taptest)$/i
+ /^(check|installcheck|plcheck|contribcheck|modulescheck|ecpgcheck|isolationcheck|upgradecheck|bincheck|recoverycheck|taptest|alltaptests)$/i
)
{
$what = uc $what;
@@ -109,6 +109,7 @@ my %command = (
BINCHECK => \&bincheck,
RECOVERYCHECK => \&recoverycheck,
UPGRADECHECK => \&upgradecheck, # no-op
+ ALLTAPTESTS => \&alltaptests,
TAPTEST => \&taptest,);
my $proc = $command{$what};
@@ -291,6 +292,9 @@ sub tap_check
$ENV{PG_REGRESS} = "$topdir/$Config/pg_regress/pg_regress";
$ENV{REGRESS_SHLIB} = "$topdir/src/test/regress/regress.dll";
+ print "============================================================\n";
+ print "Checking $dir: @args\n";
+
rmtree('tmp_check');
system(@args);
my $status = $? >> 8;
@@ -305,8 +309,7 @@ sub bincheck
my $mstat = 0;
- # Find out all the existing TAP tests by looking for t/ directories
- # in the tree.
+ # Find the TAP tests by looking for t/ directories
my @bin_dirs = glob("$topdir/src/bin/*");
# Process each test
@@ -321,6 +324,36 @@ sub bincheck
return;
}
+sub alltaptests
+{
+ InstallTemp();
+
+ my $mstat = 0;
+
+ # Find out all the existing TAP tests by looking for t/ directories
+ # in the tree.
+ my @tap_dirs = ();
+ my @top_dir = ($topdir);
+ File::Find::find(
+ { wanted => sub {
+ /^t\z/s
+ && $File::Find::name !~ /\/(kerberos|ldap|ssl|ssl_passphrase_callback)\// # opt-in
+ && push(@tap_dirs, $File::Find::name);
+ }
+ },
+ @top_dir);
+
+ # Process each test
+ foreach my $test_path (@tap_dirs)
+ {
+ my $dir = dirname($test_path);
+ my $status = tap_check($dir);
+ $mstat ||= $status;
+ }
+ exit $mstat if $mstat;
+ return;
+}
+
sub taptest
{
my $dir = shift;
@@ -671,6 +704,7 @@ sub usage
print STDERR
"Usage: vcregress.pl <mode> [<arg>]\n\n",
"Options for <mode>:\n",
+ " alltaptests run all tap tests (except kerberos, ldap, ssl, ssl_passphrase_callback)\n",
" bincheck run tests of utilities in src/bin/\n",
" check deploy instance and run regression tests on it\n",
" contribcheck run tests of modules in contrib/\n",
@@ -688,6 +722,7 @@ sub usage
"\nOptions for <arg>: (used by contribcheck and modulescheck)\n",
" install also run tests which require a new instance\n",
"\nOption for <arg>: for taptest\n",
- " TEST_DIR (required) directory where tests reside\n";
+ " TEST_DIR (required) directory where tests reside\n",
+ " PROVE_FLAGS flags to pass to prove\n";
exit(1);
}
--
2.17.1
--Sw7tCqrGA+HQ0/zt
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="0009-tmp-run-tap-tests-first.patch"
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded
@ 2023-06-14 22:23 James Addison <[email protected]>
2023-06-15 07:12 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 9+ messages in thread
From: James Addison @ 2023-06-14 22:23 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Hannu Krosing <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers
On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik <[email protected]> wrote:
>
>
>
> On 12.06.2023 3:23 PM, Pavel Borisov wrote:
> > Is the following true or not?
> >
> > 1. If we switch processes to threads but leave the amount of session
> > local variables unchanged, there would be hardly any performance gain.
> > 2. If we move some backend's local variables into shared memory then
> > the performance gain would be very near to what we get with threads
> > having equal amount of session-local variables.
> >
> > In other words, the overall goal in principle is to gain from less
> > memory copying wherever it doesn't add the burden of locks for
> > concurrent variables access?
> >
> > Regards,
> > Pavel Borisov,
> > Supabase
> >
> >
> IMHO both statements are not true.
> Switching to threads will cause less context switch overhead (because
> all threads are sharing the same memory space and so preserve TLB.
> How big will be this advantage? In my prototype I got ~10%. But may be
> it is possible to fin workloads when it is larger.
Hi Konstantin - do you have code/links that you can share for the
prototype and benchmarks used to gather those results?
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded
2023-06-14 22:23 Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
@ 2023-06-15 07:12 ` Konstantin Knizhnik <[email protected]>
2023-06-15 08:41 ` Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
0 siblings, 1 reply; 9+ messages in thread
From: Konstantin Knizhnik @ 2023-06-15 07:12 UTC (permalink / raw)
To: James Addison <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Hannu Krosing <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers
On 15.06.2023 1:23 AM, James Addison wrote:
> On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik<[email protected]> wrote:
>>
>>
>> On 12.06.2023 3:23 PM, Pavel Borisov wrote:
>>> Is the following true or not?
>>>
>>> 1. If we switch processes to threads but leave the amount of session
>>> local variables unchanged, there would be hardly any performance gain.
>>> 2. If we move some backend's local variables into shared memory then
>>> the performance gain would be very near to what we get with threads
>>> having equal amount of session-local variables.
>>>
>>> In other words, the overall goal in principle is to gain from less
>>> memory copying wherever it doesn't add the burden of locks for
>>> concurrent variables access?
>>>
>>> Regards,
>>> Pavel Borisov,
>>> Supabase
>>>
>>>
>> IMHO both statements are not true.
>> Switching to threads will cause less context switch overhead (because
>> all threads are sharing the same memory space and so preserve TLB.
>> How big will be this advantage? In my prototype I got ~10%. But may be
>> it is possible to fin workloads when it is larger.
> Hi Konstantin - do you have code/links that you can share for the
> prototype and benchmarks used to gather those results?
Sorry, I have already shared the link:
https://github.com/postgrespro/postgresql.pthreads/
As you can see last commit was 6 years ago when I stopped work on this
project.
Why? I already tried to explain it:
- benefits from switching to threads were not so large. May be I just
failed to fid proper workload, but is was more or less expected result,
because most of the code was not changed - it uses the same sync
primitives, the same local catalog/relation caches,..
To take all advantage of multithreadig model it is necessary to rewrite
many components, especially related with interprocess communication.
But maintaining such fork of Postgres and synchronize it with mainstream
requires too much efforts and I was not able to do it myself.
There are three different but related directions of improving current
Postgres:
1. Replacing processes with threads
2. Builtin connection pooler
3. Lightweight backends (shared catalog/relation/prepared statements caches)
The motivation for such changes are also similar:
1. Increase Postgres scalability
2. Reduce memory consumption
3. Make Postgres better fir cloud and serverless requirements
I am not sure now which one should be addressed first or them can be
done together.
Replacing static variables with thread-local is the first and may be the
easiest step.
It requires more or less mechanical changes. More challenging thing is
replacing private per-backend data structures
with shared ones (caches, file descriptors,...)
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded
2023-06-14 22:23 Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 07:12 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
@ 2023-06-15 08:41 ` James Addison <[email protected]>
2023-06-15 09:04 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]>
2023-06-15 19:36 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
0 siblings, 2 replies; 9+ messages in thread
From: James Addison @ 2023-06-15 08:41 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Hannu Krosing <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers
On Thu, 15 Jun 2023 at 08:12, Konstantin Knizhnik <[email protected]> wrote:
>
>
>
> On 15.06.2023 1:23 AM, James Addison wrote:
>
> On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik <[email protected]> wrote:
>
>
> On 12.06.2023 3:23 PM, Pavel Borisov wrote:
>
> Is the following true or not?
>
> 1. If we switch processes to threads but leave the amount of session
> local variables unchanged, there would be hardly any performance gain.
> 2. If we move some backend's local variables into shared memory then
> the performance gain would be very near to what we get with threads
> having equal amount of session-local variables.
>
> In other words, the overall goal in principle is to gain from less
> memory copying wherever it doesn't add the burden of locks for
> concurrent variables access?
>
> Regards,
> Pavel Borisov,
> Supabase
>
>
> IMHO both statements are not true.
> Switching to threads will cause less context switch overhead (because
> all threads are sharing the same memory space and so preserve TLB.
> How big will be this advantage? In my prototype I got ~10%. But may be
> it is possible to fin workloads when it is larger.
>
> Hi Konstantin - do you have code/links that you can share for the
> prototype and benchmarks used to gather those results?
>
>
>
> Sorry, I have already shared the link:
> https://github.com/postgrespro/postgresql.pthreads/
Nope, my mistake for not locating the existing link - thank you.
Is there a reason that parser-related files (flex/bison) are added as
part of the changeset? (I'm trying to narrow it down to only the
changes necessary for the functionality. so far it looks mostly
fairly minimal, which is good. the adjustments to progname are
another thing that look a bit unusual/maybe unnecessary for the
feature)
> As you can see last commit was 6 years ago when I stopped work on this project.
> Why? I already tried to explain it:
> - benefits from switching to threads were not so large. May be I just failed to fid proper workload, but is was more or less expected result,
> because most of the code was not changed - it uses the same sync primitives, the same local catalog/relation caches,..
> To take all advantage of multithreadig model it is necessary to rewrite many components, especially related with interprocess communication.
> But maintaining such fork of Postgres and synchronize it with mainstream requires too much efforts and I was not able to do it myself.
I get the feeling that there are probably certain query types or
patterns where a significant, order-of-magnitude speedup is possible
with threads - but yep, I haven't seen those described in detail yet
on the mailing list (but as hinted by my not noticing the github link
previously, maybe I'm not following the list closely enough).
What workloads did you try with your version of the project?
> There are three different but related directions of improving current Postgres:
> 1. Replacing processes with threads
> 2. Builtin connection pooler
> 3. Lightweight backends (shared catalog/relation/prepared statements caches)
>
> The motivation for such changes are also similar:
> 1. Increase Postgres scalability
> 2. Reduce memory consumption
> 3. Make Postgres better fir cloud and serverless requirements
>
> I am not sure now which one should be addressed first or them can be done together.
>
> Replacing static variables with thread-local is the first and may be the easiest step.
> It requires more or less mechanical changes. More challenging thing is replacing private per-backend data structures
> with shared ones (caches, file descriptors,...)
Thank you. Personally I think that motivation two (reducing memory
consumption) -- as long as it can be done without detrimentally
affecting functionality or correctness, and without making the code
harder to develop/understand -- could provide benefits for all three
of the motivating cases (and, in fact, for non-cloud/serverful use
cases too).
This is making me wonder about other performance/scalability areas
that might not have been considered due to focus on the details of the
existing codebase, but I'll save that for another thread and will try
to learn more first.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded
2023-06-14 22:23 Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 07:12 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
2023-06-15 08:41 ` Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
@ 2023-06-15 09:04 ` Hannu Krosing <[email protected]>
2023-06-15 09:07 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]>
2023-06-15 19:49 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
1 sibling, 2 replies; 9+ messages in thread
From: Hannu Krosing @ 2023-06-15 09:04 UTC (permalink / raw)
To: James Addison <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers
On Thu, Jun 15, 2023 at 10:41 AM James Addison <[email protected]> wrote:
>
> This is making me wonder about other performance/scalability areas
> that might not have been considered due to focus on the details of the
> existing codebase, but I'll save that for another thread and will try
> to learn more first.
A gradual move to more shared structures seems to be a way forward
It should get us all the benefits of threading minus the need for TLB
reloading and (in some cases) reduction of per-process virtual memory
mapping tables.
In any case we would need to implement all the locking and parallelism
management of these shared structures that are not there in the
current process architecture.
So a fair bit of work but also a clearly defined benefits of
1) reduced memory usage
2) no need to rebuild caches for each new connection
3) no need to track PREPARE statements inside connection poolers.
There can be extra complexity when different connections use the same
prepared statement name (say "PREP001") for different queries.
For this wel likely will need a good cooperation with connection
pooler where it passes some kind of client connection id along at the
transaction start
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded
2023-06-14 22:23 Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 07:12 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
2023-06-15 08:41 ` Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 09:04 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]>
@ 2023-06-15 09:07 ` Hannu Krosing <[email protected]>
2023-07-28 18:10 ` Re: Let's make PostgreSQL multi-threaded Matthias van de Meent <[email protected]>
1 sibling, 1 reply; 9+ messages in thread
From: Hannu Krosing @ 2023-06-15 09:07 UTC (permalink / raw)
To: James Addison <[email protected]>; +Cc: Konstantin Knizhnik <[email protected]>; Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers
One more unexpected benefit of having shared caches would be easing
access to other databases.
If the system caches are there for all databases anyway, then it
becomes much easier to make queries using objects from multiple
databases.
Note that this does not strictly need threads, just shared caches.
On Thu, Jun 15, 2023 at 11:04 AM Hannu Krosing <[email protected]> wrote:
>
> On Thu, Jun 15, 2023 at 10:41 AM James Addison <[email protected]> wrote:
> >
> > This is making me wonder about other performance/scalability areas
> > that might not have been considered due to focus on the details of the
> > existing codebase, but I'll save that for another thread and will try
> > to learn more first.
>
> A gradual move to more shared structures seems to be a way forward
>
> It should get us all the benefits of threading minus the need for TLB
> reloading and (in some cases) reduction of per-process virtual memory
> mapping tables.
>
> In any case we would need to implement all the locking and parallelism
> management of these shared structures that are not there in the
> current process architecture.
>
> So a fair bit of work but also a clearly defined benefits of
> 1) reduced memory usage
> 2) no need to rebuild caches for each new connection
> 3) no need to track PREPARE statements inside connection poolers.
>
> There can be extra complexity when different connections use the same
> prepared statement name (say "PREP001") for different queries.
> For this wel likely will need a good cooperation with connection
> pooler where it passes some kind of client connection id along at the
> transaction start
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded
2023-06-14 22:23 Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 07:12 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
2023-06-15 08:41 ` Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 09:04 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]>
2023-06-15 09:07 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]>
@ 2023-07-28 18:10 ` Matthias van de Meent <[email protected]>
0 siblings, 0 replies; 9+ messages in thread
From: Matthias van de Meent @ 2023-07-28 18:10 UTC (permalink / raw)
To: Hannu Krosing <[email protected]>; +Cc: James Addison <[email protected]>; Konstantin Knizhnik <[email protected]>; Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers
On Thu, 15 Jun 2023 at 11:07, Hannu Krosing <[email protected]> wrote:
>
> One more unexpected benefit of having shared caches would be easing
> access to other databases.
>
> If the system caches are there for all databases anyway, then it
> becomes much easier to make queries using objects from multiple
> databases.
We have several optimizations in our visibility code that allow us to
remove dead tuples from this database when another database still has
a connection that has an old snapshot in which the deleting
transaction of this database has not yet committed. This is allowed
because we can say with confidence that other database's connections
will never be able to see this database's tables. If we were to allow
cross-database data access, that would require cross-database snapshot
visibility checks, and that would severely hinder these optimizations.
As an example, it would increase the work we need to do for snapshots:
For the snapshot data of tables that aren't shared catalogs, we only
need to consider our own database's backends for visibility. With
cross-database visibility, we would need to consider all active
backends for all snapshots, and this can be significantly more work.
Kind regards,
Matthias van de Meent
Neon (https://neon.tech/)
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded
2023-06-14 22:23 Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 07:12 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
2023-06-15 08:41 ` Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 09:04 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]>
@ 2023-06-15 19:49 ` Konstantin Knizhnik <[email protected]>
1 sibling, 0 replies; 9+ messages in thread
From: Konstantin Knizhnik @ 2023-06-15 19:49 UTC (permalink / raw)
To: Hannu Krosing <[email protected]>; James Addison <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers
On 15.06.2023 12:04 PM, Hannu Krosing wrote:
> So a fair bit of work but also a clearly defined benefits of
> 1) reduced memory usage
> 2) no need to rebuild caches for each new connection
> 3) no need to track PREPARE statements inside connection poolers.
Shared plan cache (not only prepared statements cache) also opens way to
more sophisticated query optimizations.
Right now we are not performing some optimization (like constant
expression folding) just because them increase time of processing normal
queries.
This is why queries generated by ORMs or wizards, which can contain a
lot of dumb stuff, are not well simplified by Postgres.
With MS-Sql it is quite frequent that query execution time is much
smaller than query optimization time.
Having shared plan cache allows us to spend more time in optimization
without risk to degrade performance.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Let's make PostgreSQL multi-threaded
2023-06-14 22:23 Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 07:12 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
2023-06-15 08:41 ` Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
@ 2023-06-15 19:36 ` Konstantin Knizhnik <[email protected]>
1 sibling, 0 replies; 9+ messages in thread
From: Konstantin Knizhnik @ 2023-06-15 19:36 UTC (permalink / raw)
To: James Addison <[email protected]>; +Cc: Pavel Borisov <[email protected]>; Dilip Kumar <[email protected]>; Hannu Krosing <[email protected]>; Heikki Linnakangas <[email protected]>; pgsql-hackers
On 15.06.2023 11:41 AM, James Addison wrote:
> On Thu, 15 Jun 2023 at 08:12, Konstantin Knizhnik <[email protected]> wrote:
>>
>>
>> On 15.06.2023 1:23 AM, James Addison wrote:
>>
>> On Tue, 13 Jun 2023 at 07:55, Konstantin Knizhnik <[email protected]> wrote:
>>
>>
>> On 12.06.2023 3:23 PM, Pavel Borisov wrote:
>>
>> Is the following true or not?
>>
>> 1. If we switch processes to threads but leave the amount of session
>> local variables unchanged, there would be hardly any performance gain.
>> 2. If we move some backend's local variables into shared memory then
>> the performance gain would be very near to what we get with threads
>> having equal amount of session-local variables.
>>
>> In other words, the overall goal in principle is to gain from less
>> memory copying wherever it doesn't add the burden of locks for
>> concurrent variables access?
>>
>> Regards,
>> Pavel Borisov,
>> Supabase
>>
>>
>> IMHO both statements are not true.
>> Switching to threads will cause less context switch overhead (because
>> all threads are sharing the same memory space and so preserve TLB.
>> How big will be this advantage? In my prototype I got ~10%. But may be
>> it is possible to fin workloads when it is larger.
>>
>> Hi Konstantin - do you have code/links that you can share for the
>> prototype and benchmarks used to gather those results?
>>
>>
>>
>> Sorry, I have already shared the link:
>> https://github.com/postgrespro/postgresql.pthreads/
> Nope, my mistake for not locating the existing link - thank you.
>
> Is there a reason that parser-related files (flex/bison) are added as
> part of the changeset? (I'm trying to narrow it down to only the
> changes necessary for the functionality. so far it looks mostly
> fairly minimal, which is good. the adjustments to progname are
> another thing that look a bit unusual/maybe unnecessary for the
> feature)
Sorry, absolutely no reason - just my fault.
>> As you can see last commit was 6 years ago when I stopped work on this project.
>> Why? I already tried to explain it:
>> - benefits from switching to threads were not so large. May be I just failed to fid proper workload, but is was more or less expected result,
>> because most of the code was not changed - it uses the same sync primitives, the same local catalog/relation caches,..
>> To take all advantage of multithreadig model it is necessary to rewrite many components, especially related with interprocess communication.
>> But maintaining such fork of Postgres and synchronize it with mainstream requires too much efforts and I was not able to do it myself.
> I get the feeling that there are probably certain query types or
> patterns where a significant, order-of-magnitude speedup is possible
> with threads - but yep, I haven't seen those described in detail yet
> on the mailing list (but as hinted by my not noticing the github link
> previously, maybe I'm not following the list closely enough).
>
> What workloads did you try with your version of the project?
I do not remember now precisely (6 years passed).
But definitely I tried pgbench, especially read-only pgbench (to be more
CPU rather than disk bounded)
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2023-07-28 18:10 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-02-25 23:00 [PATCH 08/19] vcregress: add alltaptests Justin Pryzby <[email protected]>
2023-06-14 22:23 Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 07:12 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
2023-06-15 08:41 ` Re: Let's make PostgreSQL multi-threaded James Addison <[email protected]>
2023-06-15 09:04 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]>
2023-06-15 09:07 ` Re: Let's make PostgreSQL multi-threaded Hannu Krosing <[email protected]>
2023-07-28 18:10 ` Re: Let's make PostgreSQL multi-threaded Matthias van de Meent <[email protected]>
2023-06-15 19:49 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
2023-06-15 19:36 ` Re: Let's make PostgreSQL multi-threaded Konstantin Knizhnik <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox