Date: Thu, 14 Jul 2016 11:53:38 -0700
From: Andres Freund <andres@anarazel.de>
To: Vladimir Borodin <root@simply.name>
Cc: pgsql-performance@postgresql.org,
	PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: [PERFORM] 9.4 -> 9.5 regression with queries through
 pgbouncer on RHEL 6
Message-ID: <20160714185338.yo236a4rxkvlsmon@alap3.anarazel.de>
References: <26239E39-A838-4611-B7BA-0E96D2DF3253@simply.name>
 <AFD452B9-4EB3-4D4B-BD1D-A7C0FCFE0D83@gmail.com>
 <20160609222855.irpszcpo6qn4skq2@alap3.anarazel.de>
 <B8A94724-DD6E-463C-A0BC-4750FF71D69E@simply.name>
 <20160612215140.znmnl4mpzvgxjmgn@alap3.anarazel.de>
 <8C22C550-A2C3-48E1-AA00-07EB06EF0269@simply.name>
 <E8E521D6-0457-4A85-B085-59B8BA40AD2D@simply.name>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <E8E521D6-0457-4A85-B085-59B8BA40AD2D@simply.name>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

On 2016-07-04 16:30:51 +0300, Vladimir Borodin wrote:
>
> > 13 июня 2016 г., в 21:58, Vladimir Borodin <root@simply.name> написал(а):
> >
> >>
> >> 13 июня 2016 г., в 0:51, Andres Freund <andres@anarazel.de <mailto:andres@anarazel.de>> написал(а):
> >>
> >> Hi Vladimir,
> >>
> >> Thanks for these reports.
> >>
> >> On 2016-06-13 00:42:19 +0300, Vladimir Borodin wrote:
> >>> perf report -g -i pg9?_all.data >/tmp/pg9?_perf_report.txt
> >>
> >> Any chance you could redo the reports with --no-children --call-graph=fractal
> >> added? The mode that includes child overheads unfortunately makes the
> >> output hard to interpet/compare.
> >
> > Of course. Not sure if that is important but I upgraded perf for that (because --no-children option was introduced in ~3.16), so perf record and perf report were done with different perf versions.
> >
> > <pg94_perf_report.txt.gz>
> > <pg95_perf_report.txt.gz>
> > <pg96_perf_report.txt.gz>
> >
> > Also I’ve done the same test on same host (RHEL 6) but with 4.6
> > kernel/perf and writing perf data to /dev/shm for not loosing
> > events. Perf report output is also attached but important thing is
> > that the regression is not so significant:

FWIW, you can instead use -F 300 or something to reduce the sampling
frequency.

> > root@pgload05g ~ # uname -r
> > 4.6.0-1.el6.elrepo.x86_64
> > root@pgload05g ~ # cat /proc/sys/kernel/sched_autogroup_enabled
> > 1
> > root@pgload05g ~ # /tmp/run.sh
> > RHEL 6		9.4		71634		0.893
> > RHEL 6		9.5		54005		1.185
> > RHEL 6		9.6		65550		0.976
> > root@pgload05g ~ # echo 0 >/proc/sys/kernel/sched_autogroup_enabled
> > root@pgload05g ~ # /tmp/run.sh
> > RHEL 6		9.4		73041		0.876
> > RHEL 6		9.5		60105		1.065
> > RHEL 6		9.6		67984		0.941
> > root@pgload05g ~ #
> >
> > <pg96_perf_report_4.6.txt.gz>
> > <pg95_perf_report_4.6.txt.gz>
> > <pg94_perf_report_4.6.txt.gz>
>
> Andres, is there any chance that you would find time to look at those results? Are they actually useful?

I don't really see anything suspicious in the profile. This looks more
like a kernel scheduler issue than a postgres bottleneck one. It seems
that somehow using nonblocking IO (started in 9.5) causes scheduling
issues when pgbouncer is also local.

Could you do perf stat -ddd -a sleep 10 or something during both runs? I
suspect that the context switch ratios will be quite different.

Andres


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers