pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedRe: Hanging queries on dual CPU windows
15+ messages / 4 participants
[nested] [flat]
* Re: Hanging queries on dual CPU windows
@ 2006-03-09 20:07 Jan de Visser <jdevisser@digitalfairway.com>
2006-03-09 20:10 ` Re: Hanging queries on dual CPU windows Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 15+ messages in thread
From: Jan de Visser @ 2006-03-09 20:07 UTC (permalink / raw)
To: pgsql-performance
I have more information on this issue.
First of, the problem now happens after about 1-2 hours, as opposed to the 6-8
I mentioned earlier. Yey for shorter test cycles.
Furtermore, it does not happen on Linux machines, both single CPU and dual
CPU, nor on single CPU windows machines. We can only reproduce on a dual CPU
windows machine, and if we take one CPU out, it does not happen.
I executed the following after it hung:
db=# select l.pid, c.relname, l.mode, l.granted, l.page, l.tuple
from pg_locks l, pg_class c where c.oid = l.relation order by l.pid;
Which showed me that several transactions where waiting for a particular row
which was locked by another transaction. This transaction had no pending
locks (so no deadlock), but just does not complete and hence never
relinquishes the lock.
What gives? has anybody ever heard of problems like this on dual CPU windows
machines?
jan
On Monday 06 March 2006 09:38, Jan de Visser wrote:
> Hello,
>
> While doing performance tests on Windows Server 2003 we observed to
> following two problems.
>
> Environment: J2EE application running in JBoss application server, against
> pgsql 8.1 database. Load is caused by a smallish number of (very) complex
> transactions, typically about 5-10 concurrently.
>
> The first one, which bothers me the most, is that after about 6-8 hours the
> application stops processing. No errors are reported, neither by the JDBC
> driver nor by the server, but when I kill the application server, I see
> that all my connections hang in a SQL statements (which never seem to
> return):
>
> 2006-03-03 08:17:12 4504 6632560 LOG: duration: 45087000.000 ms
> statement: EXECUTE <unnamed> [PREPARE: SELECT objID FROM objects WHERE
> objID = $1 FOR UPDATE]
>
> I think I can reliably reproduce this by loading the app, and waiting a
> couple of hours.
--
--------------------------------------------------------------
Jan de Visser jdevisser@digitalfairway.com
Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
2006-03-09 20:07 Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
@ 2006-03-09 20:10 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-03-09 21:15 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 02:00 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 2 replies; 15+ messages in thread
From: Tom Lane @ 2006-03-09 20:10 UTC (permalink / raw)
To: Jan de Visser <jdevisser@digitalfairway.com>; +Cc: pgsql-performance
Jan de Visser <jdevisser@digitalfairway.com> writes:
> Furtermore, it does not happen on Linux machines, both single CPU and dual
> CPU, nor on single CPU windows machines. We can only reproduce on a dual CPU
> windows machine, and if we take one CPU out, it does not happen.
> ...
> Which showed me that several transactions where waiting for a particular row
> which was locked by another transaction. This transaction had no pending
> locks (so no deadlock), but just does not complete and hence never
> relinquishes the lock.
Is the stuck transaction still consuming CPU time, or just stopped?
Is it possible to get a stack trace from the stuck process? I dunno
if you've got anything gdb-equivalent under Windows, but that's the
first thing I'd be interested in ...
regards, tom lane
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
2006-03-09 20:07 Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
2006-03-09 20:10 ` Re: Hanging queries on dual CPU windows Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-03-09 21:15 ` Jan de Visser <jdevisser@digitalfairway.com>
1 sibling, 0 replies; 15+ messages in thread
From: Jan de Visser @ 2006-03-09 21:15 UTC (permalink / raw)
To: pgsql-performance
On Thursday 09 March 2006 15:10, Tom Lane wrote:
> Jan de Visser <jdevisser@digitalfairway.com> writes:
> > Furtermore, it does not happen on Linux machines, both single CPU and
> > dual CPU, nor on single CPU windows machines. We can only reproduce on a
> > dual CPU windows machine, and if we take one CPU out, it does not happen.
> > ...
> > Which showed me that several transactions where waiting for a particular
> > row which was locked by another transaction. This transaction had no
> > pending locks (so no deadlock), but just does not complete and hence
> > never relinquishes the lock.
>
> Is the stuck transaction still consuming CPU time, or just stopped?
CPU drops off. In fact, that's my main clue something's wrong ;-)
>
> Is it possible to get a stack trace from the stuck process? I dunno
> if you've got anything gdb-equivalent under Windows, but that's the
> first thing I'd be interested in ...
I wouldn't know. I'm hardly a windows expert. Prefer not to touch the stuff,
myself. Can do some research though...
>
> regards, tom lane
jan
--
--------------------------------------------------------------
Jan de Visser jdevisser@digitalfairway.com
Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
2006-03-09 20:07 Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
2006-03-09 20:10 ` Re: Hanging queries on dual CPU windows Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-03-10 02:00 ` Jan de Visser <jdevisser@digitalfairway.com>
1 sibling, 0 replies; 15+ messages in thread
From: Jan de Visser @ 2006-03-10 02:00 UTC (permalink / raw)
To: pgsql-performance
On Thursday 09 March 2006 15:10, Tom Lane wrote:
> Is it possible to get a stack trace from the stuck process? I dunno
> if you've got anything gdb-equivalent under Windows, but that's the
> first thing I'd be interested in ...
Here ya go:
http://www.devisser-siderius.com/stack1.jpg
http://www.devisser-siderius.com/stack2.jpg
http://www.devisser-siderius.com/stack3.jpg
There are three threads in the process. I guess thread 1 (stack1.jpg) is the
most interesting.
I also noted that cranking up concurrency in my app reproduces the problem in
about 4 minutes ;-)
With thanks to Magnus Hagander for the Process Explorer hint.
jan
--
--------------------------------------------------------------
Jan de Visser jdevisser@digitalfairway.com
Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
@ 2006-03-09 21:22 Magnus Hagander <mha@sollentuna.net>
0 siblings, 0 replies; 15+ messages in thread
From: Magnus Hagander @ 2006-03-09 21:22 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Jan de Visser <jdevisser@digitalfairway.com>; +Cc: pgsql-performance
> Is it possible to get a stack trace from the stuck process?
> I dunno if you've got anything gdb-equivalent under Windows,
> but that's the first thing I'd be interested in ...
Try Process Explorer from www.sysinternals.com.
//Magnus
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
@ 2006-03-10 09:20 Magnus Hagander <mha@sollentuna.net>
2006-03-10 14:03 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 1 reply; 15+ messages in thread
From: Magnus Hagander @ 2006-03-10 09:20 UTC (permalink / raw)
To: Jan de Visser <jdevisser@digitalfairway.com>; pgsql-performance
> > Is it possible to get a stack trace from the stuck process?
> I dunno
> > if you've got anything gdb-equivalent under Windows, but that's the
> > first thing I'd be interested in ...
>
> Here ya go:
>
> http://www.devisser-siderius.com/stack1.jpg
> http://www.devisser-siderius.com/stack2.jpg
> http://www.devisser-siderius.com/stack3.jpg
>
> There are three threads in the process. I guess thread 1
> (stack1.jpg) is the most interesting.
>
> I also noted that cranking up concurrency in my app
> reproduces the problem in about 4 minutes ;-)
Actually, stack2 looks very interesting. Does it "stay stuck" in pg_queue_signal? That's really not supposed to happen.
Also, can you confirm that stack1 actually *stops* in pgwin32_waitforsinglesocket? Or does it go out and come back? ;-)
(A good signal of this is to check the cswitch delta. If it stays at zero, then it's stuck. If it shows any values, that means it's actuall going out and coming back)
And finally, is this 8.0 or 8.1? There have been some significant changes in the handling of the signals between the two...
//Magnus
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
2006-03-10 09:20 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
@ 2006-03-10 14:03 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 14:32 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 1 reply; 15+ messages in thread
From: Jan de Visser @ 2006-03-10 14:03 UTC (permalink / raw)
To: pgsql-performance
On Friday 10 March 2006 04:20, Magnus Hagander wrote:
> > > Is it possible to get a stack trace from the stuck process?
> >
> > I dunno
> >
> > > if you've got anything gdb-equivalent under Windows, but that's the
> > > first thing I'd be interested in ...
> >
> > Here ya go:
> >
> > http://www.devisser-siderius.com/stack1.jpg
> > http://www.devisser-siderius.com/stack2.jpg
> > http://www.devisser-siderius.com/stack3.jpg
> >
> > There are three threads in the process. I guess thread 1
> > (stack1.jpg) is the most interesting.
> >
> > I also noted that cranking up concurrency in my app
> > reproduces the problem in about 4 minutes ;-)
>
Just reproduced again.
> Actually, stack2 looks very interesting. Does it "stay stuck" in
> pg_queue_signal? That's really not supposed to happen.
Yes it does.
>
> Also, can you confirm that stack1 actually *stops* in
> pgwin32_waitforsinglesocket? Or does it go out and come back? ;-)
>
> (A good signal of this is to check the cswitch delta. If it stays at zero,
> then it's stuck. If it shows any values, that means it's actuall going out
> and coming back)
I only see CSwitch change once I click OK on the thread window. Once I do
that, it goes up to 3 and back to blank again. The 'context switches' counter
does not increase like it does for other processes (like e.g. process
explorer itself).
Another thing which may or may not be of interest: Nothing is listed in the
'TCP/IP' tab for the stuck process. I would have expected to see at least the
socket of the client connection there??
>
> And finally, is this 8.0 or 8.1? There have been some significant changes
> in the handling of the signals between the two...
This is 8.1.3 on Windows 2003 Server. Also reproduced on 8.1.0 and 8.1.1 (also
on 2K3).
>
> //Magnus
jan
--
--------------------------------------------------------------
Jan de Visser jdevisser@digitalfairway.com
Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
2006-03-10 09:20 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
2006-03-10 14:03 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
@ 2006-03-10 14:32 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 14:47 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 1 reply; 15+ messages in thread
From: Jan de Visser @ 2006-03-10 14:32 UTC (permalink / raw)
To: pgsql-performance
On Friday 10 March 2006 09:03, Jan de Visser wrote:
> On Friday 10 March 2006 04:20, Magnus Hagander wrote:
> > > > Is it possible to get a stack trace from the stuck process?
> > >
> > > I dunno
> > >
> > > > if you've got anything gdb-equivalent under Windows, but that's the
> > > > first thing I'd be interested in ...
> > >
> > > Here ya go:
> > >
> > > http://www.devisser-siderius.com/stack1.jpg
> > > http://www.devisser-siderius.com/stack2.jpg
> > > http://www.devisser-siderius.com/stack3.jpg
> > >
> > > There are three threads in the process. I guess thread 1
> > > (stack1.jpg) is the most interesting.
> > >
> > > I also noted that cranking up concurrency in my app
> > > reproduces the problem in about 4 minutes ;-)
>
> Just reproduced again.
>
> > Actually, stack2 looks very interesting. Does it "stay stuck" in
> > pg_queue_signal? That's really not supposed to happen.
>
> Yes it does.
An update on that: There is actually *two* processes in this state, both
hanging in pg_queue_signal. I've looked at the source of that, and the
obvious candidate for hanging is EnterCriticalSection. I also found this:
http://blogs.msdn.com/larryosterman/archive/2005/03/02/383685.aspx
where they say:
"
In addition, for Windows 2003, SP1, the EnterCriticalSection API has a subtle
change that's intended tor resolve many of the lock convoy issues. Before
Win2003 SP1, if 10 threads were blocked on EnterCriticalSection and all 10
threads had the same priority, then EnterCriticalSection would service those
threads in a FIFO (first -in, first-out) basis. Starting in Windows 2003
SP1, the EnterCriticalSection will wake up a random thread from the waiting
threads. If all the threads are doing the same thing (like a thread pool)
this won't make much of a difference, but if the different threads are doing
different work (like the critical section protecting a widely accessed
object), this will go a long way towards removing lock convoy semantics.
"
Could it be they broke it when they did that????
>
> > Also, can you confirm that stack1 actually *stops* in
> > pgwin32_waitforsinglesocket? Or does it go out and come back? ;-)
> >
> > (A good signal of this is to check the cswitch delta. If it stays at
> > zero, then it's stuck. If it shows any values, that means it's actuall
> > going out and coming back)
>
> I only see CSwitch change once I click OK on the thread window. Once I do
> that, it goes up to 3 and back to blank again. The 'context switches'
> counter does not increase like it does for other processes (like e.g.
> process explorer itself).
>
> Another thing which may or may not be of interest: Nothing is listed in the
> 'TCP/IP' tab for the stuck process. I would have expected to see at least
> the socket of the client connection there??
>
> > And finally, is this 8.0 or 8.1? There have been some significant changes
> > in the handling of the signals between the two...
>
> This is 8.1.3 on Windows 2003 Server. Also reproduced on 8.1.0 and 8.1.1
> (also on 2K3).
>
> > //Magnus
>
> jan
--
--------------------------------------------------------------
Jan de Visser jdevisser@digitalfairway.com
Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
2006-03-10 09:20 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
2006-03-10 14:03 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 14:32 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
@ 2006-03-10 14:47 ` Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 0 replies; 15+ messages in thread
From: Jan de Visser @ 2006-03-10 14:47 UTC (permalink / raw)
To: pgsql-performance
On Friday 10 March 2006 09:32, Jan de Visser wrote:
> > > Actually, stack2 looks very interesting. Does it "stay stuck" in
> > > pg_queue_signal? That's really not supposed to happen.
> >
> > Yes it does.
>
> An update on that: There is actually *two* processes in this state, both
> hanging in pg_queue_signal. I've looked at the source of that, and the
> obvious candidate for hanging is EnterCriticalSection. I also found this:
>
> http://blogs.msdn.com/larryosterman/archive/2005/03/02/383685.aspx
>
> where they say:
>
> "
> In addition, for Windows 2003, SP1, the EnterCriticalSection API has a
> subtle change that's intended tor resolve many of the lock convoy issues.
> Before Win2003 SP1, if 10 threads were blocked on EnterCriticalSection and
> all 10 threads had the same priority, then EnterCriticalSection would
> service those threads in a FIFO (first -in, first-out) basis. Starting in
> Windows 2003 SP1, the EnterCriticalSection will wake up a random thread
> from the waiting threads. If all the threads are doing the same thing
> (like a thread pool) this won't make much of a difference, but if the
> different threads are doing different work (like the critical section
> protecting a widely accessed object), this will go a long way towards
> removing lock convoy semantics. "
>
> Could it be they broke it when they did that????
See also this:
http://bugs.mysql.com/bug.php?id=12071
It appears the mysql people ran into this and concluded it is a Windows bug
they needed to work around.
jan
--
--------------------------------------------------------------
Jan de Visser jdevisser@digitalfairway.com
Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
@ 2006-03-10 14:32 Hakan Kocaman <Hakan.Kocaman@digame.de>
0 siblings, 0 replies; 15+ messages in thread
From: Hakan Kocaman @ 2006-03-10 14:32 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Jan de Visser <jdevisser@digitalfairway.com>; +Cc: pgsql-performance
Hi,
> -----Original Message-----
> From: pgsql-performance-owner@postgresql.org
> [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Tom Lane
> Sent: Thursday, March 09, 2006 9:11 PM
> To: Jan de Visser
> Cc: pgsql-performance@postgresql.org
> Subject: Re: [PERFORM] Hanging queries on dual CPU windows
>
>
> Jan de Visser <jdevisser@digitalfairway.com> writes:
> > Furtermore, it does not happen on Linux machines, both
> single CPU and dual
> > CPU, nor on single CPU windows machines. We can only
> reproduce on a dual CPU
> > windows machine, and if we take one CPU out, it does not happen.
> > ...
> > Which showed me that several transactions where waiting for
> a particular row
> > which was locked by another transaction. This transaction
> had no pending
> > locks (so no deadlock), but just does not complete and hence never
> > relinquishes the lock.
>
> Is the stuck transaction still consuming CPU time, or just stopped?
>
> Is it possible to get a stack trace from the stuck process? I dunno
> if you've got anything gdb-equivalent under Windows, but that's the
> first thing I'd be interested in ...
Debugging Tools for Windows from Microsoft
http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
Additinonally you need a symbol-file or you use
"SRV*c:\debug\symbols*http://msdl.microsoft.com/download/symbols";
to load the symbol-file dynamically from the net.
Best regards
>
> regards, tom lane
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
Hakan Kocaman
Software-Development
digame.de GmbH
Richard-Byrd-Str. 4-8
50829 Köln
Tel.: +49 (0) 221 59 68 88 31
Fax: +49 (0) 221 59 68 88 98
Email: hakan.kocaman@digame.de
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
@ 2006-03-10 15:11 Magnus Hagander <mha@sollentuna.net>
2006-03-10 16:13 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 1 reply; 15+ messages in thread
From: Magnus Hagander @ 2006-03-10 15:11 UTC (permalink / raw)
To: Jan de Visser <jdevisser@digitalfairway.com>; pgsql-performance
> > > > I dunno
> > > >
> > > > > if you've got anything gdb-equivalent under Windows,
> but that's
> > > > > the first thing I'd be interested in ...
> > > >
> > > > Here ya go:
> > > >
> > > > http://www.devisser-siderius.com/stack1.jpg
> > > > http://www.devisser-siderius.com/stack2.jpg
> > > > http://www.devisser-siderius.com/stack3.jpg
> > > >
> > > > There are three threads in the process. I guess thread 1
> > > > (stack1.jpg) is the most interesting.
> > > >
> > > > I also noted that cranking up concurrency in my app
> reproduces the
> > > > problem in about 4 minutes ;-)
> >
> > Just reproduced again.
> >
> > > Actually, stack2 looks very interesting. Does it "stay stuck" in
> > > pg_queue_signal? That's really not supposed to happen.
> >
> > Yes it does.
>
> An update on that: There is actually *two* processes in this
> state, both hanging in pg_queue_signal. I've looked at the
> source of that, and the obvious candidate for hanging is
> EnterCriticalSection. I also found this:
>
> http://blogs.msdn.com/larryosterman/archive/2005/03/02/383685.aspx
>
> where they say:
>
> "
> In addition, for Windows 2003, SP1, the EnterCriticalSection
> API has a subtle change that's intended tor resolve many of
> the lock convoy issues. Before
> Win2003 SP1, if 10 threads were blocked on
> EnterCriticalSection and all 10 threads had the same
> priority, then EnterCriticalSection would service those
> threads in a FIFO (first -in, first-out) basis. Starting in
> Windows 2003 SP1, the EnterCriticalSection will wake up a
> random thread from the waiting threads. If all the threads
> are doing the same thing (like a thread pool) this won't make
> much of a difference, but if the different threads are doing
> different work (like the critical section protecting a widely
> accessed object), this will go a long way towards removing
> lock convoy semantics.
> "
>
> Could it be they broke it when they did that????
In theory, yes, but it still seems a bit far fetched :-(
If you have the env to rebuild, can you try changing the order of the lines:
ResetEvent(pgwin32_signal_event);
LeaveCriticalSection(&pg_signal_crit_sec);
in backend/port/win32/signal.c
And if not, can you also try disabling the stats collector and see if that makes a difference. (Could be a workaround..)
//Magnus
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
2006-03-10 15:11 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
@ 2006-03-10 16:13 ` Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 0 replies; 15+ messages in thread
From: Jan de Visser @ 2006-03-10 16:13 UTC (permalink / raw)
To: pgsql-performance
On Friday 10 March 2006 10:11, Magnus Hagander wrote:
> > Could it be they broke it when they did that????
>
> In theory, yes, but it still seems a bit far fetched :-(
Well, I rolled back SP1 and am running my test again. Looking much better,
hasn't locked up in 45mins now, whereas before it would lock up within 5mins.
So I think they broke something.
jan
--
--------------------------------------------------------------
Jan de Visser jdevisser@digitalfairway.com
Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
@ 2006-03-10 18:25 Magnus Hagander <mha@sollentuna.net>
2006-03-10 19:27 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 1 reply; 15+ messages in thread
From: Magnus Hagander @ 2006-03-10 18:25 UTC (permalink / raw)
To: Jan de Visser <jdevisser@digitalfairway.com>; pgsql-performance
> > > Could it be they broke it when they did that????
> >
> > In theory, yes, but it still seems a bit far fetched :-(
>
> Well, I rolled back SP1 and am running my test again. Looking
> much better, hasn't locked up in 45mins now, whereas before
> it would lock up within 5mins.
>
> So I think they broke something.
Wow. I guess I was lucky that I didn't say it was impossible :-)
But what really is happening. What other thread is actually holding the
critical section at this point, causing us to block? The only places it
gets held is while looping the signal queue, but it is released while
calling the signal function itself...
But they obviously *have* been messing with critical sections, so maybe
they accidentally changed something else as well...
What bothers me is that nobody else has reported this. It could be that
this was exposed by the changes to the signal handling done for 8.1, and
the ppl with this level of concurrency are either still on 8.0 or just
not on SP1 for their windows boxes yet... Do you have any other software
installed on the machine? That might possibly interfere in some way?
But let's have it run for a bit longer to confirm this does help. If so,
we could perhaps recode that part using a Mutex instead of a critical
section - since it's not a performance critical path, the difference
shouldn't be large. If I code up a patch for that, can you re-apply SP1
and test it? Or is this a production system you can't really touch?
//Magnus
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
2006-03-10 18:25 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
@ 2006-03-10 19:27 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 19:37 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 1 reply; 15+ messages in thread
From: Jan de Visser @ 2006-03-10 19:27 UTC (permalink / raw)
To: pgsql-performance
On Friday 10 March 2006 13:25, Magnus Hagander wrote:
> > > > Could it be they broke it when they did that????
> > >
> > > In theory, yes, but it still seems a bit far fetched :-(
> >
> > Well, I rolled back SP1 and am running my test again. Looking
> > much better, hasn't locked up in 45mins now, whereas before
> > it would lock up within 5mins.
> >
> > So I think they broke something.
>
> Wow. I guess I was lucky that I didn't say it was impossible :-)
>
>
> But what really is happening. What other thread is actually holding the
> critical section at this point, causing us to block? The only places it
> gets held is while looping the signal queue, but it is released while
> calling the signal function itself...
>
> But they obviously *have* been messing with critical sections, so maybe
> they accidentally changed something else as well...
>
> What bothers me is that nobody else has reported this. It could be that
> this was exposed by the changes to the signal handling done for 8.1, and
> the ppl with this level of concurrency are either still on 8.0 or just
> not on SP1 for their windows boxes yet... Do you have any other software
> installed on the machine? That might possibly interfere in some way?
Just a JDK, JBoss, cygwin (running sshd), and a VNC Server. I don't think that
interferes.
>
> But let's have it run for a bit longer to confirm this does help.
I turned it off after 2.5hr. The longest I had to wait before, with less load,
was 1.45hr.
> If so,
> we could perhaps recode that part using a Mutex instead of a critical
> section - since it's not a performance critical path, the difference
> shouldn't be large. If I code up a patch for that, can you re-apply SP1
> and test it? Or is this a production system you can't really touch?
I can do whatever the hell I want with it, so if you could cook up a patch
that would be great.
As a BTW: I reinstalled SP1 and turned stats collection off. That also seems
to work, but is not really a solution since we want to use autovacuuming.
>
> //Magnus
jan
--
--------------------------------------------------------------
Jan de Visser jdevisser@digitalfairway.com
Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Hanging queries on dual CPU windows
2006-03-10 18:25 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
2006-03-10 19:27 ` Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
@ 2006-03-10 19:37 ` Jan de Visser <jdevisser@digitalfairway.com>
0 siblings, 0 replies; 15+ messages in thread
From: Jan de Visser @ 2006-03-10 19:37 UTC (permalink / raw)
To: pgsql-performance
On Friday 10 March 2006 14:27, Jan de Visser wrote:
> As a BTW: I reinstalled SP1 and turned stats collection off. That also
> seems to work, but is not really a solution since we want to use
> autovacuuming.
I lied. I hangs now. Just takes a lot longer...
jan
--
--------------------------------------------------------------
Jan de Visser jdevisser@digitalfairway.com
Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------
^ permalink raw reply [nested|flat] 15+ messages in thread
end of thread, other threads:[~2006-03-10 19:37 UTC | newest]
Thread overview: 15+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-03-09 20:07 Re: Hanging queries on dual CPU windows Jan de Visser <jdevisser@digitalfairway.com>
2006-03-09 20:10 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-03-09 21:15 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 02:00 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-09 21:22 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
2006-03-10 09:20 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
2006-03-10 14:03 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 14:32 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 14:47 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 14:32 Re: Hanging queries on dual CPU windows Hakan Kocaman <Hakan.Kocaman@digame.de>
2006-03-10 15:11 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
2006-03-10 16:13 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 18:25 Re: Hanging queries on dual CPU windows Magnus Hagander <mha@sollentuna.net>
2006-03-10 19:27 ` Jan de Visser <jdevisser@digitalfairway.com>
2006-03-10 19:37 ` Jan de Visser <jdevisser@digitalfairway.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox