pgjdbc/pgjdbc GitHub issues and pull requests (mirror)
help / color / mirror / Atom feed[pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
20+ messages / 7 participants
[nested] [flat]
* [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-03-30 12:56 "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
0 siblings, 0 replies; 20+ messages in thread
From: PeterCsalaHbo (@PeterCsalaHbo) @ 2022-03-30 12:56 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Is this library expose some metrics like
- execution time in milliseconds
- number of records out
- number of bytes out
- number of success requests
- etc.
?
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-03-30 13:02 ` "davecramer (@davecramer)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: davecramer (@davecramer) @ 2022-03-30 13:02 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Currently there is not
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-03-30 13:17 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: PeterCsalaHbo (@PeterCsalaHbo) @ 2022-03-30 13:17 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
@davecramer Is there any plan for the near future?
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-03-30 13:20 ` "davecramer (@davecramer)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: davecramer (@davecramer) @ 2022-03-30 13:20 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
I have been contemplating it, but time is a challenge.
@PeterCsalaHbo can you provide a wishlist here ?
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-03-30 13:55 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: PeterCsalaHbo (@PeterCsalaHbo) @ 2022-03-30 13:55 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Basically the above four would be enough I think.
We can monitor the incoming traffic from the PostgreSQL cluster perspective but we can't monitor it from the application server perspective.
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-03-30 14:14 ` "davecramer (@davecramer)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: davecramer (@davecramer) @ 2022-03-30 14:14 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
So the challenge is that while you can get some numbers from the client, you need to know the server numbers to figure out the round trip overhead and the client overhead
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-03-31 09:01 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: PeterCsalaHbo (@PeterCsalaHbo) @ 2022-03-31 09:01 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Yes exactly, if we know both the postgres client's and the server's numbers then we can better locate where the problem could reside.
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-03-31 09:07 ` "davecramer (@davecramer)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: davecramer (@davecramer) @ 2022-03-31 09:07 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
FYI Postgres is never referred to as Postgre .
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-03-31 09:49 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: PeterCsalaHbo (@PeterCsalaHbo) @ 2022-03-31 09:49 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Apologize
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-06-15 06:55 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: PeterCsalaHbo (@PeterCsalaHbo) @ 2022-06-15 06:55 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
I think I killed this feature request with a typo ...
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-06-15 07:15 ` "vlsi (@vlsi)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: vlsi (@vlsi) @ 2022-06-15 07:15 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Let me reopen this, as I think the feature will be useful.
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-06-15 07:21 ` "vlsi (@vlsi)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: vlsi (@vlsi) @ 2022-06-15 07:21 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
There are driver-managed structures (e.g. number of server-prepared statements, number of SQL statement cache misses) that might be interesting to monitor.
The tricky question is what do we use for collecting and reporting the metrics.
Relevant libraries are:
* https://micrometer.io/
* https://metrics.dropwizard.io/
* http://hdrhistogram.org/
* https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/atomic/LongAdder.html
I'm not fond of reimplementing our own metric facade, and I'm not fond of breaking tons of applications with cases like "micrometer version from pgjdbc 42.x is not compatible with the one that comes with Spring Boot 3.y".
Any thoughts on the way we could collect metrics, expose them and avoid classpath clash?
Should we shade a metric collection library and expose the values via JMX?
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-06-15 13:11 ` "davecramer (@davecramer)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: davecramer (@davecramer) @ 2022-06-15 13:11 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
If we wanted to remain agnostic I would expose listeners. This however would be PostgreSQL specific as the JDBC API has no such facility
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-06-15 14:59 ` "sehrope (@sehrope)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: sehrope (@sehrope) @ 2022-06-15 14:59 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
+1 to exposing our own interfaces and let the end user bridge that to their own collector. That would give us the ability to make it match whatever level of low level detail we'd like to provide.
Perhaps we could even mark the interface itself as "experimental" to have the first couple releases maintain the flexibility / luxury of changing the interface if real world usage changes. Breaking things after they're in the field is never a good thing, but for something entirely new like this keeping that option might be worthwhile.
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-06-15 15:03 ` "svendiedrichsen (@svendiedrichsen)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: svendiedrichsen (@svendiedrichsen) @ 2022-06-15 15:03 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
IMHO it would be great if the listeners would be notified asynchronously.
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-06-15 15:06 ` "davecramer (@davecramer)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: davecramer (@davecramer) @ 2022-06-15 15:06 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
> IMHO it would be great if the listeners would be notified asynchronously.
That would mean we would have to be opinionated on what we sent it. The listener can choose to acquire whatever data it wants and then send it on asynchronously to avoid blocking if this is a requirement.
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-06-18 11:36 ` "const (@const)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: const (@const) @ 2022-06-18 11:36 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Please change time units to microseconds. java.util.time.Instant provides that time.
Also, it makes sense to organize measurement as listeners registered with driver or connection, that target output to needed data target: JFR special file or standard api, logging, or some custom output.
I think the following events are needed (time and thread when happened for each event, only additional parameters specified, id should be used to correlate events):
* Connection
* open
* close (duration)
* First statement in transaction
* transaction commit success or fail (duration of transaction)
* transaction explicit rollback (duration of transaction)
* autocommit changed
* XA
* Connection joined transaction
* Connection left transaction
* Update preparead statement
* prepare (statement, number of aguments)
* Batch execution (batch size, batch number of elements)
* Update started (size and number of input parameters, statement)
* Update finished (size and number of input parameters, number of records, duration)
* Call statement
* parepare (statement, number of arguments)
* call started (size and number of input parameters, statement)
* call finished (size and number of input/output parameters, duration)
* Query statement
* create (statement, number of agruments)
* query started (size and number of input parameters, statement)
* record received (offset in result, size, number of elements, time of reading record)
* query finished (size and number of input/output parameters, total result set amount of records, total result size, duration)
For non-prepared statement, prepare and start event need to be joined. Size of result set record, call result, and arguments calculated as it is serialized to output + overheads of driver format. So int8 + string of 20 ascii chars will be calculated as 8 (size of long) + 4 (length of text) + 20 (UTF8 byte size).
And it would nice to have a standard custom JFR output stream that records each statement with unique text only once. So log will be small and could be picked from servers later.
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2022-06-18 12:56 ` "davecramer (@davecramer)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: davecramer (@davecramer) @ 2022-06-18 12:56 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Is there a reason you wouldn't use https://github.com/p6spy/p6spy for this ?
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2024-02-14 11:38 ` "koriit-kontakt (@koriit-kontakt)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: koriit-kontakt (@koriit-kontakt) @ 2024-02-14 11:38 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
Hi, if I could add to the wishlist, I would like to monitor the prepared statement cache - current size and possibly how many prepared statements are being created and/or evicted. The goal is to detect cache trashing, which was the root cause of severe performance issues in one of our databases.
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side
@ 2024-12-12 18:34 ` "svendiedrichsen (@svendiedrichsen)" <[email protected]>
18 siblings, 0 replies; 20+ messages in thread
From: svendiedrichsen (@svendiedrichsen) @ 2024-12-12 18:34 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
In case of using JFR events for observability one could use [JFR Event Streaming](https://openjdk.org/jeps/349) to receive the events and stream them into whatever system to observe. No need for specific listeners codewise from them JDBC driver.
^ permalink raw reply [nested|flat] 20+ messages in thread
end of thread, other threads:[~2024-12-12 18:34 UTC | newest]
Thread overview: 20+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-03-30 12:56 [pgjdbc/pgjdbc] issue #2478: Collect and report metrics from the driver side "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
2022-03-30 13:02 ` "davecramer (@davecramer)" <[email protected]>
2022-03-30 13:17 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
2022-03-30 13:20 ` "davecramer (@davecramer)" <[email protected]>
2022-03-30 13:55 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
2022-03-30 14:14 ` "davecramer (@davecramer)" <[email protected]>
2022-03-31 09:01 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
2022-03-31 09:07 ` "davecramer (@davecramer)" <[email protected]>
2022-03-31 09:49 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
2022-06-15 06:55 ` "PeterCsalaHbo (@PeterCsalaHbo)" <[email protected]>
2022-06-15 07:15 ` "vlsi (@vlsi)" <[email protected]>
2022-06-15 07:21 ` "vlsi (@vlsi)" <[email protected]>
2022-06-15 13:11 ` "davecramer (@davecramer)" <[email protected]>
2022-06-15 14:59 ` "sehrope (@sehrope)" <[email protected]>
2022-06-15 15:03 ` "svendiedrichsen (@svendiedrichsen)" <[email protected]>
2022-06-15 15:06 ` "davecramer (@davecramer)" <[email protected]>
2022-06-18 11:36 ` "const (@const)" <[email protected]>
2022-06-18 12:56 ` "davecramer (@davecramer)" <[email protected]>
2024-02-14 11:38 ` "koriit-kontakt (@koriit-kontakt)" <[email protected]>
2024-12-12 18:34 ` "svendiedrichsen (@svendiedrichsen)" <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox