public inbox for [email protected]  

help / color / mirror / Atom feed

postgres table statistics
3+ messages / 3 participants
[nested] [flat]

* postgres table statistics
@ 2024-06-12 07:47 Chandy G <[email protected]>
  2024-06-12 08:27 ` Re: postgres table statistics Ron Johnson <[email protected]>
  2024-06-12 14:44 ` Re: postgres table statistics Shammat <[email protected]>
  0 siblings, 2 replies; 3+ messages in thread

From: Chandy G @ 2024-06-12 07:47 UTC (permalink / raw)
  To: pgsql-general

Hi, 
  We have postgres 13.9 running with tables thats got billions of records of varying sizes. Eventhough pg jdbc driver  provides a way to set fetch size to tune the driver to achieve better throughput, the JVM fails at the driver level when records of large size (say 200mb each) flows through.  this forces to reduce the fetch size (if were to operate at a fixed Xmx setting of client jvm).
It get a bit trickier when 100s of such tables exists with varying records sizes. trying to see if the fetch size can be set dynamically based on the row count and the record size distribution for a table. Unfortunately, trying to get this data by a query run against each table (for row size: max(length(t::text))) seem to be  quite time consuming too.
Does postgres maintain metadata about tables for the following.1. row count 
2. max row size. 

or is there some other pg metadata that can help get this data quicker.
TIA.





^ permalink  raw  reply  [nested|flat] 3+ messages in thread

* Re: postgres table statistics
  2024-06-12 07:47 postgres table statistics Chandy G <[email protected]>
@ 2024-06-12 08:27 ` Ron Johnson <[email protected]>
  1 sibling, 0 replies; 3+ messages in thread

From: Ron Johnson @ 2024-06-12 08:27 UTC (permalink / raw)
  To: pgsql-general

On Wed, Jun 12, 2024 at 3:48 AM Chandy G <[email protected]> wrote:

> Hi,
>   We have postgres 13.9 running with tables thats got billions of records
> of varying sizes. Eventhough pg jdbc driver  provides a way to set fetch
> size to tune the driver to achieve better throughput, the JVM fails at the
> driver level when records of large size (say 200mb each) flows through.
> this forces to reduce the fetch size (if were to operate at a fixed Xmx
> setting of client jvm).
>
> It get a bit trickier when 100s of such tables exists with varying records
> sizes. trying to see if the fetch size can be set dynamically based on the
> row count and the record size distribution for a table. Unfortunately,
> trying to get this data by a query run against each table (for row size:
> max(length(t::text))) seem to be  quite time consuming too.
>

Maybe create your own table with three columns:
table_name (PK; taken from pg_class.relname)
average_rec_size (taken from sum(pg_stat.avg_width))
max_rec_size (calculated yourself)

Periodically refresh it.  (How periodic depends on how often the average
and max change substantively.)

Does postgres maintain metadata about tables for the following.
> 1. row count
>

https://www.postgresql.org/docs/13/catalog-pg-class.html

pg_class.reltuples.  This is an estimate, so make sure your tables are
regularly analyzed.


> 2. max row size.
>

https://www.postgresql.org/docs/13/view-pg-stats.html

pg_stats.avg_width


> or is there some other pg metadata that can help get this data quicker.
>
> TIA.
>


^ permalink  raw  reply  [nested|flat] 3+ messages in thread

* Re: postgres table statistics
  2024-06-12 07:47 postgres table statistics Chandy G <[email protected]>
@ 2024-06-12 14:44 ` Shammat <[email protected]>
  1 sibling, 0 replies; 3+ messages in thread

From: Shammat @ 2024-06-12 14:44 UTC (permalink / raw)
  To: [email protected]


Chandy G schrieb am 12.06.2024 um 09:47:
> Eventhough pg jdbc driver  provides a way to set fetch size to tune
> the driver to achieve better throughput, the JVM fails at the driver
> level when records of large size (say 200mb each) flows through.
> this forces to reduce the fetch size (if were to operate at a fixed
> Xmx setting of client jvm).


Did you try the driver's "adaptive fetch" feature?


https://jdbc.postgresql.org/documentation/use/#connection-parameters







^ permalink  raw  reply  [nested|flat] 3+ messages in thread

end of thread, other threads:[~2024-06-12 14:44 UTC | newest]

Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-06-12 07:47 postgres table statistics Chandy G <[email protected]>
2024-06-12 08:27 ` Ron Johnson <[email protected]>
2024-06-12 14:44 ` Shammat <[email protected]>

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox