agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedplease help
20+ messages / 9 participants
[nested] [flat]
* please help
@ 2000-01-07 08:17 Shanthala Rao <S.Rao@toa.toasia.co.jp>
0 siblings, 0 replies; 20+ messages in thread
From: Shanthala Rao @ 2000-01-07 08:17 UTC (permalink / raw)
To: pgsql-hackers
I had written a pgm in Java on a Windows m/c
which accesses data from a linux m/c.The database
is stored in the linux m/c.I added postgresql.jar file(from Linux m/c)
for my proj.But when I run the pgm,the following Exception
is displayed.
The postgresql.jar file does not contain the correct
JDBC classes for this JVM.
Try rebuilding.
Exception thrown
java.lang.ClassNotFound Exception.
I want to know is there any JDBC driver for Windows
in postgresql??
If so,would u please let me know where is it available??
Thanking you,
Shanthala.
^ permalink raw reply [nested|flat] 20+ messages in thread
* Please help
@ 2002-10-21 13:42 Olivier PRENANT <ohp@pyrenet.fr>
0 siblings, 2 replies; 20+ messages in thread
From: Olivier PRENANT @ 2002-10-21 13:42 UTC (permalink / raw)
To: pgsql-hackers
Hi all,
Without modifying anything, postgresql (since today) has a strange
behavior:
All connections are rejected with No space left on device.
There's plenty of space in shm, disk...
It occurs on UNIX as well as on INET socket.
I can't even shut the postmaster down (it doesn't respond) and have a few
defunct process.
SHMAX is at 40000000 here, version is 7.2.3
Heres postgresql.conf:
#
# PostgreSQL configuration file
# -----------------------------
#
# This file consists of lines of the form
#
# name = value
#
# (The `=' is optional.) White space is collapsed, comments are
# introduced by `#' anywhere on a line. The complete list of option
# names and allowed values can be found in the PostgreSQL
# documentation. The commented-out settings shown in this file
# represent the default values.
# Any option can also be given as a command line switch to the
# postmaster, e.g., 'postmaster -c log_connections=on'. Some options
# can be changed at run-time with the 'SET' SQL command.
#========================================================================
#
# Connection Parameters
#
tcpip_socket = true
#ssl = false
max_connections = 64
port = 5432
hostname_lookup = true
#show_source_port = false
#unix_socket_directory = ''
#unix_socket_group = ''
#unix_socket_permissions = 0777
#virtual_host = ''
#krb_server_keyfile = ''
#
# Shared Memory Size
#
shared_buffers = 512 # 2*max_connections, min 16
#max_fsm_relations = 100 # min 10, fsm is free space map
#max_fsm_pages = 10000 # min 1000, fsm is free space map
#max_locks_per_transaction = 64 # min 10
#wal_buffers = 8 # min 4
#
# Non-shared Memory Sizes
#
sort_mem = 10240 # min 32
#vacuum_mem = 8192 # min 1024
#
# Write-ahead log (WAL)
#
#wal_files = 4 # range 0-64
#wal_sync_method = fsync # the default varies across platforms:
# # fsync, fdatasync, open_sync, or open_datasync
#wal_debug = 0 # range 0-16
#commit_delay = 0 # range 0-100000
#commit_siblings = 5 # range 1-1000
#checkpoint_segments = 3 # in logfile segments (16MB each), min 1
#checkpoint_timeout = 300 # in seconds, range 30-3600
#fsync = true
#
# Optimizer Parameters
#
#enable_seqscan = true
#enable_indexscan = true
#enable_tidscan = true
#enable_sort = true
#enable_nestloop = true
#enable_mergejoin = true
#enable_hashjoin = true
#ksqo = false
#effective_cache_size = 1000 # default in 8k pages
#random_page_cost = 4
#cpu_tuple_cost = 0.01
#cpu_index_tuple_cost = 0.001
#cpu_operator_cost = 0.0025
#
# GEQO Optimizer Parameters
#
#geqo = true
#geqo_selection_bias = 2.0 # range 1.5-2.0
#geqo_threshold = 11
#geqo_pool_size = 0 # default based on #tables in query, range 128-1024
#geqo_effort = 1
#geqo_generations = 0
#geqo_random_seed = -1 # auto-compute seed
#
# Debug display
#
#silent_mode = false
log_connections = true
#log_timestamp = true
log_pid = true
#debug_level = 0 # range 0-16
debug_print_query = true
#debug_print_parse = false
#debug_print_rewritten = false
#debug_print_plan = false
debug_pretty_print = true
# requires USE_ASSERT_CHECKING
#debug_assertions = true
#
# Syslog
#
# requires ENABLE_SYSLOG
syslog = 2 # range 0-2
#syslog_facility = 'LOCAL0'
#syslog_ident = 'postgres'
#
# Statistics
#
show_parser_stats = false
show_planner_stats = false
show_executor_stats = false
show_query_stats = false
# requires BTREE_BUILD_STATS
#show_btree_build_stats = false
#
# Access statistics collection
#
stats_start_collector = true
stats_reset_on_server_start = true
stats_command_string = true
stats_row_level = true
stats_block_level = true
#
# Lock Tracing
#
#trace_notify = false
# requires LOCK_DEBUG
#trace_locks = false
#trace_userlocks = false
#trace_lwlocks = false
#debug_deadlocks = false
#trace_lock_oidmin = 16384
#trace_lock_table = 0
#
# Misc
#
#dynamic_library_path = '$libdir'
#australian_timezones = false
#authentication_timeout = 60 # min 1, max 600
#deadlock_timeout = 1000
#default_transaction_isolation = 'read committed'
#max_expr_depth = 10000 # min 10
#max_files_per_process = 1000 # min 25
#password_encryption = false
#sql_inheritance = true
#transform_null_equals = false
Anyone can come with an idea?
It NEVER happend before...
--
Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
31190 AUTERIVE +33-6-07-63-80-64 (GSM)
FRANCE Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 15:02 Tom Lane <tgl@sss.pgh.pa.us>
parent: Olivier PRENANT <ohp@pyrenet.fr>
1 sibling, 1 reply; 20+ messages in thread
From: Tom Lane @ 2002-10-21 15:02 UTC (permalink / raw)
To: ohp@pyrenet.fr; +Cc: pgsql-hackers
Olivier PRENANT <ohp@pyrenet.fr> writes:
> Without modifying anything, postgresql (since today) has a strange
> behavior:
> All connections are rejected with No space left on device.
Could you be out of swap space?
I'd like to see the *exact* context in which you see this error message,
though.
regards, tom lane
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 15:16 Ian Barwick <barwick@gmx.net>
parent: Olivier PRENANT <ohp@pyrenet.fr>
1 sibling, 0 replies; 20+ messages in thread
From: Ian Barwick @ 2002-10-21 15:16 UTC (permalink / raw)
To: ohp@pyrenet.fr; pgsql-hackers
On Monday 21 October 2002 15:42, Olivier PRENANT wrote:
> Hi all,
>
> Without modifying anything, postgresql (since today) has a strange
> behavior:
>
> All connections are rejected with No space left on device.
>
> There's plenty of space in shm, disk...
I have no idea whether it's relevant, but maybe you have a problem with
semaphores? See:
http://www.ca.postgresql.org/docs/faq-english.html#3.4
(A lack of available semaphores can also produce the message
"No space left on device.")
Sorry I can't help any further.
Ian Barwick
barwick@gmx.net
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 15:22 Olivier PRENANT <ohp@pyrenet.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 20+ messages in thread
From: Olivier PRENANT @ 2002-10-21 15:22 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
Thanks for your reply.
Actually, I just found xhat happened 2 mn ago!
Last week I changed my pg_hba.conf to require an md5 password for one
specific database and updated all my script but ONE.
It seems that connection from php pg_connect not supplying a password
lives the process for a "certain ammount of time" running, then postmaster
just hangs.
Don't really know what happens here, but supplying a password on this
script made the problem go away... (at least for now)
Regards
On Mon, 21 Oct 2002, Tom Lane wrote:
> Date: Mon, 21 Oct 2002 11:02:43 -0400
> From: Tom Lane <tgl@sss.pgh.pa.us>
> To: ohp@pyrenet.fr
> Cc: pgsql-hackers list <pgsql-hackers@postgresql.org>
> Subject: Re: [HACKERS] Please help
>
> Olivier PRENANT <ohp@pyrenet.fr> writes:
> > Without modifying anything, postgresql (since today) has a strange
> > behavior:
>
> > All connections are rejected with No space left on device.
>
> Could you be out of swap space?
>
> I'd like to see the *exact* context in which you see this error message,
> though.
>
> regards, tom lane
>
--
Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
31190 AUTERIVE +33-6-07-63-80-64 (GSM)
FRANCE Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 15:35 Tom Lane <tgl@sss.pgh.pa.us>
parent: Olivier PRENANT <ohp@pyrenet.fr>
0 siblings, 1 reply; 20+ messages in thread
From: Tom Lane @ 2002-10-21 15:35 UTC (permalink / raw)
To: ohp@pyrenet.fr; +Cc: pgsql-hackers
Olivier PRENANT <ohp@pyrenet.fr> writes:
> It seems that connection from php pg_connect not supplying a password
> lives the process for a "certain ammount of time" running, then postmaster
> just hangs.
That's hard to believe. In 7.2 or later, the backend should give up and
close the connection and exit if the client doesn't finish the
authentication handshake within 60 seconds.
Can anyone else reproduce a problem with lack of a password on a PHP
connection?
regards, tom lane
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 16:23 Olivier PRENANT <ohp@pyrenet.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 20+ messages in thread
From: Olivier PRENANT @ 2002-10-21 16:23 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
Hi Tom,
I can assure you that there are no problem anymore now that I have put the
password.
However the modification pg_hba.conf has been done a few days ago and
never noticed the error until today where I had a *LOT* of visits to my
site.
So I still think it's a matter of bad connection delay.
Connexion where comming very fast and postmaster did'nt release them fast
enough.
I now have this problem when I run vacuum analyze:
Script started on Mon Oct 21 18:20:35 2002
~ 18:20:35: psql pyrenet
Password:
Welcome to psql, the PostgreSQL interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help on internal slash commands
\g or terminate with semicolon to execute query
\q to quit
pyrenet=# VACUUM ANALYZE ;
FATAL 2: could not open transaction-commit log directory (/usr/local/pgsql/data/pg_clog): Too many open files
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: NOTICE: Message from PostgreSQL backend:
The Postmaster has informed me that some other backend
died abnormally and possibly corrupted shared memory.
I have rolled back the current transaction and am
going to terminate your database system connection and exit.
Please reconnect to the database system and repeat your query.
Failed.
!# \q
~ 18:21:21: psql pyrenet
Password:
Welcome to psql, the PostgreSQL interactive terminal.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help on internal slash commands
\g or terminate with semicolon to execute query
\q to quit
pyrenet=# VACUUM ANALYZE ;
FATAL 2: could not open transaction-commit log directory (/usr/local/pgsql/data/pg_clog): Too many open files
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!# \q
script done on Mon Oct 21 18:21:51 2002
What causes (FILES=64).
Again, this is the very first time I have such problems on postgresql!!
It works so well, it's the central point of my system!!
On Mon, 21 Oct 2002, Tom Lane wrote:
> Date: Mon, 21 Oct 2002 11:35:33 -0400
> From: Tom Lane <tgl@sss.pgh.pa.us>
> To: ohp@pyrenet.fr
> Cc: pgsql-hackers list <pgsql-hackers@postgresql.org>
> Subject: Re: [HACKERS] Please help
>
> Olivier PRENANT <ohp@pyrenet.fr> writes:
> > It seems that connection from php pg_connect not supplying a password
> > lives the process for a "certain ammount of time" running, then postmaster
> > just hangs.
>
> That's hard to believe. In 7.2 or later, the backend should give up and
> close the connection and exit if the client doesn't finish the
> authentication handshake within 60 seconds.
>
> Can anyone else reproduce a problem with lack of a password on a PHP
> connection?
>
> regards, tom lane
>
--
Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
31190 AUTERIVE +33-6-07-63-80-64 (GSM)
FRANCE Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 16:52 Tom Lane <tgl@sss.pgh.pa.us>
parent: Olivier PRENANT <ohp@pyrenet.fr>
0 siblings, 1 reply; 20+ messages in thread
From: Tom Lane @ 2002-10-21 16:52 UTC (permalink / raw)
To: ohp@pyrenet.fr; +Cc: pgsql-hackers
Olivier PRENANT <ohp@pyrenet.fr> writes:
> pyrenet=# VACUUM ANALYZE ;
> FATAL 2: could not open transaction-commit log directory (/usr/local/pgsql/data/pg_clog): Too many open files
Hmm. Do you have any idea what sysconf(_SC_OPEN_MAX) returns on your
platform? You could try reducing the max_files_per_process parameter.
regards, tom lane
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 17:26 Olivier PRENANT <ohp@pyrenet.fr>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 20+ messages in thread
From: Olivier PRENANT @ 2002-10-21 17:26 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
On Mon, 21 Oct 2002, Tom Lane wrote:
> Date: Mon, 21 Oct 2002 12:52:10 -0400
> From: Tom Lane <tgl@sss.pgh.pa.us>
> To: ohp@pyrenet.fr
> Cc: pgsql-hackers list <pgsql-hackers@postgresql.org>
> Subject: Re: [HACKERS] Please help
>
> Olivier PRENANT <ohp@pyrenet.fr> writes:
> > pyrenet=# VACUUM ANALYZE ;
> > FATAL 2: could not open transaction-commit log directory (/usr/local/pgsql/data/pg_clog): Too many open files
>
> Hmm. Do you have any idea what sysconf(_SC_OPEN_MAX) returns on your
> platform? You could try reducing the max_files_per_process parameter.
The point is, it occurs today for the very first time!
Question: does (with 7.2) augmenting max_connection suffice, or do I have
to recompile?
That's the only thing that comes to my mind! I changed max_coneections
(and related parameters) in postgresql.conf only...
I say that, because I tried to change socket_directory in postgresql.conf
and clients didn't work anymore
>
> regards, tom lane
>
--
Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
31190 AUTERIVE +33-6-07-63-80-64 (GSM)
FRANCE Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 17:34 Larry Rosenman <ler@lerctr.org>
parent: Olivier PRENANT <ohp@pyrenet.fr>
0 siblings, 1 reply; 20+ messages in thread
From: Larry Rosenman @ 2002-10-21 17:34 UTC (permalink / raw)
To: ohp@pyrenet.fr; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Mon, 2002-10-21 at 12:26, Olivier PRENANT wrote:
> On Mon, 21 Oct 2002, Tom Lane wrote:
>
> > Date: Mon, 21 Oct 2002 12:52:10 -0400
> > From: Tom Lane <tgl@sss.pgh.pa.us>
> > To: ohp@pyrenet.fr
> > Cc: pgsql-hackers list <pgsql-hackers@postgresql.org>
> > Subject: Re: [HACKERS] Please help
> >
> > Olivier PRENANT <ohp@pyrenet.fr> writes:
> > > pyrenet=# VACUUM ANALYZE ;
> > > FATAL 2: could not open transaction-commit log directory (/usr/local/pgsql/data/pg_clog): Too many open files
> >
> > Hmm. Do you have any idea what sysconf(_SC_OPEN_MAX) returns on your
> > platform? You could try reducing the max_files_per_process parameter.
> The point is, it occurs today for the very first time!
> Question: does (with 7.2) augmenting max_connection suffice, or do I have
> to recompile?
You might need to up the Shared Memory parameters and the Semaphore
Parameters in your OS (UnixWare IIRC).
>
> That's the only thing that comes to my mind! I changed max_coneections
> (and related parameters) in postgresql.conf only...
>
> I say that, because I tried to change socket_directory in postgresql.conf
> and clients didn't work anymore
See above.
> >
> > regards, tom lane
> >
>
> --
> Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE +33-6-07-63-80-64 (GSM)
> FRANCE Email: ohp@pyrenet.fr
> ------------------------------------------------------------------------------
> Make your life a dream, make your dream a reality. (St Exupery)
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 17:39 Olivier PRENANT <ohp@pyrenet.fr>
parent: Larry Rosenman <ler@lerctr.org>
0 siblings, 1 reply; 20+ messages in thread
From: Olivier PRENANT @ 2002-10-21 17:39 UTC (permalink / raw)
To: Larry Rosenman <ler@lerctr.org>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi larry,
Glad to see you around...
On 21 Oct 2002, Larry Rosenman wrote:
> Date: 21 Oct 2002 12:34:48 -0500
> From: Larry Rosenman <ler@lerctr.org>
> To: ohp@pyrenet.fr
> Cc: Tom Lane <tgl@sss.pgh.pa.us>,
> pgsql-hackers list <pgsql-hackers@postgresql.org>
> Subject: Re: [HACKERS] Please help
> > The point is, it occurs today for the very first time!
> > Question: does (with 7.2) augmenting max_connection suffice, or do I have
> > to recompile?
> You might need to up the Shared Memory parameters and the Semaphore
> Parameters in your OS (UnixWare IIRC).
I did!
> >
> > That's the only thing that comes to my mind! I changed max_coneections
> > (and related parameters) in postgresql.conf only...
> >
> > I say that, because I tried to change socket_directory in postgresql.conf
> > and clients didn't work anymore
Sorry, I mis-explain!
I mean changing socket_directory in postgresql.conf and restart server did
create .s.PGSQL.5432 in the new dir, however clients (like psql) still
want it in /tmp!!
> See above.
>
>
> > >
> > > regards, tom lane
> > >
> >
> > --
> > Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
> > Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
> > 31190 AUTERIVE +33-6-07-63-80-64 (GSM)
> > FRANCE Email: ohp@pyrenet.fr
> > ------------------------------------------------------------------------------
> > Make your life a dream, make your dream a reality. (St Exupery)
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 4: Don't 'kill -9' the postmaster
> >
>
--
Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
31190 AUTERIVE +33-6-07-63-80-64 (GSM)
FRANCE Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 17:57 Larry Rosenman <ler@lerctr.org>
parent: Olivier PRENANT <ohp@pyrenet.fr>
0 siblings, 1 reply; 20+ messages in thread
From: Larry Rosenman @ 2002-10-21 17:57 UTC (permalink / raw)
To: ohp@pyrenet.fr; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Mon, 2002-10-21 at 12:39, Olivier PRENANT wrote:
> Hi larry,
>
> Glad to see you around...
> On 21 Oct 2002, Larry Rosenman wrote:
>
> > Date: 21 Oct 2002 12:34:48 -0500
> > From: Larry Rosenman <ler@lerctr.org>
> > To: ohp@pyrenet.fr
> > Cc: Tom Lane <tgl@sss.pgh.pa.us>,
> > pgsql-hackers list <pgsql-hackers@postgresql.org>
> > Subject: Re: [HACKERS] Please help
> > > The point is, it occurs today for the very first time!
> > > Question: does (with 7.2) augmenting max_connection suffice, or do I have
> > > to recompile?
> > You might need to up the Shared Memory parameters and the Semaphore
> > Parameters in your OS (UnixWare IIRC).
> I did!
Ok.
> > >
> > > That's the only thing that comes to my mind! I changed max_coneections
> > > (and related parameters) in postgresql.conf only...
> > >
> > > I say that, because I tried to change socket_directory in postgresql.conf
> > > and clients didn't work anymore
> Sorry, I mis-explain!
> I mean changing socket_directory in postgresql.conf and restart server did
> create .s.PGSQL.5432 in the new dir, however clients (like psql) still
> want it in /tmp!!
That **WOULD** take a recompile.
LER
>
> > See above.
> >
> >
> > > >
> > > > regards, tom lane
> > > >
> > >
> > > --
> > > Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
> > > Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
> > > 31190 AUTERIVE +33-6-07-63-80-64 (GSM)
> > > FRANCE Email: ohp@pyrenet.fr
> > > ------------------------------------------------------------------------------
> > > Make your life a dream, make your dream a reality. (St Exupery)
> > >
> > >
> > > ---------------------------(end of broadcast)---------------------------
> > > TIP 4: Don't 'kill -9' the postmaster
> > >
> >
>
> --
> Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
> Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
> 31190 AUTERIVE +33-6-07-63-80-64 (GSM)
> FRANCE Email: ohp@pyrenet.fr
> ------------------------------------------------------------------------------
> Make your life a dream, make your dream a reality. (St Exupery)
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2002-10-21 17:59 Larry Rosenman <ler@lerctr.org>
parent: Larry Rosenman <ler@lerctr.org>
0 siblings, 0 replies; 20+ messages in thread
From: Larry Rosenman @ 2002-10-21 17:59 UTC (permalink / raw)
To: Larry Rosenman <ler@lerctr.org>; +Cc: ohp@pyrenet.fr; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
On Mon, 2002-10-21 at 12:57, Larry Rosenman wrote:
> On Mon, 2002-10-21 at 12:39, Olivier PRENANT wrote:
> > Hi larry,
> >
> > Glad to see you around...
> > On 21 Oct 2002, Larry Rosenman wrote:
> >
> > > Date: 21 Oct 2002 12:34:48 -0500
> > > From: Larry Rosenman <ler@lerctr.org>
> > > To: ohp@pyrenet.fr
> > > Cc: Tom Lane <tgl@sss.pgh.pa.us>,
> > > pgsql-hackers list <pgsql-hackers@postgresql.org>
> > > Subject: Re: [HACKERS] Please help
> > > > The point is, it occurs today for the very first time!
> > > > Question: does (with 7.2) augmenting max_connection suffice, or do I have
> > > > to recompile?
> > > You might need to up the Shared Memory parameters and the Semaphore
> > > Parameters in your OS (UnixWare IIRC).
> > I did!
> Ok.
> > > >
> > > > That's the only thing that comes to my mind! I changed max_coneections
> > > > (and related parameters) in postgresql.conf only...
> > > >
> > > > I say that, because I tried to change socket_directory in postgresql.conf
> > > > and clients didn't work anymore
> > Sorry, I mis-explain!
> > I mean changing socket_directory in postgresql.conf and restart server did
> > create .s.PGSQL.5432 in the new dir, however clients (like psql) still
> > want it in /tmp!!
> That **WOULD** take a recompile.
Or (IIRC), changing the connect string passed from PHP to PostgreSQL.
>
> LER
> >
> > > See above.
> > >
> > >
> > > > >
> > > > > regards, tom lane
> > > > >
> > > >
> > > > --
> > > > Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
> > > > Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
> > > > 31190 AUTERIVE +33-6-07-63-80-64 (GSM)
> > > > FRANCE Email: ohp@pyrenet.fr
> > > > ------------------------------------------------------------------------------
> > > > Make your life a dream, make your dream a reality. (St Exupery)
> > > >
> > > >
> > > > ---------------------------(end of broadcast)---------------------------
> > > > TIP 4: Don't 'kill -9' the postmaster
> > > >
> > >
> >
> > --
> > Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
> > Quartier d'Harraud Turrou +33-5-61-50-97-01 (Fax)
> > 31190 AUTERIVE +33-6-07-63-80-64 (GSM)
> > FRANCE Email: ohp@pyrenet.fr
> > ------------------------------------------------------------------------------
> > Make your life a dream, make your dream a reality. (St Exupery)
> --
> Larry Rosenman http://www.lerctr.org/~ler
> Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
> US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 972-414-9812 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
^ permalink raw reply [nested|flat] 20+ messages in thread
* Please help
@ 2003-10-30 16:33 ohp@pyrenet.fr
0 siblings, 1 reply; 20+ messages in thread
From: ohp@pyrenet.fr @ 2003-10-30 16:33 UTC (permalink / raw)
To: pgsql-hackers
I've have a MAJOR crash an hour ago and postgresql doesn't start anymoe
(version 7.3.4).
Here's the error log:
Oct 30 17:16:20 server postgres[4135]: [1-2] This probably means that some data blocks are corrupted
Oct 30 17:16:20 server postgres[4135]: [1-3] and you will have to use the last backup for recovery.
Oct 30 17:16:20 server postgres[4135]: [2] LOG: checkpoint record is at 36/C27C14C0
Oct 30 17:16:20 server postgres[4135]: [3] LOG: redo record is at 36/C2782998; undo record is at 0/0; shutdown FALSE
Oct 30 17:16:20 server postgres[4135]: [4] LOG: next transaction id: 203794305; next oid: 32417798
Oct 30 17:16:20 server postgres[4135]: [5] LOG: database system was not properly shut down; automatic recovery in progress
Oct 30 17:16:20 server postgres[4135]: [6] LOG: redo starts at 36/C2782998
Oct 30 17:16:21 server postgres[4135]: [7] PANIC: Invalid page header in block 6157 of 29135442
Oct 30 17:16:21 server postgres[4132]: [1] LOG: startup process (pid 4135) was terminated by signal 6
Oct 30 17:16:21 server postgres[4132]: [2] LOG: aborting startup due to startup process failure
Is there anything I can do not to reload all backups?
Regards
--
Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
6, Chemin d'Harraud Turrou +33-5-61-50-97-01 (Fax)
31190 AUTERIVE +33-6-07-63-80-64 (GSM)
FRANCE Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2003-10-30 22:25 Tom Lane <tgl@sss.pgh.pa.us>
parent: ohp@pyrenet.fr
0 siblings, 1 reply; 20+ messages in thread
From: Tom Lane @ 2003-10-30 22:25 UTC (permalink / raw)
To: ohp@pyrenet.fr; +Cc: pgsql-hackers
ohp@pyrenet.fr writes:
> I've have a MAJOR crash an hour ago and postgresql doesn't start anymoe
> (version 7.3.4).
> Oct 30 17:16:21 server postgres[4135]: [7] PANIC: Invalid page header in block 6157 of 29135442
> Oct 30 17:16:21 server postgres[4132]: [1] LOG: startup process (pid 4135) was terminated by signal 6
> Oct 30 17:16:21 server postgres[4132]: [2] LOG: aborting startup due to startup process failure
> Is there anything I can do not to reload all backups?
You could try turning on zero_damaged_pages in postgresql.conf. If you
are lucky, the page in question is going to be rewritten from WAL anyway.
regards, tom lane
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help
@ 2003-10-30 22:59 ohp@pyrenet.fr
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 20+ messages in thread
From: ohp@pyrenet.fr @ 2003-10-30 22:59 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
Thanks Tom,
The answer came too late and I could'nt wait. pg_resetlog did nearly the
trick, Only one database was really hurt. So I reloaded all but this one
from pg_dumpall then the last one from backup...
I'm cursed
On Thu, 30 Oct 2003, Tom Lane wrote:
> Date: Thu, 30 Oct 2003 17:25:02 -0500
> From: Tom Lane <tgl@sss.pgh.pa.us>
> To: ohp@pyrenet.fr
> Cc: pgsql-hackers list <pgsql-hackers@postgresql.org>
> Subject: Re: [HACKERS] Please help
>
> ohp@pyrenet.fr writes:
> > I've have a MAJOR crash an hour ago and postgresql doesn't start anymoe
> > (version 7.3.4).
>
> > Oct 30 17:16:21 server postgres[4135]: [7] PANIC: Invalid page header in block 6157 of 29135442
> > Oct 30 17:16:21 server postgres[4132]: [1] LOG: startup process (pid 4135) was terminated by signal 6
> > Oct 30 17:16:21 server postgres[4132]: [2] LOG: aborting startup due to startup process failure
>
> > Is there anything I can do not to reload all backups?
>
> You could try turning on zero_damaged_pages in postgresql.conf. If you
> are lucky, the page in question is going to be rewritten from WAL anyway.
>
> regards, tom lane
>
--
Olivier PRENANT Tel: +33-5-61-50-97-00 (Work)
6, Chemin d'Harraud Turrou +33-5-61-50-97-01 (Fax)
31190 AUTERIVE +33-6-07-63-80-64 (GSM)
FRANCE Email: ohp@pyrenet.fr
------------------------------------------------------------------------------
Make your life a dream, make your dream a reality. (St Exupery)
^ permalink raw reply [nested|flat] 20+ messages in thread
* Please help!
@ 2006-06-13 16:08 Brandon E Hofmann <Brandon_E_Hofmann@notes.ntrs.com>
0 siblings, 1 reply; 20+ messages in thread
From: Brandon E Hofmann @ 2006-06-13 16:08 UTC (permalink / raw)
To: pgsql-hackers
When using a temp table in plpgsql functions that has columns comprised
from many tables populated by joins, how do you specify a temp table return
type when its generated by select into and dropped dynamically? I get an
error when I specify returns setof temp_table. Also when I specify a
permanent table as the return type, I get extra blank columns in the result
set which are the additional columns of the permanent table I don't need
and the data is displayed in the wrong columns.
I tried defining composite types, but get a runtime error that it isn't
available unless I first define it outside of the function. Since requests
are dynamic, its impossible for me to determine all possible composite
types required ahead of time.
In plpgsql, how do you return back a result set that is determined and
generated at runtime based on a report request? Also why does plpgsql
require you to define what is returned?
I'm relatively new to PostgreSQL and really impressed by its capabilities.
Do you have any examples showing how to define and return a dynamic result
set at runtime within a function?
I tried posting these questions to the general mailing list, but didn't
receive any feedback. Any help would be greatly appreciated.
Thanks,
Brandon
^ permalink raw reply [nested|flat] 20+ messages in thread
* Re: Please help!
@ 2006-06-16 03:22 Tom Lane <tgl@sss.pgh.pa.us>
parent: Brandon E Hofmann <Brandon_E_Hofmann@notes.ntrs.com>
0 siblings, 0 replies; 20+ messages in thread
From: Tom Lane @ 2006-06-16 03:22 UTC (permalink / raw)
To: Brandon E Hofmann <Brandon_E_Hofmann@notes.ntrs.com>; +Cc: pgsql-hackers
Brandon E Hofmann <Brandon_E_Hofmann@notes.ntrs.com> writes:
> In plpgsql, how do you return back a result set that is determined and
> generated at runtime based on a report request?
If I understand what you are asking for, you don't.
> Also why does plpgsql require you to define what is returned?
plpgsql is not imposing this, the system as a whole does. Else, the
parser would have no idea what to expand "*" to in
select * from myfunc(42);
There are facilities that let you use the same textual function "myfunc"
for different result column sets, but this doesn't get you off the hook
of having to tell the calling query what the column set is going to be.
regards, tom lane
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH] v9
@ 2021-01-14 14:59 Tomas Vondra <tomas@2ndquadrant.com>
0 siblings, 0 replies; 20+ messages in thread
From: Tomas Vondra @ 2021-01-14 14:59 UTC (permalink / raw)
---
contrib/postgres_fdw/deparse.c | 43 ++-
.../postgres_fdw/expected/postgres_fdw.out | 116 ++++++-
contrib/postgres_fdw/option.c | 14 +
contrib/postgres_fdw/postgres_fdw.c | 298 ++++++++++++++----
contrib/postgres_fdw/postgres_fdw.h | 5 +-
contrib/postgres_fdw/sql/postgres_fdw.sql | 91 ++++++
doc/src/sgml/fdwhandler.sgml | 89 +++++-
doc/src/sgml/postgres-fdw.sgml | 13 +
src/backend/executor/nodeModifyTable.c | 160 ++++++++++
src/backend/nodes/list.c | 15 +
src/include/foreign/fdwapi.h | 10 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pg_list.h | 15 +
13 files changed, 809 insertions(+), 66 deletions(-)
diff --git a/contrib/postgres_fdw/deparse.c b/contrib/postgres_fdw/deparse.c
index 3cf7b4eb1e..2d38ab25cb 100644
--- a/contrib/postgres_fdw/deparse.c
+++ b/contrib/postgres_fdw/deparse.c
@@ -1711,7 +1711,7 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs, bool doNothing,
List *withCheckOptionList, List *returningList,
- List **retrieved_attrs)
+ List **retrieved_attrs, int *values_end_len)
{
AttrNumber pindex;
bool first;
@@ -1754,6 +1754,7 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
}
else
appendStringInfoString(buf, " DEFAULT VALUES");
+ *values_end_len = buf->len;
if (doNothing)
appendStringInfoString(buf, " ON CONFLICT DO NOTHING");
@@ -1763,6 +1764,46 @@ deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
withCheckOptionList, returningList, retrieved_attrs);
}
+/*
+ * rebuild remote INSERT statement
+ *
+ */
+void
+rebuildInsertSql(StringInfo buf, char *orig_query,
+ int values_end_len, int num_cols,
+ int num_rows)
+{
+ int i, j;
+ int pindex;
+ bool first;
+
+ /* Copy up to the end of the first record from the original query */
+ appendBinaryStringInfo(buf, orig_query, values_end_len);
+
+ /* Add records to VALUES clause */
+ pindex = num_cols + 1;
+ for (i = 0; i < num_rows; i++)
+ {
+ appendStringInfoString(buf, ", (");
+
+ first = true;
+ for (j = 0; j < num_cols; j++)
+ {
+ if (!first)
+ appendStringInfoString(buf, ", ");
+ first = false;
+
+ appendStringInfo(buf, "$%d", pindex);
+ pindex++;
+ }
+
+ appendStringInfoChar(buf, ')');
+ }
+
+ /* Copy stuff after VALUES clause from the original query */
+ appendStringInfoString(buf, orig_query + values_end_len);
+}
+
/*
* deparse remote UPDATE statement
*
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index c11092f8cc..96bad17ded 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -8911,7 +8911,7 @@ DO $d$
END;
$d$;
ERROR: invalid option "password"
-HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, fetch_size
+HINT: Valid options in this context are: service, passfile, channel_binding, connect_timeout, dbname, host, hostaddr, port, options, application_name, keepalives, keepalives_idle, keepalives_interval, keepalives_count, tcp_user_timeout, sslmode, sslcompression, sslcert, sslkey, sslrootcert, sslcrl, requirepeer, ssl_min_protocol_version, ssl_max_protocol_version, gssencmode, krbsrvname, gsslib, target_session_attrs, use_remote_estimate, fdw_startup_cost, fdw_tuple_cost, extensions, updatable, fetch_size, batch_size
CONTEXT: SQL statement "ALTER SERVER loopback_nopw OPTIONS (ADD password 'dummypw')"
PL/pgSQL function inline_code_block line 3 at EXECUTE
-- If we add a password for our user mapping instead, we should get a different
@@ -9053,3 +9053,117 @@ SELECT 1 FROM ft1 LIMIT 1;
ALTER SERVER loopback OPTIONS (ADD use_remote_estimate 'off');
-- The invalid connection gets closed in pgfdw_xact_callback during commit.
COMMIT;
+-- ===================================================================
+-- batch insert
+-- ===================================================================
+BEGIN;
+CREATE SERVER batch10 FOREIGN DATA WRAPPER postgres_fdw OPTIONS( batch_size '10' );
+SELECT count(*)
+FROM pg_foreign_server
+WHERE srvname = 'batch10'
+AND srvoptions @> array['batch_size=10'];
+ count
+-------
+ 1
+(1 row)
+
+ALTER SERVER batch10 OPTIONS( SET batch_size '20' );
+SELECT count(*)
+FROM pg_foreign_server
+WHERE srvname = 'batch10'
+AND srvoptions @> array['batch_size=10'];
+ count
+-------
+ 0
+(1 row)
+
+SELECT count(*)
+FROM pg_foreign_server
+WHERE srvname = 'batch10'
+AND srvoptions @> array['batch_size=20'];
+ count
+-------
+ 1
+(1 row)
+
+CREATE FOREIGN TABLE table30 ( x int ) SERVER batch10 OPTIONS ( batch_size '30' );
+SELECT COUNT(*)
+FROM pg_foreign_table
+WHERE ftrelid = 'table30'::regclass
+AND ftoptions @> array['batch_size=30'];
+ count
+-------
+ 1
+(1 row)
+
+ALTER FOREIGN TABLE table30 OPTIONS ( SET batch_size '40');
+SELECT COUNT(*)
+FROM pg_foreign_table
+WHERE ftrelid = 'table30'::regclass
+AND ftoptions @> array['batch_size=30'];
+ count
+-------
+ 0
+(1 row)
+
+SELECT COUNT(*)
+FROM pg_foreign_table
+WHERE ftrelid = 'table30'::regclass
+AND ftoptions @> array['batch_size=40'];
+ count
+-------
+ 1
+(1 row)
+
+ROLLBACK;
+CREATE TABLE batch_table ( x int );
+CREATE FOREIGN TABLE ftable ( x int ) SERVER loopback OPTIONS ( table_name 'batch_table', batch_size '10' );
+INSERT INTO ftable SELECT * FROM generate_series(1, 10) i;
+INSERT INTO ftable SELECT * FROM generate_series(11, 31) i;
+INSERT INTO ftable VALUES (32);
+INSERT INTO ftable VALUES (33), (34);
+SELECT COUNT(*) FROM ftable;
+ count
+-------
+ 34
+(1 row)
+
+TRUNCATE batch_table;
+DROP FOREIGN TABLE ftable;
+-- Disable batch insert
+CREATE FOREIGN TABLE ftable ( x int ) SERVER loopback OPTIONS ( table_name 'batch_table', batch_size '1' );
+INSERT INTO ftable VALUES (1), (2);
+SELECT COUNT(*) FROM ftable;
+ count
+-------
+ 2
+(1 row)
+
+DROP FOREIGN TABLE ftable;
+DROP TABLE batch_table;
+-- Use partitioning
+CREATE TABLE batch_table ( x int ) PARTITION BY HASH (x);
+CREATE TABLE batch_table_p0 (LIKE batch_table);
+CREATE FOREIGN TABLE batch_table_p0f
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 3, REMAINDER 0)
+ SERVER loopback
+ OPTIONS (table_name 'batch_table_p0', batch_size '10');
+CREATE TABLE batch_table_p1 (LIKE batch_table);
+CREATE FOREIGN TABLE batch_table_p1f
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 3, REMAINDER 1)
+ SERVER loopback
+ OPTIONS (table_name 'batch_table_p1', batch_size '1');
+CREATE TABLE batch_table_p2
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 3, REMAINDER 2);
+INSERT INTO batch_table SELECT * FROM generate_series(1, 66) i;
+SELECT COUNT(*) FROM batch_table;
+ count
+-------
+ 66
+(1 row)
+
+-- Clean up
+DROP TABLE batch_table CASCADE;
diff --git a/contrib/postgres_fdw/option.c b/contrib/postgres_fdw/option.c
index 1fec3c3eea..64698c4da3 100644
--- a/contrib/postgres_fdw/option.c
+++ b/contrib/postgres_fdw/option.c
@@ -142,6 +142,17 @@ postgres_fdw_validator(PG_FUNCTION_ARGS)
errmsg("%s requires a non-negative integer value",
def->defname)));
}
+ else if (strcmp(def->defname, "batch_size") == 0)
+ {
+ int batch_size;
+
+ batch_size = strtol(defGetString(def), NULL, 10);
+ if (batch_size <= 0)
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("%s requires a non-negative integer value",
+ def->defname)));
+ }
else if (strcmp(def->defname, "password_required") == 0)
{
bool pw_required = defGetBoolean(def);
@@ -203,6 +214,9 @@ InitPgFdwOptions(void)
/* fetch_size is available on both server and table */
{"fetch_size", ForeignServerRelationId, false},
{"fetch_size", ForeignTableRelationId, false},
+ /* batch_size is available on both server and table */
+ {"batch_size", ForeignServerRelationId, false},
+ {"batch_size", ForeignTableRelationId, false},
{"password_required", UserMappingRelationId, false},
/*
diff --git a/contrib/postgres_fdw/postgres_fdw.c b/contrib/postgres_fdw/postgres_fdw.c
index 2f2d4d171c..85a072bc88 100644
--- a/contrib/postgres_fdw/postgres_fdw.c
+++ b/contrib/postgres_fdw/postgres_fdw.c
@@ -87,8 +87,10 @@ enum FdwScanPrivateIndex
* 1) INSERT/UPDATE/DELETE statement text to be sent to the remote server
* 2) Integer list of target attribute numbers for INSERT/UPDATE
* (NIL for a DELETE)
- * 3) Boolean flag showing if the remote query has a RETURNING clause
- * 4) Integer list of attribute numbers retrieved by RETURNING, if any
+ * 3) Length till the end of VALUES clause for INSERT
+ * (-1 for a DELETE/UPDATE)
+ * 4) Boolean flag showing if the remote query has a RETURNING clause
+ * 5) Integer list of attribute numbers retrieved by RETURNING, if any
*/
enum FdwModifyPrivateIndex
{
@@ -96,6 +98,8 @@ enum FdwModifyPrivateIndex
FdwModifyPrivateUpdateSql,
/* Integer list of target attribute numbers for INSERT/UPDATE */
FdwModifyPrivateTargetAttnums,
+ /* Length till the end of VALUES clause (as an integer Value node) */
+ FdwModifyPrivateLen,
/* has-returning flag (as an integer Value node) */
FdwModifyPrivateHasReturning,
/* Integer list of attribute numbers retrieved by RETURNING */
@@ -176,7 +180,10 @@ typedef struct PgFdwModifyState
/* extracted fdw_private data */
char *query; /* text of INSERT/UPDATE/DELETE command */
+ char *orig_query; /* original text of INSERT command */
List *target_attrs; /* list of target attribute numbers */
+ int values_end; /* length up to the end of VALUES */
+ int batch_size; /* value of FDW option "batch_size" */
bool has_returning; /* is there a RETURNING clause? */
List *retrieved_attrs; /* attr numbers retrieved by RETURNING */
@@ -185,6 +192,9 @@ typedef struct PgFdwModifyState
int p_nums; /* number of parameters to transmit */
FmgrInfo *p_flinfo; /* output conversion functions for them */
+ /* batch operation stuff */
+ int num_slots; /* number of slots to insert */
+
/* working memory context */
MemoryContext temp_cxt; /* context for per-tuple temporary data */
@@ -343,6 +353,12 @@ static TupleTableSlot *postgresExecForeignInsert(EState *estate,
ResultRelInfo *resultRelInfo,
TupleTableSlot *slot,
TupleTableSlot *planSlot);
+static TupleTableSlot **postgresExecForeignBatchInsert(EState *estate,
+ ResultRelInfo *resultRelInfo,
+ TupleTableSlot **slots,
+ TupleTableSlot **planSlots,
+ int *numSlots);
+static int postgresGetForeignModifyBatchSize(ResultRelInfo *resultRelInfo);
static TupleTableSlot *postgresExecForeignUpdate(EState *estate,
ResultRelInfo *resultRelInfo,
TupleTableSlot *slot,
@@ -429,20 +445,24 @@ static PgFdwModifyState *create_foreign_modify(EState *estate,
Plan *subplan,
char *query,
List *target_attrs,
+ int len,
bool has_returning,
List *retrieved_attrs);
-static TupleTableSlot *execute_foreign_modify(EState *estate,
+static TupleTableSlot **execute_foreign_modify(EState *estate,
ResultRelInfo *resultRelInfo,
CmdType operation,
- TupleTableSlot *slot,
- TupleTableSlot *planSlot);
+ TupleTableSlot **slots,
+ TupleTableSlot **planSlots,
+ int *numSlots);
static void prepare_foreign_modify(PgFdwModifyState *fmstate);
static const char **convert_prep_stmt_params(PgFdwModifyState *fmstate,
ItemPointer tupleid,
- TupleTableSlot *slot);
+ TupleTableSlot **slots,
+ int numSlots);
static void store_returning_result(PgFdwModifyState *fmstate,
TupleTableSlot *slot, PGresult *res);
static void finish_foreign_modify(PgFdwModifyState *fmstate);
+static void deallocate_query(PgFdwModifyState *fmstate);
static List *build_remote_returning(Index rtindex, Relation rel,
List *returningList);
static void rebuild_fdw_scan_tlist(ForeignScan *fscan, List *tlist);
@@ -530,6 +550,8 @@ postgres_fdw_handler(PG_FUNCTION_ARGS)
routine->PlanForeignModify = postgresPlanForeignModify;
routine->BeginForeignModify = postgresBeginForeignModify;
routine->ExecForeignInsert = postgresExecForeignInsert;
+ routine->ExecForeignBatchInsert = postgresExecForeignBatchInsert;
+ routine->GetForeignModifyBatchSize = postgresGetForeignModifyBatchSize;
routine->ExecForeignUpdate = postgresExecForeignUpdate;
routine->ExecForeignDelete = postgresExecForeignDelete;
routine->EndForeignModify = postgresEndForeignModify;
@@ -1665,6 +1687,7 @@ postgresPlanForeignModify(PlannerInfo *root,
List *returningList = NIL;
List *retrieved_attrs = NIL;
bool doNothing = false;
+ int values_end_len = -1;
initStringInfo(&sql);
@@ -1752,7 +1775,7 @@ postgresPlanForeignModify(PlannerInfo *root,
deparseInsertSql(&sql, rte, resultRelation, rel,
targetAttrs, doNothing,
withCheckOptionList, returningList,
- &retrieved_attrs);
+ &retrieved_attrs, &values_end_len);
break;
case CMD_UPDATE:
deparseUpdateSql(&sql, rte, resultRelation, rel,
@@ -1776,8 +1799,9 @@ postgresPlanForeignModify(PlannerInfo *root,
* Build the fdw_private list that will be available to the executor.
* Items in the list must match enum FdwModifyPrivateIndex, above.
*/
- return list_make4(makeString(sql.data),
+ return list_make5(makeString(sql.data),
targetAttrs,
+ makeInteger(values_end_len),
makeInteger((retrieved_attrs != NIL)),
retrieved_attrs);
}
@@ -1797,6 +1821,7 @@ postgresBeginForeignModify(ModifyTableState *mtstate,
char *query;
List *target_attrs;
bool has_returning;
+ int values_end_len;
List *retrieved_attrs;
RangeTblEntry *rte;
@@ -1812,6 +1837,8 @@ postgresBeginForeignModify(ModifyTableState *mtstate,
FdwModifyPrivateUpdateSql));
target_attrs = (List *) list_nth(fdw_private,
FdwModifyPrivateTargetAttnums);
+ values_end_len = intVal(list_nth(fdw_private,
+ FdwModifyPrivateLen));
has_returning = intVal(list_nth(fdw_private,
FdwModifyPrivateHasReturning));
retrieved_attrs = (List *) list_nth(fdw_private,
@@ -1829,6 +1856,7 @@ postgresBeginForeignModify(ModifyTableState *mtstate,
mtstate->mt_plans[subplan_index]->plan,
query,
target_attrs,
+ values_end_len,
has_returning,
retrieved_attrs);
@@ -1846,7 +1874,8 @@ postgresExecForeignInsert(EState *estate,
TupleTableSlot *planSlot)
{
PgFdwModifyState *fmstate = (PgFdwModifyState *) resultRelInfo->ri_FdwState;
- TupleTableSlot *rslot;
+ TupleTableSlot **rslot;
+ int numSlots = 1;
/*
* If the fmstate has aux_fmstate set, use the aux_fmstate (see
@@ -1855,7 +1884,36 @@ postgresExecForeignInsert(EState *estate,
if (fmstate->aux_fmstate)
resultRelInfo->ri_FdwState = fmstate->aux_fmstate;
rslot = execute_foreign_modify(estate, resultRelInfo, CMD_INSERT,
- slot, planSlot);
+ &slot, &planSlot, &numSlots);
+ /* Revert that change */
+ if (fmstate->aux_fmstate)
+ resultRelInfo->ri_FdwState = fmstate;
+
+ return rslot ? *rslot : NULL;
+}
+
+/*
+ * postgresExecForeignBatchInsert
+ * Insert multiple rows into a foreign table
+ */
+static TupleTableSlot **
+postgresExecForeignBatchInsert(EState *estate,
+ ResultRelInfo *resultRelInfo,
+ TupleTableSlot **slots,
+ TupleTableSlot **planSlots,
+ int *numSlots)
+{
+ PgFdwModifyState *fmstate = (PgFdwModifyState *) resultRelInfo->ri_FdwState;
+ TupleTableSlot **rslot;
+
+ /*
+ * If the fmstate has aux_fmstate set, use the aux_fmstate (see
+ * postgresBeginForeignInsert())
+ */
+ if (fmstate->aux_fmstate)
+ resultRelInfo->ri_FdwState = fmstate->aux_fmstate;
+ rslot = execute_foreign_modify(estate, resultRelInfo, CMD_INSERT,
+ slots, planSlots, numSlots);
/* Revert that change */
if (fmstate->aux_fmstate)
resultRelInfo->ri_FdwState = fmstate;
@@ -1863,6 +1921,23 @@ postgresExecForeignInsert(EState *estate,
return rslot;
}
+/*
+ * postgresGetForeignModifyBatchSize
+ * Report the maximum number of tuples that can be inserted in bulk
+ */
+static int
+postgresGetForeignModifyBatchSize(ResultRelInfo *resultRelInfo)
+{
+ /* Disable batching when we have to use RETURNING. */
+ if (resultRelInfo->ri_projectReturning != NULL ||
+ (resultRelInfo->ri_TrigDesc &&
+ resultRelInfo->ri_TrigDesc->trig_insert_after_row))
+ return 1;
+
+ /* Otherwise use the batch size specified for server/table. */
+ return ((PgFdwModifyState *) resultRelInfo->ri_FdwState)->batch_size;
+}
+
/*
* postgresExecForeignUpdate
* Update one row in a foreign table
@@ -1873,8 +1948,13 @@ postgresExecForeignUpdate(EState *estate,
TupleTableSlot *slot,
TupleTableSlot *planSlot)
{
- return execute_foreign_modify(estate, resultRelInfo, CMD_UPDATE,
- slot, planSlot);
+ TupleTableSlot **rslot;
+ int numSlots = 1;
+
+ rslot = execute_foreign_modify(estate, resultRelInfo, CMD_UPDATE,
+ &slot, &planSlot, &numSlots);
+
+ return rslot ? rslot[0] : NULL;
}
/*
@@ -1887,8 +1967,13 @@ postgresExecForeignDelete(EState *estate,
TupleTableSlot *slot,
TupleTableSlot *planSlot)
{
- return execute_foreign_modify(estate, resultRelInfo, CMD_DELETE,
- slot, planSlot);
+ TupleTableSlot **rslot;
+ int numSlots = 1;
+
+ rslot = execute_foreign_modify(estate, resultRelInfo, CMD_DELETE,
+ &slot, &planSlot, &numSlots);
+
+ return rslot ? rslot[0] : NULL;
}
/*
@@ -1925,6 +2010,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
RangeTblEntry *rte;
TupleDesc tupdesc = RelationGetDescr(rel);
int attnum;
+ int values_end_len;
StringInfoData sql;
List *targetAttrs = NIL;
List *retrieved_attrs = NIL;
@@ -2001,7 +2087,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
deparseInsertSql(&sql, rte, resultRelation, rel, targetAttrs, doNothing,
resultRelInfo->ri_WithCheckOptions,
resultRelInfo->ri_returningList,
- &retrieved_attrs);
+ &retrieved_attrs, &values_end_len);
/* Construct an execution state. */
fmstate = create_foreign_modify(mtstate->ps.state,
@@ -2011,6 +2097,7 @@ postgresBeginForeignInsert(ModifyTableState *mtstate,
NULL,
sql.data,
targetAttrs,
+ values_end_len,
retrieved_attrs != NIL,
retrieved_attrs);
@@ -2636,6 +2723,9 @@ postgresExplainForeignModify(ModifyTableState *mtstate,
FdwModifyPrivateUpdateSql));
ExplainPropertyText("Remote SQL", sql, es);
+
+ if (rinfo->ri_BatchSize > 0)
+ ExplainPropertyInteger("Batch Size", NULL, rinfo->ri_BatchSize, es);
}
}
@@ -3530,6 +3620,7 @@ create_foreign_modify(EState *estate,
Plan *subplan,
char *query,
List *target_attrs,
+ int values_end,
bool has_returning,
List *retrieved_attrs)
{
@@ -3538,6 +3629,7 @@ create_foreign_modify(EState *estate,
TupleDesc tupdesc = RelationGetDescr(rel);
Oid userid;
ForeignTable *table;
+ ForeignServer *server;
UserMapping *user;
AttrNumber n_params;
Oid typefnoid;
@@ -3564,7 +3656,10 @@ create_foreign_modify(EState *estate,
/* Set up remote query information. */
fmstate->query = query;
+ if (operation == CMD_INSERT)
+ fmstate->orig_query = pstrdup(fmstate->query);
fmstate->target_attrs = target_attrs;
+ fmstate->values_end = values_end;
fmstate->has_returning = has_returning;
fmstate->retrieved_attrs = retrieved_attrs;
@@ -3616,6 +3711,44 @@ create_foreign_modify(EState *estate,
Assert(fmstate->p_nums <= n_params);
+ /* Set batch_size from foreign server/table options. */
+ if (operation == CMD_INSERT)
+ {
+ /* Check the foreign table option. */
+ foreach(lc, table->options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "batch_size") == 0)
+ {
+ fmstate->batch_size = strtol(defGetString(def), NULL, 10);
+ break;
+ }
+ }
+
+ /* Check the foreign server option if the table option is not set. */
+ if (fmstate->batch_size == 0)
+ {
+ server = GetForeignServer(table->serverid);
+ foreach(lc, server->options)
+ {
+ DefElem *def = (DefElem *) lfirst(lc);
+
+ if (strcmp(def->defname, "batch_size") == 0)
+ {
+ fmstate->batch_size = strtol(defGetString(def), NULL, 10);
+ break;
+ }
+ }
+ }
+
+ /* If neither the table nor server option is set, set the default. */
+ if (fmstate->batch_size == 0)
+ fmstate->batch_size = 100;
+ }
+
+ fmstate->num_slots = 1;
+
/* Initialize auxiliary state */
fmstate->aux_fmstate = NULL;
@@ -3626,26 +3759,50 @@ create_foreign_modify(EState *estate,
* execute_foreign_modify
* Perform foreign-table modification as required, and fetch RETURNING
* result if any. (This is the shared guts of postgresExecForeignInsert,
- * postgresExecForeignUpdate, and postgresExecForeignDelete.)
+ * postgresExecForeignBatchInsert, postgresExecForeignUpdate, and
+ * postgresExecForeignDelete.)
*/
-static TupleTableSlot *
+static TupleTableSlot **
execute_foreign_modify(EState *estate,
ResultRelInfo *resultRelInfo,
CmdType operation,
- TupleTableSlot *slot,
- TupleTableSlot *planSlot)
+ TupleTableSlot **slots,
+ TupleTableSlot **planSlots,
+ int *numSlots)
{
PgFdwModifyState *fmstate = (PgFdwModifyState *) resultRelInfo->ri_FdwState;
ItemPointer ctid = NULL;
const char **p_values;
PGresult *res;
int n_rows;
+ StringInfoData sql;
/* The operation should be INSERT, UPDATE, or DELETE */
Assert(operation == CMD_INSERT ||
operation == CMD_UPDATE ||
operation == CMD_DELETE);
+ /*
+ * If the existing query was deparsed and prepared for a different number
+ * of rows, rebuild it for the proper number.
+ */
+ if (operation == CMD_INSERT && fmstate->num_slots != *numSlots)
+ {
+ /* Destroy the prepared statement created previously */
+ if (fmstate->p_name)
+ deallocate_query(fmstate);
+
+ /*
+ * Build INSERT string with numSlots records in its VALUES clause.
+ */
+ initStringInfo(&sql);
+ rebuildInsertSql(&sql, fmstate->orig_query, fmstate->values_end,
+ fmstate->p_nums, *numSlots - 1);
+ pfree(fmstate->query);
+ fmstate->query = sql.data;
+ fmstate->num_slots = *numSlots;
+ }
+
/* Set up the prepared statement on the remote server, if we didn't yet */
if (!fmstate->p_name)
prepare_foreign_modify(fmstate);
@@ -3658,7 +3815,7 @@ execute_foreign_modify(EState *estate,
Datum datum;
bool isNull;
- datum = ExecGetJunkAttribute(planSlot,
+ datum = ExecGetJunkAttribute(planSlots[0],
fmstate->ctidAttno,
&isNull);
/* shouldn't ever get a null result... */
@@ -3668,14 +3825,14 @@ execute_foreign_modify(EState *estate,
}
/* Convert parameters needed by prepared statement to text form */
- p_values = convert_prep_stmt_params(fmstate, ctid, slot);
+ p_values = convert_prep_stmt_params(fmstate, ctid, slots, *numSlots);
/*
* Execute the prepared statement.
*/
if (!PQsendQueryPrepared(fmstate->conn,
fmstate->p_name,
- fmstate->p_nums,
+ fmstate->p_nums * (*numSlots),
p_values,
NULL,
NULL,
@@ -3696,9 +3853,10 @@ execute_foreign_modify(EState *estate,
/* Check number of rows affected, and fetch RETURNING tuple if any */
if (fmstate->has_returning)
{
+ Assert(*numSlots == 1);
n_rows = PQntuples(res);
if (n_rows > 0)
- store_returning_result(fmstate, slot, res);
+ store_returning_result(fmstate, slots[0], res);
}
else
n_rows = atoi(PQcmdTuples(res));
@@ -3708,10 +3866,12 @@ execute_foreign_modify(EState *estate,
MemoryContextReset(fmstate->temp_cxt);
+ *numSlots = n_rows;
+
/*
* Return NULL if nothing was inserted/updated/deleted on the remote end
*/
- return (n_rows > 0) ? slot : NULL;
+ return (n_rows > 0) ? slots : NULL;
}
/*
@@ -3771,52 +3931,64 @@ prepare_foreign_modify(PgFdwModifyState *fmstate)
static const char **
convert_prep_stmt_params(PgFdwModifyState *fmstate,
ItemPointer tupleid,
- TupleTableSlot *slot)
+ TupleTableSlot **slots,
+ int numSlots)
{
const char **p_values;
+ int i;
+ int j;
int pindex = 0;
MemoryContext oldcontext;
oldcontext = MemoryContextSwitchTo(fmstate->temp_cxt);
- p_values = (const char **) palloc(sizeof(char *) * fmstate->p_nums);
+ p_values = (const char **) palloc(sizeof(char *) * fmstate->p_nums * numSlots);
+
+ /* ctid is provided only for UPDATE/DELETE, which don't allow batching */
+ Assert(!(tupleid != NULL && numSlots > 1));
/* 1st parameter should be ctid, if it's in use */
if (tupleid != NULL)
{
+ Assert(numSlots == 1);
/* don't need set_transmission_modes for TID output */
p_values[pindex] = OutputFunctionCall(&fmstate->p_flinfo[pindex],
PointerGetDatum(tupleid));
pindex++;
}
- /* get following parameters from slot */
- if (slot != NULL && fmstate->target_attrs != NIL)
+ /* get following parameters from slots */
+ if (slots != NULL && fmstate->target_attrs != NIL)
{
int nestlevel;
ListCell *lc;
nestlevel = set_transmission_modes();
- foreach(lc, fmstate->target_attrs)
+ for (i = 0; i < numSlots; i++)
{
- int attnum = lfirst_int(lc);
- Datum value;
- bool isnull;
+ j = (tupleid != NULL) ? 1 : 0;
+ foreach(lc, fmstate->target_attrs)
+ {
+ int attnum = lfirst_int(lc);
+ Datum value;
+ bool isnull;
- value = slot_getattr(slot, attnum, &isnull);
- if (isnull)
- p_values[pindex] = NULL;
- else
- p_values[pindex] = OutputFunctionCall(&fmstate->p_flinfo[pindex],
- value);
- pindex++;
+ value = slot_getattr(slots[i], attnum, &isnull);
+ if (isnull)
+ p_values[pindex] = NULL;
+ else
+ p_values[pindex] = OutputFunctionCall(&fmstate->p_flinfo[j],
+ value);
+ pindex++;
+ j++;
+ }
}
reset_transmission_modes(nestlevel);
}
- Assert(pindex == fmstate->p_nums);
+ Assert(pindex == fmstate->p_nums * numSlots);
MemoryContextSwitchTo(oldcontext);
@@ -3870,29 +4042,41 @@ finish_foreign_modify(PgFdwModifyState *fmstate)
Assert(fmstate != NULL);
/* If we created a prepared statement, destroy it */
- if (fmstate->p_name)
- {
- char sql[64];
- PGresult *res;
-
- snprintf(sql, sizeof(sql), "DEALLOCATE %s", fmstate->p_name);
-
- /*
- * We don't use a PG_TRY block here, so be careful not to throw error
- * without releasing the PGresult.
- */
- res = pgfdw_exec_query(fmstate->conn, sql);
- if (PQresultStatus(res) != PGRES_COMMAND_OK)
- pgfdw_report_error(ERROR, res, fmstate->conn, true, sql);
- PQclear(res);
- fmstate->p_name = NULL;
- }
+ deallocate_query(fmstate);
/* Release remote connection */
ReleaseConnection(fmstate->conn);
fmstate->conn = NULL;
}
+/*
+ * deallocate_query
+ * Deallocate a prepared statement for a foreign insert/update/delete
+ * operation
+ */
+static void
+deallocate_query(PgFdwModifyState *fmstate)
+{
+ char sql[64];
+ PGresult *res;
+
+ /* do nothing if the query is not allocated */
+ if (!fmstate->p_name)
+ return;
+
+ snprintf(sql, sizeof(sql), "DEALLOCATE %s", fmstate->p_name);
+
+ /*
+ * We don't use a PG_TRY block here, so be careful not to throw error
+ * without releasing the PGresult.
+ */
+ res = pgfdw_exec_query(fmstate->conn, sql);
+ if (PQresultStatus(res) != PGRES_COMMAND_OK)
+ pgfdw_report_error(ERROR, res, fmstate->conn, true, sql);
+ PQclear(res);
+ fmstate->p_name = NULL;
+}
+
/*
* build_remote_returning
* Build a RETURNING targetlist of a remote query for performing an
diff --git a/contrib/postgres_fdw/postgres_fdw.h b/contrib/postgres_fdw/postgres_fdw.h
index 19ea27a1bc..1f67b4d9fd 100644
--- a/contrib/postgres_fdw/postgres_fdw.h
+++ b/contrib/postgres_fdw/postgres_fdw.h
@@ -161,7 +161,10 @@ extern void deparseInsertSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs, bool doNothing,
List *withCheckOptionList, List *returningList,
- List **retrieved_attrs);
+ List **retrieved_attrs, int *values_end_len);
+extern void rebuildInsertSql(StringInfo buf, char *orig_query,
+ int values_end_len, int num_cols,
+ int num_rows);
extern void deparseUpdateSql(StringInfo buf, RangeTblEntry *rte,
Index rtindex, Relation rel,
List *targetAttrs,
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 25dbc08b98..fd5abf2471 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -2711,3 +2711,94 @@ SELECT 1 FROM ft1 LIMIT 1;
ALTER SERVER loopback OPTIONS (ADD use_remote_estimate 'off');
-- The invalid connection gets closed in pgfdw_xact_callback during commit.
COMMIT;
+
+-- ===================================================================
+-- batch insert
+-- ===================================================================
+
+BEGIN;
+
+CREATE SERVER batch10 FOREIGN DATA WRAPPER postgres_fdw OPTIONS( batch_size '10' );
+
+SELECT count(*)
+FROM pg_foreign_server
+WHERE srvname = 'batch10'
+AND srvoptions @> array['batch_size=10'];
+
+ALTER SERVER batch10 OPTIONS( SET batch_size '20' );
+
+SELECT count(*)
+FROM pg_foreign_server
+WHERE srvname = 'batch10'
+AND srvoptions @> array['batch_size=10'];
+
+SELECT count(*)
+FROM pg_foreign_server
+WHERE srvname = 'batch10'
+AND srvoptions @> array['batch_size=20'];
+
+CREATE FOREIGN TABLE table30 ( x int ) SERVER batch10 OPTIONS ( batch_size '30' );
+
+SELECT COUNT(*)
+FROM pg_foreign_table
+WHERE ftrelid = 'table30'::regclass
+AND ftoptions @> array['batch_size=30'];
+
+ALTER FOREIGN TABLE table30 OPTIONS ( SET batch_size '40');
+
+SELECT COUNT(*)
+FROM pg_foreign_table
+WHERE ftrelid = 'table30'::regclass
+AND ftoptions @> array['batch_size=30'];
+
+SELECT COUNT(*)
+FROM pg_foreign_table
+WHERE ftrelid = 'table30'::regclass
+AND ftoptions @> array['batch_size=40'];
+
+ROLLBACK;
+
+CREATE TABLE batch_table ( x int );
+
+CREATE FOREIGN TABLE ftable ( x int ) SERVER loopback OPTIONS ( table_name 'batch_table', batch_size '10' );
+INSERT INTO ftable SELECT * FROM generate_series(1, 10) i;
+INSERT INTO ftable SELECT * FROM generate_series(11, 31) i;
+INSERT INTO ftable VALUES (32);
+INSERT INTO ftable VALUES (33), (34);
+SELECT COUNT(*) FROM ftable;
+TRUNCATE batch_table;
+DROP FOREIGN TABLE ftable;
+
+-- Disable batch insert
+CREATE FOREIGN TABLE ftable ( x int ) SERVER loopback OPTIONS ( table_name 'batch_table', batch_size '1' );
+INSERT INTO ftable VALUES (1), (2);
+SELECT COUNT(*) FROM ftable;
+DROP FOREIGN TABLE ftable;
+DROP TABLE batch_table;
+
+-- Use partitioning
+CREATE TABLE batch_table ( x int ) PARTITION BY HASH (x);
+
+CREATE TABLE batch_table_p0 (LIKE batch_table);
+CREATE FOREIGN TABLE batch_table_p0f
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 3, REMAINDER 0)
+ SERVER loopback
+ OPTIONS (table_name 'batch_table_p0', batch_size '10');
+
+CREATE TABLE batch_table_p1 (LIKE batch_table);
+CREATE FOREIGN TABLE batch_table_p1f
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 3, REMAINDER 1)
+ SERVER loopback
+ OPTIONS (table_name 'batch_table_p1', batch_size '1');
+
+CREATE TABLE batch_table_p2
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 3, REMAINDER 2);
+
+INSERT INTO batch_table SELECT * FROM generate_series(1, 66) i;
+SELECT COUNT(*) FROM batch_table;
+
+-- Clean up
+DROP TABLE batch_table CASCADE;
diff --git a/doc/src/sgml/fdwhandler.sgml b/doc/src/sgml/fdwhandler.sgml
index 9c9293414c..854913ae5f 100644
--- a/doc/src/sgml/fdwhandler.sgml
+++ b/doc/src/sgml/fdwhandler.sgml
@@ -523,8 +523,9 @@ BeginForeignModify(ModifyTableState *mtstate,
Begin executing a foreign table modification operation. This routine is
called during executor startup. It should perform any initialization
needed prior to the actual table modifications. Subsequently,
- <function>ExecForeignInsert</function>, <function>ExecForeignUpdate</function> or
- <function>ExecForeignDelete</function> will be called for each tuple to be
+ <function>ExecForeignInsert/ExecForeignBatchInsert</function>,
+ <function>ExecForeignUpdate</function> or
+ <function>ExecForeignDelete</function> will be called for tuple(s) to be
inserted, updated, or deleted.
</para>
@@ -614,6 +615,81 @@ ExecForeignInsert(EState *estate,
<para>
<programlisting>
+TupleTableSlot **
+ExecForeignBatchInsert(EState *estate,
+ ResultRelInfo *rinfo,
+ TupleTableSlot **slots,
+ TupleTableSlot *planSlots,
+ int *numSlots);
+</programlisting>
+
+ Insert multiple tuples in bulk into the foreign table.
+ The parameters are the same for <function>ExecForeignInsert</function>
+ except <literal>slots</literal> and <literal>planSlots</literal> contain
+ multiple tuples and <literal>*numSlots></literal> specifies the number of
+ tuples in those arrays.
+ </para>
+
+ <para>
+ The return value is an array of slots containing the data that was
+ actually inserted (this might differ from the data supplied, for
+ example as a result of trigger actions.)
+ The passed-in <literal>slots</literal> can be re-used for this purpose.
+ The number of successfully inserted tuples is returned in
+ <literal>*numSlots</literal>.
+ </para>
+
+ <para>
+ The data in the returned slot is used only if the <command>INSERT</command>
+ statement involves a view
+ <literal>WITH CHECK OPTION</literal>; or if the foreign table has
+ an <literal>AFTER ROW</literal> trigger. Triggers require all columns,
+ but the FDW could choose to optimize away returning some or all columns
+ depending on the contents of the
+ <literal>WITH CHECK OPTION</literal> constraints.
+ </para>
+
+ <para>
+ If the <function>ExecForeignBatchInsert</function> or
+ <function>GetForeignModifyBatchSize</function> pointer is set to
+ <literal>NULL</literal>, attempts to insert into the foreign table will
+ use <function>ExecForeignInsert</function>.
+ This function is not used if the <command>INSERT</command> has the
+ <literal>RETURNING></literal> clause.
+ </para>
+
+ <para>
+ Note that this function is also called when inserting routed tuples into
+ a foreign-table partition. See the callback functions
+ described below that allow the FDW to support that.
+ </para>
+
+ <para>
+<programlisting>
+int
+GetForeignModifyBatchSize(ResultRelInfo *rinfo);
+</programlisting>
+
+ Report the maximum number of tuples that a single
+ <function>ExecForeignBatchInsert</function> call can handle for
+ the specified foreign table. That is, The executor passes at most
+ the number of tuples that this function returns to
+ <function>ExecForeignBatchInsert</function>.
+ <literal>rinfo</literal> is the <structname>ResultRelInfo</structname> struct describing
+ the target foreign table.
+ The FDW is expected to provide a foreign server and/or foreign
+ table option for the user to set this value, or some hard-coded value.
+ </para>
+
+ <para>
+ If the <function>ExecForeignBatchInsert</function> or
+ <function>GetForeignModifyBatchSize</function> pointer is set to
+ <literal>NULL</literal>, attempts to insert into the foreign table will
+ use <function>ExecForeignInsert</function>.
+ </para>
+
+ <para>
+<programlisting>
TupleTableSlot *
ExecForeignUpdate(EState *estate,
ResultRelInfo *rinfo,
@@ -741,8 +817,9 @@ BeginForeignInsert(ModifyTableState *mtstate,
in both cases when it is the partition chosen for tuple routing and the
target specified in a <command>COPY FROM</command> command. It should
perform any initialization needed prior to the actual insertion.
- Subsequently, <function>ExecForeignInsert</function> will be called for
- each tuple to be inserted into the foreign table.
+ Subsequently, <function>ExecForeignInsert</function> or
+ <function>ExecForeignBatchInsert</function> will be called for
+ tuple(s) to be inserted into the foreign table.
</para>
<para>
@@ -773,8 +850,8 @@ BeginForeignInsert(ModifyTableState *mtstate,
<para>
Note that if the FDW does not support routable foreign-table partitions
and/or executing <command>COPY FROM</command> on foreign tables, this
- function or <function>ExecForeignInsert</function> subsequently called
- must throw error as needed.
+ function or <function>ExecForeignInsert/ExecForeignBatchInsert</function>
+ subsequently called must throw error as needed.
</para>
<para>
diff --git a/doc/src/sgml/postgres-fdw.sgml b/doc/src/sgml/postgres-fdw.sgml
index e6fd2143c1..97eeb64a02 100644
--- a/doc/src/sgml/postgres-fdw.sgml
+++ b/doc/src/sgml/postgres-fdw.sgml
@@ -354,6 +354,19 @@ OPTIONS (ADD password_required 'false');
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><literal>batch_size</literal></term>
+ <listitem>
+ <para>
+ This option specifies the number of rows <filename>postgres_fdw</filename>
+ should insert in each insert operation. It can be specified for a
+ foreign table or a foreign server. The option specified on a table
+ overrides an option specified for the server.
+ The default is <literal>100</literal>.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect3>
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c
index 921e695419..fc48d8cb75 100644
--- a/src/backend/executor/nodeModifyTable.c
+++ b/src/backend/executor/nodeModifyTable.c
@@ -58,6 +58,13 @@
#include "utils/rel.h"
+static void ExecBatchInsert(ModifyTableState *mtstate,
+ ResultRelInfo *resultRelInfo,
+ TupleTableSlot **slots,
+ TupleTableSlot **planSlots,
+ int numSlots,
+ EState *estate,
+ bool canSetTag);
static bool ExecOnConflictUpdate(ModifyTableState *mtstate,
ResultRelInfo *resultRelInfo,
ItemPointer conflictTid,
@@ -389,6 +396,7 @@ ExecInsert(ModifyTableState *mtstate,
ModifyTable *node = (ModifyTable *) mtstate->ps.plan;
OnConflictAction onconflict = node->onConflictAction;
PartitionTupleRouting *proute = mtstate->mt_partition_tuple_routing;
+ MemoryContext oldContext;
/*
* If the input result relation is a partitioned table, find the leaf
@@ -441,6 +449,72 @@ ExecInsert(ModifyTableState *mtstate,
ExecComputeStoredGenerated(resultRelInfo, estate, slot,
CMD_INSERT);
+ /*
+ * Determine if the FDW supports batch insert and determine the batch
+ * size (a FDW may support batching, but it may be disabled for the
+ * server/table). Do this only once, at the beginning - we don't want
+ * the batch size to change during execution.
+ */
+ if (resultRelInfo->ri_FdwRoutine->GetForeignModifyBatchSize &&
+ resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert &&
+ resultRelInfo->ri_BatchSize == 0)
+ resultRelInfo->ri_BatchSize =
+ resultRelInfo->ri_FdwRoutine->GetForeignModifyBatchSize(resultRelInfo);
+
+ if (resultRelInfo->ri_BatchSize == 0)
+ resultRelInfo->ri_BatchSize = 1;
+
+ Assert(resultRelInfo->ri_BatchSize >= 0);
+
+ /*
+ * If the FDW supports batching, and batching is requested, accumulate
+ * rows and insert them in batches. Otherwise use the per-row inserts.
+ */
+ if (resultRelInfo->ri_BatchSize > 1)
+ {
+ /*
+ * If a certain number of tuples have already been accumulated,
+ * or a tuple has come for a different relation than that for
+ * the accumulated tuples, perform the batch insert
+ */
+ if (resultRelInfo->ri_NumSlots == resultRelInfo->ri_BatchSize)
+ {
+ ExecBatchInsert(mtstate, resultRelInfo,
+ resultRelInfo->ri_Slots,
+ resultRelInfo->ri_PlanSlots,
+ resultRelInfo->ri_NumSlots,
+ estate, canSetTag);
+ resultRelInfo->ri_NumSlots = 0;
+ }
+
+ oldContext = MemoryContextSwitchTo(estate->es_query_cxt);
+
+ if (resultRelInfo->ri_Slots == NULL)
+ {
+ resultRelInfo->ri_Slots = palloc(sizeof(TupleTableSlot *) *
+ resultRelInfo->ri_BatchSize);
+ resultRelInfo->ri_PlanSlots = palloc(sizeof(TupleTableSlot *) *
+ resultRelInfo->ri_BatchSize);
+ }
+
+ resultRelInfo->ri_Slots[resultRelInfo->ri_NumSlots] =
+ MakeSingleTupleTableSlot(slot->tts_tupleDescriptor,
+ slot->tts_ops);
+ ExecCopySlot(resultRelInfo->ri_Slots[resultRelInfo->ri_NumSlots],
+ slot);
+ resultRelInfo->ri_PlanSlots[resultRelInfo->ri_NumSlots] =
+ MakeSingleTupleTableSlot(planSlot->tts_tupleDescriptor,
+ planSlot->tts_ops);
+ ExecCopySlot(resultRelInfo->ri_PlanSlots[resultRelInfo->ri_NumSlots],
+ planSlot);
+
+ resultRelInfo->ri_NumSlots++;
+
+ MemoryContextSwitchTo(oldContext);
+
+ return NULL;
+ }
+
/*
* insert into foreign table: let the FDW do it
*/
@@ -698,6 +772,70 @@ ExecInsert(ModifyTableState *mtstate,
return result;
}
+/* ----------------------------------------------------------------
+ * ExecBatchInsert
+ *
+ * Insert multiple tuples in an efficient way.
+ * Currently, this handles inserting into a foreign table without
+ * RETURNING clause.
+ * ----------------------------------------------------------------
+ */
+static void
+ExecBatchInsert(ModifyTableState *mtstate,
+ ResultRelInfo *resultRelInfo,
+ TupleTableSlot **slots,
+ TupleTableSlot **planSlots,
+ int numSlots,
+ EState *estate,
+ bool canSetTag)
+{
+ int i;
+ int numInserted = numSlots;
+ TupleTableSlot *slot = NULL;
+ TupleTableSlot **rslots;
+
+ /*
+ * insert into foreign table: let the FDW do it
+ */
+ rslots = resultRelInfo->ri_FdwRoutine->ExecForeignBatchInsert(estate,
+ resultRelInfo,
+ slots,
+ planSlots,
+ &numInserted);
+
+ for (i = 0; i < numInserted; i++)
+ {
+ slot = rslots[i];
+
+ /*
+ * AFTER ROW Triggers or RETURNING expressions might reference the
+ * tableoid column, so (re-)initialize tts_tableOid before evaluating
+ * them.
+ */
+ slot->tts_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc);
+
+ /* AFTER ROW INSERT Triggers */
+ ExecARInsertTriggers(estate, resultRelInfo, slot, NIL,
+ mtstate->mt_transition_capture);
+
+ /*
+ * Check any WITH CHECK OPTION constraints from parent views. See the
+ * comment in ExecInsert.
+ */
+ if (resultRelInfo->ri_WithCheckOptions != NIL)
+ ExecWithCheckOptions(WCO_VIEW_CHECK, resultRelInfo, slot, estate);
+ }
+
+ if (canSetTag && numInserted > 0)
+ estate->es_processed += numInserted;
+
+ for (i = 0; i < numSlots; i++)
+ {
+ ExecDropSingleTupleTableSlot(slots[i]);
+ ExecDropSingleTupleTableSlot(planSlots[i]);
+ }
+}
+
/* ----------------------------------------------------------------
* ExecDelete
*
@@ -1937,6 +2075,9 @@ ExecModifyTable(PlanState *pstate)
ItemPointerData tuple_ctid;
HeapTupleData oldtupdata;
HeapTuple oldtuple;
+ PartitionTupleRouting *proute = node->mt_partition_tuple_routing;
+ List *relinfos = NIL;
+ ListCell *lc;
CHECK_FOR_INTERRUPTS();
@@ -2152,6 +2293,25 @@ ExecModifyTable(PlanState *pstate)
return slot;
}
+ /*
+ * Insert remaining tuples for batch insert.
+ */
+ if (proute)
+ relinfos = estate->es_tuple_routing_result_relations;
+ else
+ relinfos = estate->es_opened_result_relations;
+
+ foreach(lc, relinfos)
+ {
+ resultRelInfo = lfirst(lc);
+ if (resultRelInfo->ri_NumSlots > 0)
+ ExecBatchInsert(node, resultRelInfo,
+ resultRelInfo->ri_Slots,
+ resultRelInfo->ri_PlanSlots,
+ resultRelInfo->ri_NumSlots,
+ estate, node->canSetTag);
+ }
+
/*
* We're done, but fire AFTER STATEMENT triggers before exiting.
*/
diff --git a/src/backend/nodes/list.c b/src/backend/nodes/list.c
index c4eba6b053..dbf6b30233 100644
--- a/src/backend/nodes/list.c
+++ b/src/backend/nodes/list.c
@@ -277,6 +277,21 @@ list_make4_impl(NodeTag t, ListCell datum1, ListCell datum2,
return list;
}
+List *
+list_make5_impl(NodeTag t, ListCell datum1, ListCell datum2,
+ ListCell datum3, ListCell datum4, ListCell datum5)
+{
+ List *list = new_list(t, 5);
+
+ list->elements[0] = datum1;
+ list->elements[1] = datum2;
+ list->elements[2] = datum3;
+ list->elements[3] = datum4;
+ list->elements[4] = datum5;
+ check_list_invariants(list);
+ return list;
+}
+
/*
* Make room for a new head cell in the given (non-NIL) list.
*
diff --git a/src/include/foreign/fdwapi.h b/src/include/foreign/fdwapi.h
index 2953499fb1..248f78da45 100644
--- a/src/include/foreign/fdwapi.h
+++ b/src/include/foreign/fdwapi.h
@@ -85,6 +85,14 @@ typedef TupleTableSlot *(*ExecForeignInsert_function) (EState *estate,
TupleTableSlot *slot,
TupleTableSlot *planSlot);
+typedef TupleTableSlot **(*ExecForeignBatchInsert_function) (EState *estate,
+ ResultRelInfo *rinfo,
+ TupleTableSlot **slots,
+ TupleTableSlot **planSlots,
+ int *numSlots);
+
+typedef int (*GetForeignModifyBatchSize_function) (ResultRelInfo *rinfo);
+
typedef TupleTableSlot *(*ExecForeignUpdate_function) (EState *estate,
ResultRelInfo *rinfo,
TupleTableSlot *slot,
@@ -209,6 +217,8 @@ typedef struct FdwRoutine
PlanForeignModify_function PlanForeignModify;
BeginForeignModify_function BeginForeignModify;
ExecForeignInsert_function ExecForeignInsert;
+ ExecForeignBatchInsert_function ExecForeignBatchInsert;
+ GetForeignModifyBatchSize_function GetForeignModifyBatchSize;
ExecForeignUpdate_function ExecForeignUpdate;
ExecForeignDelete_function ExecForeignDelete;
EndForeignModify_function EndForeignModify;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 48c3f570fa..d65099c94a 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -446,6 +446,12 @@ typedef struct ResultRelInfo
/* true when modifying foreign table directly */
bool ri_usesFdwDirectModify;
+ /* batch insert stuff */
+ int ri_NumSlots; /* number of slots in the array */
+ int ri_BatchSize; /* max slots inserted in a single batch */
+ TupleTableSlot **ri_Slots; /* input tuples for batch insert */
+ TupleTableSlot **ri_PlanSlots;
+
/* list of WithCheckOption's to be checked */
List *ri_WithCheckOptions;
diff --git a/src/include/nodes/pg_list.h b/src/include/nodes/pg_list.h
index 710dcd37ef..404e03f132 100644
--- a/src/include/nodes/pg_list.h
+++ b/src/include/nodes/pg_list.h
@@ -213,6 +213,10 @@ list_length(const List *l)
#define list_make4(x1,x2,x3,x4) \
list_make4_impl(T_List, list_make_ptr_cell(x1), list_make_ptr_cell(x2), \
list_make_ptr_cell(x3), list_make_ptr_cell(x4))
+#define list_make5(x1,x2,x3,x4,x5) \
+ list_make5_impl(T_List, list_make_ptr_cell(x1), list_make_ptr_cell(x2), \
+ list_make_ptr_cell(x3), list_make_ptr_cell(x4), \
+ list_make_ptr_cell(x5))
#define list_make1_int(x1) \
list_make1_impl(T_IntList, list_make_int_cell(x1))
@@ -224,6 +228,10 @@ list_length(const List *l)
#define list_make4_int(x1,x2,x3,x4) \
list_make4_impl(T_IntList, list_make_int_cell(x1), list_make_int_cell(x2), \
list_make_int_cell(x3), list_make_int_cell(x4))
+#define list_make5_int(x1,x2,x3,x4,x5) \
+ list_make5_impl(T_IntList, list_make_int_cell(x1), list_make_int_cell(x2), \
+ list_make_int_cell(x3), list_make_int_cell(x4), \
+ list_make_int_cell(x5))
#define list_make1_oid(x1) \
list_make1_impl(T_OidList, list_make_oid_cell(x1))
@@ -235,6 +243,10 @@ list_length(const List *l)
#define list_make4_oid(x1,x2,x3,x4) \
list_make4_impl(T_OidList, list_make_oid_cell(x1), list_make_oid_cell(x2), \
list_make_oid_cell(x3), list_make_oid_cell(x4))
+#define list_make5_oid(x1,x2,x3,x4,x5) \
+ list_make5_impl(T_OidList, list_make_oid_cell(x1), list_make_oid_cell(x2), \
+ list_make_oid_cell(x3), list_make_oid_cell(x4), \
+ list_make_oid_cell(x5))
/*
* Locate the n'th cell (counting from 0) of the list.
@@ -520,6 +532,9 @@ extern List *list_make3_impl(NodeTag t, ListCell datum1, ListCell datum2,
ListCell datum3);
extern List *list_make4_impl(NodeTag t, ListCell datum1, ListCell datum2,
ListCell datum3, ListCell datum4);
+extern List *list_make5_impl(NodeTag t, ListCell datum1, ListCell datum2,
+ ListCell datum3, ListCell datum4,
+ ListCell datum5);
extern pg_nodiscard List *lappend(List *list, void *datum);
extern pg_nodiscard List *lappend_int(List *list, int datum);
--
2.26.2
--------------1CC560E9DC56D15767F1DEDE--
^ permalink raw reply [nested|flat] 20+ messages in thread
* [PATCH v2] Track skipped tables during vacuum and analyze
@ 2026-03-24 04:09 Yugo Nagata <nagata@sraoss.co.jp>
0 siblings, 0 replies; 20+ messages in thread
From: Yugo Nagata @ 2026-03-24 04:09 UTC (permalink / raw)
---
src/backend/catalog/system_views.sql | 8 ++
src/backend/commands/vacuum.c | 84 +++++++++++++++-----
src/backend/utils/activity/pgstat_relation.c | 69 ++++++++++++++++
src/backend/utils/adt/pgstatfuncs.c | 24 ++++++
src/include/catalog/pg_proc.dat | 32 ++++++++
src/include/pgstat.h | 11 +++
src/test/regress/expected/rules.out | 24 ++++++
7 files changed, 230 insertions(+), 22 deletions(-)
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index e54018004db..2ca02a0354c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -736,13 +736,21 @@ CREATE VIEW pg_stat_all_tables AS
pg_stat_get_mod_since_analyze(C.oid) AS n_mod_since_analyze,
pg_stat_get_ins_since_vacuum(C.oid) AS n_ins_since_vacuum,
pg_stat_get_last_vacuum_time(C.oid) as last_vacuum,
+ pg_stat_get_last_skipped_vacuum_time(C.oid) as last_skipped_vacuum,
pg_stat_get_last_autovacuum_time(C.oid) as last_autovacuum,
+ pg_stat_get_last_skipped_autovacuum_time(C.oid) as last_skipped_autovacuum,
pg_stat_get_last_analyze_time(C.oid) as last_analyze,
+ pg_stat_get_last_skipped_analyze_time(C.oid) as last_skipped_analyze,
pg_stat_get_last_autoanalyze_time(C.oid) as last_autoanalyze,
+ pg_stat_get_last_skipped_autoanalyze_time(C.oid) as last_skipped_autoanalyze,
pg_stat_get_vacuum_count(C.oid) AS vacuum_count,
+ pg_stat_get_skipped_vacuum_count(C.oid) AS skipped_vacuum_count,
pg_stat_get_autovacuum_count(C.oid) AS autovacuum_count,
+ pg_stat_get_skipped_autovacuum_count(C.oid) AS skipped_autovacuum_count,
pg_stat_get_analyze_count(C.oid) AS analyze_count,
+ pg_stat_get_skipped_analyze_count(C.oid) AS skipped_analyze_count,
pg_stat_get_autoanalyze_count(C.oid) AS autoanalyze_count,
+ pg_stat_get_skipped_autoanalyze_count(C.oid) AS skipped_autoanalyze_count,
pg_stat_get_total_vacuum_time(C.oid) AS total_vacuum_time,
pg_stat_get_total_autovacuum_time(C.oid) AS total_autovacuum_time,
pg_stat_get_total_analyze_time(C.oid) AS total_analyze_time,
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index bce3a2daa24..8125dfac137 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -794,6 +794,11 @@ vacuum_open_relation(Oid relid, RangeVar *relation, bits32 options,
{
rel = NULL;
rel_lock = false;
+
+ pgstat_report_skipped_vacuum_analyze(relid,
+ (options & VACOPT_VACUUM) != 0,
+ (options & VACOPT_ANALYZE) != 0,
+ AmAutoVacuumWorkerProcess());
}
/* if relation is opened, leave */
@@ -904,7 +909,6 @@ expand_vacuum_rel(VacuumRelation *vrel, MemoryContext vac_context,
Form_pg_class classForm;
bool include_children;
bool is_partitioned_table;
- int rvr_opts;
/*
* Since autovacuum workers supply OIDs when calling vacuum(), no
@@ -917,29 +921,65 @@ expand_vacuum_rel(VacuumRelation *vrel, MemoryContext vac_context,
* below, as well as find_all_inheritors's expectation that the caller
* holds some lock on the starting relation.
*/
- rvr_opts = (options & VACOPT_SKIP_LOCKED) ? RVR_SKIP_LOCKED : 0;
- relid = RangeVarGetRelidExtended(vrel->relation,
- AccessShareLock,
- rvr_opts,
- NULL, NULL);
-
- /*
- * If the lock is unavailable, emit the same log statement that
- * vacuum_rel() and analyze_rel() would.
- */
- if (!OidIsValid(relid))
+ if (!(options & VACOPT_SKIP_LOCKED))
{
- if (options & VACOPT_VACUUM)
- ereport(WARNING,
- (errcode(ERRCODE_LOCK_NOT_AVAILABLE),
- errmsg("skipping vacuum of \"%s\" --- lock not available",
- vrel->relation->relname)));
- else
- ereport(WARNING,
- (errcode(ERRCODE_LOCK_NOT_AVAILABLE),
- errmsg("skipping analyze of \"%s\" --- lock not available",
+ relid = RangeVarGetRelidExtended(vrel->relation,
+ AccessShareLock,
+ 0, NULL, NULL);
+ if (!OidIsValid(relid))
+ return vacrels;
+ }
+ else
+ {
+ /* Get relid for reporting before taking a lock */
+ relid = RangeVarGetRelid(vrel->relation, NoLock, false);
+
+ if (!ConditionalLockRelationOid(relid, AccessShareLock))
+ {
+ /*
+ * If the lock is unavailable, emit the same log statement that
+ * vacuum_rel() and analyze_rel() would.
+ */
+ if (options & VACOPT_VACUUM)
+ ereport(WARNING,
+ (errcode(ERRCODE_LOCK_NOT_AVAILABLE),
+ errmsg("skipping vacuum of \"%s\" --- lock not available",
vrel->relation->relname)));
- return vacrels;
+ else
+ ereport(WARNING,
+ (errcode(ERRCODE_LOCK_NOT_AVAILABLE),
+ errmsg("skipping analyze of \"%s\" --- lock not available",
+ vrel->relation->relname)));
+
+ pgstat_report_skipped_vacuum_analyze(relid,
+ (options & VACOPT_VACUUM) != 0,
+ (options & VACOPT_ANALYZE) != 0,
+ false);
+ return vacrels;
+ }
+
+ /*
+ * Now that we have the lock, probe to see if the relation really
+ * exists or not.
+ */
+ if (!SearchSysCacheExists1(RELOID, ObjectIdGetDatum(relid)))
+ {
+ if (options & VACOPT_VACUUM)
+ ereport(WARNING,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("skipping vacuum of \"%s\" --- relation no longer exists",
+ vrel->relation->relname)));
+ else
+ ereport(WARNING,
+ (errcode(ERRCODE_UNDEFINED_TABLE),
+ errmsg("skipping analyze of \"%s\" --- relation no longer exists",
+ vrel->relation->relname)));
+
+ /* Release useless lock */
+ UnlockRelationOid(relid, AccessShareLock);
+
+ return vacrels;
+ }
}
/*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index bc8c43b96aa..51d0f280ebf 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -17,6 +17,7 @@
#include "postgres.h"
+#include "access/htup_details.h"
#include "access/twophase_rmgr.h"
#include "access/xact.h"
#include "catalog/catalog.h"
@@ -366,6 +367,74 @@ pgstat_report_analyze(Relation rel,
(void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
}
+/*
+ * Report that the table was skipped during vacuum or/and analyze.
+ */
+void
+pgstat_report_skipped_vacuum_analyze(Oid relid, bool vacuum, bool analyze,
+ bool autovacuum)
+{
+ PgStat_EntryRef *entry_ref;
+ PgStatShared_Relation *shtabentry;
+ PgStat_StatTabEntry *tabentry;
+ TimestampTz ts;
+ HeapTuple classTup;
+ bool isshared;
+
+ if (!pgstat_track_counts || !(vacuum || analyze))
+ return;
+
+ classTup = SearchSysCache1(RELOID, ObjectIdGetDatum(relid));
+ if (!HeapTupleIsValid(classTup))
+ return; /* somebody deleted the rel, forget it */
+ isshared = ((Form_pg_class) GETSTRUCT(classTup))->relisshared;
+ ReleaseSysCache(classTup);
+
+ /* Store the data in the table's hash table entry. */
+ ts = GetCurrentTimestamp();
+
+ /* block acquiring lock for the same reason as pgstat_report_autovac() */
+ entry_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
+ isshared ? InvalidOid : MyDatabaseId,
+ relid, false);
+
+ shtabentry = (PgStatShared_Relation *) entry_ref->shared_stats;
+ tabentry = &shtabentry->stats;
+
+ if (autovacuum)
+ {
+ if (vacuum)
+ {
+ tabentry->last_skipped_autovacuum_time = ts;
+ tabentry->skipped_autovacuum_count++;
+ }
+ if (analyze)
+ {
+ tabentry->last_skipped_autoanalyze_time = ts;
+ tabentry->skipped_autoanalyze_count++;
+ }
+ }
+ else
+ {
+ if (vacuum)
+ {
+ tabentry->last_skipped_vacuum_time = ts;
+ tabentry->skipped_vacuum_count++;
+ }
+ if (analyze)
+ {
+ tabentry->last_skipped_analyze_time = ts;
+ tabentry->skipped_analyze_count++;
+ }
+ }
+
+ pgstat_unlock_entry(entry_ref);
+
+ /* see pgstat_report_vacuum() */
+ pgstat_flush_io(false);
+ (void) pgstat_flush_backend(false, PGSTAT_BACKEND_FLUSH_IO);
+}
+
/*
* count a tuple insertion of n tuples
*/
diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c
index 9185a8e6b83..18aaa3996e1 100644
--- a/src/backend/utils/adt/pgstatfuncs.c
+++ b/src/backend/utils/adt/pgstatfuncs.c
@@ -84,6 +84,18 @@ PG_STAT_GET_RELENTRY_INT64(mod_since_analyze)
/* pg_stat_get_numscans */
PG_STAT_GET_RELENTRY_INT64(numscans)
+/* pg_stat_get_skipped_analyze_count */
+PG_STAT_GET_RELENTRY_INT64(skipped_analyze_count)
+
+/* pg_stat_get_skipped_autoanalyze_count */
+PG_STAT_GET_RELENTRY_INT64(skipped_autoanalyze_count)
+
+/* pg_stat_get_skipped_autovacuum_count */
+PG_STAT_GET_RELENTRY_INT64(skipped_autovacuum_count)
+
+/* pg_stat_get_skipped_vacuum_count */
+PG_STAT_GET_RELENTRY_INT64(skipped_vacuum_count)
+
/* pg_stat_get_tuples_deleted */
PG_STAT_GET_RELENTRY_INT64(tuples_deleted)
@@ -170,6 +182,18 @@ PG_STAT_GET_RELENTRY_TIMESTAMPTZ(last_vacuum_time)
/* pg_stat_get_lastscan */
PG_STAT_GET_RELENTRY_TIMESTAMPTZ(lastscan)
+/* pg_stat_get_last_skipped_analyze_time */
+PG_STAT_GET_RELENTRY_TIMESTAMPTZ(last_skipped_analyze_time)
+
+/* pg_stat_get_last_skipped_autoanalyze_time */
+PG_STAT_GET_RELENTRY_TIMESTAMPTZ(last_skipped_autoanalyze_time)
+
+/* pg_stat_get_last_skipped_autovacuum_time */
+PG_STAT_GET_RELENTRY_TIMESTAMPTZ(last_skipped_autovacuum_time)
+
+/* pg_stat_get_last_skipped_vacuum_time */
+PG_STAT_GET_RELENTRY_TIMESTAMPTZ(last_skipped_vacuum_time)
+
/* pg_stat_get_stat_reset_time */
PG_STAT_GET_RELENTRY_TIMESTAMPTZ(stat_reset_time)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 0118e970dda..ccfe3d9e02a 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -5667,6 +5667,38 @@
proname => 'pg_stat_get_total_autoanalyze_time', provolatile => 's',
proparallel => 'r', prorettype => 'float8', proargtypes => 'oid',
prosrc => 'pg_stat_get_total_autoanalyze_time' },
+{ oid => '8142', descr => 'statistics: last skipped vacuum time for a table',
+ proname => 'pg_stat_get_last_skipped_vacuum_time', provolatile => 's',
+ proparallel => 'r', prorettype => 'timestamptz', proargtypes => 'oid',
+ prosrc => 'pg_stat_get_last_skipped_vacuum_time' },
+{ oid => '8143', descr => 'statistics: last skipped auto vacuum time for a table',
+ proname => 'pg_stat_get_last_skipped_autovacuum_time', provolatile => 's',
+ proparallel => 'r', prorettype => 'timestamptz', proargtypes => 'oid',
+ prosrc => 'pg_stat_get_last_skipped_autovacuum_time' },
+{ oid => '8144', descr => 'statistics: last skipped analyze time for a table',
+ proname => 'pg_stat_get_last_skipped_analyze_time', provolatile => 's',
+ proparallel => 'r', prorettype => 'timestamptz', proargtypes => 'oid',
+ prosrc => 'pg_stat_get_last_skipped_analyze_time' },
+{ oid => '8145', descr => 'statistics: last skipped auto analyze time for a table',
+ proname => 'pg_stat_get_last_skipped_autoanalyze_time', provolatile => 's',
+ proparallel => 'r', prorettype => 'timestamptz', proargtypes => 'oid',
+ prosrc => 'pg_stat_get_last_skipped_autoanalyze_time' },
+{ oid => '8146', descr => 'statistics: number of skipped vacuum for a table',
+ proname => 'pg_stat_get_skipped_vacuum_count', provolatile => 's',
+ proparallel => 'r', prorettype => 'int8', proargtypes => 'oid',
+ prosrc => 'pg_stat_get_skipped_vacuum_count' },
+{ oid => '8147', descr => 'statistics: number of skipped auto vacuum for a table',
+ proname => 'pg_stat_get_skipped_autovacuum_count', provolatile => 's',
+ proparallel => 'r', prorettype => 'int8', proargtypes => 'oid',
+ prosrc => 'pg_stat_get_skipped_autovacuum_count' },
+{ oid => '8148', descr => 'statistics: number of skipped analyzes for a table',
+ proname => 'pg_stat_get_skipped_analyze_count', provolatile => 's',
+ proparallel => 'r', prorettype => 'int8', proargtypes => 'oid',
+ prosrc => 'pg_stat_get_skipped_analyze_count' },
+{ oid => '8149', descr => 'statistics: number of skipped auto analyzes for a table',
+ proname => 'pg_stat_get_skipped_autoanalyze_count', provolatile => 's',
+ proparallel => 'r', prorettype => 'int8', proargtypes => 'oid',
+ prosrc => 'pg_stat_get_skipped_autoanalyze_count' },
{ oid => '1936', descr => 'statistics: currently active backend IDs',
proname => 'pg_stat_get_backend_idset', prorows => '100', proretset => 't',
provolatile => 's', proparallel => 'r', prorettype => 'int4',
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 8e3549c3752..d7f19d91783 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -479,6 +479,15 @@ typedef struct PgStat_StatTabEntry
TimestampTz last_autoanalyze_time; /* autovacuum initiated */
PgStat_Counter autoanalyze_count;
+ TimestampTz last_skipped_vacuum_time; /* user initiated vacuum */
+ PgStat_Counter skipped_vacuum_count;
+ TimestampTz last_skipped_autovacuum_time; /* autovacuum initiated */
+ PgStat_Counter skipped_autovacuum_count;
+ TimestampTz last_skipped_analyze_time; /* user initiated */
+ PgStat_Counter skipped_analyze_count;
+ TimestampTz last_skipped_autoanalyze_time; /* autovacuum initiated */
+ PgStat_Counter skipped_autoanalyze_count;
+
PgStat_Counter total_vacuum_time; /* times in milliseconds */
PgStat_Counter total_autovacuum_time;
PgStat_Counter total_analyze_time;
@@ -706,6 +715,8 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_report_skipped_vacuum_analyze(Oid relid, bool vacuum, bool analyze,
+ bool autovacuum);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 2b3cf6d8569..4ccfc43e903 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1835,13 +1835,21 @@ pg_stat_all_tables| SELECT c.oid AS relid,
pg_stat_get_mod_since_analyze(c.oid) AS n_mod_since_analyze,
pg_stat_get_ins_since_vacuum(c.oid) AS n_ins_since_vacuum,
pg_stat_get_last_vacuum_time(c.oid) AS last_vacuum,
+ pg_stat_get_last_skipped_vacuum_time(c.oid) AS last_skipped_vacuum,
pg_stat_get_last_autovacuum_time(c.oid) AS last_autovacuum,
+ pg_stat_get_last_skipped_autovacuum_time(c.oid) AS last_skipped_autovacuum,
pg_stat_get_last_analyze_time(c.oid) AS last_analyze,
+ pg_stat_get_last_skipped_analyze_time(c.oid) AS last_skipped_analyze,
pg_stat_get_last_autoanalyze_time(c.oid) AS last_autoanalyze,
+ pg_stat_get_last_skipped_autoanalyze_time(c.oid) AS last_skipped_autoanalyze,
pg_stat_get_vacuum_count(c.oid) AS vacuum_count,
+ pg_stat_get_skipped_vacuum_count(c.oid) AS skipped_vacuum_count,
pg_stat_get_autovacuum_count(c.oid) AS autovacuum_count,
+ pg_stat_get_skipped_autovacuum_count(c.oid) AS skipped_autovacuum_count,
pg_stat_get_analyze_count(c.oid) AS analyze_count,
+ pg_stat_get_skipped_analyze_count(c.oid) AS skipped_analyze_count,
pg_stat_get_autoanalyze_count(c.oid) AS autoanalyze_count,
+ pg_stat_get_skipped_autoanalyze_count(c.oid) AS skipped_autoanalyze_count,
pg_stat_get_total_vacuum_time(c.oid) AS total_vacuum_time,
pg_stat_get_total_autovacuum_time(c.oid) AS total_autovacuum_time,
pg_stat_get_total_analyze_time(c.oid) AS total_analyze_time,
@@ -2293,13 +2301,21 @@ pg_stat_sys_tables| SELECT relid,
n_mod_since_analyze,
n_ins_since_vacuum,
last_vacuum,
+ last_skipped_vacuum,
last_autovacuum,
+ last_skipped_autovacuum,
last_analyze,
+ last_skipped_analyze,
last_autoanalyze,
+ last_skipped_autoanalyze,
vacuum_count,
+ skipped_vacuum_count,
autovacuum_count,
+ skipped_autovacuum_count,
analyze_count,
+ skipped_analyze_count,
autoanalyze_count,
+ skipped_autoanalyze_count,
total_vacuum_time,
total_autovacuum_time,
total_analyze_time,
@@ -2348,13 +2364,21 @@ pg_stat_user_tables| SELECT relid,
n_mod_since_analyze,
n_ins_since_vacuum,
last_vacuum,
+ last_skipped_vacuum,
last_autovacuum,
+ last_skipped_autovacuum,
last_analyze,
+ last_skipped_analyze,
last_autoanalyze,
+ last_skipped_autoanalyze,
vacuum_count,
+ skipped_vacuum_count,
autovacuum_count,
+ skipped_autovacuum_count,
analyze_count,
+ skipped_analyze_count,
autoanalyze_count,
+ skipped_autoanalyze_count,
total_vacuum_time,
total_autovacuum_time,
total_analyze_time,
--
2.43.0
--Multipart=_Thu__26_Mar_2026_19_22_03_+0900_+1+LQ8ruMpR+Xuy_--
^ permalink raw reply [nested|flat] 20+ messages in thread
end of thread, other threads:[~2026-03-24 04:09 UTC | newest]
Thread overview: 20+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2000-01-07 08:17 please help Shanthala Rao <S.Rao@toa.toasia.co.jp>
2002-10-21 13:42 Please help Olivier PRENANT <ohp@pyrenet.fr>
2002-10-21 15:02 ` Re: Please help Tom Lane <tgl@sss.pgh.pa.us>
2002-10-21 15:22 ` Re: Please help Olivier PRENANT <ohp@pyrenet.fr>
2002-10-21 15:35 ` Re: Please help Tom Lane <tgl@sss.pgh.pa.us>
2002-10-21 16:23 ` Re: Please help Olivier PRENANT <ohp@pyrenet.fr>
2002-10-21 16:52 ` Re: Please help Tom Lane <tgl@sss.pgh.pa.us>
2002-10-21 17:26 ` Re: Please help Olivier PRENANT <ohp@pyrenet.fr>
2002-10-21 17:34 ` Re: Please help Larry Rosenman <ler@lerctr.org>
2002-10-21 17:39 ` Re: Please help Olivier PRENANT <ohp@pyrenet.fr>
2002-10-21 17:57 ` Re: Please help Larry Rosenman <ler@lerctr.org>
2002-10-21 17:59 ` Re: Please help Larry Rosenman <ler@lerctr.org>
2002-10-21 15:16 ` Re: Please help Ian Barwick <barwick@gmx.net>
2003-10-30 16:33 Please help ohp@pyrenet.fr
2003-10-30 22:25 ` Re: Please help Tom Lane <tgl@sss.pgh.pa.us>
2003-10-30 22:59 ` Re: Please help ohp@pyrenet.fr
2006-06-13 16:08 Please help! Brandon E Hofmann <Brandon_E_Hofmann@notes.ntrs.com>
2006-06-16 03:22 ` Re: Please help! Tom Lane <tgl@sss.pgh.pa.us>
2021-01-14 14:59 [PATCH] v9 Tomas Vondra <tomas@2ndquadrant.com>
2026-03-24 04:09 [PATCH v2] Track skipped tables during vacuum and analyze Yugo Nagata <nagata@sraoss.co.jp>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox