public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v37 5/6] Doc part of shared-memory based stats collector. 4+ messages / 3 participants [nested] [flat]
* [PATCH v37 5/6] Doc part of shared-memory based stats collector. @ 2020-03-19 06:11 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Kyotaro Horiguchi @ 2020-03-19 06:11 UTC (permalink / raw) --- doc/src/sgml/catalogs.sgml | 6 +- doc/src/sgml/config.sgml | 34 ++++---- doc/src/sgml/high-availability.sgml | 13 +-- doc/src/sgml/monitoring.sgml | 127 +++++++++++++--------------- doc/src/sgml/ref/pg_dump.sgml | 9 +- 5 files changed, 90 insertions(+), 99 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index de9bacd34f..69db5afc94 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -9209,9 +9209,9 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l <para> <xref linkend="view-table"/> lists the system views described here. More detailed documentation of each view follows below. - There are some additional views that provide access to the results of - the statistics collector; they are described in <xref - linkend="monitoring-stats-views-table"/>. + There are some additional views that provide access to the activity + statistics; they are described in + <xref linkend="monitoring-stats-views-table"/>. </para> <para> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 8eabf93834..cc5dc1173f 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -7192,11 +7192,11 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; <title>Run-time Statistics</title> <sect2 id="runtime-config-statistics-collector"> - <title>Query and Index Statistics Collector</title> + <title>Query and Index Activity Statistics</title> <para> - These parameters control server-wide statistics collection features. - When statistics collection is enabled, the data that is produced can be + These parameters control server-wide activity statistics features. + When activity statistics is enabled, the data that is produced can be accessed via the <structname>pg_stat</structname> and <structname>pg_statio</structname> family of system views. Refer to <xref linkend="monitoring"/> for more information. @@ -7212,14 +7212,13 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; </term> <listitem> <para> - Enables the collection of information on the currently - executing command of each session, along with the time when - that command began execution. This parameter is on by - default. Note that even when enabled, this information is not - visible to all users, only to superusers and the user owning - the session being reported on, so it should not represent a - security risk. - Only superusers can change this setting. + Enables activity tracking on the currently executing command of + each session, along with the time when that command began + execution. This parameter is on by default. Note that even when + enabled, this information is not visible to all users, only to + superusers and the user owning the session being reported on, so it + should not represent a security risk. Only superusers can change this + setting. </para> </listitem> </varlistentry> @@ -7250,9 +7249,9 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; </term> <listitem> <para> - Enables collection of statistics on database activity. + Enables tracking of database activity. This parameter is on by default, because the autovacuum - daemon needs the collected information. + daemon needs the activity information. Only superusers can change this setting. </para> </listitem> @@ -8313,7 +8312,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; <listitem> <para> Specifies the fraction of the total number of heap tuples counted in - the previous statistics collection that can be inserted without + the previously collected statistics that can be inserted without incurring an index scan at the <command>VACUUM</command> cleanup stage. This setting currently applies to B-tree indexes only. </para> @@ -8325,9 +8324,10 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; the index contains deleted pages that can be recycled during cleanup. Index statistics are considered to be stale if the number of newly inserted tuples exceeds the <varname>vacuum_cleanup_index_scale_factor</varname> - fraction of the total number of heap tuples detected by the previous - statistics collection. The total number of heap tuples is stored in - the index meta-page. Note that the meta-page does not include this data + + fraction of the total number of heap tuples in the previously + collected statistics. The total number of heap tuples is stored in the + index meta-page. Note that the meta-page does not include this data until <command>VACUUM</command> finds no dead tuples, so B-tree index scan at the cleanup stage can only be skipped if the second and subsequent <command>VACUUM</command> cycles detect no dead tuples. diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml index beb309e668..45ec7ce68f 100644 --- a/doc/src/sgml/high-availability.sgml +++ b/doc/src/sgml/high-availability.sgml @@ -2366,12 +2366,13 @@ LOG: database system is ready to accept read only connections </para> <para> - The statistics collector is active during recovery. All scans, reads, blocks, - index usage, etc., will be recorded normally on the standby. Replayed - actions will not duplicate their effects on primary, so replaying an - insert will not increment the Inserts column of pg_stat_user_tables. - The stats file is deleted at the start of recovery, so stats from primary - and standby will differ; this is considered a feature, not a bug. + The activity statistics is collected during recovery. All scans, reads, + blocks, index usage, etc., will be recorded normally on the + standby. Replayed actions will not duplicate their effects on primary, so + replaying an insert will not increment the Inserts column of + pg_stat_user_tables. The activity statistics is reset at the start of + recovery, so stats from primary and standby will differ; this is + considered a feature, not a bug. </para> <para> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 4e0193a967..7a04d58a1a 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -22,7 +22,7 @@ <para> Several tools are available for monitoring database activity and analyzing performance. Most of this chapter is devoted to describing - <productname>PostgreSQL</productname>'s statistics collector, + <productname>PostgreSQL</productname>'s activity statistics, but one should not neglect regular Unix monitoring programs such as <command>ps</command>, <command>top</command>, <command>iostat</command>, and <command>vmstat</command>. Also, once one has identified a @@ -53,7 +53,6 @@ postgres 15554 0.0 0.0 57536 1184 ? Ss 18:02 0:00 postgres: back postgres 15555 0.0 0.0 57536 916 ? Ss 18:02 0:00 postgres: checkpointer postgres 15556 0.0 0.0 57536 916 ? Ss 18:02 0:00 postgres: walwriter postgres 15557 0.0 0.0 58504 2244 ? Ss 18:02 0:00 postgres: autovacuum launcher -postgres 15558 0.0 0.0 17512 1068 ? Ss 18:02 0:00 postgres: stats collector postgres 15582 0.0 0.0 58772 3080 ? Ss 18:04 0:00 postgres: joe runbug 127.0.0.1 idle postgres 15606 0.0 0.0 58772 3052 ? Ss 18:07 0:00 postgres: tgl regression [local] SELECT waiting postgres 15610 0.0 0.0 58772 3056 ? Ss 18:07 0:00 postgres: tgl regression [local] idle in transaction @@ -65,9 +64,8 @@ postgres 15610 0.0 0.0 58772 3056 ? Ss 18:07 0:00 postgres: tgl primary server process. The command arguments shown for it are the same ones used when it was launched. The next five processes are background worker processes automatically launched by the - primary process. (The <quote>stats collector</quote> process will not be present - if you have set the system not to start the statistics collector; likewise - the <quote>autovacuum launcher</quote> process can be disabled.) + primary process. (The <quote>autovacuum launcher</quote> process will not + be present if you have set the system not to start it.) Each of the remaining processes is a server process handling one client connection. Each such process sets its command line display in the form @@ -130,20 +128,21 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </sect1> <sect1 id="monitoring-stats"> - <title>The Statistics Collector</title> + <title>The Activity Statistics</title> <indexterm zone="monitoring-stats"> <primary>statistics</primary> </indexterm> <para> - <productname>PostgreSQL</productname>'s <firstterm>statistics collector</firstterm> - is a subsystem that supports collection and reporting of information about - server activity. Presently, the collector can count accesses to tables - and indexes in both disk-block and individual-row terms. It also tracks - the total number of rows in each table, and information about vacuum and - analyze actions for each table. It can also count calls to user-defined - functions and the total time spent in each one. + <productname>PostgreSQL</productname>'s <firstterm>activity + statistics</firstterm> is a subsystem that supports tracking and reporting + of information about server activity. Presently, the activity statistics + tracks the count of accesses to tables and indexes in both disk-block and + individual-row terms. It also tracks the total number of rows in each + table, and information about vacuum and analyze actions for each table. It + can also track calls to user-defined functions and the total time spent in + each one. </para> <para> @@ -151,15 +150,15 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser information about exactly what is going on in the system right now, such as the exact command currently being executed by other server processes, and which other connections exist in the system. This facility is independent - of the collector process. + of the activity statistics. </para> <sect2 id="monitoring-stats-setup"> - <title>Statistics Collection Configuration</title> + <title>Activity Statistics Configuration</title> <para> - Since collection of statistics adds some overhead to query execution, - the system can be configured to collect or not collect information. + Since tracking for the activity statistics adds some overhead to query + execution, the system can be configured to track or not track activity. This is controlled by configuration parameters that are normally set in <filename>postgresql.conf</filename>. (See <xref linkend="runtime-config"/> for details about setting configuration parameters.) @@ -172,7 +171,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser <para> The parameter <xref linkend="guc-track-counts"/> controls whether - statistics are collected about table and index accesses. + to track activity about table and index accesses. </para> <para> @@ -196,18 +195,11 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </para> <para> - The statistics collector transmits the collected information to other - <productname>PostgreSQL</productname> processes through temporary files. - These files are stored in the directory named by the - <xref linkend="guc-stats-temp-directory"/> parameter, - <filename>pg_stat_tmp</filename> by default. - For better performance, <varname>stats_temp_directory</varname> can be - pointed at a RAM-based file system, decreasing physical I/O requirements. - When the server shuts down cleanly, a permanent copy of the statistics - data is stored in the <filename>pg_stat</filename> subdirectory, so that - statistics can be retained across server restarts. When recovery is - performed at server start (e.g., after immediate shutdown, server crash, - and point-in-time recovery), all statistics counters are reset. + down cleanly, a permanent copy of the statistics data is stored in + the <filename>pg_stat</filename> subdirectory, so that statistics can be + retained across server restarts. When recovery is performed at server + start (e.g. after immediate shutdown, server crash, and point-in-time + recovery), all statistics counters are reset. </para> </sect2> @@ -220,48 +212,46 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser linkend="monitoring-stats-dynamic-views-table"/>, are available to show the current state of the system. There are also several other views, listed in <xref - linkend="monitoring-stats-views-table"/>, available to show the results - of statistics collection. Alternatively, one can - build custom views using the underlying statistics functions, as discussed - in <xref linkend="monitoring-stats-functions"/>. + linkend="monitoring-stats-views-table"/>, available to show the activity + statistics. Alternatively, one can build custom views using the underlying + statistics functions, as discussed in + <xref linkend="monitoring-stats-functions"/>. </para> <para> - When using the statistics to monitor collected data, it is important - to realize that the information does not update instantaneously. - Each individual server process transmits new statistical counts to - the collector just before going idle; so a query or transaction still in - progress does not affect the displayed totals. Also, the collector itself - emits a new report at most once per <varname>PGSTAT_STAT_INTERVAL</varname> - milliseconds (500 ms unless altered while building the server). So the - displayed information lags behind actual activity. However, current-query - information collected by <varname>track_activities</varname> is - always up-to-date. + When using the activity statistics, it is important to realize that the + information does not update instantaneously. Each individual server writes + out new statistical counts just before going idle, not frequent than once + per <varname>PGSTAT_STAT_INTERVAL</varname> milliseconds (1 second unless + altered while building the server); so a query or transaction still in + progress does not affect the displayed totals. However, current-query + information tracked by <varname>track_activities</varname> is always + up-to-date. </para> <para> Another important point is that when a server process is asked to display - any of these statistics, it first fetches the most recent report emitted by - the collector process and then continues to use this snapshot for all - statistical views and functions until the end of its current transaction. - So the statistics will show static information as long as you continue the - current transaction. Similarly, information about the current queries of - all sessions is collected when any such information is first requested - within a transaction, and the same information will be displayed throughout - the transaction. - This is a feature, not a bug, because it allows you to perform several - queries on the statistics and correlate the results without worrying that - the numbers are changing underneath you. But if you want to see new - results with each query, be sure to do the queries outside any transaction - block. Alternatively, you can invoke + any of these statistics, it first reads the current statistics and then + continues to use this snapshot for all statistical views and functions + until the end of its current transaction. So the statistics will show + static information as long as you continue the current transaction. + Similarly, information about the current queries of all sessions is tracked + when any such information is first requested within a transaction, and the + same information will be displayed throughout the transaction. This is a + feature, not a bug, because it allows you to perform several queries on the + statistics and correlate the results without worrying that the numbers are + changing underneath you. But if you want to see new results with each + query, be sure to do the queries outside any transaction block. + Alternatively, you can invoke <function>pg_stat_clear_snapshot</function>(), which will discard the current transaction's statistics snapshot (if any). The next use of statistical information will cause a new snapshot to be fetched. </para> - + <para> - A transaction can also see its own statistics (as yet untransmitted to the - collector) in the views <structname>pg_stat_xact_all_tables</structname>, + A transaction can also see its own statistics (as yet unwritten to the + server-wide activity statistics) in the + views <structname>pg_stat_xact_all_tables</structname>, <structname>pg_stat_xact_sys_tables</structname>, <structname>pg_stat_xact_user_tables</structname>, and <structname>pg_stat_xact_user_functions</structname>. These numbers do not act as @@ -620,7 +610,7 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser kernel's I/O cache, and might therefore still be fetched without requiring a physical read. Users interested in obtaining more detailed information on <productname>PostgreSQL</productname> I/O behavior are - advised to use the <productname>PostgreSQL</productname> statistics collector + advised to use the <productname>PostgreSQL</productname> activity statistics in combination with operating system utilities that allow insight into the kernel's handling of I/O. </para> @@ -1057,10 +1047,6 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser <entry><literal>LogicalLauncherMain</literal></entry> <entry>Waiting in main loop of logical replication launcher process.</entry> </row> - <row> - <entry><literal>PgStatMain</literal></entry> - <entry>Waiting in main loop of statistics collector process.</entry> - </row> <row> <entry><literal>RecoveryWalStream</literal></entry> <entry>Waiting in main loop of startup process for WAL to arrive, during @@ -1815,6 +1801,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser </thead> <tbody> + <row> + <entry><literal>ActivityStatistics</literal></entry> + <entry>Waiting to write out activity statistics to shared memory.</entry> + </row> <row> <entry><literal>AddinShmemInit</literal></entry> <entry>Waiting to manage an extension's space allocation in shared @@ -5738,9 +5728,10 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <entry><literal>performing final cleanup</literal></entry> <entry> <command>VACUUM</command> is performing final cleanup. During this phase, - <command>VACUUM</command> will vacuum the free space map, update statistics - in <literal>pg_class</literal>, and report statistics to the statistics - collector. When this phase is completed, <command>VACUUM</command> will end. + <command>VACUUM</command> will vacuum the free space map, update + statistics in <literal>pg_class</literal>, and system-wide activity + statistics. When this phase is completed, <command>VACUUM</command> + will end. </entry> </row> </tbody> diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml index e09ed0a4c3..71bb24accf 100644 --- a/doc/src/sgml/ref/pg_dump.sgml +++ b/doc/src/sgml/ref/pg_dump.sgml @@ -1290,11 +1290,10 @@ PostgreSQL documentation </para> <para> - The database activity of <application>pg_dump</application> is - normally collected by the statistics collector. If this is - undesirable, you can set parameter <varname>track_counts</varname> - to false via <envar>PGOPTIONS</envar> or the <literal>ALTER - USER</literal> command. + The database activity of <application>pg_dump</application> is normally + collected. If this is undesirable, you can set + parameter <varname>track_counts</varname> to false + via <envar>PGOPTIONS</envar> or the <literal>ALTER USER</literal> command. </para> </refsect1> -- 2.18.4 ----Next_Part(Fri_Sep_25_09_27_26_2020_419)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v37-0006-Remove-the-GUC-stats_temp_directory.patch" ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: [fixed] Trigger test @ 2025-04-03 18:11 Tom Lane <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Tom Lane @ 2025-04-03 18:11 UTC (permalink / raw) To: Dmitrii Bondar <[email protected]>; +Cc: Paul Jungwirth <[email protected]>; PostgreSQL Hackers <[email protected]>; Lilian <[email protected]> Dmitrii Bondar <[email protected]> writes: > [ v6-0001-Triggers-test-fix-with-the-invalid-cache-in-refin.patch ] I spent a little bit of time looking over this patch. My first instinct was "we can't really change the recommended method of installing these triggers" --- but that would only matter if we thought there were actual production users of these triggers, which surely there are not (anymore). The only reason we're carrying refint.c at all is as an example of C-coded triggers. So changing the example seems fine, and you're right that this sort of change is far better done from an AFTER trigger. However ... as an example refint.c is pretty darn awful :-(. I'd never looked hard at it before, and now that I have, I'm rather horrified, particularly by the shoddy quoting practices. None of the identifiers inserted into constructed queries receive quote_identifier() protection, and the insertion of data values around line 500 is beyond awful. So while I think your v6 patch fixes the problem(s) it set out to fix, it still feels a lot like putting lipstick on a pig. I wonder if we'd be better off to nuke refint.c altogether. If we don't, I feel like we're morally obliged to spend more effort trying to make it less of an example of bad practices. Some of the things I think need to be cleaned up: * It's ridiculous that the update-cascade case is inserting data values textually at all. Even if it were quoting them properly, that's expensive and risks round-trip-conversion problems. That should be handled as an additional Param value passed into the statement. * Worse yet, that code doesn't work if used more than once, because the first value that needs to be updated gets baked into the plan that will be re-used later. So the Param approach is really essential even aside from quoting concerns. * String comparisons to detect value equality (around line 400) are not terribly cool either. Proper code would be looking up the default equality operator for the datatypes and applying that. * Some thought needs to be given to table and column names that require quoting. I guess in a way there's an advantage to not quoting the table names that come from trigger arguments: it lets the user get around the module's failure to think about schema-qualified names, by writing 'foo.bar' rather than just 'bar'. But that's not documented. If we did quote everything then we'd really have to go over to providing separate schema and name arguments for each of the other tables. In any case, not quoting the column names has nothing to recommend it. * I'll slide gently past the question of whether this should be expected to react on-the-fly to DDL changes in the tables. SPI will do some of that under the hood, but it can't fix cases where the query string would need to change (eg. table or column renames). So that's a long laundry list and we haven't even dug hard. Is it worth it? If you feel like doing the legwork then I'm willing to support the project, but I really wonder if we shouldn't cut our losses and just remove the module. (I hesitate now to look at the rest of contrib/spi/ :-() regards, tom lane ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: [fixed] Trigger test @ 2025-04-04 05:45 Dmitrii Bondar <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Dmitrii Bondar @ 2025-04-04 05:45 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Paul Jungwirth <[email protected]>; PostgreSQL Hackers <[email protected]>; Lilian <[email protected]> On 04/04/2025 01:11, Tom Lane wrote: > So that's a long laundry list and we haven't even dug hard. > Is it worth it? If you feel like doing the legwork then > I'm willing to support the project, but I really wonder if > we shouldn't cut our losses and just remove the module. > > (I hesitate now to look at the rest of contrib/spi/ :-() You wrote a note that I decided to omit. As I mentioned, the patch does not even fix the cascade update problem—there are still broken cases—because it seems impossible to address it in a gentle way (the code was patched 20 years ago; it's truly legacy). I considered removing it entirely, but that seemed too drastic a solution (and, at the very least, I don't have enough expertise to make that decision). If everything looks acceptable, I would prefer to cut the module. The |check_primary_key| and |check_foreign| functions are clearly unused, are buggy, and no one has reported any obvious problems—so refint.c can be safely removed. Autoinc.c also looks problematic. There are some question. When should we remove the module? Should we mark it as deprecated for now and remove it later? Should we handle it in another thread? Should we apply this patch in that case? Best regards, Dmitrii ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: [fixed] Trigger test @ 2025-04-05 19:54 Tom Lane <[email protected]> parent: Dmitrii Bondar <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Tom Lane @ 2025-04-05 19:54 UTC (permalink / raw) To: Dmitrii Bondar <[email protected]>; +Cc: Paul Jungwirth <[email protected]>; PostgreSQL Hackers <[email protected]>; Lilian <[email protected]> Dmitrii Bondar <[email protected]> writes: > On 04/04/2025 01:11, Tom Lane wrote: >> So that's a long laundry list and we haven't even dug hard. >> Is it worth it? If you feel like doing the legwork then >> I'm willing to support the project, but I really wonder if >> we shouldn't cut our losses and just remove the module. > You wrote a note that I decided to omit. As I mentioned, the patch does > not even fix the cascade update problem—there are still broken > cases—because it seems impossible to address it in a gentle way (the > code was patched 20 years ago; it's truly legacy). I'm not terribly concerned about whether these triggers have perfect foreign-key semantics, since no one (in their right mind) would use them as foreign-key enforcement anyway. What they're good for is as examples of writing checks and updates in C-coded triggers. As such, questions like "are identifiers and data values quoted appropriately" seem far more urgent than whether cascade update works per spec. Even just using a StringInfo rather than a fixed-size char[] variable to build the query in would be an improvement. > I considered removing it entirely, but that seemed too drastic a > solution (and, at the very least, I don't have enough expertise to make > that decision). I'm not that thrilled with giving up on refint.c either. But in its current state, it's a pretty lousy example. Are we willing to put enough effort into making it a more useful code example? regards, tom lane ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2025-04-05 19:54 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-03-19 06:11 [PATCH v37 5/6] Doc part of shared-memory based stats collector. Kyotaro Horiguchi <[email protected]> 2025-04-03 18:11 Re: [fixed] Trigger test Tom Lane <[email protected]> 2025-04-04 05:45 ` Re: [fixed] Trigger test Dmitrii Bondar <[email protected]> 2025-04-05 19:54 ` Re: [fixed] Trigger test Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox