Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1erQjI-0002Dn-4x for pgsql-admin@arkaria.postgresql.org; Thu, 01 Mar 2018 16:06:48 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1erQjH-00088D-7N for pgsql-admin@arkaria.postgresql.org; Thu, 01 Mar 2018 16:06:47 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1erQjG-000883-Up for pgsql-admin@lists.postgresql.org; Thu, 01 Mar 2018 16:06:47 +0000 Received: from mout.perfora.net ([74.208.4.197]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1erQjB-00043x-RJ for pgsql-admin@postgresql.org; Thu, 01 Mar 2018 16:06:46 +0000 Received: from [192.168.1.101] ([98.169.215.140]) by mrelay.perfora.net (mreueus002 [74.208.5.2]) with ESMTPSA (Nemesis) id 0M7Zpp-1eXZrV02S7-00xG57; Thu, 01 Mar 2018 17:06:38 +0100 Message-ID: <5A98250D.5090303@sqlexec.com> Date: Thu, 01 Mar 2018 11:06:37 -0500 From: MichaelDBA User-Agent: Postbox 5.0.22 (Windows/20171208) MIME-Version: 1.0 To: "David G. Johnston" CC: Sohel Tamboli , pgsql-admin@postgresql.org Subject: Re: Postgresql | Vacuum information References: <5A97194B.30709@sqlexec.com> <5A971D59.5050000@sqlexec.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------090302060607040601020003" X-Provags-ID: V03:K0:W7EJsfhtZ9T1LgRL2H8Rd9mQh2nB2NcF/nTqRzjO+z+KJ7Ddi3m GjXA5VIDQJ4uofwCo67lz/gT592xLC85Vukv4/Zc+SvKUAK1x3jcJ2VLqcJra+OOBUyAJ1/ 4NrcE0cfdWBwQ0V/cP9jFh0enEbAqxSQCrkgaTmHtafYtYtUZN/1N5IPxngIDs2tvliH0RP MPgZ47lKNUtA3GAUi7NdA== X-UI-Out-Filterresults: notjunk:1;V01:K0:QWvACxUD228=:yaVuHbUplLhsVS77IRuWxN S+hYQULZ2mFmd0Xzj8X/Acuf13+w1sJhRrYXl3090MqQXz7cOJAaB6QEhxRrALkOq70vHbAfN 0jyIi97nWyHYjvfTdhE4wyXfEi0VfSECTtlX9u4LZXND5beRXqD5K9BWrDm6QQPXd2+uBy6j/ 1XkpWSqDVk18Ejr8vXg8UC5hVCmMwKyGCZpUITa43vX+xrLc3F4IHdpF2w2tWm+HJkqG1A3pv kvZjmMMRFsnQaSzoU4dgwHUolWkAZtIIjO0dGnjTlqGu1Jf5G+eKgYEQpgPPDpxFFPc55ktSy unVs3v04oQ2mmHmCzEu/U7nS6u+g6yQtngvwx3jGI3g7rEImyXx4nRDKrM3uUxoPandUCpCFR MCl9mYuhlHOivagX4uK23clW05NAEdS6+rhP5eP5k8Vw5tqeBrPWBHhx1va2RTfKytDqFjFQn zJFQ30FFkI8pxbpECVHn4LtP3olQfzChY3KIF0QtincI/kZHWIk/BiFS1hpGovJY5ARn7VlR2 TEWfB7Fj4am89Es2YU/ZvBzRN+YZ+ON2ehGS3L1kYPznICEU1g1togSM9uKAKeCw2YzHaGQ3N cONAGUFpR+NyoGc4Kk2kpEpE2nK82r53IoEwT7tMUMaK/WKzaWOov0n1OtjEtRdxjLzJCYpAc U/IuM0GiYLHP+NaOPQXkjNxWd9LzP5gz0OJ5M9QfnW3FsMCGym8ca2v12UI5jG+0tKWQJqLdi ExFGUbDgsfvD/jcfj/IwiJXzUTkuzXF0hxSVMg1Z8a4lLJ/agpE8k5yK0x4= List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------090302060607040601020003 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit oh my my, think David has the correct answer here. Thanks David for your research and correction. So INSERT logic: see if space on current target page, then check FSM, then default to end of relation Regards, Michael Vitale > David G. Johnston > Wednesday, February 28, 2018 4:53 PM > On Wed, Feb 28, 2018 at 2:21 PM, MichaelDBA >wrote: > > wee bit further elaboration... > > MVCC for updates is DELETE followed by INSERT. That is what the > docs are talking about when mentioning "new rows", not the type of > "new row" that is done with an INSERT statement which is ALWAYS > appended to end of table. I actually had to ask about this one in > PostgreSQL slack chat because I wasn't 100% positive that index > clustering order does not influence INSERT location, but > apparently even clustering order does not influence a location for > INSERTs other than the end of the table. > > > ​ My (admittedly inexperienced) reading of the code > (src/backend/access/head/hio.c::RelationGetBufferForTuple) leads me to > conclude that my assumption about the behavior is indeed correct. > Both update-related insertions and isolated insertions result in the > system using the free space map (FSM) to locate existing pages with > free space on them to use for persisting the newly created tuples. > > Specifically: > > /* > * We first try to put the tuple on the same page we last inserted a tuple > * on, as cached in the BulkInsertState or relcache entry. If that > * doesn't work, we ask the Free Space Map to locate a suitable page. > * Since the FSM's info might be out of date, we have to be prepared to > * loop around and retry multiple times. (To insure this isn't an infinite > * loop, we must update the FSM with the correct amount of free space on > * each page that proves not to be suitable.) If the FSM has no record of > * a page with enough free space, we give up and extend the relation. > * > * When use_fsm is false, we either put the tuple onto the existing target > * page or extend the relation. > */ > > and > > if (targetBlock == InvalidBlockNumber && use_fsm) > { > /* > * We have no cached target page, so ask the FSM for an initial > * target. > */ > targetBlock = GetPageWithFreeSpace(relation, len + saveFreeSpace); > [...] > > David J. > > MichaelDBA > Wednesday, February 28, 2018 4:21 PM > wee bit further elaboration... > > MVCC for updates is DELETE followed by INSERT. That is what the docs > are talking about when mentioning "new rows", not the type of "new > row" that is done with an INSERT statement which is ALWAYS appended to > end of table. I actually had to ask about this one in PostgreSQL > slack chat because I wasn't 100% positive that index clustering order > does not influence INSERT location, but apparently even clustering > order does not influence a location for INSERTs other than the end of > the table. > > We all learned something here, lol > > Regards, > Michael Vitale > > > MichaelDBA > Wednesday, February 28, 2018 4:04 PM > Actually, David, that reference to the docs about "new rows" in > https://www.postgresql.org/docs/10/static/routine-vacuuming.html#VACUUM-BASICS, > recovering disk space, relates to UPDATEs and DELETEs, not INSERTs. > The main gain for reusing dead space is where UPDATES are concerned > where it will try to insert on the same page if possible. > > Regards, > Michael Vitale > > David G. Johnston > Wednesday, February 28, 2018 1:37 PM > On Wed, Feb 28, 2018 at 11:31 AM, Sohel Tamboli > >wrote: > > Hi, > > I need some information on vacuum in postgresql. I know that > "vacuum full" recreate full table and releases space to OS. Only > "vacuum" clears the dead tuples and free the space but does not > returns free space to OS, indeed it keeps free space as a part of > table. > My question is, after running only "vacuum", how does new data or > insert is written to the table? I need to know that Does new data > gets inserted in free space available in between of live tuples or > gets inserted at the end of table everytime. > > > There would be no point to non-full vacuuming if "new data [was] > inserted ... at the end of the table everytime"...​ > > ​ This logic is also documented:​ > > ​ > https://www.postgresql.org/docs/10/static/routine-vacuuming.html#VACUUM-BASICS > ​ > > ​" > The space it occupies must then be reclaimed for reuse by new rows, to > avoid unbounded growth of disk space requirements. This is done by > running VACUUM. > ​"​ > > ​ David J. > > Sohel Tamboli > Wednesday, February 28, 2018 1:31 PM > Hi, > > I need some information on vacuum in postgresql. I know that "vacuum > full" recreate full table and releases space to OS. Only "vacuum" > clears the dead tuples and free the space but does not returns free > space to OS, indeed it keeps free space as a part of table. > My question is, after running only "vacuum", how does new data or > insert is written to the table? I need to know that Does new data gets > inserted in free space available in between of live tuples or gets > inserted at the end of table everytime. > > Appreciate your quick response!! > > Thanks, > Sohel --------------090302060607040601020003 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit oh my my, think David has the correct answer here.  Thanks David for your research and correction.

So INSERT logic: see if space on current target page, then check FSM, then default to end of relation

Regards,
Michael Vitale

Wednesday, February 28, 2018 4:53 PM
On Wed, Feb 28, 2018 at 2:21 PM, MichaelDBA <MichaelDBA@sqlexec.com> wrote:
wee bit further elaboration...

MVCC for updates is DELETE followed by INSERT.  That is what the docs are talking about when mentioning "new rows", not the type of "new row" that is done with an INSERT statement which is ALWAYS appended to end of table.  I actually had to ask about this one in PostgreSQL slack chat because I wasn't 100% positive that index clustering order does not influence INSERT location, but apparently even clustering order does not influence a location for INSERTs other than the end of the table.

​ My (admittedly inexperienced) reading of the code (src/backend/access/head/hio.c::RelationGetBufferForTuple) leads me to conclude that my assumption about the behavior is indeed correct.  Both update-related insertions and isolated insertions result in the system using the free space map (FSM) to locate existing pages with free space on them to use for persisting the newly created tuples.

Specifically:

/*
* We first try to put the tuple on the same page we last inserted a tuple
* on, as cached in the BulkInsertState or relcache entry.  If that
* doesn't work, we ask the Free Space Map to locate a suitable page.
* Since the FSM's info might be out of date, we have to be prepared to
* loop around and retry multiple times. (To insure this isn't an infinite
* loop, we must update the FSM with the correct amount of free space on
* each page that proves not to be suitable.)  If the FSM has no record of
* a page with enough free space, we give up and extend the relation.
*
* When use_fsm is false, we either put the tuple onto the existing target
* page or extend the relation.
*/

and

if (targetBlock == InvalidBlockNumber && use_fsm)
{
/*
* We have no cached target page, so ask the FSM for an initial
* target.
*/
targetBlock = GetPageWithFreeSpace(relation, len + saveFreeSpace);
[...]

David J.

Wednesday, February 28, 2018 4:21 PM
wee bit further elaboration...

MVCC for updates is DELETE followed by INSERT.  That is what the docs are talking about when mentioning "new rows", not the type of "new row" that is done with an INSERT statement which is ALWAYS appended to end of table.  I actually had to ask about this one in PostgreSQL slack chat because I wasn't 100% positive that index clustering order does not influence INSERT location, but apparently even clustering order does not influence a location for INSERTs other than the end of the table.

We all learned something here, lol

Regards,
Michael Vitale


Wednesday, February 28, 2018 4:04 PM
Actually, David, that reference to the docs about "new rows" in https://www.postgresql.org/docs/10/static/routine-vacuuming.html#VACUUM-BASICS, recovering disk space, relates to UPDATEs and DELETEs, not INSERTs.  The main gain for reusing dead space is where UPDATES are concerned where it will try to insert on the same page if possible.

Regards,
Michael Vitale

Wednesday, February 28, 2018 1:37 PM
On Wed, Feb 28, 2018 at 11:31 AM, Sohel Tamboli <sohel.tamboli0016@gmail.com> wrote:
Hi, 

I need some information on vacuum in postgresql. I know that "vacuum full" recreate full table and releases space to OS. Only "vacuum" clears the dead tuples and free the space but does not returns free space to OS, indeed it keeps free space as a part of table. 
My question is, after running only "vacuum", how does new data or insert is written to the table? I need to know that Does new data gets inserted in free space available in between of live tuples or gets inserted at the end of table everytime. 


There would be no point to non-full vacuuming if "new data [was] inserted ... at the end of the table everytime"...​

​ This logic is also documented:​

​"
The space it occupies must then be reclaimed for reuse by new rows, to avoid unbounded growth of disk space requirements. This is done by running VACUUM.
​"​

​ David J.

Wednesday, February 28, 2018 1:31 PM
Hi, 

I need some information on vacuum in postgresql. I know that "vacuum full" recreate full table and releases space to OS. Only "vacuum" clears the dead tuples and free the space but does not returns free space to OS, indeed it keeps free space as a part of table. 
My question is, after running only "vacuum", how does new data or insert is written to the table? I need to know that Does new data gets inserted in free space available in between of live tuples or gets inserted at the end of table everytime. 

Appreciate your quick response!! 

Thanks, 
Sohel

--------------090302060607040601020003--