Feedback-ID: i0c21471d:Fastmail
User-Agent: Cyrus-JMAP/3.9.0-alpha0-745-g95dd7bea33-fm-20230905.001-g95dd7bea
Mime-Version: 1.0
Message-Id: <a52489dd-e6ca-425b-a23d-d0139728419f@app.fastmail.com>
In-Reply-To: 
 <169454016225.2433963.11433530945607791340@wrigleys.postgresql.org>
References: 
 <169454016225.2433963.11433530945607791340@wrigleys.postgresql.org>
Date: Wed, 13 Sep 2023 15:48:41 -0300
From: "Euler Taveira" <euler@eulerto.com>
To: postgresql@seebs.net, pgsql-docs@lists.postgresql.org
Subject: Re: index access method documentation light on details on ii_AmCache
Content-Type: multipart/alternative;
 boundary=7940dd648e284bf1996fe76b24946d34
Archived-At: 
 <https://www.postgresql.org/message-id/a52489dd-e6ca-425b-a23d-d0139728419f%40app.fastmail.com>
Precedence: bulk

--7940dd648e284bf1996fe76b24946d34
Content-Type: text/plain

On Tue, Sep 12, 2023, at 2:36 PM, PG Doc comments form wrote:
> The following documentation comment has been logged on the website:
> 
> Page: https://www.postgresql.org/docs/15/index-functions.html
> Description:
> 
> So, if I cache something in ii_AmCache during a call to my aminsert
> callback...
> 
> When, if ever, does it get freed?
> 
> Having looked at example code, I don't actually see anything doing this in
> insert paths, so presumably there's some point at which this happens
> automatically, possibly as part of the Memory Context thing, maybe related
> to the ii_Context which seems to be getting used, but I can't find anything
> anywhere documenting that. This may well be completely obvious, or intended
> to be implied by "it can allocate space in indexInfo->ii_Context", but it's
> not exceptionally obvious to me as a newcomer to the code. (By contrast, the
> ambuild docs say to palloc a data structure, but don't mention a context for
> it; no idea whether it should be in a particular context.)
> 

Maybe it is not clear by the comment in the IndexInfo struct but ii_AmCache is
a pointer to the AM state information that is stored into ii_Context memory
context (see gistinsert, gininsert or brininsert to understand how ii_Context
and ii_AmCache are used). The document that you shared also has a sentence with
this information.

Unless you change it, ii_Context is CurrentMemoryContext (see makeIndexInfo).
Hence, your AM state information is freed when the current memory context is
freed. You can use gdb to figure out what's the memory context you are in.
Start a session, create a table and an index on it. Attach gdb to this current
session and add a breakpoint into one of the AM insert function that I
mentioned in the previous paragraph. Insert a row and continue:

(gdb) b initGinState
Breakpoint 1 at 0x55a47b179d80: file ginutil.c, line 98.
(gdb) c
Continuing.

Breakpoint 1, initGinState (state=state@entry=0x55a47d40cd58, index=index@entry=0x7f1970352d40)
    at ginutil.c:98
(gdb) bt
#0  initGinState (state=state@entry=0x55a47d40cd58, index=index@entry=0x7f1970352d40) at ginutil.c:98
#1  0x000055a47b178255 in gininsert (index=0x7f1970352d40, values=0x7fff31d04a80, isnull=0x7fff31d04a60, 
    ht_ctid=0x55a47d415ef8, heapRel=<optimized out>, checkUnique=<optimized out>, indexUnchanged=false, 
    indexInfo=0x55a47d4164f8) at gininsert.c:502
#2  0x000055a47b3100f6 in ExecInsertIndexTuples (resultRelInfo=resultRelInfo@entry=0x55a47d415328, 
    slot=slot@entry=0x55a47d415ec8, estate=estate@entry=0x55a47d414e98, update=update@entry=false, 
    noDupErr=noDupErr@entry=false, specConflict=specConflict@entry=0x0, arbiterIndexes=0x0, 
    onlySummarizing=false) at execIndexing.c:432
.
.
.
(gdb) f 1
#1  0x000055a47b178255 in gininsert (index=0x7f1970352d40, values=0x7fff31d04a80, isnull=0x7fff31d04a60,
    ht_ctid=0x55a47d415ef8, heapRel=<optimized out>, checkUnique=<optimized out>, indexUnchanged=false,
    indexInfo=0x55a47d4164f8) at gininsert.c:502
(gdb) p *indexInfo->ii_Context
$3 = {type = T_AllocSetContext, isReset = false, allowInCritSection = false, mem_allocated = 17912, 
  methods = 0x55a47b9302f0 <mcxt_methods+240>, parent = 0x55a47d3fdda0, firstchild = 0x55a47d408cd0, 
  prevchild = 0x0, nextchild = 0x0, name = 0x55a47b751813 "ExecutorState", ident = 0x0, reset_cbs = 0x0}

The "ExecutorState" is a per query memory context so it is deallocated when the
query ends.

> Actually, in full generality, I have not been able to find a section of the
> documentation which explains the memory-context stuff at all. I found a blog
> post elsewhere suggesting that it's just "the memory context will be freed
> and thus everything associated with it". This implies that there's no
> straightforward way for an index to do end-of-insert maintenance after all
> the inserts from a given query are complete, except to do it after every
> tuple just in case it's the last tuple, I guess?
> 

Check src/backend/utils/mmgr/README.


--
Euler Taveira
EDB   https://www.enterprisedb.com/

--7940dd648e284bf1996fe76b24946d34
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html><html><head><title></title><style type=3D"text/css">p.Mso=
Normal,p.MsoNoSpacing{margin:0}</style></head><body><div>On Tue, Sep 12,=
 2023, at 2:36 PM, PG Doc comments form wrote:<br></div><blockquote type=
=3D"cite" id=3D"qt" style=3D""><div>The following documentation comment =
has been logged on the website:<br></div><div><br></div><div>Page:&nbsp;=
<a href=3D"https://www.postgresql.org/docs/15/index-functions.html">http=
s://www.postgresql.org/docs/15/index-functions.html</a><br></div><div>De=
scription:<br></div><div><br></div><div>So, if I cache something in ii_A=
mCache during a call to my aminsert<br></div><div>callback...<br></div><=
div><br></div><div>When, if ever, does it get freed?<br></div><div><br><=
/div><div>Having looked at example code, I don't actually see anything d=
oing this in<br></div><div>insert paths, so presumably there's some poin=
t at which this happens<br></div><div>automatically, possibly as part of=
 the Memory Context thing, maybe related<br></div><div>to the ii_Context=
 which seems to be getting used, but I can't find anything<br></div><div=
>anywhere documenting that. This may well be completely obvious, or inte=
nded<br></div><div>to be implied by "it can allocate space in indexInfo-=
&gt;ii_Context", but it's<br></div><div>not exceptionally obvious to me =
as a newcomer to the code. (By contrast, the<br></div><div>ambuild docs =
say to palloc a data structure, but don't mention a context for<br></div=
><div>it; no idea whether it should be in a particular context.)<br></di=
v><div><br></div></blockquote><div><br></div><div>Maybe it is not clear =
by the comment in the IndexInfo struct but ii_AmCache is<br></div><div>a=
 pointer to the AM state information that is stored into ii_Context memo=
ry<br></div><div>context (see gistinsert, gininsert or brininsert to und=
erstand how ii_Context<br></div><div>and ii_AmCache are used). The docum=
ent that you shared also has a sentence with<br></div><div>this informat=
ion.<br></div><div><br></div><div>Unless you change it, ii_Context is Cu=
rrentMemoryContext (see makeIndexInfo).<br></div><div>Hence, your AM sta=
te information is freed when the current memory context is<br></div><div=
>freed. You can use gdb to figure out what's the memory context you are =
in.<br></div><div>Start a session, create a table and an index on it. At=
tach gdb to this current<br></div><div>session and add a breakpoint into=
 one of the AM insert function that I<br></div><div>mentioned in the pre=
vious paragraph. Insert a row and continue:<br></div><div><br></div><div=
>(gdb) b initGinState<br></div><div>Breakpoint 1 at 0x55a47b179d80: file=
 ginutil.c, line 98.<br></div><div>(gdb) c<br></div><div>Continuing.<br>=
</div><div><br></div><div>Breakpoint 1, initGinState (state=3Dstate@entr=
y=3D0x55a47d40cd58, index=3Dindex@entry=3D0x7f1970352d40)<br></div><div>=
&nbsp;&nbsp;&nbsp; at ginutil.c:98<br></div><div>(gdb) bt<br></div><div>=
#0&nbsp; initGinState (state=3Dstate@entry=3D0x55a47d40cd58, index=3Dind=
ex@entry=3D0x7f1970352d40) at ginutil.c:98<br></div><div>#1&nbsp; 0x0000=
55a47b178255 in gininsert (index=3D0x7f1970352d40, values=3D0x7fff31d04a=
80, isnull=3D0x7fff31d04a60,&nbsp;<br></div><div>&nbsp;&nbsp;&nbsp; ht_c=
tid=3D0x55a47d415ef8, heapRel=3D&lt;optimized out&gt;, checkUnique=3D&lt=
;optimized out&gt;, indexUnchanged=3Dfalse,&nbsp;<br></div><div>&nbsp;&n=
bsp;&nbsp; indexInfo=3D0x55a47d4164f8) at gininsert.c:502<br></div><div>=
#2&nbsp; 0x000055a47b3100f6 in ExecInsertIndexTuples (resultRelInfo=3Dre=
sultRelInfo@entry=3D0x55a47d415328,&nbsp;<br></div><div>&nbsp;&nbsp;&nbs=
p; slot=3Dslot@entry=3D0x55a47d415ec8, estate=3Destate@entry=3D0x55a47d4=
14e98, update=3Dupdate@entry=3Dfalse,&nbsp;<br></div><div>&nbsp;&nbsp;&n=
bsp; noDupErr=3DnoDupErr@entry=3Dfalse, specConflict=3DspecConflict@entr=
y=3D0x0, arbiterIndexes=3D0x0,&nbsp;<br></div><div>&nbsp;&nbsp;&nbsp; on=
lySummarizing=3Dfalse) at execIndexing.c:432<br></div><div>.<br></div><d=
iv>.<br></div><div>.<br></div><div>(gdb) f 1<br></div><div>#1&nbsp; 0x00=
0055a47b178255 in gininsert (index=3D0x7f1970352d40, values=3D0x7fff31d0=
4a80, isnull=3D0x7fff31d04a60,<br></div><div>&nbsp;&nbsp;&nbsp; ht_ctid=3D=
0x55a47d415ef8, heapRel=3D&lt;optimized out&gt;, checkUnique=3D&lt;optim=
ized out&gt;, indexUnchanged=3Dfalse,<br></div><div>&nbsp;&nbsp;&nbsp; i=
ndexInfo=3D0x55a47d4164f8) at gininsert.c:502<br></div><div>(gdb) p *ind=
exInfo-&gt;ii_Context<br></div><div>$3 =3D {type =3D T_AllocSetContext, =
isReset =3D false, allowInCritSection =3D false, mem_allocated =3D 17912=
,&nbsp;<br></div><div>&nbsp; methods =3D 0x55a47b9302f0 &lt;mcxt_methods=
+240&gt;, parent =3D 0x55a47d3fdda0, firstchild =3D 0x55a47d408cd0,&nbsp=
;<br></div><div>&nbsp; prevchild =3D 0x0, nextchild =3D 0x0, name =3D 0x=
55a47b751813 "ExecutorState", ident =3D 0x0, reset_cbs =3D 0x0}<br></div=
><div><br></div><div>The "ExecutorState" is a per query memory context s=
o it is deallocated when the<br></div><div>query ends.<br></div><div><br=
></div><blockquote type=3D"cite" id=3D"qt" style=3D""><div>Actually, in =
full generality, I have not been able to find a section of the<br></div>=
<div>documentation which explains the memory-context stuff at all. I fou=
nd a blog<br></div><div>post elsewhere suggesting that it's just "the me=
mory context will be freed<br></div><div>and thus everything associated =
with it". This implies that there's no<br></div><div>straightforward way=
 for an index to do end-of-insert maintenance after all<br></div><div>th=
e inserts from a given query are complete, except to do it after every<b=
r></div><div>tuple just in case it's the last tuple, I guess?<br></div><=
div><br></div></blockquote><div><br></div><div>Check src/backend/utils/m=
mgr/README.<br></div><div><br></div><div><br></div><div id=3D"sig1124090=
80"><div class=3D"signature">--<br></div><div class=3D"signature">Euler =
Taveira<br></div><div class=3D"signature">EDB&nbsp;&nbsp; <a href=3D"htt=
ps://www.enterprisedb.com/">https://www.enterprisedb.com/</a><br></div><=
/div><div><br></div></body></html>
--7940dd648e284bf1996fe76b24946d34--