Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rh771-009fpg-Nt for pgsql-hackers@arkaria.postgresql.org; Mon, 04 Mar 2024 12:08:08 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rh770-007FY2-8k for pgsql-hackers@arkaria.postgresql.org; Mon, 04 Mar 2024 12:08:06 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rh76z-007FXu-VD for pgsql-hackers@lists.postgresql.org; Mon, 04 Mar 2024 12:08:06 +0000 Received: from mail-ej1-x631.google.com ([2a00:1450:4864:20::631]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1rh76w-002iLo-LZ for pgsql-hackers@lists.postgresql.org; Mon, 04 Mar 2024 12:08:05 +0000 Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-a44cdb2d3a6so257405966b.2 for ; Mon, 04 Mar 2024 04:08:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb.com; s=google; t=1709554081; x=1710158881; darn=lists.postgresql.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=vmY0bWWk92iHcW7CJ8a4gyRmajKTUEEBKvZJAa0FPLg=; b=jn4bc13xjGVDJUlcrFbgGEIfNfsSuXEqMIWmS5n3ByyDzz6ZTUpJn702vc7oZaOGB8 KP9KqKyoVSuDAqImDE9faM9E7z8jDW48rpIDKFoqx0YcuVrDPr7h/caVvbQx8qDfLclU SPdM+WRwU27c4z0I4nN80GnlNCYMIdIcniX4nEvvIBlwu4J0K/WavlE+uNvSB6I+fuqw JXUgpsbtmTjTDTAO9RGVy26g5P8S6MFQDYVphyR99ztP7Z5jBH0UkwXHdz3p/yZtX9jB rqTpiCM11Lirn+dicekJVU7SYQMBZE5RPjzk2C9QsIcxFKwFg3IibKcjyT2lsZCLvD9t vKag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709554081; x=1710158881; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vmY0bWWk92iHcW7CJ8a4gyRmajKTUEEBKvZJAa0FPLg=; b=he8o6G5KWYB0dL0TW18tpCZ2urruOC0weg/jMH3u/TLYPY6bDkZLz7Idgc8jn6MMvh hatHR5iNiX8ljwPOtZfWsgk8994B9U4tdglRqg1jWLXaFOUDtjphpt0Dtsy6punFi0Uz +4sqgrX5LJQNqoY8MOaq0unJeoS2e3DOLCWzK/iYMO10RdKRnniYYSvhXqlHKKRsQ2Jm UpbE1iJyPEd9Zkg1X3o24rryNQWhIm2k2K+nsboEtWYog5Hs3Vfl36dkrLF/gRGeyehc GPjhQjlU18qVmKas5FyCYoCvSB+yMLw1dftR4cV+nRR6F9Giqp831qLpKUJQBM8WRaY5 xRJw== X-Forwarded-Encrypted: i=1; AJvYcCW8JMWLI98I232cYaR5bGiDjpwJXoSJJdSaVG9aEkMN03YS7pslL8SMeJA5XuwC+hMIo2VVWXTEDMvp+Ub/ESnfGoidVpCMER7J9xFBBKBeLxQH X-Gm-Message-State: AOJu0YwE+sxfHp2b/2CXEcQb56L2/JK+WwuhCz/MzmXFa7RMPO11H14J +UqTQr8It6ltIT3/xE0kwZ2onsXU3AISQZmkvB9o/EBeget8QBgJIAbyb0hrUg== X-Google-Smtp-Source: AGHT+IGa8IwUB4MYCGixrZJVHjg3fsbIJPWUBPHUU6uqWkQ2YA2jooeAktZo5fJNnLqBrmZRtjXysQ== X-Received: by 2002:a17:906:4108:b0:a45:44a0:3635 with SMTP id j8-20020a170906410800b00a4544a03635mr1756178ejk.49.1709554081321; Mon, 04 Mar 2024 04:08:01 -0800 (PST) Received: from [10.137.0.18] (ip-86-49-229-30.bb.vodafone.cz. [86.49.229.30]) by smtp.gmail.com with ESMTPSA id bj16-20020a170906b05000b00a44adc01f13sm3237687ejb.196.2024.03.04.04.08.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 04 Mar 2024 04:08:00 -0800 (PST) Message-ID: <6718759c-2dac-48e4-bf18-282de4d82204@enterprisedb.com> Date: Mon, 4 Mar 2024 13:07:58 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Shared detoast Datum proposal To: Andy Fan Cc: Nikita Malakhov , Michael Zhilin , Peter Smith , vignesh C , Matthias van de Meent , pgsql-hackers@lists.postgresql.org References: <87il4jrk1l.fsf@163.com> <87wmskm9qu.fsf@163.com> <87v87kvboq.fsf@163.com> <71321d83-391b-473b-aaf6-618c7f682b96@postgrespro.ru> <87r0i7vos3.fsf@163.com> <874je3hcwn.fsf@163.com> <8734tmg88e.fsf@163.com> <7f4c8673-4c5d-4dbf-b8e5-0ec11cd2fba0@enterprisedb.com> <87h6i2djf9.fsf@163.com> <87jzms0xza.fsf@163.com> <87frxf1gld.fsf@163.com> <871q8z19ux.fsf@163.com> <87msrgdq93.fsf@163.com> <87frx7et1j.fsf@163.com> <87v862dbht.fsf@163.com> Content-Language: en-US From: Tomas Vondra In-Reply-To: <87v862dbht.fsf@163.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 3/4/24 02:29, Andy Fan wrote: > > Tomas Vondra writes: > >> On 3/3/24 07:10, Andy Fan wrote: >>> >>> Hi, >>> >>> Here is a updated version, the main changes are: >>> >>> 1. an shared_detoast_datum.org file which shows the latest desgin and >>> pending items during discussion. >>> 2. I removed the slot->pre_detoast_attrs totally. >>> 3. handle some pg_detoast_datum_slice use case. >>> 4. Some implementation improvement. >>> >> >> I only very briefly skimmed the patch, and I guess most of my earlier >> comments still apply. > > Yes, the overall design is not changed. > >> But I'm a bit surprised the patch needs to pass a >> MemoryContext to so many places as a function argument - that seems to >> go against how we work with memory contexts. Doubly so because it seems >> to only ever pass CurrentMemoryContext anyway. So what's that about? > > I think you are talking about the argument like this: > > /* ---------- > - * detoast_attr - > + * detoast_attr_ext - > * > * Public entry point to get back a toasted value from compression > * or external storage. The result is always non-extended varlena form. > * > + * ctx: The memory context which the final value belongs to. > + * > * Note some callers assume that if the input is an EXTERNAL or COMPRESSED > * datum, the result will be a pfree'able chunk. > * ---------- > */ > > +extern struct varlena * > +detoast_attr_ext(struct varlena *attr, MemoryContext ctx) > > This is mainly because 'detoast_attr' will apply more memory than the > "final detoast datum" , for example the code to scan toast relation > needs some memory as well, and what I want is just keeping the memory > for the final detoast datum so that other memory can be released sooner, > so I added the function argument for that. > What exactly does detoast_attr allocate in order to scan toast relation? Does that happen in master, or just with the patch? If with master, I suggest to ignore that / treat that as a separate issue and leave it for a different patch. In any case, the custom is to allocate results in the context that is set in CurrentMemoryContext at the moment of the call, and if there's substantial amount of allocations that we want to free soon, we either do that by explicit pfree() calls, or create a small temporary context in the function (but that's more expensive). I don't think we should invent a new convention where the context is passed as an argument, unless absolutely necessary. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company