Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tQJRi-001dKh-7D for pgsql-hackers@arkaria.postgresql.org; Wed, 25 Dec 2024 04:56:34 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1tQJRf-00B2Ux-Sx for pgsql-hackers@arkaria.postgresql.org; Wed, 25 Dec 2024 04:56:31 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tQJRf-00B2Uo-Gh for pgsql-hackers@lists.postgresql.org; Wed, 25 Dec 2024 04:56:31 +0000 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1tQJRd-000RaA-1m for pgsql-hackers@lists.postgresql.org; Wed, 25 Dec 2024 04:56:30 +0000 Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-216634dd574so42039865ad.2 for ; Tue, 24 Dec 2024 20:56:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735102588; x=1735707388; darn=lists.postgresql.org; h=content-transfer-encoding:subject:from:cc:content-language:to :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=hF/j2RIbH0lvGd2xn9BB5xBRH/MPIjQF9Qpolfwt6dY=; b=TQSKeQQxdm7NQv7rZhmJMfcjBar4oZXvlYVJI5D4BCwuCoOkj/Qkl/ZAX1tqDXudwZ stheNwPYv3/HZ5VgkpIGeIIG0nNUjjyGJQu8KoA1QWD785DLwGiLjsIbXs3acVBw09FR wqHTF6dcMdjTPnFVSRo3vddcWFGm8sOTvLU0niMEXJiFZQjBkLb37d8J3q+RCRBQJkML b99tXWeGUoP2ck5fwqPXtqoXNAzPjOEaKmaaLnWVsRc/uYX7M4GWNb+3FbtNSfa2Kuw3 Tfn5CzcFeY61gdqdbwtwj/J4h6h3I2vE1dt4mT1HFEGMXDPHTjvIPvxv4bMh0cmMrLa+ OHiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735102588; x=1735707388; h=content-transfer-encoding:subject:from:cc:content-language:to :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=hF/j2RIbH0lvGd2xn9BB5xBRH/MPIjQF9Qpolfwt6dY=; b=AU9ftHOR5qSvfXxxlQTQ8jquIEcr8HydqQFoLPLmvMEYZh99YuhtgH0P3s3+ZbGkP9 T9xQkBNb04gPX5Sv9np+fKliV6XIootFShfhnDri00MCbnUySqszH7zK04qS8WKwHwIG /98Xkg2Ra7jInFzxeT8W9dCE+kPHqKUyxLqDJ9C3s0S7B6d9clAnBFIN/nxd4c6QWA28 9puJhlCQ4bIz7OvU9xRQBxGeS4RPoHTQp9RLNpqb7Zcxzr+ttaiTj04+oyX4N52oY032 UrrjTG9kHIDiW5nQt2r9J5kMt6L1vgOMy5kcB/sgJnKzSAKWMqmDGs3yeywq1Jp7glHL Wrwg== X-Gm-Message-State: AOJu0YyYW86j6sn836t492DCYgktH4DJDbsyFI9hllxJYPQfE47k6VdN t3w3prQi76kEsBJ+/ffjilWbbnXcqtKmTeY1z4Ysig8jZ8RJCM44Eu6Z6Voe X-Gm-Gg: ASbGncvPugiz1M+bp/tQobTzlbOsxvYCs3NpXXSNHJKjMaXKlYUwdN2B9Dgi/Wwz8ba jWEB0Jtb5rG0JR8C7Ehrvu1h6iGkLPpOwVenMiBr+YzSHoS8e+W+Fk4jwH9vZfHpd847ra2kT5G DnfrX78bKfSu2JfEvPJuCG/WfIOlXT7slCwKw65hRJJyu12vBsF4ryGomGtoECp35aCz+bwn2/A VmTnt81pePwa+P3Zc/PJ6MOAYZGjd39wqseayDwPE0Oi00quYnolbwYUEUks92gP9+TGfORM3w5 vh2QVcaRRbuwuwNZxZ5jW/EdCqule8tgm38= X-Google-Smtp-Source: AGHT+IG9wtHEj7en9kajWPufdioMXKHC6alQckUEO8FzCtkrgQ6zYA1zFoQNDrGBRGEFj3L15DLs/Q== X-Received: by 2002:a17:903:247:b0:216:1ad2:1d5 with SMTP id d9443c01a7336-219e6f145camr267435125ad.41.1735102588364; Tue, 24 Dec 2024 20:56:28 -0800 (PST) Received: from [10.10.20.106] (node-u7g.pool-101-109.dynamic.totinternet.net. [101.109.152.236]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dc962c31sm97678855ad.47.2024.12.24.20.56.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 24 Dec 2024 20:56:27 -0800 (PST) Message-ID: <1f797d0e-4829-48b5-817e-2278466ed4ef@gmail.com> Date: Wed, 25 Dec 2024 11:56:24 +0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: PostgreSQL Hackers Content-Language: en-AU Cc: Tom Lane From: Andrei Lepikhov Subject: Short-living Memory Context in the optimiser Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Hi, Here is a rebased and slightly commented Tom's patch separated from the discussion [1]. I still occasionally see that planner peak memory consumption is triggered by selectivity estimation of massive arrays and clauses generated for multiple partitions. I think that in case of a growing number of selectivity estimations, the planner should consume memory in some scalable manner, so this thread is an attempt to draw community attention to this issue. I question some decisions in that patch. For example, field 'depth' and even 'usage' can be incorporated inside the MemoryContext structure. This can make the reset function safer. To avoid overhead, it may be used only for the 'short-living' class of memory contexts, which can be declared with an additional parameter of AllocSetContextCreate on a context creation. [1] https://www.postgresql.org/message-id/1367418.1708816059@sss.pgh.pa.us -- regards, Andrei Lepikhov