Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iEL7U-0003os-4y for pgsql-hackers@arkaria.postgresql.org; Sat, 28 Sep 2019 22:23:16 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iEL6U-0002bk-7j for pgsql-hackers@arkaria.postgresql.org; Sat, 28 Sep 2019 22:22:14 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iEL6T-0002bd-Va for pgsql-hackers@lists.postgresql.org; Sat, 28 Sep 2019 22:22:14 +0000 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iEL6R-0008Ef-O1 for pgsql-hackers@postgresql.org; Sat, 28 Sep 2019 22:22:13 +0000 Received: by mail-wm1-x341.google.com with SMTP id 7so9444332wme.1 for ; Sat, 28 Sep 2019 15:22:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=LNgkdNZ6Y/YuGu0mHlQc98bUSARBkEOOobF4nnjYqIg=; b=EoETwKrVoiJGnZncjFNBdwECrY+hs7NpWeSSDMHfG2VrPKSfTjYfbHEluEb4QNisDl 92/PLi46zCejpv1b/lCzahwvmhrl3xRp7h7wzoa49Jniuk1rSVTF3hgPPa2HWhUlONs4 Bbp5pJdGOzWrUQLzFNl+AKXKb9i2V0iRKyvr01UnopQwmYAeBHzXGyit0qNZ8TuNuSLT QK5cRcSheJA5FgaXIMLSUhGJf8tgZ9EpV4c3ap9VUqKnTasMIvPyLZ4bSLdaVyKCbQgC CRBNiMx581O0dI3q0tZTwkxEoAlLm3dRRe6HCGuUpbToPZ/Bh4hP4w9juBZS2gmBJL3X Eu9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=LNgkdNZ6Y/YuGu0mHlQc98bUSARBkEOOobF4nnjYqIg=; b=U3DyhIZRDMGuVIL6aeD0YVYGyqUQf7fWZ/5HFjKvgRRic5wyBThF/vW/GPiEkZkGb7 8BmA24ZgDhKeolFpi5W2pt+5dYJ6jjmQa0uhB4mQmE8PTtbvO0fz7Gb0e/3kGUVo3orz bb6OdeI+QnPsAbAAVFoMXcJnRIAPJrBFA2o8+5M5zETXCdCo/r7pns+SAyRrnk1LekmW VRakjCLQCtzNgNX2mfCU0JkwPkMC80Iz+4Xspg1/Rrj3VX6EHMaQELJnZ+5G/8QbUAPZ ZUm8GyeFT8HtFWk8HJxUUBuGDoAnnsFCdEt1ixi/PnRmHAfaHqU61w+D+9rYaqudlif+ oFqQ== X-Gm-Message-State: APjAAAUDhgp8klkQhg2yfw+rXjC82PFuMZcCEh9L5eAwXldR9eJboyWP v794DxUZN4ZDfEcLD2ccGT/8Hw== X-Google-Smtp-Source: APXvYqwKrIPdO7OPgeW+6V0Cvx67B8xB3CnaC+OO2ACPs6sFo1tBGP4xZJZpoA2aL8GSRjkZFVkrHg== X-Received: by 2002:a7b:c112:: with SMTP id w18mr12535344wmi.88.1569709329376; Sat, 28 Sep 2019 15:22:09 -0700 (PDT) Received: from localhost (ip-86-49-253-183.net.upcbroadband.cz. [86.49.253.183]) by smtp.gmail.com with ESMTPSA id z13sm6414507wrq.51.2019.09.28.15.22.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Sep 2019 15:22:08 -0700 (PDT) Date: Sun, 29 Sep 2019 00:22:06 +0200 From: Tomas Vondra To: Jeff Davis Cc: Melanie Plageman , Soumyadeep Chakraborty , Robert Haas , Pg Hackers , soumyadeep2007@gmail.com Subject: Re: Memory Accounting Message-ID: <20190928222206.thidzeygazl7axhg@development> References: <027a129b8525601c6a680d27ce3a7172dab61aab.camel@j-davis.com> <20190724215228.tlyj74ie4l2jn7ji@development> <67f66d2fa39e35aef90093c74531a38dcd5c0f73.camel@j-davis.com> <20190926192244.mk5ru6nkofo7am3h@development> <57780817877d4b40106b248b63261e21a7a56261.camel@j-davis.com> <20190928221249.mo2jrtlakingnc5e@development> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20190928221249.mo2jrtlakingnc5e@development> User-Agent: NeoMutt/20180716 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Sun, Sep 29, 2019 at 12:12:49AM +0200, Tomas Vondra wrote: >On Thu, Sep 26, 2019 at 01:36:46PM -0700, Jeff Davis wrote: >>On Thu, 2019-09-26 at 21:22 +0200, Tomas Vondra wrote: >>>It's worth mentioning that those bechmarks (I'm assuming we're >>>talking >>>about the numbers Rober shared in [1]) were done on patches that used >>>the eager accounting approach (i.e. walking all parent contexts and >>>updating the accounting for them). >>> >>>I'm pretty sure the current "lazy accounting" patches don't have that >>>issue, so unless someone objects and/or can show numbers >>>demonstrating >>>I'wrong I'll stick to my plan to get this committed soon. >> >>That was my conclusion, as well. >> > >I was about to commit the patch, but during the final review I've >noticed two places that I think are bugs: > >1) aset.c (AllocSetDelete) >-------------------------- > >#ifdef CLOBBER_FREED_MEMORY > wipe_mem(block, block->freeptr - ((char *) block)); >#endif > > if (block != set->keeper) > { > context->mem_allocated -= block->endptr - ((char *) block); > free(block); > } > >2) generation.c (GenerationReset) >--------------------------------- > >#ifdef CLOBBER_FREED_MEMORY > wipe_mem(block, block->blksize); >#endif > context->mem_allocated -= block->blksize; > > >Notice that when CLOBBER_FREED_MEMORY is defined, the code first calls >wipe_mem and then accesses fields of the (wiped) block. Interesringly >enough, the regression tests don't seem to exercise these bits - I've >tried adding elog(ERROR) and it still passes. For (2) that's not very >surprising because Generation context is only really used in logical >decoding (and we don't delete the context I think). Not sure about (1) >but it might be because AllocSetReset does the right thing and only >leaves behind the keeper block. > >I'm pretty sure a custom function calling the contexts explicitly would >fall over, but I haven't tried. > Oh, and one more thing - this probably needs to add at least some basic explanation of the accounting to src/backend/mmgr/README. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services