Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lp8JK-00039C-DT for pgsql-hackers@arkaria.postgresql.org; Fri, 04 Jun 2021 11:48:22 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lp8JJ-0002Y2-Bx for pgsql-hackers@arkaria.postgresql.org; Fri, 04 Jun 2021 11:48:21 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lp8JJ-0002Xu-0O for pgsql-hackers@lists.postgresql.org; Fri, 04 Jun 2021 11:48:21 +0000 Received: from mail-ej1-x631.google.com ([2a00:1450:4864:20::631]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1lp8JG-00061Z-9L for pgsql-hackers@postgresql.org; Fri, 04 Jun 2021 11:48:19 +0000 Received: by mail-ej1-x631.google.com with SMTP id c10so13992306eja.11 for ; Fri, 04 Jun 2021 04:48:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=enterprisedb-com.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=HWiDCKuI3XvLAH8f9sliKxtBGK8BRYc4xiI+XTh0Lvo=; b=qrg/n9GhMV0H5FZEKTw6lNb93bv+pqo9Zt311P960vkmHd/lcDv0rNRI5vUfcw2y+v 6NRNk+m8zCjcVgYWfjpfDHThGeWqCMQ0yUmz/HNCZbcgngk2FJeLIrYPcmhTO5QelfQT X09Nps6aCvKGbscQYE4gHSuVNYWJRBrnYzq0FTDktIxBNN7rPVn6aGpC+YEPt7KHyCTn NFqzLT59aY2hk7r5kBAo6qQARnFFvewLn2uXFTIA0fJpKv/O+3elKToGiCJ8gcmkY15X 78mbE43YVWTJE+P/0xRd7+xepgeFZSMqSP6crVq5bFYw6ib4+vEHSH87mGwvM5VoOuNN BDfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=HWiDCKuI3XvLAH8f9sliKxtBGK8BRYc4xiI+XTh0Lvo=; b=CrTKiKXBtOT0h5HoLyf0rXhwHlZyRTyvXmhHyVJOVl2+oeuthOpz2z7e97gR1NZwP9 4sxJ5Dhcu8XC8XpqYF0Qs/Hxm90TeG5yOYrzDSDeu2B1Qj+gCxtpK7Js4jnG2jaA2ZOn bU8Kmh0G166zL9BxaEX8o7R4UjrMaavi4PhvqDyl13395D+fMndobYi1jofjuFwia5u/ zU3UYo7LzpnQjqXMnAH7ROPgwFvxI1R4Wx+XtlFLPPmsXfpK6DM5nb9O01pNFcQAyt4G uAudLtVfArqawOzCeC7/fXtjWvp4/vfqPqAP4LdZo34qLFZzJeTqbOm5H+cu7v8iAxhy 4NZg== X-Gm-Message-State: AOAM5308Bs5lXrB4Pl6PaIbkNPYSDwnII0AXujBzGWmUEi08NKALg7pk zb4JWAVvClWi/EMrLrlh5DaPAJYBvOCqkYnSk2lwsFPqw5lG224gJ2zza3hQVKogXNbA7nUp/1g bw8iskueuK/IBhD+rzWXE9Ii1YBgbgLTF4tivjnWNC73WECLiZCc4NFOxDp5w1p07YOhOkjzQWh j2JEWH17TFSV3Iq3yiz2cBDNLdZqZnJbrUvwc+ahzi6eYyDrHa9vrxGmNyIc1Ey4qDrFZ8cTuqW a1DunpmVBS9WMXz1BTK8SRwMa2BMGApjkJr8BzF X-Google-Smtp-Source: ABdhPJy0Rp8ek8bOJltypyToYz4reQD3LN+NbIkd5jMrJHRKWguFm1L7OGOtIkSINYY8A5mrffmtwQ== X-Received: by 2002:a17:906:b2d1:: with SMTP id cf17mr3812967ejb.225.1622807296163; Fri, 04 Jun 2021 04:48:16 -0700 (PDT) Received: from [10.137.0.19] (ip-86-49-254-183.net.upcbroadband.cz. [86.49.254.183]) by smtp.gmail.com with ESMTPSA id u19sm2653003ejt.74.2021.06.04.04.48.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 04 Jun 2021 04:48:15 -0700 (PDT) Subject: Re: postgres_fdw batching vs. (re)creating the tuple slots To: Tom Lane , Andres Freund Cc: PostgreSQL-development References: <20210530205831.vib4p7mt23cymhpc@alap3.anarazel.de> <2399204.1622409059@sss.pgh.pa.us> <20210530212616.5k5nxnbunwnwet2a@alap3.anarazel.de> <2433405.1622416631@sss.pgh.pa.us> From: Tomas Vondra Message-ID: Date: Fri, 4 Jun 2021 13:48:13 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: <2433405.1622416631@sss.pgh.pa.us> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-CLOUD-SEC-AV-Info: enterprisedb,google_mail,monitor X-CLOUD-SEC-AV-Sent: true X-Gm-Spam: 0 X-Gm-Phishy: 0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Hi, Here's two WIP patches that fixes the regression for me. The first part is from [1], so make large batches work, 0002 just creates a copy of the tupledesc to not cause issues in resource owner, 0003 ensures we only initialize the slots once (not per batch). With the patches applied, the timings look like this: batch timing ---------------------- 1 64194.942 ms 10 7233.785 ms 100 2244.255 ms 32k 1372.175 ms which seems fine. I still need to get this properly tested etc. and make sure nothing is left over. regards [1] https://postgr.es/m/OS0PR01MB571603973C0AC2874AD6BF2594299%40OS0PR01MB5716.jpnprd01.prod.outlook.com -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company