Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q4NaR-0004FU-NN for pgsql-general@arkaria.postgresql.org; Wed, 31 May 2023 15:18:07 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1q4NaQ-0001Iv-4C for pgsql-general@arkaria.postgresql.org; Wed, 31 May 2023 15:18:06 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q4NXL-0004dg-Lv for pgsql-general@lists.postgresql.org; Wed, 31 May 2023 15:14:55 +0000 Received: from out3-smtp.messagingengine.com ([66.111.4.27]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q4NXC-003J7e-GV for pgsql-general@lists.postgresql.org; Wed, 31 May 2023 15:14:55 +0000 Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 7724C5C01F5; Wed, 31 May 2023 11:14:45 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Wed, 31 May 2023 11:14:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aklaver.com; h= cc:content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1685546085; x=1685632485; bh=VKpvE97kFGHBecDvJ4bgCTZ+naSbD1O4X9I H2g2cXeU=; b=vKaPYyOZicWk/2cDrlHNEYZ7Lc7sXhs652i3a278/BGkEcoYkaW fgHseSIwTJEVK0BiN74NYJgRh/vcroEi8hX/bOKQW3pM/D5yNX32JWoBKiM4+4Yc YbLrhrQgJCMtz1gbbpY5yMb+4fby1HYAODhChRWvbC/XDVfpzmO1TDP40628iYHd x3fAe3MccvC+9M+ia9jops/M90Ec3qi+ZgaJgfOC6HzsWrr1HaBmMwP/GjhAz1EO gEIMWDplLKCxX25MEhZX+ERUnzpy8EGFVv2tEyccduWRdWb4fthbLAk0JkP3rJV4 F4roJe8X6PcL3CwTPqNMVWCEfkiVNk94TfA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1685546085; x= 1685632485; bh=VKpvE97kFGHBecDvJ4bgCTZ+naSbD1O4X9IH2g2cXeU=; b=b ixMrtTpZpelZO7E55MgMK3KEwApQzngCVNKCK2uJJcF/KR3Of30NM/hmA5qwUtiK cKb+veJjnpzijG8lN2I+YfOnnbZJ8FU6rvVnQpCGAjGEclASLBGcucI52Z0RjHLY Ebd07d9jFO0TfZYDfSGOCVIFRcyNNP53ePkcpMrShKMlj4mzCjDrQIbpVY1Qk/kc RkeMz1EFUYzKMqBxpbBVshghEhOOlZ7DcLzpAbNSd3zP/6EFp0J1lleiHrHpn57E 9UotmmFbbZ+npxGlSgvDRAPQxmeeik6np590NCG9bwDBMa7/VTnmgz5BGUyRCQr4 GWPcwvBD0AwRzxgB0vXLA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfeekledgkeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvfhfhjggtgfesthejredttdefjeenucfhrhhomheptegurhhi rghnucfmlhgrvhgvrhcuoegrughrihgrnhdrkhhlrghvvghrsegrkhhlrghvvghrrdgtoh hmqeenucggtffrrghtthgvrhhnpeevvdekiefhudfhhfdtkeelgeffudeigfetjeegfffg hfetfeelledugfelleegveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpegrughrihgrnhdrkhhlrghvvghrsegrkhhlrghvvghrrdgtohhm X-ME-Proxy: Feedback-ID: i76984098:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 May 2023 11:14:45 -0400 (EDT) Message-ID: Date: Wed, 31 May 2023 08:14:44 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.2 Subject: Re: speed up full table scan using psql Content-Language: en-US To: Lian Jiang , pgsql-general@lists.postgresql.org References: From: Adrian Klaver In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 5/30/23 21:25, Lian Jiang wrote: > hi, > > I am using psql to periodically dump the postgres tables into json files > which are imported into snowflake. For large tables (e.g. 70M rows), it The command you are using is? Postgres version? > takes hours for psql to complete. Using spark to read the postgres table > seems not to work as the postgres read only replication is the > bottleneck so spark cluster never uses >1 worker node and the working > node timeout or out of memory. > > Will vertical scaling the postgres db speed up psql? Or any thread > related parameter of psql can help? Thanks for any hints. > > Regards > Lian -- Adrian Klaver adrian.klaver@aklaver.com