Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q4jqU-0005b8-0v for pgsql-general@arkaria.postgresql.org; Thu, 01 Jun 2023 15:04:10 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1q4jqR-0007oh-V7 for pgsql-general@arkaria.postgresql.org; Thu, 01 Jun 2023 15:04:07 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q4jqQ-0007nR-Da for pgsql-general@lists.postgresql.org; Thu, 01 Jun 2023 15:04:07 +0000 Received: from wout1-smtp.messagingengine.com ([64.147.123.24]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q4jqI-0035rj-Me for pgsql-general@lists.postgresql.org; Thu, 01 Jun 2023 15:04:05 +0000 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 0E3EE32001FC; Thu, 1 Jun 2023 11:03:54 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Thu, 01 Jun 2023 11:03:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aklaver.com; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1685631834; x=1685718234; bh=DQ6vz42okeiHAgOD45JvdCZNUSAA2SGVkmf HFYgBKw4=; b=fNyFJtzKyb+Q7usjymfwJwIaa3md4iJr7R5UkqGAD/p3u5OQyln 6gHP89pPJG43wcExUJryJO9ToJU1aD2RR+oK0T16eLBp/XQ5ouT8BcMNvczAHIbT fyoCIrdoTq9Lh/g98hpg5Nr0XDBmkGWGXHzkXaogrNJEE4tisqoKeD/1dQ2DovTs J4eVTBtTja1Wee87BSmiFFQ1M7YZ1DInIpld6jKMLptR6zUXwDPGt9bilvnPneBj NLrBSm6iDAiHoVVsYQ5KKrFp44Ao0INOR/xiNYNjYuARytE/3BOMEvqpslQcwjeg 3OXWRVhMzUAYiSUw5Rg3sae1Zw3AL97edNA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1685631834; x=1685718234; bh=DQ6vz42okeiHAgOD45JvdCZNUSAA2SGVkmf HFYgBKw4=; b=FpCrg8ys0423L9X6BFvAeL8E78TI9wIkjnBcH+esl/CsydqOD6+ 37f9kKTQ2qUZ8UL39oFaLClalghA2B67eY4ThjNnbJiL+a25wlvehxMZhoX1LRES 7wVAX73OT7HaraQZ1eKZC60tgenSm3RxdI/TfI4wAD6w+6WQjI0OC4Ou4IfL7GtF bqTp1KC/WijahS+H31yYjvNvtG1DfeQws+Y+b7VpX8KUovuzKoKHBT7hAyVMw3UJ ncHTAbeibcB2jXGj2lFAXec2L2JQif25vtkVOEHQ1HRXvrl+FNxm2CF4sdG3gvCo DqN5f/NJx9PcFi5WtAywyU8Z2Ihwj20wRGA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfeeluddgkeegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpeetughr ihgrnhcumfhlrghvvghruceorggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtg homheqnecuggftrfgrthhtvghrnhepjeegtdfhuefhtefgffdvheduleetgeejudevffek teegjeeiheeuudellefgffffnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomheprggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtghomh X-ME-Proxy: Feedback-ID: i76984098:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 1 Jun 2023 11:03:54 -0400 (EDT) Message-ID: <8bee33fc-9b8d-94d9-6fe3-256db822f75f@aklaver.com> Date: Thu, 1 Jun 2023 08:03:53 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.2 Subject: Re: speed up full table scan using psql To: Lian Jiang Cc: pgsql-general@lists.postgresql.org References: Content-Language: en-US From: Adrian Klaver In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 5/31/23 22:51, Lian Jiang wrote: > The whole command is: > > psql %(pg_uri)s -c %(sql)s | %(sed)s | %(pv)s | %(split)s) 2>&1 | %(tr)s > > where: > sql is "copy (select row_to_json(x_tmp_uniq) from public.mytable > x_tmp_uniq) to stdout" > sed, pv, split, tr together format and split the stdout into jsonl files. Well that is quite the pipeline. At this point I think you need to do some testing on your end. First create a table that is a subset of the original data to make testing a little quicker. Then break the process down into smaller actions. Start with just doing a COPY direct to CSV and one with the row_to_json to see if that makes a difference. Then COPY directly to a file before applying the above pipeline. There are more ways you can slice this depending on what the preceding shows you. > > Hope this helps. -- Adrian Klaver adrian.klaver@aklaver.com