Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q4TbI-0000Jq-6D for pgsql-general@arkaria.postgresql.org; Wed, 31 May 2023 21:43:24 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1q4TbF-0004dL-Ov for pgsql-general@arkaria.postgresql.org; Wed, 31 May 2023 21:43:21 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q4TbE-0004c7-OU for pgsql-general@lists.postgresql.org; Wed, 31 May 2023 21:43:21 +0000 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q4Tb6-003MTa-SO for pgsql-general@lists.postgresql.org; Wed, 31 May 2023 21:43:19 +0000 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 517013200961; Wed, 31 May 2023 17:43:11 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Wed, 31 May 2023 17:43:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aklaver.com; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1685569390; x=1685655790; bh=OVLFqIeCva3a96r2U5KD5x+DxtjixFfwCBV Wo/fsKQw=; b=caB7FZObjcPwYjqv3zUQkUEz/8oDWe/PbHKG41kt7vVy5ZjcK5m Bt+zCCv7jbsk8qWNJUxaWN0mgWKmc+O/xXfBIXZ0gMJnNYIFJSVmqRGgqCegJujV 3NlAgknE82S1E33y98apoBYBGWn2yA6Y4Jv/x6u22Ech741kA6scCduEIKeyeFkD p72l814veMFqs1BlfzDzDNf+4nn9xAtw+EcZuFccDkkWkeu/zT8EqXtoGShOrIIs ZoouSuJlwvOx57K7n0qMqyqDeW9oOpkkCbm4F4avKKrOfV2NyUHPz2E76OG8XokC JbB7qpS5CZQ2IU94OrREEwuzzY0hggQEiDg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1685569390; x=1685655790; bh=OVLFqIeCva3a96r2U5KD5x+DxtjixFfwCBV Wo/fsKQw=; b=kTn99XP0CmNq8956XOzAh6P9eGC3fzo7e6AmEFsToxOPNv1+yYH Kti3WCYG9/PSk6Oki7bAfxaQaAQPL+i17VB4dVlaIJ/c/R32Y+C68gha22FIeY69 LB7M6yeNf3Agz2UEYndkvPWdluNyCrpCQ5TJMPgmbz5ztzWyUWgGttohD7EviAUW xSVaGKeCyCeEt5Av/DO9bzIKiUr+6NfqvZg6muc3I80J+2KDmFqYURTNre87AMTC heuNucn6ldR6wehRjc3XJqalIqinSA2vn96RICoCkI4IXw8sRKiGQBs77n72rN1L 7mE3eUTkFp6YxBr57QBIuVejCxwMzlGUMSA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfeeltddgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtkeertddtfeejnecuhfhrohhmpeetughr ihgrnhcumfhlrghvvghruceorggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtg homheqnecuggftrfgrthhtvghrnhepvefhhefggffhveffudevgfelgeekfffhvddtjeej ffdvtdehvdeitdffveehgfegnecuffhomhgrihhnpehtohifrghrughsuggrthgrshgtih gvnhgtvgdrtghomhenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhl fhhrohhmpegrughrihgrnhdrkhhlrghvvghrsegrkhhlrghvvghrrdgtohhm X-ME-Proxy: Feedback-ID: i76984098:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 May 2023 17:43:10 -0400 (EDT) Message-ID: Date: Wed, 31 May 2023 14:43:09 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.2 Subject: Re: speed up full table scan using psql To: Lian Jiang Cc: pgsql-general@lists.postgresql.org References: Content-Language: en-US From: Adrian Klaver In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 5/31/23 13:57, Lian Jiang wrote: > The command is: psql $db_url -c "copy (select row_to_json(x_tmp_uniq) > from public.mytable x_tmp_uniq) to stdout" > postgres version:  14.7 > Does this mean COPY and java CopyManager may not help since my psql > command already uses copy? I don't think the issue is COPY itself but row_to_json(x_tmp_uniq). This: https://towardsdatascience.com/spark-essentials-how-to-read-and-write-data-with-pyspark-5c45e29227cd indicates Spark can use CSV as an input source. Given that I would just COPY the data out as CSV. > > Regarding pg_dump, it does not support json format which means extra > work is needed to convert the supported format to jsonl (or parquet) so > that they can be imported into snowflake. Still exploring but want to > call it out early. Maybe 'custom' format can be parquet? > > > Thanks > Lian -- Adrian Klaver adrian.klaver@aklaver.com