Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q4ZjV-0005Yp-R9 for pgsql-general@arkaria.postgresql.org; Thu, 01 Jun 2023 04:16:18 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1q4ZjU-0002ew-07 for pgsql-general@arkaria.postgresql.org; Thu, 01 Jun 2023 04:16:16 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q4ZjS-0002bf-J4 for pgsql-general@lists.postgresql.org; Thu, 01 Jun 2023 04:16:15 +0000 Received: from wout5-smtp.messagingengine.com ([64.147.123.21]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1q4ZjN-0030dE-RW for pgsql-general@lists.postgresql.org; Thu, 01 Jun 2023 04:16:13 +0000 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 204CA3200900; Thu, 1 Jun 2023 00:16:06 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Thu, 01 Jun 2023 00:16:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aklaver.com; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1685592965; x=1685679365; bh=8/Eed9GgGwNRmCyZAQpKBo/KsBqgKnyYw75 JSxeJGjQ=; b=HYYj4xACYp5A7P1PTGu7O2lfYsbKmWf17FCykHyFihGz0EGMpwr 9jMgy/PuQnDBymWO/J6VLjlPJUnM9j7uMYzpkBWErmDQtCKigbI9zyx60MBM4YSs bDtHF78DbemcZTlwsOJlIm7ROpkwr1+xCFm4CsB9gAI8JfMeyJrDyFfwKyxVwYmT XCo3/wEiuVvdC1MiQUZ0eCR2WjTxAQXIT/SABQCuulbYWpqqwCUFPnsLmeM3IL66 UZFcjLrdmt6Klj5QafnzRtJzJwfuCe2jCuJOYitNxof43LhoW7CUvsrGWpui0C+Y 9rDYhJYiWMCB8VvKs9obtbPwRM9DwS7aNBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1685592965; x=1685679365; bh=8/Eed9GgGwNRmCyZAQpKBo/KsBqgKnyYw75 JSxeJGjQ=; b=lL1Myd0qIXALLfBAGKuQN8ZvRFFGV7khSXnpZCf0y3v2/A4k0Eu FOSeQQUYtvIipPgISaaAnwWkSNM1w8N3osMspXQ390UnQgrDoZ4dCWa9pidi7EK7 B9yvcTDQVBrlDKIfAhoQyTb06VBREPXtjpq0qiZ3rY+PNZ+mSVtfTkXQcV84rdr5 zOLmZqNKIxeYQC1y0v091SCKY6RyzWgAMV6LcpZNnwpO/cwUWlhHcXdW8nbfP9nd NZlUARfUbBh8DhQx2Jyj8dSK7l20hp8qWc21d3WRXmPkGK7sgxQz/oC88qRMuTpI 36ZDZEk+zxd8QH6s2wIlTdna2s2bFLB3jew== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfeeltddgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtkeertddtfeejnecuhfhrohhmpeetughr ihgrnhcumfhlrghvvghruceorggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtg homheqnecuggftrfgrthhtvghrnhephedugfethfettdehieevteeukedtjedvhfevkefg udefgeevteduvdeuvdeggeegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomheprggurhhirghnrdhklhgrvhgvrhesrghklhgrvhgvrhdrtghomh X-ME-Proxy: Feedback-ID: i76984098:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 1 Jun 2023 00:16:04 -0400 (EDT) Message-ID: Date: Wed, 31 May 2023 21:16:03 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.2 Subject: Re: speed up full table scan using psql Content-Language: en-US To: Lian Jiang Cc: pgsql-general@lists.postgresql.org References: From: Adrian Klaver In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 5/31/23 13:57, Lian Jiang wrote: > The command is: psql $db_url -c "copy (select row_to_json(x_tmp_uniq) > from public.mytable x_tmp_uniq) to stdout" What is taking the stdout and what it is it doing? > postgres version:  14.7 > Does this mean COPY and java CopyManager may not help since my psql > command already uses copy? > > Regarding pg_dump, it does not support json format which means extra > work is needed to convert the supported format to jsonl (or parquet) so > that they can be imported into snowflake. Still exploring but want to > call it out early. Maybe 'custom' format can be parquet? > > > Thanks > Lian -- Adrian Klaver adrian.klaver@aklaver.com