Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gXUDW-0006HS-5B for pgsql-sql@arkaria.postgresql.org; Thu, 13 Dec 2018 16:52:06 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1gXUDU-0005Dm-Sw for pgsql-sql@arkaria.postgresql.org; Thu, 13 Dec 2018 16:52:04 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gXUDU-0005De-Gx for pgsql-sql@lists.postgresql.org; Thu, 13 Dec 2018 16:52:04 +0000 Received: from mail-ot1-x343.google.com ([2607:f8b0:4864:20::343]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1gXUDS-0000KW-4i for pgsql-sql@lists.postgresql.org; Thu, 13 Dec 2018 16:52:03 +0000 Received: by mail-ot1-x343.google.com with SMTP id s5so2534595oth.7 for ; Thu, 13 Dec 2018 08:52:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:date:subject:message-id :to; bh=L4sdaDawQVbUHk4WQx3FxS5lrtXRwsf+sBqAigVegmo=; b=uAy9hkuMbolUfJaAI7J36LsvDRR551DZHMP+rkIbXcxoFakwmfBsQXNAUGUqvH6cDI ZU6805SI7/arm9NHmCKbox3TrWCDdjAXjK6x+Ii43h8ki6kMzY/kVrT2+S0E+tLdxbt9 xabtQqP/GG5kJdBvuGiCXgrwBVY0SrYIGhREmBjtIMOqUJtbaL9SR0+fucU7S0fSZsOu wmSuP3E6uhvfVe3wG6bAW+DosMefdOle+bJrV+L8Vzlvic7Vlf0HVomWSuX3mEmoVLM7 T/FC1KuCQlcNf+oWuCfCm4tSm+vIrmKvZfNrzQAEnxLI3XG0xyHkVTcPHFJMyAd/cPSy 8iHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version:date :subject:message-id:to; bh=L4sdaDawQVbUHk4WQx3FxS5lrtXRwsf+sBqAigVegmo=; b=mhVYbNdTmoZGlwvFSZww1zFnkIQsZQofXN0gGzg768VPE4mKGNv8lcmin4ijJOD35F dKMi93ZhcSj0VjlUj2waOMdnVwsVXZQpIMhoy1189Es38wdL+fenzvxj7hM8HPxVXjaV F3RnYKuS5zfVHuUHDQfBVo5X4t+gf7ug0N2xrD1TgKFYO40iZYDXc4RyvwNNU3ARxwyi Xkqbb2LGCDcFX2WVLs8oikpylGIvId7flhYVZFFDtfMHCIYvZ7jgSIhcsqWCWBcp2RXu 9Mws48fAWsrqRaz35+l2XR4Y06m1r/9/0PTZFGmqNXSCHsg+Fkn1IhsdnwN/DyhbTNhj 4AsQ== X-Gm-Message-State: AA+aEWZeduTzyMei5s7B1c1S4VrKpsS1HSvIXaDekmhmoOWJ57bvjASK f3GbNoSS4ckRXPNdo1PKS1YYmMijis6GAg== X-Google-Smtp-Source: AFSGD/WT11hC5Avv+kDLHvg2PViAQOoA3QZNn5aNNchE2rlbrVpYIjm6bCFvxx1o13jnT7wULyLkZQ== X-Received: by 2002:a9d:346:: with SMTP id 64mr18365691otv.90.1544719920601; Thu, 13 Dec 2018 08:52:00 -0800 (PST) Received: from ?IPv6:2600:1700:e690:2d20:d434:e0c0:9c39:662c? ([2600:1700:e690:2d20:d434:e0c0:9c39:662c]) by smtp.gmail.com with ESMTPSA id q13sm1050209ota.14.2018.12.13.08.51.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Dec 2018 08:52:00 -0800 (PST) From: Shekar Tippur Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Date: Thu, 13 Dec 2018 08:51:59 -0800 Subject: Changing csv structure and corresponding etl Message-Id: To: pgsql-sql@lists.postgresql.org X-Mailer: iPhone Mail (16B92) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Hello, I am using redshift to store data from csv backups that appear at a regular i= nterval. I use pyspark (psycopg library) to perform etl. The issue is that t= he csv structure changes in between and the etl job fails.=20 The issue I found is that the column gets mixed up. For example, the original column list was A,B,C,D. In the next iteration, th= e columns can be A, B, C, X,Y, D I read from some of the other posts that it is not possible to alter a table= to add a column in a particular position within Postgres.=20 The table itself currently has millions of rows. Merging tables whenever I g= et a change may not be a good option. I.e. create a union of existing table a= nd new data, drop the original table and rename union to original. Any pointers in how to proceed?=20 Thanks, Shekar=20 Sent from my iPhone=