agora inbox for pgsql-sql@postgresql.org  

help / color / mirror / Atom feed

From: Shekar Tippur <ctippur@gmail.com>
To: pgsql-sql@lists.postgresql.org
Subject: Changing csv structure and corresponding etl
Date: Thu, 13 Dec 2018 08:51:59 -0800
Message-ID: <CED2688C-EC81-4679-8FD8-C64E249099FD@gmail.com> (raw)

Hello,

I am using redshift to store data from csv backups that appear at a regular interval. I use pyspark (psycopg library) to perform etl. The issue is that the csv structure changes in between and the etl job fails. 
The issue I found is that the column gets mixed up.
For example, the original column list was A,B,C,D. In the next iteration, the columns can be A, B, C, X,Y, D
I read from some of the other posts that it is not possible to alter a table to add a column in a particular position within Postgres. 
The table itself currently has millions of rows. Merging tables whenever I get a change may not be a good option. I.e. create a union of existing table and new data, drop the original table and rename union to original.
Any pointers in how to proceed? 

Thanks,
Shekar 
Sent from my iPhone

Message-ID: <CED2688C-EC81-4679-8FD8-C64E249099FD@gmail.com>
Permalink:  ../CED2688C-EC81-4679-8FD8-C64E249099FD@gmail.com/
Also on:    postgresql.org/message-id/CED2688C-EC81-4679-8FD8-C64E249099FD@gmail.com

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-sql@postgresql.org
  Cc: ctippur@gmail.com, pgsql-sql@lists.postgresql.org
  Subject: Re: Changing csv structure and corresponding etl
  In-Reply-To: <CED2688C-EC81-4679-8FD8-C64E249099FD@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox