Feedback-ID: i76984098:Fastmail
Message-ID: <11d5753c-578e-41a2-af17-6de956f03058@aklaver.com>
Date: Fri, 5 Jul 2024 07:54:49 -0700
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Load a csv or a avro?
To: sud <suds1434@gmail.com>,
 pgsql-general <pgsql-general@lists.postgresql.org>
References: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>
Content-Language: en-US
From: Adrian Klaver <adrian.klaver@aklaver.com>
In-Reply-To: <CAD=mzVUo9UpTw7F_8HKDK19ZmO6tE6Cfa4T-7i1J_QGfi6NpOw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/11d5753c-578e-41a2-af17-6de956f03058%40aklaver.com>
Precedence: bulk

On 7/5/24 02:08, sud wrote:
> Hello all,
> 
> Its postgres database. We have option of getting files in csv and/or in 
> avro format messages from another system to load it into our postgres 
> database. The volume will be 300million messages per day across many 
> files in batches.

Are dumping the entire contents of each file or are you pulling a 
portion of the data out?

> 
> My question was, which format should we chose in regards to faster data 
> loading performance ? and if any other aspects to it also should be 
> considered apart from just loading performance?
> 

-- 
Adrian Klaver
adrian.klaver@aklaver.com