Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sPfmV-005jpA-Hc for pgsql-general@arkaria.postgresql.org; Fri, 05 Jul 2024 10:03:07 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sPfmS-008Pr1-Fm for pgsql-general@arkaria.postgresql.org; Fri, 05 Jul 2024 10:03:05 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sPfmS-008Pqr-4f for pgsql-general@lists.postgresql.org; Fri, 05 Jul 2024 10:03:04 +0000 Received: from mail-oa1-x30.google.com ([2001:4860:4864:20::30]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1sPfmQ-000VxP-9i for pgsql-general@lists.postgresql.org; Fri, 05 Jul 2024 10:03:03 +0000 Received: by mail-oa1-x30.google.com with SMTP id 586e51a60fabf-25e0c0df558so686843fac.0 for ; Fri, 05 Jul 2024 03:03:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1720173781; x=1720778581; darn=lists.postgresql.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=/wpKqhva8Oq3NJrzr0anqF5AeoqOyOvezMm0DHEYnn8=; b=PtECx3h209QB5JCOUipG7eCBGCq9I6FM83FX0RduriSmDDbrUWh9s7M60EUP3K3ZqC gyZ/l202gf7av5AAdZcm1F+9eO29GcsXz00hluy4lIzNfvGw0fRsP+xasiZoz55kjYGV ksPRb2ENK4ZFWXjxcHRIpt4yqzl72Zcmc61USzHC8mxkd3OJRjeFrJcHW0gGMLNozkOy kDDjeYH0h6yoXysEpggH5o4LLWDvhmzGoW+pObTicqdpvgwcouArhAVtaUP6bY1QAEtK 7kTl1bXIQP/tQnjVmERkemEswsDUvXAMMEX76io3CfMMtOgjdjuHBUmLQFCXulGSUFqt MWpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720173781; x=1720778581; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/wpKqhva8Oq3NJrzr0anqF5AeoqOyOvezMm0DHEYnn8=; b=J5TR/xqpErt4TK8IlLP2pvlW17RdeaesyorEICoF47DOTPD2DS3kkcqidNQ6mSkrxS spoNFQATn0JNB7qWRwNLo14Z1fDySNEySqf8+wWrHHOql/L7/EqESLI35eYLa+qq9xZv lEy7U5L+lLmFT1Y6Vn0fdRTpBVShLP08cikRkwBhZPsXlII0erq/C3P+JmvRHQI0ih9P Tr7rv8oXs1QZbnwhtUDgHUUf0Zd92wxbpXJSs/s6iZjVuXIwPMYbxMzhreeOscynQmWa duqYWz79TBouRwcV9CT9QFTpF2O0IAGFLxWRctGTxMOR8jYaLpENm/YFBPQWNgGI3C+6 sntw== X-Gm-Message-State: AOJu0Yw1NSL3SydoKC/XES4S12EzoxpW29jltRnU/bpBgtxqzLlTzODO 2bTHBvBwRfJhK9GThIyvPICpO44ACZ82A5tlDw+4tWGsP4+ga02u0O/JKRf6CccJjppeSuk8nfe 0D6n2l7rMu8ZytYS8nG7miM43NCSo7ViC X-Google-Smtp-Source: AGHT+IGjRczEHXEVNWYSI7MOd6ZGsLlDnzKD5maLhpSXl4DNFUV5MEfLVYRoVw/eqHsCdxrW3/rBsAruiT2od4Tbx+w= X-Received: by 2002:a05:6871:5214:b0:25e:14f0:62c2 with SMTP id 586e51a60fabf-25e2b8d97c1mr3541295fac.3.1720173781574; Fri, 05 Jul 2024 03:03:01 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?Sm9zZWYgxaBpbcOhbmVr?= Date: Fri, 5 Jul 2024 12:02:50 +0200 Message-ID: Subject: Re: Load a csv or a avro? To: sud Cc: pgsql-general Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk p=C3=A1 5. 7. 2024 v 11:08 odes=C3=ADlatel sud napsal: > > Hello all, > > Its postgres database. We have option of getting files in csv and/or in a= vro format messages from another system to load it into our postgres databa= se. The volume will be 300million messages per day across many files in bat= ches. > > My question was, which format should we chose in regards to faster data l= oading performance ? and if any other aspects to it also should be consider= ed apart from just loading performance? We are able to load ~300 million rows per one day using CSV and COPY functions (https://www.postgresql.org/docs/current/libpq-copy.html#LIBPQ-CO= PY-SEND).