Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qWfxA-0041Zu-AA for pgsql-general@arkaria.postgresql.org; Thu, 17 Aug 2023 16:34:32 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1qWfx7-002D7W-7H for pgsql-general@arkaria.postgresql.org; Thu, 17 Aug 2023 16:34:29 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qWfx6-002D7O-Rw for pgsql-general@lists.postgresql.org; Thu, 17 Aug 2023 16:34:29 +0000 Received: from mail-pj1-x1029.google.com ([2607:f8b0:4864:20::1029]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1qWfx2-000hI3-M7 for pgsql-general@lists.postgresql.org; Thu, 17 Aug 2023 16:34:28 +0000 Received: by mail-pj1-x1029.google.com with SMTP id 98e67ed59e1d1-26b4dea0000so3563211a91.0 for ; Thu, 17 Aug 2023 09:34:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692290062; x=1692894862; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=OBJRo4NaPgMghae7yEOETMMydHndo1iIBrNp/27r2m0=; b=CXUhelnGhrmAO58ntKYCNnQJmLYcvsWa7tdvdx4DixLyfKjWqfvzeqM4LtLgFPjX/W i23TUqM1p/uoTrA6KPye7WR0pDTjVr1rdBCltNmzdCo44kFOv+KEt9s+IqYmpGXiPlXB MQUg2bgkpw+BcTlHmjRPlh/A80YV6PZvyDUHla8UwSVY8JLnbUIqHsPwmECCZMJk8KA1 jASiIOswJAWLv/xlwHSiqSjzncpEGrGFE8HBYOfoWvYBJV932EB4nYkIrK9K2MYVNYL5 irK3WJK76FRrNEckV7gp89cDr+lnP+Ajvx5/vfvd0ZLWNnNP0oLi8NccyM7S2nlBT4Lg 2l7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692290062; x=1692894862; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OBJRo4NaPgMghae7yEOETMMydHndo1iIBrNp/27r2m0=; b=Tp9cJFy68ga3qzultmIdylOFFb4m8kJDYrXGo8uFvPjo1AUsc/PuCHkDV8Cg/WS9xr JCJE/JK4FEdEe0LVXv2olwHJGmzxiVYhqFcbeSNtT9IIKfqDUFkagZl3jailibIk/I0w U0jmKbVSHKxiO9X3nua8LUshg8PTjDRXd0Mp0booOIaIgRZi0jQ3xl2RFdn4Deqflt00 dv7mCIpRP1pxkC1wbHAJeLKFWGv6pf4h+4YKWMw/uiCtZHBlCiqU/K897eMAnhb64DKW h2O1d2whv/y4a0ObPxplVd73ef+koC0fCv+qmNQkl8OICYM0UthPRCy6e/cnqWWl2OR4 dupw== X-Gm-Message-State: AOJu0YxpPuyFz/Jknfgf51+w67QJJtndHtZGohbSD4e1WY4O+WhQnhSj pgReceXzbAkb/0YhNM8CmCGkLIZ3L4U= X-Google-Smtp-Source: AGHT+IHlsIYpFIGvcHF7rz2zDyE2Z172MjZLaRgw3cUxCz9HmVX+z0oDfZwrmSf9Hj7xwdvMqyfajw== X-Received: by 2002:a17:90a:9ed:b0:267:a859:dfef with SMTP id 100-20020a17090a09ed00b00267a859dfefmr4758884pjo.27.1692290061838; Thu, 17 Aug 2023 09:34:21 -0700 (PDT) Received: from ?IPV6:2601:681:5a00:f790:9595:5ffa:5ed4:6f67? ([2601:681:5a00:f790:9595:5ffa:5ed4:6f67]) by smtp.gmail.com with ESMTPSA id mg18-20020a17090b371200b00262d9b4b527sm2021pjb.52.2023.08.17.09.34.20 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 17 Aug 2023 09:34:21 -0700 (PDT) Message-ID: <7adb97ac-5433-054d-32a0-a1046f7212da@gmail.com> Date: Thu, 17 Aug 2023 10:34:20 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: Fatal Error : Invalid Memory alloc request size 1236252631 Content-Language: en-CA To: pgsql-general@lists.postgresql.org References: <6BDA46BE-B926-4BF7-AEBF-C9153D59A51D@a-kretschmer.de> <1022628.1692069795@sss.pgh.pa.us> From: Rob Sargent In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 8/17/23 07:35, Sai Teja wrote: > Hi Team, > > Even I used postgreSQL Large Objects by referring this link to store > and retrieve large files (As bytea not working) > https://www.postgresql.org/docs/current/largeobjects.html > > But even now I am unable to fetch the data at once from large objects > > select lo_get(oid); > > Here I'm getting the same error message. > > But if I use select data from pg_large_object where loid = 49374 > Then I can fetch the data but in page wise (data splitting into rows > of each size 2KB) > > So, here how can I fetch the data at single step rather than page by > page without any error. > > And I'm just wondering how do many applications storing huge amount of > data in GBs? I know that there is 1GB limit for each field set by > postgreSQL. If so, how to deal with these kind of situations? Would > like to know about this to deal with real time scenarios. > > We need to store large content (huge volume of data) and retrieve it. > Currently It is not happening due to limit of field size set by > postgreSQL. > > Would request to share your insights and suggestions on this to help > me for resolving this issue. > > > My first attempt at handling large payload was to use Java Selector directly in my app.  This worked but manually chunking the data was tricky.  I switched to using Tomcat and it handles large http(s) payloads seamlessly.