Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q7byM-0004Eb-SG for pgsql-general@arkaria.postgresql.org; Fri, 09 Jun 2023 13:16:11 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1q7byL-0001j9-IJ for pgsql-general@arkaria.postgresql.org; Fri, 09 Jun 2023 13:16:09 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1q7byK-0001j0-VC for pgsql-general@lists.postgresql.org; Fri, 09 Jun 2023 13:16:09 +0000 Received: from mail-oo1-xc2a.google.com ([2607:f8b0:4864:20::c2a]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1q7byH-001Dtd-FL for pgsql-general@lists.postgresql.org; Fri, 09 Jun 2023 13:16:08 +0000 Received: by mail-oo1-xc2a.google.com with SMTP id 006d021491bc7-55ab0f7778cso1159511eaf.1 for ; Fri, 09 Jun 2023 06:16:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joeconway.com; s=google; t=1686316563; x=1688908563; h=content-transfer-encoding:in-reply-to:subject:from:references:to :content-language:user-agent:mime-version:date:message-id:from:to:cc :subject:date:message-id:reply-to; bh=wecS4uyQQwtjZ+Igf20iUi1oOpx8cELax810htKfNZk=; b=az6OPoE3DpB/bxv8xhCo+DQRkC5tlszd7vYq556llmMJGn/0kB+mNfSTHuYEHUqRX9 h2ZXPzlVNkFvZGXkXHG+sE0RN5vT6K4tvkkg9uoMA+oiFLDGVokch2x5dh+dngHxLEYd NjDyBy4T62yXZwAQXl5cMGHLh+sgJNZVz3uc0wmfpVPk9Xi/YaEqpyJc6dmmDD8HLJQy J20ky0AXOa+NsvorPjjrQ6TUtME1OyMP1kM2rTn5y0C1WYF/bor4V/pC2CMiYNODhULx HX/ddnfcLKNcBTVb0f6tYCjNrWWbU72A2Z6XjxHEMeCe/hl9kBO7V7vUyYu0fMvmif0O 71Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686316563; x=1688908563; h=content-transfer-encoding:in-reply-to:subject:from:references:to :content-language:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wecS4uyQQwtjZ+Igf20iUi1oOpx8cELax810htKfNZk=; b=B5/RPuiLFpC3kHI27bLr2eze+t1jFyKYobbduLliVlVqhgxqa9/mQ1k452iddH215a Prp5SJJzypPKmb/WUo31yNsxlGqIihmxtUI+FSO97ScwtFjZq3V1XDeb9lhePawm723Z pIJWfd9hUyPwrZEh8kuFYynHtAUTonCZIPGMr5puot1Zyo3DYKGcv1Zb0OUTYbiAEtqr LfD/XFpuKbfr8N2K0mEQRPlLwIv8T8ah7Rl8wWZXY9TS/T1bZ0h1Mk0iB6JIuTYovN5x H/YIhoh2Sdlw2msTTLNqFwoWeaJFUHhbaDlO2knvnN/ffo/ISO/HrZrI2TNuZGrmw5FX yTxw== X-Gm-Message-State: AC+VfDzHr3OOz3HBiY+Hp3D32rQvSD9IRb1DKrtYLXgrxl3qVS1Tzjvm +Z1pCUBRtE6gmCcwalnXn0QhM14ANiuNpsmfqXU= X-Google-Smtp-Source: ACHHUZ7LGYj2SNr/9H8AyAw4qienHyJzu/0JTgodhRq38pfEUoktXt5lAO4As2YVk70/Nc6brMLOcw== X-Received: by 2002:a05:6808:238a:b0:396:4a76:95b0 with SMTP id bp10-20020a056808238a00b003964a7695b0mr1832755oib.6.1686316563424; Fri, 09 Jun 2023 06:16:03 -0700 (PDT) Received: from [192.168.4.41] (162-239-31-113.lightspeed.dybhfl.sbcglobal.net. [162.239.31.113]) by smtp.gmail.com with ESMTPSA id m14-20020a819e0e000000b0055a21492192sm573674ywj.115.2023.06.09.06.16.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 09 Jun 2023 06:16:03 -0700 (PDT) Message-ID: <626833f2-50e2-342c-e8ea-cb16b0d01785@joeconway.com> Date: Fri, 9 Jun 2023 09:16:02 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US To: Pat Trainor , pgsql-general@lists.postgresql.org References: From: Joe Conway Subject: Re: How To: A large [2D] matrix, 100,000+ rows/columns In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 6/8/23 22:17, Pat Trainor wrote: > Imagine something akin to stocks, where you have a row for every stock, > and a column for every stock. Except where the same stock is the row & > col, a number is at each X-Y (row/column), and that is the big picture. > I need to have a very large matrix to maintain & query, and if not > (1,600 column limit), then how could such data be broken down to work? 100,000 rows * 100,000 columns * 8 bytes (assuming float8) = about 80 GB per matrix if I got the math correct. Is this really a dense matrix or is it sparse? What kind of operations? Does it really need to be stored as such or could it be stored as vectors that are converted to a matrix on the fly when needed? Seems like using python or R makes more sense. Perhaps it might make sense to store the data in Postgres and use plpython or plr. But it is hard to say with more details. -- Joe Conway PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com