pg.ddx.io pgsql-performance@postgresql.org mailing list archivehelp / color / mirror / Atom feed
One big table or split data? Writing data. From disk point of view. With a good storage (GBs/s, writing speed) 2+ messages / 1 participants [nested] [flat]
* One big table or split data? Writing data. From disk point of view. With a good storage (GBs/s, writing speed) @ 2018-10-12 16:27 Sam R. <samruohola@yahoo.com> 2018-10-16 03:47 ` Re: One big table or split data? Writing data. From disk point of view. With a good storage (GBs/s, writing speed) Sam R. <samruohola@yahoo.com> 0 siblings, 1 reply; 2+ messages in thread From: Sam R. @ 2018-10-12 16:27 UTC (permalink / raw) To: pgsql-performance@lists.postgresql.org <pgsql-performance@lists.postgresql.org> Hi! Could someone discuss about following? It would be great to hear comments! There is a good storage. According to "fio", write speed could be e.g. 3 GB/s. (It is First time using the command for me, so I am not certain of the real speed with "fio". E.g. with --bs=100m, direct=1, in fio. The measurement result may be faulty also. So, checking still.) Currently there is one table without partitioning. The table contains json data. In PostgreSQL, in Linux. Write speed can be e.g. 300 - 600 MB/s, through PostgreSQL. Measured with dstat while inserting. Shared buffers is large is PostgreSQL. With a storage/disk which "scales", is there some way to write faster to the disk in the system through PostgreSQL? Inside same server. Does splitting data help? Partitioned table / splitting to smaller tables? Should I test it? Change settings somewhere? Block sizes? 8 KB / 16 KB, ... "Dangerous" to change? 2nd question, sharding: If the storage / "disk" scales, could better *disk writing speed* be achieved (in total) with sharding kind of splitting of data? (Same NAS storage, which scales, in use in all shards.)Sharding or use only one server? From pure disk writing speed point of view. BR Sam ^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: One big table or split data? Writing data. From disk point of view. With a good storage (GBs/s, writing speed) 2018-10-12 16:27 One big table or split data? Writing data. From disk point of view. With a good storage (GBs/s, writing speed) Sam R. <samruohola@yahoo.com> @ 2018-10-16 03:47 ` Sam R. <samruohola@yahoo.com> 0 siblings, 0 replies; 2+ messages in thread From: Sam R. @ 2018-10-16 03:47 UTC (permalink / raw) To: Pgsql Performance <pgsql-performance@lists.postgresql.org> Hi! Still, "writing" speed to disk: With "fio" 3 GB / s.With PostgreSQL: 350 -500 MB / s, also with a partitioned table. (and something similar with an other DBMS).Monitored with dstat. With about 12 threads, then it does not get better anymore.Same results with data split to 4 partitions, in a partitioned table. CPU load increased, but not full yet.Same results with open_datasync. BR Sam On pe, lokak. 12, 2018 at 19:27, Sam R.<samruohola@yahoo.com> wrote: Hi! Could someone discuss about following? It would be great to hear comments! There is a good storage. According to "fio", write speed could be e.g. 3 GB/s. (It is First time using the command for me, so I am not certain of the real speed with "fio". E.g. with --bs=100m, direct=1, in fio. The measurement result may be faulty also. So, checking still.) Currently there is one table without partitioning. The table contains json data. In PostgreSQL, in Linux. Write speed can be e.g. 300 - 600 MB/s, through PostgreSQL. Measured with dstat while inserting. Shared buffers is large is PostgreSQL. With a storage/disk which "scales", is there some way to write faster to the disk in the system through PostgreSQL? Inside same server. Does splitting data help? Partitioned table / splitting to smaller tables? Should I test it? Change settings somewhere? Block sizes? 8 KB / 16 KB, ... "Dangerous" to change? 2nd question, sharding: If the storage / "disk" scales, could better *disk writing speed* be achieved (in total) with sharding kind of splitting of data? (Same NAS storage, which scales, in use in all shards.)Sharding or use only one server? From pure disk writing speed point of view. BR Sam ^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2018-10-16 03:47 UTC | newest] Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2018-10-12 16:27 One big table or split data? Writing data. From disk point of view. With a good storage (GBs/s, writing speed) Sam R. <samruohola@yahoo.com> 2018-10-16 03:47 ` Sam R. <samruohola@yahoo.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox