pg.ddx.io pgsql-general@postgresql.org mailing list archive
help / color / mirror / Atom feedLarge scale reliable software system
10+ messages / 8 participants
[nested] [flat]
* Large scale reliable software system
@ 2023-06-26 23:48 B M <bmmasprep@gmail.com>
0 siblings, 2 replies; 10+ messages in thread
From: B M @ 2023-06-26 23:48 UTC (permalink / raw)
To: pgsql-general@lists.postgresql.org
Dear all,
After greeting,
I taught PostgreSQL myself and developed a small scale experimental
software system using PostgreSQL in the back-end.
I would like to know your advices to develop a large scale reliable
software system using PostgreSQL in the back-end, through which i can share
the storage with the different system users where they login to the system
through the web application front-end with different passwords and
usernames , save the privacy of each user data, improve overall system
security and performance, achieve fast response, make backups and save the
stored data from loss. The system will be hosted on a cloud.
Thank you in advance.
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Large scale reliable software system
@ 2023-06-27 00:47 Adrian Klaver <adrian.klaver@aklaver.com>
parent: B M <bmmasprep@gmail.com>
1 sibling, 1 reply; 10+ messages in thread
From: Adrian Klaver @ 2023-06-27 00:47 UTC (permalink / raw)
To: B M <bmmasprep@gmail.com>; pgsql-general@lists.postgresql.org
On 6/26/23 16:48, B M wrote:
> Dear all,
>
> After greeting,
>
> I taught PostgreSQL myself and developed a small scale
> experimentalsoftware system using PostgreSQL in the back-end.
>
> I would like to know your advices to develop a large scale reliable
> software system using PostgreSQL in the back-end, through which i can
> share the storage with the different system users where they login to
> the system through the web application front-end with different
> passwords and usernames , save the privacy of each user data, improve
> overall system security and performance, achieve fast response, make
> backups and save the stored data from loss. The system will be hosted on
> a cloud.
https://www.djangoproject.com/
>
> Thank you in advance.
>
--
Adrian Klaver
adrian.klaver@aklaver.com
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Large scale reliable software system
@ 2023-06-27 01:04 Guyren Howe <guyren@gmail.com>
parent: Adrian Klaver <adrian.klaver@aklaver.com>
0 siblings, 1 reply; 10+ messages in thread
From: Guyren Howe @ 2023-06-27 01:04 UTC (permalink / raw)
To: B M <bmmasprep@gmail.com>; pgsql-general@lists.postgresql.org, Adrian Klaver <adrian.klaver@aklaver.com>
This is a reasonable answer, but I want to offer a caveat.
Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.
This heuristic is nearly always mostly wrong.
Guyren G Howe
On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:
> On 6/26/23 16:48, B M wrote:
> > Dear all,
> >
> > After greeting,
> >
> > I taught PostgreSQL myself and developed a small scale
> > experimentalsoftware system using PostgreSQL in the back-end.
> >
> > I would like to know your advices to develop a large scale reliable
> > software system using PostgreSQL in the back-end, through which i can
> > share the storage with the different system users where they login to
> > the system through the web application front-end with different
> > passwords and usernames , save the privacy of each user data, improve
> > overall system security and performance, achieve fast response, make
> > backups and save the stored data from loss. The system will be hosted on
> > a cloud.
>
> https://www.djangoproject.com/
>
> >
> > Thank you in advance.
> >
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>
>
>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Large scale reliable software system
@ 2023-06-27 04:32 Michael Nolan <htfoot@gmail.com>
parent: Guyren Howe <guyren@gmail.com>
0 siblings, 1 reply; 10+ messages in thread
From: Michael Nolan @ 2023-06-27 04:32 UTC (permalink / raw)
To: Guyren Howe <guyren@gmail.com>; +Cc: B M <bmmasprep@gmail.com>; pgsql-general@lists.postgresql.org, Adrian Klaver <adrian.klaver@aklaver.com>
It's not just Ruby, dumb databases are preferred in projects like
WordPress, Drupal and Joomla, too.
Now, if it's because they're used to using MySQL, well maybe that's
not so hard to understand. :-)
On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
>
> This is a reasonable answer, but I want to offer a caveat.
>
> Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.
>
> This heuristic is nearly always mostly wrong.
>
> Guyren G Howe
> On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:
>
> On 6/26/23 16:48, B M wrote:
>
> Dear all,
>
> After greeting,
>
> I taught PostgreSQL myself and developed a small scale
> experimentalsoftware system using PostgreSQL in the back-end.
>
> I would like to know your advices to develop a large scale reliable
> software system using PostgreSQL in the back-end, through which i can
> share the storage with the different system users where they login to
> the system through the web application front-end with different
> passwords and usernames , save the privacy of each user data, improve
> overall system security and performance, achieve fast response, make
> backups and save the stored data from loss. The system will be hosted on
> a cloud.
>
>
> https://www.djangoproject.com/
>
>
> Thank you in advance.
>
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>
>
>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Large scale reliable software system
@ 2023-06-27 05:07 Guyren Howe <guyren@gmail.com>
parent: Michael Nolan <htfoot@gmail.com>
0 siblings, 2 replies; 10+ messages in thread
From: Guyren Howe @ 2023-06-27 05:07 UTC (permalink / raw)
To: ; +Cc: B M <bmmasprep@gmail.com>; pgsql-general@lists.postgresql.org <pgsql-general@lists.postgresql.org>; Adrian Klaver <adrian.klaver@aklaver.com>
Correct. It’s a tragically wrong piece of folk wisdom that’s pretty general across web development communities.
> On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:
>
> It's not just Ruby, dumb databases are preferred in projects like
> WordPress, Drupal and Joomla, too.
>
> Now, if it's because they're used to using MySQL, well maybe that's
> not so hard to understand. :-)
>
> On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
>>
>> This is a reasonable answer, but I want to offer a caveat.
>>
>> Likely because of the influence of the originator of Ruby on Rails, it is close to holy writ in the web development community that the database must be treated as a dumb data bucket and all business logic must be implemented in the Ruby or Python or whatever back end code.
>>
>> This heuristic is nearly always mostly wrong.
>>
>> Guyren G Howe
>> On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>, wrote:
>>
>> On 6/26/23 16:48, B M wrote:
>>
>> Dear all,
>>
>> After greeting,
>>
>> I taught PostgreSQL myself and developed a small scale
>> experimentalsoftware system using PostgreSQL in the back-end.
>>
>> I would like to know your advices to develop a large scale reliable
>> software system using PostgreSQL in the back-end, through which i can
>> share the storage with the different system users where they login to
>> the system through the web application front-end with different
>> passwords and usernames , save the privacy of each user data, improve
>> overall system security and performance, achieve fast response, make
>> backups and save the stored data from loss. The system will be hosted on
>> a cloud.
>>
>>
>> https://www.djangoproject.com/
>>
>>
>> Thank you in advance.
>>
>>
>> --
>> Adrian Klaver
>> adrian.klaver@aklaver.com
>>
>>
>>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Large scale reliable software system
@ 2023-06-27 08:32 Saurabh Agrawal <mail@saurabhagrawal.net>
parent: Guyren Howe <guyren@gmail.com>
1 sibling, 0 replies; 10+ messages in thread
From: Saurabh Agrawal @ 2023-06-27 08:32 UTC (permalink / raw)
To: Guyren Howe <guyren@gmail.com>; +Cc: B M <bmmasprep@gmail.com>; pgsql-general@lists.postgresql.org <pgsql-general@lists.postgresql.org>; Adrian Klaver <adrian.klaver@aklaver.com>
> Correct. It’s a tragically wrong piece of folk wisdom that’s pretty
general across web development communities.
So, what's your advice, and is there some book / resource to get upto speed?
Thanks!
On Tue, Jun 27, 2023 at 10:38 AM Guyren Howe <guyren@gmail.com> wrote:
> Correct. It’s a tragically wrong piece of folk wisdom that’s pretty
> general across web development communities.
>
> On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:
>
> It's not just Ruby, dumb databases are preferred in projects like
> WordPress, Drupal and Joomla, too.
>
> Now, if it's because they're used to using MySQL, well maybe that's
> not so hard to understand. :-)
>
> On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
>
>
> This is a reasonable answer, but I want to offer a caveat.
>
> Likely because of the influence of the originator of Ruby on Rails, it is
> close to holy writ in the web development community that the database must
> be treated as a dumb data bucket and all business logic must be implemented
> in the Ruby or Python or whatever back end code.
>
> This heuristic is nearly always mostly wrong.
>
> Guyren G Howe
> On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>,
> wrote:
>
> On 6/26/23 16:48, B M wrote:
>
> Dear all,
>
> After greeting,
>
> I taught PostgreSQL myself and developed a small scale
> experimentalsoftware system using PostgreSQL in the back-end.
>
> I would like to know your advices to develop a large scale reliable
> software system using PostgreSQL in the back-end, through which i can
> share the storage with the different system users where they login to
> the system through the web application front-end with different
> passwords and usernames , save the privacy of each user data, improve
> overall system security and performance, achieve fast response, make
> backups and save the stored data from loss. The system will be hosted on
> a cloud.
>
>
> https://www.djangoproject.com/
>
>
> Thank you in advance.
>
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>
>
>
>
>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Large scale reliable software system
@ 2023-06-27 10:49 Tony Shelver <tshelver@gmail.com>
parent: Guyren Howe <guyren@gmail.com>
1 sibling, 1 reply; 10+ messages in thread
From: Tony Shelver @ 2023-06-27 10:49 UTC (permalink / raw)
To: pgsql-general <pgsql-general@lists.postgresql.org>
On Tue, 27 Jun 2023 at 07:08, Guyren Howe <guyren@gmail.com> wrote:
> Correct. It’s a tragically wrong piece of folk wisdom that’s pretty
> general across web development communities.
>
> On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:
>
> It's not just Ruby, dumb databases are preferred in projects like
> WordPress, Drupal and Joomla, too.
>
> Now, if it's because they're used to using MySQL, well maybe that's
> not so hard to understand. :-)
>
> On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
>
>
> This is a reasonable answer, but I want to offer a caveat.
>
> Likely because of the influence of the originator of Ruby on Rails, it is
> close to holy writ in the web development community that the database must
> be treated as a dumb data bucket and all business logic must be implemented
> in the Ruby or Python or whatever back end code.
>
> This heuristic is nearly always mostly wrong.
>
> Guyren G Howe
> On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>,
> wrote:
>
> On 6/26/23 16:48, B M wrote:
>
>
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
>
>
>
>
> The accepted front-end developer wisdom of treating the DB as a dumb data
store works under conditions, for example the DB will never be accessed
from a different ORM / framework, and where the performance attributes of
using an ORM with 'standard' datastructures are acceptable.
The moment you need to plug in something like reporting tools, or access
from a different system / API / framework / language / ORM or whatever, the
approach not having rules / views / procedures / whatever built into the
database falls apart.
Other things to consider are performance / load / overhead: we have one
system that involves processing through large amounts of data for reports /
queries. Shipping all that back through the ORM / db interface (ODBC /
JDBC / psycopg2 / whatever for resolution / filtering on the front end
application where SQL / procedures / views could do that in the DB and just
ship back the required data seems counterproductive.
Tony Shelver
>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Large scale reliable software system
@ 2023-06-27 12:15 Avin Kavish <avin@baseboard.ai>
parent: Tony Shelver <tshelver@gmail.com>
0 siblings, 0 replies; 10+ messages in thread
From: Avin Kavish @ 2023-06-27 12:15 UTC (permalink / raw)
To: Tony Shelver <tshelver@gmail.com>; +Cc: pgsql-general <pgsql-general@lists.postgresql.org>
Well, seeing as postgres isn't designed to serve http requests or to run
general purpose code that doesn't involve databases, which you can express
elegantly in python, to answer OP's question - my vote is on the original
answer - Django. It's got everything out of the box - authentication. file
storage. etc etc.
Once you get the application running you can enhance the database as
necessary.
On Tue, Jun 27, 2023 at 4:19 PM Tony Shelver <tshelver@gmail.com> wrote:
>
> On Tue, 27 Jun 2023 at 07:08, Guyren Howe <guyren@gmail.com> wrote:
>
>> Correct. It’s a tragically wrong piece of folk wisdom that’s pretty
>> general across web development communities.
>>
>> On Jun 26, 2023, at 21:32, Michael Nolan <htfoot@gmail.com> wrote:
>>
>> It's not just Ruby, dumb databases are preferred in projects like
>> WordPress, Drupal and Joomla, too.
>>
>> Now, if it's because they're used to using MySQL, well maybe that's
>> not so hard to understand. :-)
>>
>> On Mon, Jun 26, 2023 at 8:05 PM Guyren Howe <guyren@gmail.com> wrote:
>>
>>
>> This is a reasonable answer, but I want to offer a caveat.
>>
>> Likely because of the influence of the originator of Ruby on Rails, it is
>> close to holy writ in the web development community that the database must
>> be treated as a dumb data bucket and all business logic must be implemented
>> in the Ruby or Python or whatever back end code.
>>
>> This heuristic is nearly always mostly wrong.
>>
>> Guyren G Howe
>> On Jun 26, 2023 at 17:48 -0700, Adrian Klaver <adrian.klaver@aklaver.com>,
>> wrote:
>>
>> On 6/26/23 16:48, B M wrote:
>>
>>
>>
>> --
>> Adrian Klaver
>> adrian.klaver@aklaver.com
>>
>>
>>
>>
>> The accepted front-end developer wisdom of treating the DB as a dumb
> data store works under conditions, for example the DB will never be
> accessed from a different ORM / framework, and where the performance
> attributes of using an ORM with 'standard' datastructures are acceptable.
>
> The moment you need to plug in something like reporting tools, or access
> from a different system / API / framework / language / ORM or whatever, the
> approach not having rules / views / procedures / whatever built into the
> database falls apart.
>
> Other things to consider are performance / load / overhead: we have one
> system that involves processing through large amounts of data for reports /
> queries. Shipping all that back through the ORM / db interface (ODBC /
> JDBC / psycopg2 / whatever for resolution / filtering on the front end
> application where SQL / procedures / views could do that in the DB and just
> ship back the required data seems counterproductive.
>
> Tony Shelver
>
>>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Large scale reliable software system
@ 2023-06-27 14:58 Merlin Moncure <mmoncure@gmail.com>
parent: B M <bmmasprep@gmail.com>
1 sibling, 1 reply; 10+ messages in thread
From: Merlin Moncure @ 2023-06-27 14:58 UTC (permalink / raw)
To: B M <bmmasprep@gmail.com>; +Cc: pgsql-general@lists.postgresql.org
On Mon, Jun 26, 2023 at 6:49 PM B M <bmmasprep@gmail.com> wrote:
> Dear all,
>
> After greeting,
>
> I taught PostgreSQL myself and developed a small scale experimental
> software system using PostgreSQL in the back-end.
>
> I would like to know your advices to develop a large scale reliable
> software system using PostgreSQL in the back-end, through which i can share
> the storage with the different system users where they login to the system
> through the web application front-end with different passwords and
> usernames , save the privacy of each user data, improve overall system
> security and performance, achieve fast response, make backups and save the
> stored data from loss. The system will be hosted on a cloud.
>
> Thank you in advance.
>
* your sql is code, and treat it as such, check it into git etc
* robust deployment strategy is essential to scaling team
* write a lot of tests
* become intimate with pg_stat_statements
* keep your transactions as short as possible while preserving safety
* avoid developers who advocate for keeping business logic out of the
database religiously (controversial)
* try to avoid assumptions that only one technical stack interacts with
your database
* do not waste time making ERDs use a tool that generates them (i like
schemaspy)
* test your DR strategy before disaster strikes
* think about security model up front
merlin
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Large scale reliable software system
@ 2023-06-27 15:32 Adrian Klaver <adrian.klaver@aklaver.com>
parent: Merlin Moncure <mmoncure@gmail.com>
0 siblings, 0 replies; 10+ messages in thread
From: Adrian Klaver @ 2023-06-27 15:32 UTC (permalink / raw)
To: Merlin Moncure <mmoncure@gmail.com>; B M <bmmasprep@gmail.com>; +Cc: pgsql-general@lists.postgresql.org
On 6/27/23 07:58, Merlin Moncure wrote:
>
>
> On Mon, Jun 26, 2023 at 6:49 PM B M <bmmasprep@gmail.com
> <mailto:bmmasprep@gmail.com>> wrote:
>
> Dear all,
>
> After greeting,
>
> I taught PostgreSQL myself and developed a small scale
> experimentalsoftware system using PostgreSQL in the back-end.
>
> I would like to know your advices to develop a large scale reliable
> software system using PostgreSQL in the back-end, through which i
> can share the storage with the different system users where they
> login to the system through the web application front-end with
> different passwords and usernames , save the privacy of each user
> data, improve overall system security and performance, achieve fast
> response, make backups and save the stored data from loss. The
> system will be hosted on a cloud.
>
> Thank you in advance.
>
>
> * your sql is code, and treat it as such, check it into git etc
> * robust deployment strategy is essential to scaling team
> * write a lot of tests
> * become intimate with pg_stat_statements
> * keep your transactions as short as possible while preserving safety
> * avoid developers who advocate for keeping business logic out of the
> database religiously (controversial)
> * try to avoid assumptions that only one technical stack interacts with
> your database
> * do not waste time making ERDs use a tool that generates them (i like
> schemaspy)
> * test your DR strategy before disaster strikes
> * think about security model up front
+1
>
> merlin
>
--
Adrian Klaver
adrian.klaver@aklaver.com
^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2023-06-27 15:32 UTC | newest]
Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-06-26 23:48 Large scale reliable software system B M <bmmasprep@gmail.com>
2023-06-27 00:47 ` Adrian Klaver <adrian.klaver@aklaver.com>
2023-06-27 01:04 ` Guyren Howe <guyren@gmail.com>
2023-06-27 04:32 ` Michael Nolan <htfoot@gmail.com>
2023-06-27 05:07 ` Guyren Howe <guyren@gmail.com>
2023-06-27 08:32 ` Saurabh Agrawal <mail@saurabhagrawal.net>
2023-06-27 10:49 ` Tony Shelver <tshelver@gmail.com>
2023-06-27 12:15 ` Avin Kavish <avin@baseboard.ai>
2023-06-27 14:58 ` Merlin Moncure <mmoncure@gmail.com>
2023-06-27 15:32 ` Adrian Klaver <adrian.klaver@aklaver.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox