public inbox for [email protected]help / color / mirror / Atom feed
Permission to allow testing harness to send error reports for pgweb directly to mailing list. 10+ messages / 3 participants [nested] [flat]
* Permission to allow testing harness to send error reports for pgweb directly to mailing list. @ 2023-10-03 12:22 Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Magnus Hagander <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Akshat Jaimini @ 2023-10-03 12:22 UTC (permalink / raw) To: [email protected] Hello everyone, The testing harness has been successfull at catching multiple issues in pgweb in the past few months. The reports are for now being sent to me via email in case of any failure. Is it ok if I can setup the harness to send the error reports directly to the mailing list? Regards, Akshat Jaimini ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> @ 2023-10-03 18:38 ` Magnus Hagander <[email protected]> 2023-10-03 19:30 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Magnus Hagander @ 2023-10-03 18:38 UTC (permalink / raw) To: Akshat Jaimini <[email protected]>; +Cc: [email protected] On Tue, Oct 3, 2023 at 8:24 AM Akshat Jaimini <[email protected]> wrote: > > Hello everyone, > The testing harness has been successfull at catching multiple issues in pgweb in the past few months. The reports are for now being sent to me via email in case of any failure. > > Is it ok if I can setup the harness to send the error reports directly to the mailing list? Hi! I think that would be reasonable, *if* it is ensured that a report is only sent once. That is, if it finds the same issue on a later run, it must not re-send the same thing. How does it work in regards to that today? -- Magnus Hagander Me: https://www.hagander.net/ Work: https://www.redpill-linpro.com/ ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Magnus Hagander <[email protected]> @ 2023-10-03 19:30 ` Akshat Jaimini <[email protected]> 2023-10-05 15:02 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Akshat Jaimini @ 2023-10-03 19:30 UTC (permalink / raw) To: Magnus Hagander <[email protected]>; [email protected] Hello, > That is, if it finds the same issue on a later run, it must not re-send the same thing. How does it work in regards to that today? As per the current flow whenever a new commit is pushed to the pgweb repo, the tests are executed. If some tests fail, an error report is sent with the information of all the failed tests. So if that particular issue has been resolved, the same report won't be sent but if some other commit is pushed without resolving that particular issue then that particular error will be reported again. As far as the previous test runs have been considered, most of the errors that have been caught are related to broken urls or some other small problems that are usually fixed with a single commit so I don't think that a single report would be re-sent that often. If this functionality is required then I'll be more than happy to add it. We can add some sort of a log/cache type structure that checks if a given report with similar content has been sent to the mailing list in a given time interval, let's say 15 days for now. Regards, Akshat Jaimini On Wed, Oct 4, 2023 at 12:08 AM Magnus Hagander <[email protected]> wrote: > On Tue, Oct 3, 2023 at 8:24 AM Akshat Jaimini <[email protected]> > wrote: > > > > Hello everyone, > > The testing harness has been successfull at catching multiple issues in > pgweb in the past few months. The reports are for now being sent to me via > email in case of any failure. > > > > Is it ok if I can setup the harness to send the error reports directly > to the mailing list? > > > Hi! > > I think that would be reasonable, *if* it is ensured that a report is > only sent once. That is, if it finds the same issue on a later run, it > must not re-send the same thing. How does it work in regards to that > today? > > -- > Magnus Hagander > Me: https://www.hagander.net/ > Work: https://www.redpill-linpro.com/ > ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Magnus Hagander <[email protected]> 2023-10-03 19:30 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> @ 2023-10-05 15:02 ` Daniel Gustafsson <[email protected]> 2023-10-06 06:05 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Daniel Gustafsson @ 2023-10-05 15:02 UTC (permalink / raw) To: Akshat Jaimini <[email protected]>; +Cc: Magnus Hagander <[email protected]>; [email protected] > On 3 Oct 2023, at 21:30, Akshat Jaimini <[email protected]> wrote: > > That is, if it finds the same issue on a later run, it must not re-send the same thing. How does it work in regards to that today? > > As per the current flow whenever a new commit is pushed to the pgweb repo, the tests are executed. If some tests fail, an error report is sent with the information of all the failed tests. So if that particular issue has been resolved, the same report won't be sent but if some other commit is pushed without resolving that particular issue then that particular error will be reported again. That doesn't seem terribly great, while bugs and errors should be fixed when found, sending reports of them repeatedly risk reporting-fatigue. Publishing this report to a website would handle that I think. One question, would this test harness detect and report potential security issues like XSS? If so we should probably limit the audience of the report.. -- Daniel Gustafsson ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Magnus Hagander <[email protected]> 2023-10-03 19:30 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-05 15:02 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> @ 2023-10-06 06:05 ` Akshat Jaimini <[email protected]> 2023-10-06 12:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Akshat Jaimini @ 2023-10-06 06:05 UTC (permalink / raw) To: Daniel Gustafsson <[email protected]>; [email protected]; Magnus Hagander <[email protected]> > Publishing this report to a website would handle that I think. I had sent a proposal/tried to start a discussion for this a few days earlier : https://www.postgresql.org/message-id/CAMaW3Vg%2BGoQ3JPNo%2BfbLk9ajQv%3D4g4J-bzSAH0OJL7S71_qMig%40ma... It would actually make the reporting mechanism a lot easier if we can publish the results to a website. I am currently working on a small prototype in golang. Currently the reports are being stored as artifacts on Github actions(only available for 90 days) but we can use services like Supabase etc to store our reports and present them on the website. Once we integrate supabase we can get rid of Github artifacts for good. > One question, would this test harness detect and report potential security issues like XSS? Security related tests were not added in the Gsoc timeline but we are planning to add them. Maybe when we add those tests we can create a separate section on the proposed website only available to some 'admins' with all these sensitive reports being displayed there. We can actually benefit with some more discussion on this. Regards, Akshat Jaimini On Thu, Oct 5, 2023 at 8:32 PM Daniel Gustafsson <[email protected]> wrote: > > On 3 Oct 2023, at 21:30, Akshat Jaimini <[email protected]> wrote: > > > > That is, if it finds the same issue on a later run, it must not > re-send the same thing. How does it work in regards to that today? > > > > As per the current flow whenever a new commit is pushed to the pgweb > repo, the tests are executed. If some tests fail, an error report is sent > with the information of all the failed tests. So if that particular issue > has been resolved, the same report won't be sent but if some other commit > is pushed without resolving that particular issue then that particular > error will be reported again. > > That doesn't seem terribly great, while bugs and errors should be fixed > when > found, sending reports of them repeatedly risk reporting-fatigue. > Publishing > this report to a website would handle that I think. > > One question, would this test harness detect and report potential security > issues like XSS? If so we should probably limit the audience of the > report.. > > > -- > Daniel Gustafsson > > ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Magnus Hagander <[email protected]> 2023-10-03 19:30 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-05 15:02 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 2023-10-06 06:05 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> @ 2023-10-06 12:38 ` Daniel Gustafsson <[email protected]> 2023-10-06 17:12 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Daniel Gustafsson @ 2023-10-06 12:38 UTC (permalink / raw) To: Akshat Jaimini <[email protected]>; +Cc: [email protected]; Magnus Hagander <[email protected]> > On 6 Oct 2023, at 08:05, Akshat Jaimini <[email protected]> wrote: > > > Publishing this report to a website would handle that I think. > I had sent a proposal/tried to start a discussion for this a few days earlier It would probably help if you could link to a report from a run of the test suite. I clicked through the linked repo but I was unable to see an example testrun. > > One question, would this test harness detect and report potential security issues like XSS? > Security related tests were not added in the Gsoc timeline but we are planning to add them. Maybe when we add those tests we can create a separate section on the proposed website only available to some 'admins' with all these sensitive reports being displayed there. For tests like that we must really think about scope, limiting the report isn't useful if we publish the tests for anyone to run themselves and thus generate the report. Malicious actors are no doubt probing the website continuously regardless of this, but we don't necessarily need to do the job for them. -- Daniel Gustafsson ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Magnus Hagander <[email protected]> 2023-10-03 19:30 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-05 15:02 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 2023-10-06 06:05 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-06 12:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> @ 2023-10-06 17:12 ` Akshat Jaimini <[email protected]> 2023-10-09 12:42 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Akshat Jaimini @ 2023-10-06 17:12 UTC (permalink / raw) To: Daniel Gustafsson <[email protected]>; [email protected]; Magnus Hagander <[email protected]> > I clicked through the linked repo but I was unable to see an example testrun. You can find the reports here: https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124 . You can check the 'report', 'test-log' and 'failure_logs' artifacts, the other ones are experimental for now. > For tests like that we must really think about scope, limiting the report isn't useful if we publish the tests for anyone to run themselves and thus generate the report. > Malicious actors are no doubt probing the website continuously regardless of this, but we don't necessarily need to do the job for them. Oh yes, that is a valid point, I guess we might need to separate these tests then in some private repo? I don't know if this is possible though but we can think of some other approaches. Because if we keep those tests publicly available that will just create more problems for us, as you mentioned in your reply. I'll try to find more approaches to this because the private repository does not seem to go with the idea of open source. I might be wrong about this, so please let me know if I am wrong. Regards, Akshat Jaimini On Fri, Oct 6, 2023 at 6:09 PM Daniel Gustafsson <[email protected]> wrote: > > On 6 Oct 2023, at 08:05, Akshat Jaimini <[email protected]> wrote: > > > > > Publishing this report to a website would handle that I think. > > I had sent a proposal/tried to start a discussion for this a few days > earlier > > It would probably help if you could link to a report from a run of the test > suite. I clicked through the linked repo but I was unable to see an > example > testrun. > > > > One question, would this test harness detect and report potential > security issues like XSS? > > Security related tests were not added in the Gsoc timeline but we are > planning to add them. Maybe when we add those tests we can create a > separate section on the proposed website only available to some 'admins' > with all these sensitive reports being displayed there. > > For tests like that we must really think about scope, limiting the report > isn't > useful if we publish the tests for anyone to run themselves and thus > generate > the report. Malicious actors are no doubt probing the website continuously > regardless of this, but we don't necessarily need to do the job for them. > > -- > Daniel Gustafsson ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Magnus Hagander <[email protected]> 2023-10-03 19:30 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-05 15:02 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 2023-10-06 06:05 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-06 12:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 2023-10-06 17:12 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> @ 2023-10-09 12:42 ` Daniel Gustafsson <[email protected]> 2023-10-10 12:45 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Daniel Gustafsson @ 2023-10-09 12:42 UTC (permalink / raw) To: Akshat Jaimini <[email protected]>; +Cc: [email protected]; Magnus Hagander <[email protected]> > On 6 Oct 2023, at 19:12, Akshat Jaimini <[email protected]> wrote: > > You can find the reports here: https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124 <https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124; . You can check the 'report', 'test-log' and 'failure_logs' artifacts, the other ones are experimental for now. Thanks, that was a bit hidden (which is a Github UI issue and not something against this work). > I'll try to find more approaches to this because the private repository does not seem to go with the idea of open source. I might be wrong about this, so please let me know if I am wrong. Just because a project is open source doesn't mean that everything about it needs to be done in public. Security teams and security processes generally operate behind closed doors, to avoid leaking vulnerabilities before they can be patched, and then publish their work and findings once there is a remedy (either as an advisory with a CVE or some other form). -- Daniel Gustafsson ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Magnus Hagander <[email protected]> 2023-10-03 19:30 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-05 15:02 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 2023-10-06 06:05 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-06 12:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 2023-10-06 17:12 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-09 12:42 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> @ 2023-10-10 12:45 ` Akshat Jaimini <[email protected]> 2023-11-14 12:57 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 0 siblings, 1 reply; 10+ messages in thread From: Akshat Jaimini @ 2023-10-10 12:45 UTC (permalink / raw) To: Daniel Gustafsson <[email protected]>; +Cc: [email protected]; Magnus Hagander <[email protected]> > Security teams and security processes generally operate behind closed doors, to avoid leaking vulnerabilities before they can be patched, and then publish their work and findings once there is a remedy. Ok! So we can then proceed with a private repository maybe? We can fork the CI setup from the current testing harness and just add the respective security tests. The generated report can then be accessed by the security team/any concerned individuals in the deployment team. I'd be happy to host this repo if needed for now. > Thanks, that was a bit hidden Yup this is one of my main concerns with only relying on github actions also there are multiple runs for the monitoring cron job as well so these test runs usually get lost in the list. As a temporary solution I had added the github action run url in the email being sent and the reports attached with that email. I have started working on the website to view these reports, will be sharing the development prototype url shortly. Regards, Akshat Jaimini On Mon, Oct 9, 2023 at 6:12 PM Daniel Gustafsson <[email protected]> wrote: > > On 6 Oct 2023, at 19:12, Akshat Jaimini <[email protected]> wrote: > > > > > You can find the reports here: > https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124 > < > https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124; > . You can check the 'report', 'test-log' and 'failure_logs' artifacts, the > other ones are experimental for now. > > Thanks, that was a bit hidden (which is a Github UI issue and not something > against this work). > > > I'll try to find more approaches to this because the private repository > does not seem to go with the idea of open source. I might be wrong about > this, so please let me know if I am wrong. > > Just because a project is open source doesn't mean that everything about it > needs to be done in public. Security teams and security processes > generally > operate behind closed doors, to avoid leaking vulnerabilities before they > can > be patched, and then publish their work and findings once there is a remedy > (either as an advisory with a CVE or some other form). > > -- > Daniel Gustafsson > > ^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Magnus Hagander <[email protected]> 2023-10-03 19:30 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-05 15:02 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 2023-10-06 06:05 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-06 12:38 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 2023-10-06 17:12 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-09 12:42 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Daniel Gustafsson <[email protected]> 2023-10-10 12:45 ` Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> @ 2023-11-14 12:57 ` Akshat Jaimini <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Akshat Jaimini @ 2023-11-14 12:57 UTC (permalink / raw) To: Daniel Gustafsson <[email protected]>; +Cc: [email protected]; Magnus Hagander <[email protected]> Hey everyone, Sorry for the delay(got stuck in some uni stuff). I am working on this right now, so currently I am planning to use Supabase to store the failure report content instead of storing entire files. This will also give us a way to prevent the harness from reporting any repeating errors in subsequent builds and make it easier to display the data on the website for the harness(which can then be hosted easily on Cloudflare pages or whatever is preferred by the community). Does this seem to be in the right direction or should we opt for some other solution? Regards, Akshat Jaimini ^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2023-11-14 12:57 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]> 2023-10-03 18:38 ` Magnus Hagander <[email protected]> 2023-10-03 19:30 ` Akshat Jaimini <[email protected]> 2023-10-05 15:02 ` Daniel Gustafsson <[email protected]> 2023-10-06 06:05 ` Akshat Jaimini <[email protected]> 2023-10-06 12:38 ` Daniel Gustafsson <[email protected]> 2023-10-06 17:12 ` Akshat Jaimini <[email protected]> 2023-10-09 12:42 ` Daniel Gustafsson <[email protected]> 2023-10-10 12:45 ` Akshat Jaimini <[email protected]> 2023-11-14 12:57 ` Akshat Jaimini <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox