public inbox for [email protected]
help / color / mirror / Atom feedPermission to allow testing harness to send error reports for pgweb directly to mailing list.
10+ messages / 3 participants
[nested] [flat]
* Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-10-03 12:22 Akshat Jaimini <[email protected]>
0 siblings, 1 reply; 10+ messages in thread
From: Akshat Jaimini @ 2023-10-03 12:22 UTC (permalink / raw)
To: [email protected]
Hello everyone,
The testing harness has been successfull at catching multiple issues in
pgweb in the past few months. The reports are for now being sent to me via
email in case of any failure.
Is it ok if I can setup the harness to send the error reports directly to
the mailing list?
Regards,
Akshat Jaimini
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-10-03 18:38 Magnus Hagander <[email protected]>
parent: Akshat Jaimini <[email protected]>
0 siblings, 1 reply; 10+ messages in thread
From: Magnus Hagander @ 2023-10-03 18:38 UTC (permalink / raw)
To: Akshat Jaimini <[email protected]>; +Cc: [email protected]
On Tue, Oct 3, 2023 at 8:24 AM Akshat Jaimini <[email protected]> wrote:
>
> Hello everyone,
> The testing harness has been successfull at catching multiple issues in pgweb in the past few months. The reports are for now being sent to me via email in case of any failure.
>
> Is it ok if I can setup the harness to send the error reports directly to the mailing list?
Hi!
I think that would be reasonable, *if* it is ensured that a report is
only sent once. That is, if it finds the same issue on a later run, it
must not re-send the same thing. How does it work in regards to that
today?
--
Magnus Hagander
Me: https://www.hagander.net/
Work: https://www.redpill-linpro.com/
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-10-03 19:30 Akshat Jaimini <[email protected]>
parent: Magnus Hagander <[email protected]>
0 siblings, 1 reply; 10+ messages in thread
From: Akshat Jaimini @ 2023-10-03 19:30 UTC (permalink / raw)
To: Magnus Hagander <[email protected]>; [email protected]
Hello,
> That is, if it finds the same issue on a later run, it must not re-send
the same thing. How does it work in regards to that today?
As per the current flow whenever a new commit is pushed to the pgweb repo,
the tests are executed. If some tests fail, an error report is sent with
the information of all the failed tests. So if that particular issue has
been resolved, the same report won't be sent but if some other commit is
pushed without resolving that particular issue then that particular error
will be reported again.
As far as the previous test runs have been considered, most of the errors
that have been caught are related to broken urls or some other small
problems that are usually fixed with a single commit so I don't think that
a single report would be re-sent that often.
If this functionality is required then I'll be more than happy to add it.
We can add some sort of a log/cache type structure that checks if a given
report with similar content has been sent to the mailing list in a given
time interval, let's say 15 days for now.
Regards,
Akshat Jaimini
On Wed, Oct 4, 2023 at 12:08 AM Magnus Hagander <[email protected]> wrote:
> On Tue, Oct 3, 2023 at 8:24 AM Akshat Jaimini <[email protected]>
> wrote:
> >
> > Hello everyone,
> > The testing harness has been successfull at catching multiple issues in
> pgweb in the past few months. The reports are for now being sent to me via
> email in case of any failure.
> >
> > Is it ok if I can setup the harness to send the error reports directly
> to the mailing list?
>
>
> Hi!
>
> I think that would be reasonable, *if* it is ensured that a report is
> only sent once. That is, if it finds the same issue on a later run, it
> must not re-send the same thing. How does it work in regards to that
> today?
>
> --
> Magnus Hagander
> Me: https://www.hagander.net/
> Work: https://www.redpill-linpro.com/
>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-10-05 15:02 Daniel Gustafsson <[email protected]>
parent: Akshat Jaimini <[email protected]>
0 siblings, 1 reply; 10+ messages in thread
From: Daniel Gustafsson @ 2023-10-05 15:02 UTC (permalink / raw)
To: Akshat Jaimini <[email protected]>; +Cc: Magnus Hagander <[email protected]>; [email protected]
> On 3 Oct 2023, at 21:30, Akshat Jaimini <[email protected]> wrote:
> > That is, if it finds the same issue on a later run, it must not re-send the same thing. How does it work in regards to that today?
>
> As per the current flow whenever a new commit is pushed to the pgweb repo, the tests are executed. If some tests fail, an error report is sent with the information of all the failed tests. So if that particular issue has been resolved, the same report won't be sent but if some other commit is pushed without resolving that particular issue then that particular error will be reported again.
That doesn't seem terribly great, while bugs and errors should be fixed when
found, sending reports of them repeatedly risk reporting-fatigue. Publishing
this report to a website would handle that I think.
One question, would this test harness detect and report potential security
issues like XSS? If so we should probably limit the audience of the report..
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-10-06 06:05 Akshat Jaimini <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 10+ messages in thread
From: Akshat Jaimini @ 2023-10-06 06:05 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; [email protected]; Magnus Hagander <[email protected]>
> Publishing this report to a website would handle that I think.
I had sent a proposal/tried to start a discussion for this a few days
earlier :
https://www.postgresql.org/message-id/CAMaW3Vg%2BGoQ3JPNo%2BfbLk9ajQv%3D4g4J-bzSAH0OJL7S71_qMig%40ma...
It would actually make the reporting mechanism a lot easier if we can
publish the results to a website. I am currently working on a small
prototype in golang. Currently the reports are being stored as artifacts on
Github actions(only available for 90 days) but we can use services like
Supabase etc to store our reports and present them on the website. Once we
integrate supabase we can get rid of Github artifacts for good.
> One question, would this test harness detect and report potential
security issues like XSS?
Security related tests were not added in the Gsoc timeline but we are
planning to add them. Maybe when we add those tests we can create a
separate section on the proposed website only available to some 'admins'
with all these sensitive reports being displayed there.
We can actually benefit with some more discussion on this.
Regards,
Akshat Jaimini
On Thu, Oct 5, 2023 at 8:32 PM Daniel Gustafsson <[email protected]> wrote:
> > On 3 Oct 2023, at 21:30, Akshat Jaimini <[email protected]> wrote:
>
> > > That is, if it finds the same issue on a later run, it must not
> re-send the same thing. How does it work in regards to that today?
> >
> > As per the current flow whenever a new commit is pushed to the pgweb
> repo, the tests are executed. If some tests fail, an error report is sent
> with the information of all the failed tests. So if that particular issue
> has been resolved, the same report won't be sent but if some other commit
> is pushed without resolving that particular issue then that particular
> error will be reported again.
>
> That doesn't seem terribly great, while bugs and errors should be fixed
> when
> found, sending reports of them repeatedly risk reporting-fatigue.
> Publishing
> this report to a website would handle that I think.
>
> One question, would this test harness detect and report potential security
> issues like XSS? If so we should probably limit the audience of the
> report..
>
>
> --
> Daniel Gustafsson
>
>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-10-06 12:38 Daniel Gustafsson <[email protected]>
parent: Akshat Jaimini <[email protected]>
0 siblings, 1 reply; 10+ messages in thread
From: Daniel Gustafsson @ 2023-10-06 12:38 UTC (permalink / raw)
To: Akshat Jaimini <[email protected]>; +Cc: [email protected]; Magnus Hagander <[email protected]>
> On 6 Oct 2023, at 08:05, Akshat Jaimini <[email protected]> wrote:
>
> > Publishing this report to a website would handle that I think.
> I had sent a proposal/tried to start a discussion for this a few days earlier
It would probably help if you could link to a report from a run of the test
suite. I clicked through the linked repo but I was unable to see an example
testrun.
> > One question, would this test harness detect and report potential security issues like XSS?
> Security related tests were not added in the Gsoc timeline but we are planning to add them. Maybe when we add those tests we can create a separate section on the proposed website only available to some 'admins' with all these sensitive reports being displayed there.
For tests like that we must really think about scope, limiting the report isn't
useful if we publish the tests for anyone to run themselves and thus generate
the report. Malicious actors are no doubt probing the website continuously
regardless of this, but we don't necessarily need to do the job for them.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-10-06 17:12 Akshat Jaimini <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 10+ messages in thread
From: Akshat Jaimini @ 2023-10-06 17:12 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; [email protected]; Magnus Hagander <[email protected]>
> I clicked through the linked repo but I was unable to see an example
testrun.
You can find the reports here:
https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124
. You can check the 'report', 'test-log' and 'failure_logs' artifacts, the
other ones are experimental for now.
> For tests like that we must really think about scope, limiting the report
isn't useful if we publish the tests for anyone to run themselves and thus
generate the report.
> Malicious actors are no doubt probing the website continuously regardless
of this, but we don't necessarily need to do the job for them.
Oh yes, that is a valid point, I guess we might need to separate these
tests then in some private repo? I don't know if this is possible though
but we can think of some other approaches. Because if we keep those tests
publicly available that will just create more problems for us, as you
mentioned in your reply.
I'll try to find more approaches to this because the private repository
does not seem to go with the idea of open source. I might be wrong about
this, so please let me know if I am wrong.
Regards,
Akshat Jaimini
On Fri, Oct 6, 2023 at 6:09 PM Daniel Gustafsson <[email protected]> wrote:
> > On 6 Oct 2023, at 08:05, Akshat Jaimini <[email protected]> wrote:
> >
> > > Publishing this report to a website would handle that I think.
> > I had sent a proposal/tried to start a discussion for this a few days
> earlier
>
> It would probably help if you could link to a report from a run of the test
> suite. I clicked through the linked repo but I was unable to see an
> example
> testrun.
>
> > > One question, would this test harness detect and report potential
> security issues like XSS?
> > Security related tests were not added in the Gsoc timeline but we are
> planning to add them. Maybe when we add those tests we can create a
> separate section on the proposed website only available to some 'admins'
> with all these sensitive reports being displayed there.
>
> For tests like that we must really think about scope, limiting the report
> isn't
> useful if we publish the tests for anyone to run themselves and thus
> generate
> the report. Malicious actors are no doubt probing the website continuously
> regardless of this, but we don't necessarily need to do the job for them.
>
> --
> Daniel Gustafsson
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-10-09 12:42 Daniel Gustafsson <[email protected]>
parent: Akshat Jaimini <[email protected]>
0 siblings, 1 reply; 10+ messages in thread
From: Daniel Gustafsson @ 2023-10-09 12:42 UTC (permalink / raw)
To: Akshat Jaimini <[email protected]>; +Cc: [email protected]; Magnus Hagander <[email protected]>
> On 6 Oct 2023, at 19:12, Akshat Jaimini <[email protected]> wrote:
>
> You can find the reports here: https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124 <https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124; . You can check the 'report', 'test-log' and 'failure_logs' artifacts, the other ones are experimental for now.
Thanks, that was a bit hidden (which is a Github UI issue and not something
against this work).
> I'll try to find more approaches to this because the private repository does not seem to go with the idea of open source. I might be wrong about this, so please let me know if I am wrong.
Just because a project is open source doesn't mean that everything about it
needs to be done in public. Security teams and security processes generally
operate behind closed doors, to avoid leaking vulnerabilities before they can
be patched, and then publish their work and findings once there is a remedy
(either as an advisory with a CVE or some other form).
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-10-10 12:45 Akshat Jaimini <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 10+ messages in thread
From: Akshat Jaimini @ 2023-10-10 12:45 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: [email protected]; Magnus Hagander <[email protected]>
> Security teams and security processes generally operate behind closed
doors, to avoid leaking vulnerabilities before they can be patched, and
then publish their work and findings once there is a remedy.
Ok! So we can then proceed with a private repository maybe? We can fork the
CI setup from the current testing harness and just add the respective
security tests. The generated report can then be accessed by the security
team/any concerned individuals in the deployment team. I'd be happy to host
this repo if needed for now.
> Thanks, that was a bit hidden
Yup this is one of my main concerns with only relying on github actions
also there are multiple runs for the monitoring cron job as well so these
test runs usually get lost in the list. As a temporary solution I had added
the github action run url in the email being sent and the reports attached
with that email.
I have started working on the website to view these reports, will be
sharing the development prototype url shortly.
Regards,
Akshat Jaimini
On Mon, Oct 9, 2023 at 6:12 PM Daniel Gustafsson <[email protected]> wrote:
> > On 6 Oct 2023, at 19:12, Akshat Jaimini <[email protected]> wrote:
>
> >
> > You can find the reports here:
> https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124
> <
> https://github.com/destrex271/pgweb-testing-harness/actions/runs/6189299124;
> . You can check the 'report', 'test-log' and 'failure_logs' artifacts, the
> other ones are experimental for now.
>
> Thanks, that was a bit hidden (which is a Github UI issue and not something
> against this work).
>
> > I'll try to find more approaches to this because the private repository
> does not seem to go with the idea of open source. I might be wrong about
> this, so please let me know if I am wrong.
>
> Just because a project is open source doesn't mean that everything about it
> needs to be done in public. Security teams and security processes
> generally
> operate behind closed doors, to avoid leaking vulnerabilities before they
> can
> be patched, and then publish their work and findings once there is a remedy
> (either as an advisory with a CVE or some other form).
>
> --
> Daniel Gustafsson
>
>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Permission to allow testing harness to send error reports for pgweb directly to mailing list.
@ 2023-11-14 12:57 Akshat Jaimini <[email protected]>
parent: Akshat Jaimini <[email protected]>
0 siblings, 0 replies; 10+ messages in thread
From: Akshat Jaimini @ 2023-11-14 12:57 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: [email protected]; Magnus Hagander <[email protected]>
Hey everyone,
Sorry for the delay(got stuck in some uni stuff). I am working on this
right now, so currently I am planning to use Supabase to store the failure
report content instead of storing entire files. This will also give us a
way to prevent the harness from reporting any repeating errors in
subsequent builds and make it easier to display the data on the website for
the harness(which can then be hosted easily on Cloudflare pages or whatever
is preferred by the community). Does this seem to be in the right direction
or should we opt for some other solution?
Regards,
Akshat Jaimini
^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2023-11-14 12:57 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-10-03 12:22 Permission to allow testing harness to send error reports for pgweb directly to mailing list. Akshat Jaimini <[email protected]>
2023-10-03 18:38 ` Magnus Hagander <[email protected]>
2023-10-03 19:30 ` Akshat Jaimini <[email protected]>
2023-10-05 15:02 ` Daniel Gustafsson <[email protected]>
2023-10-06 06:05 ` Akshat Jaimini <[email protected]>
2023-10-06 12:38 ` Daniel Gustafsson <[email protected]>
2023-10-06 17:12 ` Akshat Jaimini <[email protected]>
2023-10-09 12:42 ` Daniel Gustafsson <[email protected]>
2023-10-10 12:45 ` Akshat Jaimini <[email protected]>
2023-11-14 12:57 ` Akshat Jaimini <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox