Subject: RFC: a new try for an official community approved certification

From: Jaime Casanova <[email-masked]>
Date: Thu, 1 Jun 2023 17:04:43 -0500

Hi everyone,

At PgCon I mentioned a was going to create this thread so here we go:

== why do we need one?

The fact that there is no official certification is a real problem
because anyone can offer theirs and have bad answers or confuse people
about how things work. You can think that people looking for a
certification should ensure first but what if the company has a good
reputation because other certifications they have are good?

And this is an example of this happening, recently I got a
certification from a less-known company, and I found at least one
question in which there weren't any correct answers so I had to choose
the answer that was at least partially correct. And of course this
could happen to anyone.

== we have tried… and failed

Several years ago there was a project for creating a pensum that all
companies that wanted to offer the "official" certification should
comply with. But being postgres developed and improved so fast we
ended up with half the pensum and we already had to adapt part of it
because of the changes in the new version.

But of course there was the problem of not really prepared trainers, a
know a guy that was certified by a well known postgres company but he
clearly understood wrong somethings because he was teaching that you
can restore a pg_dump'ed database and then apply wals from the
original server and have PITR (doh).

== so the problems

- mantaining/updating a pensum would be difficult
- is not only a matter of covering the pensum, it should be done right
- other problems are:
-- current certifications are not neutral and cover also tools from
the company doing the certification
-- even if they have the same name, certifications cover normally
different subjects in different companies

== so, my idea

- Let's focus on creating just one certification covering only
core/contrib functionalities of postgres, and let companies have their
own certifications for third party products.
- don't try to force the pensum, just let's create a database of
validated questions and validated answers. If we focus on only one
certification (something like associate postgresql or something like
that) we can let companies decide if they want to have 1, 2 or 3
modules to prepare people to give the exam.
- Who creates the questions? And who validates them? The answer to
both questions is "trainers" from companies that want to offer the
exam. So, a trainer creates a question and another trainer (hopefully
from another company) validates the question and the answers for the
question.
Questions and answers should also be validated for postgres version
and operating system.

If a company cannot successfully prepare people for the exam they
could check if they are teaching right (I mean that they are not
saying old/not exact things).

So, having a certification made from a pool of validated
questions/answers don't only will help companies choosing trainings
but will in effect improve the quality of trainers.

There are a lot of open questions yet, like where the exams will be
taken. In the community infrastructure? in the company's?

but I guess we have still enough for start commenting.

--
Jaime Casanova
Director de Servicios Profesionales
SYSTEMGUARDS - Consultores de PostgreSQL

---

From: Adrian Klaver <[email-masked]>
Date: Thu, 1 Jun 2023 15:31:32 -0700

> but I guess we have still enough for start commenting.
1) Who oversees all of the above?

2) How are the questions and answers developed without leaking the 
information out to test takers?

3) Does core mean just the community Postgres?

-- 
Adrian Klaver
[email-masked]

---

From: Valeria Kaplan <[email-masked]>
Date: Fri, 2 Jun 2023 10:57:18 +0100

Thank you for bringing this topic to discussion, Jaime!

To add to Adrian's questions here are my 5 cents

Postgres is a community that consists of people from different companies
with variety of interests:
- Postgres/open source/community-focused interests
- personal interests
- their company's interests

To create a fully "unbiased" certification there is a need for a working
group whose interests would be fully aligned.
I must admit, I'm a bit pessimistic about a unified Postgres certification
as it would put people who would be working on it in a difficult situation
when they're serving all the abovementioned interests.

Valeria

---

From: Dave Cramer <[email-masked]>
Date: Fri, 2 Jun 2023 07:04:41 -0400

> group whose interests would be fully aligned.
Agreed

> when they're serving all the abovementioned interests.

My pessimism is more around how you keep the answers from becoming public
knowledge. I don't think that should stop us from working on this however.

Dave

---

From: "Gunnar \"Nick\" Bluth" <[email-masked]>
Date: Fri, 2 Jun 2023 13:11:23 +0200

> information out to test takers?
I guess it'd make sense to ask the LPI folks how they handle this.

Maybe we could even slip under their umbrella, once the community agrees 
on a (closed) group working on the actual exam questions...?

Just my 2p,
-- 
Gunnar "Nick" Bluth

Eimermacherweg 106
D-48159 Münster

Mobil +49 172 8853339
Email: [email-masked]
__________________________________________________________________________
"Ceterum censeo SystemD esse delendam" - Cato

---

From: Katherine Mcmillan <[email-masked]>
Date: Fri, 2 Jun 2023 13:13:35 +0000

Hello all,

Not sure about this idea, but possibly a better one than having a single company (ex. Red Hat and EDB come to mind) run the show for certifications/competency rating.  I like it from this respect.

I will love the idea if you are able to, somehow, offer the training/testing in languages other than English.  PostgreSQL is an international open source project and English-only training/testing is not very appropriate.  This gives a huge advantage to the community's native English speakers (the goal here is to produce something 'unbiased' I thought).

Perhaps develop it in English, with input from those from non-English speaking countries on your working group, and then have it translated into multiple languages (also: be careful how you prioritize these, but if you have, say, a member from India on your working group, perhaps Hindi becomes one of the first).

Sincerely,
Katie
________________________________
From: Dave Cramer <[email-masked]>
Sent: 02 June 2023 07:04
To: Valeria Kaplan <[email-masked]>
Cc: Jaime Casanova <[email-masked]>; Adrian Klaver <[email-masked]>; PostgreSQL Advocacy <[email-masked]>
Subject: Re: RFC: a new try for an official community approved certification

Attention : courriel externe | external email

On Fri, 2 Jun 2023 at 05:57, Valeria Kaplan <[email-masked]<mailto:[email-masked]>> wrote:
Thank you for bringing this topic to discussion, Jaime!

Postgres is a community that consists of people from different companies with variety of interests:
- Postgres/open source/community-focused interests
- personal interests
- their company's interests

To create a fully "unbiased" certification there is a need for a working group whose interests would be fully aligned.
Agreed

I must admit, I'm a bit pessimistic about a unified Postgres certification as it would put people who would be working on it in a difficult situation when they're serving all the abovementioned interests.

My pessimism is more around how you keep the answers from becoming public knowledge. I don't think that should stop us from working on this however.

Dave

Valeria

On Thu, Jun 1, 2023 at 11:31 PM Adrian Klaver <[email-masked]<mailto:[email-masked]>> wrote:

--
Adrian Klaver
[email-masked]<mailto:[email-masked]>

---

From: Guillaume Lelarge <[email-masked]>
Date: Fri, 2 Jun 2023 16:06:26 +0200

Hi,

Le ven. 2 juin 2023 à 15:13, Katherine Mcmillan <[email-masked]> a
écrit :

> Hello all,
>
> Not sure about this idea, but possibly a better one than having a single
> company (ex. Red Hat and EDB come to mind) run the show for
> certifications/competency rating.  I like it from this respect.
>
>
There are other companies doing PostgreSQL certification. SRA in Japan, and
Dalibo in France (disclaimer: I work for Dalibo) offer certification exams
for their customers.

I will love the idea if you are able to, somehow, offer the
> training/testing in languages other than English.  PostgreSQL is an
> international open source project and English-only training/testing is not
> very appropriate.  This gives a huge advantage to the community's native
> English speakers (the goal here is to produce something 'unbiased' I
> thought).
>
> Perhaps develop it in English, with input from those from non-English
> speaking countries on your working group, and then have it translated into
> multiple languages (also: be careful how you prioritize these, but if you
> have, say, a member from India on your working group, perhaps Hindi becomes
> one of the first).
>
>
Multiple languages would be a must, I totally agree. Dalibo did its own
certification because the community didn't seem to be able to work on a
unified one, despite many tries. I'd love to see a unified Postgres
certification, and we (Dalibo) would gladly work on this project, but I
also am kinda skeptical of the outcome.

Regards.

-- 
Guillaume.

---

From: Katherine Mcmillan <[email-masked]>
Date: Fri, 2 Jun 2023 14:35:17 +0000

Hi Guillaume,

That's interesting - the PostgreSQL training/testing that Dalibo is offering is in French?

I wonder if we could work out an agreement with Dalibo and build on the training that you have already developed, rather than re-inventing the wheel, so to speak.  That way we're kind of building the unified or standardized English and French training/testing at the same time ('unified' or 'standardized' sounds better to me than 'community developed').

Just some thoughts :)

Sincerely,
Katie
________________________________
From: Guillaume Lelarge <[email-masked]>
Sent: 02 June 2023 10:06
To: Katherine Mcmillan <[email-masked]>
Cc: Valeria Kaplan <[email-masked]>; Adrian Klaver <[email-masked]>; Jaime Casanova <[email-masked]>; PostgreSQL Advocacy <[email-masked]>
Subject: Re: RFC: a new try for an official community approved certification

Le ven. 2 juin 2023 à 15:13, Katherine Mcmillan <[email-masked]<mailto:[email-masked]>> a écrit :
Hello all,

There are other companies doing PostgreSQL certification. SRA in Japan, and Dalibo in France (disclaimer: I work for Dalibo) offer certification exams for their customers.

Multiple languages would be a must, I totally agree. Dalibo did its own certification because the community didn't seem to be able to work on a unified one, despite many tries. I'd love to see a unified Postgres certification, and we (Dalibo) would gladly work on this project, but I also am kinda skeptical of the outcome.

Regards.

--
Guillaume.

---

From: Guillaume Lelarge <[email-masked]>
Date: Fri, 2 Jun 2023 16:44:54 +0200

Le ven. 2 juin 2023 à 16:35, Katherine Mcmillan <[email-masked]> a

> That's interesting - the PostgreSQL training/testing that Dalibo is
Yes, that's only available in french.

> I wonder if we could work out an agreement with Dalibo and build on the
> training that you have already developed, rather than re-inventing the
> wheel, so to speak.  That way we're kind of building the unified or
> standardized English and French training/testing at the same time
> ('unified' or 'standardized' sounds better to me than 'community
> developed').
>
>
I don't know, but it could be interesting. I'm not sure we did the work to
update it to v15, but it sure covers v14. Anyway, I'm not the one who could
decide such things :)

> Just some thoughts :)
>
> Sincerely,
> Katie
> ------------------------------
> *From:* Guillaume Lelarge <[email-masked]>
> *Sent:* 02 June 2023 10:06
> *To:* Katherine Mcmillan <[email-masked]>
> *Cc:* Valeria Kaplan <[email-masked]>; Adrian Klaver <
> [email-masked]>; Jaime Casanova <[email-masked]>;
> PostgreSQL Advocacy <[email-masked]>
> *Subject:* Re: RFC: a new try for an official community approved
> certification
>
> *Attention : courriel externe | external email*
> Hi,
>
> There are other companies doing PostgreSQL certification. SRA in Japan,
> and Dalibo in France (disclaimer: I work for Dalibo) offer certification
> exams for their customers.
>

-- 
Guillaume.

---

From: Dave Cramer <[email-masked]>
Date: Fri, 2 Jun 2023 10:54:11 -0400

On Fri, 2 Jun 2023 at 10:45, Guillaume Lelarge <[email-masked]>
wrote:

> I don't think we are discussing training. This could be an option for
companies to offer training for compensation.
I thought we were only discussing certification. In my mind that is just
the testing.

Dave

---

From: Guillaume Lelarge <[email-masked]>
Date: Fri, 2 Jun 2023 17:06:02 +0200

Le ven. 2 juin 2023 à 16:54, Dave Cramer <[email-masked]> a

> I thought we were only discussing certification. In my mind that is just
You're definitely right. My first message was to say some companies do
testing. I know mine does in french. And I think we're still interested in
participating in a community work on the testing.

Dave
>
>> I don't know, but it could be interesting. I'm not sure we did the work
>> to update it to v15, but it sure covers v14. Anyway, I'm not the one who
>> could decide such things :)
>>
>>
>>> Just some thoughts :)
>>>
>>> Sincerely,
>>> Katie

-- 
Guillaume.

---

From: Chris Mair <[email-masked]>
Date: Fri, 2 Jun 2023 18:17:46 +0200

> but I guess we have still enough for start commenting.

Note that LPI-Japan (a non profit organization) already has a Postgres-certification
exam that can be taken world-wide through Pearson-VUE:

https://oss-db.jp/eng_exam

AFAIK right now this is Japanese-only. But there have been attempts to offer this in
English too - see this old thread:

https://www.postgresql.org/message-id/flat/8ae8dc03d25c7d3364a5eea2eec81c1f%40smtp.hushmail.com#ce98b3345adb10949ceb454d567f9039

I guess LPI-Japan could have interesting input to this discussion.

Bye,
Chris.

---

From: Jaime Casanova <[email-masked]>
Date: Mon, 5 Jun 2023 20:27:31 -0500

> At PgCon I mentioned a was going to create this thread so here we go:
[...]

Hi everyone,

Thanks for your comments. Mine was a long trip to return to Ecuador
and then I had a lot of pending things to do.

First, let me mention the good ideas I read on this thread:

- Gunnar and Chris suggested that we could ask a NPO that is already
successfully doing a cert, namely LPI. Do any of you have a contact
there to ask this?
- Katherine mentioned something that is what I had in mind and maybe
didn't express it correctly. The idea is that companies doing training
work together for this to happen.

When I said "community developed" I certainly wasn't thinking of any
person to be involved. We should have a closed group, the development
of the cert should be done through a system and we need a process to
give credentials to anyone that will be involved. For putting
companies on the professional services we follow a process outlined
here https://www.postgresql.org/about/policies/services-and-hosting/
we could have something similar for this.

> 1) Who oversees all of the above?

while I spoke of teachers developing the questions and validating them
(and that could be a medium-size group),  we could have a small group
of well-known community members (not necessarily teachers) that could
act as a committee for administrative actions.

> 2) How are the questions and answers developed without leaking the information out to test takers?

My idea is to have a system, very similar to the one for adding
professional services or news or whatever. Of course, that system
publishes after validation, something we don't want to happen with
this. Not everyone has access to that system.

> 3) Does core mean just the community Postgres?

Yes. I really think that it is not correct to say that you cannot
certify because you know how to use repmgr instead of patroni or
pg_auto_failover, and we cannot force everyone to know all the tools.
Every company developing a tool could decide to create their own certs
about their tools.

---

From: Tatsuo Ishii <[email-masked]>
Date: Tue, 06 Jun 2023 10:35:16 +0900 (JST)

> successfully doing a cert, namely LPI. Do any of you have a contact

I have a contact to LPI-Japan. Can you clarify what questions/requests
you want to ask them?

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp

---

From: Jaime Casanova <[email-masked]>
Date: Wed, 7 Jun 2023 15:48:53 -0500

> I have a contact to LPI-Japan. Can you clarify what questions/requests

I think what scares people the most is about the security, meaning
that if the cert is somewhat community-made how they keep the
questions not public,
I guess part of the solution was already explained in which companies
using a software are the ones making the standard (with users having
to log in).

Probably other questions have to do with the method to take the exams,
actually once the exam is online (no controlled environment, which is
the way most companies do it I guess) no matter what we do people can
reveal the questions just by taking pictures or video of the screen.

ANother question was about who oversees the process, probably about
the organization of all this.

Basically what we need is some guidance about all of it ;)

---

From: Adrian Klaver <[email-masked]>
Date: Wed, 7 Jun 2023 16:09:36 -0700

> the organization of all this.

That is the main question. Without an organization in place the rest of 
it is wasted effort.

> Basically what we need is some guidance about all of it ;)

Who is we?

---

From: Jaime Casanova <[email-masked]>
Date: Thu, 8 Jun 2023 00:24:44 -0500

[...]
> > Basically what we need is some guidance about all of it ;)

we: the community
we: the ones that suggested to ask LPI how they organized everything
we: me and my other personality

you choose ;)

---

From: Robert Bernier <[email-masked]>
Date: Thu, 8 Jun 2023 06:26:05 -0700 (PDT)

Guys,

I still have bad memories when we tried this the last time some 15 years ago.

Let's do this in small steps:
- Create a mail list just for this thread
- Instead of creating an approved certification how's about developing an RFC of sorts outlining expected competencies for various classifications of knowledge (that should be real fun) is expected to carry out certain tasks. Then, after there's some sort of agreement in the community.

Hope this helps.

Robert Bernier
[email-masked]

---

From: James Keener <[email-masked]>
Date: Thu, 8 Jun 2023 10:41:04 -0400

I don't really have any business being in this discussion, but I figured
I'd toss this in anyway.  I realize it's not in-line with what many of you
are thinking, but it may be a good example to follow if we align
expectations with what it provides. Also, apologies if this has come up;
I've been trying to follow but there have been a lot of messages.

In the US, to get an Amerature Radio license you have to study the
requisite material, which includes the exam questions and answers. The exam
is administered by anyone who has passed the VE (Volunteer Examiner) exam
and is 2nd or 3rd level. There are currently 3 license levels: Technician,
General, and Extra. Each one has more in-depth question, but all of the
questions and answers for all levels are public. The exam taken is a
randomized subset of those questions. (I believe it's something like 10%? I
could be wrong without looking it up.)

As for mechanics of administering the exam, at least three VEs need to be
present who are not related to you and who are of the level you're testing
for or higher. Due to Covid, remote exams are now allowed/more common, but
are done with webcam on during the exam so the VE can watch.

While this is obviously easy to game, the expectation is that if you passed
the exam, even if you only studied the questions and answers, you'd have
the basic amount of knowledge needed to operate at the license level you
passed with.

Jim

On Thu, Jun 8, 2023 at 9:26 AM Robert Bernier <[email-masked]>
wrote:

> - Instead of creating an approved certification how's about developing an
> RFC of sorts outlining expected competencies for various classifications of
> knowledge (that should be real fun) is expected to carry out certain tasks.
> Then, after there's some sort of agreement in the community.
>
> Hope this helps.
>
> Robert Bernier

---

From: Adrian Klaver <[email-masked]>
Date: Thu, 8 Jun 2023 07:59:57 -0700

> - Instead of creating an approved certification how's about developing an RFC of sorts outlining expected competencies for various classifications of knowledge (that should be real fun) is expected…

+1

---

From: Charly <[email-masked]>
Date: Thu, 8 Jun 2023 12:09:13 -0300

I would like to start by stating that I don't believe in certifications. I
mean, they are good guides for people who want to start studying and enter
the niche the certification is testing, but they don't measure knowledge.
They measure how much content one is able to memorize, which is by no
means, knowledge.

That said, I like the process @James described. For written tests, just
give the candidates 1k questions book and randomly pick XX number of
questions. If they get it right, they at least went to the book and have a
good memory. If they fail, try again.

IMHO, if we really want to attempt to test knowledge, then we need to think
of having simulated "challenges" to be solved. And it doesn't matter if the
user has or not access to the internet, if they can ask or not for help. If
they are able to solve, and explain the reasons they solved the problem,
then they have shown they acquired the needed knowledge for that level of
certificate.

Those are my 2¢

On Thu, 8 Jun 2023 at 11:41, James Keener <[email-masked]> wrote:

> While this is obviously easy to game, the expectation is that if you
> passed the exam, even if you only studied the questions and answers, you'd
> have the basic amount of knowledge needed to operate at the license level
> you passed with.
>
> Jim
>

-- 
Regards,

*Charly Batista*
*Shanghai, China*
[email-masked] <[email-masked]> *
*https://www.linkedin.com/in/charlybatista/
<https://www.linkedin.com/in/charlybatista/>*

*Linux user #391083*

*“If you have an apple and I have an apple and we exchange these apples
then you and I will still each have one apple. But if you have an idea and
I have an idea and we exchange these ideas, then each of us will have two
ideas."      George Bernard Shaw (1856-1950)*

---

From: Dave Cramer <[email-masked]>
Date: Thu, 8 Jun 2023 14:46:30 -0400

> They measure how much content one is able to memorize, which is by no

Given my comments below, I'm starting to have some serious doubts about the
utility of this. On one hand in an ideal world people would actually learn
what is required and the test would require some level of competency. On
the other hand most of the answers are available with some google searches
and it will be nearly impossible to stop people from using it to pass the
test. If enough people do this the certification proves nothing.

> good memory. If they fail, try again.
Pretty hard to not use the internet to grok the answers which essentially
makes this an open book test.

>
> IMHO, if we really want to attempt to test knowledge, then we need to
> think of having simulated "challenges" to be solved. And it doesn't matter
> if the user has or not access to the internet, if they can ask or not for
> help. If they are able to solve, and explain the reasons they solved the
> problem, then they have shown they acquired the needed knowledge for that
> level of certificate.
>
> Agreed this would prove knowledge, much harder to implement though.

Adding my 2cents

Dave

>>> - Instead of creating an approved certification how's about developing
>>> an RFC of sorts outlining expected competencies for various classifications
>>> of knowledge (that should be real fun) is expected to carry out certain
>>> tasks. Then, after there's some sort of agreement in the community.
>>>
>>> Hope this helps.
>>>
>>> Robert Bernier

---

From: Tatsuo Ishii <[email-masked]>
Date: Mon, 12 Jun 2023 10:13:51 +0900 (JST)

> English too - see this old thread:

I heard from LPI-Japan that they have already offered English
PostgreSQL certifications:

https://lpi.or.jp/en/

---

From: Chris Mair <[email-masked]>
Date: Wed, 14 Jun 2023 09:14:33 +0200

> https://oss-db.jp/eng_exam

Hi all,

we have confirmation that LPI-Japan is offering their OSS-DB in english
once again! I'm forwarding, with permission, the below message that was sent
off-list.

Bye,
Chris.

> From: 浅利康二 <[email-masked]>
> Date: 14 June 2023 at 08:17:05 CEST
>
> Hello all,
>
> Thank you for the introduction, Ishii-san!
> We started to conduct the OSS-DB certification in 2011, which is a PostgreSQL certification that was taken over from SRA OSS.
> And, many candidates take the exam and the number is still increasing every year.
> Major exam takers are Japanese currently, but we also have the English version of the OSS-DB exam and you can take the exam at a number of Pearson Vue locations around the world.
> Feel free to ask us, if you have some questions.
>
> Regards,
>
> ---
> Koji Asari
> LPI-Japan
> https://lpi.or.jp/en/

---

From: Robert Haas <[email-masked]>
Date: Wed, 14 Jun 2023 08:42:06 -0400

> the answer that was at least partially correct. And of course this

I don't think a community approved certification is a good idea. The
problem that you're complaining about here is that the certification
was bad, and the way to solve that problem is to have something
better. I'm in favor of certifications being better. If they are going
to exist, they should be good, just like anything else. But having
something be community-approved doesn't make it automatically better,
because community members can make mistakes and do shoddy work just
like anybody else. Making something a community effort does the
following things:

- It makes it the official version of a thing, which means more people
will use that thing even if it's worse than some other version of the
thing.

- It means there's a group of people who are in charge of that thing,
and it's usually very hard to replace that group of people if they
stop doing the thing well.

- It means that decisions need to be made by consensus, which is good
for things where the cost of mistakes is high (like core development)
and things where by their nature only one can exist (like the
postgresql.org web site).

A great example of competition-is-good-for-the-project is psycopg2. If
there were just one Python connector for PostgreSQL, it wouldn't be
called psycopg2. Because we never picked an official one, it helped a
bunch of projects thrive, and the one most people use now is the one
that won the competition. If there had only ever been one, it probably
wouldn't be as good.

A good example of the difficulty of managing a large overhaul through
the community process is the documentation. There's a lot of great
content in our documentation, but there's also a lot of old stuff that
doesn't really get updated much and maybe isn't even really that
relevant. Much of the valuable content is buried multiple layers down
in the documentation in places where it's not necessarily that easy to
find, while things of more peripheral importance are quite prominent.
But getting agreement on how to address these problems, or even what
the problems are, is pretty hard. However, since the official
documentation is a thing of which there can by nature only be one,
we're kind of locked in to accepting only those improvements that can
make their way through that process.

But we're not locked into such a ponderous process for certifications.
The way we're going to get a great certification if someone goes and
writes a bunch of great questions and then updates them regularly and
vigorously based on feedback and changes in each new release -- and
that requires either a fanatically dedicated volunteer, or for that
person to be getting paid to do that work.

-- 
Robert Haas
EDB: http://www.enterprisedb.com

---

From: Justin Clift <[email-masked]>
Date: Thu, 15 Jun 2023 03:59:48 +1000

> person to be getting paid to do that work.

The PG project isn't exactly short of cash though, so that *might* be
worth exploring.

+ Justin