Subject: gitweb is no longer a real-time view

From: Tom Lane <[email-masked]>
Date: Sun, 03 Mar 2013 19:43:55 -0500

It used to be that http://git.postgresql.org/gitweb/?p=postgresql.git
would update within a minute or so after somebody pushed something.
This seems to have been broken for the last week or so; not only do
commits not show up for a random amount of time, but the displayed
age of the last commit is noticeably wrong.

I recall somebody muttering about having installed caching on the web
server.  If so, could we get that fixed or reverted?

			regards, tom lane

-- 
Sent via pgsql-www mailing list ([email-masked])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

---

From: Stephen Frost <[email-masked]>
Date: Sun, 3 Mar 2013 21:03:24 -0500

Tom,

> would update within a minute or so after somebody pushed something.

The cronjob which syncs the between the git master and
git.postgresql.org appears to run once every 15 minutes (unchanged since
mid-2011).  Given that commits simply aren't that frequent, I do
wonder if we could set that up as a trigger or similar instead of
polling for updates.

> age of the last commit is noticeably wrong.

The recent change was the addition of varnish to do cacheing in front
of git.postgresql.org to reduce load on the system.  The cacheing was
set to 30m/1h, which does seem a bit high.  I've lowered it down to
15m/30m.  That should help some, but obviously isn't going to give
within-a-minute response time.  I'll talk with the other admins and see
what we can do to improve it further.

	Thanks,

		Stephen

---

From: Tom Lane <[email-masked]>
Date: Sun, 03 Mar 2013 21:12:42 -0500

> mid-2011).

cron job?  I was under the impression there was some sort of push
operation driven by a commit trigger.  The web site has certainly
updated nearly immediately for as long as we've been using git.
Until this week, that is.

			regards, tom lane

---

From: Stephen Frost <[email-masked]>
Date: Sun, 3 Mar 2013 21:33:35 -0500

> Until this week, that is.

Curiously, there's two cron jobs, apparently.  There's a 'push' one and
then another, independent, 'pull' one.  I'll assume they're actually
doing different things, but I wonder if the pull isn't just a
hold-over..  In any case, the push-to-anon, which I hadn't seen
initially (looking at the pull side instead of the push side), does run
once a minute, though it looks like there's a hook mechanism which
would allow us to trigger the webserver to do a pull when a commit
happens and would still be better than a once-a-minute cronjob.

Even with that, however, the concern raised was that the gitweb perl
script is quite expensive to run for every request, hence the reason for
doing the cacheing.  I've lowered the varnish cacheing to a 5m ttl and
15m grace and I'll keep an eye on it.

	Thanks,

		Stephen

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 08:16:32 +0100

> hold-over..  In any case, the push-to-anon, which I hadn't seen

No, the pull one doesn't do the main repository - that one pulls
"third party" repositories in as mirrors, such as e.g. Bucardo. That
one was, btw, broken for several months and nobody noticed - so that's
clearly a less important one :)

> happens and would still be better than a once-a-minute cronjob.

The one that deals with the main repo mirroring runs once per minute.
The commit trigger only drops a file in the filesystem so the cronjob
knows there is something to do. So that we don't risk holding up the
actual commit in case there is a problem with the anonymous mirror
server.

> 15m grace and I'll keep an eye on it.

That's fixing the wrong problem. The cache is supposed to be
automatically purged whenever the push happens, to make sure that the
data never gets more than maybe a second stale. So clearly something
in that is not working - I didn't specifically test it with the
postgres.git repository, but I tested it with a couple of other ones
where it worked fine. But maybe we did something special with the main
one...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 08:32:35 +0100

> where it worked fine. But maybe we did something special with the main

Actually, looking closer, I'm seeing a failure when it actually tries the push:

To ssh://[email-masked]/postgresql.git
 ! [rejected]        master -> master (non-fast-forward)

So the problem might have nothing at all to do with the cacheing.

AFAICT, the three missing commits are materialized  views,
accidentally committed .orig file and \l support.

But. The *anonymous* repository also has:
bc61878682051678ade5f59da7bfd90ab72ce13b Fix
map_sql_value_to_xml_value() to treat domains like their base types.

This patch is *not* in the master repository, it's only in anonymous.
(The object is in the repository, but it's not part of any branch)

How the hell did *that* happen?

The master repo has:
commit 5db5974c692b0fc68e7608dd85a6b4e6173a0f28
Author: Peter Eisentraut <[email-masked]>

    psql: Let \l accept a pattern

commit d63977eea3ab18fdec05e370b633d10b9fd20179
Author: Kevin Grittner <[email-masked]>

    Remove accidentally-committed .orig file.

commit 3bf3ab8c563699138be02f9dc305b7b77a724307
Author: Kevin Grittner <[email-masked]>

    Add a materialized view relations.

commit b15a6da29217b14f02895af1d9271e84415a91ae
Author: Tom Lane <[email-masked]>

    Get rid of any toast table when converting a table to a view.

And the anonymous one has:
commit bc61878682051678ade5f59da7bfd90ab72ce13b
Author: Tom Lane <[email-masked]>

    Fix map_sql_value_to_xml_value() to treat domains like their base types.

commit 3bf3ab8c563699138be02f9dc305b7b77a724307
Author: Kevin Grittner <[email-masked]>

    Add a materialized view relations.

commit b15a6da29217b14f02895af1d9271e84415a91ae
Author: Tom Lane <[email-masked]>

    Get rid of any toast table when converting a table to a view.

Does anybody have an explanation for that? Did someone do a force-push
on the master repository, overwriting some old history?

We don't explicitly forbid this on the master repo, since we expect
committers to know how things work.. Maybe we need to do that, and
manually turn it off in case someone actually *needs* to do a non fast
forward push? But either way, it would be good to actually know how
tihs happened...

Copying this one to Kevin and Peter too, since they seem to be the
other people who are involved in any of these pushes.

Seems like we have a problem much worse than just the web...

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 13:50:13 +0100

> on the master repository, overwriting some old history?

This really looks like Kevin forced a push.

Could you show 
git show --pretty=raw d63977eea3ab18fdec05e370b633d10b9fd20179
git show --pretty=raw bc61878682051678ade5f59da7bfd90ab72ce13b
git show --pretty=raw 5db5974c692b0fc68e7608dd85a6b4e6173a0f28
?
I don't have all the commits here...

The raw format will explicitly tell which parents are referenced in the
commits...

> forward push? But either way, it would be good to actually know how

If you allow it the mirroring script should always force a push,
otherwise it will stop pushing in those cases.

I think the most realistic way to resolve this is to push the current
anongit state to master and cherry-pick
bc61878682051678ade5f59da7bfd90ab72ce13b ontop it. That way only
committers need to deal with rebasing their branches and not everyone
else.

Greetings,

Andres Freund

-- 
 Andres Freund	                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

---

From: Kevin Grittner <[email-masked]>
Date: Mon, 4 Mar 2013 05:11:00 -0800 (PST)

> other people who are involved in any of these pushes.

Apologies if I contributed to this.  At no point did I use a force option.

I'm using this pattern:

http://wiki.postgresql.org/wiki/Committing_with_Git#Dependent_Clone_per_Branch.2C_Pushing_and_Pulling_From_a_Local_Repository

I still have the shell windows open that I used, so (redacting
things which I'm sure are not significant, like git log commands)
there is this:

You will see a pull command used.  Here's what that is:

kgrittn@Kevin-Desktop:~/pg/master$ cat ~/bin/pull
#!/bin/bash

pushd ~/pg/postgresql.git >/dev/null && git fetch && popd >/dev/null && git pull

Here's what I have in my (still open) shell windows:

kgrittn@Kevin-Desktop:~/pg/master$ git push
Counting objects: 301, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (154/154), done.
Writing objects: 100% (155/155), 40.94 KiB, done.
Total 155 (delta 147), reused 6 (delta 1)
To /home/kgrittn/pg/postgresql
   2b78d10..3a21ef1  master -> master
kgrittn@Kevin-Desktop:~/pg/master$ cd ../postgresql.git/
kgrittn@Kevin-Desktop:~/pg/postgresql.git$ git push --dry-run
To ssh://[email-masked]/postgresql.git
 + b15a6da...3a21ef1 master -> master (forced update)
 + b15a6da...2b78d10 anonymous/master -> anonymous/master (forced update)
kgrittn@Kevin-Desktop:~/pg/postgresql.git$ git fetch
remote: Counting objects: 31, done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 16 (delta 15), reused 0 (delta 0)
Unpacking objects: 100% (16/16), done.
From ssh://gitmaster.postgresql.org/postgresql
 + 3a21ef1...b15a6da master     -> master  (forced update)
   2b78d10..b15a6da  anonymous/master -> anonymous/master
kgrittn@Kevin-Desktop:~/pg/postgresql.git$ git remote --verbose
origin  ssh://[email-masked]/postgresql.git (fetch)
origin  ssh://[email-masked]/postgresql.git (push)
kgrittn@Kevin-Desktop:~/pg/postgresql.git$ cd ../master
kgrittn@Kevin-Desktop:~/pg/master$ pull
remote: Counting objects: 31, done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 16 (delta 15), reused 0 (delta 0)
Unpacking objects: 100% (16/16), done.
From /home/kgrittn/pg/postgresql
 + 3a21ef1...b15a6da master     -> origin/master  (forced update)
First, rewinding head to replay your work on top of it...
kgrittn@Kevin-Desktop:~/pg/master$ git status
# On branch master
nothing to commit (working directory clean)
kgrittn@Kevin-Desktop:~/pg/master$ git pull
Current branch master is up to date.
kgrittn@Kevin-Desktop:~/pg/master$ git status
# On branch master
nothing to commit (working directory clean)
kgrittn@Kevin-Desktop:~/pg/master$ patch -p1 <~/patch/matview-v6.diff
[some offsets in rewriteDefine.c which looked OK on review]
kgrittn@Kevin-Desktop:~/pg/master$ git add --all
kgrittn@Kevin-Desktop:~/pg/master$ git commit
[master 3bf3ab8] Add a materialized view relations.
 103 files changed, 4240 insertions(+), 438 deletions(-)
 create mode 100644 doc/src/sgml/ref/alter_materialized_view.sgml
 create mode 100644 doc/src/sgml/ref/create_materialized_view.sgml
 create mode 100644 doc/src/sgml/ref/drop_materialized_view.sgml
 create mode 100644 doc/src/sgml/ref/refresh_materialized_view.sgml
 create mode 100644 src/backend/commands/matview.c
 create mode 100644 src/backend/rewrite/rewriteDefine.c.orig
 create mode 100644 src/include/commands/matview.h
 create mode 100644 src/test/regress/expected/matview.out
 create mode 100644 src/test/regress/sql/matview.sql
kgrittn@Kevin-Desktop:~/pg/master$ git push
Counting objects: 301, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (154/154), done.
Writing objects: 100% (155/155), 40.95 KiB, done.
Total 155 (delta 147), reused 5 (delta 1)
To /home/kgrittn/pg/postgresql
   b15a6da..3bf3ab8  master -> master
kgrittn@Kevin-Desktop:~/pg/master$ cd ../postgresql.git/
kgrittn@Kevin-Desktop:~/pg/postgresql.git$ git push --dry-run
To ssh://[email-masked]/postgresql.git
   b15a6da..3bf3ab8  master -> master
kgrittn@Kevin-Desktop:~/pg/postgresql.git$ git push
Counting objects: 301, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (155/155), 40.95 KiB, done.
Total 155 (delta 147), reused 155 (delta 147)
To ssh://[email-masked]/postgresql.git
   b15a6da..3bf3ab8  master -> master

[on review, discover accidental commit of .orig file]

kgrittn@Kevin-Desktop:~/pg/master$ git rm src/backend/rewrite/rewriteDefine.c.orig
rm 'src/backend/rewrite/rewriteDefine.c.orig'
kgrittn@Kevin-Desktop:~/pg/master$ git commit
[master d63977e] Remove accidentally-committed .orig file.
 1 file changed, 945 deletions(-)
 delete mode 100644 src/backend/rewrite/rewriteDefine.c.orig
kgrittn@Kevin-Desktop:~/pg/master$ git push
Counting objects: 9, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 440 bytes, done.
Total 5 (delta 4), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
To /home/kgrittn/pg/postgresql
   3bf3ab8..d63977e  master -> master
kgrittn@Kevin-Desktop:~/pg/master$ cd ../postgresql.git/
kgrittn@Kevin-Desktop:~/pg/postgresql.git$ git push
Counting objects: 15018, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4617/4617), done.
Writing objects: 100% (13087/13087), 2.31 MiB | 632 KiB/s, done.
Total 13087 (delta 11231), reused 9860 (delta 8431)
To ssh://[email-masked]/postgresql.git
 + bc61878...d63977e master -> master (forced update)
 + bc61878...b15a6da anonymous/master -> anonymous/master (forced update)
kgrittn@Kevin-Desktop:~/pg/postgresql.git$ cd ../master ; git diff --stat HEAD^ HEAD
 src/backend/rewrite/rewriteDefine.c.orig |  945 -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1 file changed, 945 deletions(-)
kgrittn@Kevin-Desktop:~/pg/master$ git status
# On branch master
nothing to commit (working directory clean)
kgrittn@Kevin-Desktop:~/pg/master$ pull
Current branch master is up to date.

-- 
Kevin Grittner
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 14:18:09 +0100

>    2b78d10..b15a6da  anonymous/master -> anonymous/master

^^ there it clearly says forced update. So - at least now we know what happened.

Question still remains how then.

Normally it should, AFAIK, refuse to do such a push without a --force.
But admittedly, I don't know for sure exactly how those more complex
setups work.

Do you by any chance have any other git settings either in your local
or in the repository config?

Also, can someone who knows how that stuff is set up, explain why it's
pushing anonymous/master in this case? Seems like it should just push
the actual branch, no? Are those instructions for usage wrong?

>  + 3a21ef1...b15a6da master     -> origin/master  (forced update)

^^ there's another forced update. Which seems to indicate that your
git is doing forced updates by default in both directions.

I have no idea how it would do that by default - I've always received
an error when accidentally doing something like that...

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 14:18:44 +0100

> pushd ~/pg/postgresql.git >/dev/null && git fetch && popd >/dev/null && git pull

Completely unrelated, but subshells are useful for that kind of thing,
e.g.
(cd ~/pg/postgresql.git && git fetch) && git pull
would do the same.

>  + bc61878...b15a6da anonymous/master -> anonymous/master (forced update)

So, here it pushed with force, overwriting the old HEAD? It moves master
from bc61878 to d63977e which exactly explains the missing
map_sql_value_to_xml_value() commit from Tom.

Greetings,

Andres Freund

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 14:19:27 +0100

> git show --pretty=raw 5db5974c692b0fc68e7608dd85a6b4e6173a0f28

In which of the repositories - master, anonymous, or both?

> otherwise it will stop pushing in those cases.

Yeah. But we shouldn't do it, so disallowing it seems like a good thing.

> committers need to deal with rebasing their branches and not everyone

Yeah, I agree that's probably the best way to do it - the number of
committers is a lot more limited...

Anyone else have an opinion on that part?

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 14:23:57 +0100

> ^^ there it clearly says forced update. So - at least now we know what happened.

Thats a fetch though. He pulls in Tom's commit. Which is unsurprisingly
a forced update because he locally had committed something.

> the actual branch, no? Are those instructions for usage wrong?

The defaults around this are afaik changing. Its push.default that
configures that, I suggest setting it to the future default "simple"
instead of "matching" which is the current value.

> an error when accidentally doing something like that...

Whats "git config -l" saying?

Greetings,

Andres Freund

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 14:27:00 +0100

> In which of the repositories - master, anonymous, or both?

Master. The commit-ids should be unique. And I only see the second one
in anon.

But I guess its not really necessary anymore since we know its Kevin's
second push that did the deed even we don't have a clue on why it did.

Greetings,

Andres Freund

---

From: Kevin Grittner <[email-masked]>
Date: Mon, 4 Mar 2013 05:52:42 -0800 (PST)

> an error when accidentally doing something like that...

Are there any files to look at besides these?:

kgrittn@Kevin-Desktop:~/pg/master$ cat ~/.gitconfig
[user]
        name = Kevin Grittner
        email = [email-masked]
[core]
        excludesfile = /home/kgrittn/.gitexcludesfile
        pager = less -x4
    editor = vim
kgrittn@Kevin-Desktop:~/pg/master$ cat .git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        fetch = +refs/heads/master:refs/remotes/origin/master
        url = /home/kgrittn/pg/postgresql
[branch "master"]
        remote = origin
        merge = refs/heads/master
        rebase = true
[remote "github"]
        url = [email-masked]:kgrittn/postgres.git
        fetch = +refs/heads/*:refs/remotes/github/*
kgrittn@Kevin-Desktop:~/pg/master$ cat ../postgresql.git/config
[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
[remote "origin"]
        fetch = +refs/*:refs/*
        mirror = true
        url = ssh://[email-masked]/postgresql.git

The rebase option was something I did because the Wiki instructions
told me to:

I don't think there is anything there which wasn't done per
instructions, but maybe the Wiki needs a fix?

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 14:56:02 +0100

>         repositoryformatversion = 0

^^ that's your problem.

AFAICT, that one sets it to be force-always.

> instructions, but maybe the Wiki needs a fix?

Yea, I'm pretty sure that --mirror thing is the cause of the problem.

Of course, we shuld deploy protect on the server against this, but
that's only going to cause the procedure to *fail* rather than
owerwrite the repo. It still need to be fixed.

---

From: Alvaro Herrera <[email-masked]>
Date: Mon, 4 Mar 2013 11:33:49 -0300

> committers need to deal with rebasing their branches and not everyone

I agree this seems the sanest course of action, but I think it would be
better to install defenses in the master repo so that forced pushes are
no longer allowed *before* rebasing gitmaster; otherwise some other
committer could clobber things again if they don't force-pull their
trees.

FWIW I also use a bare mirror (same setup as Kevin).

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 15:38:52 +0100

> committer could clobber things again if they don't force-pull their

Its just receive.denyNonFastForwards that needs to be set to true for
that, so it should be simple enouh.

> FWIW I also use a bare mirror (same setup as Kevin).

I think that setup is fine as far as pulling goes, but you should push
directly from the individual branches.

Greetings,

Andres Freund

---

From: Tom Lane <[email-masked]>
Date: Mon, 04 Mar 2013 09:40:38 -0500

> committer could clobber things again if they don't force-pull their

Agreed.  I'm astonished that we had this configured this way in the
first place.

FWIW, I noticed something about a "forced update" when I did a git pull
after Kevin's commits.  I failed to pay enough attention though.  Now
that I look, the xml fix is not in my local history either, and I don't
know what bc61878682051678ade5f59da7bfd90ab72ce13b is because none of
the recent commits have that hash according to my repo:

$ git status
# On branch master
nothing to commit (working directory clean)
$ git log
commit d63977eea3ab18fdec05e370b633d10b9fd20179
Author: Kevin Grittner <[email-masked]>
Date:   Sun Mar 3 18:38:44 2013 -0600

commit 3bf3ab8c563699138be02f9dc305b7b77a724307
Author: Kevin Grittner <[email-masked]>
Date:   Sun Mar 3 18:23:31 2013 -0600

    A materialized view has a rule just like a view and a heap and
    other physical properties like a table.  The rule is only used to
    populate the table, references in queries refer to the
    materialized data.
    
    This is a minimal implementation, but should still be useful in
    many cases.  Currently data is only populated "on demand" by the
    CREATE MATERIALIZED VIEW and REFRESH MATERIALIZED VIEW statements.
    It is expected that future releases will add incremental updates
    with various timings, and that a more refined concept of defining
    what is "fresh" data will be developed.  At some point it may even
    be possible to have queries use a materialized in place of
    references to underlying tables, but that requires the other
    above-mentioned features to be working first.
    
    Much of the documentation work by Robert Haas.
    Review by Noah Misch, Thom Brown, Robert Haas, Marko Tiikkaja
    Security review by KaiGai Kohei, with a decision on how best to
    implement sepgsql still pending.

commit b15a6da29217b14f02895af1d9271e84415a91ae
Author: Tom Lane <[email-masked]>
Date:   Sun Mar 3 19:05:47 2013 -0500

    Also make sure other fields of the view's pg_class entry are appropriate
    for a view; it shouldn't have relfrozenxid set for instance.
    
    This ancient omission isn't believed to have any serious consequences in
    versions 8.4-9.2, so no backpatch.  But let's fix it before it does bite
    us in some serious way.  It's just luck that the case doesn't cause
    problems for autovacuum.  (It did cause problems in 8.3, but that's out
    of support.)
    
    Andres Freund

commit 2b78d101d1d6b1d8533a7b7aeff4d82b10a915f8
Author: Tom Lane <[email-masked]>
Date:   Sun Mar 3 17:39:58 2013 -0500

    Fix SQL function execution to be safe with long-lived FmgrInfos.
    
			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 15:46:09 +0100

> the recent commits have that hash according to my repo:

That is/was your map_sql_value_to_xml_value() commit which is now not in
the normal history anymore because kevin pushed a HEAD that didn't
include it.  You should be able to see it with git show
bc61878682051678ade5f59da7bfd90ab72ce13b

Greetings,

Andres Freund

---

From: Alvaro Herrera <[email-masked]>
Date: Mon, 4 Mar 2013 11:50:24 -0300

> the recent commits have that hash according to my repo:

Maybe you can see it mentioned in a "git reflog"?

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 15:52:59 +0100

> committers need to deal with rebasing their branches and not everyone

I satarted looking at doing this, but this is backwards, right?

Commit bc61878682051678ade5f59da7bfd90ab72ce13b exists on anongit, and
only there.

So what we need is to reset master HEAD to that of anongit, and then
cherrypick 3bf3ab8c563699138be02f9dc305b7b77a724307,
d63977eea3ab18fdec05e370b633d10b9fd20179 and
5db5974c692b0fc68e7608dd85a6b4e6173a0f28 on top of that, no?

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 15:56:55 +0100

> I satarted looking at doing this, but this is backwards, right?

Yes. Sorry.

> cherrypick 3bf3ab8c563699138be02f9dc305b7b77a724307,

Uhm, that is in both, isn't it?

> 5db5974c692b0fc68e7608dd85a6b4e6173a0f28 on top of that, no?

Yes.

Greetings,

Andres Freund

---

From: Alvaro Herrera <[email-masked]>
Date: Mon, 4 Mar 2013 11:58:42 -0300

> directly from the individual branches.

From the individual repos you mean?  I don't like that very much,
because when I back-patch something, I prefer to push all commits
locally to the bare mirror repo before pushing them out together to
gitmaster.  That way, if I notice a mistake at the last minute, I can
still revoke everything and start afresh.

One difference in my setup is that my bare mirror has two remotes, one
from gitmaster and one from the anonymous mirror.  The latter fetches
from cron every 5 minutes.  The other one only fetches when I invoke it
manually (because it needs access to my passphrase-protected SSH key).

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 16:03:44 +0100

> manually (because it needs access to my passphrase-protected SSH key).

Well, then you may not use --mirror/remote.xxx.mirror mode since that a)
pushes all branches b) pushes new local branches c) forces updates.

I guess in your setup you could set --mirror with remote.anongit.mirror
so it only mirrors anongit but when pushing to ssh it doesnt...

Greetings,

Andres Freund

---

From: Alvaro Herrera <[email-masked]>
Date: Mon, 4 Mar 2013 12:05:22 -0300

> 5db5974c692b0fc68e7608dd85a6b4e6173a0f28 on top of that, no?

Uh, 3bf3ab8c5 is parent of bc6187868 so surely you don't need to
cherry-pick that one?  Only the other two.

---

From: Tom Lane <[email-masked]>
Date: Mon, 04 Mar 2013 10:09:10 -0500

> bc61878682051678ade5f59da7bfd90ab72ce13b

Ah, right, so it is still there but it's not in the chain leading to
HEAD.  So "forced update" really means "I moved HEAD in a non fast
forward way"?

			regards, tom lane

---

From: Alvaro Herrera <[email-masked]>
Date: Mon, 4 Mar 2013 12:11:28 -0300

> pushes all branches b) pushes new local branches c) forces updates.

Hm, there are no new local branches here; I fetch to the same branches
as the other remote:

git fetch -q anon-origin master:master REL9_2_STABLE:REL9_2_STABLE REL9_1_STABLE:REL9_1_STABLE REL9_0_STABLE:REL9_0_STABLE REL8_4_STABLE:REL8_4_STABLE REL8_3_STABLE:REL8_3_STABLE

Note this fails (which is what I want) when it's not a FF pull.  So my
local repository hasn't had any corruption.  In fact, to observe the
breakage, I had to do a "git fetch anon-origin master:anon-master".

I guess there clearly weren't enough protections in the gitmaster server
to prevent me from causing major breakage (hopefully there are now), but
then I'm so chicken that I triple-check these things before pushing, so
no major harm has been done by me to date, even though my setup may be
theoretically dangerous.

---

From: Alvaro Herrera <[email-masked]>
Date: Mon, 4 Mar 2013 12:15:26 -0300

> HEAD.  So "forced update" really means "I moved HEAD in a non fast

Well, to git, "non fast forward" can also mean create a merge commit of
two heads, which isn't very problematic.  A forced update might mean
(such as in this case) that one or more commits are left completely
disconnected from history -- that's much worse.

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 16:16:03 +0100

> HEAD.  So "forced update" really means "I moved HEAD in a non fast

Exactly. And normally you need to specify --force for that except in
Kevin's setup the mirror mode used (and described in the wiki) also
includes that...

Greetings,

Andres Freund

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 16:17:09 +0100

> disconnected from history -- that's much worse.

Well, but thats during pulling/merging and not during pushing. The
latter doesn't do any merges.

Greetings,

Andres Freund

---

From: Tom Lane <[email-masked]>
Date: Mon, 04 Mar 2013 10:17:30 -0500

Alvaro Herrera <[email-masked]> writes:
> to prevent me from causing major breakage (hopefully there are now),

Are there?  I concur with your recommendation that we should lock
this down *before* we set about repairing the master repo's contents.
It seems particularly important that we do so if we're going to ask
committers to rebase, because it seems like somebody who does that wrong
could easily screw things up again.

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 16:22:23 +0100

> this down *before* we set about repairing the master repo's contents.

There are. I put them in, and I'm currently working on repairing the
repo contents.

> could easily screw things up again.

Yeah.

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 16:31:15 +0100

> There are. I put them in, and I'm currently working on repairing the

Ok, both repos *should* now be where they are supposed to be, per this
discussion.

AFAICT, pushing now works fine between primary and anonymous.

And yes, the cache purging appears to be broken *as well*. The reason
for that one seems to be that I forgot to update the port number for
cache invalidation when I moved it into production. Oops. That should
start working on the next commit that someone does.

But git:// access to anonymous and ssh:// access to master should both
work, and contain the complete and correct repositorires.

---

From: Tom Lane <[email-masked]>
Date: Mon, 04 Mar 2013 10:48:59 -0500

> Ok, both repos *should* now be where they are supposed to be, per this

Being a paranoid sort, I'm doing a full new "git clone" to compare
to my old repo.  But assuming that that's okay, what instructions
do you want to give committers about recovering?  There should be
a followup with that info to the pgsql-hackers message you posted.

The other todo item was updating the wiki's instructions for committers.

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 16:59:37 +0100

> a followup with that info to the pgsql-hackers message you posted.

Yes, please do that comparison. Alvaro looked it over and found things
to look right, but one more paranoid check would be very good.

On my local clone it "just worked". The pull went through in non-force
mode. So all that would be needed was a normal rebase.

But I didn't have any active development branch against gitmaster -
only against the anonymous one... So I'm not sure if others might run
into issues with it.

> The other todo item was updating the wiki's instructions for committers.

Yeah, someone who actually knows those parts well should do that - I
just use a separate repository and manual pull myself, so I don't know
enough to do the instrucitons I think.

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 17:00:22 +0100

> a followup with that info to the pgsql-hackers message you posted.

"git pull your-ssh-pg-upstream --rebase" should do the trick.

Greetings,

Andres Freund

---

From: Tom Lane <[email-masked]>
Date: Mon, 04 Mar 2013 12:11:31 -0500

> to look right, but one more paranoid check would be very good.

AFAICT things are okay now.

One thing that would be interesting to try is to look for commits that
are unreachable from any of the branch heads, to verify that we've
not missed anything else.  I imagine git has a command for that, but
I don't know it --- any ideas?

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Mon, 4 Mar 2013 18:16:02 +0100

> I don't know it --- any ideas?

git fsck --unreachable|grep "^unreachable commit" might do it.

Greetings,

Andres Freund

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 18:19:31 +0100

> AFAICT things are okay now.

Great. I'll "release" in the other thread then.

> I don't know it --- any ideas?

I'm running the fsck command suggested by Andres. It's taking a long
time to run, but I'll report if it finds something we shouldn't have.

---

From: Tom Lane <[email-masked]>
Date: Mon, 04 Mar 2013 15:18:06 -0500

> start working on the next commit that someone does.

Seemed to work when I pushed the tm2timestamp fix just now --- the
front page updated almost immediately after the next top-of-minute.
Thanks.

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Mon, 4 Mar 2013 21:31:32 +0100

> front page updated almost immediately after the next top-of-minute.

Great, thanks for confirming.

Now I just need to get Dave to stop unintentionally purgning the cache
every 5 minutes...

---

From: Dave Page <[email-masked]>
Date: Tue, 5 Mar 2013 09:02:26 +0000

> Now I just need to get Dave to stop unintentionally purgning the cache

Meh. It's on my list.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 5 Mar 2013 14:15:59 +0100

>> Now I just need to get Dave to stop unintentionally purgning the cache

Oh, I know it is. I'm working on the theory of that eventually getting
that fixed will be a lower pain than listening to my whining about it
:)