Synchronize hg.adblockplus.org with GitHub

[fhd]

Using the hg-git extension and assuming the hg1.adblockplus.org server being integrated, the following steps should remain:

• Setting up a GitHub-account for synchronization by hg@hg1.adblockplus.org
• Configuring a github remote URL within the [path] section of the respective hgrc files
• Configuring a changegroup.github hook for those repos, e.g. hg push -q github
• Bookmarking HG branches to become recognized by hg-git
• Performing an initial synchronization / push via hg-git

Note that the latter two will probably require breaking the Git references once more.

[matze]

Here's an example hgrc file for the infrastructure repository:

[web]
contact = Mathias Hennig <***@adblockplus.org>
description = Adblock Plus Infrastructure based on Puppet 
[hooks]
pretxnchangegroup.author_check = !( hg log --template '{author}\n' -r $HG_NODE: | grep -i -v "$HGUSER")
pretxnchangegroup.github = hg push -q github || true
incoming.irc = python:sitescripts.hg.bin.irchook.hook
[paths]
github = git+ssh://git@github.com/adblockplus/infrastructure.git

According to my tests it seems to work, although I haven't had the chance to actually test it with ​https://github.com/adblockplus/infrastructure yet - due to the aforementioned reference issue, which I do not want to trigger just yet.

[matze]

Now that I found the time for a bit more testing and manually setup hg1.adblockplus.org with a more recent version of both mercuial and hg-git via pip(1), I found a way to not break references again. Well, at least not with the infrastructure module, which I've used for testing.

Here's the hgrc file that worked for me:

[hooks]
changegroup.github = hg bookmark -r default master && hg push github || true
#changegroup.legacy = hg push legacy || true
incoming.irc = python:sitescripts.hg.bin.irchook.hook
pretxnchangegroup.author_check = !( hg log --template '{author}\n' -r $HG_NODE: | grep -i -v "$HGUSER")

[paths]
#legacy = ssh://***@adblockplus.org/infrastructure
github = git+ssh://git@github.com/adblockplus/infrastructure.git

[web]
contact = Mathias Hennig <***@adblockplus.org>
description = Adblock Plus Infrastructure based on Puppet

Note the currently inactive legacy options, which are meant to synchronize with the former repository server as long as necessary. Thus the migration plan could look like this:

• Create a backup of the old hgaccess repository
• Register hg@hg1.adblockplus.org as [trusted] user with every hgaccess/repos/ item
• Disable hgweb on hg1.adblockplus.org, in order to not leak private repositories
• Clone all repos into the hg@hg1.adblockplus.org account (via SSH), except hgaccess
• Adjust all hgrc files, incl. review
• Enable hgweb on hg1.adblockplus.org
• Change the DNS record of hg.adblockplus.org accordingly
• Remove every user but the hg one from the old hgaccess/repos/
• Perform a (last) hg -R $repo pull legacy for every single repository
• Replace the existing hgaccess on hg1.adblockplus.org with the backup created

This of course requires communicating the time-schedule first, as well as the new upstream URL for those with commit privileges. Any feedback?

[kzar]

So as it stands our repositories are no longer being synchronised with GitHub at all? If so how long do do you think it will take to get this working? (I use Git for all my work.)

Also if possible could you reduce the number of times you break Git references or at least batch them together? Rebasing all my branches is kind of a pain.

Thanks for sorting this out anyway, it will be great to have the synchronisation working properly! - Dave

[matze]

@kzar
I just learned that Felix is now executing the sync on his Laptop every 30 mins. If it's on, of course. So, as of now, nothing has changed for any contributor - beside the broken references in Git. The latter was not avoidable, by the way, although I am kinda curious why you seem to speak about multiple occurrences.?

[fhd]

Yes, same situation as before with Wladimir, it's running regularly on my local machine.

I don't think we'll have to push --force again any time soon - hg-git should be avoiding changes that break the hash mapping like the plague. If I understood Matze correctly earlier today, this was a regression that's already fixed on the hg-git side, hashes should stay the same.

[fhd]

@Matze: Note that we have an actual master bookmark in some repositories, so we shouldn't explicitly set it to default, unless there is none at all.

[matze]

@fhd: Now I am a bit confused. In my understanding we have to "forward" the master bookmark for every incoming transaction, in order for hg-git to recognize the new change-sets when pushing.. See also the changegroup.github hook in the example hgrc above.

[kzar]

@matze Ace OK, shouldn't disturb me at all then :)

(The reason I thought the references might be broken again was this sentence in the issue description "Note that the latter two will probably require breaking the Git references once more.")

[matze]

The above hgrc example (including the legacy bits) works out as expected. In fact the last few pushes to infrastructure have taken the routes configured there, so far without any issues.

[matze]

This part of the migration is done, the remainder is tracked in other tickets - e.g. #2949.