Opened 6 years ago

Closed 4 years ago

#273 closed change (fixed)

Synchronize hg.adblockplus.org with GitHub

Reported by: fhd Assignee: matze
Priority: P2 Milestone:
Module: Infrastructure Keywords:
Cc: trev, fred, kzar Blocked By:
Blocking: #2906 Platform: Unknown / Cross platform
Ready: no Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

Description (last modified by matze)

Using the hg-git extension and assuming the hg1.adblockplus.org server being integrated, the following steps should remain:

  • Setting up a GitHub-account for synchronization by hg@hg1.adblockplus.org
  • Configuring a github remote URL within the [path] section of the respective hgrc files
  • Configuring a changegroup.github hook for those repos, e.g. hg push -q github
  • Bookmarking HG branches to become recognized by hg-git
  • Performing an initial synchronization / push via hg-git

Note that the latter two will probably require breaking the Git references once more.

Change History (14)

comment:1 Changed 4 years ago by matze

  • Blocking 2906 added
  • Owner set to matze
  • Platform set to Unknown / Cross platform
  • Summary changed from Automatically sync all our repositories to GitHub to Synchronize hg.adblockplus.org/ with github.com/adblockplus/
  • Tester set to Unknown

comment:2 Changed 4 years ago by matze

  • Cc fred added
  • Description modified (diff)
  • Priority changed from P4 to P2
  • Ready unset
  • Summary changed from Synchronize hg.adblockplus.org/ with github.com/adblockplus/ to Synchronize hg.adblockplus.org with GitHub

comment:3 Changed 4 years ago by matze

Here's an example hgrc file for the infrastructure repository:

[web]
contact = Mathias Hennig <***@adblockplus.org>
description = Adblock Plus Infrastructure based on Puppet 
[hooks]
pretxnchangegroup.author_check = !( hg log --template '{author}\n' -r $HG_NODE: | grep -i -v "$HGUSER")
pretxnchangegroup.github = hg push -q github || true
incoming.irc = python:sitescripts.hg.bin.irchook.hook
[paths]
github = git+ssh://git@github.com/adblockplus/infrastructure.git

According to my tests it seems to work, although I haven't had the chance to actually test it with https://github.com/adblockplus/infrastructure yet - due to the aforementioned reference issue, which I do not want to trigger just yet.

comment:4 Changed 4 years ago by matze

Now that I found the time for a bit more testing and manually setup hg1.adblockplus.org with a more recent version of both mercuial and hg-git via pip(1), I found a way to not break references again. Well, at least not with the infrastructure module, which I've used for testing.

Here's the hgrc file that worked for me:

[hooks]
changegroup.github = hg bookmark -r default master && hg push github || true
#changegroup.legacy = hg push legacy || true
incoming.irc = python:sitescripts.hg.bin.irchook.hook
pretxnchangegroup.author_check = !( hg log --template '{author}\n' -r $HG_NODE: | grep -i -v "$HGUSER")

[paths]
#legacy = ssh://***@adblockplus.org/infrastructure
github = git+ssh://git@github.com/adblockplus/infrastructure.git

[web]
contact = Mathias Hennig <***@adblockplus.org>
description = Adblock Plus Infrastructure based on Puppet

Note the currently inactive legacy options, which are meant to synchronize with the former repository server as long as necessary. Thus the migration plan could look like this:

  • Create a backup of the old hgaccess repository
  • Register hg@hg1.adblockplus.org as [trusted] user with every hgaccess/repos/ item
  • Disable hgweb on hg1.adblockplus.org, in order to not leak private repositories
  • Clone all repos into the hg@hg1.adblockplus.org account (via SSH), except hgaccess
  • Adjust all hgrc files, incl. review
  • Enable hgweb on hg1.adblockplus.org
  • Change the DNS record of hg.adblockplus.org accordingly
  • Remove every user but the hg one from the old hgaccess/repos/
  • Perform a (last) hg -R $repo pull legacy for every single repository
  • Replace the existing hgaccess on hg1.adblockplus.org with the backup created

This of course requires communicating the time-schedule first, as well as the new upstream URL for those with commit privileges. Any feedback?

Last edited 4 years ago by matze (previous) (diff)

comment:5 Changed 4 years ago by kzar

So as it stands our repositories are no longer being synchronised with GitHub at all? If so how long do do you think it will take to get this working? (I use Git for all my work.)

Also if possible could you reduce the number of times you break Git references or at least batch them together? Rebasing all my branches is kind of a pain.

Thanks for sorting this out anyway, it will be great to have the synchronisation working properly! - Dave

comment:6 Changed 4 years ago by kzar

  • Cc kzar added

comment:7 Changed 4 years ago by matze

@kzar
I just learned that Felix is now executing the sync on his Laptop every 30 mins. If it's on, of course. So, as of now, nothing has changed for any contributor - beside the broken references in Git. The latter was not avoidable, by the way, although I am kinda curious why you seem to speak about multiple occurrences.?

comment:8 Changed 4 years ago by fhd

Yes, same situation as before with Wladimir, it's running regularly on my local machine.

I don't think we'll have to push --force again any time soon - hg-git should be avoiding changes that break the hash mapping like the plague. If I understood Matze correctly earlier today, this was a regression that's already fixed on the hg-git side, hashes should stay the same.

Last edited 4 years ago by fhd (previous) (diff)

comment:9 Changed 4 years ago by fhd

@Matze: Note that we have an actual master bookmark in some repositories, so we shouldn't explicitly set it to default, unless there is none at all.

comment:10 Changed 4 years ago by matze

@fhd: Now I am a bit confused. In my understanding we have to "forward" the master bookmark for every incoming transaction, in order for hg-git to recognize the new change-sets when pushing.. See also the changegroup.github hook in the example hgrc above.

comment:11 Changed 4 years ago by kzar

@matze Ace OK, shouldn't disturb me at all then :)

(The reason I thought the references might be broken again was this sentence in the issue description "Note that the latter two will probably require breaking the Git references once more.")

comment:12 Changed 4 years ago by matze

The above hgrc example (including the legacy bits) works out as expected. In fact the last few pushes to infrastructure have taken the routes configured there, so far without any issues.

comment:13 Changed 4 years ago by matze

This part of the migration is done, the remainder is tracked in other tickets - e.g. #2949.

comment:14 Changed 4 years ago by matze

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.