Opened on 04/08/2014 at 05:43:48 AM

Closed on 08/24/2015 at 05:13:47 PM

#273 closed change (fixed)

Synchronize with GitHub

Reported by: fhd Assignee: matze
Priority: P2 Milestone:
Module: Infrastructure Keywords:
Cc: trev, fred, kzar Blocked By:
Blocking: #2906 Platform: Unknown / Cross platform
Ready: no Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

Description (last modified by matze)

Using the hg-git extension and assuming the server being integrated, the following steps should remain:

  • Setting up a GitHub-account for synchronization by
  • Configuring a github remote URL within the [path] section of the respective hgrc files
  • Configuring a changegroup.github hook for those repos, e.g. hg push -q github
  • Bookmarking HG branches to become recognized by hg-git
  • Performing an initial synchronization / push via hg-git

Note that the latter two will probably require breaking the Git references once more.

Attachments (0)

Change History (14)

comment:1 Changed on 08/15/2015 at 02:15:20 PM by matze

  • Blocking 2906 added
  • Owner set to matze
  • Platform set to Unknown / Cross platform
  • Summary changed from Automatically sync all our repositories to GitHub to Synchronize with
  • Tester set to Unknown

comment:2 Changed on 08/15/2015 at 11:11:33 PM by matze

  • Cc fred added
  • Description modified (diff)
  • Priority changed from P4 to P2
  • Ready unset
  • Summary changed from Synchronize with to Synchronize with GitHub

comment:3 Changed on 08/15/2015 at 11:18:38 PM by matze

Here's an example hgrc file for the infrastructure repository:

contact = Mathias Hennig <***>
description = Adblock Plus Infrastructure based on Puppet 
pretxnchangegroup.author_check = !( hg log --template '{author}\n' -r $HG_NODE: | grep -i -v "$HGUSER")
pretxnchangegroup.github = hg push -q github || true
incoming.irc = python:sitescripts.hg.bin.irchook.hook
github = git+ssh://

According to my tests it seems to work, although I haven't had the chance to actually test it with yet - due to the aforementioned reference issue, which I do not want to trigger just yet.

comment:4 Changed on 08/17/2015 at 03:34:31 AM by matze

Now that I found the time for a bit more testing and manually setup with a more recent version of both mercuial and hg-git via pip(1), I found a way to not break references again. Well, at least not with the infrastructure module, which I've used for testing.

Here's the hgrc file that worked for me:

changegroup.github = hg bookmark -r default master && hg push github || true
#changegroup.legacy = hg push legacy || true
incoming.irc = python:sitescripts.hg.bin.irchook.hook
pretxnchangegroup.author_check = !( hg log --template '{author}\n' -r $HG_NODE: | grep -i -v "$HGUSER")

#legacy = ssh://***
github = git+ssh://

contact = Mathias Hennig <***>
description = Adblock Plus Infrastructure based on Puppet

Note the currently inactive legacy options, which are meant to synchronize with the former repository server as long as necessary. Thus the migration plan could look like this:

  • Create a backup of the old hgaccess repository
  • Register as [trusted] user with every hgaccess/repos/ item
  • Disable hgweb on, in order to not leak private repositories
  • Clone all repos into the account (via SSH), except hgaccess
  • Adjust all hgrc files, incl. review
  • Enable hgweb on
  • Change the DNS record of accordingly
  • Remove every user but the hg one from the old hgaccess/repos/
  • Perform a (last) hg -R $repo pull legacy for every single repository
  • Replace the existing hgaccess on with the backup created

This of course requires communicating the time-schedule first, as well as the new upstream URL for those with commit privileges. Any feedback?

Last edited on 08/17/2015 at 03:38:22 AM by matze

comment:5 Changed on 08/17/2015 at 03:26:23 PM by kzar

So as it stands our repositories are no longer being synchronised with GitHub at all? If so how long do do you think it will take to get this working? (I use Git for all my work.)

Also if possible could you reduce the number of times you break Git references or at least batch them together? Rebasing all my branches is kind of a pain.

Thanks for sorting this out anyway, it will be great to have the synchronisation working properly! - Dave

comment:6 Changed on 08/17/2015 at 03:29:39 PM by kzar

  • Cc kzar added

comment:7 Changed on 08/17/2015 at 04:16:38 PM by matze

I just learned that Felix is now executing the sync on his Laptop every 30 mins. If it's on, of course. So, as of now, nothing has changed for any contributor - beside the broken references in Git. The latter was not avoidable, by the way, although I am kinda curious why you seem to speak about multiple occurrences.?

comment:8 Changed on 08/17/2015 at 05:46:18 PM by fhd

Yes, same situation as before with Wladimir, it's running regularly on my local machine.

I don't think we'll have to push --force again any time soon - hg-git should be avoiding changes that break the hash mapping like the plague. If I understood Matze correctly earlier today, this was a regression that's already fixed on the hg-git side, hashes should stay the same.

Last edited on 08/17/2015 at 05:47:21 PM by fhd

comment:9 Changed on 08/17/2015 at 05:50:43 PM by fhd

@Matze: Note that we have an actual master bookmark in some repositories, so we shouldn't explicitly set it to default, unless there is none at all.

comment:10 Changed on 08/17/2015 at 05:53:26 PM by matze

@fhd: Now I am a bit confused. In my understanding we have to "forward" the master bookmark for every incoming transaction, in order for hg-git to recognize the new change-sets when pushing.. See also the changegroup.github hook in the example hgrc above.

comment:11 Changed on 08/17/2015 at 08:42:50 PM by kzar

@matze Ace OK, shouldn't disturb me at all then :)

(The reason I thought the references might be broken again was this sentence in the issue description "Note that the latter two will probably require breaking the Git references once more.")

comment:12 Changed on 08/17/2015 at 10:29:56 PM by matze

The above hgrc example (including the legacy bits) works out as expected. In fact the last few pushes to infrastructure have taken the routes configured there, so far without any issues.

comment:13 Changed on 08/24/2015 at 05:13:22 PM by matze

This part of the migration is done, the remainder is tracked in other tickets - e.g. #2949.

comment:14 Changed on 08/24/2015 at 05:13:47 PM by matze

  • Resolution set to fixed
  • Status changed from new to closed

Add Comment

Modify Ticket

Change Properties
as closed .
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from matze.
Note: See TracTickets for help on using tickets.