Bridge support in git for mercurial and bazaar

I’ve been involved in the git project for some time now, and through the years I’ve had to work with other DSCMs, like mercurial and monotone. Not really as a user, but investigating the inner workings of them, mostly for conversion purposes, and I’ve written tools that nobody used, like mtn2git, or pidgin-git-import, but eventually I decided; why not write something that everybody can use?

So, I present to you git-remote-hg, and git-remote-bzr. What do they do?

git clone "hg::http://selenic.com/repo/hello"
git clone "bzr::lp:gnuhello"

That’s right, you can clone both mercurial and bazaar repositories now with little to no effort.

The original repository will be tracked like any git repository:

% git remote show origin
* remote origin
  Fetch URL: hg::http://selenic.com/repo/hello
  Push  URL: hg::http://selenic.com/repo/hello
  HEAD branch: master
  Remote branches:hg and mercurial. 
    branches/default tracked
    master           tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (create)

You can pull, and in fact push as well. Actually, you wouldn’t even notice that this remote is not a git repository. You can create and push new branches… everything.

You might be thinking that this code being so new might have many issues, and you might screw up a mercurial repository. While that is true to some extent, there’s already a project that tries to act as a bridge between mercurial and git: hg-git. This project has a long history, and a lot of people use it successfully. So, I took their test framework, cleaned it up, and used to test my git-remote-hg code. Eventually all the tests passed, and I cloned big mercurial repositories without any issue, and the result were exact replicas, and I know that because the SHA-1′s were the same :)

So you can even keep working on top of clones you have created with hg-git, and switch back and forth between git-remote-hg and hg-git as you wish. The advantage over hg-git, is that you don’t need to use the mercurial interface at all, you can use git :)

To enable the hg-git compatibility mode you would need this:

% git config --global remote-hg.hg-git-compat true

What about the others?

Surely I must not be the first one try something this cool, right? I don’t want to dwell into all the details why none of the other tools suited my needs, but let’s give a quick look:

hg-git

It’s for mercurial, not git. So you need to use it through the mercurial UI.

fast-export

This is a very nice tool, but only allows you to export stuff (fetch an hg repo), and you have to manually run the ‘git fast-import’ command, setup the marks, and so on. Also, it’s not very well maintained.

Another git-remote-hg

It needs hg-git.

Yet another git-remote-hg

You need quite a lot of patches on top of git, so your vanilla version of git is not going to cut it. In addition to that it doesn’t support as many features; no tags, no bookmarks. Plus it fails all my extensive tests for git-remote-hg.

git-hg

It needs separate tools, such as ‘hg convert’, and fast-export, and it doesn’t use remote helper infrastructure, so it’s not transparent: you have to run git-hg clone, git-hg fetch, and so on.

Another git-hg

It needs hg-git, and doesn’t use git’s remote helper infrastructure either: you have to do git hg-clone, git hg-pull, etc.

There might be others, but you get the point.

The situation in bazaar is not much better, but I’m not going to bother listing the tools.

More coolness

In fact, you can push and pull from and to mercurial and bazaar :)

% git remote -v
bzr	bzr::lp:~felipec/+junk/test (fetch)
bzr	bzr::lp:~felipec/+junk/test (push)
gh	gh:felipec/test.git (fetch)
gh	gh:felipec/test.git (push)
hg	hg::bb+ssh://felipec/test (fetch)
hg	hg::bb+ssh://felipec/test (push)

Isn’t that cool?

So, are there any drawbacks? Surely some information must be lost.

Nope, not really. At least not when using the hg-git compat mode, because all the extra information is saved in the commit messages.

Update: I forgot to mention the only bidirectionality problem: octopus merges. In git a merge can have any number of parent commits, but in mercurial only two are allowed. So you might have problem converting from a git repository to a mercurial one. It’s the only feature that hg-git has, that git-remote-hg doesn’t. As for bazaar, it also allows multiple parents, so it should work fine, but this hasn’t been thoroughly tested.

The only consideration is that in the case of mercurial branches are quite different from git branches, so you need to be really careful to get the behavior you want, which is why it’s best to just ignore mercurial branches, and use bookmarks instead. When you create git branches and push them to mercurial, bookmarks will be created on whatever branch is current.

Trying it out

Just copy them files (git-remote-hggit-remote-bzr) to any location available in your $PATH (.e.g ~/bin), and start cloning :)

Soon they might be merged into upstream git, so they would be distributed as other contrib scripts (e.g. /usr/share/git), and eventually possibly enabled by default. They were distributed as other contrib scripts, and they were going to be enabled by default, until the maintainer decided not to, for no reason at all.

Enjoy!

About these ads

62 thoughts on “Bridge support in git for mercurial and bazaar

  1. +Serge

    I guess this is the thread you are looking for:

    New remote-hg helper

    +Malureau

    What happened to elmarco?

    Nah, I don’t know what I’m doing. But I learn quickly :)

    +Ceyusa

    Yes, python. When working with mercurial/bazaar is a necessary evil =/

  2. This link gives me a 404:
    https://github.com/felipec/git/blob/fc-remote-hg/contrib/remote-helpers/git-remote-hg

    Just my problem? Donde esta git-remote-hg? Yo tengo git-remote-hg! (^:

    It is interesting that the issue of hg->git has been so convoluted. I also found and used one
    called git-hg-again. And it worked. Apparently github also has their own called gexport,gimport.
    Bit I dunno, I got a headache now. I just wanted to fork a bitbucket repo on github, jeez.

  3. So what about HTTPS based Mercurial repos (say things like in Bitbucket)? When I try pushing to one I get a stacktrace with:

    raise util.Abort(_(‘http authorization required’))
    mercurial.error.Abort: http authorization required

    At the end.

  4. Thank you, this is awesome!

    The only thing that I found a bit annoying was that after pushing, “git status” says my repo is still ahead of the remote. This is despite the fact that the remote contains the changes I’ve pushed. I need to do a fetch in order to sync the two up. Pure git doesn’t have this behaviour. Perhaps “git push” could be changed to do “git push && git fetch”?

    user@host:~/someRepo[master*] $ git commit -m “Some message”
    user@host:~/someRepo[master+] $ git push
    Everything up-to-date
    user@host:~/someRepo[master+] $ git status
    # On branch master
    # Your branch is ahead of ‘origin/master’ by 4 commits.
    #
    nothing to commit, working directory clean
    user@host:~/someRepo[master+] $ git fetch
    From hg::ssh://remoteHost//path/to/someRepo
    742c327..5fcb7a2 master -> origin/master
    742c327..5fcb7a2 branches/default -> origin/branches/default
    user@host:~/someRepo[master] $ git status
    # On branch master
    nothing to commit, working directory clean

  5. +Paul I answered in the github issue report. It only happens when you don’t have a ‘master’ bookmark. I attached a patch, it should work.

  6. Thanks! I haven’t had a chance to check it yet as I’m on vacation. I’ll try it out in the new year, and file bugs at github if necessary. Thanks for your work!

  7. Felipe, are you going to try to maintain this, and ultimately get it merged into the upstream git? Or would you like someone (or some group) to take it and run with it for you?

  8. I tried this on an ubuntu system that had git installed but did not have hg installed. Didn’t work. Installing: “apt-get install mercurial” solved the problem

  9. After cloning, I try to push but get the error message “mercurial.error.Abort: push creates new remote head 49ad7886bb97!”. Is there something I’m doing wrong?

  10. @Paul I will try to maintain this. If somebody ends up committing good stuff, I wouldn’t object to him/her/them taking maintainership. Ultimately it’s Junio’s call (git’s maintainer).

  11. Does the bzr bridge survive rebasing, or does this result in bzr complaining about diverged branches?

  12. I second Mathias’ question: lack of rebasing support was the reason I gave up on git-bzr-ng after a very short trial.

  13. Hello Felipe

    Thanks for this nice plugin. I’ve using it since weeks, mainly to synch gajim hg repo (http://hg.gajim.org/gajim/).

    I did a fetch today and got this:

    mmoya@andrea branches/gtk3 ~/src/gajim$ git fetch
    warning: Not updating refs/hg/origin/bookmarks/master (new tip dab765e2639914eb5033dd6681adbee81fc46c21 does not contain ae34ca21170712dbd99d64b72b246a8415728160)
    fatal: Error while running fast-import

    is this an error of the bridge plugin?

    Thanks,
    maykel

  14. Felipe, your bridge doesn’t gracefully handle importing tags with spaces in the tag name. These are legal in bzr, but not in git, so it’s not really clear how to resolve this situation. Probably it’s best to ignore them with a warning but continue.

    Currently the following happens when importing the the tag “Name with space”:
    WARNING: TODO: fetch tag ‘Name’
    error: refs/tags/Name does not point to a valid object!

  15. Let me know if I’m doing something wrong here, but it seems there is a massive showstopper: you can’t seems to merge branches that originate from different remote and push the result back upstream: I’m getting a KeyError in blob_marks at https://github.com/felipec/git/blob/fc/remote/bzr/contrib/remote-helpers/git-remote-bzr#L537. Could this be caused by the marks files being separate per remote s.t. it can’t find any the revisions and blobs from the merged-in branch?

  16. Thanks, Simon.

    It looks like this has made it into the git tree, under `contrib/remote-helpers`. If installing from source, you’ll need to install it explicitly: it’s not enabled by default, nor is the `/usr/share/git/remote-helpers` directory created automatically (that’s probably a feature of whatever distro Simon’s using).

  17. Pingback: Access Mercurial repo with Git on windows | BlogoSfera

  18. To have it working from inside Rubymine 5 I had to set it as executable and place it in the git folder with the git executable.

  19. Weirdly, though, only the Mercurial part seems to have made it into git proper. The Bazaar helper doesn’t seem to be included – or am I missing something?

  20. Cloning using git hg::ssh://user@repo name works great, but the consecutive pull failes with

    git pull
    Traceback (most recent call last):
    File “/home/wmader/local/bin/git-remote-hg”, line 805, in
    sys.exit(main(sys.argv))
    File “/home/wmader/local/bin/git-remote-hg”, line 774, in main
    repo = get_repo(url, alias)
    File “/home/wmader/local/bin/git-remote-hg”, line 280, in get_repo
    peer = hg.peer(myui, {}, url)
    File “/usr/lib/python2.7/site-packages/mercurial/hg.py”, line 121, in peer
    return _peerorrepo(rui, path, create).peer()
    File “/usr/lib/python2.7/site-packages/mercurial/hg.py”, line 101, in _peerorrepo
    obj = _peerlookup(path).instance(ui, path, create)
    File “/usr/lib/python2.7/site-packages/mercurial/sshpeer.py”, line 36, in __init__
    self._abort(error.RepoError(_(“couldn’t parse location %s”) % path))
    File “/usr/lib/python2.7/site-packages/mercurial/sshpeer.py”, line 115, in _abort
    raise exception
    mercurial.error.RepoError: couldn’t parse location ssh://d2d_repository@tesla.fdm.uni-freiburg.de

    And ideas on this?

    Thank you very much.
    Wolfgang

  21. Guys, this is not the place to report issues, for that go here:

    https://github.com/felipec/git/issues

    @Mathias Hasselmann Why would rebasing fail in bazaar? When you rebase, you recreate the commits, so it’s exactly the same as making new commits.

    @mmoya That error looks interesting. Are you able to reproduce it?

    @Wulf C. Krueger The bazaar one was introduced in 1.8.2

    @Wolfgang Weird. Is that the same URL you specified originally? ssh://d2d_repository@tesla.fdm.uni-freiburg.de

    @Paul I’m working on remote-hg again and I’ve solved many bugs. I looked at gitifyhg and I don’t see what extra value it provides.

  22. Felipe: I figured you’d just up and moved on, you were so quiet. The guys working on gitifyhg have been fairly active and responsive to bug reports. I’m grateful for the work you’ve done, but, all other things being equal, I’m going to go with the product that fixes the problems I have when I use it. That’s not a one-time decision, though, so if they don’t get things done then I’ll come back. Or fork both of your projects!

  23. @FelipeC:
    Thank you for the great code, and sorry for missusing the comment section for bug reporting. I didn’t know about the tracker (Sound like a lame excuse by now…).

    As of the comment of Paul (10328) I now use gitifyhg. They give credit to you for using your code base. Is there a reason for two projects. Are you still active on your code? Can’t the two of you join to one project again?

    Now to the error I reported earlier. I already posted how I worked around it of the gitifyh page [1] but here it is again

    The repository I cloned lives directly in the home directory of its dedicated user, this is
    /home/repoUser
    Therefore, login into the machine using ssh droped me in the correct place for the repo, and hence, the URI I used to clone was

    git clone gitifyhg::ssh//repoUserSSH

    where repoUserSSH is the ssh shortcut (~./ssh/config) for the remoe machine. This was good enough for cloning, but the pull failed. Changing the clone to

    git clone gitifyhg::ssh//repoUserSSH://home/repoUser

    fixed the pull command.

    I will investigate further and let you know when I find s.th.

    [1] https://github.com/buchuki/gitifyhg/issues/67#issuecomment-15927481

  24. @Wolfgang I was able to reproduce the issue, and I believe the issue is in mercurial. However, you can workaround it doing this:

    git clone gitifyhg::ssh//repoUserSSH/

    (Notice the slash at the end)

  25. I tried to do a remote add with a fetch and got the following error:

    “`bash
    $git remote add -f extern/$REMOTE_NAME $REMOTE_URL

    Updating extern/ipython-physics
    Traceback (most recent call last):
    File “/usr/bin/git-remote-hg”, line 1024, in
    sys.exit(main(sys.argv))
    File “/usr/bin/git-remote-hg”, line 992, in main
    repo = get_repo(url, alias)
    File “/usr/bin/git-remote-hg”, line 383, in get_repo
    os.mkdir(dirname)
    OSError: [Errno 2] No such file or directory: ‘.git/hg/extern/ipython-physics’
    error: Could not fetch extern/ipython-physics
    “`

    A few variations such as just adding the remote then doing a separate fetch didn’t work either. I am able to clone the mercurial repo, however. Ultimately I’m trying to add a remote to an existing repo to then insert it as a subtree dependency. Suggestions as to how I could achieve this would be appreciated!

  26. How did you deal with the issues with Git Mercurial roundtriping (mainly legacy problems, but not only those) described in blog post “Kiln Harmony Internals: the Basics” http://blog.fogcreek.com/kiln-harmony-internals-the-basics/

    Kiln Harmony is a tool to save commit simultaneously in Git and Mercurial repository in parallel. It is a proprietary tool from FogCreek Software, but perhaps you could get their testsuite from them…

  27. @Jakub Narebski The same way hg-git does it; by using Mercurial extras, and in Git adding the extras in the commit message. I started trying that with git notes, but that required changes in git fast-export, and you know how well received changes in Git are. Round-tripping only works properly in hg-git mode.

  28. @FelipeC: By “extras in the commit message” do you mean extra commit headers (IIUC like Kiln Harmony) or at the end of commit message like git-svn?

    I’m not sure if git notes would be good solution for converting Mercurial commit to Git one. It could be a good solution for annotating a commit which didn’t came the same on round-trip, to note changes that Mercurial brought. BTW what are problems that `git fast-export` have with git-notes? Are they not composed of ordinary git objects?

    “And you know how well received changes in Git are.” — no, I don’t. I had a quite a long hiatus in contributing to Git and watching Git mailing list, and my return to it (e.g. reviewing gitweb patches) got interrupted by HDD failing :-(

    BTW. I do wonder why Kiln Harmony uses IIRC base64 encoding to stuff multiline extra fields in Git extra headers instead of using gpginfo like multiline header (for merging in signed tags) that current Git has…

  29. @Jakub Narebski

    “extras in the commit message” do you mean extra commit headers (IIUC like Kiln Harmony) or at the end of commit message like git-svn?

    I mean extra commit headers. Mercurial allows arbitrary key-values to be stored in an “extra” dictionary.

    I’m not sure if git notes would be good solution for converting Mercurial commit to Git one. It could be a good solution for annotating a commit which didn’t came the same on round-trip, to note changes that Mercurial brought. BTW what are problems that `git fast-export` have with git-notes? Are they not composed of ordinary git objects?

    We need some way to store the extra information, either notes or in the commit message.

    The problem with fast-export is that we need that extra information before we create the Mercurial commit. So, for example; a file was explicitly renamed, we need that information as-is before creating the Mercurial commit. We can extract that from the commit message that fast-export passes, but it doesn’t pass the notes associated with that commit until much later.

    “And you know how well received changes in Git are.” — no, I don’t. I had a quite a long hiatus in contributing to Git and watching Git mailing list, and my return to it (e.g. reviewing gitweb patches) got interrupted by HDD failing

    Interesting, my external HDD where I had all my work also failed.

    Nowadays Junio rejects patches without even looking at them. Check this patch series for example:

    http://thread.gmane.org/gmane.comp.version-control.git/225475

    I’ll send a summary of the status of remote-helpers in a short while, but most likely I won’t work on them any more. There is no point if my patches are not merged.

  30. We need some way to store the extra information, either notes or in the commit message.

    The problem with fast-export is that we need that extra information before we create the Mercurial commit. So, for example; a file was explicitly renamed, we need that information as-is before creating the Mercurial commit. We can extract that from the commit message that fast-export passes, but it doesn’t pass the notes associated with that commit until much later.

    On the other hand at least in some cases importer would need notes related to commit after a commit, to perhaps rewrite them to point at changed during import commit they refer to. Well, that’s what I think, but I don’t know this area of code and those issues.

    Nowadays Junio rejects patches without even looking at them. Check this patch series for example:

    http://thread.gmane.org/gmane.comp.version-control.git/225475

    What I can see here is you wh^W complaining, but I don’t see any response from Junio. Which (the las of response) is perhaps a response in itself… though this might be because lack of tuits. “What’s cooking in git.git (Jun 2013, #02; Tue, 4)” includes fc/transport-helper-no-refspec (2013-05-21) (!)

  31. @Jakub Narebski

    On the other hand at least in some cases importer would need notes related to commit after a commit, to perhaps rewrite them to point at changed during import commit they refer to.

    I couldn’t parse that. But if if there’s a legitimate reason to send notes after commits, it should be easy to ad an option to the exporter. But I don’t see it.

    What I can see here is you wh^W complaining,

    You can see it that way, or you can see it the other way; they are the ones complaining. I’m certain I’m replying to them, not the other way around.

    “What’s cooking in git.git (Jun 2013, #02; Tue, 4)” includes fc/transport-helper-no-refspec (2013-05-21) (!)

    That’s another patch series I sent, since the first one was de facto rejected.

  32. @Jakub Narebski

    As for the discussion in the mailing list. What is so wrong about this statement?

    So, instead of saying:

    “Just one side being right, and the other side continuing to repeat
    nonsense without listening.”

    You should say:

    “Simply a matter of disagreement where the code belongs.”

  33. Seems to bug out when you need to authenticate on Google code, mercurial. It just seems to break it completely.

  34. Previously the clone worked fine. But when i tried to pull the hg repo (SDL-2.0), it gave me an error:
    TypeError: Non-hexadecimal digit found

    I tried to clone a fresh repo, still the same error. I cloned successfully 2 month ago. Don’t know whether there is something new in that repo break your script. You could try:

    git clone hg::http://hg.libsdl.org/SDL

  35. @FelipeC I’m using the msysgit on windows.

    Previouly i replaced the file in git\libexec\git-core\git-remote-hg with your script from github:
    https://github.com/felipec/git/blob/fc/master/contrib/remote-helpers/git-remote-hg.py

    You mentioned there is newer version, do you mean this one?
    https://github.com/felipec/git/blob/fc/master/contrib/hg-to-git/hg-to-git.py

    Because the file in old path only contains symbolic link (and the linked path seems invalid…)

  36. Thanks FelipeC. I tried the latest version and still the same error.

    I’ve checked the code, in updatebookmarks(), the remotemarks contains a strange k-v pair:
    k=”publishing”, v=True (a boolean value), this will raise an exception in hgbin(v).

    Since this k-v pair is the only one inside remotemarks, i just ignore it and the rest of the code works fine. Don’t know why I got this strange k-v pair … I’m using Mercurial-2.6 python module.

    Anyway, now I can work on the git now. Thanks for providing this helpful software.

  37. @fantasydr It might have something to do with the specific Mercurial repository you are using. Is there a public location I can access it?

  38. I’m getting: “fatal: Unable to find remote helper for ‘hg’” on a basic git clone attempt from an hg repo.

    I have the two .py scripts in my /usr/bin, and I’ve tested that they’re in the path and are executable from the shell I’m using.

    git 1.7.9.5.
    Mint 12

    Any ideas?

  39. I tried using hg remote helper for a while, it worked “decently” (with some ugly messages that corrupted terminal) until at some point it crashed to unrecoverable (to me) state with message:

    fatal: mark :20255 not declared
    fast-import: dumping crash report to .git/fast_import_crash_2171
    fatal: Error while running fast-import

    Can not recommend for daily use.

  40. I’ve used it for weeks now and never had many problems, today though many of my repos (>30) started acting up, same message as Marko Koivusalo gets:
    fatal: mark :11632 not declared
    fast-import: dumping crash report to .git/fast_import_crash_61709
    fatal: Error while running fast-import

    I didn’t update my Git version or anything else on my Mac, checking out an old version of the repo and running pull again doesn’t help, neither does deleting branches (including master) and checking out from origin/master again, I’ll update to the newest git-hg bridge and hope I can resolve the issue.

  41. exact same issue here:
    fatal: mark :215316 not declared
    fast-import: dumping crash report to .git/fast_import_crash_62398
    fatal: Error while running fast-import

    i didn’t do anything special to get into this state. on one of my machines, i did do an update to git 1.8.5.3 earlier today, and another machine that accesses the same local repository is still running 1.8.4.2. everything was working fine (fetches, merges, rebases, etc) until suddenly i started getting this error whenever i try to fetch.

  42. I just pushed a branch to the mercurial remote but instead it created a bookmark which caused the default branch to have 2 heads which needed to be merged using hg directly.
    Deleting the branch using git didn’t work, pushing to branches/mybranch also didn’t work. Is this not possible or how can I push a local branch to my remote mercurial repo?

    Great work nevertheless, I love Git

  43. Played around a little with a local mercurial repo and cloned it, the way it works is the following:
    Create branch in the mercurial repo: hg branch myBranch
    Commit at least once: hg ci -m “Created branch myBranch”
    Pull in your git clone repo: git pull
    Switch over to your branch: git checkout branches/myBranch

    Important is the branches/ prefix and that you need to create the branch on the mercurial side, pushing and creating from git doesn’t work, also you can’t delete the branch.
    For me it also didn’t work to push from a branch which was named differently than the remote branch like:
    git push origin branchX:branches/myBranch

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s