Merging Two Git Repositories Into One Repository Without Losing File History

A while ago my team had code for our project spread out in two different Git repositories.  Over time we realized that there was no good reason for this arrangement and was just a general hassle and source of friction, so we decided to combine our two repositories into one repository containing both halves of the code base, with each of the old repositories in its own subdirectory.  However, we wanted to preserve all of the change history from each repo and have it available in the new repository.

The good news is that Git makes this sort of thing very easy to do.  Since a repository in Git is just a directed acyclic graph, it’s trivial to glue two graphs together and make one big graph.  The bad news is that there are a few different ways to do it and some of them end up with a less desirable result (at least for our purposes) than others.  For instance, do a web search on this subject and you’ll get a lot of information about git submodules or subtree merges, both of which are kind of complex and are designed for the situation where you’re trying to bring in source code from an external project or library and you want to bring in more changes from that project in the future, or ship your changes back to them.  One side effect of this is that when you import the source code using a subtree merge all of the files show up as newly added files.  You can see the history of commits for those files in aggregate (i.e. you can view the commits in the DAG) but if you try to view the history for a specific file in your sub-project all you’ll get is one commit for that file – the subtree merge.

This is generally not a problem for the “import an external library” scenario but I was trying to do something different.  I wanted to glue to repositories together and have them look as though they had always been one repository all along.  I didn’t need the ability to extract changes and ship them back anywhere because my old repositories would be retired.  Fortunately, after much research and trial-and-error it turned out that it’s actually very easy to do what I was trying to do and it requires just a couple of straightforward git commands.

The basic idea is that we follow these steps:

  1. Create a new empty repository New.
  2. Make an initial commit because we need one before we do a merge.
  3. Add a remote to old repository OldA.
  4. Merge OldA/master to New/master.
  5. Make a subdirectory OldA.
  6. Move all files into subdirectory OldA.
  7. Commit all of the file moves.
  8. Repeat 3-6 for OldB.

A Powershell script for these steps might look like this:

# Assume the current directory is where we want the new repository to be created
# Create the new repository
git init

# Before we do a merge, we have to have an initial commit, so we’ll make a dummy commit
dir > deleteme.txt
git add .
git commit -m “Initial dummy commit”

# Add a remote for and fetch the old repo
git remote add -f old_a <OldA repo URL>

# Merge the files from old_a/master into new/master
git merge old_a/master

# Clean up our dummy file because we don’t need it any more
git rm .\deleteme.txt
git commit -m “Clean up initial file”

# Move the old_a repo files and folders into a subdirectory so they don’t collide with the other repo coming later
mkdir old_a
dir –exclude old_a | %{git mv $_.Name old_a}

# Commit the move
git commit -m “Move old_a files into subdir”

# Do the same thing for old_b
git remote add -f old_b <OldB repo URL>
git merge old_b/master
mkdir old_b
dir –exclude old_a,old_b | %{git mv $_.Name old_b}
git commit -m “Move old_b files into subdir”

Very simple.  Now we have all the files from OldA and OldB in repository New, sitting in separate subdirectories, and we have both the commit history and the individual file history for all files.  (Since we did a rename, you have to do “git log –follow <file>” to see that history, but that’s true for any file rename operation, not just for our repo-merge.)

Obviously you could instead merge old_b into old_a (which becomes the new combined repo) if you’d rather do that – modify the script to suit.

If we have in-progress feature branches in the old repositories that also need to come over to the new repository, that’s also quite easy:

# Bring over a feature branch from one of the old repos
git checkout -b feature-in-progress
git merge -s recursive -Xsubtree=old_a old_a/feature-in-progress

This is the only non-obvious part of the whole operation.  We’re doing a normal recursive merge here (the “-s recursive” part isn’t strictly necessary because that’s the default) but we’re passing an argument to the recursive merge that tells Git that we’ve renamed the target and that helps Git line them up correctly.  This is not the same thing as the thing called a “subtree merge“.

So, if you’re simply trying to merge two repositories together into one repository and make it look like it was that way all along, don’t mess with submodules or subtree merges.  Just do a few regular, normal merges and you’ll have what you want.

114 thoughts on “Merging Two Git Repositories Into One Repository Without Losing File History”

  1. Hi – Thank you for this. I am new to git. I am testing this as a method to use to move SNV repo into a GIT repo. I have applied the steps in this page. After the commit i can view differences between files’ commits using gitk, However TortoiseGit “diff with previous” doe not work on the same file. It tells me “Could not determine the last commited revision” . Why does this no longer work? Have I lost history during the merge?

  2. This is a life saver. Out of all those links on this problem out there, this worked for me! Most of the others are linus based commands. I wanted the Windows. This one suits perfect.

  3. I would recommend doing the move on the old repository before merging it into the new repo. Then you can simply merge the repos without subtree.

  4. Awesome, and very helpful! Thanks!
    Note that you don’t really need to create another new repo, i.e., if you want to merge repo_small into repo_big, you can just add repo_small as a remote for repo_big, fetch it, and then merge. It helps if you first restructure (and commit) the files in repo_small to where you want them to be in repo_big.

    1. This is actually really helpful. I was wondering about this — I don’t know how I didn’t think of that. Thanks!

  5. questions though. Lets say I want to merge repo’s A and B into new repo AB. I do a git init –bare AB.git, then clone AB, then do these remote/merge/move/commits for A and B per your instruction. I can push origin master, and push this all to the new AB shared repo. But the tags and remote branches stay behind. Questions I have are:
    1) If A and B has the tag REL2.2 in it. post-consolidation, I have a singe REL2.2.
    I need to bugfix REL2.2. If I do a simple git checkout -b fixbug REL2.2, all the moves disappear and mkdirs disappear, and I’m left with one of the repos I remoted in.

    Is that to be expected? Because I thought a tag was attached to a commit, and if post-merge, that tag exists across both A and B, creating a branch off a tag would shift me back, and I’m not sure if the REL2.2 tag is for A or B, but it’s definitely not both.

    As for remote branches, if I do a git branch -a, I’d see
    * master
    remotes/A/branch1
    remotes/A/branch2
    remotes/A/master
    remote/B/branch1
    remotes/B/mybranch
    remotes/B/master
    remotes/origin/master

    Creating a tracking branch off any of those remote branches would also send me back as if only A (if creating tracking branch branch2), B (if mybranch), or if checking out branch1, it fails on pathspec.

    If it truly behaved as if it was always one combined repo, I would be able to check out any of those branches and it would work.

    Would I need to create a local branch1 and merge each branch in like I did master, for all remote branches that are non-origin?

    And back to tags, uh, how would that work since a typical tag is on a particular commit?

  6. And as for the creating a local branch then merging each branch, would taht really work, since creating a local branch is likely initially off master’s HEAD, and would result in one or more merge conflict since it’s likely those branches are for older releases. I tried it, and I had to manually fix the merge conflict, which can be a pain depending on how many files are conflicting, and I am not sure it’s going to pick the right versions of each file/directory. I suppose I can go back and loop (something like find . -type f | xargs diff in subdirectory vs original repo (assuming a tracking branch was created in original repo). Probably want to diff on directories too.

  7. Also, I get merge conflicts in merging repo_b/master with common files. For example, if both repo_A and repo_b contain .gitignore and pom.xml at root level (repo_A/.gitignore, repo_B/.gitignore, repo_A/pom.xml, repo_B/pom.xml).

    I pull in repo_a first. Obviously, there is nothing to conflict with when merging repo_a/master to master.

    Your process is:
    init
    add deleteme.txt file
    commit (to create master branch, no problem)
    git remote (importing in repo_A)
    git merge repo_A/master (repo_A’s pom.xml and .gitignore are in root
    remove deleteme.txt file
    commit
    then you mkdir/move/commit.

    Then, when merging repo_B/master, it conflicts over .gitignore and pom.xml because it was repo_A’s files, and now it’s conflicting with the merged repo_B’s .gitignore and pom.xml

  8. Should there be a space between ‘-X’ and ‘subtree=old_a’ in ‘git merge -s recursive -Xsubtree=old_a old_a/feature-in-progress’? Having trouble with this one…

  9. I believe it worked as written at the time I wrote this post, but it’s possible that more recent versions of Git changed that. Let me know what you find out!

  10. Hi, I have been fighting with this issue as i have had a couple of projects which need merging into a single repository. I have now managed to create a script thanks to your ideas and others which handles branches and tags of the old repositories.

    Should it interest anyone, then here it is:

    http://paste.ubuntu.com/11732805/

      1. Create a file with the list of URLs of the projects you want to merge. e.g.:

        git@github.com:eitchnet/ch.eitchnet.parent.git
        git@github.com:eitchnet/ch.eitchnet.utils.git
        git@github.com:eitchnet/ch.eitchnet.privilege.git

        Then call the script giving two arguments: first the name of the new repository to create, then the path to the file containing the URLs:

        ./mergeGitRepositories.sh my_new_repo repos.list

    1. You are awsome eitch thanks for sharing 🙂 mukul you create a file a name it whatever you want put repos urls one by line and then excute like this: ./mergerepos.sh newRepo myrepos.lst

      1. Glad to be of help – who hasn’t had to merge repos and wanted to keep the history, incl. tags and any current branches?

    2. Using eitch’s script with Git 2.11:
      -change in commit_merge() function (now outputs “working tree clean”)
      CHANGES=$(git status | grep “working directory clean”)
      MERGING=$(git status | grep “merging”)
      if [[ “$CHANGES” != “” ]] && [[ “$MERGING” == “” ]] ; then
      to
      CHANGES=$(git status –porcelain)
      if [[ “$CHANGES” = “” ]] ; then

      -when merging branches add to the git merge (as it is no longer the default)
      –allow-unrelated-histories

      -If you’re using git bash on Windows, before running make sure the list of repositories has unix line endings:
      dos2unix

    3. eitch – thanks for the script, really!

      I do have one question: when I go to my new repository and try to checkout a branch, i get:
      error: pathspec ‘some_branch’ did not match any file(s) known to git.
      Though I see in the script’s logs that it had been created.
      Were you able to move to one of your old branches? It critical for me.

      Thanks

      1. Hi Mich, actually yes, i was able to move my old branches. They are just renamed a little. You should be able to list them with “git branch”. Do you see them there?

      2. Hi eitcho,

        Thanks for responding!

        I do see the branches you talk about.
        But I have many branches with the same name in different repositories (on porpoise). And I wish to be able to git checkout a branch in my new repository, and get the correct changes in each directory (that used to be a repository).

        For example assume I used to have repA, repB, repC, and a branch branch1 in all of them.
        Now i have the following branches: repA/branch1, repC/branch1, repC/branch1, but I wish to create branch1 in the new repository that will include all the branches.
        When I try to do so – as you did with master, I get many conflicts that don’t make sense, as they were all in different repositories. Most of them are untracked files from different folder (that used to be a repository).
        Maybe there is some flag to the merge I can use, or a different place I can try to do the merge from.

        Did you encountered something like that?

      3. Hi Mich

        So i tested out my script again, i found some issues with using it on latest git, but it works again. I added an entry to my blog: https://blog.eitchnet.ch/archives/324

        Can you try it with this version? Maybe the merge didn’t work and you didn’t see it? I also add a logfile, so should be easier to spot errors.

      4. Hi eitch,

        What I managed to do, is the following:
        For each repository repository1 and branch branch1
        # Go to one of the existing branches:
        git checkout repository1/branch1
        # Create a new branch1 from the existing one.
        git checkout -b branch1
        # And then for each repository: if repository2/branch1 exists then:
        git merge repository2/branch1
        # Otherwise:
        git merge repository2/master

        Also I had to remove .gitignore from each directory (that used to be a repository) and create a merged one in the new repository.
        I’m running out of time, so I won’t be able to test your updated script.

        Super thanks, the script was really helpful!

  11. I works find until we merge back some changes from original `old_a` with `git merge -Xsubtree=old_a`
    The git log result looks weird for single file

    If I use a normal log command, it shows tow commits, one is for added while ‘move to old_a’, and the other is for bulk changes from ‘merge branch … -Xsubtree=old_a’, detail changes are lost, maybe acceptible

    If I use `git log –follow old_a/path/to/any/single/file`, the history after the move commit are totally lost, looks like that file is never changed after the -Xsubtree merge, very strange behavior

    BTW, `git blame` works well in this situation

  12. So I followed the steps, and the commit history is the repository. However, when I try to show log for certain file, all history is wiped, I only see the commit of moving the files into the sub folder. Any help?

  13. This doesn’t merge the repos seamlessly. Your workflow might be broken in a fundamental way, because git log –follow won’t work with folders, but only with files. If you go to old_a and type git log –follow . or gitk –follow . it doesn’t return old_a’s commits. This is likely a current limitation of git log.

  14. Here is how to push changes upstream. It also works for other subtree merge strategies:

    git fetch old_a
    # use an integration branch, to be on the safe side
    git checkout -b old_a-integration old_a/master
    # merge changes from master using subtree strategy
    git cherry-pick -x –strategy=subtree -Xsubtree=old_a/ master

    # Or, if you organized your old_a commits directly
    # into eg old_a-integration, just rebase:

    git branch feature-reb feature-branch
    # new branch needed to rebase without losing the commits in master
    git rebase -s subtree -Xsubtree=old_a –onto old_a-integration master feature-reb
    git checkout -b old_a-integration old_a/master
    git merge feature-reb
    git push old_a HEAD:master
    git checkout master
    git branch -D feature-reb
    git branch -D old_a-integration # I actually keep this one and even push it

    Rebase is nice, you can do it interactively which means that you have a chance to edit the commits which are rebased (inserting move-related info such as the
    origin sha1 etc). You can reorder the commits, and you can
    remove them (weeding out bad or otherwise unwanted patches).

  15. I’m stuck at
    dir –exclude old_a | %{git mv $_.Name old_a}

    I’m using zsh on Mac and keep getting this error
    zsh: parse error near `}’

    Any Mac devs reading that can help?
    Thanks!

      1. is there another way for windows (GIT BASH), I’m halfway through finishing your walkthrough and this issue came to me too. What to do now?

      1. Thanks for the script! I had to modify it because of the illegal -I option on OSX sierra but it got me started on the right path. Here are the two that I used.

        for i in $(ls); do if [ $i != “old_a” ]; then git mv $i old_a; fi; done;
        for i in $(ls); do if [ $i != “old_a” ] && [ $i != “old_b” ]; then git mv $i old_b; fi; done;

      2. For linux (avoid Ma¢ and Winbug$), this may be safer (special characters)
        “`while IFS= read file; do git mv -k “$file” old_a; done < <(ls -A | grep -v '\(\.|\..\|old_a\)')“`

  16. I’m also stuck at
    dir –exclude old_a | %{git mv $_.Name old_a}
    what shell is this? Is it some Windows cmd magic (not bash that I can see!!)

      1. For linux (avoid Ma¢ and Winbug$), this may be safer (special characters)

        while IFS= read file; do git mv -k “$file” old_a; done < <(ls -A | grep -v '\(\.|\..\|old_a\)')

  17. Thanks for that – I’ll have to think about it as I don’t fancy running a script that contains lines like “INFO: Removing remote”. I’ll want to test drive it all locally before I let it loose on the real world!

    1. I understand. But the script only does local changes. The removal of remotes is local, so that the resulting merged repository does not point to the original repositories anymore.

  18. Eitch0000 script works great, but fails when directorys contain a space in their name. It appears that there is a problem under this section. But i’m not quite sure how to fix it:

    for f in $(ls -a) ; do
    if [[ “$f” == “${sub_project}” ]] ||
    [[ “$f” == “.” ]] ||
    [[ “$f” == “..” ]] ; then
    continue
    fi
    git mv -k “$f” “${sub_project}/”
    done

    1. It would be a LOT easier – at least for me…. to use this instead.

      for f in $(ls -a | tail -n +3 | grep -v ${sub_project}); do
      git mv -k ……
      done

      the “tail -n +3” starts printing on line 3, which cuts off the first two lines (. & ..)
      the “| grep -v ${sub_project}” part is used to grep remove (-v option) the ${sub_project} from the ls listing.

      This way, in the subcommand, you do all the filtering and we don’t have all the $f comparisons as well as the “continue” all inside the (now un-necessary) “if” block.

      Sorry

      Me just putting too much on one line.

      1. Argh! – I hate arrogent programmers – like when I respond with a solution without FIRST testing!!!!.

        for f in $(ls -a | tail -n +3 | grep -v ${sub_project}$); do

        Needed to add the “$” at the end for the grep or it leaves behind branches with the sub_project string embedded int it.

        my humble apologies for wasting your time with the wrong answer.

    2. For linux (avoid Ma¢ and Winbug$), this may be safer (special characters)
      while IFS= read file; do git mv -k “$file” old_a; done < <(ls -A | grep -v '\(\.|\..\|old_a\)')

  19. OK, I figured out the space issue. You have to change the internal field seperator that Bash uses to something else temporarily so that the for f in $(ls -a); loop will work properly.

    Here is an updated script with all others contributions and the IFS (Internal Field Seperator) fix.

    http://paste.ubuntu.com/25825168/

    1. The IFS fix is interesting…. but a simple “while read” as used in 2 other sections of the code, will get lines instead of space delimited strings.

      while read f ; do
      git mv -k “$f” “${sub_project}/”
      done < <(ls -a | tail -n +3 | grep -v "${sub_project}$")

      Since there is the possibility of spaces, you need to put the ${sub_project}$ into double quotes.

      None of my folders had spaces, so I will admit up front, that this is un-tested.

      1. I had many headaches with this type of thing in the past. I use something like this:

        while IFS= read file; do git mv -k “$file” old_a; done < <(ls -A | grep -v '\(\.|\..\|old_a\)')

        I read several things about it. It was some time ago, and don't remember the source, but this seems legit:
        http://wiki.bash-hackers.org/commands/builtin/read
        Notice that I _did_ leave the -r out (it works for me)

    1. Help please. I tried this solution and it worked for oldA, but when I tried to do the same thing for oldB, git does not complain, but files are not copied into my repository.

  20. I’m combining 50 repos into one and find that the history graph is very wide, I was hoping that fast-forward would flatten it out more but I understand that is on by default .. any suggestions on how to flatten the history?

  21. I’ve read through the description and the comments. Just to verify: it seems that the latest version of the script by @eitch is this one: https://blog.eitchnet.ch/archives/324. That’s what I’m using, together with git v. 2.18.0.

    One problem I’m having is that ‘git status –short’ seems to return no output even when .git/MERGE_HEAD exists (i.e., git is in ‘merging’ state). Is that a valid test, in function commit_merge, for whether or not to commit?

    I have a large project with 29 repositories to merge. I’m doing a proof-of-concept with just five small ones. When I run the script, the ‘while read branch’ loop, which calls commit_merge, seems to work just fine. When I get to the final loop for merging master branches into new repository, which also calls commit_merge, it fails during ‘git merge’ on the third old repo (ccc), because the second old repo (bbb) left the new repo in merging state. Here is the tail end of the log:

    :
    :
    INFO: Merging projects master branches into new repository...
    ====================================================
    INFO:   Merging aaa...
    Updating 71354fcf..02dcecc0
    Fast-forward
    INFO:   No commit required.
    INFO:   Merging bbb...
    Already up to date!
    Automatic merge went well; stopped before committing as requested
    INFO:   No commit required.
    INFO:   Merging ccc...
    fatal: You have not concluded your merge (MERGE_HEAD exists).
    Please, commit your changes before you merge.
    ERROR: Merging of projects failed:
    

    Note that we don’t use branch ‘master’ to do our work, but there is a branch master in the old bbb repo with one commit. There is also one commit in the old aaa repo, which the script seems to be getting past.

    1. Hi tanager

      I have updated the script on my webpage so that it now uses a more robust way of detecting if a merge is in progress. Can you try my version 0.3.2?

      Kind regards
      Robert

      1. Robert,
        Your version 0.3.2 of the script seems to have a couple typos (one of them also in 0.3.0).

        The usage in the initial comments, and in the “ERROR: Usage…” line are wrong. No arguments are mentioned.
        The IFS value in the new 0.3.2 script seems to be wrong. It is closed with a single quote, but not started with a single quote.

        After fixing these, I’m still having problems, but I’m not ready to share them until I understand them better myself. The script now (with the new way of detecting ‘merging’ state) fails much sooner than it did before. Update coming soon…

      2. OK, after adding at least one file to each master branch, the script works. We don’t use ‘master’ and most repos only had a master with a single commit (and no files) that came from when we converted from SVN to git.

        So, this segues into my next problem. We have branches representing releases and for each repo, the release branch has the same name. So, in my example above, I want to checkout branch 2.5 for aaa, bbb and ccc simultaneously. But the branches I end up with using this script are aaa/2.5, bbb/2.5 and ccc/2.5, so there is no way to check out a single branch that produces the HEAD from the old aaa project on 2.5, the HEAD of bbb and the head of ccc, all at once.

      3. Hi tanager

        I am sorry, i didn’t realize wordpress borked the formatting. I updated the site with a link to a

      4. Looks like I need to merge to release branch 2.5 instead of master (the final loop), and once for each release branch I care to keep.

      5. Hi Tanager,

        Looks like I need to merge to release branch 2.5 instead of master (the final loop), and once for each release branch I care to keep.

        Did you succeed with merging your repos into common release branches? We have exactly the same challenge.

        Best regards
        Michael

      6. With version 0.3.2 there’s a problem in line 130 with “git fetch …” inside the if-clause. With my Git 2.21.0 (Windows) for the second repo it brings the warning “no common commits”, which causes the if-clause to fail. If I remove the if-clause (and thus the error checking), the script proceeds.
        I run the Bash script from the Git Bask console.

  22. Hmm… i pressed the wrong button.

    If you need help, maybe you could send me an example of your repos and i can try and fix it in the script.

    1. Thanks. The link to the 0.3.2 script looks much better now.

      I think I’ve got the idea for handling my branching scheme by fiddling with the final loop. For each branch B that I need to preserve, I will create and checkout B, then merge ${sub_project}/B into B, just as the script does for master.

      The only thing I think will be difficult (or impossible) is preserving our tags. They are much the same as our branches, spanning all the repos. So, the state of my new combined repo, when at tag T, should mimic the state of each of the old repos when each repo is at its own tag T. I would need some way to checkout each sub_project tag T at the same time, then create the new global tag T. The only other way would be to somehow recreate the new repo in chronological order and create the tags as I go–seems impossibly complicated.

      1. After a few tweaks, I’ve got something that makes it through all my repos without dying. However, the final result is still not quite right–some missing paths still.

        One warning I’m getting a few times is as follows; wondering if it is the cause of my current issues.

        Warning: you are leaving 3 commits behind, not connected to
        any of your branches:
        
          2aeb201c9 blah, blah, blah
         816d8d7a6 kilroy was here
          fa87abe80 whatever you do, don't look down
        
        If you want to keep them by creating a new branch, this may be a good time   
        to do so with:
        
         git branch <new-branch-name> 2aeb201c9
        
        Switched to a new branch 'cfg/30.2'
        

        This happens when I’m about to do ‘creating local branches for cfg…’ (the ‘git checkout -b’ command for repo X+1) and the three stranded commits are actually from the previous repo, call it repo X. How could the script leave commits in this state? Any ideas?

  23. Hi eitch
    At first I really thanks for sharing your script and support it.
    I have two question
    1)can I use this script to merge projects in Gitlab ?
    If yes:
    2) could you share with me link of the latest version your script to reply my message because i found version 0.2.0 in: https://paste.ubuntu.com/11732805/
    thanks in advance.

    1. I found it.
      I have a problem with this script. during merge process bash ask me user and password frequently.
      how can i run script without enter user and password frequently.

      1. Hi Sorry for the late reply, here you can find the latest version: .gist table { margin-bottom: 0; } This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters #!/bin/bash # ################################################################################ ## Script to merge multiple git repositories into a new repository ## – The new repository will contain a folder for every merged repository ## – The script adds remotes for every project and then merges in every branch ##  and tag. These are renamed to have the origin project name as a prefix ## ## Usage: mergeGitRepositories.sh <new_project> <my_repo_urls.lst> ## – where <new_project> is the name of the new project to create ## – and <my_repo_urls.lst> is a file contaning the URLs to the respositories ## which are to be merged on separate lines. ## ## Author: Robert von Burg ## eitch@eitchnet.ch ## ## Version: 0.3.2 ## Created: 2018-02-05 ## ################################################################################ # # disallow using undefined variables shopt -s -o nounset # Script variables declare SCRIPT_NAME="${0##*/}" declare SCRIPT_DIR="$(cd ${0%/*} ; pwd)" declare ROOT_DIR="$PWD" IFS=$'n' # Detect proper usage if [ "$#" -ne "2" ] ; then echo -e "ERROR: Usage: $0 <new_project> <my_repo_urls.lst>" exit 1 fi ## Script variables PROJECT_NAME="${1}" PROJECT_PATH="${ROOT_DIR}/${PROJECT_NAME}" TIMESTAMP="$(date +%s)" LOG_FILE="${ROOT_DIR}/${PROJECT_NAME}_merge.${TIMESTAMP}.log" REPO_FILE="${2}" REPO_URL_FILE="${ROOT_DIR}/${REPO_FILE}" # Script functions function failed() { echo -e "ERROR: Merging of projects failed:" echo -e "ERROR: Merging of projects failed:" >>${LOG_FILE} 2>&1 echo -e "$1" exit 1 } function commit_merge() { current_branch="$(git symbolic-ref HEAD 2>/dev/null)" if [[ ! -f ".git/MERGE_HEAD" ]] ; then echo -e "INFO: No commit required." echo -e "INFO: No commit required." >>${LOG_FILE} 2>&1 else echo -e "INFO: Committing ${sub_project}…" echo -e "INFO: Committing ${sub_project}…" >>${LOG_FILE} 2>&1 if ! git commit -m "[Project] Merged branch '$1' of ${sub_project}" >>${LOG_FILE} 2>&1 ; then failed "Failed to commit merge of branch '$1' of ${sub_project} into ${current_branch}" fi fi } # Make sure the REPO_URL_FILE exists if [ ! -e "${REPO_URL_FILE}" ] ; then echo -e "ERROR: Repo file ${REPO_URL_FILE} does not exist!" exit 1 fi # Make sure the required directories don't exist if [ -e "${PROJECT_PATH}" ] ; then echo -e "ERROR: Project ${PROJECT_NAME} already exists!" exit 1 fi # create the new project echo -e "INFO: Logging to ${LOG_FILE}" echo -e "INFO: Creating new git repository ${PROJECT_NAME}…" echo -e "INFO: Creating new git repository ${PROJECT_NAME}…" >>${LOG_FILE} 2>&1 echo -e "====================================================" echo -e "====================================================" >>${LOG_FILE} 2>&1 cd ${ROOT_DIR} mkdir ${PROJECT_NAME} cd ${PROJECT_NAME} git init echo "Initial Commit" > initial_commit # Since this is a new repository we need to have at least one commit # thus were we create temporary file, but we delete it again. # Deleting it guarantees we don't have conflicts later when merging git add initial_commit git commit –quiet -m "[Project] Initial Master Repo Commit" git rm –quiet initial_commit git commit –quiet -m "[Project] Initial Master Repo Commit" echo # Merge all projects into the branches of this project echo -e "INFO: Merging projects into new repository…" echo -e "INFO: Merging projects into new repository…" >>${LOG_FILE} 2>&1 echo -e "====================================================" echo -e "====================================================" >>${LOG_FILE} 2>&1 for url in $(cat ${REPO_URL_FILE}) ; do if [[ "${url:0:1}" == '#' ]] ; then continue fi # extract the name of this project export sub_project=${url##*/} sub_project=${sub_project%*.git} echo -e "INFO: Project ${sub_project}" echo -e "INFO: Project ${sub_project}" >>${LOG_FILE} 2>&1 echo -e "—————————————————-" echo -e "—————————————————-" >>${LOG_FILE} 2>&1 # Fetch the project echo -e "INFO: Fetching ${sub_project}…" echo -e "INFO: Fetching ${sub_project}…" >>${LOG_FILE} 2>&1 git remote add "${sub_project}" "${url}" if ! git fetch –tags –quiet ${sub_project} >>${LOG_FILE} 2>&1 ; then failed "Failed to fetch project ${sub_project}" fi # add remote branches echo -e "INFO: Creating local branches for ${sub_project}…" echo -e "INFO: Creating local branches for ${sub_project}…" >>${LOG_FILE} 2>&1 while read branch ; do branch_ref=$(echo $branch | tr " " "t" | cut -f 1) branch_name=$(echo $branch | tr " " "t" | cut -f 2 | cut -d / -f 3-) echo -e "INFO: Creating branch ${branch_name}…" echo -e "INFO: Creating branch ${branch_name}…" >>${LOG_FILE} 2>&1 # create and checkout new merge branch off of master if ! git checkout -b "${sub_project}/${branch_name}" master >>${LOG_FILE} 2>&1 ; then failed "Failed preparing ${branch_name}" ; fi if ! git reset –hard ; then failed "Failed preparing ${branch_name}" >>${LOG_FILE} 2>&1 ; fi if ! git clean -d –force ; then failed "Failed preparing ${branch_name}" >>${LOG_FILE} 2>&1 ; fi # Merge the project echo -e "INFO: Merging ${sub_project}…" echo -e "INFO: Merging ${sub_project}…" >>${LOG_FILE} 2>&1 if ! git merge –allow-unrelated-histories –no-commit "remotes/${sub_project}/${branch_name}" >>${LOG_FILE} 2>&1 ; then failed "Failed to merge branch 'remotes/${sub_project}/${branch_name}' from ${sub_project}" fi # And now see if we need to commit (maybe there was a merge) commit_merge "${sub_project}/${branch_name}" # relocate projects files into own directory if [ "$(ls)" == "${sub_project}" ] ; then echo -e "WARN: Not moving files in branch ${branch_name} of ${sub_project} as already only one root level." echo -e "WARN: Not moving files in branch ${branch_name} of ${sub_project} as already only one root level." >>${LOG_FILE} 2>&1 else echo -e "INFO: Moving files in branch ${branch_name} of ${sub_project} so we have a single directory…" echo -e "INFO: Moving files in branch ${branch_name} of ${sub_project} so we have a single directory…" >>${LOG_FILE} 2>&1 mkdir ${sub_project} for f in $(ls -a) ; do if [[ "$f" == "${sub_project}" ]] || [[ "$f" == "." ]] || [[ "$f" == ".." ]] ; then continue fi git mv -k "$f" "${sub_project}/" done # commit the moving if ! git commit –quiet -m "[Project] Move ${sub_project} files into sub directory" ; then failed "Failed to commit moving of ${sub_project} files into sub directory" fi fi echo done < <(git ls-remote –heads ${sub_project}) # checkout master of sub probject if ! git checkout "${sub_project}/master" >>${LOG_FILE} 2>&1 ; then failed "sub_project ${sub_project} is missing master branch!" fi # copy remote tags echo -e "INFO: Copying tags for ${sub_project}…" echo -e "INFO: Copying tags for ${sub_project}…" >>${LOG_FILE} 2>&1 while read tag ; do tag_ref=$(echo $tag | tr " " "t" | cut -f 1) tag_name_unfixed=$(echo $tag | tr " " "t" | cut -f 2 | cut -d / -f 3) # hack for broken tag names where they are like 1.2.0^{} instead of just 1.2.0 tag_name="${tag_name_unfixed%%^*}" tag_new_name="${sub_project}/${tag_name}" echo -e "INFO: Copying tag ${tag_name_unfixed} to ${tag_new_name} for ref ${tag_ref}…" echo -e "INFO: Copying tag ${tag_name_unfixed} to ${tag_new_name} for ref ${tag_ref}…" >>${LOG_FILE} 2>&1 if ! git tag "${tag_new_name}" "${tag_ref}" >>${LOG_FILE} 2>&1 ; then echo -e "WARN: Could not copy tag ${tag_name_unfixed} to ${tag_new_name} for ref ${tag_ref}" echo -e "WARN: Could not copy tag ${tag_name_unfixed} to ${tag_new_name} for ref ${tag_ref}" >>${LOG_FILE} 2>&1 fi done < <(git ls-remote –tags –refs ${sub_project}) # Remove the remote to the old project echo -e "INFO: Removing remote ${sub_project}…" echo -e "INFO: Removing remote ${sub_project}…" >>${LOG_FILE} 2>&1 git remote rm ${sub_project} echo done # Now merge all project master branches into new master git checkout –quiet master echo -e "INFO: Merging projects master branches into new repository…" echo -e "INFO: Merging projects master branches into new repository…" >>${LOG_FILE} 2>&1 echo -e "====================================================" echo -e "====================================================" >>${LOG_FILE} 2>&1 for url in $(cat ${REPO_URL_FILE}) ; do if [[ ${url:0:1} == '#' ]] ; then continue fi # extract the name of this project export sub_project=${url##*/} sub_project=${sub_project%*.git} echo -e "INFO: Merging ${sub_project}…" echo -e "INFO: Merging ${sub_project}…" >>${LOG_FILE} 2>&1 if ! git merge –allow-unrelated-histories –no-commit "${sub_project}/master" >>${LOG_FILE} 2>&1 ; then failed "Failed to merge branch ${sub_project}/master into master" fi # And now see if we need to commit (maybe there was a merge) commit_merge "${sub_project}/master" echo done # Done cd ${ROOT_DIR} echo -e "INFO: Done." echo -e "INFO: Done." >>${LOG_FILE} 2>&1 echo exit 0 view raw mergeGitRepositories.sh hosted with ❤ by GitHub If you have to enter the password all the time, then that is because you are cloning the projects of the way you are cloning the projects. If using git, then make sure your public key is known by the remote server, and you probably don’t want to use HTTPS as this authentication scheme does not cache credentials.
  24. I have noticed you don’t monetize saintgimp.org, don’t waste
    your traffic, you can earn additional bucks every month with new monetization method.

    This is the best adsense alternative for any type of
    website (they approve all websites), for more details simply search
    in gooogle: murgrabia’s tools

  25. I am using windows termnal and I’m stuck at ,
    dir > exclude old_a | %{git mv $_.Name old_a} .
    Below is what I get,
    ‘%{git’ is not recognized as an internal or external command,
    operable program or batch file.

  26. This guide was really helpful. Thank you!

    You could perhaps streamline it slightly by just initializing an empty repo initially (no dummy file):

    git commit -m “Initializing empty merge repo” –allow-empty

  27. Bring over a feature branch from one of the old repos

    If I follow this procedure Git tells me “not something we can merge”. I tried with several open feature branches 😦

  28. Works well with the following modifications :

    After git remote add -f old_a <OldA repo URL>

    Add git fetch --all because the code to merge has to be available locally. Thus this will fetch old_a.

    And, as mentionned by @AndreaLigios, git merge old_a/master has to be replaced by:
    git merge old_a/master --allow-unrelated-histories.

    1. This one works well as well if you’re getting errors.
      notice ‘git fetch’ and I am using ‘main’ instead of master

      Add a remote for and fetch the old repo

      git remote add -f old_a

      #git fetch –all

      Merge the files from old_a/master into new/master

      git merge old_a/main –allow-unrelated-histories

  29. How does it work when i want to push the master of old repo into a featurebranch of an existing repo?

  30. Hello,
    I’m new to git, but just have one question. If we merge multiple repos, is it possible to have different builds and different pipelines for each of the merged repo? Even merged into one, I want to keep the builds and pipelines separate for each.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.