Gitlobster

A tool for full cloning all available repositories from a GitLab server.

EN | RU

Key features

• Clone all available repositories
• Clone all branches from each repository
• Upload all repositories to another GitLab server (or a group in the same GitLab)
• Download only updates (including all newly added repositories) after the first full cloning
• Preserve the group hierarchy
• Support filters (include regexp templates) for cloning only necessary repository

Install

Docker

docker run --rm -it lowitea/gitlobster:latest --help

Running pre-assembled binary files

1. Download an archive from the release page for your OS.
2. Unpack the archive.
3. Run the gitlobster file.

Cargo

cargo install gitlobster

Building from source

# clone the repository
git clone https://github.com/lowitea/gitlobster.git
# going to the downloaded directory
cd gitlobster
# build
cargo build --release
# run
./target/release/gitlobster --help

The option to run it in the developer mode without pre-build is also available.

# in the project directory
cargo run -- --help

Usage

GitLab Token

In order for the tool to work, you need to generate a GitLab token with API read rights (read_api). If SSH copying is not used, then you will also need a permission to read repositories (read_repository).

If a second GitLab is used to copy repositories there, then a token is also required for it. Full API rights are required (api). If SSH upload is not used, then you will also need write permissions for repositories (write_repository).

You can generate tokens on the settings page.

SSH

If SSH copying is used, then ssh keys must be added in GitLab.

Help command

$ gitlobster --help

A tool for cloning all available repositories in a GitLab instance

Usage: gitlobster [OPTIONS] --fu <FETCH URL> --ft <FETCH TOKEN>

Options:
      --fu <FETCH URL>             The GitLab instance URL for fetch repositories (example: https://gitlab.local/) [env: GTLBSTR_FETCH_URL=]
      --ft <FETCH TOKEN>           Your personal GitLab token for fetch repositories [env: GTLBSTR_FETCH_TOKEN=]
      --bu <BACKUP URL>            The GitLab instance URL for backup repositories (example: https://backup-gitlab.local/) [env: GTLBSTR_BACKUP_URL=]
      --bt <BACKUP TOKEN>          Your personal GitLab token for backup repositories [env: GTLBSTR_BACKUP_TOKEN=]
      --bg <BACKUP GROUP>          A target created group on backup GitLab for push repositories [env: GTLBSTR_BACKUP_GROUP=]
  -i, --include <PATTERN>          Include regexp patterns (cannot be used together with --exclude flag, may be repeated) [env: GTLBSTR_INCLUDE=]
  -x, --exclude <PATTERN>          Comma separated exclude regexp patterns (cannot be used together with --include flag, may be repeated) [env: GTLBSTR_EXCLUDE=]
  -d, --dst <DIRECTORY>            A destination local folder for save downloaded repositories [env: GTLBSTR_DST=]
  -v, --verbose...                 Verbose level (one or more, max four)
      --dry-run                    Show all projects to download
      --objects-per-page <COUNT>   Low-level option, how many projects can fetch in one request [env: GTLBSTR_OBJECTS_PER_PAGE=]
      --limit <COUNT>              Maximum projects to download [env: GTLBSTR_LIMIT=]
      --concurrency-limit <LIMIT>  Limit concurrency download [env: GTLBSTR_CONCURRENCY_LIMIT=] [default: 21]
      --only-owned                 Download projects explicitly owned by user [env: GTLBSTR_ONLY_OWNED=]
      --only-membership            Download only user's projects [env: GTLBSTR_ONLY_MEMBERSHIP=]
      --download-ssh               Enable download by ssh instead of http. An authorized ssh key is required [env: GTLBSTR_DOWNLOAD_SSH=]
      --upload-ssh                 Enable upload by ssh instead of http. An authorized ssh key is required [env: GTLBSTR_UPLOAD_SSH=]
      --disable-hierarchy          Disable saving the directory hierarchy [env: GTLBSTR_DISABLE_HIERARCHY=]
      --clear-dst                  Clear dst path before cloning [env: GTLBSTR_CLEAR_DST=]
      --only-master                Download only default branch [env: GTLBSTR_ONLY_MASTER=]
      --disable-sync-date          Disable adding sync dates in project descriptions [env: GTLBSTR_DISABLE_SYNC_DATE=]
  -h, --help                       Print help information
  -V, --version                    Print version information

Copying all repositories to a second GitLab

gitlobster \
    --ft=<FETCH_TOKEN> \
    --fu=https://gitlab.com/ \
    --bt=<UPLOAD_TOKEN> \
    --bu=https://gitlab.com/ \
    --bg=gitlobster_test/upload

Download all repositories to a local directory

gitlobster \
    --ft=<FETCH_TOKEN> \
    --fu=https://gitlab.com/ \
    -d out_directory

Simultaneous saving repositories to a local directory and a second GitLab is supported.

Using filters and filtering flags

gitlobster \
    --ft=<FETCH_TOKEN> \
    --fu=https://gitlab.com/ \
    --only-owned \
    --include="^gitlobster_test/download/project_2" \
    --include="^gitlobster_test/download/project_1" \
    -d out_directory

It's also possible to use --exclude flag to load all repositories except repositories matching a necessary template.

Simultaneous use of both --exclude and --include flags isn't allowed.

Using with Docker

docker run --rm -it -v $(pwd)/out:/out lowitea/gitlobster:latest \
    --ft=<FETCH_TOKEN> \
    --fu=https://gitlab.com/ \
    --include='^gitlobster_test/example' \
    -d /out

Notice

Empty projects will not be migrated. Because they cannot be cloned via git until no branches are created in these projects (issue 37).

Analogues

• gitlabber
• ghorg