gitlobster 1.0.0

A tool for cloning all available repositories in a GitLab instance
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137

# Gitlobster

**_A tool for full cloning all available repositories from a GitLab server._**

EN | [RU](README.RU.md)

<br>
<br>

<p align="center"><img src="https://github.com/lowitea/gitlobster/raw/master/logo.png"></p>

<br>

## Key features

- Clone all available repositories
- Clone all branches from each repository
- Upload all repositories to another GitLab server (or a group in the same GitLab)
- Download only updates (including all newly added repositories) after the first full cloning
- Preserve the group hierarchy
- Support filters (include regexp templates) for cloning only necessary repository

## Install

### Docker

```shell
docker run --rm -it lowitea/gitlobster:latest --help
```

### Running pre-assembled binary files

1. Download an archive from [the release page](https://github.com/lowitea/gitlobster/releases) for your OS.
2. Unpack the archive.
3. Run the `gitlobster` file.

### Cargo

```shell
cargo install gitlobster
```

### Building from source

```shell
# clone the repository
git clone https://github.com/lowitea/gitlobster.git
# going to the downloaded directory
cd gitlobster
# build
cargo build --release
# run
./target/release/gitlobster --help
```

_The option to run it in the developer mode without pre-build is also available._

```shell
# in the project directory
cargo run -- --help
```

## Usage

### GitLab Token

In order for the tool to work, you need to generate a GitLab token with API read rights (`read_api`). If SSH copying is not used, then you will also a need permission to read repositories (`read_repository`).

If a second GitLab is used to copy repositories there, then a token is also required for it. Full API rights are required (`api`). If SSH upload is not used, then you will also need write permissions for repositories (`write_repository`).

You can generate tokens on [the settings page](https://github.com/-/profile/personal_access_tokens).

### SSH

If SSH copying is used, then ssh keys must be [added](https://gitlab.com/-/profile/keys) in GitLab.

### Help command

```text
$ gitlobster --help

A tool for cloning all available repositories in a GitLab instance

Usage: gitlobster [OPTIONS] --fu <FETCH URL> --ft <FETCH TOKEN>

Options:
      --fu <FETCH URL>             The GitLab instance URL for fetch repositories (example: https://gitlab.local/) [env: GTLBSTR_FETCH_URL=]
      --ft <FETCH TOKEN>           Your personal GitLab token for fetch repositories [env: GTLBSTR_FETCH_TOKEN=]
      --bu <BACKUP URL>            The GitLab instance URL for backup repositories (example: https://backup-gitlab.local/) [env: GTLBSTR_BACKUP_URL=]
      --bt <BACKUP TOKEN>          Your personal GitLab token for backup repositories [env: GTLBSTR_BACKUP_TOKEN=]
      --bg <BACKUP GROUP>          A target created group on backup GitLab for push repositories [env: GTLBSTR_BACKUP_GROUP=]
      --include <PATTERN>          Include regexp patterns (cannot be used together with --exclude flag, may be repeated) [env: GTLBSTR_INCLUDE=]
      --exclude <PATTERN>          Comma separated exclude regexp patterns (cannot be used together with --include flag, may be repeated) [env: GTLBSTR_EXCLUDE=]
  -d, --dst <DIRECTORY>            A destination local folder for save downloaded repositories [env: GTLBSTR_DST=]
  -v, --verbose...                 Verbose level (one or more, max four)
      --dry-run                    Show all projects to download
      --objects-per-page <COUNT>   Low-level option, how many projects can fetch in one request [env: GTLBSTR_OBJECTS_PER_PAGE=]
      --limit <COUNT>              Maximum projects to download [env: GTLBSTR_LIMIT=]
      --concurrency-limit <LIMIT>  Limit concurrency download [env: GTLBSTR_CONCURRENCY_LIMIT=] [default: 21]
      --only-owned                 Download projects explicitly owned by user [env: GTLBSTR_ONLY_OWNED=]
      --only-membership            Download only user's projects [env: GTLBSTR_ONLY_MEMBERSHIP=]
      --download-ssh               Enable download by ssh instead of http. An authorized ssh key is required [env: GTLBSTR_DOWNLOAD_SSH=]
      --upload-ssh                 Enable upload by ssh instead of http. An authorized ssh key is required [env: GTLBSTR_UPLOAD_SSH=]
      --disable-hierarchy          Disable saving the directory hierarchy [env: GTLBSTR_DISABLE_HIERARCHY=]
      --clear-dst                  Clear dst path before cloning [env: GTLBSTR_CLEAR_DST=]
  -h, --help                       Print help information
  -V, --version                    Print version information
```

### Copying all repositories to a second GitLab

```shell
gitlobster --ft=<FETCH_TOKEN> --fu=https://gitlab.com/ --bt=<UPLOAD_TOKEN> --bu=https://gitlab.com/ --bg=gitlobster_test/upload
```

### Download all repositories to a local directory

```shell
gitlobster --ft=<FETCH_TOKEN> --fu=https://gitlab.com/ -d out_directory
```

_Simultaneous saving repositories to a local directory and a second GitLab is supported._

### Using filters and filtering flags

```shell
gitlobster --ft=<FETCH_TOKEN> --fu=https://gitlab.com/ --only-owned --include="^gitlobster_test/download/project_2" --include="^gitlobster_test/download/project_1" -d out_directory
```

_It's also possible to use `--exclude` flag to load all repositories except repositories matching a necessary template._

_Simultaneous use of both `--exclude` and `--include` flags isn't allowed._

## Analogues

- [gitlabber](https://github.com/ezbz/gitlabber)
- [ghorg](https://github.com/gabrie30/ghorg)