pazi 0.5.0

A fast autojump helper
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209

# Benchmarking pazi

Pazi is benchmarked against other autojump utilities. Currently, it's
benchmarked against [`fasd`](https://github.com/clvv/fasd/), [`autojump`](https://github.com/wting/autojump), and [`z`](https://github.com/rupa/z).

The benchmarks are maintained under the 'tests' directory in this repository.

## Benchmark methodology

### Test system

These benchmarks were all run on a machine with an i5-4670K and a Samsung 850 EVO SSD.
The following versions of relevant software were installed:

| software | version |
|:--------:|:-------:|
| bash     | `GNU bash, version 4.4.12` |
| zsh | `zsh 5.5.1` |
| autojump | commit [6a529f4](https://github.com/wting/autojump/commit/6a529f4f929042a37e7394d8c91ba31b479189a2) |
| fasd | commit [90b531a](https://github.com/clvv/fasd/commit/90b531a5daaa545c74c7d98974b54cbdb92659fc) |
| pazi | commit [e56d571](https://github.com/euank/pazi/commit/e56d5713d8f492cccde46cdbeba9e9fbf2d2ed2c) |
| z | commit [ea5ec78](https://github.com/rupa/z/commit/ea5ec7834398ee2244de26953b3d1ef785d3f942) |
| rust | 1.29.0-nightly |

### Procedure

Each benchmark was run once to warm up, and then once to get the value used. Benchmarks were run from the repository via:

```
$ cargo build --release && cd tests
$ cargo bench --features=nightly,cgroups2 <benchmark name>
```

Analysis is based on my understanding of the benchmarks and software under analysis.

## Benchmarks

The below is a list of benchmarks which I think can be meaningfully interpreted.

### cd bench

The `cd_bench_normal` benchmark tries to evaluate the cost of the autojump program
maintaining its frecency list during regular usage.
The benchmark accesses directories, causing the autojump program under test to
add the directory to its history.

This benchmark also includes a benchmark for performing the same operations
with no autojump program installed. This provides a baseline for how slow the
shell+vte+test harness are without any additional autojump accounting.

Without further adieu, the results:

| Autojump Program | Shell  | `cd`/s   |  ms/`cd`  |
|------------------|--------|----------|-----------|
| None             | `bash` | 10870    |  .092     |
| None             | `zsh`  | 32258    |  .031     |
| `autojump`       | `bash` | 60       |  16.771   |
| `autojump`       | `zsh`  | 56       |  17.765   |
| `fasd`           | `bash` | 66       |  15.218   |
| `fasd`           | `zsh`  | 73       |  13.770   |
| `pazi`           | `bash` | 599      |   1.669   |
| `pazi`           | `zsh`  | 702      |   1.423   |
| `z`              | `bash` | 754      |   1.327   |


<!--
test bench::cd_bench_normal_autojump_bash     ... bench:  16,771,453 ns/iter (+/- 9,867,754)
test bench::cd_bench_normal_autojump_zsh      ... bench:  17,764,760 ns/iter (+/- 7,924,858)
test bench::cd_bench_normal_fasd_bash         ... bench:  15,218,044 ns/iter (+/- 553,219)
test bench::cd_bench_normal_fasd_zsh          ... bench:  13,769,654 ns/iter (+/- 542,622)
test bench::cd_bench_normal_nojumper_bash     ... bench:      92,764 ns/iter (+/- 4,920)
test bench::cd_bench_normal_nojumper_zsh      ... bench:      31,373 ns/iter (+/- 3,654)
test bench::cd_bench_normal_pazi_bash         ... bench:   1,669,345 ns/iter (+/- 215,517)
test bench::cd_bench_normal_pazi_zsh          ... bench:   1,422,955 ns/iter (+/- 150,071)
test bench::cd_bench_normal_z_bash            ... bench:   1,327,210 ns/iter (+/- 281,970)
-->

Using an autojump program clearly has a significant impact on a shell's performance.
However, with pazi that impact is clearly very small.

Using pazi introduces around 1.5 milliseconds of delay between a command completing and a new shell prompt being available. This 1.5 milliseconds will often not be noticible, especially since 60hz displays are only refreshing every 16ms.

The fastest autojumper in this specific benchmark is `z`, which is
neck-and-neck with pazi. Unfortunately, under `zsh`, `z` exhibits a serious bug
often enough that it cannot be benchmarked in this method.

This test does not show all of the differences between the impact these programs have though.
In addition to the visible "time to next command prompt" this benchmark shows, there's the "time to directory processed". This time is the same for pazi and fasd.
However, `autojump` and `z` both fork work into the background during `cd`s.
Even if a new shell prompt is visible in a millisecond with `z`, it continues
to consume resources for a short while more.

The next benchmark tries to measure that difference as well.

## cd sync bench

The `cd_bench` benchmark above, simply measures how long it takes a new shell prompt to be available for use after running a `cd` command.
This does not capture any work an autojumper might do in the background.
The `cd_sync` benchmark is meant to measure that as well. It operates by
running `cd`, and then waiting for a shell to be visible *and* for no
additional background processes to be running under the shell.

This captures how long it takes autojumpers that fork into the background to quit using resources.

The results for this benchmark are:

| Autojump Program | Shell  | `cd`/s   |  ms/`cd`  |
|------------------|--------|----------|-----------|
| None             | `bash` | 10000    |  .100     |
| None             | `zsh`  | 32258    |  .031     |
| `autojump`       | `bash` | 19       |  52.293   |
| `autojump`       | `zsh`  | 19       |  52.123   |
| `fasd`           | `bash` | 66       |  15.237   |
| `fasd`           | `zsh`  | 73       |  13.728   |
| `pazi`           | `bash` | 644      |   1.552   |
| `pazi`           | `zsh`  | 733      |   1.364   |
| `z`              | `bash` | 262      |   3.813   |
| `z`              | `zsh`  | 319      |   3.133   |


<!--
test bench::cd_bench_sync_autojump_bash       ... bench:  52,292,682 ns/iter (+/- 1,024,399)
test bench::cd_bench_sync_autojump_zsh        ... bench:  52,123,162 ns/iter (+/- 1,738,625)
test bench::cd_bench_sync_fasd_bash           ... bench:  15,237,464 ns/iter (+/- 369,652)
test bench::cd_bench_sync_fasd_zsh            ... bench:  13,728,382 ns/iter (+/- 326,905)
test bench::cd_bench_sync_nojumper_bash       ... bench:     100,451 ns/iter (+/- 3,779)
test bench::cd_bench_sync_nojumper_zsh        ... bench:      30,548 ns/iter (+/- 1,674)
test bench::cd_bench_sync_pazi_bash           ... bench:   1,551,545 ns/iter (+/- 168,553)
test bench::cd_bench_sync_pazi_zsh            ... bench:   1,363,778 ns/iter (+/- 102,843)
test bench::cd_bench_sync_z_bash              ... bench:   3,812,556 ns/iter (+/- 267,864)
test bench::cd_bench_sync_z_zsh               ... bench:   3,132,578 ns/iter (+/- 138,502)
-->

For this benchmark, we can run `z` under both `bash` and `zsh` since the bugs
which prevented  it from working in the previous cd benchmark happen to be
caused by multiple processes forking and modifying its data file in parallel.

This benchmark shows that, when taking into account processes forking into the background, pazi is clearly the fastest. Vroom vroom.

## jump bench

The `jump_bench` benchmark attempts to measure how long it takes from typing `z
<directory>` to having your shell's CWD changed to that directory.

Notably, this benchmark is performed with only a minimal number of entries in
the benchmark program's database, meaning the autojump software need not load
much data from disk.

| Autojump Program | Shell  | jumps/s                |  ms/jump  |
|------------------|--------|------------------------|-----------|
| `autojump`       | `bash` | 18                     |  55.875   |
| `autojump`       | `zsh`  | 18                     |  56.440   |
| `fasd`           | `bash` | 32                     |  31.463   |
| `fasd`           | `zsh`  | 33                     |  29.888   |
| `pazi`           | `bash` | 502                    |   1.993   |
| `pazi`           | `zsh`  | 348                    |   2.873   |
| `z`              | `bash` | 242                    |   4.126   |

<!--
test bench::jump_bench_autojump_bash          ... bench:  55,874,896 ns/iter (+/- 2,322,104)
test bench::jump_bench_autojump_zsh           ... bench:  56,440,482 ns/iter (+/- 1,956,512)
test bench::jump_bench_fasd_bash              ... bench:  31,462,861 ns/iter (+/- 739,573)
test bench::jump_bench_fasd_zsh               ... bench:  29,888,266 ns/iter (+/- 396,160)
test bench::jump_bench_pazi_bash              ... bench:   1,992,635 ns/iter (+/- 149,604)
test bench::jump_bench_pazi_zsh               ... bench:   2,872,545 ns/iter (+/- 334,004)
test bench::jump_bench_z_bash                 ... bench:   4,125,595 ns/iter (+/- 428,407)
-->

We can see that pazi is twice as quick as the alternatives, though again we can't measure `z` on `zsh` due to bugs.

Both `autojump` and `fasd` end up over 10 times slower.
I suspect autojump's slowness her, and elsewhere, is in no small part due  to loading the python interpreter.
fasd's speed, I suspect, is limited by its `awk` program used to parse its
database, though I have not done in depth profiling to be sure.

## Large database jump bench

The `jump_large_db` benchmark measures the same thing as the previous jump
benchmark, but after filling the database with 1000 entries.

This measures more typical jump performance since the database will usually have quite a few entries.

| Autojump Program | Shell  | jumps/s                |  ms/jump  |
|------------------|--------|------------------------|-----------|
| `autojump`       | `bash` | 22                     |  95.099   |
| `autojump`       | `zsh`  | 21                     |  94.594   |
| `fasd`           | `bash` | 49                     |  35.512   |
| `fasd`           | `zsh`  | 40                     |  34.214   |
| `pazi`           | `bash` | 49                     |  17.763   |
| `pazi`           | `zsh`  | 40                     |  22.021   |
| `z`              | `bash` | 40                     |  17.811   |


<!--
test bench::jump_large_db_bench_autojump_bash ... bench:  95,099,457 ns/iter (+/- 2,912,535)
test bench::jump_large_db_bench_autojump_zsh  ... bench:  94,594,256 ns/iter (+/- 2,973,848)
test bench::jump_large_db_bench_fasd_bash     ... bench:  35,512,474 ns/iter (+/- 1,097,605)
test bench::jump_large_db_bench_fasd_zsh      ... bench:  34,214,217 ns/iter (+/- 645,826)
test bench::jump_large_db_bench_pazi_bash     ... bench:  17,762,955 ns/iter (+/- 538,958)
test bench::jump_large_db_bench_pazi_zsh      ... bench:  22,021,296 ns/iter (+/- 343,753)
test bench::jump_large_db_bench_z_bash        ... bench:  17,811,464 ns/iter (+/- 1,512,789)
-->

Pazi does better than the competition here, but its performance degrades
significantly worse than `fasd`s. This is a clear sign that pazi's use of
msgpack may be non-optimial. Further investigation will be done to improve this
performance in Pazi.

Pazi is also  close enough to `z` we might as well call that comparison a tie.