Written by Karl Ding
on 

Fixing a crash in git-absorb without knowing any Rust

I fixed a crash in git-absorb, and learned how to bisect Rust crates for the first time without knowing how to write any Rust.

I was using git-absorb and hit a segmentation fault when trying to absorb some changes:

# in the repo with changes
$ git add -u .
$ git absorb
Segmentation fault (core dumped)

I had been using the pre-built binaries (absorb-0.6.7-x86_64-unknown-linux-musl.tar.gz), which are statically linked against musl. However, since the pre-built binaries don't have debug symbols, I tried building from source to hopefully allow me to gather a backtrace and see where the problem was.

# in the tummychow/git-absorb repo
$ git log -n1 HEAD
219e386ff665b4fd360b397752ce51a658c5e1d6
$ cargo clean && cargo build --locked

After building, before performing any further debugging I first ensured that I could still reproduce the failure. This gave the following message:

# in the repo with changes
$ git absorb
free(): double free detected in tcache 2
Aborted (core dumped)

Running this under rust-gdb gave the following backtrace:

 >>> bt
 #0  0x00007ffff7dac00b in raise () from /lib/x86_64-linux-gnu/libc.so.6
 #1  0x00007ffff7d8b859 in abort () from /lib/x86_64-linux-gnu/libc.so.6
 #2  0x00007ffff7df626e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 #3  0x00007ffff7dfe2fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 #4  0x00007ffff7dfff6d in ?? () from /lib/x86_64-linux-gnu/libc.so.6
 #5  0x000055555564fe5b in stdalloc__free (ptr=0x5555559ee2e0) at libgit2/src/allocators/stdalloc.c:104
 #6  0x0000555555611277 in git_mwindow_close_lru_window () at libgit2/src/mwindow.c:269
 #7  0x00005555556114c2 in new_window (fd=7, size=92396285398, offset=49063184399) at libgit2/src/mwindow.c:337
 #8  0x00005555556116df in git_mwindow_open (mwf=0x5555559d50e0, cursor=0x7fffffff1f58, offset=49063184399, extra=20, left=0x7fffffff1ee0) at libgit2/src/mwindow.c:407
 #9  0x000055555564c156 in git_packfile_unpack_header (size_p=0x7fffffff1f70, type_p=0x7fffffff1f4c, mwf=0x5555559d50e0, w_curs=0x7fffffff1f58, curpos=0x7fffffff1f60) at libgit2/src/pack.c:455
 #10 0x000055555564c6c2 in pack_dependency_chain (chain_out=0x7fffffff2040, cached_out=0x7fffffff2010, cached_off=0x7fffffff2920, small_stack=0x7fffffff20a0, stack_sz=0x7fffffff2018, p=0x5555559d50e0, obj_offset=49063184399) at libgit2/src/pack.c:581
 #11 0x000055555564c8bc in git_packfile_unpack (obj=0x7fffffff2900, p=0x5555559d50e0, obj_offset=0x7fffffff2920) at libgit2/src/pack.c:637
 #12 0x0000555555667e74 in pack_backend__read (buffer_p=0x7fffffff29b0, len_p=0x7fffffff29b8, type_p=0x7fffffff29c0, backend=0x5555559c4f90, oid=0x555555abb145) at libgit2/src/odb_pack.c:400
 #13 0x000055555564292d in odb_read_1 (out=0x7fffffff2a88, db=0x5555559ca3d0, id=0x555555abb145, only_refreshed=false) at libgit2/src/odb.c:1065
 #14 0x0000555555642b3c in git_odb_read (out=0x7fffffff2a88, db=0x5555559ca3d0, id=0x555555abb145) at libgit2/src/odb.c:1116
 #15 0x0000555555611ff5 in git_object_lookup_prefix (object_out=0x7fffffff2b50, repo=0x5555559b4fe0, id=0x555555abb145, len=40, type=GIT_OBJECT_TREE) at libgit2/src/object.c:222
 #16 0x00005555556120c7 in git_object_lookup (object_out=0x7fffffff2b50, repo=0x5555559b4fe0, id=0x555555abb145, type=GIT_OBJECT_TREE) at libgit2/src/object.c:253
 #17 0x00005555556097aa in git_tree_lookup (out=0x7fffffff2b50, repo=0x5555559b4fe0, id=0x555555abb145) at libgit2/src/object_api.c:56
 #18 0x000055555566ab6c in tree_iterator_frame_push (iter=0x5555559cdd70, entry=0x555555ab6c48) at libgit2/src/iterator.c:660
 #19 0x000055555566b130 in tree_iterator_advance (out=0x7fffffff2ca8, i=0x5555559cdd70) at libgit2/src/iterator.c:813
 #20 0x0000555555625006 in git_iterator_advance (entry=0x7fffffff2ca8, iter=0x5555559cdd70) at libgit2/src/iterator.h:184
 #21 0x000055555562739c in iterator_advance (entry=0x7fffffff2ca8, iterator=0x5555559cdd70) at libgit2/src/diff_generate.c:925
 #22 0x0000555555627aaf in handle_matched_item (diff=0x5555559cb200, info=0x7fffffff2c90) at libgit2/src/diff_generate.c:1179
 #23 0x0000555555627cf7 in git_diff__from_iterators (out=0x7fffffff2d20, repo=0x5555559b4fe0, old_iter=0x5555559cdd70, new_iter=0x5555559ca200, opts=0x7fffffff3730) at libgit2/src/diff_generate.c:1249
 #24 0x00005555556280a0 in git_diff_tree_to_tree (out=0x7fffffff2e08, repo=0x5555559b4fe0, old_tree=0x5555559d8f50, new_tree=0x5555559cc120, opts=0x7fffffff3730) at libgit2/src/diff_generate.c:1319
 #25 0x00005555555eb58e in git2::repo::Repository::diff_tree_to_tree (self=0x7fffffff34c0, old_tree=..., new_tree=..., opts=...) at src/repo.rs:2374
 #26 0x00005555555b5791 in git_absorb::run (config=0x7fffffffce00) at src/lib.rs:42
 #27 0x000055555559c416 in git_absorb::main () at src/main.rs:102

Here we see that the backtrace points to libgit2.

Looking at the Cargo.lock file, I saw that the libgit2-sys crate was rather old compared to the latest on crates.io.

So I first tried updating the crate to see if someone had already fixed the issue upstream:

# in the tummychow/git-absorb repo
$ cargo clean && cargo update && cargo build --locked

This updated the transitive libgit2-sys dependency in the Cargo.lock file from 0.12.13+1.0.1 to 0.12.26+1.3.0.

name = "libgit2-sys"
-version = "0.12.13+1.0.1"
+version = "0.12.26+1.3.0"

Then running the newly built binary with the updated libgit2 succeeded:

# in the repo with changes
$ git absorb
(success)

Since this worked, it indicated that the issue I was seeing was fixed upstream already.

I then tried explicitly adding libgit2-sys as a dependency and setting an exact version to override the Cargo dependency resolver (without updating git2). The first libgit2-sys after the current version in the Cargo.lock file that fixes this seems to be =0.12.14+1.1.0.

That release bumps the libgit2 dependency, so most likely this was fixed by some change in libgit2 between 1.0.0 and 1.1.0.

I then set up the libgit2 repository locally to replace the libgit2-sys, and bisecting between those two tags indeed led to da3288ded5bce2a37566126be51fe75ff0316c2d.