Latest rustc nightlies seem to have broken incremental compilation [0].
I apparently have the worst luck in this department, as I've gotten multiple
ICEs within the last few hours alone. So, I'm putting future me out of
her misery.
[0]: https://github.com/rust-lang/rust/issues/139110
This uses `zlib-rs`, a native Rust library that is comparable in
performance to `zlib-ng`. Since there’s no complicated C build
and gitoxide only has one hashing backend now, this lets us drop our
`packaging` feature without adding any awkward build requirements.
`zlib-rs` is generally faster at decompression than
`zlib-ng`, and faster at compression on levels 6 and 9; see
<https://trifectatech.org/blog/zlib-rs-is-faster-than-c/>
for details.
I couldn’t get reliable‐looking benchmark results out of my
temperamental laptop; `hyperfine` seemed to think that some random
`jj` workloads I tested might be slightly slower than with `zlib-ng`,
but it wasn’t unambiguously distinguishable from noise, so I’d
like to see measurements from others.
It’s certainly a lot faster than the previous default, and I
think it’s likely that `zlib-rs` will continue to get faster
and that it’s more than worth avoiding the headaches of a native
library with a CMake build dependency. (Though on the other hand,
if distributions move in the direction of shipping `zlib-ng` by
default, maybe there will be more motivation to make `libz-ng-sys`
support system libraries.)
Overall, building in the test profile should significantly speed up
the overall build pipeline because so many less cycles are spent (on
GHA runners that are certainly at high load). The goal here is to help
reduce CI flake outs due to things timing out; I suspect part of the
problem may be a lot of the ~15 minute time limit being used up just
compiling things.
This is a partial revert of b714592952400ab, which removed this previous
override of the Flake `checks`.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Some of our builds have been timing out, and one angle I want to look at
is whether any tests are hanging. Nextest can help us keep track of slow
tests in CI where the underlying hosts will have significantly higher
load. In general this should also help keep our tests healthy, IMO.
I don't see any reason why any existing test should take over 20
seconds, but it should at least help control for really slow runners
that have huge latency spikes. we can start there and see how it goes
in practice.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Currently, the Git subprocess tests only work on Linux due to a default
path being used for the `git` executable when `$PATH` is unset. This can
break if Git isn't installed at the expected path. Also, I believe it is
currently necessary to set the `$TEST_GIT_EXECUTABLE_PATH` environment
variable on Windows for tests to pass. Instead, we should use the user's
`$PATH` to locate the `git` executable, as well as any other executables
that are needed. This also makes `$TEST_GIT_EXECUTABLE_PATH` no longer
necessary, so it can be removed.
bacon 3.6.0 -> 3.8.0, shows output with nextest 0.9.86
https://github.com/Canop/bacon/issues/280
Add python3 to the devshell to be able to run tools installed with uv
The `RUSTFLAGS` enviroment variable needs to be set in a `shellHook` to
allow the command to be interpreted by the shell. Otherwise, it will
just be passed as a string into the `rustc` binary:
```
cargo nextest run --workspace
error: failed to run `rustc` to learn about target-specific information
Caused by:
process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Zthreads=0 -C 'link-arg=--ld-path=$(unset' 'DEVELOPER_DIR;' /usr/bin/xcrun --find 'ld)' -C link-arg=-ld_new --target aarch64-apple-darwin --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 1)
--- stderr
error: Unrecognized option: 'find'
```
`zstd` is only used to write files in the native backend right now. For now,
jettison it, to unbundle some C code that we don't really need.
(Ideally, a future compression library would be pure Rust, but we'll cross that
bridge when we get to it...)
Signed-off-by: Austin Seipp <aseipp@pobox.com>
I noticed miniz_oxide appears in perf samples. While miniz_oxide is safer, I
think zlib-ng is pretty reliable.
https://docs.rs/gix/latest/gix/#performance
libz-ng-sys is downgraded to 1.1.16 due to the Windows linking issue. The
benchmark result is obtained with libz-ng-sys 1.1.20.
https://github.com/rust-lang/libz-sys/issues/225
```
% hyperfine --sort command --warmup 3 --runs 10 -L bin jj-0,jj-1 \
'target/release-with-debug/{bin} --ignore-working-copy log README.md'
Benchmark 1: target/release-with-debug/jj-0 ..
Time (mean ± σ): 256.6 ms ± 4.3 ms [User: 214.1 ms, System: 38.6 ms]
Range (min … max): 245.4 ms … 261.2 ms 10 runs
Benchmark 2: target/release-with-debug/jj-1 ..
Time (mean ± σ): 223.0 ms ± 4.2 ms [User: 174.7 ms, System: 44.4 ms]
Range (min … max): 212.4 ms … 225.8 ms 10 runs
```
As reported by @lilyball on Discord, the recent nixpkgs update this week brought
with it an entirely new Darwin stdenv, which had a behavioral change versus
the prior one, causing builds inside the developer shell to fail, as the new
super-duper parallel linker could not be used.
However, rather than giving up and throwing away *billions* of precious CPU
cycles, @emilazy and @lilyball instead doubled down and diagnosed the issue. The
result is a new impenetrable line of Nix code, which is fully explained by the
comments within.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Our docs are built with MkDocs, which requires Python and several deps.
Previously those deps were managed with Poetry, which is also written in Python.
This commit replaces Poetry with `uv`, a Rust-based Python
project/package manager, and thus removes several steps from the docs
build process.
Before:
<install Python>
<install pipx>
pipx install poetry
poetry install
poetry run -- mkdocs serve
After:
<install uv>
uv run mkdocs serve
The most relevant part (and the reason for this change) is the addition
of the `mainProgram` attribute. This allows getting the executable name
from inside nix expressions with ease:
```
# before
lib.getExe' jujutsu "jj"
# or
"${jujutsu}/bin/jj"
```
```
# now
lib.getExe jujutsu
```
In practice, `packages` sets the value for `nativeBuildInputs`. This has the following advantages:
- `buildInputs` is for runtime dependencies. `nativeBuildInputs` is for
build dependencies (this usually not an issue unless cross compiling)
- `shellHook`s will actually run, allowing packages to safely append to
variables like `XDG_DATA_DIRS`
- It's more idiomatic and how it's recommended to be used in the docs
See the included comment. This helps avoid a large dependency on the rustc
toolchain and helps reduce closure size for Nix users.
Partial revert of 749481e5527c092da9096504e08c6aa065f598bb.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
The new parallel macOS linker reduces link time for the debug `jj` binary from
3s to 0.7s on my M2 Macbook Air, which is a significant reduction for nearly
no cost at all. This only assumes that you have a new enough Xcode environment
as your default (where `/usr/bin/ld` resides.)
This change requires Sonoma and Xcode 15, but in theory I think we could target
a lower macOS SDK version in order to produce binaries that are more backwards
compatible, so the only real cost is that developers who also use Nix would
require Sonoma.
This is already included with the `ourRustVersion` expression since it includes
a complete Rust toolchain with all extensions.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
This cuts the size of a final debug binary in half (~250MiB -> 127MiB)
in my dev shell, and saves about 15% of total `target/` directory size
when a full build from scratch happens (2GiB -> 1.7GiB). Let's take an
easy free win.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Only active within `devShell`; the default `.#jujutsu` package is unaffected
by this change. This somewhat increases CPU utilization and has a marginal
improvement on my local compile times, but it's also nice if you have to
recompile from scratch too.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: I07ab28991843ca3723185569db5f38f2ed076875
This is a change that does affect the nix built package, and is tested as part
of the CI Matrix, so there is now an actively maintained (by me) nightly build
that can be checked for regressions.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: If9a134de7e1d496d47936259c59f73c4a62a341f
When the `jj-proc-macros` crate was introduced, it triggered an underlying
bug in `nextest`, which is the test harness we use in the Nix build. This is
upstream Nextest bug 267. The long and short of it is that `rustc` fails to
find needed libraries whenever the proc macros are loaded.
This can easily be worked around however, by setting
`DYLD_FALLBACK_LIBRARY_PATH` to an appropriate value in the devShell and in the
`preCheck` phase of the main expression.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
When running the `nix build`, the `buildRustPackage` function -- which builds
the `jj` crates -- calls `cargo build --release` with flags like `HOST_CXX`
set. This is called the `buildPhase`. Then, it runs the `checkPhase`, which
calls `cargo nextest`, in our case. However, it does not set `HOST_CXX`, for
some reason.
The intent of `buildRustPackage` is that the `buildPhase` and `checkPhase`
have their compilation options fully aligned so that they reuse the local cargo
`target/` cache directory, avoiding a full recompilation. However, the lack
of `HOST_CXX` above among others causes a cache miss, and a bunch of cascading
recompilations. The net impact is that we compile all of the codebase once in
`buildPhase`, then again in `checkPhase`, making the Nix CI build 2x slower on
average than the other Linux runners; 2-3 minutes versus 7 minutes, on average.
Instead, re-introduce a 'check' into the Flake directly, which overrides the
`jujustsu` package, but stubs out the `buildPhase`. This means it will only run
`checkPhase`, which is what we want. Then, modify the CI to run `nix flake check`
again, like it used to in the past.
Unfortunately, this doesn't fix the cache miss when running `nix build`
yourself, it recompiles from scratch in both phases still. That needs to be
fixed in the future, but it's tolerable for now.
This reverts most of 71a3045032f512.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
This matches the behavior of the actual `nix build` more closely, and might also
help Anton, since he was debugging some recompilation issues on his machine,
where `RUSTFLAGS` might have become inconsistent due to VS Code.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Summary: Pure QOL improvement, not that this was too awful as it stood. But this
ultimately makes all builds faster, effectively for free; on my Linux machine,
debug link time for jj-cli goes from 3s to 1s.
This was originally submitted as part of #1864, and backed out as part of #1982,
because the default `mold` package did not have the proper "wrapper script"
that allowed it to find the right external libraries. Using the `mold-wrapped`
package however fixes this.
On top of that, let's go ahead and make `mold` the default linker in the Nix
package/flake as well; this should improve the `nix build` speed since a lot of
tests need to be linked.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Recognize signature metadata from git commit objects, implement
a basic version of that for the native backend.
Extract the signed data (a commit binary repr without the signature) to
be verified later.
Summary: Without `devShell` providing the needed Darwin-specific inputs, `cargo
build` does not work inside a `nix develop` or `direnv` environment; libgit2 in
particular fails on being able to find the Security framework.
The actual `nix build` invocation however *does* work because we correctly
include those dependencies in the package `buildInputs`. So just factor them
out, and use them in both places.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: I484bf381ca31c29c4c39fb6d184bdd21
Summary: Apparently this was broken. Maybe I broke it. Maybe something upstream
changed and caused a regression. But without it, we get the stable `rustfmt` in
the `nix develop` shell environment, not the nightly version. Fix it.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: I602ed8e5691c4d48f8db575d62624955
Summary: Since 066032b6e69 was merged, the `nix flake check` build no longer
overrides the 'cargo test' profile explicitly, to save disk space. The CI seems
to be in a better spot. This will stem the tide for a while hopefully.
However, with that change in place, the `nix flake check` build was
essentially a redundant, nearly-identical copy of a normal `nix build` with no
differentiating features, except: `RUST_BACKTRACE` is set to 1.
Delete all this code, and remove it from the CI matrix, and instead just export
`RUST_BACKTRACE` on the `checkPhase` of the normal `nix build` instead, which is
functionally equivalent.
Also does some minor, no-functional-change touchups to `flake.nix` while I was
there (whitespace, etc.)
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: I87336b16e2a0b973343ecbde8ffd7b8f
Summary: The Nix CI has been failing recently due to (what I assume is) disk
space issues. But only the `flake check` step is failing. Right now, `nix flake
check` runs the Cargo tests with the debug profile to help get more debug info,
which is even heftier in terms of debug info than the normal 'test' profile. For
reference, a single build of 'cargo test' in a clean working copy results in a
15 gigabyte `target/` directory.
Turn off the debug profile for `nix flake check`, which should hopefully stem
the bleeding a bit. I believe the 'test' profile should still have enough
symbols for backtraces, so panics should still be useful.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: Idde10ac15847a1ad1e6f4e48a2497eca
Summary: By default, and to be explicit, the local flake turns off default
features. But, 70f6e0a turned off watchman support for the Nix build by
accident. Enable default features so this doesn't happen again.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
Change-Id: Ib7dd935eceab328fa5b0333b10368377
