Jade Lovelace
473d2d56fc
Remove 100s of CPU time (10%) from build times (1465s -> 1302s)
...
Result's from Mic92's framework 13th Gen Intel Core i7-1360P:
Before: 3595.92s user 183.01s system 1360% cpu 4:37.74 total
After: 3486.07s user 168.93s system 1354% cpu 4:29.79 total
I saw that boost/lexical_cast was costing about 100s in CPU time on our
compiles. We can fix this trivially by doing explicit template
instantiation in exactly one place and eliminating all other includes of
it, which is a code improvement anyway by hiding the boost.
Before:
```
lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin
Analyzing build trace from 'buildtimeold.bin'...
**** Time summary:
Compilation (551 times):
Parsing (frontend): 1465.3 s
Codegen & opts (backend): 1110.9 s
<snip>
**** Expensive headers:
178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via:
40x: command.hh
5x: command-installable-value.hh
3x: installable-flake.hh
2x: <direct include>
2x: installable-attr-path.hh
176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via:
36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
12x: globals.hh config.hh experimental-features.hh
11x: file-system.hh file-descriptor.hh
6x: serialise.hh strings.hh
6x: <direct include>
6x: archive.hh serialise.hh strings.hh
...
173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via:
55x: <direct include>
39x: command.hh installable-value.hh installables.hh
7x: libexpr.hh
4x: local-store.hh
4x: command-installable-value.hh installable-value.hh installables.hh
3x: binary-cache-store.hh
...
170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via:
37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh
14x: store-api.hh nar-info.hh hash.hh
11x: <direct include>
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh
7x: libexpr.hh value.hh source-path.hh archive.hh
6x: fetchers.hh hash.hh
...
169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via:
40x: command.hh installable-value.hh
5x: command-installable-value.hh installable-value.hh
3x: installable-flake.hh installable-value.hh
2x: <direct include>
1x: installable-derived-path.hh
1x: installable-value.hh
...
159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via:
37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh
19x: <direct include>
14x: store-api.hh nar-info.hh hash.hh serialise.hh
11x: serialise.hh
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh
7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh
...
156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via:
42x: <direct include>
7x: command-installable-value.hh
2x: installable-attr-path.hh
150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh
11x: file-system.hh
10x: globals.hh
6x: fetchers.hh
6x: serialise.hh strings.hh error.hh
5x: archive.hh
...
133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via
:
37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh
19x: file-system.hh
11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh
...
132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
16x: gtest.h
11x: file-system.hh types.hh ref.hh
10x: globals.hh types.hh ref.hh
10x: json.hpp
6x: serialise.hh
...
done in 0.6s.
```
After:
```
lix/lix2 » maintainers/buildtime_report.sh build
Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it.
Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'...
**** Time summary:
Compilation (551 times):
Parsing (frontend): 1302.1 s
Codegen & opts (backend): 956.3 s
<snip>
**** Expensive headers:
178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via:
36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh
12x: globals.hh config.hh experimental-features.hh
11x: file-system.hh file-descriptor.hh
6x: <direct include>
6x: serialise.hh strings.hh
6x: fetchers.hh hash.hh serialise.hh strings.hh
...
154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via:
40x: command.hh
5x: command-installable-value.hh
3x: installable-flake.hh
2x: <direct include>
2x: installable-attr-path.hh
153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via:
55x: <direct include>
39x: command.hh installable-value.hh installables.hh
7x: libexpr.hh
4x: local-store.hh
4x: command-installable-value.hh installable-value.hh installables.hh
3x: binary-cache-store.hh
...
149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh
11x: file-system.hh
10x: globals.hh
6x: fetchers.hh
6x: serialise.hh strings.hh error.hh
5x: archive.hh
...
144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via:
40x: command.hh installable-value.hh
5x: command-installable-value.hh installable-value.hh
3x: installable-flake.hh installable-value.hh
2x: <direct include>
1x: installable-value.hh
1x: installable-derived-path.hh
...
136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via:
42x: <direct include>
7x: command-installable-value.hh
2x: installable-attr-path.hh
133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via:
36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh
16x: gtest.h
11x: file-system.hh types.hh ref.hh
10x: globals.hh types.hh ref.hh
10x: json.hpp
6x: serialise.hh
...
89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via:
37x: command.hh installable-value.hh installables.hh
25x: store-api.hh realisation.hh
7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh
6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh
6x: libexpr.hh value.hh context.hh
6x: shared.hh
...
87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via:
35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h
12x: regex sstream istream
10x: file-system.hh types.hh ref.hh memory unique_ptr.h
10x: gtest.h memory unique_ptr.h
10x: globals.hh types.hh ref.hh memory unique_ptr.h
6x: fetchers.hh types.hh ref.hh memory unique_ptr.h
...
85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via:
37x: command.hh installable-value.hh installables.hh derived-path.hh
20x: globals.hh
20x: logging.hh
16x: store-api.hh logging.hh
6x: <direct include>
6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh
...
done in 0.5s.
```
Adapated from 18aa3e1d57
2024-05-31 13:00:09 +02:00
Robert Hensing
ef5c846e25
Merge pull request #10768 from obsidiansystems/legacy-ssh-expose-ssh-master-for-hydra
...
Create `CommonSSHStoreConfig::createSSHMaster`
2024-05-29 22:53:29 +02:00
Robert Hensing
1054ff0873
Merge pull request #10789 from nix-windows/windows-substitution-goal
...
More work on the scheduler for windows
2024-05-29 22:45:55 +02:00
Robert Hensing
154769544d
Merge pull request #10805 from hercules-ci/issue-10774
...
libcmd: Fix #10774
2024-05-29 22:39:00 +02:00
Robert Hensing
5df42223e2
Merge pull request #10797 from obsidiansystems/shellcheck-tests
...
Shellcheck some test scripts
2024-05-29 22:27:00 +02:00
Robert Hensing
1c70eb8eee
libcmd: Fix #10774
2024-05-29 21:42:53 +02:00
Qyriad
18ac6545fc
print type and value in "flake attr is not a derivation" errors
...
This turns errors like:
error: flake output attribute 'hydraJobs' is not a derivation or path
into errors like:
error: expected flake output attribute 'hydraJobs' to be a derivation or
path but found a set: { binaryTarball = «thunk»; build = «thunk»; etc> }
This change affects all InstallableFlake commands.
Source: 20981461d4
2024-05-29 20:51:32 +02:00
Jörg Thalheim
e2182d07d9
fixup extension of changelog entry
2024-05-29 20:51:32 +02:00
Jörg Thalheim
5786e1ae7c
docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation ( #10803 )
...
* docs: mention importNative/exec in allow-unsafe-native-code-during-evaluation
Both of these still needs their own actual documentation, but they are
at least now mentioned that they exist and what they're enabled by.
Co-authored-by: Qyriad <qyriad@qyriad.me>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2024-05-29 07:50:51 +00:00
Philipp
2bd66922ee
add empty line to documentation comments after @brief field ( #10800 )
...
* add empty line to documentation comments after `@brief` field
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2024-05-28 23:05:40 +00:00
John Ericson
10f864c5ae
Ensure all functional scripts are (a) executable (b) have shebang
...
This is good for shebang, and also good for future build system
simplifications
2024-05-28 12:46:24 -04:00
John Ericson
2e12b58126
Shellcheck some test scripts
...
Progress on #10795
2024-05-28 12:32:22 -04:00
John Ericson
bcdee80a0d
More work on the scheduler for windows
...
- Get a rump derivation goal: hook instance will come later, local
derivation goal will come after that.
- Start cleaning up the channel / waiting code with an abstraction.
2024-05-28 11:39:49 -04:00
John Ericson
1e2b26734b
Merge pull request #10799 from hercules-ci/safer-tab-completion
...
Add repl completion test
2024-05-28 11:30:56 -04:00
John Ericson
aa5f013d64
Merge pull request #10794 from obsidiansystems/per-hook-excludes
...
dev shell: excludes are per hook
2024-05-28 09:56:04 -04:00
Eelco Dolstra
7b471547e6
Merge pull request #10793 from NixOS/dependabot/github_actions/zeebe-io/backport-action-3.0.2
...
Bump zeebe-io/backport-action from 2.5.0 to 3.0.2
2024-05-28 15:54:15 +02:00
John Ericson
ebc29017fc
dev shell: excludes are per hook
...
As suggested by Robert in
https://github.com/NixOS/nix/pull/10787#discussion_r1617145374
2024-05-28 09:28:36 -04:00
dependabot[bot]
d7b04d61a9
Bump zeebe-io/backport-action from 2.5.0 to 3.0.2
...
Bumps [zeebe-io/backport-action](https://github.com/zeebe-io/backport-action ) from 2.5.0 to 3.0.2.
- [Release notes](https://github.com/zeebe-io/backport-action/releases )
- [Commits](https://github.com/zeebe-io/backport-action/compare/v2.5.0...v3.0.2 )
---
updated-dependencies:
- dependency-name: zeebe-io/backport-action
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 13:10:48 +00:00
John Ericson
cc98fce039
Merge pull request #10787 from NixOS/shellcheck-tests
...
Start getting all shell scripts passing shellcheck
2024-05-28 09:09:37 -04:00
John Ericson
ef96a58ed7
Merge pull request #10791 from obsidiansystems/fix-format
...
Fix format
2024-05-27 22:58:04 -04:00
John Ericson
567265ae67
Start getting all shell scripts passing shellcheck
...
Like with the formatter, we are blacklisting most files by default.
Do a few files to get us started, and get a sense of what this looks
like.
2024-05-27 22:39:56 -04:00
John Ericson
1d5d748fe4
Fix format
...
39b2a399ad1d6c2316a9
2024-05-27 22:32:52 -04:00
John Ericson
d0c7da131f
Merge pull request #10678 from nix-windows/windows-substitution-goal
...
Start building the scheduler for Windows
2024-05-27 17:47:29 -04:00
John Ericson
3e9c3738d3
Create CommonSSHStoreConfig::createSSHMaster
...
By moving `host` to the config, we can do a lot further cleanups and
dedups. This anticipates a world where we always go `StoreReference` ->
`*StoreConfig` -> `Store*` rather than skipping the middle step too.
Progress on #10766
Progress on https://github.com/NixOS/hydra/issues/1164
2024-05-27 16:12:53 -04:00
John Ericson
17964441d9
Merge pull request #10781 from obsidiansystems/build-mode-parse
...
Worker proto use proper serialiser for `BuildMode`
2024-05-27 16:07:54 -04:00
John Ericson
263905da4b
Merge pull request #10785 from mjoerg/fix-typos
...
fix typos
2024-05-27 10:24:53 -04:00
John Ericson
8527f4e7fa
Merge pull request #10754 from qwqawawow/master
...
nix repl: make runNix() isInteractive is true by default
2024-05-27 10:07:15 -04:00
Martin Joerg
e7ea5591a2
fix typos
2024-05-27 15:56:52 +02:00
John Ericson
aa4a2927a7
Merge pull request #10778 from Mic92/safer-tab-completion
...
repl: do not crash when tab-completing import errors
2024-05-27 09:54:29 -04:00
Robert Hensing
ebfada36a1
Add repl completion test
2024-05-27 09:58:49 +02:00
John Ericson
8ebd99c74e
Back in enum values for BuildMode serializer
...
We don't want to rely on how C assigns numbers for enums in the wire
format. Sure, this is totally determined by the ABI, but it obscures the
code and makes it harder to safely change the enum definition (should we
need to) without accidentally breaking the wire format.
2024-05-27 00:22:55 -04:00
John Ericson
eeb89c28b0
Worker proto use proper serialiser for BuildMode
...
Do this instead of an unchecked cast
I redid this to use the serialisation framework (including a unit test),
but I am keeping the reference to credit Jade for spotting the issue.
Change-Id: Icf6af7935e8f139bef36b40ad475e973aa48855c
(adapted from commit 2a7a824d83dc5fb33326b8b89625685f283a743b)
Co-Authored-By: Jade Lovelace <lix@jade.fyi>
2024-05-27 00:22:55 -04:00
John Ericson
7de033d63f
Merge pull request #10777 from Mic92/copy-path
...
libstore: remove unused copyPath function
2024-05-26 10:58:38 -04:00
Pierre Bourdon
ffe6ba69d6
repl: do not crash when tab-completing import errors
...
File not found while importing is not currently caught by the tab-completion handler.
Original bug report: https://git.lix.systems/lix-project/lix/issues/340
Fix has been adapted from https://gerrit.lix.systems/c/lix/+/1189
Example crash:
$ cat /tmp/foo.nix
{
someImport = import ./this_file_does_not_exist;
}
$ ./src/nix/nix repl --file /tmp/foo.nix
warning: unknown experimental feature 'repl-flake'
Nix 2.23.0pre20240517_dirty
Type :? for help.
Loading installable ''...
Added 1 variables.
nix-repl> someImport.<TAB>
2024-05-25 23:15:28 +02:00
Jörg Thalheim
f97da4b11c
libutil/source-accessor: custom error if source does not exist
...
This allows better error handling by catching this error in particular.
2024-05-25 23:06:57 +02:00
eldritch horrors
2de4589e46
libstore: remove unused copyPath function
...
Source: 47523944c5
2024-05-25 22:28:41 +02:00
John Ericson
5cfa75ea16
Merge pull request #10737 from poweredbypie/mingw-stackSize
...
Implement `setStackSize` on Windows
2024-05-25 09:56:02 -04:00
John Ericson
e0c94b91ee
Merge pull request #10757 from obsidiansystems/fix-4977
...
Require `drvPath` attribute to end with `.drv`
2024-05-24 12:14:59 -04:00
Robert Hensing
c90a763273
Merge pull request #10767 from hercules-ci/fix-c-api-primop-for-strict-initializers
...
C API: Fix custom primops
2024-05-24 08:32:02 +02:00
PoweredByPie
0b7da099d1
Commit more stack size in some windows binaries
...
This way we can commit the same amount of stack size (64 MB) without a conditional.
Includes nix, libnixexpr-tests, libnixfetchers-tests, libnixstore-tests, libnixutil-tests.
2024-05-23 17:42:55 -07:00
Robert Hensing
97c3463291
C API: Refactor: use NIX_VALUE_CALL
2024-05-23 21:25:23 +02:00
Robert Hensing
2497d10351
C API: Add nix_value_call_multi, NIX_VALUE_CALL
...
_multi can be implemented more efficiently.
NIX_VALUE_CALL is a convenient way to invoke it.
2024-05-23 21:24:06 +02:00
Robert Hensing
ab106c5ca3
C API: Test arity 2 primop
2024-05-23 21:23:15 +02:00
Robert Hensing
4bc4fb40ea
C API: builtin -> custom function
...
Not all primops will be in `builtins`.
2024-05-23 21:23:14 +02:00
Robert Hensing
8ef6efc184
C API: Require non-thunk value from primop definition
2024-05-23 21:22:59 +02:00
Robert Hensing
8884227045
C API: Require initialized value from primop definition
2024-05-23 21:22:21 +02:00
Robert Hensing
a942a34469
C API: Fix nix_c_primop_wrapper for strict initializers
...
https://github.com/NixOS/nix/pull/10555 added a check requiring
that output parameters always have an uninitialized Value as argument.
Unfortunately the output parameter of the primop callback received
a thunk instead.
See the comment for implementation considerations.
2024-05-23 18:32:49 +02:00
John Ericson
0f9099b517
Merge pull request #10764 from obsidiansystems/machine-ctor-field-comments
...
Document field being initialized in `Machine` constructor
2024-05-23 11:53:21 -04:00
John Ericson
5384ceacc3
Document field being initialized in Machine constructor
2024-05-23 11:28:25 -04:00
John Ericson
2c42e7b8d9
Merge pull request #10763 from obsidiansystems/expose-parsing-machines
...
Restore exposing machine file parsing
2024-05-23 09:58:55 -04:00
