Commit Graph

65 Commits

Author SHA1 Message Date
Silvan Mosberger
e130ee33a1 pkgs/test/nixpkgs-check-by-name/scripts: Various improvements
- trace function, avoids littering `echo >&2` all throughout
- Avoid `eval`, remove unneeded shellcheck

Co-Authored-By: Victor Engmark <victor@engmark.name>
2023-12-22 00:04:16 +01:00
Silvan Mosberger
1968beeabd check-by-name: Pass --base in CI and local running script
This enables the ratchet checks for pkgs/by-name, allowing gradual
migrations!
2023-12-20 17:38:06 +01:00
Silvan Mosberger
9e03178865 tests.nixpkgs-check-by-name: Add documentation for scripts 2023-12-20 17:38:05 +01:00
Silvan Mosberger
92238ac52d tests.nixpkgs-check-by-name: Create script to run locally
Due to the check soon depending on the base branch (see `--base`),
the CI check can't reasonably share all code with a local check.
We can still make a script to run it locally, just not sharing all code.
2023-12-20 17:38:03 +01:00
Silvan Mosberger
3869ba4127 workflows/check-by-name: Move tool fetching into script
This part of the CI can also be reproduced locally
2023-12-19 23:15:01 +01:00
Silvan Mosberger
fc2d26939d tests.nixpkgs-check-by-name: Improve check clarity 2023-12-15 17:27:26 +01:00
Silvan Mosberger
74e8b38dbe tests.nixpkgs-check-by-name: Move interface description into code
This would be duplicated otherwise
2023-12-15 02:28:55 +01:00
Silvan Mosberger
79618ff8cb tests.nixpkgs-check-by-name: Improve docs, introduce "ratchet" term 2023-12-15 02:14:48 +01:00
Silvan Mosberger
413dd9c03e tests.nixpkgs-check-by-name: Minor improvements from review 2023-12-15 01:02:49 +01:00
Silvan Mosberger
53b43ce0e3 tests.nixpkgs-check-by-name: Fix and document behavior without --base
Previously, not passing `--base` would enforce the most strict checks.
While there's currently no actual violation of these stricter checks,
this does not match the previous behavior.

This won't matter once CI passes `--base`, the code handling the
optionality can be removed then.
2023-12-15 01:01:14 +01:00
Silvan Mosberger
bb08bfc2d3 tests.nixpkgs-check-by-name: Test for gradual transition
This implements the ability to test gradual transitions in check
strictness, and adds one such test for the empty non-auto-called
arguments check.
2023-12-14 04:05:03 +01:00
Silvan Mosberger
d487a975cc tests.nixpkgs-check-by-name: Gradual migration from base Nixpkgs
This implements the option for a gradual migration to stricter checks.
For now this is only done for the check against empty non-auto-called
callPackage arguments, but in the future this can be used to ensure all
new packages make use of `pkgs/by-name`.

This is implemented by adding a `--base <BASE_NIXPKGS>` flag, which then
compares the base nixpkgs against the main nixpkgs version, making sure
that there are no regressions.

The `--version` flag is removed. While it was implemented, it was never
used in CI, so this is fine.
2023-12-14 03:51:10 +01:00
Silvan Mosberger
a6ba4cae31 tests.nixpkgs-check-by-name: Intermediate refactor
This prepares the code base for the removal of the `--version` flag, to
be replaced with a flag that can specify a base version to compare the
main Nixpkgs against, in order to have gradual transitions to stricter
checks.

This refactoring does:
- Introduce the `version` module that can house the logic to increase
  strictness, with a `version::Nixpkgs` struct that contains the
  strictness conformity of a single Nixpkgs version
- Make the check return `version::Nixpkgs`
- Handle the behavior of the still-existing `--version` flag with `version::Nixpkgs`
- Introduce an intermediate `process` function to handle the top-level
  logic, especially useful in the next commit
2023-12-14 03:40:49 +01:00
Silvan Mosberger
e98d22851b tests.nixpkgs-check-by-name: Introduce result_map
Convenience function to run another validation over a successful validation result.

This will be usable in more locations in future commits, making the code
nicer.
2023-12-14 03:21:41 +01:00
Silvan Mosberger
b8e4d555b4 tests.nixpkgs-check-by-name: Minor refactor, allow more simultaneous problems
This makes it such that these two errors can both be thrown for a single
package:
- The attribute value not being a derivation
- The attribute not being a proper callPackage

The tests had to be adjusted to only throw the error they were testing
for
2023-12-14 03:13:36 +01:00
Silvan Mosberger
6361b22370
Merge pull request #267048 from narang99/hariom/257748/check-by-name-msg
fix long error messages in nix-instantiate
2023-11-21 01:35:11 +01:00
Hariom Narang
ca97497bfd made EXPR_PATH point to local instead of store
- we need this behavior in dev shell
- the main build still uses the store file
2023-11-21 01:11:06 +05:30
Silvan Mosberger
def5df923f tests.nixpkgs-check-by-name: More reliable .envrc reloading
See also https://github.com/nix-community/nix-direnv/issues/408 and https://github.com/nix-community/nix-direnv/pull/407
2023-11-18 04:54:05 +01:00
Hariom Narang
437b2054f4 pass eval.nix as a file instead of expression
- passing it as expression gives large error messages
  which are not very readable
- this commits puts the file in nix-store
  and patches the final program to have access to
  the path to the file as env.
- We simply pass this file to nix-instantiate
2023-11-14 15:05:45 +05:30
Silvan Mosberger
7753969628 tests.nixpkgs-check-by-name: Remove PackageContext helper
Was not really necessary anymore
2023-10-24 20:02:15 +02:00
Silvan Mosberger
82e708c192 tests.nixpkgs-check-by-name: Custom Validation type and improvements
Co-authored-by: Wanja Hentze
2023-10-24 19:58:52 +02:00
Silvan Mosberger
03c58ad1d6 tests.nixpkgs-check-by-name: Minor doc updates 2023-10-24 01:18:46 +02:00
Silvan Mosberger
8be41ace99 tests.nixpkgs-check-by-name: Separate file for all problems
And introduce a function for some smaller indentation
2023-10-24 01:18:46 +02:00
Silvan Mosberger
eac0b69063 tests.nixpkgs-check-by-name: Redesign and document check_result functions 2023-10-24 01:18:44 +02:00
Silvan Mosberger
3d60440799 tests.nixpkgs-check-by-name: Remove error writer 2023-10-24 01:18:13 +02:00
Silvan Mosberger
e58bc75444 tests.nixpkgs-check-by-name: Remove Nixpkgs struct
Isn't necessary anymore with the refactoring
2023-10-24 01:18:13 +02:00
Silvan Mosberger
d65f3ddb89 tests.nixpkgs-check-by-name: Make reference check part of structural check 2023-10-24 01:18:13 +02:00
Silvan Mosberger
0475238ec0 tests.nixpkgs-check-by-name: Make structural check a global function 2023-10-24 01:18:12 +02:00
Silvan Mosberger
83b887504c tests.nixpkgs-check-by-name: Support for combining check results 2023-10-24 01:18:09 +02:00
Silvan Mosberger
bb89ca72df tests.nixpkgs-check-by-name: Refactor 2023-10-24 01:15:58 +02:00
Silvan Mosberger
571eaed155 tests.nixpkgs-check-by-name: Intermediate ShardNonDir error 2023-10-24 01:15:58 +02:00
Silvan Mosberger
b7ace0198c tests.nixpkgs-check-by-name: Intermediate InvalidShardName error 2023-10-24 01:15:58 +02:00
Silvan Mosberger
143e267ad2 tests.nixpkgs-check-by-name: Intermediate PackageNonDir error 2023-10-24 01:15:58 +02:00
Silvan Mosberger
935f82267a tests.nixpkgs-check-by-name: Intermediate CaseSensitiveDuplicate error 2023-10-24 01:15:58 +02:00
Silvan Mosberger
e7d9cc96ed tests.nixpkgs-check-by-name: Intermediate InvalidPackageName error 2023-10-24 01:15:57 +02:00
Silvan Mosberger
b011d53bda tests.nixpkgs-check-by-name: Intermediate IncorrectShard error 2023-10-24 01:15:57 +02:00
Silvan Mosberger
64f5eb616e tests.nixpkgs-check-by-name: Intermediate PackageNixNonExistent error 2023-10-24 01:15:57 +02:00
Silvan Mosberger
e3979d14cd tests.nixpkgs-check-by-name: Intermediate PackageNixDir error 2023-10-24 01:15:57 +02:00
Silvan Mosberger
4f17b9367d tests.nixpkgs-check-by-name: Intermediate UndefinedAttr error 2023-10-24 01:15:57 +02:00
Silvan Mosberger
b688da8189 tests.nixpkgs-check-by-name: Intermediate WrongCallPackage error 2023-10-24 01:15:57 +02:00
Silvan Mosberger
9a0ef88623 tests.nixpkgs-check-by-name: Intermediate NonDerivation error 2023-10-24 01:15:57 +02:00
Silvan Mosberger
4897b63ae6 tests.nixpkgs-check-by-name: Intermediate Symlink errors 2023-10-24 01:15:57 +02:00
Silvan Mosberger
9a3abc4383 tests.nixpkgs-check-by-name: Intermediate CouldNotParseNix error 2023-10-24 01:15:57 +02:00
Silvan Mosberger
96f6a350fa tests.nixpkgs-check-by-name: Intermediate PathInterpolation error 2023-10-24 01:15:56 +02:00
Silvan Mosberger
a755aa7d02 tests.nixpkgs-check-by-name: Intermediate SearchPath error 2023-10-24 01:15:56 +02:00
Silvan Mosberger
ed56d74c08 tests.nixpkgs-check-by-name: Intermediate path reference errors 2023-10-24 01:15:56 +02:00
Silvan Mosberger
37f8f6681c tests.nixpkgs-check-by-name: Intermediate error type refactoring prep
Currently the tool prints problems right as it is checking the code
without an intermediate error representation. However for various reasons
it would be beneficial to have an intermediate error type:
- It makes the code cleaner, having all errors in one place
- It allows printing the error in different ways, e.g. for a future
  --json mode

This commit prepares for an incremental refactoring for an intermediate
error/problem representation. Most notable is that we want to be able to collect
multiple errors/problems and not just exit on the first one.

We introduce the type alias CheckResult and CheckError (later renamed to
NixpkgsProblem), where CheckError allows collecting multiple
CheckErrors using the utility function flatten_check_results (later
renamed to check_result::sequence)

The write_check_result function is only temporarily introduced to help
refactoring, it's removed again in later commits.
2023-10-24 01:11:57 +02:00
Silvan Mosberger
d3bf6133e8 tests.nixpkgs-check-by-name: Disallow empty all-packages.nix overrides
Only enabled with `--version v1`
2023-10-12 02:28:34 +02:00
Silvan Mosberger
23541bed72 tests.nixpkgs-check-by-name: Introduce --version 2023-10-12 02:28:09 +02:00
Silvan Mosberger
fcaa408d00 tests.nixpkgs-check-by-name: auto-calling differentiation
Allows detecting whether attributes are overridden in all-packages.nix.
In a future commit we'll use this to detect empty arguments being set in
all-packages.nix and complain about that.
2023-10-12 02:27:55 +02:00