Commit Graph

181 Commits

Author SHA1 Message Date
Sandro Jäckel
025a028c47 fetchgit: fix private fetching via netrc
This fixes a regression introduced in https://github.com/NixOS/nixpkgs/pull/256628
which broke fetching with private = true through a netrc file.
Tested locally with a really special github enterprise.
2023-10-05 13:54:00 +00:00
Alyssa Ross
7d40fbbc04
nix-prefetch-git: ignore global and user git config
nix-prefetch-git is either run as part of a build, usually sandboxed,
or outside a build, unsandboxed, to prefetch something that will later
be used in a build.  It's important that the latter use produces
hashes that can be reproduced by the former.

One way that they can differ is if the user's git config does
something that changes the result of git clone.  I ran into this,
because my global git config automatically enables git-lfs, whereas
nix-prefetch-git otherwise only uses git-lfs if specifically
requested.  This led to very confusing hash mismatches.
2023-09-22 06:59:40 +00:00
Silvan Mosberger
7cbc8215fd
Merge pull request #252865 from emily-is-my-username/fix/fetchgit-deepclone
`fetchgit`: don't shallow clone if `deepClone` is requested
2023-09-06 14:08:06 +02:00
github-actions[bot]
17a15a50ab
Merge master into staging-next 2023-07-21 06:01:05 +00:00
Theodore Ni
63804228e5
fetchgit: clean up md5 references 2023-07-20 21:24:13 -07:00
PedroHLC ☭
0142902fc4
nix-prefetch-git: add sri-hash to output 2023-07-09 14:46:58 -03:00
Emily
ad94541662 fetchgit: don't shallow clone if deepClone is requested
fixes: https://github.com/NixOS/nixpkgs/issues/179788
2023-07-01 00:10:26 +02:00
Mario Rodas
922d55bf2d fetchgit: add a default NIX_BUILD_CORES variable
NIX_BUILD_CORES might not be defined at runtime, so the command would
fail.
2023-06-24 04:20:00 +00:00
Domen Kožar
4f72b6af8b fetchgit: fetch submodules in parallel 2023-06-20 10:36:10 +01:00
Rick van Schijndel
6cdf7259d0
Merge pull request #235898 from panicgh/fetchgit-sparse-checkout-failhard
fetchgit: require sparseCheckout be a list of strings
2023-06-05 22:53:19 +02:00
Nicolas Benes
52f3a1c42c fetchgit: require sparseCheckout be a list of strings
Passing a (multi-line) string was deprecated in #200082 in favour of
list of strings, but still supported (with warning). Now, enforce use of
list of strings.
2023-06-04 10:04:31 +02:00
fetsorn
29f3903f62 nix-prefetch-git: fix typo 2023-05-31 04:00:32 +04:00
Ryan Hendrickson
0f6712d45a nix-prefetch-git: redirect git lfs to stderr 2023-05-21 09:05:49 +02:00
piegames
794b05a09d
Merge pull request #158968: Make fetchFromGitHub & friends overridable 2023-05-07 21:59:36 +02:00
piegames
a205795c8a treewide: Make some fetchers overridable 2023-04-22 22:29:23 +02:00
Artturi
ddc418f3b1
Merge pull request #187210 from pinn3/fetchgit/remove-outdated-comment 2023-04-19 03:49:19 +03:00
Felix Buehler
cdb39a86e0 treewide: use optionalString 2023-02-13 21:52:34 +01:00
Artturin
fe1c7a1945 treewide: remove usages of header and stopNest
they're obsolete
2023-01-16 00:08:12 +02:00
Artturi
1121c8fcb7
Merge pull request #185005 from amjoseph-nixpkgs/pr/fetchgit/fix-comment 2023-01-14 19:50:20 +02:00
Artturin
c01f509e44 treewide: source .attrs in builders
if theres a source $stdenv then this is needed

for structuredAttrs
2022-12-08 21:09:02 +02:00
Nicolas Benes
f6b07f0e2f fetchgit: make sparseCheckout a list of strings
The `sparseCheckout` argument allows the user to specify directories or
patterns of files, which Git uses to filter files it should check-out.

Git expects a multi-line string on stdin ("newline-delimited list", see
`git-sparse-checkout(1)`), but within nixpkgs it is more consistent to
use a list of strings instead. The list elements are joined to a
multi-line string only before passing it to the builder script.

A deprecation warning is emitted if a (multi-line) string is passed to
`sparseCheckout`, but for the time being it is still accepted.
2022-11-15 19:45:33 +01:00
pinn3
bab4cdd43a fetchgit: Remove comment regarding path needing to be a string
It was changed in 2019 to be of the actual path type, and has apparently
been working since then.

Closes #143846
2022-08-18 04:50:32 +02:00
Winter
dbd18a63a7 fetchgit: allow disabling cone mode for sparse checkouts, fix test 2022-08-14 23:03:07 +02:00
Adam Joseph
c8a1838d30 fetchgit: remove "security" from comment about "security risk"
Closes #178410
2022-08-03 05:07:34 -07:00
Sebastian Jordan
783e2ef46e Revert "nix-prefetch-git: Fix inconsistency with fetchgit regarding deepClone"
This reverts commit 1dfaad73ed.
2022-06-30 11:52:36 +00:00
Maximilian Bosch
e4bf881ee0
Merge pull request #177575 from seppeljordan/nix-prefetch-git-fix
nix-prefetch-git: Fix inconsistency with fetchgit regarding deepClone
2022-06-28 14:03:41 +02:00
Sebastian Jordan
1dfaad73ed nix-prefetch-git: Fix inconsistency with fetchgit regarding deepClone
The fetchgit function in nixpkgs sets the leaveDotGit argument to true
if deepClone is set to true. nix-prefetch-git did behave differently. It
would not assume --leave-dotGit if --deepClone is specified. With this
change the inconsistency is addressed by assuming --leave-dotGit if
--deepClone is specified.
2022-06-14 00:23:16 +02:00
Adam Joseph
5ed4944130 fetchgit: allow passing allowedRequisites through to stdenv.mkDerivation
When maintainers override stages of `fetchgit' (e.g. `postPatch`) it
is very easy for them to accidentally leak the outpath-hash of their
current `stdenv` into `fetchgit''s output, and therefore into the
value they paste into `sha256`.

This is a problem, because the resulting expression will break
whenever any change is made to `stdenv` or when anybody attempts to
build the expression on a different platform than the one used by the
original maintainer.

Almost as much of a problem is the fact that CI **does not catch**
these problems.  The `fetchgit` is run only once, then its output goes
into cachix, and all future builds (hydra, CI, ofborg) pull from
cachix.

Let's offer maintainers the option to check that they aren't making
this mistake, by passing through `allowedRequisites`.  The default
value is `null`, but it might be worth changing that at some point in
the future.

It is also sometimes difficult to communicate to package maintainers
why their expression is problematic.  Having `allowedRequisites`
passed through makes it easier to do this: "look, when I switch on
`allowedRequisites` your package breaks; are you sure you meant to
hardcode the hash today's `x86_64-linux.stdenv` into your expression?`

For an example use case, see https://github.com/NixOS/nixpkgs/pull/171223

The issue above is part of a larger problem with nixpkgs infra: there
large parts of cachix cannot be reproduced easily if they are lost.
Once something ends goes into cachix, we never ever again reverify the
procedure by which it was placed into cachix.
2022-06-11 12:42:33 -07:00
Jan Tojnar
7aae279ad9 unstableGitUpdater: fix updating fetchzip-based sources
a67950f20b added `url` attribute
from `fetchurl` and therefore also from `fetchzip`.
We previously relied on `url` from fetchgit-based fetchers
to find the repo URL but now it will just return tarballs
in the case of `fetchFrom{GitHub,GitLab}`.

Let’s add an attribute to `fetch{git,FromGitHub,FromGitLab}`
to expose a repo URL consistently.
2022-05-24 19:12:12 +02:00
Robert Hensing
c071530ca5 testers.invalidateFetcherByDrvHash: Move from top-level 2022-05-05 12:08:50 +02:00
Sandro Jäckel
538a5a28b7
fetchgit: allow passing thru meta 2022-04-18 03:06:02 +02:00
Maciej Krüger
ba83787f0d
fetchgit: split off deterministic-git functions 2022-02-19 14:33:20 +01:00
Zhong Jianxin
36cc03e151 fetchgit: Add test for sparseCheckout 2022-01-24 11:23:56 +08:00
Zhong Jianxin
d03a07d5a7 fetchgit: Support sparse checkout
This allow git checkout small parts of a large repo, and avoid fetching
unnecessary blobs from server.
2022-01-24 11:23:56 +08:00
Tomas Bravo
80ab168c4a nix-prefetch-git: fix incorrect mktemp usage 2021-12-22 16:45:18 +01:00
Maciej Krüger
5c2b1b6a29
fetchgit: support hash parameter alongside sha256
Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2021-11-09 07:40:20 +01:00
Sandro Jäckel
6f53c06748 fetchgit: add support for netrc file through impure NIX_GIT_SSL_CAINFO env 2021-09-28 15:00:11 +02:00
Robert Hensing
1648346444
Merge pull request #136022 from hercules-ci/fetcher-testing
Fetcher testing
2021-09-21 14:50:28 +02:00
Robert Hensing
0fa3d10d7d fetchgit.tests: init 2021-08-28 10:29:10 +02:00
Bjørn Forsman
9bd51f0db9 nix-prefetch-git: add fetchLFS flag to the JSON output
Similarly to the other flags (e.g. fetchSubmodules).
2021-08-28 10:14:48 +02:00
Bjørn Forsman
f43829a1d8 nix-prefetch-git: provide fallback for $TMPDIR
$TMPDIR is not set by default in NixOS (nor Ubuntu), so without this
change nix-prefetch-git fails when using git-lfs.
2021-08-28 10:14:48 +02:00
Atemu
11df41199b nix-prefetch-git: don't fetch everything when given a hash
It's hugely inefficient as we can't use shallow cloning (--depth=1).

This has been tested and adapted for quite a few hosts fetchgit is used on in
Nixpkgs. For those where fetching the hash directly doesn't work (most notably
git.savannah.gnu.org), we simply fall back to the old method.
2021-07-12 20:50:52 +02:00
Bjørn Forsman
2e8f6e0d93 fetchgit: document --fetch-lfs flag
Fixes: 74128561 ("fetchgit: add lfs support")
2021-05-07 15:38:54 -07:00
Sandro
d6df2c3fab
Merge pull request #111782 from Luflosi/fetchgit-escape-regex
fetchgit: escape dot in regex
2021-03-21 05:34:53 +01:00
taku0
a964d7094a
Merge pull request #104714 from codedownio/tree-hashes
fetchgit: support passing tree hashes as "rev"
2021-02-20 15:22:57 +09:00
Michael Weiss
2aadb9a53d
nix-prefetch-git: Run git-init with --initial-branch=master
The reason for this change is simply to avoid the following messages
that are unnecessary and can be confusing (and these messages will be
repeated for each submodule):
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint:   git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint:   git branch -m <name>

With this change the behaviour remains unchanged (apart from the
suppressed "warning" in the console output of course) and therefore this
doesn't cause any hashes to change and by default nix-prefetch-git uses
the "fetchgit" branch anyway (branchName can be set to override the
default):
Switched to a new branch 'fetchgit'

For that reason the initial branch name doesn't matter anyway and since
we're not relying on / hardcoding "master" we could simply switch to
"main" (which seems most common nowadays). See [0] for more details on
why this wouldn't break anything.
However, since the initial branch name doesn't matter and to avoid any
additional risks it was "decided" to keep using "master" (s. #113313).

[0]: https://github.com/NixOS/nixpkgs/pull/113313#issuecomment-780589516
2021-02-18 11:34:34 +01:00
Tom McLaughlin
6779902b32 fetchgit: support passing tree hashes as "rev" 2021-02-13 22:12:08 -08:00
Luflosi
e238ff073f
fetchgit: escape dot in regex
This regex should match files ending in `.git`, not any character and `git` after that.
2021-02-03 18:41:50 +01:00
Ben Siraphob
02ee14b9d5 treewide: stdenvNoCC.lib -> lib 2021-01-27 13:01:51 +07:00
Jonathan Ringer
741285611f fetchgit: add lfs support 2021-01-11 09:41:06 -08:00