This is useful when running tools like NixOps or nix-review
on workstations where the upload to the builder is significantly
slower then downloading the source on the builder itself.
The script would parse the output of `git submodule status` but
didn't handle paths with spaces in them. This would result in the
following error when trying to determine the URL of the submodule:
error: key does not contain a section: .url
Summary:
According to git-submodule manpage,
"git submodule status" prefixes the hash with a '-' if it is not
initialized, and other chars in other circumstances.
(this is consistent on the various git versions tested)
nix-prefetch-git runs "git submodule init" which does you'd think,
but apparently despite this earlier versions of git before 2.16
would still give the hash the '-' suffix.
In particular this is the behavior when using 2.15 and 2.14.1
from the nixos-17.09 and nixos-17.03 channels respectively.
The script then used awk to drop the first char of the first field
which does the wrong thing when there is no prefix emitted:
while there is a space character before the hash, this is not
part of the field and so we ended up eating the first character
of the hash.
To fix this in a way that also works with the previous behavior,
this commit instead uses awk to grab the hash field
and uses tr to delete any '-' chars should they be present.
This seems to work in my testing, and for example can now
successfully fetch the source for "nginxModules.brotli"
where previously it would generate an error:
fatal: '22564a95d9ab58865a096b8d9f7324ea5f2e03e' is not a commit and a branch 'fetchgit' cannot be created from it
(we dropped a '2' from the beginning of the hash)
Currently we wrap ssh so it can find the config file passed in by
<ssh-config-file>. If one however uses ProxyCommand ssh, then ssh that
is on PATH is taken (which is also unavailable when using nix-shell
--pure), which is the plain ${openssh}/bin/ssh.
This commit makes sure our wrapped ssh is available on PATH.
They're additional commits from #26877.
Changing names of the fetched stuff was changing very many hashes,
and I think it's better to avoid that for the moment to reduce work
needed by nixpkgs users. The fetchers are expected to be commonly
used even outside nixpkgs, and the current naming wasn't that bad
usually.
(commit analogical to d10c3cc5eedf58e80e2; I haven't noticed the part of
the PR has already got to master)
Deprecation warnings should not be used in Nixpkgs because they spam
innocent "nix-env -qa" users with (in this case) dozens of messages
that they can't do anything about.
This also reverts commit 2ca8833383.
This commit extends fetchFromGitHub with ability to fetch GitHub
repositories with submodules, so we can use the function consistently
with all GitHub repositories.
Note it doesn't change the previous behavior.
This fixes a regression caused by commit f56ab9e
("nix-prefetch-git: Include the date in the machine-readable [...]")
where a couple of directory paths printed by pushd/popd appeared before
the JSON output on stdout (thus breaking it). Fix it by redirecting the
extraneous output to /dev/null.
Reported by Michael Alan Dorman <mdorman@ironicdesign.com>.
stdout, in strict ISO 8601 format.
This will be helpful for automatically updating fetchgit expressions
and the dates in version numbers associated with them.
Source of this change goes back to 2009 and original version of
fetchgit at 205fb0c87e.
The nondeterminism is really caused by changing .git so leave other
files alone as they might be interesting.
Note: this causes a hash mismatch with Hydra's version of Git Plugin
which we should fix to comply.
Previously, nix-prefetch-git would report the same JSON whether submodules were being fetched or not; with this change, the --fetch-submodules option will cause the JSON output to include "fetchSubmodules": true, so that fetchgit (builtins.fromJSON (builtins.readFile ./path/to/output.json)) will work.
This commit fixes#6651.
Before this change the `nix-prefetch-git` script would use a different store
name than nix's `fetchgit` function. Because of that it was not possible to
use `nix-prefetch-git` as a way to pre-populate the store (for example when
the user it using private git dependencies that needs access to the ssh agent)
The comment related to the `deepClone` and `no-deepClone` options was
misleading as these options have no relation with submodules, but on the
the depth in `git clone --depth n`.
These environment variables allow using fetchgit with git:// URLs using
the SOCKS proxy technique described in 'Using Git with a SOCKS proxy':
http://www.patthoyts.tk/blog/using-git-with-socks-proxy.html
Briefly, GIT_PROXY_COMMAND is set to a script which invokes connect[1],
which reads SOCKS_PROXY, which might be pointing to a local instance of
'ssh -D'.
[1] pkgs/tools/networking/connect
If "fetcher" is a string, then Nix will execute it with bash already, so
the additional bash argument in that string was redundant and apparently
causes trouble on non-Linux platforms.
Hopefully fixes https://github.com/NixOS/nixpkgs/issues/11496.
Close#9790.
This fixes checkouting for a nasty combination:
1. To be checkouted is a revision which corresponds to tag in a form "<tag>^{}".
2. This revision is not fetched by default.
This is useful when `leaveDotGit = true` and some other derivation
expects some branch name to exist.
Previously, `nix-prefetch-git` always created a branch with a
hard-coded name (`fetchgit`).
This patch resolves https://github.com/NixOS/nixpkgs/issues/6395. Deep
cloning is useful in combination with 'leaveDotGit' for builds that want
to run "git describe" to obtain a proper version string, etc., like the
'haskellngPackages.cabal2nix' package does.
The name detection didn't work for e.g. http://git.suckless.org/sinit/.
I tested the tarball builds now.
@shlevy claimed nixpkgs requires nix-1.8 features anyway,
so the additional check with message were superfluous.
Without this, the generated pack files are non-deterministic.
I didn't notice this issue in my earlier testing, because my test repo
had too few commits for the thread scheduling to take effect. (Test repo
had about 10 commits.)
Add more files to the delete list:
* .git/FETCH_HEAD
* .git/ORIG_HEAD
* .git/refs/remotes/origin/HEAD
* .git/config
Further, remove all remote branches, remove tags not reachable from the
given 'rev', do a full repack and then garbage collect unreferenced
objects.
According to my testing, the result is fully deterministic. As in "any
change done to the upstream repo, ahead of 'rev', will not affect the
hash of the resulting 'clone'". Even changing the clone URL will not
change the output hash, because .git/config is removed.
A new version of git can of course change store format, but that's
unavoidable.
For big repositories, the repack operation may be a bit heavy. But as
far as I can see there is no cheaper way to determinism.
Fixes a regression on OS X introduced by f83af95.
Don't use --tmpdir for mktemp, because that flag doesn't exist on OS X.
However, using -t is deprecated in GNU coreutils, so as suggested by
@ip1981 we're now using parameter expansion on ${TMPDIR:-/tmp} to
provide /tmp as a fallback if TMPDIR is not set and use it instead.
Also use this approach for nix-prefetch-cvs now in order to stay
consistent.
Reported-by: Vladimir Kirillov <proger@wilab.org.ua>
Tested-by: Igor Pashev <pashev.igor@gmail.com>
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Instead of relying on $$ to not collide with an existing path.
Quoting the Bash manual about $$:
> Expands to the process ID of the shell. In a () subshell, it expands
> to the process ID of the current shell, not the subshell.
So, this is different from $BASHPID:
> Expands to the process ID of the current bash process. This differs
> from $$ under certain circumstances, such as subshells that do not
> require bash to be re-initialized.
But even $BASHPID is prone to race conditions if the process IDs wrap
around, so to be on the safe side, we're using mktemp here.
Closes#3784.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
There was a few files containing timestamp, so we now remove them.
It shouldn't be a problem for logs. However, index might be. Anyway,
that's better than nothing.
If the user explictly gives a ref such as "refs/heads/master", `git
rev-parse` failed because we only checked out the `fetchgit`
branch. Now, we also try `git rev-parse fetchgit` if the first call
fails, which fixes the issue.
nix-prefetch-git does not convert relative submodule urls into absolute
urls based on the parent's origin. This patch adds support for
repositories which are using the relative url syntax.
The nix-prefect git script was broken when trying to parse certain
groups of submodules. This patch fixes the url detection for submodule
repositories to use the more reliable `git config` commands.
