The way in which Nixpks builds Ruby gems means that certain operations
by bundler *will not work*, namely `bundle install --redownload`.
According to the source the _cache/_ directory should have been kept,
however it seems through revisions to the file it has been purged.
Here was the comment from the original commit that introduced
buildRubyGem:
```
# Note:
# We really do need to keep the $out/${ruby.gemPath}/cache.
# This is very important in order for many parts of RubyGems/Bundler to not blow up.
# See https://github.com/bundler/bundler/issues/3327
```
Why is the _cache_ directory needed?
Bundler and RubyGems uses the cache as a source of truth.
When bundler executes `bundler install --redownload`, any gems it
discovers in the _GEM_PATH_ it assums must have their _.gem_ file
present in the cache (unaware it was installed from Nix).
Rather than downloading the gem from RubyGems the bundler code forcibly
re-installs the gem from the cache directory instead and **fails** if it
does not exist.
I've opened https://github.com/rubygems/rubygems/issues/4088 to see if
this failure should be soft and not so explicit; or fallback to fetching
the gem from scratch.
Without this change the following is the error:
```bash
> [nix-shell:~/code/nix/playground/jruby-bundler-rake]$ bundle install --force
[DEPRECATED] The `--force` option has been renamed to `--redownload`
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/nix/store/fis6nzrpw9pmcivr84qh5byfgm07qn10-jruby-9.2.13.0/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Fetching gem metadata from https://rubygems.org/.
Using bundler 2.1.4
Installing hello-world 1.2.0
Bundler::GemNotFound: Could not find hello-world-1.2.0.gem for installation
An error occurred while installing hello-world (1.2.0), and Bundler
cannot continue.
Make sure that `gem install hello-world -v '1.2.0' --source
'https://rubygems.org/'` succeeds before bundling.
```
Wth the fix the following no woccurs:
```bash
[nix-shell:~/code/nix/playground/jruby-bundler-rake]$ bundle install --redownload
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/nix/store/69wjlj4yirp48rv1q03zxgd4xvf0150d-jruby-9.2.13.0/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Fetching gem metadata from https://rubygems.org/.
Using bundler 2.1.4
Installing hello-world 1.2.0
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
```
```
[nix-shell:~/code/nix/playground/jruby-bundler-rake]$ ls -l /nix/store/cwl9n5073hqgpfhnw4wic13nrrgg9dn8-gem-env/lib/jruby/gems/2.5.0/cache/
total 8
lrwxrwxrwx 1 fmzakari primarygroup 102 Dec 31 1969 bundler-2.1.4.gem -> /nix/store/ifc8a0gsfkrhkv953rd4rz8bcspahi8y-bundler-2.1.4/lib/jruby/gems/2.5.0/cache/bundler-2.1.4.gem
lrwxrwxrwx 1 fmzakari primarygroup 110 Dec 31 1969 hello-world-1.2.0.gem -> /nix/store/xi9ln6n1mz2is5ppykjxqhhkpjq9zm6i-hello-world-1.2.0/lib/jruby/gems/2.5.0/cache/hello-world-1.2.0.gem
```
I have a minimal project that demonstrates this issue at https://github.com/fzakaria/jruby-bundler-nix-failure
Changed ruby version to 2.7.x to match upstream.
Added a gem config for gitlab-pg_query as it tries to download a source
tarball during the build process.
Also removed a patch for gitaly that has become obsolete by upstream fix
[here](de04077c25).
The way ruby loads gems and keeps track of their paths seems to not
always work very well when the gems are accessed through
symlinks. Ruby will then complain that the same files are loaded
multiple times; it relies on the file's full path to determine whether
the file is loaded or not.
This adds an option to simply copy all gem files into the environment
instead, which gets rid of this issue, but may instead result in major
file duplication.
According to https://endoflife.software/programming-languages/server-side-scripting/ruby
ruby 2.4 will go end-of-life in march, where the new release of nixpkgs
will be cut. We won't be able to support it for security updates.
Remove all references to ruby_2_4 and add ruby_2_7 instead where
missing.
Mark packages that depend on ruby 2.4 as broken:
* chefdk
* sonic-pi
Setting a Bundler version with GEM_PATH doesn't seem to work in Ruby
2.7, so we need to use the LOAD_PATH instead. Without this,
bundlerEnv environments will always use the version of Bundler that
comes with Ruby, which won't necessarily work because it isn't the
version that was used to generate the bundle.
For example, building ronn with Ruby 2.7 without this change results
in a broken executable, but it works (when built with all packaged
Ruby versions) after this change.
In building a gem whose native extension is a Rakefile, the previous
version of this code will call essentially `rake ""`, when it means to
call `rake`.
This change converts `""` into `[]` rather than `[""]`.
This fixes
nix-shell -p 'ruby.withPackages (const [])' ruby.devdoc
which otherwise wouldn't find documentation, unlike
nix-shell -p ruby ruby.devdoc
which would, because ruby has setup hooks to accomodate for this, that
were being masked by the withPackages wrapper.
Go beyond the obvious setup hooks now, with a bit of sed, with a skipped case:
- cc-wrapper's `dontlink`, because it already is handled.
Also, in nix files escaping was manually added.
EMP
Because the gemdir was referenced on the derivation, it would cause the
whole gemdir to get added to the store, which would in turn force the
derivation to be rebuilt whenever unrelated folder files would change.
The `gemset` field can be an attribute set as well in `buildRubyGem`,
however attribute sets can't be coerced to strings for the environment
set.
The impact should be relatively small as the environment variables are
only used by the `runCommand` script for `bunlderApp` which doesn't
refer to `gemset` at all.
This has two benefits:
- Anyone else using the `mathematical` gem will benefit from the config.
- The gem config overrides the default one, so we were losing the fixes
for other gems.
I had to change how the update script works. Now it looks at the `gems`
passthru from `bundlerApp`.
The manual documents both bundlerEnv and bundlerApp as providing `env`
and `wrappedRuby` attributes on the resulting derivations. However only
bundlerEnv actually had these attributes. Update bundlerApp to provide
the same passthru that bundlerEnv does.
In case of the gem type 'git', nix-bundle-install.rb was called with
wrong path to the git checkout.
${src} does contain the sources, but not the newly generated .git dir,
which is created in the buildPhase of pkgs/development/ruby-modules/gem/default.nix
In some rare cases, this .git dir is needed at installPhase.
Note: on macOS, it works without this config. Testing on Ubuntu/Debian with the parent sha will produce an error about extconf.rb failing to find openssl/sha.h.
`digest-sha3` is a C-extension gem which fails to build on Nix because
it uses non-literals as format strings which is forbidden by the default
Nix hardening settings. There is a pull request to fix that ([1]), but
the gem seems to be abandoned.
This PR disables the "format" hardening for `digest-sha3`.
[1]: https://github.com/phusion/digest-sha3-ruby/pull/8
"default" isn't really a group, it's more the absence of one. With
Bundler, this means that a gem should be installed unconditionally,
regardless of which groups are specified. It doesn't really make sense
to allow these gems to be omitted from a bundlerEnv.
This wasn't really an issue until the latest minor release of Bundix
(2.4), because prior to then Bundix didn't emit group attributes, and so
this functionality of bundlerEnv wasn't really used. However, it is now
apparent that a better default for bundlerEnv would be to include all
gem groups by default, not just the default group. This matches the
behavior of Bundler, and makes more sense, because the default group
alone isn't necessarily useful for anything -- consider a Rails app with
production, development, and test groups. It has the additional benefit
of being backwards compatible with how this would have worked before the
Bundix update.
Suppose I have a Gemfile like this:
source "https://rubygems.org"
gem "actioncable"
gem "websocket-driver", group: :test
The gemset.nix generated by Bundix 2.4.1 will set ActionCable's groups
to [ "default" ], and websocket-driver's to [ "test" ]. This means that
the generated bundlerEnv wouldn't include websocket-driver unless the
test group was included, even though it's required by the default group.
This is arguably a bug in Bundix (websocket-driver's groups should
probably be [ "default" "test" ] or just [ "default" ]), but there's no
reason bundlerEnv should omit dependencies even given such an input --
it won't necessarily come from Bundix, and it would be good for
bundlerEnv to do the right thing.
To fix this, filterGemset is now a recursive function, that adds
dependencies of gems in the group to the filtered gemset until it
stabilises on the gems that match the required groups, and all of their
recursive dependencies.
> /nix/store/r2vsi140pys7jnzyk0qz1fj9aji6sq40-ruby2.5.3-rb-readline-0.5.5/lib/ruby/gems/2.5.0/gems/rb-readline-0.5.5/lib/rbreadline.rb:1097:in `<module:RbReadline>': HOME environment variable (or HOMEDRIVE and HOMEPATH) must be set and point to a directory (RuntimeError)
It would be reasonable to have a Ruby program that depends on some other
program being in the PATH. In this case, the obvious thing to do would
be something like this:
bundlerApp {
# ...
buildInputs = [ makeWrapper ];
postBuild = ''
wrapProgram "$out/bin/foo" \
--prefix PATH : ${lib.makeBinPath [ dep ]}
'';
}
However, this doesn't work, because even though it just forwards most of
its arguments to `runCommand`, `bundlerApp` won't take a `buildInputs`
parameter. It doesn't even specify its own `buildInputs`, which means
that the `scripts` parameter to `bundlerApp` (which depends on
`makeWrapper`) is completely broken, and, as far as I can tell, has been
since its inception. I've added a `makeWrapper` build input if the
scripts parameter is present to fix this.
I've added a `buildInputs` option to `bundlerApp`. It's also passed
through to bundled-common because `postBuild` scripts are run there as
well. This actually means that in this example we'd end up going through
two layers of wrappers (one from `bundlerApp` and one from
bundled-common), but that has always been the case and isn't likely to
break anything. That oddity does suggest that it might be prudent to
not forward `postBuild` to bundled-common (or to at least use a
different option) though...
FWIW, as far as I can tell no package in nixpkgs uses either the
`scripts` or `postBuild` options to `bundlerApp`.
> WARNING: The next major version of capybara-webkit will require at
> least version 5.0 of Qt. You're using version 4.8.7.
I went to 5.9 instead of 5.11 because 5.11 doesn't currently build on
Darwin, whereas 5.9 can build on both Darwin and Linux, and is still
well within the >=5.0 requirement.
This fixes the bug introduced by 8686b98612 which broke bundlerEnv
exprs when gemdir was a string (thus making gemset a string by
`gemset = gemdir + "/gemset.nix"`) which made it being treated as a
set.
Don't default `doCheck` to false,
and use the default set of phases
so the phases list does not need to be overriden
in order to add checkPhase or installCheckPhase.
This just defers to `fetchurl` for fetching.
Additionally, update the `nix-bundle-install.rb` script to handle gems
installed from a path, i.e. those with a `url` source.
Some parts of that script have been disabled in the `path` case
that likely shouldn't be, but cause errors and aren't necessary to get
`vagrant` to work.
Keep the `source` attrset distinct to prevent its entries from merging
with the top level attrs.
Since each type of source has a different set of entries for `source`,
this is the easiest way to keep them together.
This will pave the way for a new `url` type of source.
This is a mass-rebuild of many ruby packages,
notably those using `git` type sources.
This allows patching the gemset output by bundix from a default.nix
file, making it easier to perform updates since the bundix update no
longer has to be manually updated.
