From 2ac1f685b699fe25bcb1dcb684ffc8b85fe4010e Mon Sep 17 00:00:00 2001 From: Paul Meyer <49727155+katexochen@users.noreply.github.com> Date: Fri, 18 Oct 2024 09:53:29 +0200 Subject: [PATCH] docs: update Go section after buildGoPackage removal Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com> --- doc/languages-frameworks/go.section.md | 256 +++++++----------- doc/redirects.json | 30 +- .../manual/release-notes/rl-2505.section.md | 2 +- 3 files changed, 115 insertions(+), 173 deletions(-) diff --git a/doc/languages-frameworks/go.section.md b/doc/languages-frameworks/go.section.md index 3100124e0ce5..4529639f3d4a 100644 --- a/doc/languages-frameworks/go.section.md +++ b/doc/languages-frameworks/go.section.md @@ -7,32 +7,6 @@ The function `buildGoModule` builds Go programs managed with Go modules. It buil - An intermediate fetcher derivation called `goModules`. This derivation will be used to fetch all the dependencies of the Go module. - A final derivation will use the output of the intermediate derivation to build the binaries and produce the final output. -### Attributes of `buildGoModule` {#buildgomodule-parameters} - -The `buildGoModule` function accepts the following parameters in addition to the [attributes accepted by both Go builders](#ssec-go-common-attributes): - -- `vendorHash`: is the hash of the output of the intermediate fetcher derivation (the dependencies of the Go modules). - - `vendorHash` can be set to `null`. - In that case, rather than fetching the dependencies, the dependencies already vendored in the `vendor` directory of the source repo will be used. - - To avoid updating this field when dependencies change, run `go mod vendor` in your source repo and set `vendorHash = null;`. - You can read more about [vendoring in the Go documentation](https://go.dev/ref/mod#vendoring). - - To obtain the actual hash, set `vendorHash = lib.fakeHash;` and run the build ([more details here](#sec-source-hashes)). -- `proxyVendor`: If `true`, the intermediate fetcher downloads dependencies from the - [Go module proxy](https://go.dev/ref/mod#module-proxy) (using `go mod download`) instead of vendoring them. The resulting - [module cache](https://go.dev/ref/mod#module-cache) is then passed to the final derivation. - - This is useful if your code depends on C code and `go mod tidy` does not include the needed sources to build or - if any dependency has case-insensitive conflicts which will produce platform-dependent `vendorHash` checksums. - - Defaults to `false`. -- `modPostBuild`: Shell commands to run after the build of the goModules executes `go mod vendor`, and before calculating fixed output derivation's `vendorHash`. - Note that if you change this attribute, you need to update `vendorHash` attribute. -- `modRoot`: The root directory of the Go module that contains the `go.mod` file. - Defaults to `./`, which is the root of `src`. - ### Example for `buildGoModule` {#ex-buildGoModule} The following is an example expression using `buildGoModule`: @@ -62,17 +36,45 @@ The following is an example expression using `buildGoModule`: } ``` -### Obtaining and overriding `vendorHash` for `buildGoModule` {#buildGoModule-vendorHash} +## Attributes of `buildGoModule` {#buildgomodule-parameters} + +Many attributes [controlling the build phase](#variables-controlling-the-build-phase) are respected by `buildGoModule`. Note that `buildGoModule` reads the following attributes also when building the `vendor/` goModules fixed output derivation as well: + +- [`sourceRoot`](#var-stdenv-sourceRoot) +- [`prePatch`](#var-stdenv-prePatch) +- [`patches`](#var-stdenv-patches) +- [`patchFlags`](#var-stdenv-patchFlags) +- [`postPatch`](#var-stdenv-postPatch) +- [`preBuild`](#var-stdenv-preBuild) +- `env`: useful for passing down variables such as `GOWORK`. + +To control test execution of the build derivation, the following attributes are of interest: + +- [`checkInputs`](#var-stdenv-checkInputs) +- [`preCheck`](#var-stdenv-preCheck) +- [`checkFlags`](#var-stdenv-checkFlags) + +In addition to the above attributes, and the many more variables respected also by `stdenv.mkDerivation`, `buildGoModule` respects Go-specific attributes that tweak them to behave slightly differently: + +### `vendorHash` {#var-go-vendorHash} + +Hash of the output of the intermediate fetcher derivation (the dependencies of the Go modules). + +`vendorHash` can be set to `null`. +In that case, rather than fetching the dependencies, the dependencies already vendored in the `vendor` directory of the source repo will be used. + +To avoid updating this field when dependencies change, run `go mod vendor` in your source repo and set `vendorHash = null;`. +You can read more about [vendoring in the Go documentation](https://go.dev/ref/mod#vendoring). + +To obtain the hash, set `vendorHash = lib.fakeHash;` and run the build. ([more details here](#sec-source-hashes)). +Another way is to use use `nix-prefetch` to obtain the hash. The following command gets the value of `vendorHash` for package `pet`: -We can use `nix-prefetch` to obtain the actual hash. The following command gets the value of `vendorHash` for package `pet`: ```sh cd path/to/nixpkgs nix-prefetch -E "{ sha256 }: ((import ./. { }).my-package.overrideAttrs { vendorHash = sha256; }).goModules" ``` -To obtain the hash without external tools, set `vendorHash = lib.fakeHash;` and run the build. ([more details here](#sec-source-hashes)). - `vendorHash` can be overridden with `overrideAttrs`. Override the above example like this: ```nix @@ -91,144 +93,29 @@ To obtain the hash without external tools, set `vendorHash = lib.fakeHash;` and } ``` -### Overriding `goModules` {#buildGoModule-goModules-override} +### `proxyVendor` {#var-go-proxyVendor} -Overriding `.goModules` by calling `goModules.overrideAttrs` is unsupported. Still, it is possible to override the `vendorHash` (`goModules`'s `outputHash`) and the `pre`/`post` hooks for both the build and patch phases of the primary and `goModules` derivation. Alternatively, the primary derivation provides an overridable `passthru.overrideModAttrs` function to store the attribute overlay implicitly taken by `goModules.overrideAttrs`. Here's an example usage of `overrideModAttrs`: +If `true`, the intermediate fetcher downloads dependencies from the +[Go module proxy](https://go.dev/ref/mod#module-proxy) (using `go mod download`) instead of vendoring them. The resulting +[module cache](https://go.dev/ref/mod#module-cache) is then passed to the final derivation. -```nix -{ - pet-overridden = pet.overrideAttrs ( - finalAttrs: previousAttrs: { - passthru = previousAttrs.passthru // { - # If the original package has an `overrideModAttrs` attribute set, you'd - # want to extend it, and not replace it. Hence we use - # `lib.composeExtensions`. If you are sure the `overrideModAttrs` of the - # original package trivially does nothing, you can safely replace it - # with your own by not using `lib.composeExtensions`. - overrideModAttrs = lib.composeExtensions previousAttrs.passthru.overrideModAttrs ( - finalModAttrs: previousModAttrs: { - # goModules-specific overriding goes here - postBuild = '' - # Here you have access to the `vendor` directory. - substituteInPlace vendor/github.com/example/repo/file.go \ - --replace-fail "panic(err)" "" - ''; - } - ); - }; - } - ); -} -``` +This is useful if your code depends on C code and `go mod tidy` does not include the needed sources to build or +if any dependency has case-insensitive conflicts which will produce platform-dependent `vendorHash` checksums. -## `buildGoPackage` (legacy) {#ssec-go-legacy} +Defaults to `false`. -The function `buildGoPackage` builds legacy Go programs, not supporting Go modules. -::: {.warning} -`buildGoPackage` is deprecated and will be removed for the 25.05 release. -::: +### `modPostBuild` {#var-go-modPostBuild} -### Migrating from `buildGoPackage` to `buildGoModule` {#buildGoPackage-migration} +Shell commands to run after the build of the goModules executes `go mod vendor`, and before calculating fixed output derivation's `vendorHash`. +Note that if you change this attribute, you need to update `vendorHash` attribute. -Go modules, released 6y ago, are now widely adopted in the ecosystem. -Most upstream projects are using Go modules, and the tooling previously used for dependency management in Go is mostly deprecated, archived or at least unmaintained at this point. -In case a project doesn't have external dependencies or dependencies are vendored in a way understood by `go mod init`, migration can be done with a few changes in the package. +### `modRoot` {#var-go-modRoot} -- Switch the builder from `buildGoPackage` to `buildGoModule` -- Remove `goPackagePath` and other attributes specific to `buildGoPackage` -- Set `vendorHash = null;` -- Run `go mod init ` in `postPatch` +The root directory of the Go module that contains the `go.mod` file. -In case the package has external dependencies that aren't vendored or the build setup is more complex the upstream source might need to be patched. -Examples for the migration can be found in the [issue tracking migration withing nixpkgs](https://github.com/NixOS/nixpkgs/issues/318069). - -### Example for `buildGoPackage` {#example-for-buildgopackage} - -In the following is an example expression using `buildGoPackage`, the following arguments are of special significance to the function: - -- `goPackagePath` specifies the package's canonical Go import path. -- `goDeps` is where the Go dependencies of a Go program are listed as a list of package source identified by Go import path. It could be imported as a separate `deps.nix` file for readability. The dependency data structure is described below. - -```nix -{ - deis = buildGoPackage rec { - pname = "deis"; - version = "1.13.0"; - - goPackagePath = "github.com/deis/deis"; - - src = fetchFromGitHub { - owner = "deis"; - repo = "deis"; - rev = "v${version}"; - hash = "sha256-XCPD4LNWtAd8uz7zyCLRfT8rzxycIUmTACjU03GnaeM="; - }; - - goDeps = ./deps.nix; - }; -} -``` - -The `goDeps` attribute can be imported from a separate `nix` file that defines which Go libraries are needed and should be included in `GOPATH` for `buildPhase`: - -```nix -# deps.nix -[ # goDeps is a list of Go dependencies. - { - # goPackagePath specifies Go package import path. - goPackagePath = "gopkg.in/yaml.v2"; - fetch = { - # `fetch type` that needs to be used to get package source. - # If `git` is used there should be `url`, `rev` and `hash` defined next to it. - type = "git"; - url = "https://gopkg.in/yaml.v2"; - rev = "a83829b6f1293c91addabc89d0571c246397bbf4"; - hash = "sha256-EMrdy0M0tNuOcITaTAmT5/dPSKPXwHDKCXFpkGbVjdQ="; - }; - } - { - goPackagePath = "github.com/docopt/docopt-go"; - fetch = { - type = "git"; - url = "https://github.com/docopt/docopt-go"; - rev = "784ddc588536785e7299f7272f39101f7faccc3f"; - hash = "sha256-Uo89zjE+v3R7zzOq/gbQOHj3SMYt2W1nDHS7RCUin3M="; - }; - } -] -``` - -To extract dependency information from a Go package in automated way use [go2nix (deprecated)](https://github.com/kamilchm/go2nix). It can produce complete derivation and `goDeps` file for Go programs. - -You may use Go packages installed into the active Nix profiles by adding the following to your ~/.bashrc: - -```bash -for p in $NIX_PROFILES; do - GOPATH="$p/share/go:$GOPATH" -done -``` - -## Attributes used by both builders {#ssec-go-common-attributes} - -Many attributes [controlling the build phase](#variables-controlling-the-build-phase) are respected by both `buildGoModule` and `buildGoPackage`. Note that `buildGoModule` reads the following attributes also when building the `vendor/` goModules fixed output derivation as well: - -- [`sourceRoot`](#var-stdenv-sourceRoot) -- [`prePatch`](#var-stdenv-prePatch) -- [`patches`](#var-stdenv-patches) -- [`patchFlags`](#var-stdenv-patchFlags) -- [`postPatch`](#var-stdenv-postPatch) -- [`preBuild`](#var-stdenv-preBuild) -- `env`: useful for passing down variables such as `GOWORK`. - -To control test execution of the build derivation, the following attributes are of interest: - -- [`checkInputs`](#var-stdenv-checkInputs) -- [`preCheck`](#var-stdenv-preCheck) -- [`checkFlags`](#var-stdenv-checkFlags) - -In addition to the above attributes, and the many more variables respected also by `stdenv.mkDerivation`, both `buildGoModule` and `buildGoPackage` respect Go-specific attributes that tweak them to behave slightly differently: +Defaults to `./`, which is the root of `src`. ### `ldflags` {#var-go-ldflags} @@ -326,6 +213,38 @@ Whether the build result should be allowed to contain references to the Go tool Defaults to `false` +## Overriding `goModules` {#buildGoModule-goModules-override} + +Overriding `.goModules` by calling `goModules.overrideAttrs` is unsupported. Still, it is possible to override the `vendorHash` (`goModules`'s `outputHash`) and the `pre`/`post` hooks for both the build and patch phases of the primary and `goModules` derivation. + +Alternatively, the primary derivation provides an overridable `passthru.overrideModAttrs` function to store the attribute overlay implicitly taken by `goModules.overrideAttrs`. Here's an example usage of `overrideModAttrs`: + +```nix +{ + pet-overridden = pet.overrideAttrs ( + finalAttrs: previousAttrs: { + passthru = previousAttrs.passthru // { + # If the original package has an `overrideModAttrs` attribute set, you'd + # want to extend it, and not replace it. Hence we use + # `lib.composeExtensions`. If you are sure the `overrideModAttrs` of the + # original package trivially does nothing, you can safely replace it + # with your own by not using `lib.composeExtensions`. + overrideModAttrs = lib.composeExtensions previousAttrs.passthru.overrideModAttrs ( + finalModAttrs: previousModAttrs: { + # goModules-specific overriding goes here + postBuild = '' + # Here you have access to the `vendor` directory. + substituteInPlace vendor/github.com/example/repo/file.go \ + --replace-fail "panic(err)" "" + ''; + } + ); + }; + } + ); +} +``` + ## Controlling the Go environment {#ssec-go-environment} The Go build can be further tweaked by setting environment variables. In most cases, this isn't needed. Possible values can be found in the [Go documentation of accepted environment variables](https://pkg.go.dev/cmd/go#hdr-Environment_variables). Notice that some of these flags are set by the builder itself and should not be set explicitly. If in doubt, grep the implementation of the builder. @@ -364,4 +283,23 @@ If a larger amount of tests should be skipped, the following pattern can be used ``` To disable tests altogether, set `doCheck = false;`. -`buildGoPackage` does not execute tests by default. + +## Migrating from `buildGoPackage` to `buildGoModule` {#buildGoPackage-migration} + +::: {.warning} +`buildGoPackage` was removed for the 25.05 release. It was used to build legacy Go programs +that do not support Go modules. +::: + +Go modules, released 6y ago, are now widely adopted in the ecosystem. +Most upstream projects are using Go modules, and the tooling previously used for dependency management in Go is mostly deprecated, archived or at least unmaintained at this point. + +In case a project doesn't have external dependencies or dependencies are vendored in a way understood by `go mod init`, migration can be done with a few changes in the package. + +- Switch the builder from `buildGoPackage` to `buildGoModule` +- Remove `goPackagePath` and other attributes specific to `buildGoPackage` +- Set `vendorHash = null;` +- Run `go mod init ` in `postPatch` + +In case the package has external dependencies that aren't vendored or the build setup is more complex the upstream source might need to be patched. +Examples for the migration can be found in the [issue tracking migration withing nixpkgs](https://github.com/NixOS/nixpkgs/issues/318069). diff --git a/doc/redirects.json b/doc/redirects.json index de640eed00c1..99f30ff667a2 100644 --- a/doc/redirects.json +++ b/doc/redirects.json @@ -2718,29 +2718,20 @@ "index.html#ssec-language-go" ], "buildgomodule-parameters": [ - "index.html#buildgomodule-parameters" + "index.html#buildgomodule-parameters", + "index.html#ssec-go-common-attributes" ], "ex-buildGoModule": [ "index.html#ex-buildGoModule" ], - "buildGoModule-vendorHash": [ - "index.html#buildGoModule-vendorHash" - ], "buildGoModule-goModules-override": [ "index.html#buildGoModule-goModules-override" ], - "ssec-go-legacy": [ - "index.html#ssec-go-legacy" - ], "buildGoPackage-migration": [ - "index.html#buildGoPackage-migration" - ], - "example-for-buildgopackage": [ + "index.html#buildGoPackage-migration", + "index.html#ssec-go-legacy", "index.html#example-for-buildgopackage" ], - "ssec-go-common-attributes": [ - "index.html#ssec-go-common-attributes" - ], "var-go-ldflags": [ "index.html#var-go-ldflags" ], @@ -2756,6 +2747,9 @@ "var-go-excludedPackages": [ "index.html#var-go-excludedPackages" ], + "var-go-proxyVendor": [ + "index.html#var-go-proxyVendor" + ], "var-go-CGO_ENABLED": [ "index.html#var-go-CGO_ENABLED" ], @@ -2765,6 +2759,16 @@ "var-go-allowGoReference": [ "index.html#var-go-allowGoReference" ], + "var-go-vendorHash": [ + "index.html#var-go-vendorHash", + "index.html#buildGoModule-vendorHash" + ], + "var-go-modPostBuild": [ + "index.html#var-go-modPostBuild" + ], + "var-go-modRoot": [ + "index.html#var-go-modRoot" + ], "ssec-go-environment": [ "index.html#ssec-go-environment" ], diff --git a/nixos/doc/manual/release-notes/rl-2505.section.md b/nixos/doc/manual/release-notes/rl-2505.section.md index 83141891294c..f184b5b44502 100644 --- a/nixos/doc/manual/release-notes/rl-2505.section.md +++ b/nixos/doc/manual/release-notes/rl-2505.section.md @@ -20,7 +20,7 @@ -- `buildGoPackage` has been removed. Use `buildGoModule` instead. See the Go section in the nixpkgs manual for details. +- `buildGoPackage` has been removed. Use `buildGoModule` instead. See the [Go section in the nixpkgs manual](https://nixos.org/manual/nixpkgs/unstable/#sec-language-go) for details. - `kanata` was updated to v1.7.0, which introduces several breaking changes. See the release notes of