ValidPathInfo JSON format should use null not omit field

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

doc/manual/rl-next/store-object-info.md

@@ -0,0 +1,11 @@
+---
+synopsis: Store object info JSON format now uses `null` rather than omitting fields.
+prs: 9995
+---
+
+The [store object info JSON format](@docroot@/protocols/json/store-object-info.md), used for e.g. `nix path-info`, no longer omits fields to indicate absent information, but instead includes the fields with a `null` value.
+For example, `"ca": null` is used to to indicate a store object that isn't content-addressed rather than omitting the `ca` field entirely.
+This makes records of this sort more self-describing, and easier to consume programmatically.
+
+We will follow this design principle going forward;
+the [JSON guidelines](@docroot@/contributing/json-guideline.md) in the contributing section have been updated accordingly.

doc/manual/src/glossary.md

@@ -12,7 +12,7 @@
   For how Nix uses content addresses, see:
 
     - [Content-Addressing File System Objects](@docroot@/store/file-system-object/content-address.md)
-    - [content-addressed store object](#gloss-content-addressed-store-object)
+    - [Content-Addressing Store Objects](@docroot@/store/store-object/content-address.md)
     - [content-addressed derivation](#gloss-content-addressed-derivation)
 
   Software Heritage's writing on [*Intrinsic and Extrinsic identifiers*](https://www.softwareheritage.org/2020/07/09/intrinsic-vs-extrinsic-identifiers) is also a good introduction to the value of content-addressing over other referencing schemes.
@@ -137,9 +137,12 @@
 
 - [content-addressed store object]{#gloss-content-addressed-store-object}
 
-  A [store object] whose [store path] is determined by its contents.
+  A [store object] which is [content-addressed](#gloss-content-address),
+  i.e. whose [store path] is determined by its contents.
   This includes derivations, the outputs of [content-addressed derivations](#gloss-content-addressed-derivation), and the outputs of [fixed-output derivations](#gloss-fixed-output-derivation).
 
+  See [Content-Addressing Store Objects](@docroot@/store/store-object/content-address.md) for details.
+
 - [substitute]{#gloss-substitute}
 
   A substitute is a command invocation stored in the [Nix database] that

doc/manual/src/protocols/json/store-object-info.md

@@ -24,9 +24,11 @@ Info about a [store object].
 
   An array of [store paths][store path], possibly including this one.
 
-* `ca` (optional):
+* `ca`:
 
-  Content address of this store object's file system object, used to compute its store path.
+  If the store object is [content-addressed],
+  this is the content address of this store object's file system object, used to compute its store path.
+  Otherwise (i.e. if it is [input-addressed]), this is `null`.
 
 [store path]: @docroot@/store/store-path.md
 [file system object]: @docroot@/store/file-system-object.md
@@ -37,28 +39,30 @@ Info about a [store object].
 These are not intrinsic properties of the store object.
 In other words, the same store object residing in different store could have different values for these properties.
 
-* `deriver` (optional):
+* `deriver`:
 
-  The path to the [derivation] from which this store object is produced.
+  If known, the path to the [derivation] from which this store object was produced.
+  Otherwise `null`.
 
   [derivation]: @docroot@/glossary.md#gloss-store-derivation
 
 * `registrationTime` (optional):
 
-  When this derivation was added to the store.
+  If known, when this derivation was added to the store.
+  Otherwise `null`.
 
-* `ultimate` (optional):
+* `ultimate`:
 
   Whether this store object is trusted because we built it ourselves, rather than substituted a build product from elsewhere.
 
-* `signatures` (optional):
+* `signatures`:
 
   Signatures claiming that this store object is what it claims to be.
   Not relevant for [content-addressed] store objects,
   but useful for [input-addressed] store objects.
 
-  [content-addressed]: @docroot@/glossary.md#gloss-content-addressed-store-object
-  [input-addressed]: @docroot@/glossary.md#gloss-input-addressed-store-object
+[content-addressed]: @docroot@/store/store-object/content-address.md
+[input-addressed]: @docroot@/glossary.md#gloss-input-addressed-store-object
 
 ### `.narinfo` extra fields
 

src/libstore/path-info.cc

@@ -161,28 +161,23 @@ nlohmann::json UnkeyedValidPathInfo::toJSON(
     jsonObject["narSize"] = narSize;
 
     {
-        auto& jsonRefs = (jsonObject["references"] = json::array());
+        auto & jsonRefs = jsonObject["references"] = json::array();
         for (auto & ref : references)
             jsonRefs.emplace_back(store.printStorePath(ref));
     }
 
-    if (ca)
-        jsonObject["ca"] = renderContentAddress(ca);
+    jsonObject["ca"] = ca ? (std::optional { renderContentAddress(*ca) }) : std::nullopt;
 
     if (includeImpureInfo) {
-        if (deriver)
-            jsonObject["deriver"] = store.printStorePath(*deriver);
+        jsonObject["deriver"] = deriver ? (std::optional { store.printStorePath(*deriver) }) : std::nullopt;
 
-        if (registrationTime)
-            jsonObject["registrationTime"] = registrationTime;
+        jsonObject["registrationTime"] = registrationTime ? (std::optional { registrationTime }) : std::nullopt;
 
-        if (ultimate)
-            jsonObject["ultimate"] = ultimate;
+        jsonObject["ultimate"] = ultimate;
 
-        if (!sigs.empty()) {
-            for (auto & sig : sigs)
-                jsonObject["signatures"].push_back(sig);
-        }
+        auto & sigsObj = jsonObject["signatures"] = json::array();
+        for (auto & sig : sigs)
+            sigsObj.push_back(sig);
     }
 
     return jsonObject;
@@ -210,20 +205,25 @@ UnkeyedValidPathInfo UnkeyedValidPathInfo::fromJSON(
         throw;
     }
 
+    // New format as this as nullable but mandatory field; handling
+    // missing is for back-compat.
     if (json.contains("ca"))
-        res.ca = ContentAddress::parse(getString(valueAt(json, "ca")));
+        if (auto * rawCa = getNullable(valueAt(json, "ca")))
+            res.ca = ContentAddress::parse(getString(*rawCa));
 
     if (json.contains("deriver"))
-        res.deriver = store.parseStorePath(getString(valueAt(json, "deriver")));
+        if (auto * rawDeriver = getNullable(valueAt(json, "deriver")))
+            res.deriver = store.parseStorePath(getString(*rawDeriver));
 
     if (json.contains("registrationTime"))
-        res.registrationTime = getInteger(valueAt(json, "registrationTime"));
+        if (auto * rawRegistrationTime = getNullable(valueAt(json, "registrationTime")))
+            res.registrationTime = getInteger(*rawRegistrationTime);
 
     if (json.contains("ultimate"))
         res.ultimate = getBoolean(valueAt(json, "ultimate"));
 
     if (json.contains("signatures"))
-        res.sigs = valueAt(json, "signatures");
+        res.sigs = getStringSet(valueAt(json, "signatures"));
 
     return res;
 }

src/libutil/json-utils.cc

@@ -39,12 +39,9 @@ std::optional<nlohmann::json> optionalValueAt(const nlohmann::json::object_t & m
 }
 
 
-std::optional<nlohmann::json> getNullable(const nlohmann::json & value)
+const nlohmann::json * getNullable(const nlohmann::json & value)
 {
-    if (value.is_null())
-        return std::nullopt;
-
-    return value.get<nlohmann::json>();
+    return value.is_null() ? nullptr : &value;
 }
 
 /**

src/libutil/json-utils.hh

@@ -29,7 +29,7 @@ std::optional<nlohmann::json> optionalValueAt(const nlohmann::json::object_t & v
  * Downcast the json object, failing with a nice error if the conversion fails.
  * See https://json.nlohmann.me/features/types/
  */
-std::optional<nlohmann::json> getNullable(const nlohmann::json & value);
+const nlohmann::json * getNullable(const nlohmann::json & value);
 const nlohmann::json::object_t & getObject(const nlohmann::json & value);
 const nlohmann::json::array_t & getArray(const nlohmann::json & value);
 const nlohmann::json::string_t & getString(const nlohmann::json & value);

tests/functional/signing.sh

@@ -15,9 +15,9 @@ outPath=$(nix-build dependencies.nix --no-out-link --secret-key-files "$TEST_ROO
 
 # Verify that the path got signed.
 info=$(nix path-info --json $outPath)
-[[ $info =~ '"ultimate":true' ]]
-[[ $info =~ 'cache1.example.org' ]]
-[[ $info =~ 'cache2.example.org' ]]
+echo $info | jq -e '.[] | .ultimate == true'
+echo $info | jq -e '.[] | .signatures.[] | select(startswith("cache1.example.org"))'
+echo $info | jq -e '.[] | .signatures.[] | select(startswith("cache2.example.org"))'
 
 # Test "nix store verify".
 nix store verify -r $outPath
@@ -39,8 +39,8 @@ nix store verify -r $outPath
 
 # Verify that the path did not get signed but does have the ultimate bit.
 info=$(nix path-info --json $outPath2)
-[[ $info =~ '"ultimate":true' ]]
-(! [[ $info =~ 'signatures' ]])
+echo $info | jq -e '.[] | .ultimate == true'
+echo $info | jq -e '.[] | .signatures == []'
 
 # Test "nix store verify".
 nix store verify -r $outPath2
@@ -57,7 +57,7 @@ nix store verify -r $outPath2 --sigs-needed 1 --trusted-public-keys $pk1
 # Build something content-addressed.
 outPathCA=$(IMPURE_VAR1=foo IMPURE_VAR2=bar nix-build ./fixed.nix -A good.0 --no-out-link)
 
-[[ $(nix path-info --json $outPathCA) =~ '"ca":"fixed:md5:' ]]
+nix path-info --json $outPathCA | jq -e '.[] | .ca | startswith("fixed:md5:")'
 
 # Content-addressed paths don't need signatures, so they verify
 # regardless of --sigs-needed.
@@ -73,15 +73,15 @@ nix copy --to file://$cacheDir $outPath2
 
 # Verify that signatures got copied.
 info=$(nix path-info --store file://$cacheDir --json $outPath2)
-(! [[ $info =~ '"ultimate":true' ]])
-[[ $info =~ 'cache1.example.org' ]]
-(! [[ $info =~ 'cache2.example.org' ]])
+echo $info | jq -e '.[] | .ultimate == false'
+echo $info | jq -e '.[] | .signatures.[] | select(startswith("cache1.example.org"))'
+echo $info | expect 4 jq -e '.[] | .signatures.[] | select(startswith("cache2.example.org"))'
 
 # Verify that adding a signature to a path in a binary cache works.
 nix store sign --store file://$cacheDir --key-file $TEST_ROOT/sk2 $outPath2
 info=$(nix path-info --store file://$cacheDir --json $outPath2)
-[[ $info =~ 'cache1.example.org' ]]
-[[ $info =~ 'cache2.example.org' ]]
+echo $info | jq -e '.[] | .signatures.[] | select(startswith("cache1.example.org"))'
+echo $info | jq -e '.[] | .signatures.[] | select(startswith("cache2.example.org"))'
 
 # Copying to a diverted store should fail due to a lack of signatures by trusted keys.
 chmod -R u+w $TEST_ROOT/store0 || true

tests/unit/libstore/data/path-info/empty_impure.json

@@ -0,0 +1,10 @@
+{
+  "ca": null,
+  "deriver": null,
+  "narHash": "sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc=",
+  "narSize": 0,
+  "references": [],
+  "registrationTime": null,
+  "signatures": [],
+  "ultimate": false
+}

tests/unit/libstore/data/path-info/empty_pure.json

@@ -0,0 +1,6 @@
+{
+  "ca": null,
+  "narHash": "sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc=",
+  "narSize": 0,
+  "references": []
+}

tests/unit/libstore/path-info.cc

@@ -19,7 +19,15 @@ class PathInfoTest : public CharacterizationTest, public LibStoreTest
     }
 };
 
-static UnkeyedValidPathInfo makePathInfo(const Store & store, bool includeImpureInfo) {
+static UnkeyedValidPathInfo makeEmpty()
+{
+    return {
+        Hash::parseSRI("sha256-FePFYIlMuycIXPZbWi7LGEiMmZSX9FMbaQenWBzm1Sc="),
+    };
+}
+
+static UnkeyedValidPathInfo makeFull(const Store & store, bool includeImpureInfo)
+{
     UnkeyedValidPathInfo info = ValidPathInfo {
         store,
         "foo",
@@ -50,22 +58,21 @@ static UnkeyedValidPathInfo makePathInfo(const Store & store, bool includeImpure
     return info;
 }
 
-#define JSON_TEST(STEM, PURE)                                             \
+#define JSON_TEST(STEM, OBJ, PURE)                                        \
     TEST_F(PathInfoTest, PathInfo_ ## STEM ## _from_json) {               \
         readTest(#STEM, [&](const auto & encoded_) {                      \
             auto encoded = json::parse(encoded_);                         \
             UnkeyedValidPathInfo got = UnkeyedValidPathInfo::fromJSON(    \
                 *store,                                                   \
                 encoded);                                                 \
-            auto expected = makePathInfo(*store, PURE);                   \
+            auto expected = OBJ;                                          \
             ASSERT_EQ(got, expected);                                     \
         });                                                               \
     }                                                                     \
                                                                           \
     TEST_F(PathInfoTest, PathInfo_ ## STEM ## _to_json) {                 \
         writeTest(#STEM, [&]() -> json {                                  \
-            return makePathInfo(*store, PURE)                             \
-                .toJSON(*store, PURE, HashFormat::SRI);                   \
+            return OBJ.toJSON(*store, PURE, HashFormat::SRI);             \
         }, [](const auto & file) {                                        \
             return json::parse(readFile(file));                           \
         }, [](const auto & file, const auto & got) {                      \
@@ -73,7 +80,10 @@ static UnkeyedValidPathInfo makePathInfo(const Store & store, bool includeImpure
         });                                                               \
     }
 
-JSON_TEST(pure, false)
-JSON_TEST(impure, true)
+JSON_TEST(empty_pure, makeEmpty(), false)
+JSON_TEST(empty_impure, makeEmpty(), true)
+
+JSON_TEST(pure, makeFull(*store, false), false)
+JSON_TEST(impure, makeFull(*store, true), true)
 
 }

tests/unit/libutil/json-utils.cc

@@ -175,13 +175,16 @@ TEST(optionalValueAt, empty) {
 TEST(getNullable, null) {
     auto json = R"(null)"_json;
 
-    ASSERT_EQ(getNullable(json), std::nullopt);
+    ASSERT_EQ(getNullable(json), nullptr);
 }
 
 TEST(getNullable, empty) {
     auto json = R"({})"_json;
 
-    ASSERT_EQ(getNullable(json), std::optional { R"({})"_json });
+    auto * p = getNullable(json);
+
+    ASSERT_NE(p, nullptr);
+    ASSERT_EQ(*p, R"({})"_json);
 }
 
 } /* namespace nix */