
As initially designed, `lib.packagesFromDirectoryRecursive` allowed passing a string for the `directory` argument. This is necessary for several reasons: - `outPath` on derivations and Flake inputs is not a path. - Derivations can be coerced to their `outPath` in string interpolation, but that produces strings, not paths. - `builtins.path`, bizarrely, returns a string instead of a path (not that the documentation makes this clear). If a path is used instead of a string here, then Nix will dutifully copy the entire directory into a new path in the Nix store (ignored as WONTFIX by Eelco in https://github.com/NixOS/nix/issues/9428). For industrial use cases, this can result in an extra 10-15 seconds on every single eval just to copy files from one spot in the Nix store to another spot in the Nix store. In #361424, this was changed so that `directory` must be a path, breaking these use-cases. I'm not really sure what happened here -- #361424 has very little justification for why it exists, only a reference to a previous version of the PR (#359941), which itself had very little justification given. The description on #359941 explained that it would "Shrink the function's code by ~2/3rd 🎉", but 60% of the reduction in size was just deleting comments (!) and bindings like `directoryEntryIsPackage` that helped clarify the intent of the implementation. As a result, the new implementation is (to my eyes) more challenging to read and understand. I think the whole thing was in service of #392800, which adds a `newScope` argument in order "to create nested scopes for each (sub)directory (not just the top-level one) when `newScope` is given." Nobody noticed this regression until after the commit was merged. After @phanirithvij pointed out the regression, @nbraud said they would "shortly prepare a PR to fix this" [1] but did not. Later, they would explain that they were "quite ill the last month(s)" [2], which explains why this got forgotten about. @nbraud also requested a review from @Gabriella439 [3], as she had reviewed the original PR adding `lib.packagesFromDirectoryRecursive`, but not from me, the original author of that PR. @Gabriella439 did not review the "refactor" PR, and no attempt to contact her or myself was made after that initial request. This behavior is admittedly rather subtle, so I'm not sure either Gabriella or myself would have noticed the change (especially since the relevant PR restructures the entire implementation). While I find this a bit frustrating, I should have added a test for this use-case in my original PR; if there was a test that relied on passing paths in as a string, perhaps the authors modifying this code would have noticed that the implementation was not an accident. [1]: https://github.com/NixOS/nixpkgs/pull/361424#discussion_r1912407693 [2]: https://github.com/NixOS/nixpkgs/pull/359984#issuecomment-2775768808 [3]: https://github.com/NixOS/nixpkgs/pull/361424#issuecomment-2521308983
Nixpkgs lib
This directory contains the implementation, documentation and tests for the Nixpkgs lib
library.
Overview
The evaluation entry point for lib
is default.nix
.
This file evaluates to an attribute set containing two separate kinds of attributes:
-
Sub-libraries: Attribute sets grouping together similar functionality. Each sub-library is defined in a separate file usually matching its attribute name.
Example:
lib.lists
is a sub-library containing list-related functionality such aslib.lists.take
andlib.lists.imap0
. These are defined in the filelists.nix
. -
Aliases: Attributes that point to an attribute of the same name in some sub-library.
Example:
lib.take
is an alias forlib.lists.take
.
Most files in this directory are definitions of sub-libraries, but there are a few others:
minver.nix
: A string of the minimum version of Nix that is required to evaluate Nixpkgs.tests
: Tests, see Running testsrelease.nix
: A derivation aggregating all testsmisc.nix
: Evaluation unit tests for most sub-libraries*.sh
: Bash scripts that run tests for specific sub-libraries- All other files in this directory exist to support the tests
systems
: Thelib.systems
sub-library, structured into a directory instead of a file due to its complexitypath
: Thelib.path
sub-library, which includes tests as well as a document describing the design goals oflib.path
- All other files in this directory are sub-libraries
Module system
The module system spans multiple sub-libraries:
modules.nix
:lib.modules
for the core functions and anything not relating to option definitionsoptions.nix
:lib.options
for anything relating to option definitionstypes.nix
:lib.types
for module system types
PR Guidelines
Follow these guidelines for proposing a change to the interface of lib
.
Provide a Motivation
Clearly describe why the change is necessary and its use cases.
Make sure that the change benefits the user more than the added mental effort of looking it up and keeping track of its definition. If the same can reasonably be done with the existing interface, consider just updating the documentation with more examples and links. This is also known as the Fairbairn Threshold.
Through this principle we avoid the human cost of duplicated functionality in an overly large library.
Make one PR for each change
Don't have multiple changes in one PR, instead split it up into multiple ones.
This keeps the conversation focused and has a higher chance of getting merged.
Name the interface appropriately
When introducing new names to the interface, such as new function, or new function attributes, make sure to name it appropriately.
Names should be self-explanatory and consistent with the rest of lib
.
If there's no obvious best name, include the alternatives you considered.
Write documentation
Update the reference documentation to reflect the change.
Be generous with links to related functionality.
Write tests
Add good test coverage for the change, including:
-
Tests for edge cases, such as empty values or lists.
-
Tests for tricky inputs, such as a string with string context or a path that doesn't exist.
-
Test all code paths, such as
if-then-else
branches and returned attributes. -
If the tests for the sub-library are written in bash, test messages of custom errors, such as
throw
orabortMsg
,At the time this is only not necessary for sub-libraries tested with
tests/misc.nix
.
See running tests for more details on the test suites.
Write tidy code
Name variables well, even if they're internal. The code should be as self-explanatory as possible. Be generous with code comments when appropriate.
As a baseline, follow the Nixpkgs code conventions.
Write efficient code
Nix generally does not have free abstractions. Be aware that seemingly straightforward changes can cause more allocations and a decrease in performance. That said, don't optimise prematurely, especially in new code.
Reference documentation
Reference documentation for library functions is written above each function as a multi-line comment. These comments are processed using nixdoc and rendered in the Nixpkgs manual. The nixdoc README describes the comment format.
See doc/README.md for how to build the manual.
Running tests
All library tests can be run by building the derivation in tests/release.nix
:
nix-build tests/release.nix
Some commands for quicker iteration over parts of the test suite are also available:
# Run all evaluation unit tests in tests/misc.nix
# if the resulting list is empty, all tests passed
nix-instantiate --eval --strict tests/misc.nix
# Run the module system tests
tests/modules.sh
# Run the lib.sources tests
tests/sources.sh
# Run the lib.filesystem tests
tests/filesystem.sh
# Run the lib.path property tests
path/tests/prop.sh
# Run the lib.fileset tests
fileset/tests.sh
Commit conventions
-
Make sure you read about the commit conventions common to Nixpkgs as a whole.
-
Format the commit messages in the following way:
lib.(section): (init | add additional argument | refactor | etc) (Motivation for change. Additional information.)
Examples:
-
lib.getExe': check arguments
-
lib.fileset: Add an additional argument in the design docs
Closes #264537
-