Changes:
- Add a basic automatic resolution plan for unsatisiable SAT problems
- Fix regression in pkg info -R
- Add a mechanism to finding libs when packaging base
- Fix some warnings on arm
- Fix hardlink extraction with --rootdir
- Improved error reporting
- Update libucl
- Fixes on locked package handling
- Add ip_version in pkg -vv output
- Add !~ in pkg query (negated glob match)
- Implement "vital" packages: prevent a package from being manually removed
- Add support for arm hardfloat
- Multiple bug fixes
Prefetching option fix:
Until now, if the "prefetch suitable packages" option was enabled, it
could happen that "synth force" and "synth test" commands wouldn't do
anything. Moreover, even though it wouldn't build, it would delete the
existing package and refetch it because force/test always pre-deletes
existing packages.
Now, ports listed with "force" and "test" commands are always built,
even if suitable remote packages are available.
Improvements with test mode:
1) Changes to /var/tmp are now ignored. These are legal modifications
2) Access time modification to the builder mount are now ignored. This
was possible if changes to /tmp or /var propagates upward (legally).
3) A new environment variable is added for test command, "LOCK". If it
defined to any value, the builder's localbase will be remounted as
read-only during the configure and build phases. This is a
diagnostic tool to help pinpoint where ports are committing file
system violations by writing to localbase during these phases.
New notice to user regard CONSERVATIVE_UPGRADE:
At the end of the "synth rebuild-repository" command, Synth will probe
the host pkg(8) to see if it's configured with CONSERVATIVE_UPGRADE.
If so, Synth will emit a notice because chances are good that the
user built repository for the local system and may try to upgrade the
system with pkg(8), and in this case, they may wish to turn off the
CONSERVATIVE_UPGRADE setting.
prepare-system, upgrade-system package deletion bug:
These two commands had a serious logic flaw that resulted in newly-built
packages to be immediately deleted prior the pkg(8) repo command. (And
the obvious result of the system not getting updated without re-running
the same command.)
The cause was that the first package validity scan marked the package
for deletion, and this setting was never reset. The obsolete package
was correctly deleted and rebuilt, but then it was quickly deleted
again during the second package scan because the previous deletion mark
was still present. The fix was simple: Reset the deletion marks before
rebuilding the repository so the second scan proceeds accurately.
Test case provided by Wapcaplet in FreeBSD forums
Pkg(8) bug workaround:
The pkg rquery man page is incorrect. It lists the invocation of
pkg rquery as "-U -r reponame" but if it's invoked in that order, it
will squawk errors if any of the repositories are misconfigured. The
command must be "pkg rquery -r reponame -U" in order to limit pkg(8) to
a single repository.
The order sensitivity came into play with "synth upgrade-system" command
on a brand new system. The 00_synth.conf pkg(8) configuration file was
created but there were no repository files created yet, cause pkg(8) to
emit errors on Synth repo when checking the official FreeBSD package
repo. This resulted in no packages prefetching even though "synth status"
indicated some would be fetched. Rearranging the internal rquery
commands works around the misleading pkg(8) documentation (or bug?).
FreeBSD-specific system root validity check.
The $SYSROOT/boot directory doesn't have to be present, but if it is,
there must also be present $SYSROOT/boot/modules directory. Synth now
checks for this on FreeBSD and stops with an informative message when
the modules directory needs to be created by the root user.
This is an enhancement release that contains a bug fix.
Description of bug:
If synth is launched from mountpoint of the ports directory
(e.g. /usr/ports), it may malfunction with strange messages such
as "invalid origins" and printing strange symbols to the screen
along with parts of a directory.
As a result, Synth now detects this launch location (as it already
did with /usr/local) and refuses to run until the current directory
is changed outside of the ports tree.
Enhancement 1:
It is now possible to remove alternate profiles. When more than
one profile exists, a menu option "<" appears that provides the
user with an opportunity to remove one to all inactive profiles
from the configuration. The man page has been updated as well.
Enhancement 2:
Synth will automatically convert any directory inputs on the
configure command to the true path. For example, if somebody types
in "/usr/xports" for the ports directory, but that's just a
symbolic link to /vcs/freebsd-ports", the entry will automatically
convert to its true resolved path (e.g. /vcs/freebsd-ports). This
enables the cwd detection of the bug fix above to always work.
Two minor bug fixes:
* A specific check during test mode would emit a failure to stdout when
testing devel/py-setuptools27. It turns out that there's a file there
with a space in the filename. The filename was an argument for
/usr/bin//file and it wasn't escaped. The file in question had
parentheses too which the shell was trying to process. The fix was
to escape the filename in the /usr/bin/file command.
* The builders were mounting the source directory from "/usr/src", not
$sysroot/usr/src as intended. This potentially causes breakage when
the $sysroot reflects a different versions/release than the host
machine has (e.g. making FreeBSD 10.2 packages on FreeBSD 11-current).
Now the source directory mount is relative to profile's $sysroot.
With portest, you can create, update, patch and revert a port tree with
svn, git and portsnap. Portest can also do portlint testing, test a port
with port test, or poudriere, and also generate a list of ports a
patchfile(s) will modify. Portest is written in shell with minimal
dependencies mostly using FreeBSD-base
WWW: https://github.com/Ultima1252/portest
PR: 207810
Submitted by: Ultima1252@gmail.com
Fix distinfo for the offending ports.
lang/yorick's tag was moved, and the added patch was no longer needed.
PR: 207644
Submitted by: mat
Exp-run by by: antoine
Sponsored by: Absolight
Differential Revision: https://reviews.freebsd.org/D4268
I knew the repository signing enhancements weren't fully tested, but it
was important to get the regressions fixed. I thought that if there were
any problems with the signing, people not using those features (e.g. 99%
of users) would not be affected. Unfortunately, that's not true.
Due to a simple boolean logic error, an inaccurate and harmless warning
about not finding the signature fingerprint will show every time the
repository is rebuild. This minor releases fixes that logic.
This was meant to be a pure enhancement release to support the signing
of the repository, but there were some regressions brought in with
version 1.1x and also some long standing bugs found.
Fixed:
* Regression: Interactive test mode
Changing the TERM setting to "dumb" semi-broke the interactive shell
when ENTERAFTER is used with the test command. The fix was to set
TERM back to cons25 in this particular case.
* Regression (serious): install command
The improvement in speed of package validation for the rebuilding of
the repository resulted in a new bug that caused the "synth install"
command to install every port in the repository. It was caused by
reusing a container that I thought was available, but it turns out
that the "install" command still needed it.
* Bug: handle exception that occurred when Synth asked a question
when no TTY was attached. Running a command like "force" would result
in a question, and if Synth was running in a cron job or having args
piped to it via xargs, no terminal would be available and Synth would
throw an exception. Now in this case, Synth just assumes answer to
question is "No" and ends gracefully.
* Bug: purge-distfiles command would delete everything if any error was
encountered. It should have just stopped, which it does now.
* Bug: purge-distfiles container had a bug which prevented all the
distfiles from being stored, which caused removal of good distfiles.
* Bug: purge-distfiles container didn't check for duplicates, resulting
in unhandled exceptions. Now fixed, purge-distfiles command works now.
enhancements:
* Show version information on status commands. On the display, a symbol
of "N" (New), "R" (result), or "U" (upgrade) shows next to each line.
Additionaly for "U", the old version and new version are shown. The
same information is shown in the log, but in addition the actual
version for "N" and "R" categories is also listed.
* Test mode: Add filters from poudriere to prevent false test failures
with leftover and modified files. Mostly it involves files that get
modified, but it's okay so the MD5 check failures need ignoring
* Support signing repository with local RSA keys. Adjust man page with
two new FILES (public key and private key) along with an example on
how to generate those files.
* Support signing repository with external signing command and
fingerprints. Adjust man page with two more new FILES and an example
of how configure this feature.
This fixes a regression in building ports that have dependences that
install kernel modules. When DTrace support was added by providing a
read-only mount of /boot to the builder, the kernel modules could no
longer be installed at /boot/modules by pkg(8).
Previously, although successful, module installs would have caused a file
system violation on test mode checks. Since /boot is now excluded from
checks (since DTrace support), leftovers in /boot/modules will not be
detected in test mode. The fix is too elaborate and FreeBSD-specific
to worry about (plus there's the philosophy question about why the ports
framework is even allowed to modify the base but that's out of scope).
This release addresses unacceptably long repository rebuild times for the
worst cases (FreeBSD [1], slow CPU, slow mechanical disk). Until now,
rebuilding the repository required a full tree scan (nearly 26k ports).
While I only saw around 4 minutes on a 4-year DragonFly machine equipped
with a SSD, others reported times exceeding 20 minutes.
This new method scans existing packages twice -- first to eliminate those
packages where the port was removed and also those with a mismatching
version (parallel). This sets up a second pass to serially and
recursively scan the ports of the remaining packages. That leads to the
final validation (same as previously done) and the actual repository
generation. Now the repository generation time is much shorter, but
corresponds to the number of build packages in the packages directory.
The long repository generation times were identified prior to the 1.0
release, but I targetted 1.1 for the enhancement.
bug fixes:
* cron operation fixed
The terminal is unset for cron jobs. The setting of TERM to "cons25"
in the environment caused a failure, but it's fixed by changing the
value to "dumb". Note that curses must be off in the profile invoked
by cron.
* dtrace building fixed [1]
On FreeBSD, /boot is now read-only null-mounted if it exists in the
profile's system root. This fixes building dtrace probes when the
option is set.
* fix muted pipe exit status
Pipes have to be read even when no output is expected in order to get
the correct exit status
* fix pipe child exit status
The previous implementation mixed child and parent exit statuses in
error; now they are isolated and returned correctly.
Enhancement:
If a muted pipe command fails, an appropriate error message is given
including which command failed. This helps identify the real error
instead of repercussions appearing later which mask the problem.
PR: 207361 [1]
Currently, the application is a MVP (Minimum Viable Product) which
provides only the following features: packages listing and searching,
installation and deinstallation of packages, and filtering based on
the package state.
WWW: https://github.com/culot/portal
Any attempt to alter echo and control flags when the standard input
stream is not connected to a tty will fail. Fix the resulting nuisance
messages by skipping these steps if a tty is not present.
Reported by kmoore@ on jenkins
The echo and control-flow flags weren't being restored correctly in
general, and they weren't being restored at program exit. The result
on PuTTY was that nothing would echo to the terminal after Synth
exited, but the new version fixes it.
I'm pleased to finally issue the first release of Synth. There were some
significant changes since the last release candidate:
* Rework piped command handling to fix command parsing
Synth was using ada-util to execute external commands and receive the
output. Unfortunately, this implementation currently can not handle
either extra whitespace nor can it support quoted values. The author
is going to fix this soon using popen and pclose for Unix, but I just
implemented the functionality by binding to those functions myself.
* This was the only use of ada-util (I originally expected to use more
of this library's functionality) so it's been dropped as a dependency.
This reduces the stripped executable size by about 30%.
* The environment was prepopulated with UNAME_[x] variables. This is
required for Synth to support building packages for different releases
or even architectures (e.g. building 10.2 packages on 11.0-CURRENT).
* Define OSREL in the builder's /etc/make.conf. This seemed to be
required for emulators/virtio-kmod which uses <pre> although it's not
clear defining OSREL is necessary.
* Fix graceful shutdown handling in text mode.
When the graceful shutdown key was changed from Escape to Control-Q, it
stopped working in text mode (i.e. when it wasn't in curses). I believe
that's because the curses display uses the "raw" tty mode and lets all
control characters through. In text mode, Control-Q has a flow control
functionality. I wrote a routine to disable that flow control and give
Synth access to control-Q keypress in all cases.
* The same routine disables TTY echoes when they are unwanted
* Fix "status-everything" dry-run issue where a graceful shutdown resulted
in an unwanted "sorry" message.
* In test mode file system violation check, mark changes to /var/run
during building to be excluded from checks.
* Add a guard that checks if Synth is launched when the current directory
is <sysroot>/usr/local (or a subdirectory of it). Synth will fail
to mount everything in this case, so now it stops and tells the user
to change directories and try again.
/usr/local/share/poudriere/MANIFESTS, where poudriere (as of version
3.1.11) checks for pre-distributed MANIFEST files before fetching them
from the ftp/http/https server.
This allows poudriere to ensure that the bits it is downloading and
installing really match the bits provided by the release engineering
team, and have not been subtly trojanned in transit. (Note that this
does not apply if poudriere is creating a jail from -STABLE or -CURRENT
since we cannot pre-distribute those MANIFESTs.)
The MANIFEST files were obtained by
(a) finding the GPG-signed announcements for 9.0 and later releases;
(b) verifying those signatures against the GPG keys in the FreeBSD
documentation repository;
(c) downloading all of the relevant bootonly ISOs;
(d) verifying the ISOs against the hashes listed in the signed release
announcements; and
(e) extracting the MANIFEST files from those ISOs.
Reviewed by: bdrewery
Security: If someone could trick you into building packages in a
world which they tampered with, they could do all sorts
of nasty things to those packages...
Changes:
- Optimize jail startup/cloning/rollback and mtree comparisons.
- api: Make more useful
- check_fs_violation: Use mktemp rather than hard-coded tmpfile
- Fix false-positive 'Leftover processes' warning after disabling jexecd
- Conditionalize the jexec logic around USE_JEXECD and add back missing jkills.
- QEMU: Remove _MAKE_JOBS not present in release-3.1 and noted to break
builds by sbruno
- Kill processes before remounting/rolling back the builder FS, rather than
after.
- jail -c: Support older FREEBSD_HOST=ftp.freebsd.org URL schemes still. [1]
- jail -c: Support pre-distributed MANIFESTs for checksum comparing.
PR: 206848 [1]
Unfortunately, there's been a bit too much change since 0.99_6 to
confidently release version 1.00, so another release candidate is
necessary. Both new features and bug fixes were added.
New features:
* Provide ability to define environment variables in a profile
(/usr/local/etc/synth/<profile>-environment)
* Support fetching by proxy using these environment variables
* Add zsh and bash completion scripts
* Accept port origins with trailing file separators (so people
using completion scripts don't have to backtrack to remove them)
* In text (non-curses) mode, output the current package build
tally every 200 seconds (approximately)
Bug fixes:
* Fix support for system roots that don't match host (e.g.
ARCH, OSRELEASE, OSVERSION, etc
* Fix ABI check for system roots that don't match host
* Remove effect of system /etc/make.conf (originally seen when
MAKE_JOBS_NUMBER was defined there and disabled synth)
Changes:
- Fix hardlinks extraction with --rootdir
- Relax url scheme acceptance to allow file:/ and file://
- Accept pkg+ for url scheme where it was never intended to be used, because users used it anyway
- Remove pkg-message
Torsten has accepted the challenge of whipping portmaster back into
shape. In addition to changing MAINTAINER, he's starting with fixing
both non-default options by providing their necessary RUN_DEPENDS.
PR: 207075
I had hoped that 0.99_5 would be bug-free and the basis for the first
release (1.00), but couple were found. If use of 0.99_6 reveals no
further issues after a week or so, I'll re-release it as v1.00.
bugs fixed:
* if the origin started with a directory separator, an exception would
occur. Now it properly labels it as an invalid origin.
* The "extract" stage was labelled as "checksum". Internally everything
was fine, but on the display, the order was checksum, extract-depends,
checksum instead of checksum, extract-depends, extract.
* During one phase (build), the DEVELOPER flag was set unconditionally.
This was a regression as it wasn't always the case. This code was
tweaked several times since 0.99_5 and now the DEVELOPER setting has
been moved the builder's make.conf to ensure it's consistently
present or absent (as needed).
* It turns out that the ports tree scan is affected by the DEVELOPER
flag. It turns out the setting can affect the dependencies list so
it needs to be set (or absent) appropriately to match how it will be
on the builders. The make.conf solution above solves this too.
* There was "NO_BACKUP" set in the builders make.conf. This line is
for the DragonFly src builder and it's presence caused no harm, but
it's been removed now.
* Make ports makefile respect CFLAGS
new feature:
* Provide ability to break into a build at a specific point and
interact with it.
- Only available on "test" command
- Only active when one (1) port origin is given to "test" command
- Only active when ENTERAFTER is defined in environment as:
> extract
> patch
> configure
> build
> stage
> install
> deinstall
- All dependencies are built first with typical display and
DEVELOPER=1 set. Afterwards, Synth converts to text mode and
builds the specified port up to and including the phase specified
by ENTERAFTER. Then it launches a tcsh shell and gives control
to the user at the builder's root directory
- The user ends the interactive session with the shell cmd "exit"
- Synth will clean up and exit (it will not try to continue the
build due to possible corruption from the users)
* This is a port developers tool. The average user does not need it.
* The average user might use "test" command to generate a log to submit
as a FreeBSD Bugzilla PR attachment.
A fairly recent change caused a regression after a build was complete.
Previously a "tally" or summary of the build would appear after the
ncurses screen was restored to the regular terminal mode. It would
list how many ports were built, failed, etc. After the regressin, it
just ended abruptly.
This commit restores the tally to show as it did previously.
While processing Issue 206746 [1] for a security update to
security/py-rsa (For versions < 3.3), it was noticed that Portscout
had not identified the the newer version, released on 2016-01-13.
Investigation revealed that the PyPI SiteHandler in Portscout only
processed the first url/filename returned by PyPI, which in many cases
is not a tar.gz, the default EXTRACT_SUFFIX for source distribution
(sdist) files:
[py-rsa] VersionCheck()
[py-rsa] Checking site: https://pypi.python.org/packages/source/r/rsa/
Does site handler exist ... Yes
(Portscout::SiteHandler::PyPI) GET https://pypi.python.org/pypi/rsa/json
(Portscout::SiteHandler::PyPI) GET success: 200 Filename: rsa-3.3-py2.py3-none-any.whl
FindNewest: Checking rsa-3.3-py2.py3-none-any.whl ... against port DISTFILES.
FindNewest: Checking DISTFILE ... rsa-3.1.4.tar.gz (ver: 3.1.4, sufx: .tar.gz)
[py-rsa] Done
This change backports a commit [1] made to Portroach which adds a loop to
enumerate all URLs/filenames in the PyPI JSON response, not just the
first.
[1] e93b8331f6
PR: 206746 [1]
Obtained from: https://github.com/jasperla/portroach
This is a minor update to synth, which includes:
* Support for the WHYFAIL environment variable. If this variable
is defined (to any value) in the environment, Synth will turn on
the "debug" mode for dependency and option sanity checks. This
mode will provide exact details on how the package failed the check.
* README.md: editorial corrections, 3 images replaced to reflect current
version of Synth
* Man page: editorial correction, WHYFAIL documented, and the "Impulse"
indicator was documented (in NOTES section)
* Significantly improve ports scan error messages. In particular,
eliminate the 'bad value ""' messages that are caused by ports that
are partially or completely missing. Also propagate exception
messages when helping.
* Log 03 (ignored ports) did not list the actual ports, only the reason
the port was ignored. Fix bug to show category/port too.
Erratum on previous commit message: The "Graceful Shutdown" is initiated
with Control-Q, not Control-C! The typo is doubly unfortunate because
Control-C will exit Synth without cleaning up the mounts.
WARNING: rebuild-repository command has changed action! see below!
The follow changes have been made since v0.99_2:
* Change the graceful shutdown key from "Escape" to Control-C.
The former was easy to hit inadvertently (reported) and could be
interfered with by terminal ANSI codes and/or mouse wheels. The
documentation has also been modified to reflect this change.
* Fixed bug where installed packages that no longer had a port
might cause the scan to fail rather than be ignored as advertised
* New feature: SYNTHPROFILE environment variable
When SYNTHPROFILE is set toTill be loaded rather than the default
profile. This is aimed for synth's use in scripts.
* The "rebuild-repository" command has been renamed to "prepare-system".
This is partly because the former command will be repurposed.
* A new command assumed the name "rebuild-repository"; it performs a
sanity check on all the built packages, removes the bad ones, and
rebuilds the local Synth repository on command. It is primarily for
scripting use, but it has other legitimate uses.
* Fix case where prefetching packages would try to update a non-existent
local Synth directory. As a consequence, prefetching is only done
from a single external repository (the normal use case thought)
1) Fixed false "fetched failed" messages that always appear after
prebuilt packages are fetched
2) Fix bug where "synth configure" command would not run if any directories
were invalid. For new systems, /usr/ports/distfiles is always invalid
3) Following 2), greatly improve error message by saying exactly which
directory is missing and which configuration letter it pertains to
4) If synth is configured to a non-existent /usr/ports/distfiles directory,
also add a recommendation to consider a better location outside of the
ports tree and remind them to set DISTDIR in /etc/make.conf too.
Changes:
- jail -c: Always copy in QEMU emulator, even if not building from src.
- jail -c: Ensure MANIFEST is not empty or give an error.
- Clear CMD_ENV to disallow sneaking in environment vars
- Update libnv to the latest head @ r293361
- Update sh from head @ r293359
- Null-mount read-only /rescue and /nxb-bin into the jails as they will not
change.
- Add a MUTABLE_BASE option (default yes) to experiment with null-mounting of
base.
- Prefer https://download.FreeBSD.org for snapshot downloads from FreeBSD.
- Remove duplicate NM from make.nxb.conf.
I've been ordered by portmgr to remove DEPRECATION designation because
others have indicated they believe people should not be so directly
informed of its poor state. Despite the fact that there was no expiration
date set and that functionality was not affected in any way (leaving now
informed people free to use this unmaintained port), it was considered a
disruptive change.
Let the record show that I strongly object to this decision and that I
firmly believe that portmaster is a port that *must* have a competent
maintainer that can *develop* it. It should *not* be allowed to be
unmaintained and still maintain a presence in FreeBSD documentation.
1) When using prefetch option, list the packages that failed to download
rather than just say, "at least one failed to download"
2) sysutils/htop requires linprocfs but doesn't set USE_LINUX. Set this
port to mount linprocfs based on its origin
3) Fix linprocfs implementation, it was mounting out of order, basically
resulting in that it was non-functional
4) Close all the logs in the case where no packages are built. In that
case, the logs were never modified. Changes discarded?
Changes:
- Allow to limit the valid url scheme in pkg.conf
- Add support for SOURCE_DATE_EPOCH: https://reproducible-builds.org/specs/source-date-epoch/
- Fix pkg check -d
- Fix testsuite with kyua 0.12
- Fix completion on pkg info -l
- Reduce te probability to get an infinite loop on pkg version check
- Fix pkg version -q
- Update libfetch to the latest version from head
- Debug mode is now verbose when fetching
- Fix segfault with empty HTTP_USER_AGENT
- Fix segfault when parsing invalid URL
- Fix build on OS X
- Better error reporint on issue during pkg create
Note that this release will be merged to quarterly branch in 10 days
