1) Create a platform-specific (FreeBSD/DF, NetBSD, Linux, Solaris) version
of /etc/shells for the builders. This resolves installation errors on
packages that manipulate /etc/shells.
2) Give in and finally mount /usr/games if available. This is a
performance impact ultimately caused by removing the strfile
application from ports with the faulty logic that the 400-line C
program can't exist because it's in a non-standard location in base.
The alternative is that Synth can't build any applications that
require strfile from games, e.g. all fortune ports. Thanks! <sarcasm/>
This version blocks TTY signals to and from background processes, allowing
lang/sbcl to build as well as avoiding disruption of an active ncurses
display by "you've got mail" message.
It also includes a minor duration time formatting fix on build logs and
the many-thousands-open-file-descriptor problem fix on purge-distfiles
command provided by 1.66_1 between releases.
There is also a change in the build order algorithm. Previously the
categories were ordered alphabetically, but rather scanned in the order
that appears by walking the directory. Now when the "nothing depends on
these ports" ports block starts building, it's in A-Z order by category/
portname whereas before ony the portname was ordered A-Z (the categories
were blocks in random order). There is no real benefit to the reordering;
it's mainly to satisfy those bothered by disorder.
The purge-distfiles command reads the distinfo file of every port in the
tree, but unfortunately never closed the file descriptor when the scan
was complete. This resulted in 26,000+ open file descriptors, possibly
exceeding the host server's FD limit. (issue #70)
The primary reason for the creation of v1.66 is to provide support for
pkgsrc on FreeBSD, DragonFly, NetBSD and initial support for Linux and
Solaris. It represents over 100 commits and was successful. Along the
way, some improvements were made for the ports version. A (likely
incomplete) list includes:
- Improve watchdog kill message by including inactive time for trigger
- Add SYNTHPROFILE variable and value to builders' mk.conf which can be
used by custom local makefiles
- Force users to define TERM in environment if it is not present
- Ensure fetch(1) doesn't check for valid SSL certificates. Since all
downloads are hash-checked for authenticity, misconfigured SSL of a
webserver is not a good reason to break fetching.
- New 6th log which documents why existing packages are obsolete (same
as setting WHYFAIL in env.) "06_obsolete_packages.log"
- New SELFTEST environment command for test mode which forces new "test"
phase to execute (for ports with regression tests)
- Updated man page
Skip new "base" category during ports scanning
The new base category was never meant to be part of the build so the
category makefile was intentionally left off so poudriere and friends
would skip it.
Unfortunately, this idea didn't account for Synth which has a different
behavior: If no category makefile exists, *ALL* subdirectories are
scanned instead of none of them. (DPorts doesn't use category Makefiles
because by definition all ports are valid; there are no "unhooked" ones)
The proper solution would be to include a practically blank
ports/head/base/Makefile to maintain consistency with the rest of
the ports categories, but synth now actively excludes the base category.
Change name of 05 log
The command output log name has changed from "05_abnormal_cmd.out" to
"05_abnormal_command_output.log". The former tended to be considered
a binary file by webservers and would download rather than display.
Final tally added to end of main 00_last_results.log file
At the end of a bulk run, the final tally (queue size, built, failed,
skipped, ignored) would be displayed in curses mode. This change adds
the same useful information to the end the main bulk run log.
Change in hook behavior:
Now the initial building of pkg(8) triggers a success or failure hook
after building. Before ports-mgmt/pkg was the only port that could
never trigger these hooks. The "bulk run start" hook is triggered
afterwords as pkg(8) is a prerequisite to preparing the bulk run.
Fix potential bad unmount of /usr/src
In the case where /usr/src exists, but the profiles uses a system root
that is not "/" and doesn't have an "usr/src" subdirectory, an error
would be emitted as the unmount of the non-existant mount failed. This
uncommon use case has been fixed.
Curses display builder "Elapsed" label changed to "Duration"
This matches the format of the web-based report
DragonFly only: Support File(1) v2.0 version parsing
The output of file(1) version on DragonFly will change with version 2.0.
For example, what is now version 4.0.702 will be displayed as 4.7.2 on
newer versions of file(1). Moreover, the parsing for DragonFly 4.10+
would not have worked. Now both versions are supports and the double-
digit minor versions are properly handled as well.
Fix web report regression caused by cleaning of history files.
The history files were being removed too late. Any ignored/skipped
files erased after writing (but would return with first built/failed
port). If more than 500 ports were ignored+skipped, the web report
would fail to show the history completely and would not recover. The
old history files are now removed before the ignore/skip cascades.
Cosmetic: curses display
Capitalize "Swap" and "Skipped" field labels
Cosmetic: web report
After the build completes, fade out the builders table in 2.5 seconds.
Nobody needs to see a table showing all the builders in "shutdown" mode.
Bandwidth: json files
A history.json file holds up to 500 log entries. 16 bytes of spaces were
removed which reduces the history file size by 8000 bytes. Spaces were
also removed from summary.json which can account for 20%+ of the size.
Only rewrite history files upon change:
The last history.json file was being written/rewritten every 3 seconds
regardless if there were changes to it or not. Now it's only written
if it's changed since the previous write. This might help reduce errors
seen by in nginx access log which are thought to occur when file changes
as the file is being served.
Address "[N] Fetch prebuilt packages" bug:
Once the /usr/local/etc/pkg/repos/00_synth.conf file was created, there
was a good change that fetching prebuilt packages would stop with a
fetch error that the checksum didn't match. This seems to be caused by
pkg(8) using both the FreeBSD and the Synth repositories and often
deferring to the latter. This is incorrect behavior as only the FreeBSD
repository should be used for prefetching. This issue is hopefully
resolved now.
Web report behavior change:
Any existing ??_history.json files are now deleted at the start of
each run. They were only litter as the javascript did not pull in old
files, so this change just removes them.
While here, rename the "Elapsed" column on the builders table to
"Duration" to match the history table lable and also adjust the widths
of the origin and information columns on the history table.
Major new feature: Web-based front-end
A dynamic html report is now generated for each build. The report is
placed at <profile log directory>/Report/index.html.
The report is updated 10 times per minute. The entire build history
is retained and is searchable, but that history is reset at the start
of the next run.
In addition, several fields are clickable and trigger a quick-filter
on the history log including the "Built", "Failed", "Ignored",
"Skipped", "No." column and "ID" column (see tiptool for action hint)
and the "Total" field clears the search pattern quickly.
An earlier but representative version of the web report can be seen at:
https://github.com/jrmarino/synth#web-interface
The Synth web report pulls 2 orders of magnitude less data than the
equivalent poudriere report because the json data is split into
multiple files limited to 500 ports each (e.g. the poudriere data file
might be 1.8 Mb on a big bulk run polled every 8 seconds where as
Synth only pulls the latest segment file (50kb avg) every 6 seconds.
Synth also stops polling when the run is complete.
Minor enhancement: The log files were cleaned up
The data/phase headers and footers were modified so that the build
log appears a lot cleaner now (subjective of course).
It turns out that the C source files were getting built with the headers
from the base ncurses. The recent update in Synth's display exposed a
resulting problem with the add chtype routines, resulting in garbled
output. Since adacurses is a static library, synth requires a revbump.
Handles remaining resizing exceptions and improves display handling.
Yesterday's work handled most of the common display exceptions, but others
were still possible. Now all possible exceptions are handled.
Several improvements were made to the display:
1) lines no longer wrap if the size width is resized too narrow; they
get truncated as always intended
2) Elements such as the elapse timer don't get displayed in the wrong
place when the screen is too narrow (they just don't show)
3) The dashes now get restored if the screen is sized small and then
big again (or started small and then expanded). In many cases those
lines just never came back before.
4) The "full" refresh frequency was increased a period of 30 seconds to
a period of 4 seconds. This has a side benefit to text-mode watchdog
as well since that's the same timer for the log inspection.
5) The history window height ranges from 10 to 50 rows. If the xterm
window starts small, the history will be 10 lines. If it starts
big, the number of lines will be dictated by the original size of
the xterm window. Making the screen small and then bigger again will
reveal the full number of log lines.
Major bug fix: ncurses display resize hang fixed
Until now, resizing the window why synth is running in ncurses mode
caused synth to hang (it would finish the builds it was working on
but the display wouldn't update and no new jobs would start). This
was due to an unhandled exception thown by ncurses binding as a result
of the resize event, and now these are handled.
Minor fix: Ports with @info in pkg-plist now pass in test mode
The mtree exclusion file was improved to allow these leftover info
directories to be ignored (as is done in poudriere. Before only
info/dir was ignored, but the presence of "dir" prevented "info" from
being removed by pkg(8) upon deinstallation.
enhancement: Augment text mode (requested)
Now when a builder starts on a new package, the port origin will be
shown in the running log (before only the completion was logged.)
Fix regression in text-mode caused by activation of watchdog.
The watchdog is checking the lengths of the build logs to figure out if
a builder has stalled. It turns out that the logs were only being
inspected in ncurses display mode, so any port that took longer than
20 minutes to build would be aborted by the watchdog.
While here, bump the *BASE* time limit for the build phase from 20 to
25 minutes based on extreme causes (normally involving gcc or tex ports)
and also bump the check-plist phase limit from 3 minutes to 10 minutes.
Some ports have tens of thousands of files in them which takes a long
time to check under test mode, especially if the server is loaded.
This release changes the watchdog time limits from a fixed value to
one that is affected by the average load. Using fixed values didn't
work well on extremely (and intentionally) loaded servers, e.g. an
overage load of 200 on an server with ncpu of 32.
The base times which were long to account for loading have been reduced,
and a multiplier of average-5 load / ncpu rounded to the nearest tenth
has been implemented (with 1.0 as the floor value). For example, a 4-core
machine with an average 5-minute load of 6.0 will multiply the base
timeout by 1.5. This approach worked well in an extreme test against the
entire ports tree.
This release improves robustness and activates the watchdog.
It leveraes the procctl functionality to ensure all processes spawned
from a builder are reaped, which in turn ensures that tmpfs mounts can
be dismounted. Previously stuck processes could prevent those dismounts,
trapping them as new mounts get placed on top.
This also finally enables the watchdog that will kill runaway builds.
The watchdog has a specific time limit per build phase where it will
kill the build if the log doesn't grow over the previous X minutes.
No activity timeout limits per phase are:
check_sanity : 1 minute
pkg_depends : 3 minutes
fetch : 480 minutes
checksum : 480 minutes (fetches if required)
extract_depends : 3 minutes
extract : 30 minutes
patch_depends : 3 minutes
patch : 3 minutes
build_depends : 5 minutes
build : 20 minutes
run_depends : 10 minutes
stage : 20 minutes
check_plist : 3 minutes
pkg_package : 120 minutes
install_mtree : 3 minutes
install : 10 minutes
deinstall : 10 minutes
A minor change regarding the swap display: If there is no swap installed,
it will now display "n/a" instead of "100%"
This is a minor bug fix version. Changes include:
* Set close-on-exec operation mode on popen. This is required to stop
leaking file descriptors in highly concurrent modes (e.g. 32 builders).
It's not supported on FreeBSD 9 or DragonFly 4.4 (and earlier) so this
modification is removed for those platforms.
* Remove procfs mount for lang/rust and lang/rust-nightly. This was
necessary for DragonFly, but rust has been fixed for DF 4.6 and later
* Add a stage-QA exception for entries left /var/spool/*. The presence
of these are not an identication of a port issue
* Sanitize synthexec to verify file descriptors 0 .. 2 are in use and
automatically close any file descriptor > 2 before execv fork.
At one point during development of last changeset, the default.gpr
file needed to have a LOCALBASE correction, but that's no longer true.
This line shouldn't have made it into the previous commit.
This is a bug fix version. Changes include:
* Fix potential exception seen with a large number of builders. The
height of the log area was determined by subtracting the height of
the other two zones from the overall xterm height. This allowed for
the possibility of a negative height for the log area resulting in a
thrown exception. Now the log is defined to be at least 10 lines
log, but ncurses will just truncate as necessary.
* Detection of bad OPSYS value added. There's a rare case where pkg
can corrupt the configuration value of the Operating System. If it
happens on FreeBSD, Synth thinks it is running on DragonFly which
causes all sorts of problems. The corruption can't be auto-fixed,
but Synth will gracefully abort with instructions on how to correct
the issue.
* Fix filtering of port-mgmt/pkg in list of ports. Synth was designed
to always skip building pkg and thus remove it from a given list of
ports, but the filtering didn't work correctly resulting in an
exception with a misleading message.
* Redirect failed jail setup command output to a new log:
/var/log/synth/05_abnormal_cmd.out. Under normal operation, the
log will be empty, but any abnormal output will now be sent there
rather than corrupt the ncurses display as it before now.
* Fix mount cleanup procedure. The output of "df -h" was being
sorted alphabetically which caused mounts-on-mounts (an exception
condition that dillon has been getting with H2 testing) to fail
during dismount. The df output is already in the correct order so
it only needs to be iterated in reverse without the pre-sort.
Change:
* Synth now links curses statically so it should not pull in any
additional packages when the Synth binary package is installed.
The only known bug is the curses display will freeze if running in an
xterm window that gets resized. The cause is still unknown because in
theory Synth should correctly react to a window-resize event. Some
people may have success with ^z followed by fg command, but it's best
to avoid resizing the window at all.
This release adds detection and handling of circular dependencies.
Two users that tried replacing OpenSSL with LibreSSL found out that
ftp/curl depends on itself when its built with GNUTLS. That's because
Synth goes into an infinity dependency loop, runs out of stack space,
and crashes.
Now Synth can detect a circular dependency and abort gracefully and
informatively, e.g.
> synth status ftp/curl
ftp/curl scan aborted because a circular dependency on ftp/curl was detected.
... backtrace devel/cmake
... backtrace emulators/tpm-emulator
... backtrace security/trousers
... backtrace security/gnutls
... backtrace ftp/curl
While here, build synth with stack-checking on so out-of-stack situations
result in an exception and not an illegal instruction crash.
This release brings a behavioral change to Synth. The action of building
package repositories is much more conservative now; packages that would
fail sanity checks are only removed under the following commands:
* synth prepare-system
* synth upgrade-system
* synth rebuild-repository
* synth everything
That means the following commands will never result in built packages
being removed now:
* synth install
* synth build (followed by "yes" to rebuild repo question)
* synth force (followed by "yes" to rebuild repo question)
Other changes/fixes:
* The PORTSCAN.SEEK_FAILURE exception is usually a false positive and
should be fixed now
* Under test mode leftover check, all changes to /var/db between
installation and deinstallation are ignored rather than being flagged
as a failure (see databases/mysql56-server test failure)
* Add support for DragonFly's /boot/modules.local (similar to FreeBSD's
/boot/modules)
* Bring in ports patch to use installed libraries instead of the bundled
versions
Now that the Ada Framework is based on GCC6 by default, it is no
longer required for Synth to bundle and build a local version of
devel/adacurses and misc/ini_file_manager.
Switch the port to normal BUILD_DEPENDS and remove the bundle
distfile information. No bump is required because these libraries
are statically linked into Synth, so there's no practical difference
between older and newer packages.
* Incorporate prefetch partial-revert patch (1.33_1) and remove the
patch and files directory.
* Add lang/rust and lang/rust-nightly to the list of ports that require
procfs to build. This is only true for DragonFly. The plan is to
eliminate this requirement, but it's a bit tricky right now. Once
done, the rust ports will be removed from the list again.
* Move /var/db/fontconfig directory exception from post-build to all
stages. The doxygen port has shown that /var/db/fontconfig changes
during the build phase and this is legitimate.
Prefetching option fix:
Until now, if the "prefetch suitable packages" option was enabled, it
could happen that "synth force" and "synth test" commands wouldn't do
anything. Moreover, even though it wouldn't build, it would delete the
existing package and refetch it because force/test always pre-deletes
existing packages.
Now, ports listed with "force" and "test" commands are always built,
even if suitable remote packages are available.
Improvements with test mode:
1) Changes to /var/tmp are now ignored. These are legal modifications
2) Access time modification to the builder mount are now ignored. This
was possible if changes to /tmp or /var propagates upward (legally).
3) A new environment variable is added for test command, "LOCK". If it
defined to any value, the builder's localbase will be remounted as
read-only during the configure and build phases. This is a
diagnostic tool to help pinpoint where ports are committing file
system violations by writing to localbase during these phases.
New notice to user regard CONSERVATIVE_UPGRADE:
At the end of the "synth rebuild-repository" command, Synth will probe
the host pkg(8) to see if it's configured with CONSERVATIVE_UPGRADE.
If so, Synth will emit a notice because chances are good that the
user built repository for the local system and may try to upgrade the
system with pkg(8), and in this case, they may wish to turn off the
CONSERVATIVE_UPGRADE setting.
prepare-system, upgrade-system package deletion bug:
These two commands had a serious logic flaw that resulted in newly-built
packages to be immediately deleted prior the pkg(8) repo command. (And
the obvious result of the system not getting updated without re-running
the same command.)
The cause was that the first package validity scan marked the package
for deletion, and this setting was never reset. The obsolete package
was correctly deleted and rebuilt, but then it was quickly deleted
again during the second package scan because the previous deletion mark
was still present. The fix was simple: Reset the deletion marks before
rebuilding the repository so the second scan proceeds accurately.
Test case provided by Wapcaplet in FreeBSD forums
Pkg(8) bug workaround:
The pkg rquery man page is incorrect. It lists the invocation of
pkg rquery as "-U -r reponame" but if it's invoked in that order, it
will squawk errors if any of the repositories are misconfigured. The
command must be "pkg rquery -r reponame -U" in order to limit pkg(8) to
a single repository.
The order sensitivity came into play with "synth upgrade-system" command
on a brand new system. The 00_synth.conf pkg(8) configuration file was
created but there were no repository files created yet, cause pkg(8) to
emit errors on Synth repo when checking the official FreeBSD package
repo. This resulted in no packages prefetching even though "synth status"
indicated some would be fetched. Rearranging the internal rquery
commands works around the misleading pkg(8) documentation (or bug?).
FreeBSD-specific system root validity check.
The $SYSROOT/boot directory doesn't have to be present, but if it is,
there must also be present $SYSROOT/boot/modules directory. Synth now
checks for this on FreeBSD and stops with an informative message when
the modules directory needs to be created by the root user.
This is an enhancement release that contains a bug fix.
Description of bug:
If synth is launched from mountpoint of the ports directory
(e.g. /usr/ports), it may malfunction with strange messages such
as "invalid origins" and printing strange symbols to the screen
along with parts of a directory.
As a result, Synth now detects this launch location (as it already
did with /usr/local) and refuses to run until the current directory
is changed outside of the ports tree.
Enhancement 1:
It is now possible to remove alternate profiles. When more than
one profile exists, a menu option "<" appears that provides the
user with an opportunity to remove one to all inactive profiles
from the configuration. The man page has been updated as well.
Enhancement 2:
Synth will automatically convert any directory inputs on the
configure command to the true path. For example, if somebody types
in "/usr/xports" for the ports directory, but that's just a
symbolic link to /vcs/freebsd-ports", the entry will automatically
convert to its true resolved path (e.g. /vcs/freebsd-ports). This
enables the cwd detection of the bug fix above to always work.
Two minor bug fixes:
* A specific check during test mode would emit a failure to stdout when
testing devel/py-setuptools27. It turns out that there's a file there
with a space in the filename. The filename was an argument for
/usr/bin//file and it wasn't escaped. The file in question had
parentheses too which the shell was trying to process. The fix was
to escape the filename in the /usr/bin/file command.
* The builders were mounting the source directory from "/usr/src", not
$sysroot/usr/src as intended. This potentially causes breakage when
the $sysroot reflects a different versions/release than the host
machine has (e.g. making FreeBSD 10.2 packages on FreeBSD 11-current).
Now the source directory mount is relative to profile's $sysroot.
