The default VM size may depend on the architecture. In particular,
it is currently larged on riscv64 due to a toolchain issue which
results in bloated binaries.
MFC after: 3 days
Fixes: 59c21ed6e8 "release: Bump default VM size for riscv64 to 6 GB"
Sponsored by: Amazon
Due to issues with the riscv64 toolchain, some binaries end up
significantly larger on riscv64 than they should be. This results
in riscv64 VM images -- and at present *only* riscv64 images -- not
fitting within the default 5 GB filesystem size.
Bump the default size for riscv64 to 6 GB until the toolchain issues
can be resolved.
MFC after: 1 week
Sponsored by: Amazon
VM, BASIC-CI, and BASIC-CLOUDWARE images are published with the
filesystem type (UFS or ZFS) in the file name. For backwards
compatibility we have published the UFS images with no filesystem
in the filename (aka with the same file names as we used before we
supported building ZFS VM images); those are going away now.
This will not be MFCed.
If WITH_CLOUDWARE is not set, we didn't build BASIC-CLOUDWARE; don't try
to install it into the /VM-IMAGES/ directory as part of ftp-stage.
Fixes: 4771c2e9d1 "release: Publish non-uploaded cloudware bits"
MFC after: 6 days
While most "cloudware" (AWS, Azure, GCP, Vagrant) images get uploaded
to their respective clouds, we have two images -- BASIC-CI and
BASIC-CLOUDINIT -- which are published via the FTP mirrors. We have
been handling these using a manual and error-prone process of copying
them from the internal build directories into the FTP staging tree,
but they should be handled using the "make install ftp-stage"
mechanism which is used for all the other published images.
This commit 1. Adds a new 'make cloud-install' target which is run as
part of 'make install' and installs BASIC-CI and BASIC-CLOUDINIT images,
and 2. Extends 'make ftp-stage' in Makefile.mirrors to copy those bits
into the FTP staging area.
As a side benefit, this will also make BASIC-CI and BASIC-CLOUDINIT
images available for arm64; until now they have only been available
for amd64.
MFC after: 1 week
This breaks the weekly snapshot builds. We'll try again once Warner has
some time to debug the qemu-user-static-devel port.
This reverts commit a04ecddfc7.
Remove archivers/unzip (now in base) and emulators/linux_base-c7 (old
and unlikely to be useful without other linux packages being installed),
ports-mgmt/portmaster (now largely obsolete and discouraged in favour
of using pkg and binary packages) and x11-drivers/xf86-video-vmware
(questionably useful).
Replace devel/git with devel/git@lite (sufficient for most purposes),
and adjust the "ensure the ports exist to sanitize the list" code to
ignore the @lite part when checking that /usr/ports/devel/git exists.
Add sysutils/seatd and x11-wm/sway for wayland support.
MFC after: 1 minute
Differential Revision: https://reviews.freebsd.org/D45278
For historical reasons, the vm-release and cloudware-release targets
were written as recursive makes of ${VMTARGETS} and ${CLOUDTARGETS};
this worked fine until we started running `make release -jN` and
had both VM and cloud targets depending on ${QEMUTGT}, at which
point things exploded when we tried to build the port twice at the
same time.
Switch vm-release and cloudware-release to have make dependencies on
their individual components; this way a single make process runs and
is aware of the duplicate dependency (and only runs it once).
MFC after: 5 days
The vm_extra_pre_umount function in vmimage.subr served two purposes:
It removed /etc/resolv.conf and /qemu (if cross-building), and it
provided a function for cloudware to override in order to make cloud
specific changes to the filesystem before constructing a disk image.
This resulted in a number of bugs:
1. When cross-building, the emulator binary was left as /qemu in the
Azure, GCE, Openstack and Vagrant images.
2. The build host's resolv.conf was left as /etc/resolv.conf in the
basic-ci and basic-cloudinit images.
3. When building GCE images, a Google-specific resolv.conf file was
constructed, and then deleted before the disk image was created.
Move the bits needed for running code inside a VM staging directory
from vm_install_base into a new vm_emulation_setup routine, and move
the corresponding cleanup bits from vm_extra_pre_umount to a new
vm_emulation_cleanup routine.
Remove the /qemu and /etc/resolv.conf cleanups from the cloudware
configuration files (where they exist) since we will now be running
vm_emulation_cleanup to remove those even when vm_extra_pre_umount
has been overridden.
Override vm_emulation_cleanup in gce.conf since in that one case (and
*only* that one case) we don't want to clean up resolv.conf (since it
was constructed for the VM image rather than copied from the host).
releng/14.1 candidate.
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva
For a bit over 5 years, we have used qemu when cross-building cloudware
images; in particular, it's necessary when installing packages which
might include post-install scripts.
Use qemu in the vm-images target too; while "generic" vm images don't
install packages, they still run newaliases and /etc/rc.d/ldconfig,
both of which fail without appropriate emulation.
MFC after: 1 week
When the VM image building code was updated to support building
non-UFS images, the vm-images-stage target was not updated to
install those newly built images to the FTP site. As a result, we
have been sending weekly snapshot announcements since August claiming
that ZFS VM images are available when they are not in fact present
anywhere publicly accessible.
Fixes: 32ae9a6b39 "release: Build UFS and ZFS VM images"
Reported by: Michael Dexter
MFC after: 5 days
A bug in release/packages/generate-ucl.sh causes package dependencies
(other than shlib depends) to not be generated correctly, meaning
packages are missing their dependencies.
generate-ucl.sh creates the UCL file by:
1. copying ${uclsource} (template.ucl) to ${uclfile}
2. appending dependencies to ${uclfile}
3. calling generate-ucl.lua on ${uclsource} to create ${uclfile}
This breaks because the dependencies added in step 2 are overwritten in
step 3.
Fix this by calling generate-ucl.lua with ${uclfile} as both the input
and output file, so anything we added to ${uclfile} is preserved.
Reviewed by: des, imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/1203
Move auditing runtime (auditd, etc.) into the new FreeBSD-audit package.
Also move the runtime OpenBSM manual pages from libbsm into auditd so
they get installed with the right package.
Add an UPDATING entry noting the new packages.
Reviewed by: imp, manu
Pull Request: https://github.com/freebsd/freebsd-src/pull/1197
This isn't actually packaged by the sysutils/rpi-firmware port, so we
cannot include it in the image. Fix the RPI release build until an
interested party adds it to the port and packages are available.
Reported by: cperciva
Allow a new variable SE_xxxPACKAGE to be used to place an example group
into a package.
Move the following examples into existing appropriate packages:
- bhyve examples into FreeBSD-bhyve
- bootforth examples into FreeBSD-bootloader
- csh examples into FreeBSD-csh
- ipfw examples into FreeBSD-ipfw
- jail examples into FreeBSD-jail
- pf examples into FreeBSD-pf
- ppp examples into FreeBSD-ppp
- printing examples into FreeBSD-lp
- uefisign examples into FreeBSD-efi-tools
- ypldap examples into FreeBSD-yp
- hast examples into FreeBSD-hast
Move all other examples into the new 'FreeBSD-examples' package.
This removes a significant number of files from the FreeBSD-utilities
package.
Reviewed by: imp, manu
Pull Request: https://github.com/freebsd/freebsd-src/pull/1176
This missing dependency is only visible with -jN (which was otherwise
broken until recently) and only when building DVD images (which aren't
included in weekly snapshots and thereby slipped past testing).
MFC after: 3 days
With multiple filesystems and disk image formats, we now use different
staging directories; we need to include them all in CLEANDIRS.
Fixes: 97bd53ef4d "Makefile.vm: Fix duplicate rc.conf files"
MFC after: 3 days
- Point --rootdir at the installed destdir in the dvd tree. This
causes pkg to determine the ABI from the installed destdir instead
of the host's binaries. Previously the result was that packages
for the host's ABI were always downloaded breaking cross-releases
(e.g. arm64 releases built on an amd64 host included amd64
packages on the DVD ISO image rather than arm64 packages). This
also handles version mismatches, and I tested this by cross-building
a 15.x arm64 release on a 14.x amd64 host.
- As a result, pkg now does a chdir(3) to the rootdir before running,
so the -o argument to fetch needs to be updated to be relative to
rootdir instead of the CWD as make runs.
- Add a new ROOTDIR variable to limit references to "dvd" to one
place. Ideally ROOTDIR would be an argument to this script so
that it didn't really know about the dvd layout at all.
- While here, simplify creation of symlinks by just using a longer
path to the link name instead of using 'cd' in the shell before
invoking ln(1). Also use ln -sf to create the pkg.pkg symlink
rather than rm + ln.
PR: 278273
Reported by: gatekeeper <tiago.gasiba@gmail.com>
Reviewed by: imp, delphij
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D44749
Make basic-cloudinit available both in qcow2 and raw formats
MFC After: 3 days
Co-Authored-By: bapt
Sponsored by: OVHCloud (original patch from bapt)
Differential Revision: https://reviews.freebsd.org/D44747
Confusingly, uuencode writes to stdout by default, but uudecode does not.
Fixes: 6c3e01bf85 "release: Randomize powerpc boot block file name"
MFC after: 3 days
With parallel builds enabled, we can end up building multiple ISOs at
once, which causes problems if mkisoimages.sh uses a consistent file
name "/tmp/hfs-boot-block".
MFC after: 3 days
Set WRKDIRPREFIX=/tmp/ports DISTDIR=/tmp/distfiles when building tools
needed for uploading cloudware images.
While I'm here, adjust the bsdec2-image-upload build target to match
the style used by the Azure/GCE/Vagrant Makefiles.
MFC after: 3 days
Build qemu (if needed) with WRKDIRPREFIX=/tmp/ports DISTDIR=/tmp/distfiles
so that we can have a read-only /usr/ports and don't contaminate it. This
became an issue when I enabled parallel release building, since one image
might be creating its ports.txz file at the same time as we're building
qemu as a prerequisite for building another image.
MFC after: 5 days
Make basic-cloudinit available both in qcow2 and raw formats
MFC After: 1 week
Reviewed by: Allanjude
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44747
With the latest changes to release/Makefile, it is now possible to
run `make release -jN` without the build failing (at least in my
latest tests).
This reverts commit 7b707e797b.
MFC after: 1 week
The disc1 and bootonly directories have files distributed into them
for use in "full" and "mini" images; the former are disc1.iso and
memstick.img, and the latter is bootonly.iso and mini-memstick.img.
Unfortunately the scripts which package a directory tree into an ISO
or memory stick image also modify the directory, for example to
create an appropriate /etc/fstab file; so creating two images at the
same time breaks.
Resolve this by copying disc1 to disc1-disc1 and disc1-memstick,
and copying bootonly to bootonly-bootonly and bootonly-memstick,
before using those directories for constructing the ISO+memstick
images.
MFC after: 1 week
Place instances of "cd foo && bar" inside subshells for compatibility
with modern make(8) which uses a single shell for the duration of a
makefile target.
MFC after: 1 week
Prior to this commit, we install git from ports if there is a ports
tree available and git is not installed, and we install git from pkg
otherwise -- including the case where git is already installed.
Rework the logic to not (re)install git at all if it is already
installed.
MFC after: 3 days
The framework do not support multiple format images, so this last minute
addition of qcow2 format was clearly not a good idea.
while here fix a typo
Reported by: cperciva
This is made in order to be able to find add the post-install scripts
for the kernel, where PKGNAME varies for each KERNCONF but we don't want
to dynamically duplicated the kernel.ucl file.
At the same time we don't want the *-dbg* packages to actually include
those post-install scripts
Rework how ucl manifest are generated leveraging ucl features and flua
now the ucl generation is done via a lua script which uses libucl to
ingest the template and use variables as defined in its command line.
the template will include only if it exist a ucl file named after the
package name which will complement the template or overwrite what was
defined in the template if defined in this specific ucl file
this allows to overwrite license, but add script only to the packages
who actually needs them.
As a results the post install scripts are now only added to the right
package and not also added to the subpackages like -man or -dev
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D44374
Provide both zfs and ufs images which a 1MB partition reserved for the
config drive wearing a GPT Label "config-drive" to allow consumer to
know where they should push the config drive on the provided image.
2 formats available: qcow2 and raw
This has been tested on OVHCloud baremetal via "bring your own image"
Also tested on openstack
Reviewed by: emaste
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44369
GDB was the final GNU binutils component included in the base system,
but was removed in 2020. Nothing provides a pkgbase binutils package
any longer.
Fixes: 1c0ea326aa ("Retire obsolete GDB 6.1.1")
Sponsored by: The FreeBSD Foundation
Currently there is no support for generating armv7 vm images in the
release artifacts. In fact in terms of release artifacts and
architecture there is no good reason to have a vm release artifact for
armv7 as those are mostly used in SOCs or embedded boards. However
considering that developers actually do need an easy way to test armv7
with a vm running this is really important. As part of pre-commit ci for
developers this can be really helpful for the end developers.
Approved by: cperciva, imp, re
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D43952
Create two partitions in the existing space at the beginning of the
image (8MB). These are intended to hold u-boot SPL and u-boot proper.
The partition types selected are compatible with SiFive boards, e.g. the
HiFive Unmatched. They can easily be overridden for a platform that uses
the same scheme but different partition types, e.g. the StarFive
VisionFive v2. Firmware is not actually installed, this too is left for
the user.
It is not as simple to create the firmware partitions after the fact,
e.g. with partition indices 3 and 4. It is a shortcoming of current day
u-boot that the SPL loader looks for a specific partition index, rather
than the partition type, meaning that we will fail to boot if partition
2 doesn't contain u-boot.
Thus, our GENERICSD images become more generically usable with current
RISC-V hardware/firmware platforms.
Reviewed by: manu, karels, imp
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D43849