1
0
mirror of https://git.FreeBSD.org/src.git synced 2024-12-23 11:18:54 +00:00
freebsd/sys
Maxim Sobolev 8f8cb840b0 Improve mkuzip(8) and geom_uzip(4), merge in LZMA support from mkulzma(8)
and geom_uncompress(4):

1. mkuzip(8):

 - Proper support for eliminating all-zero blocks when compressing an
   image. This feature is already supported by the geom_uzip(4) module
   and CLOOP format in general, so it's just a matter of making mkuzip(8)
   match. It should be noted, however that this feature while it sounds
   great, results in very slight improvement in the overall compression
   ratio, since compressing default 16k all-zero block produces only 39
   bytes compressed output block, which is 99.8% compression ratio. With
   typical average compression ratio of amd64 binaries and data being
   around 60-70% the difference between 99.8% and 100.0% is not that
   great further diluted by the ratio of number of zero blocks in the
   uncompressed image to the overall number of blocks being less than
   0.5 (typically). However, this may be important from performance
   standpoint, so that kernel are not spinning its wheels decompressing
   those empty blocks every time this zero region is read. It could also
   be important when you create huge image mostly filled with zero
   blocks for testing purposes.

 - New feature allowing to de-duplicate output image. It turns out that
   if you twist CLOOP format a bit you can do that as well. And unlike
   zero-blocks elimination, this gives a noticeable improvement in the
   overall compression ratio, reducing output image by something like
   3-4% on my test UFS2 3GB image consisting of full FreeBSD base system
   plus some of the packages (openjdk, apache etc), about 2.3GB worth of
   file data (800+MB compressed). The only caveat is that images created
   with this feature "on" would not work on older versions of FeeBSDxi
   kernel, hence it's turned off by default.

 - provide options to control both features and document them in manual
   page.

 - merge in all relevant LZMA compression support from the mkulzma(8),
   add new option to select between both.

 - switch license from ad-hoc beerware into standard 2-clause BSD.

2. geom_uzip(4):

 - implement support for de-duplicated images;

 - optimize some code paths to handle "all-zero" blocks without reading
   any compressed data;

 - beef up manual page to explain that geom_uzip(4) is not limited only
   to md(4) images. The compressed data can be written to the block
   device and accessed directly via magic of GEOM(4) and devfs(4),
   including to mount root fs from a compressed drive.

 - convert debug log code from being compiled in conditionally into
   being present all the time and provide two sysctls to turn it on or
   off. Due to intended use of the module, it can be used in
   environments where there may not be a luxury to put new kernel with
   debug code enabled. Having those options handy allows debug issues
   without as much problem by just having access to serial console or
   network shell access to a box/appliance. The resulting additional
   CPU cycles are just few int comparisons and branches, and those are
   minuscule when compared to data decompression which is the main
   feature of the module.

 - hopefully improve robustness and resiliency of the geom_uzip(4) by
   performing some of the data validation / range checking on the TOC
   entries and rejecting to attach to an image if those checks fail.

 - merge in all relevant LZMA decompression support from the
   geom_uncompress(4), enable automatically when appropriate format is
   indicated in the header.

 - move compilation work into its own worker thread so that it does not
   clog g_up. This allows multiple instances work in parallel utilizing
   smp cores.

 - document new knobs in the manual page.

Reviewed by:		adrian
MFC after:		1 month
Differential Revision:	https://reviews.freebsd.org/D5333
2016-02-23 23:59:08 +00:00
..
amd64 As <machine/vm.h> is included from <vm/vm.h>, there is no need to 2016-02-22 09:10:23 +00:00
arm Remove redundant __ARM_ARCH >= 6 check in armv6 specific files. 2016-02-23 10:12:50 +00:00
arm64 Add Intel 10Gb support to ARM64 GENERIC kernel config 2016-02-22 13:34:43 +00:00
boot o Grab physical memory regions information from the device tree. 2016-02-23 14:21:46 +00:00
bsm
cam Remove a stray else. It isn't needed (due to the return at the end of 2016-02-18 15:12:52 +00:00
cddl As <machine/vmparam.h> is included from <vm/vm_param.h>, there is no 2016-02-22 09:08:04 +00:00
compat Make handling of mmap()'s prot argument more strict. 2016-02-23 09:22:00 +00:00
conf Improve mkuzip(8) and geom_uzip(4), merge in LZMA support from mkulzma(8) 2016-02-23 23:59:08 +00:00
contrib Optimize ROL and ROR emulations and fix comments. 2016-02-18 23:03:37 +00:00
crypto
ddb
dev Silence PVS-Studio warning (V595). 2016-02-23 23:09:45 +00:00
fs Ext2: cleanup setting of ctime/mtime/birthtime. 2016-02-19 15:53:08 +00:00
gdb
geom Improve mkuzip(8) and geom_uzip(4), merge in LZMA support from mkulzma(8) 2016-02-23 23:59:08 +00:00
gnu Update our copy of the Linux dts files to be in sync with Linux 4.5-rc1. We 2016-02-09 16:42:32 +00:00
i386 As <machine/pmap.h> is included from <vm/pmap.h>, there is no need to 2016-02-22 09:02:20 +00:00
isa Replace several bus_alloc_resource() calls using default arguments with bus_alloc_resource_any() 2016-02-19 03:37:56 +00:00
kern Fix build after r295934. 2016-02-23 23:37:10 +00:00
kgssapi kcrypto_aes: Use separate sessions for AES and SHA1 2016-02-02 00:14:51 +00:00
libkern
mips As <machine/param.h> is included from <sys/param.h>, there is no need 2016-02-22 09:04:36 +00:00
modules Improve mkuzip(8) and geom_uzip(4), merge in LZMA support from mkulzma(8) 2016-02-23 23:59:08 +00:00
net Fix regression introduced on 272446r. 2016-02-19 06:35:53 +00:00
net80211 net80211: fix TIM cleanup. 2016-02-23 21:11:42 +00:00
netgraph These files were getting sys/malloc.h and vm/uma.h with header pollution 2016-02-01 17:41:21 +00:00
netinet Fix build after r29592. 2016-02-23 21:21:47 +00:00
netinet6 Don't leak an address in an error path. 2016-02-23 18:50:34 +00:00
netipsec MfH @r295202 2016-02-03 11:49:51 +00:00
netnatm
netpfil in pf_print_state_parts, do not use skw->proto to print the protocol but our 2016-02-20 12:53:53 +00:00
netsmb
nfs MFP r287070,r287073: split radix implementation and route table structure. 2016-01-25 06:33:15 +00:00
nfsclient
nfsserver
nlm
ofed LinuxKPI list updates: 2016-01-26 15:12:31 +00:00
opencrypto
pc98 Convert rman to use rman_res_t instead of u_long 2016-01-27 02:23:54 +00:00
powerpc Allow the size argument for law_enable() to be non-power-of-2. 2016-02-23 02:28:19 +00:00
riscv o Grab physical memory regions information from the device tree. 2016-02-23 14:21:46 +00:00
rpc These files were getting sys/malloc.h and vm/uma.h with header pollution 2016-02-01 17:41:21 +00:00
security Busy the mount point which is the owner of the audit vnode, around 2016-01-16 10:06:33 +00:00
sparc64 As <machine/pmap.h> is included from <vm/pmap.h>, there is no need to 2016-02-22 09:02:20 +00:00
sys Fix the definition of RM_MAX_END. 2016-02-20 01:34:13 +00:00
teken
tests
tools Fix MFS builds when both MD_ROOT_SIZE and MFS_IMAGE are specified 2016-02-02 07:02:51 +00:00
ufs Remove ffs_mountroot() prototype; seems to be long gone. 2016-01-28 12:21:23 +00:00
vm Include sys/_task.h into uma_int.h, so that taskqueue.h isn't a 2016-02-09 20:22:35 +00:00
x86 Silence PVS-Studio warning (V595). It can never be NULL here. 2016-02-23 23:57:24 +00:00
xdr
xen xenbus: add a comment with the names of the generated accessors 2016-01-15 14:34:31 +00:00
Makefile