2005-01-07 02:29:27 +00:00
|
|
|
/*-
|
2017-11-30 15:48:35 +00:00
|
|
|
* SPDX-License-Identifier: (BSD-3-Clause AND MIT-CMU)
|
2017-11-20 19:43:44 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Copyright (c) 1991, 1993
|
|
|
|
* The Regents of the University of California. All rights reserved.
|
|
|
|
*
|
|
|
|
* This code is derived from software contributed to Berkeley by
|
|
|
|
* The Mach Operating System project at Carnegie-Mellon University.
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
2017-02-28 23:42:47 +00:00
|
|
|
* 3. Neither the name of the University nor the names of its contributors
|
1994-05-24 10:09:53 +00:00
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
* without specific prior written permission.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
*
|
1994-08-02 07:55:43 +00:00
|
|
|
* from: @(#)vm_glue.c 8.6 (Berkeley) 1/5/94
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
|
|
|
*
|
|
|
|
* Copyright (c) 1987, 1990 Carnegie-Mellon University.
|
|
|
|
* All rights reserved.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Permission to use, copy, modify and distribute this software and
|
|
|
|
* its documentation is hereby granted, provided that both the copyright
|
|
|
|
* notice and this permission notice appear in all copies of the
|
|
|
|
* software, derivative works or modified versions, and any portions
|
|
|
|
* thereof, and that both notices appear in supporting documentation.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
|
|
|
* CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS"
|
|
|
|
* CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND
|
1994-05-24 10:09:53 +00:00
|
|
|
* FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Carnegie Mellon requests users of this software to return to
|
|
|
|
*
|
|
|
|
* Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU
|
|
|
|
* School of Computer Science
|
|
|
|
* Carnegie Mellon University
|
|
|
|
* Pittsburgh PA 15213-3890
|
|
|
|
*
|
|
|
|
* any improvements or extensions that they make and grant Carnegie the
|
|
|
|
* rights to redistribute these changes.
|
|
|
|
*/
|
|
|
|
|
2003-06-11 23:50:51 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
__FBSDID("$FreeBSD$");
|
|
|
|
|
1998-09-29 17:33:59 +00:00
|
|
|
#include "opt_vm.h"
|
2003-07-31 01:25:05 +00:00
|
|
|
#include "opt_kstack_pages.h"
|
|
|
|
#include "opt_kstack_max_pages.h"
|
2014-10-04 18:38:14 +00:00
|
|
|
#include "opt_kstack_usage_prof.h"
|
1996-12-22 23:17:09 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/param.h>
|
|
|
|
#include <sys/systm.h>
|
2003-04-29 13:36:06 +00:00
|
|
|
#include <sys/limits.h>
|
2001-05-01 08:13:21 +00:00
|
|
|
#include <sys/lock.h>
|
2013-08-07 06:21:20 +00:00
|
|
|
#include <sys/malloc.h>
|
2001-05-01 08:13:21 +00:00
|
|
|
#include <sys/mutex.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/proc.h>
|
2011-04-05 20:23:59 +00:00
|
|
|
#include <sys/racct.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/resourcevar.h>
|
2013-03-09 02:32:23 +00:00
|
|
|
#include <sys/rwlock.h>
|
2005-12-16 18:34:14 +00:00
|
|
|
#include <sys/sched.h>
|
|
|
|
#include <sys/sf_buf.h>
|
1995-03-28 07:58:53 +00:00
|
|
|
#include <sys/shm.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <sys/vmmeter.h>
|
2013-08-07 06:21:20 +00:00
|
|
|
#include <sys/vmem.h>
|
2001-03-28 11:52:56 +00:00
|
|
|
#include <sys/sx.h>
|
1997-12-06 02:23:36 +00:00
|
|
|
#include <sys/sysctl.h>
|
2011-12-16 10:56:16 +00:00
|
|
|
#include <sys/_kstack_cache.h>
|
2009-09-01 11:41:51 +00:00
|
|
|
#include <sys/eventhandler.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
#include <sys/kernel.h>
|
2000-09-07 01:33:02 +00:00
|
|
|
#include <sys/ktr.h>
|
1997-04-07 07:16:06 +00:00
|
|
|
#include <sys/unistd.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_param.h>
|
|
|
|
#include <vm/pmap.h>
|
|
|
|
#include <vm/vm_map.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm_page.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
#include <vm/vm_pageout.h>
|
2002-07-07 23:05:27 +00:00
|
|
|
#include <vm/vm_object.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm_kern.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_extern.h>
|
2002-07-07 23:05:27 +00:00
|
|
|
#include <vm/vm_pager.h>
|
2002-12-15 19:17:57 +00:00
|
|
|
#include <vm/swap_pager.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
|
2014-10-04 18:38:14 +00:00
|
|
|
#include <machine/cpu.h>
|
|
|
|
|
2002-06-22 01:26:02 +00:00
|
|
|
/*
|
|
|
|
* MPSAFE
|
2003-01-20 17:46:48 +00:00
|
|
|
*
|
|
|
|
* WARNING! This code calls vm_map_check_protection() which only checks
|
|
|
|
* the associated vm_map_entry range. It does not determine whether the
|
|
|
|
* contents of the memory is actually readable or writable. In most cases
|
|
|
|
* just checking the vm_map_entry is sufficient within the kernel's address
|
|
|
|
* space.
|
2002-06-22 01:26:02 +00:00
|
|
|
*/
|
1994-05-24 10:09:53 +00:00
|
|
|
int
|
2018-01-12 03:50:44 +00:00
|
|
|
kernacc(void *addr, int len, int rw)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
|
|
|
boolean_t rv;
|
|
|
|
vm_offset_t saddr, eaddr;
|
1999-10-30 06:32:05 +00:00
|
|
|
vm_prot_t prot;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-01-17 16:46:26 +00:00
|
|
|
KASSERT((rw & ~VM_PROT_ALL) == 0,
|
1999-10-30 06:32:05 +00:00
|
|
|
("illegal ``rw'' argument to kernacc (%x)\n", rw));
|
2005-01-22 19:21:29 +00:00
|
|
|
|
|
|
|
if ((vm_offset_t)addr + len > kernel_map->max_offset ||
|
|
|
|
(vm_offset_t)addr + len < (vm_offset_t)addr)
|
|
|
|
return (FALSE);
|
|
|
|
|
1999-10-30 06:32:05 +00:00
|
|
|
prot = rw;
|
1998-10-13 08:24:45 +00:00
|
|
|
saddr = trunc_page((vm_offset_t)addr);
|
|
|
|
eaddr = round_page((vm_offset_t)addr + len);
|
2003-11-10 01:37:40 +00:00
|
|
|
vm_map_lock_read(kernel_map);
|
1994-05-24 10:09:53 +00:00
|
|
|
rv = vm_map_check_protection(kernel_map, saddr, eaddr, prot);
|
2003-11-10 01:37:40 +00:00
|
|
|
vm_map_unlock_read(kernel_map);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
return (rv == TRUE);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
2002-06-22 01:26:02 +00:00
|
|
|
/*
|
|
|
|
* MPSAFE
|
2003-01-20 17:46:48 +00:00
|
|
|
*
|
|
|
|
* WARNING! This code calls vm_map_check_protection() which only checks
|
|
|
|
* the associated vm_map_entry range. It does not determine whether the
|
|
|
|
* contents of the memory is actually readable or writable. vmapbuf(),
|
|
|
|
* vm_fault_quick(), or copyin()/copout()/su*()/fu*() functions should be
|
2016-05-02 20:16:29 +00:00
|
|
|
* used in conjunction with this call.
|
2002-06-22 01:26:02 +00:00
|
|
|
*/
|
1994-05-24 10:09:53 +00:00
|
|
|
int
|
2018-01-12 03:50:44 +00:00
|
|
|
useracc(void *addr, int len, int rw)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
|
|
|
boolean_t rv;
|
1999-10-30 06:32:05 +00:00
|
|
|
vm_prot_t prot;
|
2002-09-21 22:07:17 +00:00
|
|
|
vm_map_t map;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-01-17 16:46:26 +00:00
|
|
|
KASSERT((rw & ~VM_PROT_ALL) == 0,
|
1999-10-30 06:32:05 +00:00
|
|
|
("illegal ``rw'' argument to useracc (%x)\n", rw));
|
|
|
|
prot = rw;
|
2002-09-21 22:07:17 +00:00
|
|
|
map = &curproc->p_vmspace->vm_map;
|
|
|
|
if ((vm_offset_t)addr + len > vm_map_max(map) ||
|
|
|
|
(vm_offset_t)addr + len < (vm_offset_t)addr) {
|
1994-05-25 09:21:21 +00:00
|
|
|
return (FALSE);
|
|
|
|
}
|
2003-11-10 01:37:40 +00:00
|
|
|
vm_map_lock_read(map);
|
2002-09-21 22:07:17 +00:00
|
|
|
rv = vm_map_check_protection(map, trunc_page((vm_offset_t)addr),
|
|
|
|
round_page((vm_offset_t)addr + len), prot);
|
2003-11-10 01:37:40 +00:00
|
|
|
vm_map_unlock_read(map);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
return (rv == TRUE);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
2004-03-05 22:03:11 +00:00
|
|
|
int
|
2004-03-15 06:43:51 +00:00
|
|
|
vslock(void *addr, size_t len)
|
2004-03-05 22:03:11 +00:00
|
|
|
{
|
2004-03-15 09:11:23 +00:00
|
|
|
vm_offset_t end, last, start;
|
|
|
|
vm_size_t npages;
|
|
|
|
int error;
|
2004-03-05 22:03:11 +00:00
|
|
|
|
2004-03-15 09:11:23 +00:00
|
|
|
last = (vm_offset_t)addr + len;
|
2004-03-15 06:42:40 +00:00
|
|
|
start = trunc_page((vm_offset_t)addr);
|
2004-03-15 09:11:23 +00:00
|
|
|
end = round_page(last);
|
|
|
|
if (last < (vm_offset_t)addr || end < (vm_offset_t)addr)
|
2004-03-05 22:03:11 +00:00
|
|
|
return (EINVAL);
|
|
|
|
npages = atop(end - start);
|
|
|
|
if (npages > vm_page_max_wired)
|
|
|
|
return (ENOMEM);
|
|
|
|
#if 0
|
|
|
|
/*
|
|
|
|
* XXX - not yet
|
|
|
|
*
|
|
|
|
* The limit for transient usage of wired pages should be
|
|
|
|
* larger than for "permanent" wired pages (mlock()).
|
|
|
|
*
|
|
|
|
* Also, the sysctl code, which is the only present user
|
|
|
|
* of vslock(), does a hard loop on EAGAIN.
|
|
|
|
*/
|
2014-03-22 10:26:09 +00:00
|
|
|
if (npages + vm_cnt.v_wire_count > vm_page_max_wired)
|
2004-03-05 22:03:11 +00:00
|
|
|
return (EAGAIN);
|
|
|
|
#endif
|
2004-03-15 06:42:40 +00:00
|
|
|
error = vm_map_wire(&curproc->p_vmspace->vm_map, start, end,
|
2004-05-07 11:43:24 +00:00
|
|
|
VM_MAP_WIRE_SYSTEM | VM_MAP_WIRE_NOHOLES);
|
2004-03-15 06:42:40 +00:00
|
|
|
/*
|
|
|
|
* Return EFAULT on error to match copy{in,out}() behaviour
|
|
|
|
* rather than returning ENOMEM like mlock() would.
|
|
|
|
*/
|
|
|
|
return (error == KERN_SUCCESS ? 0 : EFAULT);
|
2004-03-05 22:03:11 +00:00
|
|
|
}
|
|
|
|
|
2004-03-15 06:42:40 +00:00
|
|
|
void
|
2004-03-15 06:43:51 +00:00
|
|
|
vsunlock(void *addr, size_t len)
|
2004-03-05 22:03:11 +00:00
|
|
|
{
|
|
|
|
|
2004-03-15 06:42:40 +00:00
|
|
|
/* Rely on the parameter sanity checks performed by vslock(). */
|
|
|
|
(void)vm_map_unwire(&curproc->p_vmspace->vm_map,
|
|
|
|
trunc_page((vm_offset_t)addr), round_page((vm_offset_t)addr + len),
|
2004-03-15 06:43:51 +00:00
|
|
|
VM_MAP_WIRE_SYSTEM | VM_MAP_WIRE_NOHOLES);
|
2004-03-05 22:03:11 +00:00
|
|
|
}
|
|
|
|
|
2005-12-16 18:34:14 +00:00
|
|
|
/*
|
|
|
|
* Pin the page contained within the given object at the given offset. If the
|
|
|
|
* page is not resident, allocate and load it using the given object's pager.
|
|
|
|
* Return the pinned page if successful; otherwise, return NULL.
|
|
|
|
*/
|
|
|
|
static vm_page_t
|
2013-08-05 08:55:35 +00:00
|
|
|
vm_imgact_hold_page(vm_object_t object, vm_ooffset_t offset)
|
2005-12-16 18:34:14 +00:00
|
|
|
{
|
2015-06-12 11:32:20 +00:00
|
|
|
vm_page_t m;
|
2005-12-16 18:34:14 +00:00
|
|
|
vm_pindex_t pindex;
|
|
|
|
int rv;
|
|
|
|
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_WLOCK(object);
|
2005-12-16 18:34:14 +00:00
|
|
|
pindex = OFF_TO_IDX(offset);
|
2016-08-14 22:00:45 +00:00
|
|
|
m = vm_page_grab(object, pindex, VM_ALLOC_NORMAL | VM_ALLOC_NOBUSY);
|
2009-06-07 19:38:26 +00:00
|
|
|
if (m->valid != VM_PAGE_BITS_ALL) {
|
2016-08-14 22:00:45 +00:00
|
|
|
vm_page_xbusy(m);
|
A change to KPI of vm_pager_get_pages() and underlying VOP_GETPAGES().
o With new KPI consumers can request contiguous ranges of pages, and
unlike before, all pages will be kept busied on return, like it was
done before with the 'reqpage' only. Now the reqpage goes away. With
new interface it is easier to implement code protected from race
conditions.
Such arrayed requests for now should be preceeded by a call to
vm_pager_haspage() to make sure that request is possible. This
could be improved later, making vm_pager_haspage() obsolete.
Strenghtening the promises on the business of the array of pages
allows us to remove such hacks as swp_pager_free_nrpage() and
vm_pager_free_nonreq().
o New KPI accepts two integer pointers that may optionally point at
values for read ahead and read behind, that a pager may do, if it
can. These pages are completely owned by pager, and not controlled
by the caller.
This shifts the UFS-specific readahead logic from vm_fault.c, which
should be file system agnostic, into vnode_pager.c. It also removes
one VOP_BMAP() request per hard fault.
Discussed with: kib, alc, jeff, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix
2015-12-16 21:30:45 +00:00
|
|
|
rv = vm_pager_get_pages(object, &m, 1, NULL, NULL);
|
2009-06-06 20:13:14 +00:00
|
|
|
if (rv != VM_PAGER_OK) {
|
2010-04-30 00:46:43 +00:00
|
|
|
vm_page_lock(m);
|
2005-12-16 18:34:14 +00:00
|
|
|
vm_page_free(m);
|
2010-04-30 00:46:43 +00:00
|
|
|
vm_page_unlock(m);
|
2005-12-16 18:34:14 +00:00
|
|
|
m = NULL;
|
|
|
|
goto out;
|
|
|
|
}
|
2016-08-14 22:00:45 +00:00
|
|
|
vm_page_xunbusy(m);
|
2005-12-16 18:34:14 +00:00
|
|
|
}
|
2013-08-05 08:55:35 +00:00
|
|
|
vm_page_lock(m);
|
|
|
|
vm_page_hold(m);
|
2014-08-13 05:44:08 +00:00
|
|
|
vm_page_activate(m);
|
2013-08-05 08:55:35 +00:00
|
|
|
vm_page_unlock(m);
|
2005-12-16 18:34:14 +00:00
|
|
|
out:
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_WUNLOCK(object);
|
2005-12-16 18:34:14 +00:00
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Return a CPU private mapping to the page at the given offset within the
|
|
|
|
* given object. The page is pinned before it is mapped.
|
|
|
|
*/
|
|
|
|
struct sf_buf *
|
|
|
|
vm_imgact_map_page(vm_object_t object, vm_ooffset_t offset)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
2013-08-05 08:55:35 +00:00
|
|
|
m = vm_imgact_hold_page(object, offset);
|
2005-12-16 18:34:14 +00:00
|
|
|
if (m == NULL)
|
|
|
|
return (NULL);
|
|
|
|
sched_pin();
|
|
|
|
return (sf_buf_alloc(m, SFB_CPUPRIVATE));
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Destroy the given CPU private mapping and unpin the page that it mapped.
|
|
|
|
*/
|
|
|
|
void
|
2013-08-05 08:55:35 +00:00
|
|
|
vm_imgact_unmap_page(struct sf_buf *sf)
|
2005-12-16 18:34:14 +00:00
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
|
|
|
m = sf_buf_page(sf);
|
|
|
|
sf_buf_free(sf);
|
|
|
|
sched_unpin();
|
2013-08-05 08:55:35 +00:00
|
|
|
vm_page_lock(m);
|
|
|
|
vm_page_unhold(m);
|
|
|
|
vm_page_unlock(m);
|
2005-12-16 18:34:14 +00:00
|
|
|
}
|
|
|
|
|
2009-10-21 18:38:02 +00:00
|
|
|
void
|
|
|
|
vm_sync_icache(vm_map_t map, vm_offset_t va, vm_offset_t sz)
|
|
|
|
{
|
|
|
|
|
|
|
|
pmap_sync_icache(map->pmap, va, sz);
|
|
|
|
}
|
|
|
|
|
2011-12-16 10:56:16 +00:00
|
|
|
struct kstack_cache_entry *kstack_cache;
|
2009-09-01 11:41:51 +00:00
|
|
|
static int kstack_cache_size = 128;
|
|
|
|
static int kstacks;
|
|
|
|
static struct mtx kstack_cache_mtx;
|
2012-10-26 17:31:35 +00:00
|
|
|
MTX_SYSINIT(kstack_cache, &kstack_cache_mtx, "kstkch", MTX_DEF);
|
|
|
|
|
2009-09-01 11:41:51 +00:00
|
|
|
SYSCTL_INT(_vm, OID_AUTO, kstack_cache_size, CTLFLAG_RW, &kstack_cache_size, 0,
|
|
|
|
"");
|
|
|
|
SYSCTL_INT(_vm, OID_AUTO, kstacks, CTLFLAG_RD, &kstacks, 0,
|
|
|
|
"");
|
|
|
|
|
2003-06-14 23:23:55 +00:00
|
|
|
/*
|
|
|
|
* Create the kernel stack (including pcb for i386) for a new thread.
|
|
|
|
* This routine directly affects the fork perf for a process and
|
|
|
|
* create performance for a thread.
|
|
|
|
*/
|
2007-11-05 11:36:16 +00:00
|
|
|
int
|
2003-06-14 23:23:55 +00:00
|
|
|
vm_thread_new(struct thread *td, int pages)
|
|
|
|
{
|
|
|
|
vm_object_t ksobj;
|
|
|
|
vm_offset_t ks;
|
2017-08-09 04:23:04 +00:00
|
|
|
vm_page_t ma[KSTACK_MAX_PAGES];
|
2009-09-01 11:41:51 +00:00
|
|
|
struct kstack_cache_entry *ks_ce;
|
2003-06-14 23:23:55 +00:00
|
|
|
int i;
|
|
|
|
|
|
|
|
/* Bounds check */
|
|
|
|
if (pages <= 1)
|
2015-08-10 17:18:21 +00:00
|
|
|
pages = kstack_pages;
|
2003-06-14 23:23:55 +00:00
|
|
|
else if (pages > KSTACK_MAX_PAGES)
|
|
|
|
pages = KSTACK_MAX_PAGES;
|
2009-09-01 11:41:51 +00:00
|
|
|
|
2015-08-10 17:18:21 +00:00
|
|
|
if (pages == kstack_pages) {
|
2009-09-01 11:41:51 +00:00
|
|
|
mtx_lock(&kstack_cache_mtx);
|
|
|
|
if (kstack_cache != NULL) {
|
|
|
|
ks_ce = kstack_cache;
|
|
|
|
kstack_cache = ks_ce->next_ks_entry;
|
|
|
|
mtx_unlock(&kstack_cache_mtx);
|
|
|
|
|
|
|
|
td->td_kstack_obj = ks_ce->ksobj;
|
|
|
|
td->td_kstack = (vm_offset_t)ks_ce;
|
2015-08-10 17:18:21 +00:00
|
|
|
td->td_kstack_pages = kstack_pages;
|
2009-09-01 11:41:51 +00:00
|
|
|
return (1);
|
|
|
|
}
|
|
|
|
mtx_unlock(&kstack_cache_mtx);
|
|
|
|
}
|
|
|
|
|
2003-06-14 23:23:55 +00:00
|
|
|
/*
|
|
|
|
* Allocate an object for the kstack.
|
|
|
|
*/
|
|
|
|
ksobj = vm_object_allocate(OBJT_DEFAULT, pages);
|
2008-03-19 06:19:01 +00:00
|
|
|
|
2003-06-14 23:23:55 +00:00
|
|
|
/*
|
|
|
|
* Get a kernel virtual address for this thread's kstack.
|
|
|
|
*/
|
2010-04-18 22:32:07 +00:00
|
|
|
#if defined(__mips__)
|
|
|
|
/*
|
|
|
|
* We need to align the kstack's mapped address to fit within
|
|
|
|
* a single TLB entry.
|
|
|
|
*/
|
2013-08-07 06:21:20 +00:00
|
|
|
if (vmem_xalloc(kernel_arena, (pages + KSTACK_GUARD_PAGES) * PAGE_SIZE,
|
|
|
|
PAGE_SIZE * 2, 0, 0, VMEM_ADDR_MIN, VMEM_ADDR_MAX,
|
|
|
|
M_BESTFIT | M_NOWAIT, &ks)) {
|
|
|
|
ks = 0;
|
|
|
|
}
|
2010-04-18 22:32:07 +00:00
|
|
|
#else
|
2013-08-07 06:21:20 +00:00
|
|
|
ks = kva_alloc((pages + KSTACK_GUARD_PAGES) * PAGE_SIZE);
|
2010-04-18 22:32:07 +00:00
|
|
|
#endif
|
2007-11-05 11:36:16 +00:00
|
|
|
if (ks == 0) {
|
|
|
|
printf("vm_thread_new: kstack allocation failed\n");
|
|
|
|
vm_object_deallocate(ksobj);
|
|
|
|
return (0);
|
|
|
|
}
|
2009-09-01 11:41:51 +00:00
|
|
|
|
|
|
|
atomic_add_int(&kstacks, 1);
|
2003-06-14 23:23:55 +00:00
|
|
|
if (KSTACK_GUARD_PAGES != 0) {
|
|
|
|
pmap_qremove(ks, KSTACK_GUARD_PAGES);
|
|
|
|
ks += KSTACK_GUARD_PAGES * PAGE_SIZE;
|
|
|
|
}
|
2007-11-05 11:36:16 +00:00
|
|
|
td->td_kstack_obj = ksobj;
|
2003-06-14 23:23:55 +00:00
|
|
|
td->td_kstack = ks;
|
|
|
|
/*
|
|
|
|
* Knowing the number of pages allocated is useful when you
|
|
|
|
* want to deallocate them.
|
|
|
|
*/
|
|
|
|
td->td_kstack_pages = pages;
|
|
|
|
/*
|
|
|
|
* For the length of the stack, link in a real page of ram for each
|
|
|
|
* page of stack.
|
|
|
|
*/
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_WLOCK(ksobj);
|
2017-08-11 16:29:22 +00:00
|
|
|
(void)vm_page_grab_pages(ksobj, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOBUSY |
|
2017-08-09 04:23:04 +00:00
|
|
|
VM_ALLOC_WIRED, ma, pages);
|
|
|
|
for (i = 0; i < pages; i++)
|
|
|
|
ma[i]->valid = VM_PAGE_BITS_ALL;
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_WUNLOCK(ksobj);
|
2003-06-14 23:23:55 +00:00
|
|
|
pmap_qenter(ks, ma, pages);
|
2007-11-05 11:36:16 +00:00
|
|
|
return (1);
|
2003-06-14 23:23:55 +00:00
|
|
|
}
|
|
|
|
|
2009-09-01 11:41:51 +00:00
|
|
|
static void
|
|
|
|
vm_thread_stack_dispose(vm_object_t ksobj, vm_offset_t ks, int pages)
|
2003-06-14 23:23:55 +00:00
|
|
|
{
|
|
|
|
vm_page_t m;
|
2009-09-01 11:41:51 +00:00
|
|
|
int i;
|
2003-06-14 23:23:55 +00:00
|
|
|
|
2009-09-01 11:41:51 +00:00
|
|
|
atomic_add_int(&kstacks, -1);
|
2003-06-14 23:23:55 +00:00
|
|
|
pmap_qremove(ks, pages);
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_WLOCK(ksobj);
|
2003-06-14 23:23:55 +00:00
|
|
|
for (i = 0; i < pages; i++) {
|
|
|
|
m = vm_page_lookup(ksobj, i);
|
|
|
|
if (m == NULL)
|
|
|
|
panic("vm_thread_dispose: kstack already missing?");
|
2010-04-30 00:46:43 +00:00
|
|
|
vm_page_lock(m);
|
2015-10-09 03:38:58 +00:00
|
|
|
vm_page_unwire(m, PQ_NONE);
|
2003-06-14 23:23:55 +00:00
|
|
|
vm_page_free(m);
|
2010-04-30 00:46:43 +00:00
|
|
|
vm_page_unlock(m);
|
2003-06-14 23:23:55 +00:00
|
|
|
}
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_WUNLOCK(ksobj);
|
2003-06-14 23:23:55 +00:00
|
|
|
vm_object_deallocate(ksobj);
|
2013-08-07 06:21:20 +00:00
|
|
|
kva_free(ks - (KSTACK_GUARD_PAGES * PAGE_SIZE),
|
2003-06-14 23:23:55 +00:00
|
|
|
(pages + KSTACK_GUARD_PAGES) * PAGE_SIZE);
|
2009-09-01 11:41:51 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Dispose of a thread's kernel stack.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_thread_dispose(struct thread *td)
|
|
|
|
{
|
|
|
|
vm_object_t ksobj;
|
|
|
|
vm_offset_t ks;
|
|
|
|
struct kstack_cache_entry *ks_ce;
|
|
|
|
int pages;
|
|
|
|
|
|
|
|
pages = td->td_kstack_pages;
|
|
|
|
ksobj = td->td_kstack_obj;
|
|
|
|
ks = td->td_kstack;
|
2007-11-05 11:36:16 +00:00
|
|
|
td->td_kstack = 0;
|
2009-09-01 11:41:51 +00:00
|
|
|
td->td_kstack_pages = 0;
|
2015-08-10 17:18:21 +00:00
|
|
|
if (pages == kstack_pages && kstacks <= kstack_cache_size) {
|
2009-09-01 11:41:51 +00:00
|
|
|
ks_ce = (struct kstack_cache_entry *)ks;
|
|
|
|
ks_ce->ksobj = ksobj;
|
|
|
|
mtx_lock(&kstack_cache_mtx);
|
|
|
|
ks_ce->next_ks_entry = kstack_cache;
|
|
|
|
kstack_cache = ks_ce;
|
|
|
|
mtx_unlock(&kstack_cache_mtx);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
vm_thread_stack_dispose(ksobj, ks, pages);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
vm_thread_stack_lowmem(void *nulll)
|
|
|
|
{
|
|
|
|
struct kstack_cache_entry *ks_ce, *ks_ce1;
|
|
|
|
|
|
|
|
mtx_lock(&kstack_cache_mtx);
|
|
|
|
ks_ce = kstack_cache;
|
|
|
|
kstack_cache = NULL;
|
|
|
|
mtx_unlock(&kstack_cache_mtx);
|
|
|
|
|
|
|
|
while (ks_ce != NULL) {
|
|
|
|
ks_ce1 = ks_ce;
|
|
|
|
ks_ce = ks_ce->next_ks_entry;
|
|
|
|
|
|
|
|
vm_thread_stack_dispose(ks_ce1->ksobj, (vm_offset_t)ks_ce1,
|
2015-08-10 17:18:21 +00:00
|
|
|
kstack_pages);
|
2009-09-01 11:41:51 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
kstack_cache_init(void *nulll)
|
|
|
|
{
|
|
|
|
|
|
|
|
EVENTHANDLER_REGISTER(vm_lowmem, vm_thread_stack_lowmem, NULL,
|
|
|
|
EVENTHANDLER_PRI_ANY);
|
2003-06-14 23:23:55 +00:00
|
|
|
}
|
|
|
|
|
2009-09-01 11:41:51 +00:00
|
|
|
SYSINIT(vm_kstacks, SI_SUB_KTHREAD_INIT, SI_ORDER_ANY, kstack_cache_init, NULL);
|
|
|
|
|
2014-10-04 18:38:14 +00:00
|
|
|
#ifdef KSTACK_USAGE_PROF
|
|
|
|
/*
|
|
|
|
* Track maximum stack used by a thread in kernel.
|
|
|
|
*/
|
|
|
|
static int max_kstack_used;
|
|
|
|
|
|
|
|
SYSCTL_INT(_debug, OID_AUTO, max_kstack_used, CTLFLAG_RD,
|
|
|
|
&max_kstack_used, 0,
|
|
|
|
"Maxiumum stack depth used by a thread in kernel");
|
|
|
|
|
|
|
|
void
|
|
|
|
intr_prof_stack_use(struct thread *td, struct trapframe *frame)
|
|
|
|
{
|
|
|
|
vm_offset_t stack_top;
|
|
|
|
vm_offset_t current;
|
|
|
|
int used, prev_used;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Testing for interrupted kernel mode isn't strictly
|
|
|
|
* needed. It optimizes the execution, since interrupts from
|
|
|
|
* usermode will have only the trap frame on the stack.
|
|
|
|
*/
|
|
|
|
if (TRAPF_USERMODE(frame))
|
|
|
|
return;
|
|
|
|
|
|
|
|
stack_top = td->td_kstack + td->td_kstack_pages * PAGE_SIZE;
|
|
|
|
current = (vm_offset_t)(uintptr_t)&stack_top;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Try to detect if interrupt is using kernel thread stack.
|
|
|
|
* Hardware could use a dedicated stack for interrupt handling.
|
|
|
|
*/
|
|
|
|
if (stack_top <= current || current < td->td_kstack)
|
|
|
|
return;
|
|
|
|
|
|
|
|
used = stack_top - current;
|
|
|
|
for (;;) {
|
|
|
|
prev_used = max_kstack_used;
|
|
|
|
if (prev_used >= used)
|
|
|
|
break;
|
|
|
|
if (atomic_cmpset_int(&max_kstack_used, prev_used, used))
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#endif /* KSTACK_USAGE_PROF */
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* Implement fork's actions on an address space.
|
|
|
|
* Here we arrange for the address space to be copied or referenced,
|
|
|
|
* allocate a user struct (pcb and kernel stack), then call the
|
|
|
|
* machine-dependent layer to fill those in and make the new process
|
1997-04-07 07:16:06 +00:00
|
|
|
* ready to run. The new process is set up so that it returns directly
|
|
|
|
* to user mode to avoid stack copying and relocation problems.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2007-11-05 11:36:16 +00:00
|
|
|
int
|
2018-01-12 03:50:44 +00:00
|
|
|
vm_forkproc(struct thread *td, struct proc *p2, struct thread *td2,
|
|
|
|
struct vmspace *vm2, int flags)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2001-09-12 08:38:13 +00:00
|
|
|
struct proc *p1 = td->td_proc;
|
2007-11-05 11:36:16 +00:00
|
|
|
int error;
|
1994-05-25 09:21:21 +00:00
|
|
|
|
1999-12-06 04:53:08 +00:00
|
|
|
if ((flags & RFPROC) == 0) {
|
|
|
|
/*
|
|
|
|
* Divorce the memory, if it is shared, essentially
|
|
|
|
* this changes shared memory amongst threads, into
|
|
|
|
* COW locally.
|
|
|
|
*/
|
|
|
|
if ((flags & RFMEM) == 0) {
|
|
|
|
if (p1->p_vmspace->vm_refcnt > 1) {
|
2007-11-05 11:36:16 +00:00
|
|
|
error = vmspace_unshare(p1);
|
|
|
|
if (error)
|
|
|
|
return (error);
|
1999-12-06 04:53:08 +00:00
|
|
|
}
|
|
|
|
}
|
2002-02-07 20:58:47 +00:00
|
|
|
cpu_fork(td, p2, td2, flags);
|
2007-11-05 11:36:16 +00:00
|
|
|
return (0);
|
1999-12-06 04:53:08 +00:00
|
|
|
}
|
|
|
|
|
1997-04-13 01:48:35 +00:00
|
|
|
if (flags & RFMEM) {
|
|
|
|
p2->p_vmspace = p1->p_vmspace;
|
2004-07-27 03:53:41 +00:00
|
|
|
atomic_add_int(&p1->p_vmspace->vm_refcnt, 1);
|
1997-04-13 01:48:35 +00:00
|
|
|
}
|
|
|
|
|
1999-09-17 04:56:40 +00:00
|
|
|
while (vm_page_count_severe()) {
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_wait_severe();
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1997-04-13 01:48:35 +00:00
|
|
|
if ((flags & RFMEM) == 0) {
|
2007-11-05 11:36:16 +00:00
|
|
|
p2->p_vmspace = vm2;
|
1997-04-07 07:16:06 +00:00
|
|
|
if (p1->p_vmspace->vm_shm)
|
|
|
|
shmfork(p1, p2);
|
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
1997-04-07 07:16:06 +00:00
|
|
|
* cpu_fork will copy and update the pcb, set up the kernel stack,
|
|
|
|
* and make the child ready to run.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2002-02-07 20:58:47 +00:00
|
|
|
cpu_fork(td, p2, td2, flags);
|
2007-11-05 11:36:16 +00:00
|
|
|
return (0);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2001-09-10 04:28:58 +00:00
|
|
|
|
|
|
|
/*
|
2016-05-02 20:16:29 +00:00
|
|
|
* Called after process has been wait(2)'ed upon and is being reaped.
|
2001-09-10 04:28:58 +00:00
|
|
|
* The idea is to reclaim resources that we could not reclaim while
|
|
|
|
* the process was still executing.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_waitproc(p)
|
|
|
|
struct proc *p;
|
|
|
|
{
|
|
|
|
|
2002-02-05 21:23:05 +00:00
|
|
|
vmspace_exitfree(p); /* and clean-out the vmspace */
|
2001-09-10 04:28:58 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
|
If a thread that is swapped out is made runnable, then the setrunnable()
routine wakes up proc0 so that proc0 can swap the thread back in.
Historically, this has been done by waking up proc0 directly from
setrunnable() itself via a wakeup(). When waking up a sleeping thread
that was swapped out (the usual case when waking proc0 since only sleeping
threads are eligible to be swapped out), this resulted in a bit of
recursion (e.g. wakeup() -> setrunnable() -> wakeup()).
With sleep queues having separate locks in 6.x and later, this caused a
spin lock LOR (sleepq lock -> sched_lock/thread lock -> sleepq lock).
An attempt was made to fix this in 7.0 by making the proc0 wakeup use
the ithread mechanism for doing the wakeup. However, this required
grabbing proc0's thread lock to perform the wakeup. If proc0 was asleep
elsewhere in the kernel (e.g. waiting for disk I/O), then this degenerated
into the same LOR since the thread lock would be some other sleepq lock.
Fix this by deferring the wakeup of the swapper until after the sleepq
lock held by the upper layer has been locked. The setrunnable() routine
now returns a boolean value to indicate whether or not proc0 needs to be
woken up. The end result is that consumers of the sleepq API such as
*sleep/wakeup, condition variables, sx locks, and lockmgr, have to wakeup
proc0 if they get a non-zero return value from sleepq_abort(),
sleepq_broadcast(), or sleepq_signal().
Discussed with: jeff
Glanced at by: sam
Tested by: Jurgen Weber jurgen - ish com au
MFC after: 2 weeks
2008-08-05 20:02:31 +00:00
|
|
|
void
|
|
|
|
kick_proc0(void)
|
2005-05-23 23:01:53 +00:00
|
|
|
{
|
|
|
|
|
If a thread that is swapped out is made runnable, then the setrunnable()
routine wakes up proc0 so that proc0 can swap the thread back in.
Historically, this has been done by waking up proc0 directly from
setrunnable() itself via a wakeup(). When waking up a sleeping thread
that was swapped out (the usual case when waking proc0 since only sleeping
threads are eligible to be swapped out), this resulted in a bit of
recursion (e.g. wakeup() -> setrunnable() -> wakeup()).
With sleep queues having separate locks in 6.x and later, this caused a
spin lock LOR (sleepq lock -> sched_lock/thread lock -> sleepq lock).
An attempt was made to fix this in 7.0 by making the proc0 wakeup use
the ithread mechanism for doing the wakeup. However, this required
grabbing proc0's thread lock to perform the wakeup. If proc0 was asleep
elsewhere in the kernel (e.g. waiting for disk I/O), then this degenerated
into the same LOR since the thread lock would be some other sleepq lock.
Fix this by deferring the wakeup of the swapper until after the sleepq
lock held by the upper layer has been locked. The setrunnable() routine
now returns a boolean value to indicate whether or not proc0 needs to be
woken up. The end result is that consumers of the sleepq API such as
*sleep/wakeup, condition variables, sx locks, and lockmgr, have to wakeup
proc0 if they get a non-zero return value from sleepq_abort(),
sleepq_broadcast(), or sleepq_signal().
Discussed with: jeff
Glanced at by: sam
Tested by: Jurgen Weber jurgen - ish com au
MFC after: 2 weeks
2008-08-05 20:02:31 +00:00
|
|
|
wakeup(&proc0);
|
|
|
|
}
|