This patch fixes UTF-8 sequence validation logic in
teken_utf8_bytes_to_codepoint() and fixes fallback behaviour in
ttydisc_rubchar() when an invalid UTF8 sequence is encountered. The code
previously used __bitcount() to extract sequence length information from
the leading byte. However, this assumption breaks for certain code
points that have additional bits set in the first half of the leading
byte (e.g. Cyrillic characters). This lead to incorrect behaviour when
deleting those characters using backspaces. The code now checks the
number of consecutive set bits in the leading byte starting from the
MSB, as per RFC 3629.
Reviewed by: christos
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D42147
The use of bitcount() triggered a build error because it couldn't be
located. __bitcount() on the other hand is defined in sys/types.h, which
is included in teken/teken.h.
MFC after: 2 weeks
This patch adds additional logic in ttydisc_rubchar() to properly handle
backspace behaviour for UTF-8 characters.
Currently, typing in a backspace after a UTF8 character will delete only
one byte from the byte sequence, leaving garbled output in the tty's
output queue. With this change all of the character's bytes are deleted.
This change is only active when the IUTF8 flag is set (see
19054eb6053189144aa962b2ecc1bf5087758a3e "(s)tty: add support for IUTF8
input flag")
The code uses the teken_wcwidth() function to properly handle character
column widths for different code points, and adds the
teken_utf8_bytes_to_codepoint() function that converts a UTF-8 byte
sequence to a codepoint, as specified in RFC3629.
Reported by: christos
Reviewed by: christos, imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D42067
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
The console escape code standard (ECMA-48) specifies color #3 (escape
code 33) as yellow. A brown console color is an artifact of the VGA
palette, which replaces dim (but not bright) yellow with brown.
Reviewed by: adrian, imp
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34531
Draw console on efi.
Add vbe framebuffer for BIOS loader (vbe off, vbe on, vbe list,
vbe set xxx).
autoload font (/boot/fonts) based on resolution and font size.
Add command loadfont (set font by file) and
variable screen.font (set font by size). Pass loaded font to kernel.
Export variables:
screen.height
screen.width
screen.depth
Add gfx primitives to draw the screen and put png image on the screen.
Rework menu draw to iterate list of consoles to enamble device specific
output.
Probably something else I forgot...
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D27420
Repeating the default WARNS here makes it slightly more difficult to
experiment with default WARNS changes, e.g. if we did something absolutely
bananas and introduced a WARNS=7 and wanted to try lifting the default to
that.
Drop most of them; there is one in the blake2 kernel module, but I suspect
it should be dropped -- the default WARNS in the rest of the build doesn't
currently apply to kernel modules, and I haven't put too much thought into
whether it makes sense to make it so.
The flag for the driver capability of supporting the fix is independent
of the flag for cons25 mode so that it can be managed independently, but
I forget to preserve it when resetting the terminal.
Right now it's possible to invoke the REP escape sequence with a maximum
of tens of millions of iterations. In practice, there is never any need
to do this. Calling it more frequently than the number of cells in the
terminal hardly makes any sense. By placing a limit on it, we can
prevent users from exhausting resources in inside the terminal emulator.
As support for this escape sequence is not present in any of the stable
branches, there is no need to MFC.
Reported by: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=11255
when TEKEN_CONS25 is configured. Fix this by adding a function to
set the flag that enables the fix and always calling this function
for syscons.
Expand the man page for teken_set_cons25(). This function is not
very useful since it can only set but not clear 1 flag. In practice,
it is only used when TEKEN_CONS25 is configured and all that does is
choose the the default emulation for syscons at compile time.
kbd(4) (but only documented in atkbd(4)) maintains a table of strings
for 96 function keys. Using teken broke this 9+ years ago for the
most usable first 12 function keys and for 10 cursor keys, by supplying
its own non-programmable strings so that the keyboard driver's strings
are not used.
Fix this by supplying NULL in the teken layer for syscons in cons25 mode
so that the the strings are found in the kbd(4) layer.
vt needs more changes to use kbd(4)'s tables. Teken's cons25 table is
still needed to supply nonempty strings for vt in cons25 mode.
Keep using teken's xterm tables for both syscons and vt in xterm mode.
Function keys should at least default to xterm values in xterm mode,
and kbd(4) doesn't support this.
teken_set_cons25() sets a sticky flag to ask for the fix, and space is
reserved for another new flag. vt should set this flag when it uses
kbd(4)'s tables.
PR: 226553 (for vt)
The escape sequence (e.g. `^[[2 q`) was unsupported before and the
letter `q` was displayed as a typed character. The sequence is used by
Neovim for instance.
Now, it is properly parsed. However, it is ignored, so it won't change
the cursor style.
Because the escape sequence contains a space character, the
`gensequences` script had to be modified to support that. In the
`sequences` file, a space is represented as the string `SP`.
... to process input, instead of inside each smaller operations such as
appending a character or moving the cursor forward.
In other words, before we were doing (oversimplified):
teken_input()
<for each input character>
vtterm_putchar()
VTBUF_LOCK()
VTBUF_UNLOCK()
vtterm_cursor_position()
VTBUF_LOCK()
VTBUF_UNLOCK()
Now, we are doing:
vtterm_pre_input()
VTBUF_LOCK()
teken_input()
<for each input character>
vtterm_putchar()
vtterm_cursor_position()
vtterm_post_input()
VTBUF_UNLOCK()
The situation was even worse when the vtterm_copy() and vtterm_fill()
callbacks were involved.
The new callbacks are:
* struct terminal_class->tc_pre_input()
* struct terminal_class->tc_post_input()
They are called in teken_input(), surrounding the while() loop.
The goal is to improve input processing speed of vt(4). As a benchmark,
here is the time taken to write a text file of 360 000 lines (26 MiB) on
`ttyv0`:
* vt(4), unmodified: 1500 ms
* vt(4), with this patch: 1200 ms
* syscons(4): 700 ms
This is on a Haswell laptop with a GENERIC-NODEBUG kernel.
At the same time, the locking is changed in the vt_flush() function
which is responsible to draw the text on screen. So instead of
(indirectly) using VTBUF_LOCK() just to read and reset the dirty area
of the internal buffer, the lock is held for about the entire function,
including the drawing part.
The change is mostly visible while content is scrolling fast: before,
lines could appear garbled while scrolling because the internal buffer
was accessed without locks (once the scrolling was finished, the output
was correct). Now, the scrolling appears correct.
In the end, the locking model is closer to what syscons(4) does.
Differential Revision: https://reviews.freebsd.org/D15302
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
No functional change intended.
was aliased to a vt sequence, causing and fixing various bugs.
For syscons, this restores support for arg 2 which sets blinking block
too forcefully, and restores bugs for arg 0 and 1. Arg 2 is used for
vs in the cons25 entry in termcap, but I've never noticed an application
that uses this. The bugs involve replacing local settings by global
ones and need better handling of defaults to fix.
For vt, this requires moving the aliasing code from teken to vt where
it belongs. This sequences is very important for cons25 compatibility
in vt since it is used by the cons25 termcap entries for ve, vi and
vs. vt can't properly support vs for either cons25 or xterm since it
doesn't support blinking. For xterm, the termcap entry for vs asks
for something different using 12;25h instead of 25h.
Rename C25CURS for this to C25LCT and change its description to be closer
to echoing the old comment about it. CURS is too generic.
Fix missing syscons escape sequence for setting the global cursor shape
(and type). Only support this in syscons since vt can't emulate anything
in it.
In a recent commit, I forgot to expand an X to an abbreviation of "BORDER".
Fix this and some nearby bad names.
The descriptions were copied from comments in scterm-sc.c, but some
of these are bad. The border [color] was inconsistently described as
a property of the "display", but I had changed this to "adapter" to
match the descriptions for other color settings. All colors supported
by the cons25 sequences are actually properties of the current vty and
that should not be described. But the other colors are defaults.
Change "adapter" to "default" for them and remove "adapter" for the
border. Reduce the verbosity of the abbreviation from AD to D.
that used to work via the bold hack).
Fix the table entry for bright black. Fix spelling of plain black in
nearby table entries (use the macro for black everywhere everywhere).
Fix the currently-unused non-bright color table to not have bright
colors in entries 9-15.
Improve nearby comments. Start converting to the xterm terminology
and default rendering of "bright" instead of "light" for bright
colors.
Syscons wasn't affected by the bug since I optimized it a little by
converting colors 0-15 directly. This also fixes the layering of
the conversion for these colors.
Apply the same optimization to vt (actually the layer above it). This
also moves the conversion 1 closer to the correct layer for colors
0-15.
The optimization of just avoiding 2 calls to a trivial function is worth
about 10% for simple output to the virtual buffer with occasional
rendering. The optimization is so large because the 2 calls are done
on every character, so although there are too many other calls and
other instructions per character, there are only about 10 times as
many. Old versions of syscons were about 10 times faster for simple
output, by using a fast path with about 12 instructions per character.
Rendering to even slow hardware takes relatively little time provided
it is rarely actually done.
for vt. Restore syscons' rendering of background (bg) brightness as
foreground (fg) blinking and vice versa, and add rendering of blinking
as background brightness to vt.
Bright/saturated is conflated with light/white in the implementation
and in this description.
Bright colors were broken in all cases, but appeared to work in the
only case shown by "vidcontrol show". A boldness hack was applied
only in 1 layering-violation place (for some syscons sequences) where
it made some cases seem to work but was undone by clearing bold using
ANSI sequences, and more seriously was not undone when setting
ANSI/xterm dark colors so left them bright. Move this hack to drivers.
The boldness hack is only for fg brightness. Restore/add a similar hack
for bg brightness rendered as fg blinking and vice versa. This works
even better for vt, since vt changes the default text mode to give the
more useful bg brightness instead of fg blinking.
The brightness bit in colors was unnecessarily removed by the boldness
hack. In other cases, it was lost later by teken_256to8(). Use
teken_256to16() to not lose it. teken_256to8() was intended to be
used for bg colors to allow finer or bg-specific control for the more
difficult reduction to 8; however, since 16 bg colors actually work
on VGA except in syscons text mode and the conversion isn't subtle
enough to significantly in that mode, teken_256to8() is not used now.
There are still bugs, especially in vidcontrol, if bright/blinking
background colors are set.
Restore XOR logic for bold/bright fg in syscons (don't change OR
logic for vt). Remove broken ifdef on FG_UNDERLINE and its wrong
or missing bit and restore the correct hard-coded bit. FG_UNDERLINE
is only for mono mode which is not really supported.
Restore XOR logic for blinking/bright bg in syscons (in vt, add
OR logic and render as bright bg). Remove related broken ifdef
on BG_BLINKING and its missing bit and restore the correct
hard-coded bit. The same bit means blinking or bright bg depending
on the mode, and we want to ignore the difference everywhere.
Simplify conversions of attributes in syscons. Don't pretend to
support bold fonts. Don't support unusual encodings of brightness.
It is as good as possible to map 16 VGA colors to 16 xterm-16
colors. E.g., VGA brown -> xterm-16 Olive will be converted back
to VGA brown, so we don't need to convert to xterm-256 Brown. Teken
cons25 compatibility code already does the same, and duplicates some
small tables. This is mostly for the sc -> te direction. The other
direction uses teken_256to16() which is too generic.
codes. This will be used to fix bright colors.
Improve teken_256to8(). Use a lookup table instead of calculations. The
calculations were inaccurate since they used indexes into the xterm-256
6x6x6 color map instead of actual xterm colors. Also, change the threshold
for converting to a primary color: require the primary's component to be
2 or more higher instead of just higher. This affects about 1/5 of the
table entries and gives uniformly distributed colors in the 6x6x6 submap
except for greys (35 entries each for red, green, blue, cyan, brown and
magenta, instead of approx. only 15 each for the mixed colors). Even
more mixed colors would be better for matching colors, but uniform
distribution is best for preserving contrast.
For teken_256to16(), bright colors are just the ones with luminosity >=
60%. These are actually light colors (more white instead of more
saturation), while xterm bright colors except for white itself are
actually saturated with no white, so have luminosity only 50%.
These functions are layering violations. teken cannot do correct
conversions since it shouldn't know the color maps of anything except
xterm. Translating through xterm-16 colors loses information. This
gives bugs like xterm-256 near-brown -> xterm-16 red -> VGA red.
After fixing the 16-bits integer arithmetic overflow in 286981, we
should also make sure to fix the VPA sequence. Bring HPA and VPA in sync
with how we now implement CUP.
PR: 202612
Reported by: kcwu csie org
MFC after: 1 month
When updating the row number when the cursor position escape sequence is
issued, we should make sure to store the intermediate result in a 32-bit
integer. If we fail to do this, the cursor may be set above the origin
region, which is bad.
This could cause libteken to crash when INVARIANTS is enabled, due to
the strict set of assertions that libteken has.
PR: 202540
Reported by: kcwu csie org
MFC after: 1 month
The fix that I applied in r286798 is already good, but it assumes that
sizeof(int) > sizeof(short). Express the upperbound in terms of
UINT_MAX. By dividing that by 100, we're sure that the resulting value
is never larger than approximately UINT_MAX / 10, which is safe.
PR: 202326
Discussed with: kcwu csie org
MFC after: 1 month
There is no need for us to support parsing values that are larger than
the maximum terminal window size. In this case that would be the maximum
of unsigned short.
The problem with parsing larger values is that they can cause integer
overflows when adjusting the cursor position, leading to all sorts of
failing assertions.
PR: 202326
Reported by: kcwu csie org
MFC after: 1 month
Instead of only wrapping when in the 'wrapped state', also force
wrapping when the character to be rendered does not fit on the line
anymore.
Tested by: lwhsu
Introduce a new formatting bit (TF_CJK_RIGHT) that is set when putting a
cell that is the right part of a CJK fullwidth character. This will
allow drivers like vt(9) to support fullwidth characters properly.
emaste@ has a patch to extend vt(9)'s font handling to increase the
number of Unicode -> glyph maps from 2 ({normal,bold)} to 4
({normal,bold} x {left,right}). This will need to use this formatting
bit to determine whether to draw the left or right glyph.
Reviewed by: emaste
