Commit Graph

80 Commits

Author SHA1 Message Date
Russ Cox 28fd9fa8a6 unicode: add CategoryAliases, Cn, LC
CategoryAliases is for regexp to use, for things like \p{Letter} as an alias for \p{L}.
Cn and LC are special-case categories that were never implemented
but should have been.

These changes were generated by the updated generator in CL 641395.

Fixes #70780.

Change-Id: Ibba20ff76191c8ae9631ac5ba19965790fe0cc81
Reviewed-on: https://go-review.googlesource.com/c/go/+/641376
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
2025-04-18 14:13:31 -07:00
Jes Cok 1e92ff11f5 unicode/utf8: use builtin max function to simplify code
Change-Id: I6a73b645d074baaa4d09480bdf4192816a8c2450
GitHub-Last-Rev: 202d498eb0
GitHub-Pull-Request: golang/go#71945
Reviewed-on: https://go-review.googlesource.com/c/go/+/652177
Auto-Submit: Keith Randall <khr@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Keith Randall <khr@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
2025-02-25 10:16:36 -08:00
Luka Krmpotic 372f2d8022 unicode/utf8: remove init from utf8_test
TestConstants and init test the same thing, remove init,
it does not exist in utf16_test.go either.

Fixes #71579

Change-Id: Ie0afd640bebde822733b6eac0bf98a17872f4e5f
GitHub-Last-Rev: d7224c1837
GitHub-Pull-Request: golang/go#71582
Reviewed-on: https://go-review.googlesource.com/c/go/+/647335
Commit-Queue: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Keith Randall <khr@google.com>
Reviewed-by: qiu laidongfeng2 <2645477756@qq.com>
Auto-Submit: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2025-02-06 14:13:50 -08:00
Cuong Manh Le 3d33437c45 unicode/utf8: speedup RuneCount
CL 612617 did speedup RuneCountInString, thus we can now use it to
speedup RuneCount, too.

name                         old time/op    new time/op    delta
RuneCountTenASCIIChars-8       8.69ns ± 1%    3.59ns ± 2%  -58.66%  (p=0.000 n=9+9)
RuneCountTenJapaneseChars-8    49.8ns ± 2%    40.9ns ± 0%  -17.94%  (p=0.000 n=10+8)

Change-Id: I311750c00efc79af35fb0ca3b482a5d94e0a7977
Reviewed-on: https://go-review.googlesource.com/c/go/+/612955
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Tim King <taking@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-09-15 02:05:37 +00:00
Cuong Manh Le 9bd34ebb7e unicode/utf8: add test that RuneCount does zero allocations
See disccusion in CL 612955.

Change-Id: I2de582321648f1798929ffb80d2f087e41146ead
Reviewed-on: https://go-review.googlesource.com/c/go/+/613315
Commit-Queue: Ian Lance Taylor <iant@google.com>
Reviewed-by: Tim King <taking@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Cuong Manh Le <cuong.manhle.vn@gmail.com>
2024-09-15 02:04:55 +00:00
Cuong Manh Le 751fbf9e5a unicode/utf8: use range loop in RuneCountInString
CL 28490 speeded up non-ASCII rune decoding, and ASCII rune is also
decoded faster now.

Benchmark using:

    perflock -governor 70% go test -run=NONE -bench=BenchmarkRuneCountInString -count=10

Result:

name                                 old time/op  new time/op  delta
RuneCountInStringTenASCIIChars-8     10.2ns ± 0%   7.1ns ± 1%  -30.53%  (p=0.000 n=8+9)
RuneCountInStringTenJapaneseChars-8  49.3ns ± 2%  38.5ns ± 2%  -21.84%  (p=0.000 n=8+8)

Fixes #13162

Change-Id: Ifb01f3799c5c93e7f7c7af13a95becfde85ae807
Reviewed-on: https://go-review.googlesource.com/c/go/+/612617
Reviewed-by: Tim King <taking@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Cuong Manh Le <cuong.manhle.vn@gmail.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Tim King <taking@google.com>
2024-09-12 19:10:46 +00:00
nlwkobe30 7cd0a4be5c all: omit unnecessary 0 in slice expression
All changes are related to the code, except for the comments in src/regexp/syntax/parse.go and src/slices/slices.go.

Change-Id: I73c5d3c54099749b62210aa7f3182c5eb84bb6a6
GitHub-Last-Rev: 794aa9b053
GitHub-Pull-Request: golang/go#69170
Reviewed-on: https://go-review.googlesource.com/c/go/+/609678
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
2024-09-03 20:55:15 +00:00
Charlie Vieth b8e533a7cd unicode: improve SimpleFold performance by 2x for non-foldable code points
Change SimpleFold to search the CaseRanges table only once when no
folding is specified for the rune (previously up to two searches could
be performed). This improves performance by 2x for runes that have no
folds or are already upper case. As a side effect this improves the
performance of To by roughly ~15%

goos: darwin
goarch: arm64
pkg: unicode
cpu: Apple M1 Max
                     │ base.10.txt  │             new.10.txt              │
                     │    sec/op    │   sec/op     vs base                │
ToUpper-10             11.860n ± 1%   9.731n ± 1%  -17.95% (p=0.000 n=10)
ToLower-10              12.31n ± 1%   10.34n ± 1%  -16.00% (p=0.000 n=10)
SimpleFold/Upper-10     19.16n ± 0%   15.98n ± 1%  -16.64% (p=0.000 n=10)
SimpleFold/Lower-10     32.41n ± 1%   17.09n ± 1%  -47.27% (p=0.000 n=10)
SimpleFold/Fold-10      8.884n ± 4%   8.856n ± 8%        ~ (p=0.700 n=10)
SimpleFold/NoFold-10    30.87n ± 0%   15.49n ± 3%  -49.84% (p=0.000 n=10)
geomean                 17.09n        12.47n       -26.99%

Change-Id: I6e5c7554106842955aadeef7b266c4c7944d3a97
Reviewed-on: https://go-review.googlesource.com/c/go/+/454958
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
2024-09-03 20:54:56 +00:00
apocelipes c0eac35a4c bytes,strings,unicode/utf16: use slices to clean up tests
Replace reflect.DeepEqual with slices.Equal, which is much faster.
Remove some redundant helper functions.

Change-Id: I51b32a3d0c3fc5ad0d3b6ff0dd03f39c507e5762
GitHub-Last-Rev: e21f46d4a0
GitHub-Pull-Request: golang/go#67609
Reviewed-on: https://go-review.googlesource.com/c/go/+/587937
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Commit-Queue: Ian Lance Taylor <iant@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Robert Griesemer <gri@google.com>
2024-07-24 18:45:08 +00:00
Diego Augusto Molina 20e18c9550 unicode/utf8: AppendRune and EncodeRune performance improvement
- Prefer the evaluation of the valid higher byte-width runes branches
over the one for invalid ones
- Avoid the evaluation of the bytes of the RuneError constant, and
instead hard code its byte values
- EncodeRune only: inline for fast handling of ASCII

goos: linux
goarch: amd64
pkg: unicode/utf8
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
                              │ baseline.test-append.txt │ append-rune-invalid-case-last.test.txt │
                              │          sec/op          │     sec/op      vs base                │
AppendASCIIRune-8                           0.2135n ± 0%     0.2135n ± 0%        ~ (p=0.578 n=20)
AppendSpanishRune-8                          1.645n ± 1%      1.509n ± 2%   -8.30% (p=0.000 n=20)
AppendJapaneseRune-8                         2.196n ± 1%      2.004n ± 1%   -8.74% (p=0.000 n=20)
AppendMaxRune-8                              2.670n ± 1%      2.349n ± 3%  -12.01% (p=0.000 n=20)
AppendInvalidRuneMaxPlusOne-8                2.214n ± 2%      1.798n ± 3%  -18.77% (p=0.000 n=20)
AppendInvalidRuneSurrogate-8                 2.258n ± 1%      1.793n ± 2%  -20.59% (p=0.000 n=20)
AppendInvalidRuneNegative-8                  2.171n ± 2%      1.767n ± 2%  -18.61% (p=0.000 n=20)
geomean                                      1.559n           1.361n       -12.69%

goos: linux
goarch: amd64
pkg: unicode/utf8
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
                              │ baseline.test-encode.txt │     encode-rune-invalid-last.txt     │
                              │          sec/op          │    sec/op     vs base                │
EncodeASCIIRune-8                           1.0950n ± 1%   0.2140n ± 0%  -80.46% (p=0.000 n=20)
EncodeSpanishRune-8                          1.499n ± 0%    1.414n ± 2%   -5.64% (p=0.000 n=20)
EncodeJapaneseRune-8                         1.960n ± 2%    1.716n ± 4%  -12.43% (p=0.000 n=20)
EncodeMaxRune-8                              2.145n ± 2%    2.227n ± 1%   +3.78% (p=0.000 n=20)
EncodeInvalidRuneMaxPlusOne-8                1.955n ± 2%    1.802n ± 2%   -7.80% (p=0.000 n=20)
EncodeInvalidRuneSurrogate-8                 1.946n ± 3%    1.777n ± 2%   -8.68% (p=0.000 n=20)
EncodeInvalidRuneNegative-8                  1.968n ± 2%    1.766n ± 2%  -10.29% (p=0.000 n=20)
geomean                                      1.757n         1.308n       -25.57%

Fixes #68131

Change-Id: Ibcafa75d63cca07a2e78cd06f6f1e382cb8c716e
Reviewed-on: https://go-review.googlesource.com/c/go/+/594115
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
2024-07-22 21:20:30 +00:00
Jes Cok 69583738eb unicode/utf8: update doc for RuneLen
As CL 569755 did, for consistency, this CL slightly improves
the documentation for RuneLen.

Change-Id: Ic9776648baf2809af36cd16a94d1313938bb0e52
Reviewed-on: https://go-review.googlesource.com/c/go/+/569816
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Commit-Queue: Ian Lance Taylor <iant@google.com>
2024-03-08 02:00:36 +00:00
Jes Cok ef4f2a0597 unicode/utf16: add func RuneLen
This CL adds func RuneLen, while here, also uses RuneLen to simplify
code in Encode.

Fixes #44940

Change-Id: Ifd3b537f69880dfd32a69a6733d8d3c2b5d4ecba
Reviewed-on: https://go-review.googlesource.com/c/go/+/569755
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Commit-Queue: Ian Lance Taylor <iant@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
2024-03-07 19:08:48 +00:00
cui fliter 954a9630c9 unicode: add available godoc link
Change-Id: I2273274249f05b0492950c27dc5a654422cefc79
Reviewed-on: https://go-review.googlesource.com/c/go/+/539856
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Run-TryBot: shuang cui <imcusg@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
2023-11-06 20:02:46 +00:00
Jes Cok 267323ef2d all: calculate the median uniformly
Like sort.Search, use "h := int(uint(i+j) >> 1)" style code to calculate
the median.

Change-Id: Ifb1a19dde1c6ed6c1654bc642fc9565a8b6c5fc4
GitHub-Last-Rev: e2213b7388
GitHub-Pull-Request: golang/go#62503
Reviewed-on: https://go-review.googlesource.com/c/go/+/526496
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2023-09-09 01:46:03 +00:00
weebney def0be5e34 all: upgrade Unicode from 13.0.0 to 15.0.0
Update unicode/tables.go to reflect changes in the Unicode Standard up to
Unicode 15.0.0, released 13 Sept 2022.

In order to accommodate this update, strconv/isPrint has been updated to
reflect changes in printable characters.

Also changed is template/exec_test.go for both text and html packages- in
the test "TestJSEscaping", rune U+FDFF was used as a placeholder for an
unprintable character. This codepoint was assigned and made printable in
Unicode 14.0.0, breaking this test. It has been replaced with the assigned
and never-printable U+FFFE to fix the test and provide resiliency in the
future.

This upgrade bypasses Unicode 14.0.0, but is compatible.

Updates https://github.com/golang/go/issues/48621
Fixes https://github.com/golang/go/issues/55079

Change-Id: I40efd097eb746db0727ebf7437280916d1242e47
GitHub-Last-Rev: c8885cab7a
GitHub-Pull-Request: golang/go#57265
Reviewed-on: https://go-review.googlesource.com/c/go/+/456837
Reviewed-by: Robert Griesemer <gri@google.com>
Run-TryBot: Rob Pike <r@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Robert Griesemer <gri@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Rob Pike <r@golang.org>
2023-02-06 04:29:53 +00:00
qmuntal 2423370136 utf16: reduce utf16.Decode allocations
This CL avoids allocating in utf16.Decode for code point sequences
with less than 64 elements. It does so by splitting the function in two,
one that can be inlined that preallocates a buffer and the other that
does the heavy-lifting.

The mid-stack inliner will allocate the buffer in the caller stack,
and in many cases this will be enough to avoid the allocation.

unicode/utf16 benchmarks:

name                         old time/op    new time/op    delta
DecodeValidASCII-12            60.1ns ± 3%    16.0ns ±20%   -73.40%  (p=0.000 n=8+10)
DecodeValidJapaneseChars-12    61.3ns ±10%    14.9ns ±39%   -75.71%  (p=0.000 n=10+10)

name                         old alloc/op   new alloc/op   delta
DecodeValidASCII-12             48.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)
DecodeValidJapaneseChars-12     48.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)

name                         old allocs/op  new allocs/op  delta
DecodeValidASCII-12              1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
DecodeValidJapaneseChars-12      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)

I've also benchmarked os.File.ReadDir with this change applied
to demonstrate that it does make a difference in the caller site, in this
case via syscall.UTF16ToString:

name        old time/op    new time/op    delta
ReadDir-12     592µs ± 8%     620µs ±16%     ~     (p=0.280 n=10+10)

name        old alloc/op   new alloc/op   delta
ReadDir-12    30.4kB ± 0%    22.4kB ± 0%  -26.10%  (p=0.000 n=8+10)

name        old allocs/op  new allocs/op  delta
ReadDir-12       402 ± 0%       272 ± 0%  -32.34%  (p=0.000 n=10+10)

Change-Id: I65cf5caa3fd3b3a466c0ed837a50a96e975bbe6b
Reviewed-on: https://go-review.googlesource.com/c/go/+/453415
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
2023-01-23 20:59:01 +00:00
cuiweixie 6a2ff105f1 unicode/utf8: use strings.Builder
Change-Id: I88b55f61eccb5764cac2a9397fd99a62f8735a9a
Reviewed-on: https://go-review.googlesource.com/c/go/+/428281
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
2022-09-07 06:17:15 +00:00
qmuntal 48297f1fb1 unicode/utf16: add AppendRune
AppendRune appends the UTF-16 encoding of a rune to a []uint16.

    BenchmarkEncodeValidASCII-12                24.61ns 16B 1allocs
    BenchmarkEncodeValidJapaneseChars-12        18.79ns 8B  1allocs
    BenchmarkAppendRuneValidASCII-12            6.826ns 0B  0allocs
    BenchmarkAppendRuneValidJapaneseChars-12    3.547ns 0B  0allocs

The ASCII case is written to be inlineable.

Fixes #51896

Change-Id: I593b1029f603297ef6e80e036f2fee2a0938d38d
Reviewed-on: https://go-review.googlesource.com/c/go/+/409054
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Joedian Reid <joedian@golang.org>
2022-08-19 16:45:08 +00:00
Russ Cox 19309779ac all: gofmt main repo
[This CL is part of a sequence implementing the proposal #51082.
The design doc is at https://go.dev/s/godocfmt-design.]

Run the updated gofmt, which reformats doc comments,
on the main repository. Vendored files are excluded.

For #51082.

Change-Id: I7332f099b60f716295fb34719c98c04eb1a85407
Reviewed-on: https://go-review.googlesource.com/c/go/+/384268
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2022-04-11 16:34:30 +00:00
Russ Cox 690ac4071f all: remove trailing blank doc comment lines
A future change to gofmt will rewrite

	// Doc comment.
	//
	func f()

to

	// Doc comment.
	func f()

Apply that change preemptively to all doc comments.

For #51082.

Change-Id: I4023e16cfb0729b64a8590f071cd92f17343081d
Reviewed-on: https://go-review.googlesource.com/c/go/+/384259
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-04-01 18:18:07 +00:00
Alan Donovan fd2e1e743a unicode/utf8: optimize Valid to parity with ValidString
The benchmarks added in this change revealed that ValidString
runs ~17% faster than Valid([]byte) on the ASCII prefix
of the input. Inspection of the assembly revealed that the
code generated for p[8:] required recomputing the slice capacity
to handle the cap=0 special case, which added an ADD -8 instruction.
By making len=cap, the capacity becomes a common subexpression
with the length, saving the ADD instruction.
(Thanks to khr for the tip.)

Incidentally, I tried a number of other optimizations but was
unable to make consistent gains across all benchmarks. The most
promising was to retain the bitmask of non-ASCII bytes from the
fast loop; the slow loop would shift it, and when it becomes zero,
return to the fast loop. This made the MostlyASCII benchmark 4x
faster, but made the other cases slower by up to 10%.

cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
benchmark                                   old ns/op     new ns/op     delta
BenchmarkValidTenASCIIChars-16              4.09          4.06          -0.85%
BenchmarkValid100KASCIIChars-16             9325          7747          -16.92%
BenchmarkValidTenJapaneseChars-16           27.0          27.2          +0.85%
BenchmarkValidLongMostlyASCII-16            57277         58361         +1.89%
BenchmarkValidLongJapanese-16               94002         93131         -0.93%
BenchmarkValidStringTenASCIIChars-16        4.15          4.07          -1.74%
BenchmarkValidString100KASCIIChars-16       7980          8019          +0.49%
BenchmarkValidStringTenJapaneseChars-16     26.0          25.9          -0.38%
BenchmarkValidStringLongMostlyASCII-16      58550         58006         -0.93%
BenchmarkValidStringLongJapanese-16         97964         100038        +2.12%

Change-Id: Ic9d585dedd9af83c27dd791ecd805150ac949f15
Reviewed-on: https://go-review.googlesource.com/c/go/+/375594
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Trust: Alex Rakoczy <alex@golang.org>
2022-03-02 11:33:18 +00:00
jiahua wang 75952abc6a unicode/utf8: add AppendRune Example
Also, correct TestAppendRune error message.

Change-Id: I3ca3ac7051af1ae6d449381b78efa86c2f6be8ac
Reviewed-on: https://go-review.googlesource.com/c/go/+/354529
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Trust: Robert Findley <rfindley@google.com>
Trust: Cherry Mui <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
2021-11-05 21:29:18 +00:00
Pedro Lopez Mareque 7cef83162b unicode: use IsSpace not IsUpper in IsSpace example test
Change-Id: Ie3017e5507f57cbb2ae9c8b737b378cef91fefeb
Reviewed-on: https://go-review.googlesource.com/c/go/+/354509
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com>
TryBot-Result: Go Bot <gobot@golang.org>
2021-10-08 00:18:29 +00:00
Pedro Lopez Mareque 4707a6c284 unicode: add examples for the Is functions
Change-Id: If4afe33985dc0a758db32564244095190b82e5c4
Reviewed-on: https://go-review.googlesource.com/c/go/+/353691
Trust: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Rob Pike <r@golang.org>
2021-10-06 23:19:13 +00:00
Joe Tsai f371b30f32 unicode/utf8: add AppendRune
AppendRune appends the UTF-8 encoding of a rune to a []byte.
It is a generally more user friendly than EncodeRune.

    EncodeASCIIRune-4     2.35ns ± 2%
    EncodeJapaneseRune-4  4.60ns ± 2%
    AppendASCIIRune-4     0.30ns ± 3%
    AppendJapaneseRune-4  4.70ns ± 2%

The ASCII case is written to be inlineable.

Fixes #47609

Change-Id: If4f71eedffd2bd4ef0d7f960cb55b41c637eec54
Reviewed-on: https://go-review.googlesource.com/c/go/+/345571
Trust: Joe Tsai <joetsai@digital-static.net>
Reviewed-by: Rob Pike <r@golang.org>
Run-TryBot: Rob Pike <r@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
2021-08-28 01:49:50 +00:00
David Benjamin 3780529255 unicode: correctly handle negative runes
Is and isExcludingLatin did not handle negative runes when dispatching
to is16. TestNegativeRune covers this along with the existing uint32
casts in IsGraphic, etc. (For tests, I picked the smallest non-Latin-1
code point in each range.)

Updates #43254

Change-Id: I17261b91f0d2b5b5125d19219411b45c480df74f
Reviewed-on: https://go-review.googlesource.com/c/go/+/280493
Run-TryBot: Rob Pike <r@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
2021-02-24 04:00:46 +00:00
Ainar Garipov a3868028ac unicode/utf8: document the handling of runes out of range in EncodeRune
Document the way EncodeRune currently handles runes which are
out of range.  Also add an example showing that behaviour.

Change-Id: I0f8e7645ae053474ec319085a2bb6d7f73bc137c
Reviewed-on: https://go-review.googlesource.com/c/go/+/255998
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Giovanni Bajo <rasky@develer.com>
Trust: Giovanni Bajo <rasky@develer.com>
Run-TryBot: Giovanni Bajo <rasky@develer.com>
TryBot-Result: Go Bot <gobot@golang.org>
2020-09-19 09:43:15 +00:00
eric fang a1762c2cc6 unicode/utf8: refactor benchmarks for FullRune function
BenchmarkFullASCIIRune tests the performance of function utf8.FullRune,
which will be inlined in BenchmarkFullASCIIRune. Since the return value
of FullRune is not referenced, it will be removed as dead code.

This CL makes the FullRune functions return value referenced by a global
variable to avoid this point. In addition, this CL adds one more benchmark
to cover more code paths, and puts them together as sub benchmarks of
BenchmarkFullRune.

Change-Id: I6e79f4c087adf70e351498a4b58d7482dcd1ec4a
Reviewed-on: https://go-review.googlesource.com/c/go/+/233979
Run-TryBot: eric fang <eric.fang@arm.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-09-10 20:25:45 +00:00
Marcel van Lohuizen 8ec5a052ec unicode: upgrade to Unicode 13.0.0
Fixes #40755

Change-Id: I14b3977317994095db8ae1bd873c174641209356
Reviewed-on: https://go-review.googlesource.com/c/go/+/248765
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-08-20 13:41:13 +00:00
Martin Möhrmann e93d5b5e05 unicode/utf8: optimize Valid and ValidString for ASCII checks
Add a fastpath that uses 32bit loads and compares to check
8 ASCII characters per loop iteration.

This avoids the overhead of comparing and branching
for every byte individually.

Combining two 32bit loads into an uint32 allows the same
code to be used for 32bit and 64bit platforms.

amd64 (Intel i7-3520M):
name                         old time/op  new time/op  delta
ValidTenASCIIChars           15.6ns ± 4%   8.5ns ±14%  -45.27%  (p=0.000 n=10+10)
ValidTenJapaneseChars        50.0ns ± 2%  52.7ns ±15%     ~     (p=0.469 n=10+10)
ValidStringTenASCIIChars     13.5ns ± 1%   7.9ns ± 5%  -41.56%  (p=0.000 n=10+10)
ValidStringTenJapaneseChars  46.3ns ± 2%  45.8ns ± 2%     ~     (p=0.085 n=10+10)

arm (Raspberry Pi 3):
name                         old time/op  new time/op  delta
ValidTenASCIIChars           87.5ns ± 0%  58.5ns ± 0%  -33.11%  (p=0.000 n=9+10)
ValidTenJapaneseChars         359ns ± 0%   384ns ± 0%   +6.96%  (p=0.000 n=10+9)
ValidStringTenASCIIChars     87.5ns ± 0%  57.5ns ± 0%  -34.31%  (p=0.000 n=10+10)
ValidStringTenJapaneseChars   356ns ± 0%   377ns ± 0%   +5.90%  (p=0.000 n=10+10)

Change-Id: I9da942bddb250ee1f0ef7aabb4a8cb48edd9053e
Reviewed-on: https://go-review.googlesource.com/c/go/+/228823
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-04-22 14:14:34 +00:00
Tim Cooper c6e8426386 all: fix typo in RuneSelf, runeSelf comments
Fixes #36396

Change-Id: I52190f450fa9ac52fbf4ecdc814e954dc29029cd
Reviewed-on: https://go-review.googlesource.com/c/go/+/213377
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-01-06 02:46:02 +00:00
Marcel van Lohuizen 79a00a3fe8 unicode: upgrade to Unicode 12
This does not include an upgrade of golang.org/x/net.
This is optional and best done as a separate CL.

Change-Id: Ifecc3fb6e3b7fe026b4ddefbe637186a3445b0bc
Reviewed-on: https://go-review.googlesource.com/c/go/+/204658
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2019-11-05 14:34:56 +00:00
Serhat Giydiren fc8bef06c4 unicode/utf8: add link to formal UTF-8 description.
Fixes #31590

Change-Id: I7fd6dcc5c34496776439ff0295f18b5fb5cb538a
Reviewed-on: https://go-review.googlesource.com/c/go/+/199141
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2019-10-06 00:33:51 +00:00
Marcel van Lohuizen 34fc6ced76 all: update to Unicode 11
This does *not* update the vendored tables.
A commit updating these tables should follow soon,

Mostly generated running UNICODE_VERSION=11.0.0 in x/text.

Manually updated next.txt file.

Updates golang/go#27945.

Change-Id: I939a01e235aeca898ee9afc99a531e7ad8444e12
Reviewed-on: https://go-review.googlesource.com/c/go/+/154420
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
2019-04-25 20:33:22 +00:00
Josh Bleecher Snyder 7596ad0b24 unicode/utf8: remove some bounds checks from DecodeRune
The compiler couldn't quite see that reading p[2] and p[3] was safe.
This change provides a few hints to help it.
First, make sz an int throughout, rather than just when checking the input length.
Second, use <= instead of == in later comparisons.

name                  old time/op  new time/op  delta
DecodeASCIIRune-8     2.62ns ± 3%  2.60ns ± 5%     ~     (p=0.126 n=18+19)
DecodeJapaneseRune-8  4.46ns ±10%  4.01ns ± 5%  -10.00%  (p=0.000 n=19+20)

Change-Id: I2f78a17e38156fbf8b0f5dd6c07c20d6a47e9209
Reviewed-on: https://go-review.googlesource.com/c/go/+/173662
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-25 00:01:23 +00:00
Josh Bleecher Snyder 2be64d3645 unicode/utf8: use binary literals
We were using hex literals and had the binary literal in a comment.
When I was working with this code, I always referred to the comment.
That's an indicator that we should just use the binary literal directly.

Updates #19308

Change-Id: I2279cb8efb4ae5f2e1558c15979058ab09eb4f6f
Reviewed-on: https://go-review.googlesource.com/c/go/+/173663
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-24 23:51:16 +00:00
Marcel van Lohuizen e308d41fe5 unicode: update table using new generator in x/text
The changes in Unicode 11 exposes a bug in maketables.go.
We update the Unicode 10 tables using a new generator
to minimize the changes upgrading to Unicode 11.

This change switches over the generation from core to that in
x/text. To properly update the tables one needs to run the generate
in x/text anyway, so this makes that official.

The RangeTable generator in x/text also generates slightly compacter
tables.

Updates golang/go#27945

See CL 154443

Change-Id: I6c59e082d5b8cd9e9332a32d8971061228581d66
Reviewed-on: https://go-review.googlesource.com/c/go/+/169617
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-24 15:22:37 +00:00
Marcel van Lohuizen 8a20fde697 unicode: remove script test
The script test requires a manual update on each new
Unicode release, which interupts the automated flow.
The test is removed in favor of one that fits within the
automated scripts.
See https://go-review.googlesource.com/c/text/+/169638.

Updates golang/go#27945

Change-Id: I9c9f1e2a77f7baef17040ea09eec28d7cf55b5e4
Reviewed-on: https://go-review.googlesource.com/c/go/+/169619
Reviewed-by: Russ Cox <rsc@golang.org>
2019-04-24 14:09:01 +00:00
Josh Bleecher Snyder 97dfbc54e6 unicode/utf8: make acceptRanges bigger
This avoids bounds checks in the calling code.
The nominal increased size of the array in the binary
is compensated for by the decreased size of the functions that call it.

The benchmark changes are a bit scattered, but overall positive.

name                                 old time/op  new time/op  delta
RuneCountTenASCIIChars-8             8.86ns ± 4%  7.93ns ± 5%  -10.45%  (p=0.000 n=45+49)
RuneCountTenJapaneseChars-8          38.2ns ± 2%  37.2ns ± 1%   -2.63%  (p=0.000 n=44+41)
RuneCountInStringTenASCIIChars-8     7.82ns ± 2%  8.70ns ± 2%  +11.19%  (p=0.000 n=43+43)
RuneCountInStringTenJapaneseChars-8  39.3ns ± 9%  40.0ns ± 5%   +1.59%  (p=0.043 n=50+50)
ValidTenASCIIChars-8                 8.68ns ± 5%  8.74ns ± 5%     ~     (p=0.070 n=50+48)
ValidTenJapaneseChars-8              34.1ns ± 5%  36.8ns ± 4%   +8.09%  (p=0.000 n=45+50)
ValidStringTenASCIIChars-8           9.76ns ± 7%  8.33ns ± 3%  -14.59%  (p=0.000 n=48+47)
ValidStringTenJapaneseChars-8        37.7ns ± 8%  36.5ns ± 5%   -3.12%  (p=0.011 n=50+47)
EncodeASCIIRune-8                    2.60ns ± 1%  2.59ns ± 2%   -0.24%  (p=0.018 n=43+36)
EncodeJapaneseRune-8                 3.75ns ± 2%  4.56ns ± 6%  +21.71%  (p=0.000 n=41+50)
DecodeASCIIRune-8                    2.59ns ± 2%  2.59ns ± 2%     ~     (p=0.350 n=44+41)
DecodeJapaneseRune-8                 4.29ns ± 2%  4.31ns ± 2%   +0.61%  (p=0.001 n=48+39)
FullASCIIRune-8                      0.87ns ± 6%  0.29ns ± 5%  -67.31%  (p=0.000 n=49+43)
FullJapaneseRune-8                   0.65ns ± 6%  0.65ns ± 4%     ~     (p=0.375 n=50+49)
[Geo mean]                           7.02ns       6.51ns        -7.19%


Change-Id: I8d5d69c8d33ce2bff94785fba39a2203f9315cb0
Reviewed-on: https://go-review.googlesource.com/c/go/+/173537
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-24 01:10:42 +00:00
Wil Selwood d704b5c956 unicode: improve generated comments for categories
The comments on the category range tables in the unicode package are fairly
redundent and require an external source to translate into human readable
category names.

This adds a look up table with the category descriptions and uses it if
available when generating the comments for the range tables.

Fixes #28954

Change-Id: I853e2d270def6492c2c1dd2ad0ec761a74c04e5d
Reviewed-on: https://go-review.googlesource.com/c/151297
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
2018-11-28 03:12:48 +00:00
Brad Fitzpatrick 05f8b44d5e unicode: fix SpecialCase to follow its docs & respect explict no-op mappings
If SpecialCase contains an explicit CaseRange with zero deltas,
respect those and don't fall back to the default behavior.

Fixes #25636

Change-Id: Ic554c6b3dd462b1b39c75194eec469b6ff4aa55b
Reviewed-on: https://go-review.googlesource.com/117155
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
2018-06-17 08:48:39 +00:00
Tim Cooper 161874da2a all: update comment URLs from HTTP to HTTPS, where possible
Each URL was manually verified to ensure it did not serve up incorrect
content.

Change-Id: I4dc846227af95a73ee9a3074d0c379ff0fa955df
Reviewed-on: https://go-review.googlesource.com/115798
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
2018-06-01 21:52:00 +00:00
Martin Möhrmann b9a59d9f2e cmd/compile: optimize len([]rune(string))
Adds a new runtime function to count runes in a string.
Modifies the compiler to detect the pattern len([]rune(string))
and replaces it with the new rune counting runtime function.

RuneCount/lenruneslice/ASCII                  27.8ns ± 2%  14.5ns ± 3%  -47.70%  (p=0.000 n=10+10)
RuneCount/lenruneslice/Japanese                126ns ± 2%    60ns ± 2%  -52.03%  (p=0.000 n=10+10)
RuneCount/lenruneslice/MixedLength             104ns ± 2%    50ns ± 1%  -51.71%  (p=0.000 n=10+9)

Fixes #24923

Change-Id: Ie9c7e7391a4e2cca675c5cdcc1e5ce7d523948b9
Reviewed-on: https://go-review.googlesource.com/108985
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2018-05-06 05:31:01 +00:00
Joe Tsai b53088a634 Revert "go/printer: forbid empty line before first comment in block"
This reverts commit 08f19bbde1.

Reason for revert:
The changed transformation takes effect on a larger set
of code snippets than expected.

For example, this:
    func foo() {

        // Comment
        bar()

    }
becomes:
    func foo() {
        // Comment
        bar()

    }

This is an unintended consequence.

Change-Id: Ifca88d6267dab8a8170791f7205124712bf8ace8
Reviewed-on: https://go-review.googlesource.com/81335
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Joe Tsai <joetsai@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-12-01 01:12:26 +00:00
Joe Tsai 08f19bbde1 go/printer: forbid empty line before first comment in block
To improve readability when exported fields are removed,
forbid the printer from emitting an empty line before the first comment
in a const, var, or type block.
Also, when printing the "Has filtered or unexported fields." message,
add an empty line before it to separate the message from the struct
or interfact contents.

Before the change:
<<<
type NamedArg struct {

        // Name is the name of the parameter placeholder.
        //
        // If empty, the ordinal position in the argument list will be
        // used.
        //
        // Name must omit any symbol prefix.
        Name string

        // Value is the value of the parameter.
        // It may be assigned the same value types as the query
        // arguments.
        Value interface{}
        // contains filtered or unexported fields
}
>>>

After the change:
<<<
type NamedArg struct {
        // Name is the name of the parameter placeholder.
        //
        // If empty, the ordinal position in the argument list will be
        // used.
        //
        // Name must omit any symbol prefix.
        Name string

        // Value is the value of the parameter.
        // It may be assigned the same value types as the query
        // arguments.
        Value interface{}

        // contains filtered or unexported fields
}
>>>

Fixes #18264

Change-Id: I9fe17ca39cf92fcdfea55064bd2eaa784ce48c88
Reviewed-on: https://go-review.googlesource.com/71990
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
2017-11-02 18:17:22 +00:00
Marcel van Lohuizen 2fd9549797 unicode: update to Unicode 10.0.0
Also includes all derived values as well as
vendored packages.

Generated by running
    UNICODE_VERSION=10.0.0 go generate
in golang.org/x/text

and modified by hand to add the tests and
entries in next.txt for new script and properties.

Closes Issue #21471

Change-Id: I1d10ee3887bd1fd3d5a756ee0d04bd6ec2814ba1
Reviewed-on: https://go-review.googlesource.com/63953
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2017-10-24 12:42:35 +00:00
Marvin Stenger 90d71fe99e all: revert "all: prefer strings.IndexByte over strings.Index"
This reverts https://golang.org/cl/65930.

Fixes #22148

Change-Id: Ie0712621ed89c43bef94417fc32de9af77607760
Reviewed-on: https://go-review.googlesource.com/68430
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-10-05 23:19:10 +00:00
Marvin Stenger f22ba1f247 all: prefer strings.IndexByte over strings.Index
strings.IndexByte was introduced in go1.2 and it can be used
effectively wherever the second argument to strings.Index is
exactly one byte long.

This avoids generating unnecessary string symbols and saves
a few calls to strings.Index.

Change-Id: I1ab5edb7c4ee9058084cfa57cbcc267c2597e793
Reviewed-on: https://go-review.googlesource.com/65930
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-09-25 17:35:41 +00:00
Daniel Martí 57e7d62455 all: use sort.Slice in a few more places
Do the low-hanging fruit - tiny Less functions that are used exactly
once. This reduces the amount of code and puts the logic in a single
place.

Change-Id: I9d4544cd68de5a95e55019bdad1fca0a1dbfae9c
Reviewed-on: https://go-review.googlesource.com/63171
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2017-09-22 12:39:14 +00:00
Marcel van Lohuizen 3dd96e9d82 unicode: allow version to be passed by env var
This, in turn, to make it work with x/text’s
go generate.

Also eliminates need to manually update version
string in maketables.go.

Change-Id: Id5a8b8e27bdce5b1b5920eb9223a2d27b889149a
Reviewed-on: https://go-review.googlesource.com/63952
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Nigel Tao <nigeltao@golang.org>
2017-09-19 10:58:07 +00:00