diff --git a/.github/ISSUE_TEMPLATE/00-bug.md b/.github/ISSUE_TEMPLATE/00-bug.md
deleted file mode 100644
index f056dab7dd..0000000000
--- a/.github/ISSUE_TEMPLATE/00-bug.md
+++ /dev/null
@@ -1,45 +0,0 @@
----
-name: Bugs
-about: The go command, standard library, or anything else
-title: "affected/package: "
----
-
-
-
-### What version of Go are you using (`go version`)?
-
-
-$ go version
-
-
-
-### Does this issue reproduce with the latest release?
-
-
-
-### What operating system and processor architecture are you using (`go env`)?
-
-go env Output
-$ go env
-
-
-$ go version
-
-
-
-### Does this issue reproduce at the latest version of golang.org/x/vuln?
-
-
-
-### What operating system and processor architecture are you using (`go env`)?
-
-go env Output
-$ go env
-
-
Special instructions
+
+
+The PCALIGN pseudo-instruction is used to indicate that the next instruction should be aligned
+to a specified boundary by padding with no-op instructions.
+
+
+
+It is currently supported on arm64, amd64, ppc64, loong64 and riscv64.
+
+For example, the start of the MOVD instruction below is aligned to 32 bytes:
+
+PCALIGN $32
+MOVD $2, R0
+
+
+
Interacting with Go types and constants
diff --git a/doc/go1.17_spec.html b/doc/go1.17_spec.html
index 15e73c3867..c87d9aff3c 100644
--- a/doc/go1.17_spec.html
+++ b/doc/go1.17_spec.html
@@ -7,8 +7,11 @@
Introduction
-This is a reference manual for the Go programming language. For
-more information and other documents, see golang.org.
+This is the reference manual for the Go programming language as it was for
+language version 1.17, in October 2021, before the introduction of generics.
+It is provided for historical interest.
+The current reference manual can be found here.
+For more information and other documents, see go.dev.
diff --git a/doc/go1.22.html b/doc/go1.22.html
deleted file mode 100644
index 287ee77bb5..0000000000
--- a/doc/go1.22.html
+++ /dev/null
@@ -1,105 +0,0 @@
-
-
-
-
-
-
-
DRAFT RELEASE NOTES — Introduction to Go 1.22
-
-
-
- Go 1.22 is not yet released. These are work-in-progress
- release notes. Go 1.22 is expected to be released in February 2024.
-
-
-
-Changes to the language
-
-
- TODO: complete this section
-
-
-
-
-Go command
-
-
- TODO: complete this section, or delete if not needed
-
-
-Cgo
-
-
-
-Runtime
-
-
- TODO: complete this section, or delete if not needed
-
-
-Compiler
-
-
- TODO: complete this section, or delete if not needed
-
-
-Linker
-
-
- TODO: complete this section, or delete if not needed
-
-
-Core library
-
-Minor changes to the library
-
-
- As always, there are various minor changes and updates to the library,
- made with the Go 1 promise of compatibility
- in mind.
- There are also various performance improvements, not enumerated here.
-
-
-
- TODO: complete this section
-
-
-- database/sql
- -
-
- The new Null[T] type
- provide a way to scan nullable columns for any column types.
-
-
-
-
-- reflect
- -
-
- The Value.IsZero
- method will now return true for a floating-point or complex
- negative zero, and will return true for a struct value if a
- blank field (a field named _) somehow has a
- non-zero value.
- These changes make IsZero consistent with comparing
- a value to zero using the language == operator.
-
-
-
-
-Ports
-
-
- TODO: complete this section, or delete if not needed
-
-
diff --git a/doc/go_mem.html b/doc/go_mem.html
index 026c1172e3..c0b81d3fac 100644
--- a/doc/go_mem.html
+++ b/doc/go_mem.html
@@ -98,12 +98,12 @@ which in turn are made up of memory operations.
A memory operation is modeled by four details:
-- its kind, indicating whether it is an ordinary data read, an ordinary data write,
-or a synchronizing operation such as an atomic data access,
-a mutex operation, or a channel operation,
-
- its location in the program,
-
- the memory location or variable being accessed, and
-
- the values read or written by the operation.
+
- its kind, indicating whether it is an ordinary data read, an ordinary data write,
+ or a synchronizing operation such as an atomic data access,
+ a mutex operation, or a channel operation,
+ - its location in the program,
+ - the memory location or variable being accessed, and
+ - the values read or written by the operation.
Some memory operations are read-like, including read, atomic read, mutex lock, and channel receive.
@@ -162,8 +162,8 @@ where visible means that both of the following hold:
-- w happens before r.
-
- w does not happen before any other write w' (to x) that happens before r.
+
- w happens before r.
+ - w does not happen before any other write w' (to x) that happens before r.
diff --git a/doc/go_spec.html b/doc/go_spec.html
index 18f88d5ead..8f48f7444b 100644
--- a/doc/go_spec.html
+++ b/doc/go_spec.html
@@ -1,6 +1,6 @@
@@ -10,7 +10,7 @@
This is the reference manual for the Go programming language.
The pre-Go1.18 version, without generics, can be found
here.
-For more information and other documents, see golang.org.
+For more information and other documents, see go.dev.
@@ -70,6 +70,14 @@ enumerations or code snippets that are not further specified. The character
+
+A link of the form [Go 1.xx] indicates that a described
+language feature (or some aspect of it) was changed or added with language version 1.xx and
+thus requires at minimum that language version to build.
+For details, see the linked section
+in the appendix.
+
+
Source code representation
@@ -263,7 +271,8 @@ continue for import return var
The following character sequences represent operators
-(including assignment operators) and punctuation:
+(including assignment operators) and punctuation
+[Go 1.18]:
+ & += &= && == != ( )
@@ -281,7 +290,8 @@ An integer literal is a sequence of digits representing an
integer constant.
An optional prefix sets a non-decimal base: 0b or 0B
for binary, 0, 0o, or 0O for octal,
-and 0x or 0X for hexadecimal.
+and 0x or 0X for hexadecimal
+[Go 1.13].
A single 0 is considered a decimal zero.
In hexadecimal literals, letters a through f
and A through F represent values 10 through 15.
@@ -347,7 +357,8 @@ prefix, an integer part (hexadecimal digits), a radix point, a fractional part (
and an exponent part (p or P followed by an optional sign and decimal digits).
One of the integer part or the fractional part may be elided; the radix point may be elided as well,
but the exponent part is required. (This syntax matches the one given in IEEE 754-2008 §5.12.3.)
-An exponent value exp scales the mantissa (integer and fractional part) by 2exp.
+An exponent value exp scales the mantissa (integer and fractional part) by 2exp
+[Go 1.13].
@@ -411,7 +422,8 @@ It consists of an integer or
floating-point literal
followed by the lowercase letter i.
The value of an imaginary literal is the value of the respective
-integer or floating-point literal multiplied by the imaginary unit i.
+integer or floating-point literal multiplied by the imaginary unit i
+[Go 1.13]
@@ -1340,6 +1352,7 @@ interface{}
For convenience, the predeclared type any is an alias for the empty interface.
+[Go 1.18]
@@ -1375,13 +1388,15 @@ as the File interface.
In a slightly more general form
an interface T may use a (possibly qualified) interface type
name E as an interface element. This is called
-embedding interface E in T.
+embedding interface E in T
+[Go 1.14].
The type set of T is the intersection of the type sets
defined by T's explicitly declared methods and the type sets
of T’s embedded interfaces.
In other words, the type set of T is the set of all types that implement all the
explicitly declared methods of T and also all the methods of
-E.
+E
+[Go 1.18].
@@ -1420,7 +1435,8 @@ type ReadCloser interface {
In their most general form, an interface element may also be an arbitrary type term
T, or a term of the form ~T specifying the underlying type T,
-or a union of terms t1|t2|…|tn.
+or a union of terms t1|t2|…|tn
+[Go 1.18].
Together with method specifications, these elements enable the precise
definition of an interface's type set as follows:
@@ -1666,6 +1682,7 @@ maps grow to accommodate the number of items
stored in them, with the exception of nil maps.
A nil map is equivalent to an empty map except that no elements
may be added.
+
Channel types
@@ -2303,7 +2320,9 @@ as an operand, and in a
The following identifiers are implicitly declared in the
-universe block:
+universe block
+[Go 1.18]
+[Go 1.21]:
Types:
@@ -2487,7 +2506,8 @@ TypeSpec = AliasDecl | TypeDef .
Alias declarations
-An alias declaration binds an identifier to the given type.
+An alias declaration binds an identifier to the given type
+[Go 1.9].
@@ -2636,7 +2656,8 @@ func (l *List[T]) Len() int { … }
A type parameter list declares the type parameters of a generic function or type declaration.
The type parameter list looks like an ordinary function parameter list
except that the type parameter names must all be present and the list is enclosed
-in square brackets rather than parentheses.
+in square brackets rather than parentheses
+[Go 1.18].
@@ -2719,7 +2740,8 @@ type T6[P int] struct{ f *T6[P] } // ok: reference to T6 is not in type para
A type constraint is an interface that defines the
set of permissible type arguments for the respective type parameter and controls the
-operations supported by values of that type parameter.
+operations supported by values of that type parameter
+[Go 1.18].
@@ -2749,7 +2771,8 @@ other interfaces based on their type sets. But this should get us going for now.
The predeclared
interface type comparable
denotes the set of all non-interface types that are
-strictly comparable.
+strictly comparable
+[Go 1.18].
@@ -2782,7 +2805,8 @@ if T is an element of the type set defined by C; i.e.,
if T implements C.
As an exception, a strictly comparable
type constraint may also be satisfied by a comparable
-(not necessarily strictly comparable) type argument.
+(not necessarily strictly comparable) type argument
+[Go 1.20].
More precisely:
@@ -4306,7 +4330,7 @@ with the same underlying array.
A generic function or type is instantiated by substituting type arguments
-for the type parameters.
+for the type parameters [Go 1.18].
Instantiation proceeds in two steps:
@@ -4759,6 +4783,7 @@ to the type of the other operand.
The right operand in a shift expression must have integer type
+[Go 1.13]
or be an untyped constant representable by a
value of type uint.
If the left operand of a non-constant shift expression is an untyped constant,
@@ -5426,7 +5451,8 @@ in any of these cases:
x is a string and T is a slice of bytes or runes.
- x is a slice, T is an array or a pointer to an array,
+ x is a slice, T is an array [Go 1.20]
+ or a pointer to an array [Go 1.17],
and the slice and array types have identical element types.
@@ -6516,7 +6542,6 @@ additionally it may specify an init
and a post statement, such as an assignment,
an increment or decrement statement. The init statement may be a
short variable declaration, but the post statement must not.
-Variables declared by the init statement are re-used in each iteration.
@@ -6548,12 +6573,54 @@ for cond { S() } is the same as for ; cond ; { S() }
for { S() } is the same as for true { S() }
+
+Each iteration has its own separate declared variable (or variables)
+[Go 1.22].
+The variable used by the first iteration is declared by the init statement.
+The variable used by each subsequent iteration is declared implicitly before
+executing the post statement and initialized to the value of the previous
+iteration's variable at that moment.
+
+
+
+var prints []func()
+for i := 0; i < 5; i++ {
+ prints = append(prints, func() { println(i) })
+ i++
+}
+for _, p := range prints {
+ p()
+}
+
+
+
+prints
+
+
+
+1
+3
+5
+
+
+
+Prior to [Go 1.22], iterations share one set of variables
+instead of having their own separate variables.
+In that case, the example above prints
+
+
+
+6
+6
+6
+
+
For statements with range clause
A "for" statement with a "range" clause
iterates through all entries of an array, slice, string or map, values received on
-a channel, or integer values from zero to an upper limit.
+a channel, or integer values from zero to an upper limit [Go 1.22].
For each entry it assigns iteration values
to corresponding iteration variables if present and then executes the block.
@@ -6595,7 +6662,7 @@ array or slice a [n]E, *[n]E, or []E index i int a[i] E
string s string type index i int see below rune
map m map[K]V key k K m[k] V
channel c chan E, <-chan E element e E
-integer n integer type I value i I
+integer n integer type value i see below
@@ -6637,25 +6704,33 @@ is nil, the range expression blocks forever.
-
For an integer value
n, the iteration values 0 through n-1
-are produced in increasing order, with the same type as n.
+are produced in increasing order.
If n <= 0, the loop does not run any iterations.
-
-The iteration values are assigned to the respective
-iteration variables as in an assignment statement.
-
-
The iteration variables may be declared by the "range" clause using a form of
short variable declaration
(:=).
-In this case their types are set to the types of the respective iteration values
-and their scope is the block of the "for"
-statement; they are re-used in each iteration.
-If the iteration variables are declared outside the "for" statement,
-after execution their values will be those of the last iteration.
+In this case their scope is the block of the "for" statement
+and each iteration has its own new variables [Go 1.22]
+(see also "for" statements with a ForClause).
+If the range expression is a (possibly untyped) integer expression n,
+the variable has the same type as if it was
+declared with initialization
+expression n.
+Otherwise, the variables have the types of their respective iteration values.
+
+
+
+If the iteration variables are not explicitly declared by the "range" clause,
+they must be preexisting.
+In this case, the iteration values are assigned to the respective variables
+as in an assignment statement.
+If the range expression is a (possibly untyped) integer expression n,
+n too must be assignable to the iteration variable;
+if there is no iteration variable, n must be assignable to int.
@@ -6698,6 +6773,11 @@ for i := range 10 {
// type of i is int (default type for untyped constant 10)
f(i)
}
+
+// invalid: 256 cannot be assigned to uint8
+var u uint8
+for u = range 256 {
+}
@@ -7221,7 +7301,7 @@ The number of elements copied is the minimum of
len(src) and len(dst).
As a special case, if the destination's core type is []byte,
copy also accepts a source argument with core type
- bytestring.
+bytestring.
This form copies the bytes from the byte slice or string into the byte slice.
@@ -7249,7 +7329,8 @@ n3 := copy(b, "Hello, World!") // n3 == 5, b is []byte("Hello")
The built-in function clear takes an argument of map,
slice, or type parameter type,
-and deletes or zeroes out all elements.
+and deletes or zeroes out all elements
+[Go 1.21].
@@ -7516,7 +7597,8 @@ The precise behavior is implementation-dependent.
The built-in functions min and max compute the
smallest—or largest, respectively—value of a fixed number of
arguments of ordered types.
-There must be at least one argument.
+There must be at least one argument
+[Go 1.21].
@@ -8232,8 +8314,8 @@ of if the general conversion rules take care of this.
A Pointer is a pointer type but a Pointer
value may not be dereferenced.
-Any pointer or value of underlying type uintptr can be
-converted to a type of underlying type Pointer and vice versa.
+Any pointer or value of core type uintptr can be
+converted to a type of core type Pointer and vice versa.
The effect of converting between Pointer and uintptr is implementation-defined.
@@ -8244,6 +8326,10 @@ bits = *(*uint64)(unsafe.Pointer(&f))
type ptr unsafe.Pointer
bits = *(*uint64)(ptr(&f))
+func f[P ~*B, B any](p P) uintptr {
+ return uintptr(unsafe.Pointer(p))
+}
+
var p ptr = nil
@@ -8292,7 +8378,8 @@ of constant size.
The function Add adds len to ptr
-and returns the updated pointer unsafe.Pointer(uintptr(ptr) + uintptr(len)).
+and returns the updated pointer unsafe.Pointer(uintptr(ptr) + uintptr(len))
+[Go 1.17].
The len argument must be of integer type or an untyped constant.
A constant len argument must be representable by a value of type int;
if it is an untyped constant it is given type int.
@@ -8312,7 +8399,8 @@ and whose length and capacity are len.
except that, as a special case, if ptr
is nil and len is zero,
-Slice returns nil.
+Slice returns nil
+[Go 1.17].
@@ -8321,14 +8409,16 @@ A constant len argument must be non-negative and run-time panic occurs.
+a run-time panic occurs
+[Go 1.17].
The function SliceData returns a pointer to the underlying array of the slice argument.
If the slice's capacity cap(slice) is not zero, that pointer is &slice[:1][0].
If slice is nil, the result is nil.
-Otherwise it is a non-nil pointer to an unspecified memory address.
+Otherwise it is a non-nil pointer to an unspecified memory address
+[Go 1.20].
@@ -8337,12 +8427,14 @@ The function String returns a string value whose under
The same requirements apply to the ptr and len argument as in the function
Slice. If len is zero, the result is the empty string "".
Since Go strings are immutable, the bytes passed to String must not be modified afterwards.
+[Go 1.20]
The function StringData returns a pointer to the underlying bytes of the str argument.
For an empty string the return value is unspecified, and may be nil.
-Since Go strings are immutable, the bytes returned by StringData must not be modified.
+Since Go strings are immutable, the bytes returned by StringData must not be modified
+[Go 1.20].
Size and alignment guarantees
@@ -8383,6 +8475,145 @@ A struct or array type has size zero if it contains no fields (or elements, resp
Appendix
+Language versions
+
+
+The Go 1 compatibility guarantee ensures that
+programs written to the Go 1 specification will continue to compile and run
+correctly, unchanged, over the lifetime of that specification.
+More generally, as adjustments are made and features added to the language,
+the compatibility guarantee ensures that a Go program that works with a
+specific Go language version will continue to work with any subsequent version.
+
+
+
+For instance, the ability to use the prefix 0b for binary
+integer literals was introduced with Go 1.13, indicated
+by [Go 1.13] in the section on
+integer literals.
+Source code containing an integer literal such as 0b1011
+will be rejected if the implied or required language version used by
+the compiler is older than Go 1.13.
+
+
+
+The following table describes the minimum language version required for
+features introduced after Go 1.
+
+
+Go 1.9
+
+
+Go 1.13
+
+-
+Integer literals may use the prefixes
0b, 0B, 0o,
+and 0O for binary, and octal literals, respectively.
+
+-
+Hexadecimal floating-point literals may be written using the prefixes
+
0x and 0X.
+
+-
+The imaginary suffix
i may be used with any (binary, decimal, hexadecimal)
+integer or floating-point literal, not just decimal literals.
+
+-
+The digits of any number literal may be separated (grouped)
+using underscores
_.
+
+-
+The shift count in a shift operation may be a signed integer type.
+
+
+
+Go 1.14
+
+-
+Emdedding a method more than once through different embedded interfaces
+is not an error.
+
+
+
+Go 1.17
+
+-
+A slice may be converted to an array pointer if the slice and array element
+types match, and the array is not longer than the slice.
+
+-
+The built-in package
unsafe includes the new functions
+Add and Slice.
+
+
+
+Go 1.18
+
+The 1.18 release adds polymorphic functions and types ("generics") to the language.
+Specifically:
+
+
+
+Go 1.20
+
+-
+A slice may be converted to an array if the slice and array element
+types match and the array is not longer than the slice.
+
+-
+The built-in package
unsafe includes the new functions
+SliceData, String, and StringData.
+
+-
+Comparable types (such as ordinary interfaces) may satisfy
+
comparable constraints, even if the type arguments are not strictly comparable.
+
+
+
+Go 1.21
+
+-
+The set of predeclared functions includes the new functions
+
min, max, and clear.
+
+-
+Type inference uses the types of interface methods for inference.
+It also infers type arguments for generic functions assigned to variables or
+passed as arguments to other (possibly generic) functions.
+
+
+
+Go 1.22
+
+-
+In a "for" statement, each iteration has its own set of iteration
+variables rather than sharing the same variables in each iteration.
+
+-
+A "for" statement with "range" clause may iterate over
+integer values from zero to an upper limit.
+
+
+
Type unification rules
diff --git a/doc/godebug.md b/doc/godebug.md
index 9235635bdd..2b8852a7ec 100644
--- a/doc/godebug.md
+++ b/doc/godebug.md
@@ -126,6 +126,25 @@ for example,
see the [runtime documentation](/pkg/runtime#hdr-Environment_Variables)
and the [go command documentation](/cmd/go#hdr-Build_and_test_caching).
+### Go 1.23
+
+Go 1.23 changed the mode bits reported by [`os.Lstat`](/pkg/os#Lstat) and [`os.Stat`](/pkg/os#Stat)
+for reparse points, which can be controlled with the `winsymlink` setting.
+As of Go 1.23 (`winsymlink=1`), mount points no longer have [`os.ModeSymlink`](/pkg/os#ModeSymlink)
+set, and reparse points that are not symlinks, Unix sockets, or dedup files now
+always have [`os.ModeIrregular`](/pkg/os#ModeIrregular) set. As a result of these changes,
+[`filepath.EvalSymlinks`](/pkg/path/filepath#EvalSymlinks) no longer evaluates
+mount points, which was a source of many inconsistencies and bugs.
+At previous versions (`winsymlink=0`), mount points are treated as symlinks,
+and other reparse points with non-default [`os.ModeType`](/pkg/os#ModeType) bits
+(such as [`os.ModeDir`](/pkg/os#ModeDir)) do not have the `ModeIrregular` bit set.
+
+Go 1.23 changed [`os.Readlink`](/pkg/os#Readlink) and [`filepath.EvalSymlinks`](/pkg/path/filepath#EvalSymlinks)
+to avoid trying to normalize volumes to drive letters, which was not always even possible.
+This behavior is controlled by the `winreadlinkvolume` setting.
+For Go 1.23, it defaults to `winreadlinkvolume=1`.
+Previous versions default to `winreadlinkvolume=0`.
+
### Go 1.22
Go 1.22 adds a configurable limit to control the maximum acceptable RSA key size
@@ -148,7 +167,7 @@ for the explicit representation of [type aliases](/ref/spec#Type_declarations).
Whether the type checker produces `Alias` types or not is controlled by the
[`gotypesalias` setting](/pkg/go/types#Alias).
For Go 1.22 it defaults to `gotypesalias=0`.
-For Go 1.23, `gotypealias=1` will become the default.
+For Go 1.23, `gotypesalias=1` will become the default.
This setting will be removed in a future release, Go 1.24 at the earliest.
Go 1.22 changed the default minimum TLS version supported by both servers
@@ -159,6 +178,41 @@ Go 1.22 changed the default TLS cipher suites used by clients and servers when
not explicitly configured, removing the cipher suites which used RSA based key
exchange. The default can be revert using the [`tlsrsakex` setting](/pkg/crypto/tls/#Config).
+Go 1.22 disabled
+[`ConnectionState.ExportKeyingMaterial`](/pkg/crypto/tls/#ConnectionState.ExportKeyingMaterial)
+when the connection supports neither TLS 1.3 nor Extended Master Secret
+(implemented in Go 1.21). It can be reenabled with the [`tlsunsafeekm`
+setting](/pkg/crypto/tls/#ConnectionState.ExportKeyingMaterial).
+
+Go 1.22 changed how the runtime interacts with transparent huge pages on Linux.
+In particular, a common default Linux kernel configuration can result in
+significant memory overheads, and Go 1.22 no longer works around this default.
+To work around this issue without adjusting kernel settings, transparent huge
+pages can be disabled for Go memory with the
+[`disablethp` setting](/pkg/runtime#hdr-Environment_Variable).
+This behavior was backported to Go 1.21.1, but the setting is only available
+starting with Go 1.21.6.
+This setting may be removed in a future release, and users impacted by this issue
+should adjust their Linux configuration according to the recommendations in the
+[GC guide](/doc/gc-guide#Linux_transparent_huge_pages), or switch to a Linux
+distribution that disables transparent huge pages altogether.
+
+Go 1.22 added contention on runtime-internal locks to the [`mutex`
+profile](/pkg/runtime/pprof#Profile). Contention on these locks is always
+reported at `runtime._LostContendedRuntimeLock`. Complete stack traces of
+runtime locks can be enabled with the [`runtimecontentionstacks`
+setting](/pkg/runtime#hdr-Environment_Variable). These stack traces have
+non-standard semantics, see setting documentation for details.
+
+Go 1.22 added a new [`crypto/x509.Certificate`](/pkg/crypto/x509/#Certificate)
+field, [`Policies`](/pkg/crypto/x509/#Certificate.Policies), which supports
+certificate policy OIDs with components larger than 31 bits. By default this
+field is only used during parsing, when it is populated with policy OIDs, but
+not used during marshaling. It can be used to marshal these larger OIDs, instead
+of the existing PolicyIdentifiers field, by using the
+[`x509usepolicies` setting.](/pkg/crypto/x509/#CreateCertificate).
+
+
### Go 1.21
Go 1.21 made it a run-time error to call `panic` with a nil interface value,
diff --git a/doc/initial/1-intro.md b/doc/initial/1-intro.md
new file mode 100644
index 0000000000..e28191ca9e
--- /dev/null
+++ b/doc/initial/1-intro.md
@@ -0,0 +1,12 @@
+
+
+
+
+## Introduction to Go 1.XX {#introduction}
+
diff --git a/doc/initial/2-language.md b/doc/initial/2-language.md
new file mode 100644
index 0000000000..61030bd676
--- /dev/null
+++ b/doc/initial/2-language.md
@@ -0,0 +1,3 @@
+## Changes to the language {#language}
+
+
diff --git a/doc/initial/3-tools.md b/doc/initial/3-tools.md
new file mode 100644
index 0000000000..5638f240a5
--- /dev/null
+++ b/doc/initial/3-tools.md
@@ -0,0 +1,6 @@
+## Tools {#tools}
+
+### Go command {#go-command}
+
+### Cgo {#cgo}
+
diff --git a/doc/initial/4-runtime.md b/doc/initial/4-runtime.md
new file mode 100644
index 0000000000..1f8e445e0b
--- /dev/null
+++ b/doc/initial/4-runtime.md
@@ -0,0 +1 @@
+## Runtime {#runtime}
diff --git a/doc/initial/5-toolchain.md b/doc/initial/5-toolchain.md
new file mode 100644
index 0000000000..0f4a816479
--- /dev/null
+++ b/doc/initial/5-toolchain.md
@@ -0,0 +1,7 @@
+## Compiler {#compiler}
+
+## Assembler {#assembler}
+
+## Linker {#linker}
+
+
diff --git a/doc/initial/6-stdlib/0-heading.md b/doc/initial/6-stdlib/0-heading.md
new file mode 100644
index 0000000000..a992170d43
--- /dev/null
+++ b/doc/initial/6-stdlib/0-heading.md
@@ -0,0 +1,2 @@
+## Standard library {#library}
+
diff --git a/doc/initial/6-stdlib/99-minor/0-heading.md b/doc/initial/6-stdlib/99-minor/0-heading.md
new file mode 100644
index 0000000000..a98105e8cc
--- /dev/null
+++ b/doc/initial/6-stdlib/99-minor/0-heading.md
@@ -0,0 +1,3 @@
+### Minor changes to the library {#minor_library_changes}
+
+
diff --git a/doc/initial/6-stdlib/99-minor/README b/doc/initial/6-stdlib/99-minor/README
new file mode 100644
index 0000000000..fac778de05
--- /dev/null
+++ b/doc/initial/6-stdlib/99-minor/README
@@ -0,0 +1 @@
+API changes and other small changes to the standard library go here.
diff --git a/doc/initial/7-ports.md b/doc/initial/7-ports.md
new file mode 100644
index 0000000000..8bea3f8fbc
--- /dev/null
+++ b/doc/initial/7-ports.md
@@ -0,0 +1,2 @@
+## Ports {#ports}
+
diff --git a/doc/next/1-intro.md b/doc/next/1-intro.md
new file mode 100644
index 0000000000..639550f92a
--- /dev/null
+++ b/doc/next/1-intro.md
@@ -0,0 +1,12 @@
+
+
+
+
+## Introduction to Go 1.23 {#introduction}
+
diff --git a/doc/next/2-language.md b/doc/next/2-language.md
new file mode 100644
index 0000000000..61030bd676
--- /dev/null
+++ b/doc/next/2-language.md
@@ -0,0 +1,3 @@
+## Changes to the language {#language}
+
+
diff --git a/doc/next/3-tools.md b/doc/next/3-tools.md
new file mode 100644
index 0000000000..bdbe6c0771
--- /dev/null
+++ b/doc/next/3-tools.md
@@ -0,0 +1,12 @@
+## Tools {#tools}
+
+### Go command {#go-command}
+
+Setting the `GOROOT_FINAL` environment variable no longer has an effect
+([#62047](https://go.dev/issue/62047)).
+Distributions that install the `go` command to a location other than
+`$GOROOT/bin/go` should install a symlink instead of relocating
+or copying the `go` binary.
+
+### Cgo {#cgo}
+
diff --git a/doc/next/4-runtime.md b/doc/next/4-runtime.md
new file mode 100644
index 0000000000..1f8e445e0b
--- /dev/null
+++ b/doc/next/4-runtime.md
@@ -0,0 +1 @@
+## Runtime {#runtime}
diff --git a/doc/next/5-toolchain.md b/doc/next/5-toolchain.md
new file mode 100644
index 0000000000..0f4a816479
--- /dev/null
+++ b/doc/next/5-toolchain.md
@@ -0,0 +1,7 @@
+## Compiler {#compiler}
+
+## Assembler {#assembler}
+
+## Linker {#linker}
+
+
diff --git a/doc/next/6-stdlib/0-heading.md b/doc/next/6-stdlib/0-heading.md
new file mode 100644
index 0000000000..a992170d43
--- /dev/null
+++ b/doc/next/6-stdlib/0-heading.md
@@ -0,0 +1,2 @@
+## Standard library {#library}
+
diff --git a/doc/next/6-stdlib/99-minor/0-heading.md b/doc/next/6-stdlib/99-minor/0-heading.md
new file mode 100644
index 0000000000..a98105e8cc
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/0-heading.md
@@ -0,0 +1,3 @@
+### Minor changes to the library {#minor_library_changes}
+
+
diff --git a/doc/next/6-stdlib/99-minor/README b/doc/next/6-stdlib/99-minor/README
new file mode 100644
index 0000000000..fac778de05
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/README
@@ -0,0 +1 @@
+API changes and other small changes to the standard library go here.
diff --git a/doc/next/6-stdlib/99-minor/database/sql/64707.md b/doc/next/6-stdlib/99-minor/database/sql/64707.md
new file mode 100644
index 0000000000..70aad889ff
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/database/sql/64707.md
@@ -0,0 +1,4 @@
+Errors returned by [`driver.Valuer`](/database/sql/driver#Driver)
+implementations are now wrapped for improved error handling during
+operations like [`Query`](/database/sql#DB.Query), [`Exec`](/database/sql#DB.Exec),
+and [`QueryRow`](/database/sql#DB.QueryRow).
diff --git a/doc/next/6-stdlib/99-minor/net/62254.md b/doc/next/6-stdlib/99-minor/net/62254.md
new file mode 100644
index 0000000000..012b7ede5e
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/net/62254.md
@@ -0,0 +1,4 @@
+The new type [`KeepAliveConfig`](/net#KeepAliveConfig) permits fine-tuning
+the keep-alive options for TCP connections, via a new
+[`TCPConn.SetKeepAliveConfig`](/net#TCPConn.SetKeepAliveConfig) method and
+new KeepAliveConfig fields for [`Dialer`](/net#Dialer) and [`ListenConfig`](/net#ListenConfig).
diff --git a/doc/next/6-stdlib/99-minor/net/http/64910.md b/doc/next/6-stdlib/99-minor/net/http/64910.md
new file mode 100644
index 0000000000..020e18b97b
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/net/http/64910.md
@@ -0,0 +1,2 @@
+The patterns used by [`net/http.ServeMux`](//net/http#ServeMux) allow
+multiple spaces matching regexp '[ \t]+'.
diff --git a/doc/next/6-stdlib/99-minor/os/33357.md b/doc/next/6-stdlib/99-minor/os/33357.md
new file mode 100644
index 0000000000..3e80943263
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/os/33357.md
@@ -0,0 +1,3 @@
+The [`os.Stat`](/os#Stat) function now sets the [`os.ModeSocket`](/os#ModeSocket)
+bit for files that are Unix sockets on Windows. These files are identified by
+having a reparse tag set to `IO_REPARSE_TAG_AF_UNIX`.
diff --git a/doc/next/6-stdlib/99-minor/os/61893.md b/doc/next/6-stdlib/99-minor/os/61893.md
new file mode 100644
index 0000000000..b2dd537039
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/os/61893.md
@@ -0,0 +1,7 @@
+On Windows, the mode bits reported by [`os.Lstat`](/pkg/os#Lstat) and [`os.Stat`](/pkg/os#Stat)
+for reparse points changed. Mount points no longer have [`os.ModeSymlink`](/pkg/os#ModeSymlink) set,
+and reparse points that are not symlinks, Unix sockets, or dedup files now
+always have [`os.ModeIrregular`](/pkg/os#ModeIrregular) set.
+This behavior is controlled by the `winsymlink` setting.
+For Go 1.23, it defaults to `winsymlink=1`.
+Previous versions default to `winsymlink=0`.
diff --git a/doc/next/6-stdlib/99-minor/os/62484.md b/doc/next/6-stdlib/99-minor/os/62484.md
new file mode 100644
index 0000000000..81abb4bc68
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/os/62484.md
@@ -0,0 +1,2 @@
+The [`CopyFS`](/os#CopyFS) function copies an [`io/fs.FS`](/io/fs#FS)
+into the local filesystem.
diff --git a/doc/next/6-stdlib/99-minor/os/63703.md b/doc/next/6-stdlib/99-minor/os/63703.md
new file mode 100644
index 0000000000..abde9448f5
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/os/63703.md
@@ -0,0 +1,5 @@
+On Windows, [`os.Readlink`](/os#Readlink) no longer tries
+to normalize volumes to drive letters, which was not always even possible.
+This behavior is controlled by the `winreadlinkvolume` setting.
+For Go 1.23, it defaults to `winreadlinkvolume=1`.
+Previous versions default to `winreadlinkvolume=0`.
\ No newline at end of file
diff --git a/doc/next/6-stdlib/99-minor/path/filepath/57151.md b/doc/next/6-stdlib/99-minor/path/filepath/57151.md
new file mode 100644
index 0000000000..67e84894fe
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/path/filepath/57151.md
@@ -0,0 +1,2 @@
+The new [`Localize`](/path/filepath#Localize) function safely converts
+a slash-separated path into an operating system path.
diff --git a/doc/next/6-stdlib/99-minor/path/filepath/63703.md b/doc/next/6-stdlib/99-minor/path/filepath/63703.md
new file mode 100644
index 0000000000..0aa0ba6fe3
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/path/filepath/63703.md
@@ -0,0 +1,11 @@
+On Windows, [`filepath.EvalSymlinks`](/pkg/path/filepath#EvalSymlinks) no longer evaluates
+mount points, which was a source of many inconsistencies and bugs.
+This behavior is controlled by the `winsymlink` setting.
+For Go 1.23, it defaults to `winsymlink=1`.
+Previous versions default to `winsymlink=0`.
+
+On Windows, [`filepath.EvalSymlinks`](/pkg/path/filepath#EvalSymlinks) no longer tries
+to normalize volumes to drive letters, which was not always even possible.
+This behavior is controlled by the `winreadlinkvolume` setting.
+For Go 1.23, it defaults to `winreadlinkvolume=1`.
+Previous versions default to `winreadlinkvolume=0`.
\ No newline at end of file
diff --git a/doc/next/6-stdlib/99-minor/reflect/60427.md b/doc/next/6-stdlib/99-minor/reflect/60427.md
new file mode 100644
index 0000000000..92230cde1a
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/reflect/60427.md
@@ -0,0 +1,6 @@
+The new methods synonymous with the method of the same name in [`reflect.Value`](/pkg/reflect#Value)
+are added to [`reflect.Type`](/pkg/reflect#Type):
+1. [`OverflowComplex`](/pkg/reflect#Type.OverflowComplex)
+2. [`OverflowFloat`](/pkg/reflect#Type.OverflowFloat)
+3. [`OverflowInt`](/pkg/reflect#Type.OverflowInt)
+4. [`OverflowUint`](/pkg/reflect#Type.OverflowUint)
diff --git a/doc/next/6-stdlib/99-minor/runtime/debug/42888.md b/doc/next/6-stdlib/99-minor/runtime/debug/42888.md
new file mode 100644
index 0000000000..d75c86900f
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/runtime/debug/42888.md
@@ -0,0 +1,8 @@
+
+The [`debug.SetCrashOutput`](/runtime#SetCrashOutput) function allows
+the user to specify an alternate file to which the runtime should
+write its fatal crash report
+([#42888](https://github.com/golang/go/issues/42888)).
+It may be used to construct an automated reporting mechanism for all
+unexpected crashes, not just those in goroutines that explicitly use
+`recover`.
diff --git a/doc/next/6-stdlib/99-minor/sync/61696.md b/doc/next/6-stdlib/99-minor/sync/61696.md
new file mode 100644
index 0000000000..173076ca5e
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/sync/61696.md
@@ -0,0 +1,4 @@
+The [`(*sync.Map) Clear()`](//sync#Map.Clear) method deletes
+all the entries, resulting in an empty map
+([#61696](https://github.com/golang/go/issues/61696)).
+It is analogous to `clear`.
\ No newline at end of file
diff --git a/doc/next/6-stdlib/99-minor/syscall/62254.md b/doc/next/6-stdlib/99-minor/syscall/62254.md
new file mode 100644
index 0000000000..fe9651a178
--- /dev/null
+++ b/doc/next/6-stdlib/99-minor/syscall/62254.md
@@ -0,0 +1 @@
+The syscall package now defines WSAENOPROTOOPT on Windows.
diff --git a/doc/next/7-ports.md b/doc/next/7-ports.md
new file mode 100644
index 0000000000..796cc4bf1b
--- /dev/null
+++ b/doc/next/7-ports.md
@@ -0,0 +1,8 @@
+## Ports {#ports}
+
+### Darwin {#darwin}
+
+
+As [announced](go1.22#darwin) in the Go 1.22 release notes,
+Go 1.23 requires macOS 11 Big Sur or later;
+support for previous versions has been discontinued.
diff --git a/lib/time/mkzip.go b/lib/time/mkzip.go
index 3920b11b6c..2f4d4dc71a 100644
--- a/lib/time/mkzip.go
+++ b/lib/time/mkzip.go
@@ -31,7 +31,7 @@ import (
)
func usage() {
- fmt.Fprintf(os.Stderr, "usage: go run mkzip.go ../../zoneinfo.zip\n")
+ fmt.Fprintf(os.Stderr, "usage: go run mkzip.go zoneinfo.zip\n")
os.Exit(2)
}
diff --git a/lib/time/update.bash b/lib/time/update.bash
index 605afa76d3..bed82b4f40 100755
--- a/lib/time/update.bash
+++ b/lib/time/update.bash
@@ -24,8 +24,8 @@
# in the CL match the update.bash in the CL.
# Versions to use.
-CODE=2023c
-DATA=2023c
+CODE=2024a
+DATA=2024a
set -e
diff --git a/lib/time/zoneinfo.zip b/lib/time/zoneinfo.zip
index 417ee2b194..bb38801b7a 100644
Binary files a/lib/time/zoneinfo.zip and b/lib/time/zoneinfo.zip differ
diff --git a/misc/wasm/go_wasip1_wasm_exec b/misc/wasm/go_wasip1_wasm_exec
index dc110327af..cd16b96ea7 100755
--- a/misc/wasm/go_wasip1_wasm_exec
+++ b/misc/wasm/go_wasip1_wasm_exec
@@ -14,8 +14,15 @@ case "$GOWASIRUNTIME" in
exec wazero run -mount /:/ -env-inherit -cachedir "${TMPDIR:-/tmp}"/wazero ${GOWASIRUNTIMEARGS:-} "$1" "${@:2}"
;;
"wasmtime" | "")
- # TODO(go.dev/issue/63718): Switch to the new CLI offered in the major version 14 of Wasmtime.
- exec env WASMTIME_NEW_CLI=0 wasmtime run --dir=/ --env PWD="$PWD" --env PATH="$PATH" --max-wasm-stack 1048576 ${GOWASIRUNTIMEARGS:-} "$1" -- "${@:2}"
+ # Match the major version in "wasmtime-cli 14.0.0". For versions before 14
+ # we need to use the old CLI. This requires Bash v3.0 and above.
+ # TODO(johanbrandhorst): Remove this condition once 1.22 is released.
+ # From 1.23 onwards we'll only support the new wasmtime CLI.
+ if [[ "$(wasmtime --version)" =~ wasmtime-cli[[:space:]]([0-9]+)\.[0-9]+\.[0-9]+ && "${BASH_REMATCH[1]}" -lt 14 ]]; then
+ exec wasmtime run --dir=/ --env PWD="$PWD" --env PATH="$PATH" --max-wasm-stack 1048576 ${GOWASIRUNTIMEARGS:-} "$1" -- "${@:2}"
+ else
+ exec wasmtime run --dir=/ --env PWD="$PWD" --env PATH="$PATH" -W max-wasm-stack=1048576 ${GOWASIRUNTIMEARGS:-} "$1" "${@:2}"
+ fi
;;
*)
echo "Unknown Go WASI runtime specified: $GOWASIRUNTIME"
diff --git a/src/archive/tar/common.go b/src/archive/tar/common.go
index e507d559cb..4910908f81 100644
--- a/src/archive/tar/common.go
+++ b/src/archive/tar/common.go
@@ -614,8 +614,6 @@ func (fi headerFileInfo) String() string {
// sysStat, if non-nil, populates h from system-dependent fields of fi.
var sysStat func(fi fs.FileInfo, h *Header) error
-var loadUidAndGid func(fi fs.FileInfo, uid, gid *int)
-
const (
// Mode constants from the USTAR spec:
// See http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html#tag_20_92_13_06
@@ -641,10 +639,6 @@ const (
// Since fs.FileInfo's Name method only returns the base name of
// the file it describes, it may be necessary to modify Header.Name
// to provide the full path name of the file.
-//
-// If fi implements [FileInfoNames]
-// the Gname and Uname of the header are
-// provided by the methods of the interface.
func FileInfoHeader(fi fs.FileInfo, link string) (*Header, error) {
if fi == nil {
return nil, errors.New("archive/tar: FileInfo is nil")
@@ -717,38 +711,12 @@ func FileInfoHeader(fi fs.FileInfo, link string) (*Header, error) {
}
}
}
- if iface, ok := fi.(FileInfoNames); ok {
- var err error
- if loadUidAndGid != nil {
- loadUidAndGid(fi, &h.Uid, &h.Gid)
- }
- h.Gname, err = iface.Gname(h.Gid)
- if err != nil {
- return nil, err
- }
- h.Uname, err = iface.Uname(h.Uid)
- if err != nil {
- return nil, err
- }
- return h, nil
- }
if sysStat != nil {
return h, sysStat(fi, h)
}
return h, nil
}
-// FileInfoNames extends [FileInfo] to translate UID/GID to names.
-// Passing an instance of this to [FileInfoHeader] permits the caller
-// to control UID/GID resolution.
-type FileInfoNames interface {
- fs.FileInfo
- // Uname should translate a UID into a user name.
- Uname(uid int) (string, error)
- // Gname should translate a GID into a group name.
- Gname(gid int) (string, error)
-}
-
// isHeaderOnlyType checks if the given type flag is of the type that has no
// data section even if a size is specified.
func isHeaderOnlyType(flag byte) bool {
diff --git a/src/archive/tar/stat_unix.go b/src/archive/tar/stat_unix.go
index 5b23d3c830..0f3428bc24 100644
--- a/src/archive/tar/stat_unix.go
+++ b/src/archive/tar/stat_unix.go
@@ -17,7 +17,6 @@ import (
func init() {
sysStat = statUnix
- loadUidAndGid = loadUidAndGidFunc
}
// userMap and groupMap caches UID and GID lookups for performance reasons.
@@ -100,12 +99,3 @@ func statUnix(fi fs.FileInfo, h *Header) error {
}
return nil
}
-
-func loadUidAndGidFunc(fi fs.FileInfo, uid, gid *int) {
- sys, ok := fi.Sys().(*syscall.Stat_t)
- if !ok {
- return
- }
- *uid = int(sys.Uid)
- *gid = int(sys.Gid)
-}
diff --git a/src/archive/tar/tar_test.go b/src/archive/tar/tar_test.go
index 49d31bb757..a476f5eb01 100644
--- a/src/archive/tar/tar_test.go
+++ b/src/archive/tar/tar_test.go
@@ -848,71 +848,3 @@ func Benchmark(b *testing.B) {
})
}
-
-const (
- testUid = 10
- testGid = 20
-)
-
-type fileInfoNames struct{}
-
-func (f *fileInfoNames) Name() string {
- return "tmp"
-}
-
-func (f *fileInfoNames) Size() int64 {
- return 0
-}
-
-func (f *fileInfoNames) Mode() fs.FileMode {
- return 0777
-}
-
-func (f *fileInfoNames) ModTime() time.Time {
- return time.Time{}
-}
-
-func (f *fileInfoNames) IsDir() bool {
- return false
-}
-
-func (f *fileInfoNames) Sys() any {
- return nil
-}
-
-func (f *fileInfoNames) Uname(uid int) (string, error) {
- if uid == testUid {
- return "Uname", nil
- }
- return "", nil
-}
-
-func (f *fileInfoNames) Gname(gid int) (string, error) {
- if gid == testGid {
- return "Gname", nil
- }
- return "", nil
-}
-
-func TestFileInfoHeaderUseFileInfoNames(t *testing.T) {
- origLoadUidAndGid := loadUidAndGid
- defer func() {
- loadUidAndGid = origLoadUidAndGid
- }()
- loadUidAndGid = func(fi fs.FileInfo, uid, gid *int) {
- *uid = testUid
- *gid = testGid
- }
-
- info := &fileInfoNames{}
- header, err := FileInfoHeader(info, "")
- if err != nil {
- t.Fatal(err)
- }
- if header.Uname != "Uname" {
- t.Fatalf("header.Uname: got %v, want %v", header.Uname, "Uname")
- }
- if header.Gname != "Gname" {
- t.Fatalf("header.Gname: got %v, want %v", header.Gname, "Gname")
- }
-}
diff --git a/src/archive/tar/writer_test.go b/src/archive/tar/writer_test.go
index a9936d6cd5..8b113a34e0 100644
--- a/src/archive/tar/writer_test.go
+++ b/src/archive/tar/writer_test.go
@@ -581,10 +581,10 @@ func TestPaxSymlink(t *testing.T) {
t.Fatal(err)
}
hdr, err := FileInfoHeader(fileinfo, "")
- hdr.Typeflag = TypeSymlink
if err != nil {
t.Fatalf("os.Stat:1 %v", err)
}
+ hdr.Typeflag = TypeSymlink
// Force a PAX long linkname to be written
longLinkname := strings.Repeat("1234567890/1234567890", 10)
hdr.Linkname = longLinkname
@@ -761,10 +761,10 @@ func TestUSTARLongName(t *testing.T) {
t.Fatal(err)
}
hdr, err := FileInfoHeader(fileinfo, "")
- hdr.Typeflag = TypeDir
if err != nil {
t.Fatalf("os.Stat:1 %v", err)
}
+ hdr.Typeflag = TypeDir
// Force a PAX long name to be written. The name was taken from a practical example
// that fails and replaced ever char through numbers to anonymize the sample.
longName := "/0000_0000000/00000-000000000/0000_0000000/00000-0000000000000/0000_0000000/00000-0000000-00000000/0000_0000000/00000000/0000_0000000/000/0000_0000000/00000000v00/0000_0000000/000000/0000_0000000/0000000/0000_0000000/00000y-00/0000/0000/00000000/0x000000/"
diff --git a/src/archive/zip/writer.go b/src/archive/zip/writer.go
index e33df2431c..9e2dcff713 100644
--- a/src/archive/zip/writer.go
+++ b/src/archive/zip/writer.go
@@ -433,6 +433,10 @@ func writeHeader(w io.Writer, h *header) error {
// [Writer.CreateHeader], [Writer.CreateRaw], or [Writer.Close].
//
// In contrast to [Writer.CreateHeader], the bytes passed to Writer are not compressed.
+//
+// CreateRaw's argument is stored in w. If the argument is a pointer to the embedded
+// [FileHeader] in a [File] obtained from a [Reader] created from in-memory data,
+// then w will refer to all of that memory.
func (w *Writer) CreateRaw(fh *FileHeader) (io.Writer, error) {
if err := w.prepare(fh); err != nil {
return nil, err
@@ -471,7 +475,10 @@ func (w *Writer) Copy(f *File) error {
if err != nil {
return err
}
- fw, err := w.CreateRaw(&f.FileHeader)
+ // Copy the FileHeader so w doesn't store a pointer to the data
+ // of f's entire archive. See #65499.
+ fh := f.FileHeader
+ fw, err := w.CreateRaw(&fh)
if err != nil {
return err
}
diff --git a/src/builtin/builtin.go b/src/builtin/builtin.go
index da0ace1498..668c799ca7 100644
--- a/src/builtin/builtin.go
+++ b/src/builtin/builtin.go
@@ -284,9 +284,10 @@ func panic(v any)
// by restoring normal execution and retrieves the error value passed to the
// call of panic. If recover is called outside the deferred function it will
// not stop a panicking sequence. In this case, or when the goroutine is not
-// panicking, or if the argument supplied to panic was nil, recover returns
-// nil. Thus the return value from recover reports whether the goroutine is
-// panicking.
+// panicking, recover returns nil.
+//
+// Prior to Go 1.21, recover would also return nil if panic is called with
+// a nil argument. See [panic] for details.
func recover() any
// The print built-in function formats its arguments in an
diff --git a/src/bytes/boundary_test.go b/src/bytes/boundary_test.go
index f9855fcb05..67f377e089 100644
--- a/src/bytes/boundary_test.go
+++ b/src/bytes/boundary_test.go
@@ -98,3 +98,18 @@ func TestIndexNearPageBoundary(t *testing.T) {
}
q[len(q)-1] = 0
}
+
+func TestCountNearPageBoundary(t *testing.T) {
+ t.Parallel()
+ b := dangerousSlice(t)
+ for i := range b {
+ c := Count(b[i:], []byte{1})
+ if c != 0 {
+ t.Fatalf("Count(b[%d:], {1})=%d, want 0\n", i, c)
+ }
+ c = Count(b[:i], []byte{0})
+ if c != i {
+ t.Fatalf("Count(b[:%d], {0})=%d, want %d\n", i, c, i)
+ }
+ }
+}
diff --git a/src/bytes/bytes.go b/src/bytes/bytes.go
index 0679b43a20..1871814c6e 100644
--- a/src/bytes/bytes.go
+++ b/src/bytes/bytes.go
@@ -525,7 +525,7 @@ func Join(s [][]byte, sep []byte) []byte {
n += len(v)
}
- b := bytealg.MakeNoZero(n)
+ b := bytealg.MakeNoZero(n)[:n:n]
bp := copy(b, s[0])
for _, v := range s[1:] {
bp += copy(b[bp:], sep)
@@ -610,7 +610,7 @@ func Repeat(b []byte, count int) []byte {
chunkMax = len(b)
}
}
- nb := bytealg.MakeNoZero(n)
+ nb := bytealg.MakeNoZero(n)[:n:n]
bp := copy(nb, b)
for bp < n {
chunk := bp
@@ -640,7 +640,7 @@ func ToUpper(s []byte) []byte {
// Just return a copy.
return append([]byte(""), s...)
}
- b := bytealg.MakeNoZero(len(s))
+ b := bytealg.MakeNoZero(len(s))[:len(s):len(s)]
for i := 0; i < len(s); i++ {
c := s[i]
if 'a' <= c && c <= 'z' {
@@ -670,7 +670,7 @@ func ToLower(s []byte) []byte {
if !hasUpper {
return append([]byte(""), s...)
}
- b := bytealg.MakeNoZero(len(s))
+ b := bytealg.MakeNoZero(len(s))[:len(s):len(s)]
for i := 0; i < len(s); i++ {
c := s[i]
if 'A' <= c && c <= 'Z' {
diff --git a/src/bytes/bytes_test.go b/src/bytes/bytes_test.go
index f0733edd3f..5e8cf85fd9 100644
--- a/src/bytes/bytes_test.go
+++ b/src/bytes/bytes_test.go
@@ -629,6 +629,11 @@ func BenchmarkEqual(b *testing.B) {
})
sizes := []int{1, 6, 9, 15, 16, 20, 32, 4 << 10, 4 << 20, 64 << 20}
+
+ b.Run("same", func(b *testing.B) {
+ benchBytes(b, sizes, bmEqual(func(a, b []byte) bool { return Equal(a, a) }))
+ })
+
benchBytes(b, sizes, bmEqual(Equal))
}
diff --git a/src/bytes/example_test.go b/src/bytes/example_test.go
index 5a66b1e436..54df5f74e5 100644
--- a/src/bytes/example_test.go
+++ b/src/bytes/example_test.go
@@ -102,7 +102,7 @@ func ExampleBuffer_Read() {
fmt.Println(n)
fmt.Println(b.String())
fmt.Println(string(rdbuf))
- // Output
+ // Output:
// 1
// bcde
// a
@@ -118,7 +118,7 @@ func ExampleBuffer_ReadByte() {
}
fmt.Println(c)
fmt.Println(b.String())
- // Output
+ // Output:
// 97
// bcde
}
diff --git a/src/cmd/addr2line/addr2line_test.go b/src/cmd/addr2line/addr2line_test.go
index 0ea8994b6a..e5b0a0fdae 100644
--- a/src/cmd/addr2line/addr2line_test.go
+++ b/src/cmd/addr2line/addr2line_test.go
@@ -109,32 +109,18 @@ func testAddr2Line(t *testing.T, dbgExePath, addr string) {
srcPath = filepath.FromSlash(srcPath)
fi2, err := os.Stat(srcPath)
- // If GOROOT_FINAL is set and srcPath is not the file we expect, perhaps
- // srcPath has had GOROOT_FINAL substituted for GOROOT and GOROOT hasn't been
- // moved to its final location yet. If so, try the original location instead.
- if gorootFinal := os.Getenv("GOROOT_FINAL"); gorootFinal != "" &&
- (os.IsNotExist(err) || (err == nil && !os.SameFile(fi1, fi2))) {
- // srcPath is clean, but GOROOT_FINAL itself might not be.
- // (See https://golang.org/issue/41447.)
- gorootFinal = filepath.Clean(gorootFinal)
-
- if strings.HasPrefix(srcPath, gorootFinal) {
- fi2, err = os.Stat(runtime.GOROOT() + strings.TrimPrefix(srcPath, gorootFinal))
- }
- }
-
if err != nil {
t.Fatalf("Stat failed: %v", err)
}
if !os.SameFile(fi1, fi2) {
t.Fatalf("addr2line_test.go and %s are not same file", srcPath)
}
- if srcLineNo != "138" {
- t.Fatalf("line number = %v; want 138", srcLineNo)
+ if want := "124"; srcLineNo != want {
+ t.Fatalf("line number = %v; want %s", srcLineNo, want)
}
}
-// This is line 137. The test depends on that.
+// This is line 123. The test depends on that.
func TestAddr2Line(t *testing.T) {
testenv.MustHaveGoBuild(t)
diff --git a/src/cmd/api/api_test.go b/src/cmd/api/api_test.go
index 910e046f12..ba358d364d 100644
--- a/src/cmd/api/api_test.go
+++ b/src/cmd/api/api_test.go
@@ -285,6 +285,25 @@ func TestIssue41358(t *testing.T) {
}
}
+func TestIssue64958(t *testing.T) {
+ defer func() {
+ if x := recover(); x != nil {
+ t.Errorf("expected no panic; recovered %v", x)
+ }
+ }()
+
+ testenv.MustHaveGoBuild(t)
+
+ for _, context := range contexts {
+ w := NewWalker(context, "testdata/src/issue64958")
+ pkg, err := w.importFrom("p", "", 0)
+ if err != nil {
+ t.Errorf("expected no error importing; got %T", err)
+ }
+ w.export(pkg)
+ }
+}
+
func TestCheck(t *testing.T) {
if !*flagCheck {
t.Skip("-check not specified")
diff --git a/src/cmd/api/main_test.go b/src/cmd/api/main_test.go
index 94e159e7d8..0a3d44ddd0 100644
--- a/src/cmd/api/main_test.go
+++ b/src/cmd/api/main_test.go
@@ -851,6 +851,16 @@ func (w *Walker) writeType(buf *bytes.Buffer, typ types.Type) {
buf.WriteByte('.')
}
buf.WriteString(typ.Obj().Name())
+ if targs := typ.TypeArgs(); targs.Len() > 0 {
+ buf.WriteByte('[')
+ for i := 0; i < targs.Len(); i++ {
+ if i > 0 {
+ buf.WriteString(", ")
+ }
+ w.writeType(buf, targs.At(i))
+ }
+ buf.WriteByte(']')
+ }
case *types.TypeParam:
// Type parameter names may change, so use a placeholder instead.
@@ -957,17 +967,17 @@ func (w *Walker) emitType(obj *types.TypeName) {
if w.isDeprecated(obj) {
w.emitf("type %s //deprecated", name)
}
+ typ := obj.Type()
+ if obj.IsAlias() {
+ w.emitf("type %s = %s", name, w.typeString(typ))
+ return
+ }
if tparams := obj.Type().(*types.Named).TypeParams(); tparams != nil {
var buf bytes.Buffer
buf.WriteString(name)
w.writeTypeParams(&buf, tparams, true)
name = buf.String()
}
- typ := obj.Type()
- if obj.IsAlias() {
- w.emitf("type %s = %s", name, w.typeString(typ))
- return
- }
switch typ := typ.Underlying().(type) {
case *types.Struct:
w.emitStructType(name, typ)
diff --git a/src/cmd/api/testdata/src/issue64958/p/p.go b/src/cmd/api/testdata/src/issue64958/p/p.go
new file mode 100644
index 0000000000..feba86797f
--- /dev/null
+++ b/src/cmd/api/testdata/src/issue64958/p/p.go
@@ -0,0 +1,3 @@
+package p
+
+type BasicAlias = uint8
diff --git a/src/cmd/api/testdata/src/pkg/p4/golden.txt b/src/cmd/api/testdata/src/pkg/p4/golden.txt
index eec0598dcd..1ceae17386 100644
--- a/src/cmd/api/testdata/src/pkg/p4/golden.txt
+++ b/src/cmd/api/testdata/src/pkg/p4/golden.txt
@@ -1,4 +1,4 @@
-pkg p4, func NewPair[$0 interface{ M }, $1 interface{ ~int }]($0, $1) Pair
+pkg p4, func NewPair[$0 interface{ M }, $1 interface{ ~int }]($0, $1) Pair[$0, $1]
pkg p4, method (Pair[$0, $1]) Second() $1
pkg p4, method (Pair[$0, $1]) First() $0
pkg p4, type Pair[$0 interface{ M }, $1 interface{ ~int }] struct
diff --git a/src/cmd/asm/internal/asm/asm.go b/src/cmd/asm/internal/asm/asm.go
index 375ef803bb..949b688bbd 100644
--- a/src/cmd/asm/internal/asm/asm.go
+++ b/src/cmd/asm/internal/asm/asm.go
@@ -16,6 +16,7 @@ import (
"cmd/asm/internal/lex"
"cmd/internal/obj"
"cmd/internal/obj/ppc64"
+ "cmd/internal/obj/riscv"
"cmd/internal/obj/x86"
"cmd/internal/sys"
)
@@ -46,7 +47,11 @@ func (p *Parser) append(prog *obj.Prog, cond string, doLabel bool) {
p.errorf("%v", err)
return
}
-
+ case sys.RISCV64:
+ if err := riscv.ParseSuffix(prog, cond); err != nil {
+ p.errorf("unrecognized suffix .%q", cond)
+ return
+ }
default:
p.errorf("unrecognized suffix .%q", cond)
return
diff --git a/src/cmd/asm/internal/asm/endtoend_test.go b/src/cmd/asm/internal/asm/endtoend_test.go
index a2de63685c..6e1aa1cd95 100644
--- a/src/cmd/asm/internal/asm/endtoend_test.go
+++ b/src/cmd/asm/internal/asm/endtoend_test.go
@@ -141,11 +141,17 @@ Diff:
// Turn relative (PC) into absolute (PC) automatically,
// so that most branch instructions don't need comments
// giving the absolute form.
- if len(f) > 0 && strings.HasSuffix(printed, "(PC)") {
- last := f[len(f)-1]
- n, err := strconv.Atoi(last[:len(last)-len("(PC)")])
+ if len(f) > 0 && strings.Contains(printed, "(PC)") {
+ index := len(f) - 1
+ suf := "(PC)"
+ for !strings.HasSuffix(f[index], suf) {
+ index--
+ suf = "(PC),"
+ }
+ str := f[index]
+ n, err := strconv.Atoi(str[:len(str)-len(suf)])
if err == nil {
- f[len(f)-1] = fmt.Sprintf("%d(PC)", seq+n)
+ f[index] = fmt.Sprintf("%d%s", seq+n, suf)
}
}
@@ -372,10 +378,10 @@ func Test386EndToEnd(t *testing.T) {
}
func TestARMEndToEnd(t *testing.T) {
- defer func(old int) { buildcfg.GOARM = old }(buildcfg.GOARM)
+ defer func(old int) { buildcfg.GOARM.Version = old }(buildcfg.GOARM.Version)
for _, goarm := range []int{5, 6, 7} {
t.Logf("GOARM=%d", goarm)
- buildcfg.GOARM = goarm
+ buildcfg.GOARM.Version = goarm
testEndToEnd(t, "arm", "arm")
if goarm == 6 {
testEndToEnd(t, "arm", "armv6")
diff --git a/src/cmd/asm/internal/asm/parse.go b/src/cmd/asm/internal/asm/parse.go
index ef6c840dc2..7a52e54090 100644
--- a/src/cmd/asm/internal/asm/parse.go
+++ b/src/cmd/asm/internal/asm/parse.go
@@ -217,8 +217,8 @@ next:
for {
tok = p.nextToken()
if len(operands) == 0 && len(items) == 0 {
- if p.arch.InFamily(sys.ARM, sys.ARM64, sys.AMD64, sys.I386) && tok == '.' {
- // Suffixes: ARM conditionals or x86 modifiers.
+ if p.arch.InFamily(sys.ARM, sys.ARM64, sys.AMD64, sys.I386, sys.RISCV64) && tok == '.' {
+ // Suffixes: ARM conditionals, RISCV rounding mode or x86 modifiers.
tok = p.nextToken()
str := p.lex.Text()
if tok != scanner.Ident {
diff --git a/src/cmd/asm/internal/asm/testdata/arm.s b/src/cmd/asm/internal/asm/testdata/arm.s
index 2ba22c71de..93edc8854e 100644
--- a/src/cmd/asm/internal/asm/testdata/arm.s
+++ b/src/cmd/asm/internal/asm/testdata/arm.s
@@ -870,10 +870,13 @@ jmp_label_3:
BIC.S R0@>R1, R2 // 7021d2e1
// SRL
+ SRL $0, R5, R6 // 0560a0e1
+ SRL $1, R5, R6 // a560a0e1
SRL $14, R5, R6 // 2567a0e1
SRL $15, R5, R6 // a567a0e1
SRL $30, R5, R6 // 256fa0e1
SRL $31, R5, R6 // a56fa0e1
+ SRL $32, R5, R6 // 2560a0e1
SRL.S $14, R5, R6 // 2567b0e1
SRL.S $15, R5, R6 // a567b0e1
SRL.S $30, R5, R6 // 256fb0e1
@@ -892,10 +895,13 @@ jmp_label_3:
SRL.S R5, R7 // 3775b0e1
// SRA
+ SRA $0, R5, R6 // 0560a0e1
+ SRA $1, R5, R6 // c560a0e1
SRA $14, R5, R6 // 4567a0e1
SRA $15, R5, R6 // c567a0e1
SRA $30, R5, R6 // 456fa0e1
SRA $31, R5, R6 // c56fa0e1
+ SRA $32, R5, R6 // 4560a0e1
SRA.S $14, R5, R6 // 4567b0e1
SRA.S $15, R5, R6 // c567b0e1
SRA.S $30, R5, R6 // 456fb0e1
@@ -914,6 +920,8 @@ jmp_label_3:
SRA.S R5, R7 // 5775b0e1
// SLL
+ SLL $0, R5, R6 // 0560a0e1
+ SLL $1, R5, R6 // 8560a0e1
SLL $14, R5, R6 // 0567a0e1
SLL $15, R5, R6 // 8567a0e1
SLL $30, R5, R6 // 056fa0e1
@@ -935,6 +943,20 @@ jmp_label_3:
SLL R5, R7 // 1775a0e1
SLL.S R5, R7 // 1775b0e1
+// Ops with zero shifts should encode as left shifts
+ ADD R0<<0, R1, R2 // 002081e0
+ ADD R0>>0, R1, R2 // 002081e0
+ ADD R0->0, R1, R2 // 002081e0
+ ADD R0@>0, R1, R2 // 002081e0
+ MOVW R0<<0(R1), R2 // 002091e7
+ MOVW R0>>0(R1), R2 // 002091e7
+ MOVW R0->0(R1), R2 // 002091e7
+ MOVW R0@>0(R1), R2 // 002091e7
+ MOVW R0, R1<<0(R2) // 010082e7
+ MOVW R0, R1>>0(R2) // 010082e7
+ MOVW R0, R1->0(R2) // 010082e7
+ MOVW R0, R1@>0(R2) // 010082e7
+
// MULA / MULS
MULAWT R1, R2, R3, R4 // c23124e1
MULAWB R1, R2, R3, R4 // 823124e1
diff --git a/src/cmd/asm/internal/asm/testdata/arm64.s b/src/cmd/asm/internal/asm/testdata/arm64.s
index 46ea6645af..ecad08b37a 100644
--- a/src/cmd/asm/internal/asm/testdata/arm64.s
+++ b/src/cmd/asm/internal/asm/testdata/arm64.s
@@ -981,6 +981,14 @@ again:
ADR next, R11 // ADR R11 // 2b000010
next:
NOP
+ ADR -2(PC), R10 // 0a000010
+ ADR 2(PC), R16 // 10000010
+ ADR -26(PC), R1 // 01000010
+ ADR 12(PC), R2 // 02000010
+ ADRP -2(PC), R10 // 0a000090
+ ADRP 2(PC), R16 // 10000090
+ ADRP -26(PC), R1 // 01000090
+ ADRP 12(PC), R2 // 02000090
// LDP/STP
LDP (R0), (R0, R1) // 000440a9
@@ -1003,6 +1011,7 @@ next:
LDP -8(R0), (R1, R2) // 01887fa9
LDP x(SB), (R1, R2)
LDP x+8(SB), (R1, R2)
+ LDP 8(R1), (ZR, R2) // 3f8840a9
LDPW -5(R0), (R1, R2) // 1b1400d1610b4029
LDPW (R0), (R1, R2) // 01084029
LDPW 4(R0), (R1, R2) // 01884029
@@ -1020,6 +1029,7 @@ next:
LDPW 1024(RSP), (R1, R2) // fb031091610b4029
LDPW x(SB), (R1, R2)
LDPW x+8(SB), (R1, R2)
+ LDPW 8(R1), (ZR, R2) // 3f084129
LDPSW (R0), (R1, R2) // 01084069
LDPSW 4(R0), (R1, R2) // 01884069
LDPSW -4(R0), (R1, R2) // 01887f69
@@ -1036,6 +1046,7 @@ next:
LDPSW 1024(RSP), (R1, R2) // fb031091610b4069
LDPSW x(SB), (R1, R2)
LDPSW x+8(SB), (R1, R2)
+ LDPSW 8(R1), (ZR, R2) // 3f084169
STP (R3, R4), (R5) // a31000a9
STP (R3, R4), 8(R5) // a39000a9
STP.W (R3, R4), 8(R5) // a39080a9
diff --git a/src/cmd/asm/internal/asm/testdata/arm64error.s b/src/cmd/asm/internal/asm/testdata/arm64error.s
index e1eafa2b46..3ac8788424 100644
--- a/src/cmd/asm/internal/asm/testdata/arm64error.s
+++ b/src/cmd/asm/internal/asm/testdata/arm64error.s
@@ -66,7 +66,6 @@ TEXT errors(SB),$0
LDP.W 8(R3), (R2, R3) // ERROR "constrained unpredictable behavior"
LDP (R1), (R2, R2) // ERROR "constrained unpredictable behavior"
LDP (R0), (F0, F1) // ERROR "invalid register pair"
- LDP (R0), (R3, ZR) // ERROR "invalid register pair"
LDXPW (RSP), (R2, R2) // ERROR "constrained unpredictable behavior"
LDAXPW (R5), (R2, R2) // ERROR "constrained unpredictable behavior"
MOVD.P 300(R2), R3 // ERROR "offset out of range [-256,255]"
diff --git a/src/cmd/asm/internal/asm/testdata/ppc64.s b/src/cmd/asm/internal/asm/testdata/ppc64.s
index 01052b49e7..da0b25c1ac 100644
--- a/src/cmd/asm/internal/asm/testdata/ppc64.s
+++ b/src/cmd/asm/internal/asm/testdata/ppc64.s
@@ -230,10 +230,10 @@ TEXT asmtest(SB),DUPOK|NOSPLIT,$0
OR $-32767, R5, R6 // 3be080017fe62b78
OR $-32768, R6 // 3be080007fe63378
OR $-32768, R6, R7 // 3be080007fe73378
- OR $1234567, R5 // 641f001263ffd6877fe52b78
- OR $1234567, R5, R3 // 641f001263ffd6877fe32b78
+ OR $1234567, R5 // 64a5001260a5d687
+ OR $1234567, R5, R3 // 64a300126063d687
OR $2147483648, R5, R3 // 64a38000
- OR $2147483649, R5, R3 // 641f800063ff00017fe32b78
+ OR $2147483649, R5, R3 // 64a3800060630001
ORIS $255, R3, R4 // 646400ff
OR $16711680, R3, R4 // 646400ff
@@ -249,8 +249,8 @@ TEXT asmtest(SB),DUPOK|NOSPLIT,$0
XOR $-32767, R5, R6 // 3be080017fe62a78
XOR $-32768, R6 // 3be080007fe63278
XOR $-32768, R6, R7 // 3be080007fe73278
- XOR $1234567, R5 // 641f001263ffd6877fe52a78
- XOR $1234567, R5, R3 // 641f001263ffd6877fe32a78
+ XOR $1234567, R5 // 6ca5001268a5d687
+ XOR $1234567, R5, R3 // 6ca300126863d687
XORIS $15, R3, R4 // 6c64000f
XOR $983040, R3, R4 // 6c64000f
diff --git a/src/cmd/asm/internal/asm/testdata/riscv64.s b/src/cmd/asm/internal/asm/testdata/riscv64.s
index 072302b225..a5ab254eaa 100644
--- a/src/cmd/asm/internal/asm/testdata/riscv64.s
+++ b/src/cmd/asm/internal/asm/testdata/riscv64.s
@@ -233,11 +233,31 @@ start:
// 11.7: Single-Precision Floating-Point Conversion and Move Instructions
FCVTWS F0, X5 // d31200c0
+ FCVTWS.RNE F0, X5 // d30200c0
+ FCVTWS.RTZ F0, X5 // d31200c0
+ FCVTWS.RDN F0, X5 // d32200c0
+ FCVTWS.RUP F0, X5 // d33200c0
+ FCVTWS.RMM F0, X5 // d34200c0
FCVTLS F0, X5 // d31220c0
+ FCVTLS.RNE F0, X5 // d30220c0
+ FCVTLS.RTZ F0, X5 // d31220c0
+ FCVTLS.RDN F0, X5 // d32220c0
+ FCVTLS.RUP F0, X5 // d33220c0
+ FCVTLS.RMM F0, X5 // d34220c0
FCVTSW X5, F0 // 538002d0
FCVTSL X5, F0 // 538022d0
FCVTWUS F0, X5 // d31210c0
+ FCVTWUS.RNE F0, X5 // d30210c0
+ FCVTWUS.RTZ F0, X5 // d31210c0
+ FCVTWUS.RDN F0, X5 // d32210c0
+ FCVTWUS.RUP F0, X5 // d33210c0
+ FCVTWUS.RMM F0, X5 // d34210c0
FCVTLUS F0, X5 // d31230c0
+ FCVTLUS.RNE F0, X5 // d30230c0
+ FCVTLUS.RTZ F0, X5 // d31230c0
+ FCVTLUS.RDN F0, X5 // d32230c0
+ FCVTLUS.RUP F0, X5 // d33230c0
+ FCVTLUS.RMM F0, X5 // d34230c0
FCVTSWU X5, F0 // 538012d0
FCVTSLU X5, F0 // 538032d0
FSGNJS F1, F0, F2 // 53011020
@@ -277,11 +297,31 @@ start:
// 12.5: Double-Precision Floating-Point Conversion and Move Instructions
FCVTWD F0, X5 // d31200c2
+ FCVTWD.RNE F0, X5 // d30200c2
+ FCVTWD.RTZ F0, X5 // d31200c2
+ FCVTWD.RDN F0, X5 // d32200c2
+ FCVTWD.RUP F0, X5 // d33200c2
+ FCVTWD.RMM F0, X5 // d34200c2
FCVTLD F0, X5 // d31220c2
+ FCVTLD.RNE F0, X5 // d30220c2
+ FCVTLD.RTZ F0, X5 // d31220c2
+ FCVTLD.RDN F0, X5 // d32220c2
+ FCVTLD.RUP F0, X5 // d33220c2
+ FCVTLD.RMM F0, X5 // d34220c2
FCVTDW X5, F0 // 538002d2
FCVTDL X5, F0 // 538022d2
FCVTWUD F0, X5 // d31210c2
+ FCVTWUD.RNE F0, X5 // d30210c2
+ FCVTWUD.RTZ F0, X5 // d31210c2
+ FCVTWUD.RDN F0, X5 // d32210c2
+ FCVTWUD.RUP F0, X5 // d33210c2
+ FCVTWUD.RMM F0, X5 // d34210c2
FCVTLUD F0, X5 // d31230c2
+ FCVTLUD.RNE F0, X5 // d30230c2
+ FCVTLUD.RTZ F0, X5 // d31230c2
+ FCVTLUD.RDN F0, X5 // d32230c2
+ FCVTLUD.RUP F0, X5 // d33230c2
+ FCVTLUD.RMM F0, X5 // d34230c2
FCVTDWU X5, F0 // 538012d2
FCVTDLU X5, F0 // 538032d2
FCVTSD F0, F1 // d3001040
diff --git a/src/cmd/asm/internal/asm/testdata/s390x.s b/src/cmd/asm/internal/asm/testdata/s390x.s
index 82aa445356..977190678f 100644
--- a/src/cmd/asm/internal/asm/testdata/s390x.s
+++ b/src/cmd/asm/internal/asm/testdata/s390x.s
@@ -419,9 +419,9 @@ TEXT main·foo(SB),DUPOK|NOSPLIT,$16-0 // TEXT main.foo(SB), DUPOK|NOSPLIT, $16-
KMC R2, R6 // b92f0026
KLMD R2, R8 // b93f0028
KIMD R0, R4 // b93e0004
- KDSA R0, R8 // b93a0008
- KMA R6, R2, R4 // b9296024
- KMCTR R6, R2, R4 // b92d6024
+ KDSA R0, R8 // b93a0008
+ KMA R2, R6, R4 // b9296024
+ KMCTR R2, R6, R4 // b92d6024
// vector add and sub instructions
VAB V3, V4, V4 // e743400000f3
diff --git a/src/cmd/cgo/internal/test/callback_windows.go b/src/cmd/cgo/internal/test/callback_windows.go
index 95e97c9af9..77bdfa4dd3 100644
--- a/src/cmd/cgo/internal/test/callback_windows.go
+++ b/src/cmd/cgo/internal/test/callback_windows.go
@@ -29,7 +29,7 @@ USHORT backtrace(ULONG FramesToCapture, PVOID *BackTrace) {
}
ControlPc = context.Rip;
- // Check if we left the user range.
+ // Check if we left the user range.
if (ControlPc < 0x10000) {
break;
}
@@ -65,32 +65,17 @@ func testCallbackCallersSEH(t *testing.T) {
// TODO: support SEH on other architectures.
t.Skip("skipping on non-amd64")
}
- const cgoexpPrefix = "_cgoexp_"
+ // Only frames in the test package are checked.
want := []string{
- "runtime.asmcgocall_landingpad",
- "runtime.asmcgocall",
- "runtime.cgocall",
"test._Cfunc_backtrace",
"test.testCallbackCallersSEH.func1.1",
"test.testCallbackCallersSEH.func1",
"test.goCallback",
- cgoexpPrefix + "goCallback",
- "runtime.cgocallbackg1",
- "runtime.cgocallbackg",
- "runtime.cgocallbackg",
- "runtime.cgocallback",
- "crosscall2",
- "runtime.asmcgocall_landingpad",
- "runtime.asmcgocall",
- "runtime.cgocall",
"test._Cfunc_callback",
"test.nestedCall.func1",
"test.nestedCall",
"test.testCallbackCallersSEH",
"test.TestCallbackCallersSEH",
- "testing.tRunner",
- "testing.(*T).Run.gowrap1",
- "runtime.goexit",
}
pc := make([]uintptr, 100)
n := 0
@@ -105,26 +90,17 @@ func testCallbackCallersSEH(t *testing.T) {
}
fname := f.Name()
switch fname {
- case "goCallback", "callback":
- // TODO(qmuntal): investigate why these functions don't appear
+ case "goCallback":
+ // TODO(qmuntal): investigate why this function doesn't appear
// when using the external linker.
continue
}
- // Skip cgo-generated functions, the runtime might not know about them,
- // depending on the link mode.
- if strings.HasPrefix(fname, "_cgo_") {
- continue
- }
- // Remove the cgo-generated random prefix.
- if strings.HasPrefix(fname, cgoexpPrefix) {
- idx := strings.Index(fname[len(cgoexpPrefix):], "_")
- if idx >= 0 {
- fname = cgoexpPrefix + fname[len(cgoexpPrefix)+idx+1:]
- }
- }
// In module mode, this package has a fully-qualified import path.
// Remove it if present.
fname = strings.TrimPrefix(fname, "cmd/cgo/internal/")
+ if !strings.HasPrefix(fname, "test.") {
+ continue
+ }
got = append(got, fname)
}
if !reflect.DeepEqual(want, got) {
diff --git a/src/cmd/cgo/internal/test/cgo_stubs_ppc64x_internal_linking_test.go b/src/cmd/cgo/internal/test/cgo_stubs_ppc64x_internal_linking_test.go
new file mode 100644
index 0000000000..015a62ccbe
--- /dev/null
+++ b/src/cmd/cgo/internal/test/cgo_stubs_ppc64x_internal_linking_test.go
@@ -0,0 +1,15 @@
+// Copyright 2024 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:build (ppc64 || ppc64le) && internal
+
+package cgotest
+
+import "testing"
+
+// If gcc is used, and linking internally, __mulsc3 and __muldc3
+// will be linked in from libgcc which make several R_PPC64_TOC16_DS
+// relocations which may not be resolvable with the internal linker.
+func test8694(t *testing.T) { t.Skip("not supported on ppc64/ppc64le with internal linking") }
+func test9510(t *testing.T) { t.Skip("not supported on ppc64/ppc64le with internal linking") }
diff --git a/src/cmd/cgo/internal/test/issue4339.c b/src/cmd/cgo/internal/test/issue4339.c
index 15d0004078..d0e64878d1 100644
--- a/src/cmd/cgo/internal/test/issue4339.c
+++ b/src/cmd/cgo/internal/test/issue4339.c
@@ -1,3 +1,7 @@
+// Copyright 2013 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
#include
#include "issue4339.h"
diff --git a/src/cmd/cgo/internal/test/issue4339.h b/src/cmd/cgo/internal/test/issue4339.h
index 20f6cebb6b..99a09960e2 100644
--- a/src/cmd/cgo/internal/test/issue4339.h
+++ b/src/cmd/cgo/internal/test/issue4339.h
@@ -1,3 +1,7 @@
+// Copyright 2013 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
typedef struct Issue4339 Issue4339;
struct Issue4339 {
diff --git a/src/cmd/cgo/internal/test/issue8694.go b/src/cmd/cgo/internal/test/issue8694.go
index 3b8f065d27..117547cfa4 100644
--- a/src/cmd/cgo/internal/test/issue8694.go
+++ b/src/cmd/cgo/internal/test/issue8694.go
@@ -2,7 +2,7 @@
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
-//go:build !android
+//go:build !android && !((ppc64 || ppc64le) && internal)
package cgotest
diff --git a/src/cmd/cgo/internal/test/issue8756.go b/src/cmd/cgo/internal/test/issue8756.go
index 817f449e96..d8eadfde6d 100644
--- a/src/cmd/cgo/internal/test/issue8756.go
+++ b/src/cmd/cgo/internal/test/issue8756.go
@@ -1,3 +1,7 @@
+// Copyright 2016 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
package cgotest
/*
diff --git a/src/cmd/cgo/internal/test/issue8756/issue8756.go b/src/cmd/cgo/internal/test/issue8756/issue8756.go
index 223397f067..02a1424b9f 100644
--- a/src/cmd/cgo/internal/test/issue8756/issue8756.go
+++ b/src/cmd/cgo/internal/test/issue8756/issue8756.go
@@ -1,3 +1,7 @@
+// Copyright 2016 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
package issue8756
/*
diff --git a/src/cmd/cgo/internal/test/issue8828/trivial.go b/src/cmd/cgo/internal/test/issue8828/trivial.go
index e7b9a4e573..9f2619654f 100644
--- a/src/cmd/cgo/internal/test/issue8828/trivial.go
+++ b/src/cmd/cgo/internal/test/issue8828/trivial.go
@@ -1,3 +1,7 @@
+// Copyright 2014 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
package issue8828
//void foo();
diff --git a/src/cmd/cgo/internal/test/issue9026/issue9026.go b/src/cmd/cgo/internal/test/issue9026/issue9026.go
index ff269ca9eb..13bc180321 100644
--- a/src/cmd/cgo/internal/test/issue9026/issue9026.go
+++ b/src/cmd/cgo/internal/test/issue9026/issue9026.go
@@ -1,3 +1,7 @@
+// Copyright 2014 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
package issue9026
// This file appears in its own package since the assertion tests the
diff --git a/src/cmd/cgo/internal/test/issue9400/asm_mips64x.s b/src/cmd/cgo/internal/test/issue9400/asm_mips64x.s
index 1f492eafe9..3edba3dd82 100644
--- a/src/cmd/cgo/internal/test/issue9400/asm_mips64x.s
+++ b/src/cmd/cgo/internal/test/issue9400/asm_mips64x.s
@@ -1,4 +1,4 @@
-// Copyright 2016 The Go Authors. All rights reserved.
+// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
diff --git a/src/cmd/cgo/internal/test/issue9400/asm_riscv64.s b/src/cmd/cgo/internal/test/issue9400/asm_riscv64.s
index fa34f6bd37..0f10e3a326 100644
--- a/src/cmd/cgo/internal/test/issue9400/asm_riscv64.s
+++ b/src/cmd/cgo/internal/test/issue9400/asm_riscv64.s
@@ -1,4 +1,4 @@
-// Copyright 2020 The Go Authors. All rights reserved.
+// Copyright 2020 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
diff --git a/src/cmd/cgo/internal/test/issue9510.go b/src/cmd/cgo/internal/test/issue9510.go
index 7f0aff4fe4..c000a047f6 100644
--- a/src/cmd/cgo/internal/test/issue9510.go
+++ b/src/cmd/cgo/internal/test/issue9510.go
@@ -2,7 +2,7 @@
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
-//go:build cgo
+//go:build cgo && !((ppc64 || ppc64le) && internal)
// Test that we can link together two different cgo packages that both
// use the same libgcc function.
diff --git a/src/cmd/cgo/internal/test/issue9510a/a.go b/src/cmd/cgo/internal/test/issue9510a/a.go
index 1a5224b8c6..f0a0128d10 100644
--- a/src/cmd/cgo/internal/test/issue9510a/a.go
+++ b/src/cmd/cgo/internal/test/issue9510a/a.go
@@ -1,3 +1,7 @@
+// Copyright 2015 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
package issue9510a
/*
diff --git a/src/cmd/cgo/internal/test/issue9510b/b.go b/src/cmd/cgo/internal/test/issue9510b/b.go
index 5016b39597..6e22508c32 100644
--- a/src/cmd/cgo/internal/test/issue9510b/b.go
+++ b/src/cmd/cgo/internal/test/issue9510b/b.go
@@ -1,3 +1,7 @@
+// Copyright 2015 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
package issue9510b
/*
diff --git a/src/cmd/cgo/internal/test/seh_internal_windows_test.go b/src/cmd/cgo/internal/test/seh_internal_windows_test.go
new file mode 100644
index 0000000000..708ffdc6f6
--- /dev/null
+++ b/src/cmd/cgo/internal/test/seh_internal_windows_test.go
@@ -0,0 +1,16 @@
+// Copyright 2024 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:build cgo && windows && internal
+
+package cgotest
+
+import (
+ "internal/testenv"
+ "testing"
+)
+
+func TestCallbackCallersSEH(t *testing.T) {
+ testenv.SkipFlaky(t, 65116)
+}
diff --git a/src/cmd/cgo/internal/test/cgo_windows_test.go b/src/cmd/cgo/internal/test/seh_windows_test.go
similarity index 87%
rename from src/cmd/cgo/internal/test/cgo_windows_test.go
rename to src/cmd/cgo/internal/test/seh_windows_test.go
index 7bbed5b04e..4a8d5bbd4d 100644
--- a/src/cmd/cgo/internal/test/cgo_windows_test.go
+++ b/src/cmd/cgo/internal/test/seh_windows_test.go
@@ -2,7 +2,7 @@
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
-//go:build cgo && windows
+//go:build cgo && windows && !internal
package cgotest
diff --git a/src/cmd/cgo/internal/testgodefs/testdata/fieldtypedef.go b/src/cmd/cgo/internal/testgodefs/testdata/fieldtypedef.go
index b0c507477f..d3ab1902c1 100644
--- a/src/cmd/cgo/internal/testgodefs/testdata/fieldtypedef.go
+++ b/src/cmd/cgo/internal/testgodefs/testdata/fieldtypedef.go
@@ -1,4 +1,4 @@
-// Copyright 2018 The Go Authors. All rights reserve d.
+// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
diff --git a/src/cmd/cgo/internal/testsanitizers/cc_test.go b/src/cmd/cgo/internal/testsanitizers/cc_test.go
index e212a4fd98..e650de835a 100644
--- a/src/cmd/cgo/internal/testsanitizers/cc_test.go
+++ b/src/cmd/cgo/internal/testsanitizers/cc_test.go
@@ -16,8 +16,10 @@ import (
"encoding/json"
"errors"
"fmt"
+ "internal/testenv"
"os"
"os/exec"
+ "os/user"
"path/filepath"
"regexp"
"strconv"
@@ -266,12 +268,28 @@ func compilerSupportsLocation() bool {
case "gcc":
return compiler.major >= 10
case "clang":
+ // TODO(65606): The clang toolchain on the LUCI builders is not built against
+ // zlib, the ASAN runtime can't actually symbolize its own stack trace. Once
+ // this is resolved, one way or another, switch this back to 'true'. We still
+ // have coverage from the 'gcc' case above.
+ if inLUCIBuild() {
+ return false
+ }
return true
default:
return false
}
}
+// inLUCIBuild returns true if we're currently executing in a LUCI build.
+func inLUCIBuild() bool {
+ u, err := user.Current()
+ if err != nil {
+ return false
+ }
+ return testenv.Builder() != "" && u.Username == "swarming"
+}
+
// compilerRequiredTsanVersion reports whether the compiler is the version required by Tsan.
// Only restrictions for ppc64le are known; otherwise return true.
func compilerRequiredTsanVersion(goos, goarch string) bool {
diff --git a/src/cmd/cgo/internal/testsanitizers/libfuzzer_test.go b/src/cmd/cgo/internal/testsanitizers/libfuzzer_test.go
index f84c9f37ae..3f5b1d91c7 100644
--- a/src/cmd/cgo/internal/testsanitizers/libfuzzer_test.go
+++ b/src/cmd/cgo/internal/testsanitizers/libfuzzer_test.go
@@ -7,11 +7,14 @@
package sanitizers_test
import (
+ "internal/testenv"
"strings"
"testing"
)
func TestLibFuzzer(t *testing.T) {
+ testenv.MustHaveGoBuild(t)
+ testenv.MustHaveCGO(t)
goos, err := goEnv("GOOS")
if err != nil {
t.Fatal(err)
diff --git a/src/cmd/cgo/internal/testsanitizers/msan_test.go b/src/cmd/cgo/internal/testsanitizers/msan_test.go
index 1a22b5246c..83d66f6660 100644
--- a/src/cmd/cgo/internal/testsanitizers/msan_test.go
+++ b/src/cmd/cgo/internal/testsanitizers/msan_test.go
@@ -8,11 +8,14 @@ package sanitizers_test
import (
"internal/platform"
+ "internal/testenv"
"strings"
"testing"
)
func TestMSAN(t *testing.T) {
+ testenv.MustHaveGoBuild(t)
+ testenv.MustHaveCGO(t)
goos, err := goEnv("GOOS")
if err != nil {
t.Fatal(err)
diff --git a/src/cmd/compile/abi-internal.md b/src/cmd/compile/abi-internal.md
index 43dc39689b..eae230dc07 100644
--- a/src/cmd/compile/abi-internal.md
+++ b/src/cmd/compile/abi-internal.md
@@ -633,6 +633,56 @@ modifying or saving the FPCR.
Functions are allowed to modify it between calls (as long as they
restore it), but as of this writing Go code never does.
+### loong64 architecture
+
+The loong64 architecture uses R4 – R19 for integer arguments and integer results.
+
+It uses F0 – F15 for floating-point arguments and results.
+
+Registers R20 - R21, R23 – R28, R30 - R31, F16 – F31 are permanent scratch registers.
+
+Register R2 is reserved and never used.
+
+Register R20, R21 is Used by runtime.duffcopy, runtime.duffzero.
+
+Special-purpose registers used within Go generated code and Go assembly code
+are as follows:
+
+| Register | Call meaning | Return meaning | Body meaning |
+| --- | --- | --- | --- |
+| R0 | Zero value | Same | Same |
+| R1 | Link register | Link register | Scratch |
+| R3 | Stack pointer | Same | Same |
+| R20,R21 | Scratch | Scratch | Used by duffcopy, duffzero |
+| R22 | Current goroutine | Same | Same |
+| R29 | Closure context pointer | Same | Same |
+| R30, R31 | used by the assembler | Same | Same |
+
+*Rationale*: These register meanings are compatible with Go’s stack-based
+calling convention.
+
+#### Stack layout
+
+The stack pointer, R3, grows down and is aligned to 8 bytes.
+
+A function's stack frame, after the frame is created, is laid out as
+follows:
+
+ +------------------------------+
+ | ... locals ... |
+ | ... outgoing arguments ... |
+ | return PC | ← R3 points to
+ +------------------------------+ ↓ lower addresses
+
+This stack layout is used by both register-based (ABIInternal) and
+stack-based (ABI0) calling conventions.
+
+The "return PC" is loaded to the link register, R1, as part of the
+loong64 `JAL` operation.
+
+#### Flags
+All bits in CSR are system flags and are not modified by Go.
+
### ppc64 architecture
The ppc64 architecture uses R3 – R10 and R14 – R17 for integer arguments
diff --git a/src/cmd/compile/default.pgo b/src/cmd/compile/default.pgo
index 2ba79688d4..0f925ec69c 100644
Binary files a/src/cmd/compile/default.pgo and b/src/cmd/compile/default.pgo differ
diff --git a/src/cmd/compile/doc.go b/src/cmd/compile/doc.go
index 0a60368afa..507899e222 100644
--- a/src/cmd/compile/doc.go
+++ b/src/cmd/compile/doc.go
@@ -295,5 +295,27 @@ The declaration of lower.f may also have a linkname directive with a
single argument, f. This is optional, but helps alert the reader that
the function is accessed from outside the package.
+ //go:wasmimport importmodule importname
+
+The //go:wasmimport directive is wasm-only and must be followed by a
+function declaration.
+It specifies that the function is provided by a wasm module identified
+by ``importmodule`` and ``importname``.
+
+ //go:wasmimport a_module f
+ func g()
+
+The types of parameters and return values to the Go function are translated to
+Wasm according to the following table:
+
+ Go types Wasm types
+ int32, uint32 i32
+ int64, uint64 i64
+ float32 f32
+ float64 f64
+ unsafe.Pointer i32
+
+Any other parameter types are disallowed by the compiler.
+
*/
package main
diff --git a/src/cmd/compile/internal/arm/galign.go b/src/cmd/compile/internal/arm/galign.go
index 23e52bacbf..43d811832e 100644
--- a/src/cmd/compile/internal/arm/galign.go
+++ b/src/cmd/compile/internal/arm/galign.go
@@ -15,7 +15,7 @@ func Init(arch *ssagen.ArchInfo) {
arch.LinkArch = &arm.Linkarm
arch.REGSP = arm.REGSP
arch.MAXWIDTH = (1 << 32) - 1
- arch.SoftFloat = buildcfg.GOARM == 5
+ arch.SoftFloat = buildcfg.GOARM.SoftFloat
arch.ZeroRange = zerorange
arch.Ginsnop = ginsnop
diff --git a/src/cmd/compile/internal/arm/ssa.go b/src/cmd/compile/internal/arm/ssa.go
index 7fcbb4d024..638ed3ed4e 100644
--- a/src/cmd/compile/internal/arm/ssa.go
+++ b/src/cmd/compile/internal/arm/ssa.go
@@ -289,7 +289,7 @@ func ssaGenValue(s *ssagen.State, v *ssa.Value) {
case ssa.OpARMANDconst, ssa.OpARMBICconst:
// try to optimize ANDconst and BICconst to BFC, which saves bytes and ticks
// BFC is only available on ARMv7, and its result and source are in the same register
- if buildcfg.GOARM == 7 && v.Reg() == v.Args[0].Reg() {
+ if buildcfg.GOARM.Version == 7 && v.Reg() == v.Args[0].Reg() {
var val uint32
if v.Op == ssa.OpARMANDconst {
val = ^uint32(v.AuxInt)
@@ -646,7 +646,7 @@ func ssaGenValue(s *ssagen.State, v *ssa.Value) {
default:
}
}
- if buildcfg.GOARM >= 6 {
+ if buildcfg.GOARM.Version >= 6 {
// generate more efficient "MOVB/MOVBU/MOVH/MOVHU Reg@>0, Reg" on ARMv6 & ARMv7
genshift(s, v, v.Op.Asm(), 0, v.Args[0].Reg(), v.Reg(), arm.SHIFT_RR, 0)
return
diff --git a/src/cmd/compile/internal/base/debug.go b/src/cmd/compile/internal/base/debug.go
index a85f0139fc..420ad1305e 100644
--- a/src/cmd/compile/internal/base/debug.go
+++ b/src/cmd/compile/internal/base/debug.go
@@ -36,11 +36,11 @@ type DebugFlags struct {
Gossahash string `help:"hash value for use in debugging the compiler"`
InlFuncsWithClosures int `help:"allow functions with closures to be inlined" concurrent:"ok"`
InlStaticInit int `help:"allow static initialization of inlined calls" concurrent:"ok"`
- InterfaceCycles int `help:"allow anonymous interface cycles"`
Libfuzzer int `help:"enable coverage instrumentation for libfuzzer"`
LoopVar int `help:"shared (0, default), 1 (private loop variables), 2, private + log"`
LoopVarHash string `help:"for debugging changes in loop behavior. Overrides experiment and loopvar flag."`
LocationLists int `help:"print information about DWARF location list creation"`
+ MaxShapeLen int `help:"hash shape names longer than this threshold (default 500)" concurrent:"ok"`
Nil int `help:"print information about nil checks"`
NoOpenDefer int `help:"disable open-coded defers" concurrent:"ok"`
NoRefName int `help:"do not include referenced symbol names in object file" concurrent:"ok"`
diff --git a/src/cmd/compile/internal/base/flag.go b/src/cmd/compile/internal/base/flag.go
index e2e15c3c9c..5b3c3ad8c6 100644
--- a/src/cmd/compile/internal/base/flag.go
+++ b/src/cmd/compile/internal/base/flag.go
@@ -124,7 +124,7 @@ type CmdFlags struct {
TraceProfile string "help:\"write an execution trace to `file`\""
TrimPath string "help:\"remove `prefix` from recorded source file paths\""
WB bool "help:\"enable write barrier\"" // TODO: remove
- PgoProfile string "help:\"read profile from `file`\""
+ PgoProfile string "help:\"read profile or pre-process profile from `file`\""
ErrorURL bool "help:\"print explanatory URL with error message if applicable\""
// Configuration derived from flags; not a flag itself.
@@ -176,6 +176,7 @@ func ParseFlags() {
Flag.WB = true
Debug.ConcurrentOk = true
+ Debug.MaxShapeLen = 500
Debug.InlFuncsWithClosures = 1
Debug.InlStaticInit = 1
Debug.PGOInline = 1
diff --git a/src/cmd/compile/internal/base/hashdebug.go b/src/cmd/compile/internal/base/hashdebug.go
index de7f01f09e..4e36c8d549 100644
--- a/src/cmd/compile/internal/base/hashdebug.go
+++ b/src/cmd/compile/internal/base/hashdebug.go
@@ -197,14 +197,13 @@ func NewHashDebug(ev, s string, file io.Writer) *HashDebug {
break
}
if i == 0 {
- hd.matches = append(hd.matches, toHashAndMask(s, fmt.Sprintf("%s", ev)))
+ hd.matches = append(hd.matches, toHashAndMask(s, ev))
} else {
hd.matches = append(hd.matches, toHashAndMask(s, fmt.Sprintf("%s%d", ev, i-1)))
}
i++
}
return hd
-
}
// TODO: Delete when we switch to bisect-only.
diff --git a/src/cmd/compile/internal/devirtualize/devirtualize.go b/src/cmd/compile/internal/devirtualize/devirtualize.go
index 7b3a869d8e..5d1b952627 100644
--- a/src/cmd/compile/internal/devirtualize/devirtualize.go
+++ b/src/cmd/compile/internal/devirtualize/devirtualize.go
@@ -18,39 +18,27 @@ import (
"cmd/compile/internal/types"
)
-// Static devirtualizes calls within fn where possible when the concrete callee
+// StaticCall devirtualizes the given call if possible when the concrete callee
// is available statically.
-func Static(fn *ir.Func) {
- ir.CurFunc = fn
+func StaticCall(call *ir.CallExpr) {
+ // For promoted methods (including value-receiver methods promoted
+ // to pointer-receivers), the interface method wrapper may contain
+ // expressions that can panic (e.g., ODEREF, ODOTPTR,
+ // ODOTINTER). Devirtualization involves inlining these expressions
+ // (and possible panics) to the call site. This normally isn't a
+ // problem, but for go/defer statements it can move the panic from
+ // when/where the call executes to the go/defer statement itself,
+ // which is a visible change in semantics (e.g., #52072). To prevent
+ // this, we skip devirtualizing calls within go/defer statements
+ // altogether.
+ if call.GoDefer {
+ return
+ }
- // For promoted methods (including value-receiver methods promoted to pointer-receivers),
- // the interface method wrapper may contain expressions that can panic (e.g., ODEREF, ODOTPTR, ODOTINTER).
- // Devirtualization involves inlining these expressions (and possible panics) to the call site.
- // This normally isn't a problem, but for go/defer statements it can move the panic from when/where
- // the call executes to the go/defer statement itself, which is a visible change in semantics (e.g., #52072).
- // To prevent this, we skip devirtualizing calls within go/defer statements altogether.
- goDeferCall := make(map[*ir.CallExpr]bool)
- ir.VisitList(fn.Body, func(n ir.Node) {
- switch n := n.(type) {
- case *ir.GoDeferStmt:
- if call, ok := n.Call.(*ir.CallExpr); ok {
- goDeferCall[call] = true
- }
- return
- case *ir.CallExpr:
- if !goDeferCall[n] {
- staticCall(n)
- }
- }
- })
-}
-
-// staticCall devirtualizes the given call if possible when the concrete callee
-// is available statically.
-func staticCall(call *ir.CallExpr) {
if call.Op() != ir.OCALLINTER {
return
}
+
sel := call.Fun.(*ir.SelectorExpr)
r := ir.StaticValue(sel.X)
if r.Op() != ir.OCONVIFACE {
@@ -70,7 +58,7 @@ func staticCall(call *ir.CallExpr) {
return
}
- // If typ *has* a shape type, then it's an shaped, instantiated
+ // If typ *has* a shape type, then it's a shaped, instantiated
// type like T[go.shape.int], and its methods (may) have an extra
// dictionary parameter. We could devirtualize this call if we
// could derive an appropriate dictionary argument.
diff --git a/src/cmd/compile/internal/devirtualize/pgo.go b/src/cmd/compile/internal/devirtualize/pgo.go
index 05b37d6be6..5cc9fab54c 100644
--- a/src/cmd/compile/internal/devirtualize/pgo.go
+++ b/src/cmd/compile/internal/devirtualize/pgo.go
@@ -107,9 +107,6 @@ func ProfileGuided(fn *ir.Func, p *pgo.Profile) {
name := ir.LinkFuncName(fn)
- // Can't devirtualize go/defer calls. See comment in Static.
- goDeferCall := make(map[*ir.CallExpr]bool)
-
var jsonW *json.Encoder
if base.Debug.PGODebug >= 3 {
jsonW = json.NewEncoder(os.Stdout)
@@ -121,12 +118,6 @@ func ProfileGuided(fn *ir.Func, p *pgo.Profile) {
return n
}
- if gds, ok := n.(*ir.GoDeferStmt); ok {
- if call, ok := gds.Call.(*ir.CallExpr); ok {
- goDeferCall[call] = true
- }
- }
-
ir.EditChildren(n, edit)
call, ok := n.(*ir.CallExpr)
@@ -156,7 +147,7 @@ func ProfileGuided(fn *ir.Func, p *pgo.Profile) {
fmt.Printf("%v: PGO devirtualize considering call %v\n", ir.Line(call), call)
}
- if goDeferCall[call] {
+ if call.GoDefer {
if base.Debug.PGODebug >= 2 {
fmt.Printf("%v: can't PGO devirtualize go/defer call %v\n", ir.Line(call), call)
}
@@ -749,7 +740,7 @@ func findHotConcreteCallee(p *pgo.Profile, caller *ir.Func, call *ir.CallExpr, e
}
if base.Debug.PGODebug >= 2 {
- fmt.Printf("%v call %s:%d: hottest callee %s (weight %d)\n", ir.Line(call), callerName, callOffset, hottest.Dst.Name(), hottest.Weight)
+ fmt.Printf("%v: call %s:%d: hottest callee %s (weight %d)\n", ir.Line(call), callerName, callOffset, hottest.Dst.Name(), hottest.Weight)
}
return hottest.Dst.AST, hottest.Weight
}
diff --git a/src/cmd/compile/internal/dwarfgen/dwarf.go b/src/cmd/compile/internal/dwarfgen/dwarf.go
index e9553d1185..512d8d22e7 100644
--- a/src/cmd/compile/internal/dwarfgen/dwarf.go
+++ b/src/cmd/compile/internal/dwarfgen/dwarf.go
@@ -16,6 +16,7 @@ import (
"cmd/compile/internal/reflectdata"
"cmd/compile/internal/ssa"
"cmd/compile/internal/ssagen"
+ "cmd/compile/internal/typecheck"
"cmd/compile/internal/types"
"cmd/internal/dwarf"
"cmd/internal/obj"
@@ -100,7 +101,23 @@ func Info(fnsym *obj.LSym, infosym *obj.LSym, curfn obj.Func) (scopes []dwarf.Sc
}
}
- decls, dwarfVars := createDwarfVars(fnsym, isODCLFUNC, fn, apdecls)
+ var closureVars map[*ir.Name]int64
+ if fn.Needctxt() {
+ closureVars = make(map[*ir.Name]int64)
+ csiter := typecheck.NewClosureStructIter(fn.ClosureVars)
+ for {
+ n, _, offset := csiter.Next()
+ if n == nil {
+ break
+ }
+ closureVars[n] = offset
+ if n.Heapaddr != nil {
+ closureVars[n.Heapaddr] = offset
+ }
+ }
+ }
+
+ decls, dwarfVars := createDwarfVars(fnsym, isODCLFUNC, fn, apdecls, closureVars)
// For each type referenced by the functions auto vars but not
// already referenced by a dwarf var, attach an R_USETYPE relocation to
@@ -137,18 +154,18 @@ func declPos(decl *ir.Name) src.XPos {
// createDwarfVars process fn, returning a list of DWARF variables and the
// Nodes they represent.
-func createDwarfVars(fnsym *obj.LSym, complexOK bool, fn *ir.Func, apDecls []*ir.Name) ([]*ir.Name, []*dwarf.Var) {
+func createDwarfVars(fnsym *obj.LSym, complexOK bool, fn *ir.Func, apDecls []*ir.Name, closureVars map[*ir.Name]int64) ([]*ir.Name, []*dwarf.Var) {
// Collect a raw list of DWARF vars.
var vars []*dwarf.Var
var decls []*ir.Name
var selected ir.NameSet
if base.Ctxt.Flag_locationlists && base.Ctxt.Flag_optimize && fn.DebugInfo != nil && complexOK {
- decls, vars, selected = createComplexVars(fnsym, fn)
+ decls, vars, selected = createComplexVars(fnsym, fn, closureVars)
} else if fn.ABI == obj.ABIInternal && base.Flag.N != 0 && complexOK {
- decls, vars, selected = createABIVars(fnsym, fn, apDecls)
+ decls, vars, selected = createABIVars(fnsym, fn, apDecls, closureVars)
} else {
- decls, vars, selected = createSimpleVars(fnsym, apDecls)
+ decls, vars, selected = createSimpleVars(fnsym, apDecls, closureVars)
}
if fn.DebugInfo != nil {
// Recover zero sized variables eliminated by the stackframe pass
@@ -159,7 +176,7 @@ func createDwarfVars(fnsym *obj.LSym, complexOK bool, fn *ir.Func, apDecls []*ir
types.CalcSize(n.Type())
if n.Type().Size() == 0 {
decls = append(decls, n)
- vars = append(vars, createSimpleVar(fnsym, n))
+ vars = append(vars, createSimpleVar(fnsym, n, closureVars))
vars[len(vars)-1].StackOffset = 0
fnsym.Func().RecordAutoType(reflectdata.TypeLinksym(n.Type()))
}
@@ -212,16 +229,16 @@ func createDwarfVars(fnsym *obj.LSym, complexOK bool, fn *ir.Func, apDecls []*ir
// Args not of SSA-able type are treated here; they
// are homed on the stack in a single place for the
// entire call.
- vars = append(vars, createSimpleVar(fnsym, n))
+ vars = append(vars, createSimpleVar(fnsym, n, closureVars))
decls = append(decls, n)
continue
}
typename := dwarf.InfoPrefix + types.TypeSymName(n.Type())
decls = append(decls, n)
- abbrev := dwarf.DW_ABRV_AUTO_LOCLIST
+ tag := dwarf.DW_TAG_variable
isReturnValue := (n.Class == ir.PPARAMOUT)
if n.Class == ir.PPARAM || n.Class == ir.PPARAMOUT {
- abbrev = dwarf.DW_ABRV_PARAM_LOCLIST
+ tag = dwarf.DW_TAG_formal_parameter
}
if n.Esc() == ir.EscHeap {
// The variable in question has been promoted to the heap.
@@ -233,7 +250,7 @@ func createDwarfVars(fnsym *obj.LSym, complexOK bool, fn *ir.Func, apDecls []*ir
if n.InlFormal() || n.InlLocal() {
inlIndex = posInlIndex(n.Pos()) + 1
if n.InlFormal() {
- abbrev = dwarf.DW_ABRV_PARAM_LOCLIST
+ tag = dwarf.DW_TAG_formal_parameter
}
}
}
@@ -241,7 +258,8 @@ func createDwarfVars(fnsym *obj.LSym, complexOK bool, fn *ir.Func, apDecls []*ir
vars = append(vars, &dwarf.Var{
Name: n.Sym().Name,
IsReturnValue: isReturnValue,
- Abbrev: abbrev,
+ Tag: tag,
+ WithLoclist: true,
StackOffset: int32(n.FrameOffset()),
Type: base.Ctxt.Lookup(typename),
DeclFile: declpos.RelFilename(),
@@ -250,6 +268,7 @@ func createDwarfVars(fnsym *obj.LSym, complexOK bool, fn *ir.Func, apDecls []*ir
InlIndex: int32(inlIndex),
ChildIndex: -1,
DictIndex: n.DictIndex,
+ ClosureOffset: closureOffset(n, closureVars),
})
// Record go type of to insure that it gets emitted by the linker.
fnsym.Func().RecordAutoType(reflectdata.TypeLinksym(n.Type()))
@@ -333,7 +352,7 @@ func preInliningDcls(fnsym *obj.LSym) []*ir.Name {
// createSimpleVars creates a DWARF entry for every variable declared in the
// function, claiming that they are permanently on the stack.
-func createSimpleVars(fnsym *obj.LSym, apDecls []*ir.Name) ([]*ir.Name, []*dwarf.Var, ir.NameSet) {
+func createSimpleVars(fnsym *obj.LSym, apDecls []*ir.Name, closureVars map[*ir.Name]int64) ([]*ir.Name, []*dwarf.Var, ir.NameSet) {
var vars []*dwarf.Var
var decls []*ir.Name
var selected ir.NameSet
@@ -343,14 +362,14 @@ func createSimpleVars(fnsym *obj.LSym, apDecls []*ir.Name) ([]*ir.Name, []*dwarf
}
decls = append(decls, n)
- vars = append(vars, createSimpleVar(fnsym, n))
+ vars = append(vars, createSimpleVar(fnsym, n, closureVars))
selected.Add(n)
}
return decls, vars, selected
}
-func createSimpleVar(fnsym *obj.LSym, n *ir.Name) *dwarf.Var {
- var abbrev int
+func createSimpleVar(fnsym *obj.LSym, n *ir.Name, closureVars map[*ir.Name]int64) *dwarf.Var {
+ var tag int
var offs int64
localAutoOffset := func() int64 {
@@ -367,9 +386,9 @@ func createSimpleVar(fnsym *obj.LSym, n *ir.Name) *dwarf.Var {
switch n.Class {
case ir.PAUTO:
offs = localAutoOffset()
- abbrev = dwarf.DW_ABRV_AUTO
+ tag = dwarf.DW_TAG_variable
case ir.PPARAM, ir.PPARAMOUT:
- abbrev = dwarf.DW_ABRV_PARAM
+ tag = dwarf.DW_TAG_formal_parameter
if n.IsOutputParamInRegisters() {
offs = localAutoOffset()
} else {
@@ -387,7 +406,7 @@ func createSimpleVar(fnsym *obj.LSym, n *ir.Name) *dwarf.Var {
if n.InlFormal() || n.InlLocal() {
inlIndex = posInlIndex(n.Pos()) + 1
if n.InlFormal() {
- abbrev = dwarf.DW_ABRV_PARAM
+ tag = dwarf.DW_TAG_formal_parameter
}
}
}
@@ -396,7 +415,7 @@ func createSimpleVar(fnsym *obj.LSym, n *ir.Name) *dwarf.Var {
Name: n.Sym().Name,
IsReturnValue: n.Class == ir.PPARAMOUT,
IsInlFormal: n.InlFormal(),
- Abbrev: abbrev,
+ Tag: tag,
StackOffset: int32(offs),
Type: base.Ctxt.Lookup(typename),
DeclFile: declpos.RelFilename(),
@@ -405,6 +424,7 @@ func createSimpleVar(fnsym *obj.LSym, n *ir.Name) *dwarf.Var {
InlIndex: int32(inlIndex),
ChildIndex: -1,
DictIndex: n.DictIndex,
+ ClosureOffset: closureOffset(n, closureVars),
}
}
@@ -413,11 +433,11 @@ func createSimpleVar(fnsym *obj.LSym, n *ir.Name) *dwarf.Var {
// hybrid approach in which register-resident input params are
// captured with location lists, and all other vars use the "simple"
// strategy.
-func createABIVars(fnsym *obj.LSym, fn *ir.Func, apDecls []*ir.Name) ([]*ir.Name, []*dwarf.Var, ir.NameSet) {
+func createABIVars(fnsym *obj.LSym, fn *ir.Func, apDecls []*ir.Name, closureVars map[*ir.Name]int64) ([]*ir.Name, []*dwarf.Var, ir.NameSet) {
// Invoke createComplexVars to generate dwarf vars for input parameters
// that are register-allocated according to the ABI rules.
- decls, vars, selected := createComplexVars(fnsym, fn)
+ decls, vars, selected := createComplexVars(fnsym, fn, closureVars)
// Now fill in the remainder of the variables: input parameters
// that are not register-resident, output parameters, and local
@@ -432,7 +452,7 @@ func createABIVars(fnsym *obj.LSym, fn *ir.Func, apDecls []*ir.Name) ([]*ir.Name
}
decls = append(decls, n)
- vars = append(vars, createSimpleVar(fnsym, n))
+ vars = append(vars, createSimpleVar(fnsym, n, closureVars))
selected.Add(n)
}
@@ -441,7 +461,7 @@ func createABIVars(fnsym *obj.LSym, fn *ir.Func, apDecls []*ir.Name) ([]*ir.Name
// createComplexVars creates recomposed DWARF vars with location lists,
// suitable for describing optimized code.
-func createComplexVars(fnsym *obj.LSym, fn *ir.Func) ([]*ir.Name, []*dwarf.Var, ir.NameSet) {
+func createComplexVars(fnsym *obj.LSym, fn *ir.Func, closureVars map[*ir.Name]int64) ([]*ir.Name, []*dwarf.Var, ir.NameSet) {
debugInfo := fn.DebugInfo.(*ssa.FuncDebug)
// Produce a DWARF variable entry for each user variable.
@@ -456,7 +476,7 @@ func createComplexVars(fnsym *obj.LSym, fn *ir.Func) ([]*ir.Name, []*dwarf.Var,
ssaVars.Add(debugInfo.Slots[slot].N)
}
- if dvar := createComplexVar(fnsym, fn, ssa.VarID(varID)); dvar != nil {
+ if dvar := createComplexVar(fnsym, fn, ssa.VarID(varID), closureVars); dvar != nil {
decls = append(decls, n)
vars = append(vars, dvar)
}
@@ -466,16 +486,16 @@ func createComplexVars(fnsym *obj.LSym, fn *ir.Func) ([]*ir.Name, []*dwarf.Var,
}
// createComplexVar builds a single DWARF variable entry and location list.
-func createComplexVar(fnsym *obj.LSym, fn *ir.Func, varID ssa.VarID) *dwarf.Var {
+func createComplexVar(fnsym *obj.LSym, fn *ir.Func, varID ssa.VarID, closureVars map[*ir.Name]int64) *dwarf.Var {
debug := fn.DebugInfo.(*ssa.FuncDebug)
n := debug.Vars[varID]
- var abbrev int
+ var tag int
switch n.Class {
case ir.PAUTO:
- abbrev = dwarf.DW_ABRV_AUTO_LOCLIST
+ tag = dwarf.DW_TAG_variable
case ir.PPARAM, ir.PPARAMOUT:
- abbrev = dwarf.DW_ABRV_PARAM_LOCLIST
+ tag = dwarf.DW_TAG_formal_parameter
default:
return nil
}
@@ -488,7 +508,7 @@ func createComplexVar(fnsym *obj.LSym, fn *ir.Func, varID ssa.VarID) *dwarf.Var
if n.InlFormal() || n.InlLocal() {
inlIndex = posInlIndex(n.Pos()) + 1
if n.InlFormal() {
- abbrev = dwarf.DW_ABRV_PARAM_LOCLIST
+ tag = dwarf.DW_TAG_formal_parameter
}
}
}
@@ -497,19 +517,21 @@ func createComplexVar(fnsym *obj.LSym, fn *ir.Func, varID ssa.VarID) *dwarf.Var
Name: n.Sym().Name,
IsReturnValue: n.Class == ir.PPARAMOUT,
IsInlFormal: n.InlFormal(),
- Abbrev: abbrev,
+ Tag: tag,
+ WithLoclist: true,
Type: base.Ctxt.Lookup(typename),
// The stack offset is used as a sorting key, so for decomposed
// variables just give it the first one. It's not used otherwise.
// This won't work well if the first slot hasn't been assigned a stack
// location, but it's not obvious how to do better.
- StackOffset: ssagen.StackOffset(debug.Slots[debug.VarSlots[varID][0]]),
- DeclFile: declpos.RelFilename(),
- DeclLine: declpos.RelLine(),
- DeclCol: declpos.RelCol(),
- InlIndex: int32(inlIndex),
- ChildIndex: -1,
- DictIndex: n.DictIndex,
+ StackOffset: ssagen.StackOffset(debug.Slots[debug.VarSlots[varID][0]]),
+ DeclFile: declpos.RelFilename(),
+ DeclLine: declpos.RelLine(),
+ DeclCol: declpos.RelCol(),
+ InlIndex: int32(inlIndex),
+ ChildIndex: -1,
+ DictIndex: n.DictIndex,
+ ClosureOffset: closureOffset(n, closureVars),
}
list := debug.LocationLists[varID]
if len(list) != 0 {
@@ -592,3 +614,7 @@ func RecordPackageName() {
base.Ctxt.Data = append(base.Ctxt.Data, s)
s.P = []byte(types.LocalPkg.Name)
}
+
+func closureOffset(n *ir.Name, closureVars map[*ir.Name]int64) int64 {
+ return closureVars[n]
+}
diff --git a/src/cmd/compile/internal/dwarfgen/dwinl.go b/src/cmd/compile/internal/dwarfgen/dwinl.go
index 655e7c66ac..bb3ef84df8 100644
--- a/src/cmd/compile/internal/dwarfgen/dwinl.go
+++ b/src/cmd/compile/internal/dwarfgen/dwinl.go
@@ -358,7 +358,7 @@ func dumpInlCalls(inlcalls dwarf.InlCalls) {
func dumpInlVars(dwvars []*dwarf.Var) {
for i, dwv := range dwvars {
typ := "local"
- if dwv.Abbrev == dwarf.DW_ABRV_PARAM_LOCLIST || dwv.Abbrev == dwarf.DW_ABRV_PARAM {
+ if dwv.Tag == dwarf.DW_TAG_formal_parameter {
typ = "param"
}
ia := 0
diff --git a/src/cmd/compile/internal/escape/call.go b/src/cmd/compile/internal/escape/call.go
index bf40de0544..4a3753ada9 100644
--- a/src/cmd/compile/internal/escape/call.go
+++ b/src/cmd/compile/internal/escape/call.go
@@ -155,10 +155,17 @@ func (e *escape) call(ks []hole, call ir.Node) {
e.discard(call.X)
e.discard(call.Y)
- case ir.ODELETE, ir.OMAX, ir.OMIN, ir.OPRINT, ir.OPRINTLN, ir.ORECOVERFP:
+ case ir.ODELETE, ir.OPRINT, ir.OPRINTLN, ir.ORECOVERFP:
call := call.(*ir.CallExpr)
- for i := range call.Args {
- e.discard(call.Args[i])
+ for _, arg := range call.Args {
+ e.discard(arg)
+ }
+ e.discard(call.RType)
+
+ case ir.OMIN, ir.OMAX:
+ call := call.(*ir.CallExpr)
+ for _, arg := range call.Args {
+ argument(ks[0], arg)
}
e.discard(call.RType)
diff --git a/src/cmd/compile/internal/escape/graph.go b/src/cmd/compile/internal/escape/graph.go
index f3baa67223..75e2546a7b 100644
--- a/src/cmd/compile/internal/escape/graph.go
+++ b/src/cmd/compile/internal/escape/graph.go
@@ -38,7 +38,7 @@ import (
// e.value(k, n.Left)
// }
-// An location represents an abstract location that stores a Go
+// A location represents an abstract location that stores a Go
// variable.
type location struct {
n ir.Node // represented variable or expression, if any
diff --git a/src/cmd/compile/internal/gc/main.go b/src/cmd/compile/internal/gc/main.go
index a19962dabb..7e5069fced 100644
--- a/src/cmd/compile/internal/gc/main.go
+++ b/src/cmd/compile/internal/gc/main.go
@@ -9,10 +9,10 @@ import (
"bytes"
"cmd/compile/internal/base"
"cmd/compile/internal/coverage"
- "cmd/compile/internal/devirtualize"
"cmd/compile/internal/dwarfgen"
"cmd/compile/internal/escape"
"cmd/compile/internal/inline"
+ "cmd/compile/internal/inline/interleaved"
"cmd/compile/internal/ir"
"cmd/compile/internal/logopt"
"cmd/compile/internal/loopvar"
@@ -224,30 +224,15 @@ func Main(archInit func(*ssagen.ArchInfo)) {
}
}
- base.Timer.Start("fe", "pgo-devirtualization")
- if profile != nil && base.Debug.PGODevirtualize > 0 {
- // TODO(prattmic): No need to use bottom-up visit order. This
- // is mirroring the PGO IRGraph visit order, which also need
- // not be bottom-up.
- ir.VisitFuncsBottomUp(typecheck.Target.Funcs, func(list []*ir.Func, recursive bool) {
- for _, fn := range list {
- devirtualize.ProfileGuided(fn, profile)
- }
- })
- ir.CurFunc = nil
- }
+ // Interleaved devirtualization and inlining.
+ base.Timer.Start("fe", "devirtualize-and-inline")
+ interleaved.DevirtualizeAndInlinePackage(typecheck.Target, profile)
- // Inlining
- base.Timer.Start("fe", "inlining")
- if base.Flag.LowerL != 0 {
- inline.InlinePackage(profile)
- }
noder.MakeWrappers(typecheck.Target) // must happen after inlining
- // Devirtualize and get variable capture right in for loops
+ // Get variable capture right in for loops.
var transformed []loopvar.VarAndLoop
for _, fn := range typecheck.Target.Funcs {
- devirtualize.Static(fn)
transformed = append(transformed, loopvar.ForCapture(fn)...)
}
ir.CurFunc = nil
diff --git a/src/cmd/compile/internal/inline/inl.go b/src/cmd/compile/internal/inline/inl.go
index 2677ae3086..8d2de22473 100644
--- a/src/cmd/compile/internal/inline/inl.go
+++ b/src/cmd/compile/internal/inline/inl.go
@@ -29,6 +29,7 @@ package inline
import (
"fmt"
"go/constant"
+ "internal/buildcfg"
"strconv"
"cmd/compile/internal/base"
@@ -76,8 +77,8 @@ var (
inlineHotMaxBudget int32 = 2000
)
-// pgoInlinePrologue records the hot callsites from ir-graph.
-func pgoInlinePrologue(p *pgo.Profile, funcs []*ir.Func) {
+// PGOInlinePrologue records the hot callsites from ir-graph.
+func PGOInlinePrologue(p *pgo.Profile) {
if base.Debug.PGOInlineCDFThreshold != "" {
if s, err := strconv.ParseFloat(base.Debug.PGOInlineCDFThreshold, 64); err == nil && s >= 0 && s <= 100 {
inlineCDFHotCallSiteThresholdPercent = s
@@ -118,7 +119,7 @@ func pgoInlinePrologue(p *pgo.Profile, funcs []*ir.Func) {
// a percent, is the lower bound of weight for nodes to be considered hot
// (currently only used in debug prints) (in case of equal weights,
// comparing with the threshold may not accurately reflect which nodes are
-// considiered hot).
+// considered hot).
func hotNodesFromCDF(p *pgo.Profile) (float64, []pgo.NamedCallEdge) {
cum := int64(0)
for i, n := range p.NamedEdgeMap.ByWeight {
@@ -134,79 +135,43 @@ func hotNodesFromCDF(p *pgo.Profile) (float64, []pgo.NamedCallEdge) {
return 0, p.NamedEdgeMap.ByWeight
}
-// InlinePackage finds functions that can be inlined and clones them before walk expands them.
-func InlinePackage(p *pgo.Profile) {
- if base.Debug.PGOInline == 0 {
- p = nil
+// CanInlineFuncs computes whether a batch of functions are inlinable.
+func CanInlineFuncs(funcs []*ir.Func, profile *pgo.Profile) {
+ if profile != nil {
+ PGOInlinePrologue(profile)
}
- inlheur.SetupScoreAdjustments()
-
- InlineDecls(p, typecheck.Target.Funcs, true)
-
- // Perform a garbage collection of hidden closures functions that
- // are no longer reachable from top-level functions following
- // inlining. See #59404 and #59638 for more context.
- garbageCollectUnreferencedHiddenClosures()
-
- if base.Debug.DumpInlFuncProps != "" {
- inlheur.DumpFuncProps(nil, base.Debug.DumpInlFuncProps)
- }
- if inlheur.Enabled() {
- postProcessCallSites(p)
- inlheur.TearDown()
- }
-}
-
-// InlineDecls applies inlining to the given batch of declarations.
-func InlineDecls(p *pgo.Profile, funcs []*ir.Func, doInline bool) {
- if p != nil {
- pgoInlinePrologue(p, funcs)
+ if base.Flag.LowerL == 0 {
+ return
}
- doCanInline := func(n *ir.Func, recursive bool, numfns int) {
- if !recursive || numfns > 1 {
- // We allow inlining if there is no
- // recursion, or the recursion cycle is
- // across more than one function.
- CanInline(n, p)
- } else {
- if base.Flag.LowerM > 1 && n.OClosure == nil {
- fmt.Printf("%v: cannot inline %v: recursive\n", ir.Line(n), n.Nname)
+ ir.VisitFuncsBottomUp(funcs, func(funcs []*ir.Func, recursive bool) {
+ numfns := numNonClosures(funcs)
+
+ for _, fn := range funcs {
+ if !recursive || numfns > 1 {
+ // We allow inlining if there is no
+ // recursion, or the recursion cycle is
+ // across more than one function.
+ CanInline(fn, profile)
+ } else {
+ if base.Flag.LowerM > 1 && fn.OClosure == nil {
+ fmt.Printf("%v: cannot inline %v: recursive\n", ir.Line(fn), fn.Nname)
+ }
}
- }
- if inlheur.Enabled() {
- analyzeFuncProps(n, p)
- }
- }
-
- ir.VisitFuncsBottomUp(funcs, func(list []*ir.Func, recursive bool) {
- numfns := numNonClosures(list)
- // We visit functions within an SCC in fairly arbitrary order,
- // so by computing inlinability for all functions in the SCC
- // before performing any inlining, the results are less
- // sensitive to the order within the SCC (see #58905 for an
- // example).
-
- // First compute inlinability for all functions in the SCC ...
- for _, n := range list {
- doCanInline(n, recursive, numfns)
- }
- // ... then make a second pass to do inlining of calls.
- if doInline {
- for _, n := range list {
- InlineCalls(n, p)
+ if inlheur.Enabled() {
+ analyzeFuncProps(fn, profile)
}
}
})
}
-// garbageCollectUnreferencedHiddenClosures makes a pass over all the
+// GarbageCollectUnreferencedHiddenClosures makes a pass over all the
// top-level (non-hidden-closure) functions looking for nested closure
// functions that are reachable, then sweeps through the Target.Decls
// list and marks any non-reachable hidden closure function as dead.
// See issues #59404 and #59638 for more context.
-func garbageCollectUnreferencedHiddenClosures() {
+func GarbageCollectUnreferencedHiddenClosures() {
liveFuncs := make(map[*ir.Func]bool)
@@ -253,7 +218,7 @@ func garbageCollectUnreferencedHiddenClosures() {
}
// inlineBudget determines the max budget for function 'fn' prior to
-// analyzing the hairyness of the body of 'fn'. We pass in the pgo
+// analyzing the hairiness of the body of 'fn'. We pass in the pgo
// profile if available (which can change the budget), also a
// 'relaxed' flag, which expands the budget slightly to allow for the
// possibility that a call to the function might have its score
@@ -265,7 +230,7 @@ func inlineBudget(fn *ir.Func, profile *pgo.Profile, relaxed bool, verbose bool)
if profile != nil {
if n, ok := profile.WeightedCG.IRNodes[ir.LinkFuncName(fn)]; ok {
if _, ok := candHotCalleeMap[n]; ok {
- budget = int32(inlineHotMaxBudget)
+ budget = inlineHotMaxBudget
if verbose {
fmt.Printf("hot-node enabled increased budget=%v for func=%v\n", budget, ir.PkgFuncName(fn))
}
@@ -336,7 +301,7 @@ func CanInline(fn *ir.Func, profile *pgo.Profile) {
visitor := hairyVisitor{
curFunc: fn,
- isBigFunc: isBigFunc(fn),
+ isBigFunc: IsBigFunc(fn),
budget: budget,
maxBudget: budget,
extraCallCost: cc,
@@ -348,20 +313,27 @@ func CanInline(fn *ir.Func, profile *pgo.Profile) {
}
n.Func.Inl = &ir.Inline{
- Cost: budget - visitor.budget,
- Dcl: pruneUnusedAutos(n.Func.Dcl, &visitor),
- HaveDcl: true,
-
+ Cost: budget - visitor.budget,
+ Dcl: pruneUnusedAutos(n.Func.Dcl, &visitor),
+ HaveDcl: true,
CanDelayResults: canDelayResults(fn),
}
+ if base.Flag.LowerM != 0 || logopt.Enabled() {
+ noteInlinableFunc(n, fn, budget-visitor.budget)
+ }
+}
+// noteInlinableFunc issues a message to the user that the specified
+// function is inlinable.
+func noteInlinableFunc(n *ir.Name, fn *ir.Func, cost int32) {
if base.Flag.LowerM > 1 {
- fmt.Printf("%v: can inline %v with cost %d as: %v { %v }\n", ir.Line(fn), n, budget-visitor.budget, fn.Type(), ir.Nodes(fn.Body))
+ fmt.Printf("%v: can inline %v with cost %d as: %v { %v }\n", ir.Line(fn), n, cost, fn.Type(), ir.Nodes(fn.Body))
} else if base.Flag.LowerM != 0 {
fmt.Printf("%v: can inline %v\n", ir.Line(fn), n)
}
+ // JSON optimization log output.
if logopt.Enabled() {
- logopt.LogOpt(fn.Pos(), "canInlineFunction", "inline", ir.FuncName(fn), fmt.Sprintf("cost: %d", budget-visitor.budget))
+ logopt.LogOpt(fn.Pos(), "canInlineFunction", "inline", ir.FuncName(fn), fmt.Sprintf("cost: %d", cost))
}
}
@@ -585,7 +557,7 @@ opSwitch:
// Check whether we'd actually inline this call. Set
// log == false since we aren't actually doing inlining
// yet.
- if canInlineCallExpr(v.curFunc, n, callee, v.isBigFunc, false) {
+ if ok, _ := canInlineCallExpr(v.curFunc, n, callee, v.isBigFunc, false); ok {
// mkinlcall would inline this call [1], so use
// the cost of the inline body as the cost of
// the call, as that is what will actually
@@ -732,14 +704,16 @@ opSwitch:
// particular, to avoid breaking the existing inlinability regress
// tests), we need to compensate for this here.
//
- // See also identical logic in isBigFunc.
- if init := n.Rhs[0].Init(); len(init) == 1 {
- if _, ok := init[0].(*ir.AssignListStmt); ok {
- // 4 for each value, because each temporary variable now
- // appears 3 times (DCL, LHS, RHS), plus an extra DCL node.
- //
- // 1 for the extra "tmp1, tmp2 = f()" assignment statement.
- v.budget += 4*int32(len(n.Lhs)) + 1
+ // See also identical logic in IsBigFunc.
+ if len(n.Rhs) > 0 {
+ if init := n.Rhs[0].Init(); len(init) == 1 {
+ if _, ok := init[0].(*ir.AssignListStmt); ok {
+ // 4 for each value, because each temporary variable now
+ // appears 3 times (DCL, LHS, RHS), plus an extra DCL node.
+ //
+ // 1 for the extra "tmp1, tmp2 = f()" assignment statement.
+ v.budget += 4*int32(len(n.Lhs)) + 1
+ }
}
}
@@ -771,12 +745,15 @@ opSwitch:
return ir.DoChildren(n, v.do)
}
-func isBigFunc(fn *ir.Func) bool {
+// IsBigFunc reports whether fn is a "big" function.
+//
+// Note: The criteria for "big" is heuristic and subject to change.
+func IsBigFunc(fn *ir.Func) bool {
budget := inlineBigFunctionNodes
return ir.Any(fn, func(n ir.Node) bool {
// See logic in hairyVisitor.doNode, explaining unified IR's
// handling of "a, b = f()" assignments.
- if n, ok := n.(*ir.AssignListStmt); ok && n.Op() == ir.OAS2 {
+ if n, ok := n.(*ir.AssignListStmt); ok && n.Op() == ir.OAS2 && len(n.Rhs) > 0 {
if init := n.Rhs[0].Init(); len(init) == 1 {
if _, ok := init[0].(*ir.AssignListStmt); ok {
budget += 4*len(n.Lhs) + 1
@@ -789,128 +766,40 @@ func isBigFunc(fn *ir.Func) bool {
})
}
-// InlineCalls/inlnode walks fn's statements and expressions and substitutes any
-// calls made to inlineable functions. This is the external entry point.
-func InlineCalls(fn *ir.Func, profile *pgo.Profile) {
- if inlheur.Enabled() && !fn.Wrapper() {
- inlheur.ScoreCalls(fn)
- defer inlheur.ScoreCallsCleanup()
+// TryInlineCall returns an inlined call expression for call, or nil
+// if inlining is not possible.
+func TryInlineCall(callerfn *ir.Func, call *ir.CallExpr, bigCaller bool, profile *pgo.Profile) *ir.InlinedCallExpr {
+ if base.Flag.LowerL == 0 {
+ return nil
}
- if base.Debug.DumpInlFuncProps != "" && !fn.Wrapper() {
- inlheur.DumpFuncProps(fn, base.Debug.DumpInlFuncProps)
+ if call.Op() != ir.OCALLFUNC {
+ return nil
}
- savefn := ir.CurFunc
- ir.CurFunc = fn
- bigCaller := isBigFunc(fn)
- if bigCaller && base.Flag.LowerM > 1 {
- fmt.Printf("%v: function %v considered 'big'; reducing max cost of inlinees\n", ir.Line(fn), fn)
- }
- var inlCalls []*ir.InlinedCallExpr
- var edit func(ir.Node) ir.Node
- edit = func(n ir.Node) ir.Node {
- return inlnode(fn, n, bigCaller, &inlCalls, edit, profile)
- }
- ir.EditChildren(fn, edit)
-
- // If we inlined any calls, we want to recursively visit their
- // bodies for further inlining. However, we need to wait until
- // *after* the original function body has been expanded, or else
- // inlCallee can have false positives (e.g., #54632).
- for len(inlCalls) > 0 {
- call := inlCalls[0]
- inlCalls = inlCalls[1:]
- ir.EditChildren(call, edit)
+ if call.GoDefer || call.NoInline {
+ return nil
}
- ir.CurFunc = savefn
-}
-
-// inlnode recurses over the tree to find inlineable calls, which will
-// be turned into OINLCALLs by mkinlcall. When the recursion comes
-// back up will examine left, right, list, rlist, ninit, ntest, nincr,
-// nbody and nelse and use one of the 4 inlconv/glue functions above
-// to turn the OINLCALL into an expression, a statement, or patch it
-// in to this nodes list or rlist as appropriate.
-// NOTE it makes no sense to pass the glue functions down the
-// recursion to the level where the OINLCALL gets created because they
-// have to edit /this/ n, so you'd have to push that one down as well,
-// but then you may as well do it here. so this is cleaner and
-// shorter and less complicated.
-// The result of inlnode MUST be assigned back to n, e.g.
-//
-// n.Left = inlnode(n.Left)
-func inlnode(callerfn *ir.Func, n ir.Node, bigCaller bool, inlCalls *[]*ir.InlinedCallExpr, edit func(ir.Node) ir.Node, profile *pgo.Profile) ir.Node {
- if n == nil {
- return n
- }
-
- switch n.Op() {
- case ir.ODEFER, ir.OGO:
- n := n.(*ir.GoDeferStmt)
- switch call := n.Call; call.Op() {
- case ir.OCALLMETH:
- base.FatalfAt(call.Pos(), "OCALLMETH missed by typecheck")
- case ir.OCALLFUNC:
- call := call.(*ir.CallExpr)
- call.NoInline = true
- }
- case ir.OTAILCALL:
- n := n.(*ir.TailCallStmt)
- n.Call.NoInline = true // Not inline a tail call for now. Maybe we could inline it just like RETURN fn(arg)?
-
- // TODO do them here (or earlier),
- // so escape analysis can avoid more heapmoves.
- case ir.OCLOSURE:
- return n
- case ir.OCALLMETH:
- base.FatalfAt(n.Pos(), "OCALLMETH missed by typecheck")
- case ir.OCALLFUNC:
- n := n.(*ir.CallExpr)
- if n.Fun.Op() == ir.OMETHEXPR {
- // Prevent inlining some reflect.Value methods when using checkptr,
- // even when package reflect was compiled without it (#35073).
- if meth := ir.MethodExprName(n.Fun); meth != nil {
- s := meth.Sym()
- if base.Debug.Checkptr != 0 {
- switch types.ReflectSymName(s) {
- case "Value.UnsafeAddr", "Value.Pointer":
- return n
- }
- }
+ // Prevent inlining some reflect.Value methods when using checkptr,
+ // even when package reflect was compiled without it (#35073).
+ if base.Debug.Checkptr != 0 && call.Fun.Op() == ir.OMETHEXPR {
+ if method := ir.MethodExprName(call.Fun); method != nil {
+ switch types.ReflectSymName(method.Sym()) {
+ case "Value.UnsafeAddr", "Value.Pointer":
+ return nil
}
}
}
- lno := ir.SetPos(n)
-
- ir.EditChildren(n, edit)
-
- // with all the branches out of the way, it is now time to
- // transmogrify this node itself unless inhibited by the
- // switch at the top of this function.
- switch n.Op() {
- case ir.OCALLMETH:
- base.FatalfAt(n.Pos(), "OCALLMETH missed by typecheck")
-
- case ir.OCALLFUNC:
- call := n.(*ir.CallExpr)
- if call.NoInline {
- break
- }
- if base.Flag.LowerM > 3 {
- fmt.Printf("%v:call to func %+v\n", ir.Line(n), call.Fun)
- }
- if ir.IsIntrinsicCall(call) {
- break
- }
- if fn := inlCallee(callerfn, call.Fun, profile); fn != nil && typecheck.HaveInlineBody(fn) {
- n = mkinlcall(callerfn, call, fn, bigCaller, inlCalls)
- }
+ if base.Flag.LowerM > 3 {
+ fmt.Printf("%v:call to func %+v\n", ir.Line(call), call.Fun)
}
-
- base.Pos = lno
-
- return n
+ if ir.IsIntrinsicCall(call) {
+ return nil
+ }
+ if fn := inlCallee(callerfn, call.Fun, profile); fn != nil && typecheck.HaveInlineBody(fn) {
+ return mkinlcall(callerfn, call, fn, bigCaller)
+ }
+ return nil
}
// inlCallee takes a function-typed expression and returns the underlying function ONAME
@@ -961,9 +850,10 @@ var InlineCall = func(callerfn *ir.Func, call *ir.CallExpr, fn *ir.Func, inlInde
// inlineCostOK returns true if call n from caller to callee is cheap enough to
// inline. bigCaller indicates that caller is a big function.
//
-// If inlineCostOK returns false, it also returns the max cost that the callee
-// exceeded.
-func inlineCostOK(n *ir.CallExpr, caller, callee *ir.Func, bigCaller bool) (bool, int32) {
+// In addition to the "cost OK" boolean, it also returns the "max
+// cost" limit used to make the decision (which may differ depending
+// on func size), and the score assigned to this specific callsite.
+func inlineCostOK(n *ir.CallExpr, caller, callee *ir.Func, bigCaller bool) (bool, int32, int32) {
maxCost := int32(inlineMaxBudget)
if bigCaller {
// We use this to restrict inlining into very big functions.
@@ -977,12 +867,11 @@ func inlineCostOK(n *ir.CallExpr, caller, callee *ir.Func, bigCaller bool) (bool
if ok {
metric = int32(score)
}
-
}
if metric <= maxCost {
// Simple case. Function is already cheap enough.
- return true, 0
+ return true, 0, metric
}
// We'll also allow inlining of hot functions below inlineHotMaxBudget,
@@ -992,7 +881,7 @@ func inlineCostOK(n *ir.CallExpr, caller, callee *ir.Func, bigCaller bool) (bool
csi := pgo.CallSiteInfo{LineOffset: lineOffset, Caller: caller}
if _, ok := candHotEdgeMap[csi]; !ok {
// Cold
- return false, maxCost
+ return false, maxCost, metric
}
// Hot
@@ -1001,47 +890,49 @@ func inlineCostOK(n *ir.CallExpr, caller, callee *ir.Func, bigCaller bool) (bool
if base.Debug.PGODebug > 0 {
fmt.Printf("hot-big check disallows inlining for call %s (cost %d) at %v in big function %s\n", ir.PkgFuncName(callee), callee.Inl.Cost, ir.Line(n), ir.PkgFuncName(caller))
}
- return false, maxCost
+ return false, maxCost, metric
}
if metric > inlineHotMaxBudget {
- return false, inlineHotMaxBudget
+ return false, inlineHotMaxBudget, metric
}
if !base.PGOHash.MatchPosWithInfo(n.Pos(), "inline", nil) {
// De-selected by PGO Hash.
- return false, maxCost
+ return false, maxCost, metric
}
if base.Debug.PGODebug > 0 {
fmt.Printf("hot-budget check allows inlining for call %s (cost %d) at %v in function %s\n", ir.PkgFuncName(callee), callee.Inl.Cost, ir.Line(n), ir.PkgFuncName(caller))
}
- return true, 0
+ return true, 0, metric
}
-// canInlineCallsite returns true if the call n from caller to callee can be
-// inlined. bigCaller indicates that caller is a big function. log indicates
-// that the 'cannot inline' reason should be logged.
+// canInlineCallExpr returns true if the call n from caller to callee
+// can be inlined, plus the score computed for the call expr in
+// question. bigCaller indicates that caller is a big function. log
+// indicates that the 'cannot inline' reason should be logged.
//
// Preconditions: CanInline(callee) has already been called.
-func canInlineCallExpr(callerfn *ir.Func, n *ir.CallExpr, callee *ir.Func, bigCaller bool, log bool) bool {
+func canInlineCallExpr(callerfn *ir.Func, n *ir.CallExpr, callee *ir.Func, bigCaller bool, log bool) (bool, int32) {
if callee.Inl == nil {
// callee is never inlinable.
if log && logopt.Enabled() {
logopt.LogOpt(n.Pos(), "cannotInlineCall", "inline", ir.FuncName(callerfn),
fmt.Sprintf("%s cannot be inlined", ir.PkgFuncName(callee)))
}
- return false
+ return false, 0
}
- if ok, maxCost := inlineCostOK(n, callerfn, callee, bigCaller); !ok {
+ ok, maxCost, callSiteScore := inlineCostOK(n, callerfn, callee, bigCaller)
+ if !ok {
// callee cost too high for this call site.
if log && logopt.Enabled() {
logopt.LogOpt(n.Pos(), "cannotInlineCall", "inline", ir.FuncName(callerfn),
fmt.Sprintf("cost %d of %s exceeds max caller cost %d", callee.Inl.Cost, ir.PkgFuncName(callee), maxCost))
}
- return false
+ return false, 0
}
if callee == callerfn {
@@ -1049,7 +940,7 @@ func canInlineCallExpr(callerfn *ir.Func, n *ir.CallExpr, callee *ir.Func, bigCa
if log && logopt.Enabled() {
logopt.LogOpt(n.Pos(), "cannotInlineCall", "inline", fmt.Sprintf("recursive call to %s", ir.FuncName(callerfn)))
}
- return false
+ return false, 0
}
if base.Flag.Cfg.Instrumenting && types.IsNoInstrumentPkg(callee.Sym().Pkg) {
@@ -1063,7 +954,7 @@ func canInlineCallExpr(callerfn *ir.Func, n *ir.CallExpr, callee *ir.Func, bigCa
logopt.LogOpt(n.Pos(), "cannotInlineCall", "inline", ir.FuncName(callerfn),
fmt.Sprintf("call to runtime function %s in instrumented build", ir.PkgFuncName(callee)))
}
- return false
+ return false, 0
}
if base.Flag.Race && types.IsNoRacePkg(callee.Sym().Pkg) {
@@ -1071,7 +962,7 @@ func canInlineCallExpr(callerfn *ir.Func, n *ir.CallExpr, callee *ir.Func, bigCa
logopt.LogOpt(n.Pos(), "cannotInlineCall", "inline", ir.FuncName(callerfn),
fmt.Sprintf(`call to into "no-race" package function %s in race build`, ir.PkgFuncName(callee)))
}
- return false
+ return false, 0
}
// Check if we've already inlined this function at this particular
@@ -1094,24 +985,24 @@ func canInlineCallExpr(callerfn *ir.Func, n *ir.CallExpr, callee *ir.Func, bigCa
fmt.Sprintf("repeated recursive cycle to %s", ir.PkgFuncName(callee)))
}
}
- return false
+ return false, 0
}
}
- return true
+ return true, callSiteScore
}
-// If n is a OCALLFUNC node, and fn is an ONAME node for a
-// function with an inlinable body, return an OINLCALL node that can replace n.
-// The returned node's Ninit has the parameter assignments, the Nbody is the
-// inlined function body, and (List, Rlist) contain the (input, output)
-// parameters.
+// mkinlcall returns an OINLCALL node that can replace OCALLFUNC n, or
+// nil if it cannot be inlined. callerfn is the function that contains
+// n, and fn is the function being called.
+//
// The result of mkinlcall MUST be assigned back to n, e.g.
//
// n.Left = mkinlcall(n.Left, fn, isddd)
-func mkinlcall(callerfn *ir.Func, n *ir.CallExpr, fn *ir.Func, bigCaller bool, inlCalls *[]*ir.InlinedCallExpr) ir.Node {
- if !canInlineCallExpr(callerfn, n, fn, bigCaller, true) {
- return n
+func mkinlcall(callerfn *ir.Func, n *ir.CallExpr, fn *ir.Func, bigCaller bool) *ir.InlinedCallExpr {
+ ok, score := canInlineCallExpr(callerfn, n, fn, bigCaller, true)
+ if !ok {
+ return nil
}
typecheck.AssertFixedCall(n)
@@ -1169,7 +1060,12 @@ func mkinlcall(callerfn *ir.Func, n *ir.CallExpr, fn *ir.Func, bigCaller bool, i
}
if base.Flag.LowerM != 0 {
- fmt.Printf("%v: inlining call to %v\n", ir.Line(n), fn)
+ if buildcfg.Experiment.NewInliner {
+ fmt.Printf("%v: inlining call to %v with score %d\n",
+ ir.Line(n), fn, score)
+ } else {
+ fmt.Printf("%v: inlining call to %v\n", ir.Line(n), fn)
+ }
}
if base.Flag.LowerM > 2 {
fmt.Printf("%v: Before inlining: %+v\n", ir.Line(n), n)
@@ -1189,8 +1085,6 @@ func mkinlcall(callerfn *ir.Func, n *ir.CallExpr, fn *ir.Func, bigCaller bool, i
inlheur.UpdateCallsiteTable(callerfn, n, res)
}
- *inlCalls = append(*inlCalls, res)
-
return res
}
@@ -1294,7 +1188,7 @@ func isAtomicCoverageCounterUpdate(cn *ir.CallExpr) bool {
return v
}
-func postProcessCallSites(profile *pgo.Profile) {
+func PostProcessCallSites(profile *pgo.Profile) {
if base.Debug.DumpInlCallSiteScores != 0 {
budgetCallback := func(fn *ir.Func, prof *pgo.Profile) (int32, bool) {
v := inlineBudget(fn, prof, false, false)
diff --git a/src/cmd/compile/internal/inline/inlheur/analyze.go b/src/cmd/compile/internal/inline/inlheur/analyze.go
index 6c3db92afe..1fb502ac2a 100644
--- a/src/cmd/compile/internal/inline/inlheur/analyze.go
+++ b/src/cmd/compile/internal/inline/inlheur/analyze.go
@@ -95,15 +95,16 @@ func AnalyzeFunc(fn *ir.Func, canInline func(*ir.Func), budgetForFunc func(*ir.F
// only after the closures it contains have been processed, so
// iterate through the list in reverse order. Once a function has
// been analyzed, revisit the question of whether it should be
- // inlinable; if it is over the default hairyness limit and it
+ // inlinable; if it is over the default hairiness limit and it
// doesn't have any interesting properties, then we don't want
// the overhead of writing out its inline body.
+ nameFinder := newNameFinder(fn)
for i := len(funcs) - 1; i >= 0; i-- {
f := funcs[i]
if f.OClosure != nil && !f.InlinabilityChecked() {
canInline(f)
}
- funcProps := analyzeFunc(f, inlineMaxBudget)
+ funcProps := analyzeFunc(f, inlineMaxBudget, nameFinder)
revisitInlinability(f, funcProps, budgetForFunc)
if f.Inl != nil {
f.Inl.Properties = funcProps.SerializeToString()
@@ -122,11 +123,11 @@ func TearDown() {
scoreCallsCache.csl = nil
}
-func analyzeFunc(fn *ir.Func, inlineMaxBudget int) *FuncProps {
+func analyzeFunc(fn *ir.Func, inlineMaxBudget int, nf *nameFinder) *FuncProps {
if funcInlHeur, ok := fpmap[fn]; ok {
return funcInlHeur.props
}
- funcProps, fcstab := computeFuncProps(fn, inlineMaxBudget)
+ funcProps, fcstab := computeFuncProps(fn, inlineMaxBudget, nf)
file, line := fnFileLine(fn)
entry := fnInlHeur{
fname: fn.Sym().Name,
@@ -153,7 +154,7 @@ func revisitInlinability(fn *ir.Func, funcProps *FuncProps, budgetForFunc func(*
if fn.Inl == nil {
return
}
- maxAdj := int32(largestScoreAdjustment(fn, funcProps))
+ maxAdj := int32(LargestNegativeScoreAdjustment(fn, funcProps))
budget := budgetForFunc(fn)
if fn.Inl.Cost+maxAdj > budget {
fn.Inl = nil
@@ -163,7 +164,7 @@ func revisitInlinability(fn *ir.Func, funcProps *FuncProps, budgetForFunc func(*
// computeFuncProps examines the Go function 'fn' and computes for it
// a function "properties" object, to be used to drive inlining
// heuristics. See comments on the FuncProps type for more info.
-func computeFuncProps(fn *ir.Func, inlineMaxBudget int) (*FuncProps, CallSiteTab) {
+func computeFuncProps(fn *ir.Func, inlineMaxBudget int, nf *nameFinder) (*FuncProps, CallSiteTab) {
if debugTrace&debugTraceFuncs != 0 {
fmt.Fprintf(os.Stderr, "=-= starting analysis of func %v:\n%+v\n",
fn, fn)
@@ -171,13 +172,13 @@ func computeFuncProps(fn *ir.Func, inlineMaxBudget int) (*FuncProps, CallSiteTab
funcProps := new(FuncProps)
ffa := makeFuncFlagsAnalyzer(fn)
analyzers := []propAnalyzer{ffa}
- analyzers = addResultsAnalyzer(fn, analyzers, funcProps, inlineMaxBudget)
- analyzers = addParamsAnalyzer(fn, analyzers, funcProps)
+ analyzers = addResultsAnalyzer(fn, analyzers, funcProps, inlineMaxBudget, nf)
+ analyzers = addParamsAnalyzer(fn, analyzers, funcProps, nf)
runAnalyzersOnFunction(fn, analyzers)
for _, a := range analyzers {
a.setResults(funcProps)
}
- cstab := computeCallSiteTable(fn, fn.Body, nil, ffa.panicPathTable(), 0)
+ cstab := computeCallSiteTable(fn, fn.Body, nil, ffa.panicPathTable(), 0, nf)
return funcProps, cstab
}
diff --git a/src/cmd/compile/internal/inline/inlheur/analyze_func_callsites.go b/src/cmd/compile/internal/inline/inlheur/analyze_func_callsites.go
index 3e285d5181..36ebe18b82 100644
--- a/src/cmd/compile/internal/inline/inlheur/analyze_func_callsites.go
+++ b/src/cmd/compile/internal/inline/inlheur/analyze_func_callsites.go
@@ -14,23 +14,37 @@ import (
)
type callSiteAnalyzer struct {
+ fn *ir.Func
+ *nameFinder
+}
+
+type callSiteTableBuilder struct {
+ fn *ir.Func
+ *nameFinder
cstab CallSiteTab
- fn *ir.Func
ptab map[ir.Node]pstate
nstack []ir.Node
loopNest int
isInit bool
}
-func makeCallSiteAnalyzer(fn *ir.Func, cstab CallSiteTab, ptab map[ir.Node]pstate, loopNestingLevel int) *callSiteAnalyzer {
- isInit := fn.IsPackageInit() || strings.HasPrefix(fn.Sym().Name, "init.")
+func makeCallSiteAnalyzer(fn *ir.Func) *callSiteAnalyzer {
return &callSiteAnalyzer{
- fn: fn,
- cstab: cstab,
- ptab: ptab,
- isInit: isInit,
- loopNest: loopNestingLevel,
- nstack: []ir.Node{fn},
+ fn: fn,
+ nameFinder: newNameFinder(fn),
+ }
+}
+
+func makeCallSiteTableBuilder(fn *ir.Func, cstab CallSiteTab, ptab map[ir.Node]pstate, loopNestingLevel int, nf *nameFinder) *callSiteTableBuilder {
+ isInit := fn.IsPackageInit() || strings.HasPrefix(fn.Sym().Name, "init.")
+ return &callSiteTableBuilder{
+ fn: fn,
+ cstab: cstab,
+ ptab: ptab,
+ isInit: isInit,
+ loopNest: loopNestingLevel,
+ nstack: []ir.Node{fn},
+ nameFinder: nf,
}
}
@@ -39,22 +53,22 @@ func makeCallSiteAnalyzer(fn *ir.Func, cstab CallSiteTab, ptab map[ir.Node]pstat
// specific subtree within the AST for a function. The main intended
// use cases are for 'region' to be either A) an entire function body,
// or B) an inlined call expression.
-func computeCallSiteTable(fn *ir.Func, region ir.Nodes, cstab CallSiteTab, ptab map[ir.Node]pstate, loopNestingLevel int) CallSiteTab {
- csa := makeCallSiteAnalyzer(fn, cstab, ptab, loopNestingLevel)
+func computeCallSiteTable(fn *ir.Func, region ir.Nodes, cstab CallSiteTab, ptab map[ir.Node]pstate, loopNestingLevel int, nf *nameFinder) CallSiteTab {
+ cstb := makeCallSiteTableBuilder(fn, cstab, ptab, loopNestingLevel, nf)
var doNode func(ir.Node) bool
doNode = func(n ir.Node) bool {
- csa.nodeVisitPre(n)
+ cstb.nodeVisitPre(n)
ir.DoChildren(n, doNode)
- csa.nodeVisitPost(n)
+ cstb.nodeVisitPost(n)
return false
}
for _, n := range region {
doNode(n)
}
- return csa.cstab
+ return cstb.cstab
}
-func (csa *callSiteAnalyzer) flagsForNode(call *ir.CallExpr) CSPropBits {
+func (cstb *callSiteTableBuilder) flagsForNode(call *ir.CallExpr) CSPropBits {
var r CSPropBits
if debugTrace&debugTraceCalls != 0 {
@@ -63,21 +77,21 @@ func (csa *callSiteAnalyzer) flagsForNode(call *ir.CallExpr) CSPropBits {
}
// Set a bit if this call is within a loop.
- if csa.loopNest > 0 {
+ if cstb.loopNest > 0 {
r |= CallSiteInLoop
}
// Set a bit if the call is within an init function (either
// compiler-generated or user-written).
- if csa.isInit {
+ if cstb.isInit {
r |= CallSiteInInitFunc
}
// Decide whether to apply the panic path heuristic. Hack: don't
// apply this heuristic in the function "main.main" (mostly just
// to avoid annoying users).
- if !isMainMain(csa.fn) {
- r = csa.determinePanicPathBits(call, r)
+ if !isMainMain(cstb.fn) {
+ r = cstb.determinePanicPathBits(call, r)
}
return r
@@ -88,15 +102,15 @@ func (csa *callSiteAnalyzer) flagsForNode(call *ir.CallExpr) CSPropBits {
// panic/exit. Do this by walking back up the node stack to see if we
// can find either A) an enclosing panic, or B) a statement node that
// we've determined leads to a panic/exit.
-func (csa *callSiteAnalyzer) determinePanicPathBits(call ir.Node, r CSPropBits) CSPropBits {
- csa.nstack = append(csa.nstack, call)
+func (cstb *callSiteTableBuilder) determinePanicPathBits(call ir.Node, r CSPropBits) CSPropBits {
+ cstb.nstack = append(cstb.nstack, call)
defer func() {
- csa.nstack = csa.nstack[:len(csa.nstack)-1]
+ cstb.nstack = cstb.nstack[:len(cstb.nstack)-1]
}()
- for ri := range csa.nstack[:len(csa.nstack)-1] {
- i := len(csa.nstack) - ri - 1
- n := csa.nstack[i]
+ for ri := range cstb.nstack[:len(cstb.nstack)-1] {
+ i := len(cstb.nstack) - ri - 1
+ n := cstb.nstack[i]
_, isCallExpr := n.(*ir.CallExpr)
_, isStmt := n.(ir.Stmt)
if isCallExpr {
@@ -104,7 +118,7 @@ func (csa *callSiteAnalyzer) determinePanicPathBits(call ir.Node, r CSPropBits)
}
if debugTrace&debugTraceCalls != 0 {
- ps, inps := csa.ptab[n]
+ ps, inps := cstb.ptab[n]
fmt.Fprintf(os.Stderr, "=-= callpar %d op=%s ps=%s inptab=%v stmt=%v\n", i, n.Op().String(), ps.String(), inps, isStmt)
}
@@ -112,7 +126,7 @@ func (csa *callSiteAnalyzer) determinePanicPathBits(call ir.Node, r CSPropBits)
r |= CallSiteOnPanicPath
break
}
- if v, ok := csa.ptab[n]; ok {
+ if v, ok := cstb.ptab[n]; ok {
if v == psCallsPanic {
r |= CallSiteOnPanicPath
break
@@ -126,16 +140,15 @@ func (csa *callSiteAnalyzer) determinePanicPathBits(call ir.Node, r CSPropBits)
}
// propsForArg returns property bits for a given call argument expression arg.
-func (csa *callSiteAnalyzer) propsForArg(arg ir.Node) ActualExprPropBits {
- _, islit := isLiteral(arg)
- if islit {
+func (cstb *callSiteTableBuilder) propsForArg(arg ir.Node) ActualExprPropBits {
+ if cval := cstb.constValue(arg); cval != nil {
return ActualExprConstant
}
- if isConcreteConvIface(arg) {
+ if cstb.isConcreteConvIface(arg) {
return ActualExprIsConcreteConvIface
}
- fname, isfunc, _ := isFuncName(arg)
- if isfunc {
+ fname := cstb.funcName(arg)
+ if fname != nil {
if fn := fname.Func; fn != nil && typecheck.HaveInlineBody(fn) {
return ActualExprIsInlinableFunc
}
@@ -149,11 +162,11 @@ func (csa *callSiteAnalyzer) propsForArg(arg ir.Node) ActualExprPropBits {
// expression; these will be stored in the CallSite object for a given
// call and then consulted when scoring. If no arg has any interesting
// properties we try to save some space and return a nil slice.
-func (csa *callSiteAnalyzer) argPropsForCall(ce *ir.CallExpr) []ActualExprPropBits {
+func (cstb *callSiteTableBuilder) argPropsForCall(ce *ir.CallExpr) []ActualExprPropBits {
rv := make([]ActualExprPropBits, len(ce.Args))
somethingInteresting := false
for idx := range ce.Args {
- argProp := csa.propsForArg(ce.Args[idx])
+ argProp := cstb.propsForArg(ce.Args[idx])
somethingInteresting = somethingInteresting || (argProp != 0)
rv[idx] = argProp
}
@@ -163,9 +176,9 @@ func (csa *callSiteAnalyzer) argPropsForCall(ce *ir.CallExpr) []ActualExprPropBi
return rv
}
-func (csa *callSiteAnalyzer) addCallSite(callee *ir.Func, call *ir.CallExpr) {
- flags := csa.flagsForNode(call)
- argProps := csa.argPropsForCall(call)
+func (cstb *callSiteTableBuilder) addCallSite(callee *ir.Func, call *ir.CallExpr) {
+ flags := cstb.flagsForNode(call)
+ argProps := cstb.argPropsForCall(call)
if debugTrace&debugTraceCalls != 0 {
fmt.Fprintf(os.Stderr, "=-= props %+v for call %v\n", argProps, call)
}
@@ -173,12 +186,12 @@ func (csa *callSiteAnalyzer) addCallSite(callee *ir.Func, call *ir.CallExpr) {
cs := &CallSite{
Call: call,
Callee: callee,
- Assign: csa.containingAssignment(call),
+ Assign: cstb.containingAssignment(call),
ArgProps: argProps,
Flags: flags,
- ID: uint(len(csa.cstab)),
+ ID: uint(len(cstb.cstab)),
}
- if _, ok := csa.cstab[call]; ok {
+ if _, ok := cstb.cstab[call]; ok {
fmt.Fprintf(os.Stderr, "*** cstab duplicate entry at: %s\n",
fmtFullPos(call.Pos()))
fmt.Fprintf(os.Stderr, "*** call: %+v\n", call)
@@ -189,38 +202,38 @@ func (csa *callSiteAnalyzer) addCallSite(callee *ir.Func, call *ir.CallExpr) {
// on heuristics.
cs.Score = int(callee.Inl.Cost)
- if csa.cstab == nil {
- csa.cstab = make(CallSiteTab)
+ if cstb.cstab == nil {
+ cstb.cstab = make(CallSiteTab)
}
- csa.cstab[call] = cs
+ cstb.cstab[call] = cs
if debugTrace&debugTraceCalls != 0 {
fmt.Fprintf(os.Stderr, "=-= added callsite: caller=%v callee=%v n=%s\n",
- csa.fn, callee, fmtFullPos(call.Pos()))
+ cstb.fn, callee, fmtFullPos(call.Pos()))
}
}
-func (csa *callSiteAnalyzer) nodeVisitPre(n ir.Node) {
+func (cstb *callSiteTableBuilder) nodeVisitPre(n ir.Node) {
switch n.Op() {
case ir.ORANGE, ir.OFOR:
if !hasTopLevelLoopBodyReturnOrBreak(loopBody(n)) {
- csa.loopNest++
+ cstb.loopNest++
}
case ir.OCALLFUNC:
ce := n.(*ir.CallExpr)
callee := pgo.DirectCallee(ce.Fun)
if callee != nil && callee.Inl != nil {
- csa.addCallSite(callee, ce)
+ cstb.addCallSite(callee, ce)
}
}
- csa.nstack = append(csa.nstack, n)
+ cstb.nstack = append(cstb.nstack, n)
}
-func (csa *callSiteAnalyzer) nodeVisitPost(n ir.Node) {
- csa.nstack = csa.nstack[:len(csa.nstack)-1]
+func (cstb *callSiteTableBuilder) nodeVisitPost(n ir.Node) {
+ cstb.nstack = cstb.nstack[:len(cstb.nstack)-1]
switch n.Op() {
case ir.ORANGE, ir.OFOR:
if !hasTopLevelLoopBodyReturnOrBreak(loopBody(n)) {
- csa.loopNest--
+ cstb.loopNest--
}
}
}
@@ -281,8 +294,8 @@ func hasTopLevelLoopBodyReturnOrBreak(loopBody ir.Nodes) bool {
// call to a pair of auto-temps, then the second one assigning the
// auto-temps to the user-visible vars. This helper will return the
// second (outer) of these two.
-func (csa *callSiteAnalyzer) containingAssignment(n ir.Node) ir.Node {
- parent := csa.nstack[len(csa.nstack)-1]
+func (cstb *callSiteTableBuilder) containingAssignment(n ir.Node) ir.Node {
+ parent := cstb.nstack[len(cstb.nstack)-1]
// assignsOnlyAutoTemps returns TRUE of the specified OAS2FUNC
// node assigns only auto-temps.
@@ -315,12 +328,12 @@ func (csa *callSiteAnalyzer) containingAssignment(n ir.Node) ir.Node {
// OAS1({x,y},OCONVNOP(OAS2FUNC({auto1,auto2},OCALLFUNC(bar))))
//
if assignsOnlyAutoTemps(parent) {
- par2 := csa.nstack[len(csa.nstack)-2]
+ par2 := cstb.nstack[len(cstb.nstack)-2]
if par2.Op() == ir.OAS2 {
return par2
}
if par2.Op() == ir.OCONVNOP {
- par3 := csa.nstack[len(csa.nstack)-3]
+ par3 := cstb.nstack[len(cstb.nstack)-3]
if par3.Op() == ir.OAS2 {
return par3
}
@@ -378,18 +391,23 @@ func UpdateCallsiteTable(callerfn *ir.Func, n *ir.CallExpr, ic *ir.InlinedCallEx
loopNestLevel = 1
}
ptab := map[ir.Node]pstate{ic: icp}
- icstab := computeCallSiteTable(callerfn, ic.Body, nil, ptab, loopNestLevel)
+ nf := newNameFinder(nil)
+ icstab := computeCallSiteTable(callerfn, ic.Body, nil, ptab, loopNestLevel, nf)
// Record parent callsite. This is primarily for debug output.
for _, cs := range icstab {
cs.parent = oldcs
}
- // Score the calls in the inlined body. Note the setting of "doCallResults"
- // to false here: at the moment there isn't any easy way to localize
- // or region-ize the work done by "rescoreBasedOnCallResultUses", which
- // currently does a walk over the entire function to look for uses
- // of a given set of results.
+ // Score the calls in the inlined body. Note the setting of
+ // "doCallResults" to false here: at the moment there isn't any
+ // easy way to localize or region-ize the work done by
+ // "rescoreBasedOnCallResultUses", which currently does a walk
+ // over the entire function to look for uses of a given set of
+ // results. Similarly we're passing nil to makeCallSiteAnalyzer,
+ // so as to run name finding without the use of static value &
+ // friends.
+ csa := makeCallSiteAnalyzer(nil)
const doCallResults = false
- scoreCallsRegion(callerfn, ic.Body, icstab, doCallResults, ic)
+ csa.scoreCallsRegion(callerfn, ic.Body, icstab, doCallResults, ic)
}
diff --git a/src/cmd/compile/internal/inline/inlheur/analyze_func_flags.go b/src/cmd/compile/internal/inline/inlheur/analyze_func_flags.go
index 588d2f4f59..d86fd7d71b 100644
--- a/src/cmd/compile/internal/inline/inlheur/analyze_func_flags.go
+++ b/src/cmd/compile/internal/inline/inlheur/analyze_func_flags.go
@@ -66,34 +66,24 @@ func (ffa *funcFlagsAnalyzer) setResults(funcProps *FuncProps) {
funcProps.Flags = rv
}
-func (ffa *funcFlagsAnalyzer) getstate(n ir.Node) pstate {
- val, ok := ffa.nstate[n]
- if !ok {
- base.Fatalf("funcFlagsAnalyzer: fn %q node %s line %s: internal error, no setting for node:\n%+v\n", ffa.fn.Sym().Name, n.Op().String(), ir.Line(n), n)
- }
- return val
+func (ffa *funcFlagsAnalyzer) getState(n ir.Node) pstate {
+ return ffa.nstate[n]
}
-func (ffa *funcFlagsAnalyzer) setstate(n ir.Node, st pstate) {
- if _, ok := ffa.nstate[n]; ok {
- base.Fatalf("funcFlagsAnalyzer: fn %q internal error, existing setting for node:\n%+v\n", ffa.fn.Sym().Name, n)
- } else {
+func (ffa *funcFlagsAnalyzer) setState(n ir.Node, st pstate) {
+ if st != psNoInfo {
ffa.nstate[n] = st
}
}
-func (ffa *funcFlagsAnalyzer) updatestate(n ir.Node, st pstate) {
- if _, ok := ffa.nstate[n]; !ok {
- base.Fatalf("funcFlagsAnalyzer: fn %q internal error, expected existing setting for node:\n%+v\n", ffa.fn.Sym().Name, n)
+func (ffa *funcFlagsAnalyzer) updateState(n ir.Node, st pstate) {
+ if st == psNoInfo {
+ delete(ffa.nstate, n)
} else {
ffa.nstate[n] = st
}
}
-func (ffa *funcFlagsAnalyzer) setstateSoft(n ir.Node, st pstate) {
- ffa.nstate[n] = st
-}
-
func (ffa *funcFlagsAnalyzer) panicPathTable() map[ir.Node]pstate {
return ffa.nstate
}
@@ -144,7 +134,7 @@ func branchCombine(p1, p2 pstate) pstate {
}
// stateForList walks through a list of statements and computes the
-// state/diposition for the entire list as a whole, as well
+// state/disposition for the entire list as a whole, as well
// as updating disposition of intermediate nodes.
func (ffa *funcFlagsAnalyzer) stateForList(list ir.Nodes) pstate {
st := psTop
@@ -164,13 +154,13 @@ func (ffa *funcFlagsAnalyzer) stateForList(list ir.Nodes) pstate {
// line 10 will be on a panic path).
for i := len(list) - 1; i >= 0; i-- {
n := list[i]
- psi := ffa.getstate(n)
+ psi := ffa.getState(n)
if debugTrace&debugTraceFuncFlags != 0 {
fmt.Fprintf(os.Stderr, "=-= %v: stateForList n=%s ps=%s\n",
ir.Line(n), n.Op().String(), psi.String())
}
st = blockCombine(psi, st)
- ffa.updatestate(n, st)
+ ffa.updateState(n, st)
}
if st == psTop {
st = psNoInfo
@@ -237,8 +227,6 @@ func (ffa *funcFlagsAnalyzer) nodeVisitPost(n ir.Node) {
ir.Line(n), n.Op().String(), shouldVisit(n))
}
if !shouldVisit(n) {
- // invoke soft set, since node may be shared (e.g. ONAME)
- ffa.setstateSoft(n, psNoInfo)
return
}
var st pstate
@@ -361,7 +349,7 @@ func (ffa *funcFlagsAnalyzer) nodeVisitPost(n ir.Node) {
fmt.Fprintf(os.Stderr, "=-= %v: visit n=%s returns %s\n",
ir.Line(n), n.Op().String(), st.String())
}
- ffa.setstate(n, st)
+ ffa.setState(n, st)
}
func (ffa *funcFlagsAnalyzer) nodeVisitPre(n ir.Node) {
diff --git a/src/cmd/compile/internal/inline/inlheur/analyze_func_params.go b/src/cmd/compile/internal/inline/inlheur/analyze_func_params.go
index 0ce0af43a2..f6bd84c3f5 100644
--- a/src/cmd/compile/internal/inline/inlheur/analyze_func_params.go
+++ b/src/cmd/compile/internal/inline/inlheur/analyze_func_params.go
@@ -19,6 +19,7 @@ type paramsAnalyzer struct {
params []*ir.Name
top []bool
*condLevelTracker
+ *nameFinder
}
// getParams returns an *ir.Name slice containing all params for the
@@ -34,8 +35,8 @@ func getParams(fn *ir.Func) []*ir.Name {
// new list. If the function in question doesn't have any interesting
// parameters then the analyzer list is returned unchanged, and the
// params flags in "fp" are updated accordingly.
-func addParamsAnalyzer(fn *ir.Func, analyzers []propAnalyzer, fp *FuncProps) []propAnalyzer {
- pa, props := makeParamsAnalyzer(fn)
+func addParamsAnalyzer(fn *ir.Func, analyzers []propAnalyzer, fp *FuncProps, nf *nameFinder) []propAnalyzer {
+ pa, props := makeParamsAnalyzer(fn, nf)
if pa != nil {
analyzers = append(analyzers, pa)
} else {
@@ -44,11 +45,11 @@ func addParamsAnalyzer(fn *ir.Func, analyzers []propAnalyzer, fp *FuncProps) []p
return analyzers
}
-// makeParamAnalyzer creates a new helper object to analyze parameters
+// makeParamsAnalyzer creates a new helper object to analyze parameters
// of function fn. If the function doesn't have any interesting
// params, a nil helper is returned along with a set of default param
// flags for the func.
-func makeParamsAnalyzer(fn *ir.Func) (*paramsAnalyzer, []ParamPropBits) {
+func makeParamsAnalyzer(fn *ir.Func, nf *nameFinder) (*paramsAnalyzer, []ParamPropBits) {
params := getParams(fn) // includes receiver if applicable
if len(params) == 0 {
return nil, nil
@@ -98,6 +99,7 @@ func makeParamsAnalyzer(fn *ir.Func) (*paramsAnalyzer, []ParamPropBits) {
params: params,
top: top,
condLevelTracker: new(condLevelTracker),
+ nameFinder: nf,
}
return pa, nil
}
@@ -162,7 +164,7 @@ func (pa *paramsAnalyzer) callCheckParams(ce *ir.CallExpr) {
return
}
sel := ce.Fun.(*ir.SelectorExpr)
- r := ir.StaticValue(sel.X)
+ r := pa.staticValue(sel.X)
if r.Op() != ir.ONAME {
return
}
@@ -193,8 +195,8 @@ func (pa *paramsAnalyzer) callCheckParams(ce *ir.CallExpr) {
return name == p, false
})
} else {
- cname, isFunc, _ := isFuncName(called)
- if isFunc {
+ cname := pa.funcName(called)
+ if cname != nil {
pa.deriveFlagsFromCallee(ce, cname.Func)
}
}
@@ -238,7 +240,7 @@ func (pa *paramsAnalyzer) deriveFlagsFromCallee(ce *ir.CallExpr, callee *ir.Func
}
// See if one of the caller's parameters is flowing unmodified
// into this actual expression.
- r := ir.StaticValue(arg)
+ r := pa.staticValue(arg)
if r.Op() != ir.ONAME {
return
}
@@ -247,7 +249,13 @@ func (pa *paramsAnalyzer) deriveFlagsFromCallee(ce *ir.CallExpr, callee *ir.Func
return
}
callerParamIdx := pa.findParamIdx(name)
- if callerParamIdx == -1 || pa.params[callerParamIdx] == nil {
+ // note that callerParamIdx may return -1 in the case where
+ // the param belongs not to the current closure func we're
+ // analyzing but to an outer enclosing func.
+ if callerParamIdx == -1 {
+ return
+ }
+ if pa.params[callerParamIdx] == nil {
panic("something went wrong")
}
if !pa.top[callerParamIdx] &&
diff --git a/src/cmd/compile/internal/inline/inlheur/analyze_func_returns.go b/src/cmd/compile/internal/inline/inlheur/analyze_func_returns.go
index 58b0f54697..2aaa68d1b7 100644
--- a/src/cmd/compile/internal/inline/inlheur/analyze_func_returns.go
+++ b/src/cmd/compile/internal/inline/inlheur/analyze_func_returns.go
@@ -20,6 +20,7 @@ type resultsAnalyzer struct {
props []ResultPropBits
values []resultVal
inlineMaxBudget int
+ *nameFinder
}
// resultVal captures information about a specific result returned from
@@ -28,7 +29,7 @@ type resultsAnalyzer struct {
// the same function, etc. This container stores info on a the specific
// scenarios we're looking for.
type resultVal struct {
- lit constant.Value
+ cval constant.Value
fn *ir.Name
fnClo bool
top bool
@@ -40,8 +41,8 @@ type resultVal struct {
// new list. If the function in question doesn't have any returns (or
// any interesting returns) then the analyzer list is left as is, and
// the result flags in "fp" are updated accordingly.
-func addResultsAnalyzer(fn *ir.Func, analyzers []propAnalyzer, fp *FuncProps, inlineMaxBudget int) []propAnalyzer {
- ra, props := makeResultsAnalyzer(fn, inlineMaxBudget)
+func addResultsAnalyzer(fn *ir.Func, analyzers []propAnalyzer, fp *FuncProps, inlineMaxBudget int, nf *nameFinder) []propAnalyzer {
+ ra, props := makeResultsAnalyzer(fn, inlineMaxBudget, nf)
if ra != nil {
analyzers = append(analyzers, ra)
} else {
@@ -54,7 +55,7 @@ func addResultsAnalyzer(fn *ir.Func, analyzers []propAnalyzer, fp *FuncProps, in
// in function fn. If the function doesn't have any interesting
// results, a nil helper is returned along with a set of default
// result flags for the func.
-func makeResultsAnalyzer(fn *ir.Func, inlineMaxBudget int) (*resultsAnalyzer, []ResultPropBits) {
+func makeResultsAnalyzer(fn *ir.Func, inlineMaxBudget int, nf *nameFinder) (*resultsAnalyzer, []ResultPropBits) {
results := fn.Type().Results()
if len(results) == 0 {
return nil, nil
@@ -84,6 +85,7 @@ func makeResultsAnalyzer(fn *ir.Func, inlineMaxBudget int) (*resultsAnalyzer, []
props: props,
values: vals,
inlineMaxBudget: inlineMaxBudget,
+ nameFinder: nf,
}
return ra, nil
}
@@ -143,29 +145,6 @@ func (ra *resultsAnalyzer) nodeVisitPost(n ir.Node) {
}
}
-// isFuncName returns the *ir.Name for the func or method
-// corresponding to node 'n', along with a boolean indicating success,
-// and another boolean indicating whether the func is closure.
-func isFuncName(n ir.Node) (*ir.Name, bool, bool) {
- sv := ir.StaticValue(n)
- if sv.Op() == ir.ONAME {
- name := sv.(*ir.Name)
- if name.Sym() != nil && name.Class == ir.PFUNC {
- return name, true, false
- }
- }
- if sv.Op() == ir.OCLOSURE {
- cloex := sv.(*ir.ClosureExpr)
- return cloex.Func.Nname, true, true
- }
- if sv.Op() == ir.OMETHEXPR {
- if mn := ir.MethodExprName(sv); mn != nil {
- return mn, true, false
- }
- }
- return nil, false, false
-}
-
// analyzeResult examines the expression 'n' being returned as the
// 'ii'th argument in some return statement to see whether has
// interesting characteristics (for example, returns a constant), then
@@ -173,18 +152,22 @@ func isFuncName(n ir.Node) (*ir.Name, bool, bool) {
// previous result (for the given return slot) that we've already
// processed.
func (ra *resultsAnalyzer) analyzeResult(ii int, n ir.Node) {
- isAllocMem := isAllocatedMem(n)
- isConcConvItf := isConcreteConvIface(n)
- lit, isConst := isLiteral(n)
- rfunc, isFunc, isClo := isFuncName(n)
+ isAllocMem := ra.isAllocatedMem(n)
+ isConcConvItf := ra.isConcreteConvIface(n)
+ constVal := ra.constValue(n)
+ isConst := (constVal != nil)
+ isNil := ra.isNil(n)
+ rfunc := ra.funcName(n)
+ isFunc := (rfunc != nil)
+ isClo := (rfunc != nil && rfunc.Func.OClosure != nil)
curp := ra.props[ii]
- dprops, isDerivedFromCall := deriveReturnFlagsFromCallee(n)
+ dprops, isDerivedFromCall := ra.deriveReturnFlagsFromCallee(n)
newp := ResultNoInfo
- var newlit constant.Value
+ var newcval constant.Value
var newfunc *ir.Name
if debugTrace&debugTraceResults != 0 {
- fmt.Fprintf(os.Stderr, "=-= %v: analyzeResult n=%s ismem=%v isconcconv=%v isconst=%v isfunc=%v isclo=%v\n", ir.Line(n), n.Op().String(), isAllocMem, isConcConvItf, isConst, isFunc, isClo)
+ fmt.Fprintf(os.Stderr, "=-= %v: analyzeResult n=%s ismem=%v isconcconv=%v isconst=%v isnil=%v isfunc=%v isclo=%v\n", ir.Line(n), n.Op().String(), isAllocMem, isConcConvItf, isConst, isNil, isFunc, isClo)
}
if ra.values[ii].top {
@@ -201,7 +184,10 @@ func (ra *resultsAnalyzer) analyzeResult(ii int, n ir.Node) {
newfunc = rfunc
case isConst:
newp = ResultAlwaysSameConstant
- newlit = lit
+ newcval = constVal
+ case isNil:
+ newp = ResultAlwaysSameConstant
+ newcval = nil
case isDerivedFromCall:
newp = dprops
ra.values[ii].derived = true
@@ -214,17 +200,20 @@ func (ra *resultsAnalyzer) analyzeResult(ii int, n ir.Node) {
// the previous returns.
switch curp {
case ResultIsAllocatedMem:
- if isAllocatedMem(n) {
+ if isAllocMem {
newp = ResultIsAllocatedMem
}
case ResultIsConcreteTypeConvertedToInterface:
- if isConcreteConvIface(n) {
+ if isConcConvItf {
newp = ResultIsConcreteTypeConvertedToInterface
}
case ResultAlwaysSameConstant:
- if isConst && isSameLiteral(lit, ra.values[ii].lit) {
+ if isNil && ra.values[ii].cval == nil {
newp = ResultAlwaysSameConstant
- newlit = lit
+ newcval = nil
+ } else if isConst && constant.Compare(constVal, token.EQL, ra.values[ii].cval) {
+ newp = ResultAlwaysSameConstant
+ newcval = constVal
}
case ResultAlwaysSameFunc:
if isFunc && isSameFuncName(rfunc, ra.values[ii].fn) {
@@ -236,7 +225,7 @@ func (ra *resultsAnalyzer) analyzeResult(ii int, n ir.Node) {
}
ra.values[ii].fn = newfunc
ra.values[ii].fnClo = isClo
- ra.values[ii].lit = newlit
+ ra.values[ii].cval = newcval
ra.props[ii] = newp
if debugTrace&debugTraceResults != 0 {
@@ -245,15 +234,6 @@ func (ra *resultsAnalyzer) analyzeResult(ii int, n ir.Node) {
}
}
-func isAllocatedMem(n ir.Node) bool {
- sv := ir.StaticValue(n)
- switch sv.Op() {
- case ir.OMAKESLICE, ir.ONEW, ir.OPTRLIT, ir.OSLICELIT:
- return true
- }
- return false
-}
-
// deriveReturnFlagsFromCallee tries to set properties for a given
// return result where we're returning call expression; return value
// is a return property value and a boolean indicating whether the
@@ -270,7 +250,7 @@ func isAllocatedMem(n ir.Node) bool {
// set foo's return property to that of bar. In the case of "two", however,
// even though each return path returns a constant, we don't know
// whether the constants are identical, hence we need to be conservative.
-func deriveReturnFlagsFromCallee(n ir.Node) (ResultPropBits, bool) {
+func (ra *resultsAnalyzer) deriveReturnFlagsFromCallee(n ir.Node) (ResultPropBits, bool) {
if n.Op() != ir.OCALLFUNC {
return 0, false
}
@@ -282,8 +262,8 @@ func deriveReturnFlagsFromCallee(n ir.Node) (ResultPropBits, bool) {
if called.Op() != ir.ONAME {
return 0, false
}
- cname, isFunc, _ := isFuncName(called)
- if !isFunc {
+ cname := ra.funcName(called)
+ if cname == nil {
return 0, false
}
calleeProps := propsForFunc(cname.Func)
@@ -295,41 +275,3 @@ func deriveReturnFlagsFromCallee(n ir.Node) (ResultPropBits, bool) {
}
return calleeProps.ResultFlags[0], true
}
-
-func isLiteral(n ir.Node) (constant.Value, bool) {
- sv := ir.StaticValue(n)
- switch sv.Op() {
- case ir.ONIL:
- return nil, true
- case ir.OLITERAL:
- return sv.Val(), true
- }
- return nil, false
-}
-
-// isSameLiteral checks to see if 'v1' and 'v2' correspond to the same
-// literal value, or if they are both nil.
-func isSameLiteral(v1, v2 constant.Value) bool {
- if v1 == nil && v2 == nil {
- return true
- }
- if v1 == nil || v2 == nil {
- return false
- }
- return constant.Compare(v1, token.EQL, v2)
-}
-
-func isConcreteConvIface(n ir.Node) bool {
- sv := ir.StaticValue(n)
- if sv.Op() != ir.OCONVIFACE {
- return false
- }
- return !sv.(*ir.ConvExpr).X.Type().IsInterface()
-}
-
-func isSameFuncName(v1, v2 *ir.Name) bool {
- // NB: there are a few corner cases where pointer equality
- // doesn't work here, but this should be good enough for
- // our purposes here.
- return v1 == v2
-}
diff --git a/src/cmd/compile/internal/inline/inlheur/names.go b/src/cmd/compile/internal/inline/inlheur/names.go
new file mode 100644
index 0000000000..022385087b
--- /dev/null
+++ b/src/cmd/compile/internal/inline/inlheur/names.go
@@ -0,0 +1,129 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package inlheur
+
+import (
+ "cmd/compile/internal/ir"
+ "go/constant"
+)
+
+// nameFinder provides a set of "isXXX" query methods for clients to
+// ask whether a given AST node corresponds to a function, a constant
+// value, and so on. These methods use an underlying ir.ReassignOracle
+// to return more precise results in cases where an "interesting"
+// value is assigned to a singly-defined local temp. Example:
+//
+// const q = 101
+// fq := func() int { return q }
+// copyOfConstant := q
+// copyOfFunc := f
+// interestingCall(copyOfConstant, copyOfFunc)
+//
+// A name finder query method invoked on the arguments being passed to
+// "interestingCall" will be able detect that 'copyOfConstant' always
+// evaluates to a constant (even though it is in fact a PAUTO local
+// variable). A given nameFinder can also operate without using
+// ir.ReassignOracle (in cases where it is not practical to look
+// at the entire function); in such cases queries will still work
+// for explicit constant values and functions.
+type nameFinder struct {
+ ro *ir.ReassignOracle
+}
+
+// newNameFinder returns a new nameFinder object with a reassignment
+// oracle initialized based on the function fn, or if fn is nil,
+// without an underlying ReassignOracle.
+func newNameFinder(fn *ir.Func) *nameFinder {
+ var ro *ir.ReassignOracle
+ if fn != nil {
+ ro = &ir.ReassignOracle{}
+ ro.Init(fn)
+ }
+ return &nameFinder{ro: ro}
+}
+
+// funcName returns the *ir.Name for the func or method
+// corresponding to node 'n', or nil if n can't be proven
+// to contain a function value.
+func (nf *nameFinder) funcName(n ir.Node) *ir.Name {
+ sv := n
+ if nf.ro != nil {
+ sv = nf.ro.StaticValue(n)
+ }
+ if name := ir.StaticCalleeName(sv); name != nil {
+ return name
+ }
+ return nil
+}
+
+// isAllocatedMem returns true if node n corresponds to a memory
+// allocation expression (make, new, or equivalent).
+func (nf *nameFinder) isAllocatedMem(n ir.Node) bool {
+ sv := n
+ if nf.ro != nil {
+ sv = nf.ro.StaticValue(n)
+ }
+ switch sv.Op() {
+ case ir.OMAKESLICE, ir.ONEW, ir.OPTRLIT, ir.OSLICELIT:
+ return true
+ }
+ return false
+}
+
+// constValue returns the underlying constant.Value for an AST node n
+// if n is itself a constant value/expr, or if n is a singly assigned
+// local containing constant expr/value (or nil not constant).
+func (nf *nameFinder) constValue(n ir.Node) constant.Value {
+ sv := n
+ if nf.ro != nil {
+ sv = nf.ro.StaticValue(n)
+ }
+ if sv.Op() == ir.OLITERAL {
+ return sv.Val()
+ }
+ return nil
+}
+
+// isNil returns whether n is nil (or singly
+// assigned local containing nil).
+func (nf *nameFinder) isNil(n ir.Node) bool {
+ sv := n
+ if nf.ro != nil {
+ sv = nf.ro.StaticValue(n)
+ }
+ return sv.Op() == ir.ONIL
+}
+
+func (nf *nameFinder) staticValue(n ir.Node) ir.Node {
+ if nf.ro == nil {
+ return n
+ }
+ return nf.ro.StaticValue(n)
+}
+
+func (nf *nameFinder) reassigned(n *ir.Name) bool {
+ if nf.ro == nil {
+ return true
+ }
+ return nf.ro.Reassigned(n)
+}
+
+func (nf *nameFinder) isConcreteConvIface(n ir.Node) bool {
+ sv := n
+ if nf.ro != nil {
+ sv = nf.ro.StaticValue(n)
+ }
+ if sv.Op() != ir.OCONVIFACE {
+ return false
+ }
+ return !sv.(*ir.ConvExpr).X.Type().IsInterface()
+}
+
+func isSameFuncName(v1, v2 *ir.Name) bool {
+ // NB: there are a few corner cases where pointer equality
+ // doesn't work here, but this should be good enough for
+ // our purposes here.
+ return v1 == v2
+}
diff --git a/src/cmd/compile/internal/inline/inlheur/score_callresult_uses.go b/src/cmd/compile/internal/inline/inlheur/score_callresult_uses.go
index 1d31f09ac0..b95ea37d59 100644
--- a/src/cmd/compile/internal/inline/inlheur/score_callresult_uses.go
+++ b/src/cmd/compile/internal/inline/inlheur/score_callresult_uses.go
@@ -46,7 +46,7 @@ type resultUseAnalyzer struct {
// rescoreBasedOnCallResultUses examines how call results are used,
// and tries to update the scores of calls based on how their results
// are used in the function.
-func rescoreBasedOnCallResultUses(fn *ir.Func, resultNameTab map[*ir.Name]resultPropAndCS, cstab CallSiteTab) {
+func (csa *callSiteAnalyzer) rescoreBasedOnCallResultUses(fn *ir.Func, resultNameTab map[*ir.Name]resultPropAndCS, cstab CallSiteTab) {
enableDebugTraceIfEnv()
rua := &resultUseAnalyzer{
resultNameTab: resultNameTab,
@@ -65,7 +65,7 @@ func rescoreBasedOnCallResultUses(fn *ir.Func, resultNameTab map[*ir.Name]result
disableDebugTrace()
}
-func examineCallResults(cs *CallSite, resultNameTab map[*ir.Name]resultPropAndCS) map[*ir.Name]resultPropAndCS {
+func (csa *callSiteAnalyzer) examineCallResults(cs *CallSite, resultNameTab map[*ir.Name]resultPropAndCS) map[*ir.Name]resultPropAndCS {
if debugTrace&debugTraceScoring != 0 {
fmt.Fprintf(os.Stderr, "=-= examining call results for %q\n",
EncodeCallSiteKey(cs))
@@ -103,7 +103,7 @@ func examineCallResults(cs *CallSite, resultNameTab map[*ir.Name]resultPropAndCS
if rprop&interesting == 0 {
continue
}
- if ir.Reassigned(n) {
+ if csa.nameFinder.reassigned(n) {
continue
}
if resultNameTab == nil {
diff --git a/src/cmd/compile/internal/inline/inlheur/scoring.go b/src/cmd/compile/internal/inline/inlheur/scoring.go
index 2b210fce8e..3de95d46b4 100644
--- a/src/cmd/compile/internal/inline/inlheur/scoring.go
+++ b/src/cmd/compile/internal/inline/inlheur/scoring.go
@@ -182,13 +182,14 @@ func mustToMay(x scoreAdjustTyp) scoreAdjustTyp {
return 0
}
-// computeCallSiteScore takes a given call site whose ir node is 'call' and
-// callee function is 'callee' and with previously computed call site
-// properties 'csflags', then computes a score for the callsite that
-// combines the size cost of the callee with heuristics based on
-// previously parameter and function properties, then stores the score
-// and the adjustment mask in the appropriate fields in 'cs'
-func (cs *CallSite) computeCallSiteScore(calleeProps *FuncProps) {
+// computeCallSiteScore takes a given call site whose ir node is
+// 'call' and callee function is 'callee' and with previously computed
+// call site properties 'csflags', then computes a score for the
+// callsite that combines the size cost of the callee with heuristics
+// based on previously computed argument and function properties,
+// then stores the score and the adjustment mask in the appropriate
+// fields in 'cs'
+func (cs *CallSite) computeCallSiteScore(csa *callSiteAnalyzer, calleeProps *FuncProps) {
callee := cs.Callee
csflags := cs.Flags
call := cs.Call
@@ -353,7 +354,7 @@ func setupFlagToAdjMaps() {
}
}
-// largestScoreAdjustment tries to estimate the largest possible
+// LargestNegativeScoreAdjustment tries to estimate the largest possible
// negative score adjustment that could be applied to a call of the
// function with the specified props. Example:
//
@@ -372,7 +373,7 @@ func setupFlagToAdjMaps() {
// given call _could_ be rescored down as much as -35 points-- thus if
// the size of "bar" is 100 (for example) then there is at least a
// chance that scoring will enable inlining.
-func largestScoreAdjustment(fn *ir.Func, props *FuncProps) int {
+func LargestNegativeScoreAdjustment(fn *ir.Func, props *FuncProps) int {
if resultFlagToPositiveAdj == nil {
setupFlagToAdjMaps()
}
@@ -397,6 +398,14 @@ func largestScoreAdjustment(fn *ir.Func, props *FuncProps) int {
return score
}
+// LargestPositiveScoreAdjustment tries to estimate the largest possible
+// positive score adjustment that could be applied to a given callsite.
+// At the moment we don't have very many positive score adjustments, so
+// this is just hard-coded, not table-driven.
+func LargestPositiveScoreAdjustment(fn *ir.Func) int {
+ return adjValues[panicPathAdj] + adjValues[initFuncAdj]
+}
+
// callSiteTab contains entries for each call in the function
// currently being processed by InlineCalls; this variable will either
// be set to 'cstabCache' below (for non-inlinable routines) or to the
@@ -438,8 +447,13 @@ type scoreCallsCacheType struct {
// after foo has been analyzed, but it's conceivable that CanInline
// might visit bar before foo for this SCC.
func ScoreCalls(fn *ir.Func) {
+ if len(fn.Body) == 0 {
+ return
+ }
enableDebugTraceIfEnv()
+ nameFinder := newNameFinder(fn)
+
if debugTrace&debugTraceScoring != 0 {
fmt.Fprintf(os.Stderr, "=-= ScoreCalls(%v)\n", ir.FuncName(fn))
}
@@ -461,21 +475,25 @@ func ScoreCalls(fn *ir.Func) {
fmt.Fprintf(os.Stderr, "=-= building cstab for non-inl func %s\n",
ir.FuncName(fn))
}
- cstab = computeCallSiteTable(fn, fn.Body, scoreCallsCache.tab, nil, 0)
+ cstab = computeCallSiteTable(fn, fn.Body, scoreCallsCache.tab, nil, 0,
+ nameFinder)
}
+ csa := makeCallSiteAnalyzer(fn)
const doCallResults = true
- scoreCallsRegion(fn, fn.Body, cstab, doCallResults, nil)
+ csa.scoreCallsRegion(fn, fn.Body, cstab, doCallResults, nil)
+
+ disableDebugTrace()
}
// scoreCallsRegion assigns numeric scores to each of the callsites in
// region 'region' within function 'fn'. This can be called on
// an entire function, or with 'region' set to a chunk of
// code corresponding to an inlined call.
-func scoreCallsRegion(fn *ir.Func, region ir.Nodes, cstab CallSiteTab, doCallResults bool, ic *ir.InlinedCallExpr) {
+func (csa *callSiteAnalyzer) scoreCallsRegion(fn *ir.Func, region ir.Nodes, cstab CallSiteTab, doCallResults bool, ic *ir.InlinedCallExpr) {
if debugTrace&debugTraceScoring != 0 {
- fmt.Fprintf(os.Stderr, "=-= scoreCallsRegion(%v, %s)\n",
- ir.FuncName(fn), region[0].Op().String())
+ fmt.Fprintf(os.Stderr, "=-= scoreCallsRegion(%v, %s) len(cstab)=%d\n",
+ ir.FuncName(fn), region[0].Op().String(), len(cstab))
}
// Sort callsites to avoid any surprises with non deterministic
@@ -510,13 +528,13 @@ func scoreCallsRegion(fn *ir.Func, region ir.Nodes, cstab CallSiteTab, doCallRes
continue
}
}
- cs.computeCallSiteScore(cprops)
+ cs.computeCallSiteScore(csa, cprops)
if doCallResults {
if debugTrace&debugTraceScoring != 0 {
fmt.Fprintf(os.Stderr, "=-= examineCallResults at %s: flags=%d score=%d funcInlHeur=%v deser=%v\n", fmtFullPos(cs.Call.Pos()), cs.Flags, cs.Score, fihcprops, desercprops)
}
- resultNameTab = examineCallResults(cs, resultNameTab)
+ resultNameTab = csa.examineCallResults(cs, resultNameTab)
}
if debugTrace&debugTraceScoring != 0 {
@@ -525,7 +543,7 @@ func scoreCallsRegion(fn *ir.Func, region ir.Nodes, cstab CallSiteTab, doCallRes
}
if resultNameTab != nil {
- rescoreBasedOnCallResultUses(fn, resultNameTab, cstab)
+ csa.rescoreBasedOnCallResultUses(fn, resultNameTab, cstab)
}
disableDebugTrace()
@@ -572,7 +590,7 @@ func GetCallSiteScore(fn *ir.Func, call *ir.CallExpr) (int, bool) {
// BudgetExpansion returns the amount to relax/expand the base
// inlining budget when the new inliner is turned on; the inliner
-// will add the returned value to the hairyness budget.
+// will add the returned value to the hairiness budget.
//
// Background: with the new inliner, the score for a given callsite
// can be adjusted down by some amount due to heuristics, however we
@@ -599,7 +617,7 @@ var allCallSites CallSiteTab
// along with info on call site scoring and the adjustments made to a
// given score. Here profile is the PGO profile in use (may be
// nil), budgetCallback is a callback that can be invoked to find out
-// the original pre-adjustment hairyness limit for the function, and
+// the original pre-adjustment hairiness limit for the function, and
// inlineHotMaxBudget is the constant of the same name used in the
// inliner. Sample output lines:
//
@@ -611,7 +629,7 @@ var allCallSites CallSiteTab
//
// In the dump above, "Score" is the final score calculated for the
// callsite, "Adjustment" is the amount added to or subtracted from
-// the original hairyness estimate to form the score. "Status" shows
+// the original hairiness estimate to form the score. "Status" shows
// whether anything changed with the site -- did the adjustment bump
// it down just below the threshold ("PROMOTED") or instead bump it
// above the threshold ("DEMOTED"); this will be blank ("---") if no
diff --git a/src/cmd/compile/internal/inline/interleaved/interleaved.go b/src/cmd/compile/internal/inline/interleaved/interleaved.go
new file mode 100644
index 0000000000..e55b0f1aee
--- /dev/null
+++ b/src/cmd/compile/internal/inline/interleaved/interleaved.go
@@ -0,0 +1,190 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+// Package interleaved implements the interleaved devirtualization and
+// inlining pass.
+package interleaved
+
+import (
+ "cmd/compile/internal/base"
+ "cmd/compile/internal/devirtualize"
+ "cmd/compile/internal/inline"
+ "cmd/compile/internal/inline/inlheur"
+ "cmd/compile/internal/ir"
+ "cmd/compile/internal/pgo"
+ "cmd/compile/internal/typecheck"
+ "fmt"
+)
+
+// DevirtualizeAndInlinePackage interleaves devirtualization and inlining on
+// all functions within pkg.
+func DevirtualizeAndInlinePackage(pkg *ir.Package, profile *pgo.Profile) {
+ if profile != nil && base.Debug.PGODevirtualize > 0 {
+ // TODO(mdempsky): Integrate into DevirtualizeAndInlineFunc below.
+ ir.VisitFuncsBottomUp(typecheck.Target.Funcs, func(list []*ir.Func, recursive bool) {
+ for _, fn := range list {
+ devirtualize.ProfileGuided(fn, profile)
+ }
+ })
+ ir.CurFunc = nil
+ }
+
+ if base.Flag.LowerL != 0 {
+ inlheur.SetupScoreAdjustments()
+ }
+
+ var inlProfile *pgo.Profile // copy of profile for inlining
+ if base.Debug.PGOInline != 0 {
+ inlProfile = profile
+ }
+
+ // First compute inlinability of all functions in the package.
+ inline.CanInlineFuncs(pkg.Funcs, inlProfile)
+
+ // Now we make a second pass to do devirtualization and inlining of
+ // calls. Order here should not matter.
+ for _, fn := range pkg.Funcs {
+ DevirtualizeAndInlineFunc(fn, inlProfile)
+ }
+
+ if base.Flag.LowerL != 0 {
+ // Perform a garbage collection of hidden closures functions that
+ // are no longer reachable from top-level functions following
+ // inlining. See #59404 and #59638 for more context.
+ inline.GarbageCollectUnreferencedHiddenClosures()
+
+ if base.Debug.DumpInlFuncProps != "" {
+ inlheur.DumpFuncProps(nil, base.Debug.DumpInlFuncProps)
+ }
+ if inlheur.Enabled() {
+ inline.PostProcessCallSites(inlProfile)
+ inlheur.TearDown()
+ }
+ }
+}
+
+// DevirtualizeAndInlineFunc interleaves devirtualization and inlining
+// on a single function.
+func DevirtualizeAndInlineFunc(fn *ir.Func, profile *pgo.Profile) {
+ ir.WithFunc(fn, func() {
+ if base.Flag.LowerL != 0 {
+ if inlheur.Enabled() && !fn.Wrapper() {
+ inlheur.ScoreCalls(fn)
+ defer inlheur.ScoreCallsCleanup()
+ }
+ if base.Debug.DumpInlFuncProps != "" && !fn.Wrapper() {
+ inlheur.DumpFuncProps(fn, base.Debug.DumpInlFuncProps)
+ }
+ }
+
+ bigCaller := base.Flag.LowerL != 0 && inline.IsBigFunc(fn)
+ if bigCaller && base.Flag.LowerM > 1 {
+ fmt.Printf("%v: function %v considered 'big'; reducing max cost of inlinees\n", ir.Line(fn), fn)
+ }
+
+ match := func(n ir.Node) bool {
+ switch n := n.(type) {
+ case *ir.CallExpr:
+ return true
+ case *ir.TailCallStmt:
+ n.Call.NoInline = true // can't inline yet
+ }
+ return false
+ }
+
+ edit := func(n ir.Node) ir.Node {
+ call, ok := n.(*ir.CallExpr)
+ if !ok { // previously inlined
+ return nil
+ }
+
+ devirtualize.StaticCall(call)
+ if inlCall := inline.TryInlineCall(fn, call, bigCaller, profile); inlCall != nil {
+ return inlCall
+ }
+ return nil
+ }
+
+ fixpoint(fn, match, edit)
+ })
+}
+
+// fixpoint repeatedly edits a function until it stabilizes.
+//
+// First, fixpoint applies match to every node n within fn. Then it
+// iteratively applies edit to each node satisfying match(n).
+//
+// If edit(n) returns nil, no change is made. Otherwise, the result
+// replaces n in fn's body, and fixpoint iterates at least once more.
+//
+// After an iteration where all edit calls return nil, fixpoint
+// returns.
+func fixpoint(fn *ir.Func, match func(ir.Node) bool, edit func(ir.Node) ir.Node) {
+ // Consider the expression "f(g())". We want to be able to replace
+ // "g()" in-place with its inlined representation. But if we first
+ // replace "f(...)" with its inlined representation, then "g()" will
+ // instead appear somewhere within this new AST.
+ //
+ // To mitigate this, each matched node n is wrapped in a ParenExpr,
+ // so we can reliably replace n in-place by assigning ParenExpr.X.
+ // It's safe to use ParenExpr here, because typecheck already
+ // removed them all.
+
+ var parens []*ir.ParenExpr
+ var mark func(ir.Node) ir.Node
+ mark = func(n ir.Node) ir.Node {
+ if _, ok := n.(*ir.ParenExpr); ok {
+ return n // already visited n.X before wrapping
+ }
+
+ ok := match(n)
+
+ ir.EditChildren(n, mark)
+
+ if ok {
+ paren := ir.NewParenExpr(n.Pos(), n)
+ paren.SetType(n.Type())
+ paren.SetTypecheck(n.Typecheck())
+
+ parens = append(parens, paren)
+ n = paren
+ }
+
+ return n
+ }
+ ir.EditChildren(fn, mark)
+
+ // Edit until stable.
+ for {
+ done := true
+
+ for i := 0; i < len(parens); i++ { // can't use "range parens" here
+ paren := parens[i]
+ if new := edit(paren.X); new != nil {
+ // Update AST and recursively mark nodes.
+ paren.X = new
+ ir.EditChildren(new, mark) // mark may append to parens
+ done = false
+ }
+ }
+
+ if done {
+ break
+ }
+ }
+
+ // Finally, remove any parens we inserted.
+ if len(parens) == 0 {
+ return // short circuit
+ }
+ var unparen func(ir.Node) ir.Node
+ unparen = func(n ir.Node) ir.Node {
+ if paren, ok := n.(*ir.ParenExpr); ok {
+ n = paren.X
+ }
+ ir.EditChildren(n, unparen)
+ return n
+ }
+ ir.EditChildren(fn, unparen)
+}
diff --git a/src/cmd/compile/internal/ir/check_reassign_no.go b/src/cmd/compile/internal/ir/check_reassign_no.go
new file mode 100644
index 0000000000..8290a7da7e
--- /dev/null
+++ b/src/cmd/compile/internal/ir/check_reassign_no.go
@@ -0,0 +1,9 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:build !checknewoldreassignment
+
+package ir
+
+const consistencyCheckEnabled = false
diff --git a/src/cmd/compile/internal/ir/check_reassign_yes.go b/src/cmd/compile/internal/ir/check_reassign_yes.go
new file mode 100644
index 0000000000..30876cca20
--- /dev/null
+++ b/src/cmd/compile/internal/ir/check_reassign_yes.go
@@ -0,0 +1,9 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+//go:build checknewoldreassignment
+
+package ir
+
+const consistencyCheckEnabled = true
diff --git a/src/cmd/compile/internal/ir/expr.go b/src/cmd/compile/internal/ir/expr.go
index ca2a2d5008..345828c163 100644
--- a/src/cmd/compile/internal/ir/expr.go
+++ b/src/cmd/compile/internal/ir/expr.go
@@ -190,7 +190,8 @@ type CallExpr struct {
RType Node `mknode:"-"` // see reflectdata/helpers.go
KeepAlive []*Name // vars to be kept alive until call returns
IsDDD bool
- NoInline bool
+ GoDefer bool // whether this call is part of a go or defer statement
+ NoInline bool // whether this call must not be inlined
}
func NewCallExpr(pos src.XPos, op Op, fun Node, args []Node) *CallExpr {
@@ -349,7 +350,7 @@ func NewKeyExpr(pos src.XPos, key, value Node) *KeyExpr {
return n
}
-// A StructKeyExpr is an Field: Value composite literal key.
+// A StructKeyExpr is a Field: Value composite literal key.
type StructKeyExpr struct {
miniExpr
Field *types.Field
@@ -855,13 +856,19 @@ func IsAddressable(n Node) bool {
// "g()" expression.
func StaticValue(n Node) Node {
for {
- if n.Op() == OCONVNOP {
- n = n.(*ConvExpr).X
- continue
- }
-
- if n.Op() == OINLCALL {
- n = n.(*InlinedCallExpr).SingleResult()
+ switch n1 := n.(type) {
+ case *ConvExpr:
+ if n1.Op() == OCONVNOP {
+ n = n1.X
+ continue
+ }
+ case *InlinedCallExpr:
+ if n1.Op() == OINLCALL {
+ n = n1.SingleResult()
+ continue
+ }
+ case *ParenExpr:
+ n = n1.X
continue
}
@@ -922,6 +929,8 @@ FindRHS:
// NB: global variables are always considered to be re-assigned.
// TODO: handle initial declaration not including an assignment and
// followed by a single assignment?
+// NOTE: any changes made here should also be made in the corresponding
+// code in the ReassignOracle.Init method.
func Reassigned(name *Name) bool {
if name.Op() != ONAME {
base.Fatalf("reassigned %v", name)
diff --git a/src/cmd/compile/internal/ir/func.go b/src/cmd/compile/internal/ir/func.go
index 303c5e4fd0..a74bb6ebda 100644
--- a/src/cmd/compile/internal/ir/func.go
+++ b/src/cmd/compile/internal/ir/func.go
@@ -539,7 +539,7 @@ func FuncPC(pos src.XPos, n Node, wantABI obj.ABI) Node {
if abi != wantABI {
base.ErrorfAt(pos, 0, "internal/abi.FuncPC%s expects an %v function, %s is defined as %v", wantABI, wantABI, name.Sym().Name, abi)
}
- var e Node = NewLinksymExpr(pos, name.Sym().LinksymABI(abi), types.Types[types.TUINTPTR])
+ var e Node = NewLinksymExpr(pos, name.LinksymABI(abi), types.Types[types.TUINTPTR])
e = NewAddrExpr(pos, e)
e.SetType(types.Types[types.TUINTPTR].PtrTo())
e = NewConvExpr(pos, OCONVNOP, types.Types[types.TUINTPTR], e)
diff --git a/src/cmd/compile/internal/ir/node.go b/src/cmd/compile/internal/ir/node.go
index 6513386f03..21d181dba6 100644
--- a/src/cmd/compile/internal/ir/node.go
+++ b/src/cmd/compile/internal/ir/node.go
@@ -152,7 +152,7 @@ const (
// OCALLFUNC, OCALLMETH, and OCALLINTER have the same structure.
// Prior to walk, they are: X(Args), where Args is all regular arguments.
// After walk, if any argument whose evaluation might requires temporary variable,
- // that temporary variable will be pushed to Init, Args will contains an updated
+ // that temporary variable will be pushed to Init, Args will contain an updated
// set of arguments.
OCALLFUNC // X(Args) (function call f(args))
OCALLMETH // X(Args) (direct method call x.Method(args))
diff --git a/src/cmd/compile/internal/ir/reassign_consistency_check.go b/src/cmd/compile/internal/ir/reassign_consistency_check.go
new file mode 100644
index 0000000000..06a6c88962
--- /dev/null
+++ b/src/cmd/compile/internal/ir/reassign_consistency_check.go
@@ -0,0 +1,46 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package ir
+
+import (
+ "cmd/compile/internal/base"
+ "cmd/internal/src"
+ "fmt"
+ "path/filepath"
+ "strings"
+)
+
+// checkStaticValueResult compares the result from ReassignOracle.StaticValue
+// with the corresponding result from ir.StaticValue to make sure they agree.
+// This method is called only when turned on via build tag.
+func checkStaticValueResult(n Node, newres Node) {
+ oldres := StaticValue(n)
+ if oldres != newres {
+ base.Fatalf("%s: new/old static value disagreement on %v:\nnew=%v\nold=%v", fmtFullPos(n.Pos()), n, newres, oldres)
+ }
+}
+
+// checkReassignedResult compares the result from ReassignOracle.Reassigned
+// with the corresponding result from ir.Reassigned to make sure they agree.
+// This method is called only when turned on via build tag.
+func checkReassignedResult(n *Name, newres bool) {
+ origres := Reassigned(n)
+ if newres != origres {
+ base.Fatalf("%s: new/old reassigned disagreement on %v (class %s) newres=%v oldres=%v", fmtFullPos(n.Pos()), n, n.Class.String(), newres, origres)
+ }
+}
+
+// fmtFullPos returns a verbose dump for pos p, including inlines.
+func fmtFullPos(p src.XPos) string {
+ var sb strings.Builder
+ sep := ""
+ base.Ctxt.AllPos(p, func(pos src.Pos) {
+ fmt.Fprintf(&sb, sep)
+ sep = "|"
+ file := filepath.Base(pos.Filename())
+ fmt.Fprintf(&sb, "%s:%d:%d", file, pos.Line(), pos.Col())
+ })
+ return sb.String()
+}
diff --git a/src/cmd/compile/internal/ir/reassignment.go b/src/cmd/compile/internal/ir/reassignment.go
new file mode 100644
index 0000000000..9974292471
--- /dev/null
+++ b/src/cmd/compile/internal/ir/reassignment.go
@@ -0,0 +1,205 @@
+// Copyright 2023 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package ir
+
+import (
+ "cmd/compile/internal/base"
+)
+
+// A ReassignOracle efficiently answers queries about whether local
+// variables are reassigned. This helper works by looking for function
+// params and short variable declarations (e.g.
+// https://go.dev/ref/spec#Short_variable_declarations) that are
+// neither address taken nor subsequently re-assigned. It is intended
+// to operate much like "ir.StaticValue" and "ir.Reassigned", but in a
+// way that does just a single walk of the containing function (as
+// opposed to a new walk on every call).
+type ReassignOracle struct {
+ fn *Func
+ // maps candidate name to its defining assignment (or for
+ // for params, defining func).
+ singleDef map[*Name]Node
+}
+
+// Init initializes the oracle based on the IR in function fn, laying
+// the groundwork for future calls to the StaticValue and Reassigned
+// methods. If the fn's IR is subsequently modified, Init must be
+// called again.
+func (ro *ReassignOracle) Init(fn *Func) {
+ ro.fn = fn
+
+ // Collect candidate map. Start by adding function parameters
+ // explicitly.
+ ro.singleDef = make(map[*Name]Node)
+ sig := fn.Type()
+ numParams := sig.NumRecvs() + sig.NumParams()
+ for _, param := range fn.Dcl[:numParams] {
+ if IsBlank(param) {
+ continue
+ }
+ // For params, use func itself as defining node.
+ ro.singleDef[param] = fn
+ }
+
+ // Walk the function body to discover any locals assigned
+ // via ":=" syntax (e.g. "a := ").
+ var findLocals func(n Node) bool
+ findLocals = func(n Node) bool {
+ if nn, ok := n.(*Name); ok {
+ if nn.Defn != nil && !nn.Addrtaken() && nn.Class == PAUTO {
+ ro.singleDef[nn] = nn.Defn
+ }
+ } else if nn, ok := n.(*ClosureExpr); ok {
+ Any(nn.Func, findLocals)
+ }
+ return false
+ }
+ Any(fn, findLocals)
+
+ outerName := func(x Node) *Name {
+ if x == nil {
+ return nil
+ }
+ n, ok := OuterValue(x).(*Name)
+ if ok {
+ return n.Canonical()
+ }
+ return nil
+ }
+
+ // pruneIfNeeded examines node nn appearing on the left hand side
+ // of assignment statement asn to see if it contains a reassignment
+ // to any nodes in our candidate map ro.singleDef; if a reassignment
+ // is found, the corresponding name is deleted from singleDef.
+ pruneIfNeeded := func(nn Node, asn Node) {
+ oname := outerName(nn)
+ if oname == nil {
+ return
+ }
+ defn, ok := ro.singleDef[oname]
+ if !ok {
+ return
+ }
+ // any assignment to a param invalidates the entry.
+ paramAssigned := oname.Class == PPARAM
+ // assignment to local ok iff assignment is its orig def.
+ localAssigned := (oname.Class == PAUTO && asn != defn)
+ if paramAssigned || localAssigned {
+ // We found an assignment to name N that doesn't
+ // correspond to its original definition; remove
+ // from candidates.
+ delete(ro.singleDef, oname)
+ }
+ }
+
+ // Prune away anything that looks assigned. This code modeled after
+ // similar code in ir.Reassigned; any changes there should be made
+ // here as well.
+ var do func(n Node) bool
+ do = func(n Node) bool {
+ switch n.Op() {
+ case OAS:
+ asn := n.(*AssignStmt)
+ pruneIfNeeded(asn.X, n)
+ case OAS2, OAS2FUNC, OAS2MAPR, OAS2DOTTYPE, OAS2RECV, OSELRECV2:
+ asn := n.(*AssignListStmt)
+ for _, p := range asn.Lhs {
+ pruneIfNeeded(p, n)
+ }
+ case OASOP:
+ asn := n.(*AssignOpStmt)
+ pruneIfNeeded(asn.X, n)
+ case ORANGE:
+ rs := n.(*RangeStmt)
+ pruneIfNeeded(rs.Key, n)
+ pruneIfNeeded(rs.Value, n)
+ case OCLOSURE:
+ n := n.(*ClosureExpr)
+ Any(n.Func, do)
+ }
+ return false
+ }
+ Any(fn, do)
+}
+
+// StaticValue method has the same semantics as the ir package function
+// of the same name; see comments on [StaticValue].
+func (ro *ReassignOracle) StaticValue(n Node) Node {
+ arg := n
+ for {
+ if n.Op() == OCONVNOP {
+ n = n.(*ConvExpr).X
+ continue
+ }
+
+ if n.Op() == OINLCALL {
+ n = n.(*InlinedCallExpr).SingleResult()
+ continue
+ }
+
+ n1 := ro.staticValue1(n)
+ if n1 == nil {
+ if consistencyCheckEnabled {
+ checkStaticValueResult(arg, n)
+ }
+ return n
+ }
+ n = n1
+ }
+}
+
+func (ro *ReassignOracle) staticValue1(nn Node) Node {
+ if nn.Op() != ONAME {
+ return nil
+ }
+ n := nn.(*Name).Canonical()
+ if n.Class != PAUTO {
+ return nil
+ }
+
+ defn := n.Defn
+ if defn == nil {
+ return nil
+ }
+
+ var rhs Node
+FindRHS:
+ switch defn.Op() {
+ case OAS:
+ defn := defn.(*AssignStmt)
+ rhs = defn.Y
+ case OAS2:
+ defn := defn.(*AssignListStmt)
+ for i, lhs := range defn.Lhs {
+ if lhs == n {
+ rhs = defn.Rhs[i]
+ break FindRHS
+ }
+ }
+ base.Fatalf("%v missing from LHS of %v", n, defn)
+ default:
+ return nil
+ }
+ if rhs == nil {
+ base.Fatalf("RHS is nil: %v", defn)
+ }
+
+ if _, ok := ro.singleDef[n]; !ok {
+ return nil
+ }
+
+ return rhs
+}
+
+// Reassigned method has the same semantics as the ir package function
+// of the same name; see comments on [Reassigned] for more info.
+func (ro *ReassignOracle) Reassigned(n *Name) bool {
+ _, ok := ro.singleDef[n]
+ result := !ok
+ if consistencyCheckEnabled {
+ checkReassignedResult(n, result)
+ }
+ return result
+}
diff --git a/src/cmd/compile/internal/loong64/galign.go b/src/cmd/compile/internal/loong64/galign.go
index 99ab7bdfb5..a613165054 100644
--- a/src/cmd/compile/internal/loong64/galign.go
+++ b/src/cmd/compile/internal/loong64/galign.go
@@ -20,4 +20,6 @@ func Init(arch *ssagen.ArchInfo) {
arch.SSAMarkMoves = func(s *ssagen.State, b *ssa.Block) {}
arch.SSAGenValue = ssaGenValue
arch.SSAGenBlock = ssaGenBlock
+ arch.LoadRegResult = loadRegResult
+ arch.SpillArgReg = spillArgReg
}
diff --git a/src/cmd/compile/internal/loong64/ssa.go b/src/cmd/compile/internal/loong64/ssa.go
index 6e81da3ef8..e7298bdb9f 100644
--- a/src/cmd/compile/internal/loong64/ssa.go
+++ b/src/cmd/compile/internal/loong64/ssa.go
@@ -10,6 +10,7 @@ import (
"cmd/compile/internal/base"
"cmd/compile/internal/ir"
"cmd/compile/internal/logopt"
+ "cmd/compile/internal/objw"
"cmd/compile/internal/ssa"
"cmd/compile/internal/ssagen"
"cmd/compile/internal/types"
@@ -144,6 +145,18 @@ func ssaGenValue(s *ssagen.State, v *ssa.Value) {
p.From.Type = obj.TYPE_REG
p.From.Reg = r
ssagen.AddrAuto(&p.To, v)
+ case ssa.OpArgIntReg, ssa.OpArgFloatReg:
+ // The assembler needs to wrap the entry safepoint/stack growth code with spill/unspill
+ // The loop only runs once.
+ for _, a := range v.Block.Func.RegArgs {
+ // Pass the spill/unspill information along to the assembler, offset by size of
+ // the saved LR slot.
+ addr := ssagen.SpillSlotAddr(a, loong64.REGSP, base.Ctxt.Arch.FixedFrameSize)
+ s.FuncInfo().AddSpill(
+ obj.RegSpill{Reg: a.Reg, Addr: addr, Unspill: loadByType(a.Type, a.Reg), Spill: storeByType(a.Type, a.Reg)})
+ }
+ v.Block.Func.RegArgs = nil
+ ssagen.CheckArgReg(v)
case ssa.OpLOONG64ADDV,
ssa.OpLOONG64SUBV,
ssa.OpLOONG64AND,
@@ -362,13 +375,12 @@ func ssaGenValue(s *ssagen.State, v *ssa.Value) {
p.To.Type = obj.TYPE_REG
p.To.Reg = v.Reg()
case ssa.OpLOONG64DUFFZERO:
- // runtime.duffzero expects start address in R19
+ // runtime.duffzero expects start address in R20
p := s.Prog(obj.ADUFFZERO)
p.To.Type = obj.TYPE_MEM
p.To.Name = obj.NAME_EXTERN
p.To.Sym = ir.Syms.Duffzero
p.To.Offset = v.AuxInt
-
case ssa.OpLOONG64LoweredZero:
// MOVx R0, (Rarg0)
// ADDV $sz, Rarg0
@@ -797,3 +809,22 @@ func ssaGenBlock(s *ssagen.State, b, next *ssa.Block) {
b.Fatalf("branch not implemented: %s", b.LongString())
}
}
+
+func loadRegResult(s *ssagen.State, f *ssa.Func, t *types.Type, reg int16, n *ir.Name, off int64) *obj.Prog {
+ p := s.Prog(loadByType(t, reg))
+ p.From.Type = obj.TYPE_MEM
+ p.From.Name = obj.NAME_AUTO
+ p.From.Sym = n.Linksym()
+ p.From.Offset = n.FrameOffset() + off
+ p.To.Type = obj.TYPE_REG
+ p.To.Reg = reg
+ return p
+}
+
+func spillArgReg(pp *objw.Progs, p *obj.Prog, f *ssa.Func, t *types.Type, reg int16, n *ir.Name, off int64) *obj.Prog {
+ p = pp.Append(p, storeByType(t, reg), obj.TYPE_REG, reg, 0, obj.TYPE_MEM, 0, n.FrameOffset()+off)
+ p.To.Name = obj.NAME_PARAM
+ p.To.Sym = n.Linksym()
+ p.Pos = p.Pos.WithNotStmt()
+ return p
+}
diff --git a/src/cmd/compile/internal/loopvar/loopvar_test.go b/src/cmd/compile/internal/loopvar/loopvar_test.go
index c8e11dbd07..64cfdb77d9 100644
--- a/src/cmd/compile/internal/loopvar/loopvar_test.go
+++ b/src/cmd/compile/internal/loopvar/loopvar_test.go
@@ -251,7 +251,7 @@ func TestLoopVarVersionEnableFlag(t *testing.T) {
t.Logf(m)
- yCount := strings.Count(m, "opt.go:16:6: loop variable private now per-iteration, heap-allocated (loop inlined into ./opt.go:30)")
+ yCount := strings.Count(m, "opt.go:16:6: loop variable private now per-iteration, heap-allocated (loop inlined into ./opt.go:29)")
nCount := strings.Count(m, "shared")
if yCount != 1 {
@@ -290,7 +290,7 @@ func TestLoopVarVersionEnableGoBuild(t *testing.T) {
t.Logf(m)
- yCount := strings.Count(m, "opt-122.go:18:6: loop variable private now per-iteration, heap-allocated (loop inlined into ./opt-122.go:32)")
+ yCount := strings.Count(m, "opt-122.go:18:6: loop variable private now per-iteration, heap-allocated (loop inlined into ./opt-122.go:31)")
nCount := strings.Count(m, "shared")
if yCount != 1 {
@@ -329,7 +329,7 @@ func TestLoopVarVersionDisableFlag(t *testing.T) {
t.Logf(m) // expect error
- yCount := strings.Count(m, "opt.go:16:6: loop variable private now per-iteration, heap-allocated (loop inlined into ./opt.go:30)")
+ yCount := strings.Count(m, "opt.go:16:6: loop variable private now per-iteration, heap-allocated (loop inlined into ./opt.go:29)")
nCount := strings.Count(m, "shared")
if yCount != 0 {
@@ -368,7 +368,7 @@ func TestLoopVarVersionDisableGoBuild(t *testing.T) {
t.Logf(m) // expect error
- yCount := strings.Count(m, "opt-121.go:18:6: loop variable private now per-iteration, heap-allocated (loop inlined into ./opt-121.go:32)")
+ yCount := strings.Count(m, "opt-121.go:18:6: loop variable private now per-iteration, heap-allocated (loop inlined into ./opt-121.go:31)")
nCount := strings.Count(m, "shared")
if yCount != 0 {
diff --git a/src/cmd/compile/internal/loopvar/testdata/opt-121.go b/src/cmd/compile/internal/loopvar/testdata/opt-121.go
index 131033b13c..4afb658fc8 100644
--- a/src/cmd/compile/internal/loopvar/testdata/opt-121.go
+++ b/src/cmd/compile/internal/loopvar/testdata/opt-121.go
@@ -19,7 +19,6 @@ func inline(j, k int) []*int {
a = append(a, &private)
}
return a
-
}
//go:noinline
diff --git a/src/cmd/compile/internal/loopvar/testdata/opt-122.go b/src/cmd/compile/internal/loopvar/testdata/opt-122.go
index 0ed6feee04..9dceab9175 100644
--- a/src/cmd/compile/internal/loopvar/testdata/opt-122.go
+++ b/src/cmd/compile/internal/loopvar/testdata/opt-122.go
@@ -19,7 +19,6 @@ func inline(j, k int) []*int {
a = append(a, &private)
}
return a
-
}
//go:noinline
diff --git a/src/cmd/compile/internal/loopvar/testdata/opt.go b/src/cmd/compile/internal/loopvar/testdata/opt.go
index 1bcd73614d..82c8616bcd 100644
--- a/src/cmd/compile/internal/loopvar/testdata/opt.go
+++ b/src/cmd/compile/internal/loopvar/testdata/opt.go
@@ -17,7 +17,6 @@ func inline(j, k int) []*int {
a = append(a, &private)
}
return a
-
}
//go:noinline
diff --git a/src/cmd/compile/internal/noder/helpers.go b/src/cmd/compile/internal/noder/helpers.go
index 1f7b497599..0bff71e658 100644
--- a/src/cmd/compile/internal/noder/helpers.go
+++ b/src/cmd/compile/internal/noder/helpers.go
@@ -80,7 +80,7 @@ func idealType(tv syntax.TypeAndValue) types2.Type {
// types2 mostly satisfies this expectation already. But there are a few
// cases where the Go spec doesn't require converting to concrete type,
// and so types2 leaves them untyped. So we need to fix those up here.
- typ := tv.Type
+ typ := types2.Unalias(tv.Type)
if basic, ok := typ.(*types2.Basic); ok && basic.Info()&types2.IsUntyped != 0 {
switch basic.Kind() {
case types2.UntypedNil:
@@ -99,6 +99,8 @@ func idealType(tv syntax.TypeAndValue) types2.Type {
typ = types2.Typ[types2.Bool] // expression in "if" or "for" condition
case types2.UntypedString:
typ = types2.Typ[types2.String] // argument to "append" or "copy" calls
+ case types2.UntypedRune:
+ typ = types2.Typ[types2.Int32] // range over rune
default:
return nil
}
@@ -107,13 +109,14 @@ func idealType(tv syntax.TypeAndValue) types2.Type {
}
func isTypeParam(t types2.Type) bool {
- _, ok := t.(*types2.TypeParam)
+ _, ok := types2.Unalias(t).(*types2.TypeParam)
return ok
}
// isNotInHeap reports whether typ is or contains an element of type
// runtime/internal/sys.NotInHeap.
func isNotInHeap(typ types2.Type) bool {
+ typ = types2.Unalias(typ)
if named, ok := typ.(*types2.Named); ok {
if obj := named.Obj(); obj.Name() == "nih" && obj.Pkg().Path() == "runtime/internal/sys" {
return true
diff --git a/src/cmd/compile/internal/noder/irgen.go b/src/cmd/compile/internal/noder/irgen.go
index 46511d1f97..e0b7bb946d 100644
--- a/src/cmd/compile/internal/noder/irgen.go
+++ b/src/cmd/compile/internal/noder/irgen.go
@@ -92,23 +92,22 @@ func checkFiles(m posMap, noders []*noder) (*types2.Package, *types2.Info) {
}
// Check for anonymous interface cycles (#56103).
- if base.Debug.InterfaceCycles == 0 {
- var f cycleFinder
- for _, file := range files {
- syntax.Inspect(file, func(n syntax.Node) bool {
- if n, ok := n.(*syntax.InterfaceType); ok {
- if f.hasCycle(n.GetTypeInfo().Type.(*types2.Interface)) {
- base.ErrorfAt(m.makeXPos(n.Pos()), errors.InvalidTypeCycle, "invalid recursive type: anonymous interface refers to itself (see https://go.dev/issue/56103)")
+ // TODO(gri) move this code into the type checkers (types2 and go/types)
+ var f cycleFinder
+ for _, file := range files {
+ syntax.Inspect(file, func(n syntax.Node) bool {
+ if n, ok := n.(*syntax.InterfaceType); ok {
+ if f.hasCycle(types2.Unalias(n.GetTypeInfo().Type).(*types2.Interface)) {
+ base.ErrorfAt(m.makeXPos(n.Pos()), errors.InvalidTypeCycle, "invalid recursive type: anonymous interface refers to itself (see https://go.dev/issue/56103)")
- for typ := range f.cyclic {
- f.cyclic[typ] = false // suppress duplicate errors
- }
+ for typ := range f.cyclic {
+ f.cyclic[typ] = false // suppress duplicate errors
}
- return false
}
- return true
- })
- }
+ return false
+ }
+ return true
+ })
}
base.ExitIfErrors()
@@ -172,7 +171,7 @@ func (f *cycleFinder) hasCycle(typ *types2.Interface) bool {
// visit recursively walks typ0 to check any referenced interface types.
func (f *cycleFinder) visit(typ0 types2.Type) bool {
for { // loop for tail recursion
- switch typ := typ0.(type) {
+ switch typ := types2.Unalias(typ0).(type) {
default:
base.Fatalf("unexpected type: %T", typ)
diff --git a/src/cmd/compile/internal/noder/reader.go b/src/cmd/compile/internal/noder/reader.go
index c1145f980e..25d6fb53e3 100644
--- a/src/cmd/compile/internal/noder/reader.go
+++ b/src/cmd/compile/internal/noder/reader.go
@@ -5,6 +5,7 @@
package noder
import (
+ "encoding/hex"
"fmt"
"go/constant"
"internal/buildcfg"
@@ -15,12 +16,14 @@ import (
"cmd/compile/internal/base"
"cmd/compile/internal/dwarfgen"
"cmd/compile/internal/inline"
+ "cmd/compile/internal/inline/interleaved"
"cmd/compile/internal/ir"
"cmd/compile/internal/objw"
"cmd/compile/internal/reflectdata"
"cmd/compile/internal/staticinit"
"cmd/compile/internal/typecheck"
"cmd/compile/internal/types"
+ "cmd/internal/notsha256"
"cmd/internal/obj"
"cmd/internal/objabi"
"cmd/internal/src"
@@ -660,9 +663,24 @@ func (pr *pkgReader) objInstIdx(info objInfo, dict *readerDict, shaped bool) ir.
}
// objIdx returns the specified object, instantiated with the given
-// type arguments, if any. If shaped is true, then the shaped variant
-// of the object is returned instead.
+// type arguments, if any.
+// If shaped is true, then the shaped variant of the object is returned
+// instead.
func (pr *pkgReader) objIdx(idx pkgbits.Index, implicits, explicits []*types.Type, shaped bool) ir.Node {
+ n, err := pr.objIdxMayFail(idx, implicits, explicits, shaped)
+ if err != nil {
+ base.Fatalf("%v", err)
+ }
+ return n
+}
+
+// objIdxMayFail is equivalent to objIdx, but returns an error rather than
+// failing the build if this object requires type arguments and the incorrect
+// number of type arguments were passed.
+//
+// Other sources of internal failure (such as duplicate definitions) still fail
+// the build.
+func (pr *pkgReader) objIdxMayFail(idx pkgbits.Index, implicits, explicits []*types.Type, shaped bool) (ir.Node, error) {
rname := pr.newReader(pkgbits.RelocName, idx, pkgbits.SyncObject1)
_, sym := rname.qualifiedIdent()
tag := pkgbits.CodeObj(rname.Code(pkgbits.SyncCodeObj))
@@ -671,22 +689,25 @@ func (pr *pkgReader) objIdx(idx pkgbits.Index, implicits, explicits []*types.Typ
assert(!sym.IsBlank())
switch sym.Pkg {
case types.BuiltinPkg, types.UnsafePkg:
- return sym.Def.(ir.Node)
+ return sym.Def.(ir.Node), nil
}
if pri, ok := objReader[sym]; ok {
- return pri.pr.objIdx(pri.idx, nil, explicits, shaped)
+ return pri.pr.objIdxMayFail(pri.idx, nil, explicits, shaped)
}
if sym.Pkg.Path == "runtime" {
- return typecheck.LookupRuntime(sym.Name)
+ return typecheck.LookupRuntime(sym.Name), nil
}
base.Fatalf("unresolved stub: %v", sym)
}
- dict := pr.objDictIdx(sym, idx, implicits, explicits, shaped)
+ dict, err := pr.objDictIdx(sym, idx, implicits, explicits, shaped)
+ if err != nil {
+ return nil, err
+ }
sym = dict.baseSym
if !sym.IsBlank() && sym.Def != nil {
- return sym.Def.(*ir.Name)
+ return sym.Def.(*ir.Name), nil
}
r := pr.newReader(pkgbits.RelocObj, idx, pkgbits.SyncObject1)
@@ -722,7 +743,7 @@ func (pr *pkgReader) objIdx(idx pkgbits.Index, implicits, explicits []*types.Typ
name := do(ir.OTYPE, false)
setType(name, r.typ())
name.SetAlias(true)
- return name
+ return name, nil
case pkgbits.ObjConst:
name := do(ir.OLITERAL, false)
@@ -730,7 +751,7 @@ func (pr *pkgReader) objIdx(idx pkgbits.Index, implicits, explicits []*types.Typ
val := FixValue(typ, r.Value())
setType(name, typ)
setValue(name, val)
- return name
+ return name, nil
case pkgbits.ObjFunc:
if sym.Name == "init" {
@@ -765,7 +786,7 @@ func (pr *pkgReader) objIdx(idx pkgbits.Index, implicits, explicits []*types.Typ
}
rext.funcExt(name, nil)
- return name
+ return name, nil
case pkgbits.ObjType:
name := do(ir.OTYPE, true)
@@ -802,13 +823,13 @@ func (pr *pkgReader) objIdx(idx pkgbits.Index, implicits, explicits []*types.Typ
r.needWrapper(typ)
}
- return name
+ return name, nil
case pkgbits.ObjVar:
name := do(ir.ONAME, false)
setType(name, r.typ())
rext.varExt(name)
- return name
+ return name, nil
}
}
@@ -882,7 +903,16 @@ func shapify(targ *types.Type, basic bool) *types.Type {
under = types.NewPtr(types.Types[types.TUINT8])
}
- sym := types.ShapePkg.Lookup(under.LinkString())
+ // Hash long type names to bound symbol name length seen by users,
+ // particularly for large protobuf structs (#65030).
+ uls := under.LinkString()
+ if base.Debug.MaxShapeLen != 0 &&
+ len(uls) > base.Debug.MaxShapeLen {
+ h := notsha256.Sum256([]byte(uls))
+ uls = hex.EncodeToString(h[:])
+ }
+
+ sym := types.ShapePkg.Lookup(uls)
if sym.Def == nil {
name := ir.NewDeclNameAt(under.Pos(), ir.OTYPE, sym)
typ := types.NewNamed(name)
@@ -896,7 +926,7 @@ func shapify(targ *types.Type, basic bool) *types.Type {
}
// objDictIdx reads and returns the specified object dictionary.
-func (pr *pkgReader) objDictIdx(sym *types.Sym, idx pkgbits.Index, implicits, explicits []*types.Type, shaped bool) *readerDict {
+func (pr *pkgReader) objDictIdx(sym *types.Sym, idx pkgbits.Index, implicits, explicits []*types.Type, shaped bool) (*readerDict, error) {
r := pr.newReader(pkgbits.RelocObjDict, idx, pkgbits.SyncObject1)
dict := readerDict{
@@ -907,7 +937,7 @@ func (pr *pkgReader) objDictIdx(sym *types.Sym, idx pkgbits.Index, implicits, ex
nexplicits := r.Len()
if nimplicits > len(implicits) || nexplicits != len(explicits) {
- base.Fatalf("%v has %v+%v params, but instantiated with %v+%v args", sym, nimplicits, nexplicits, len(implicits), len(explicits))
+ return nil, fmt.Errorf("%v has %v+%v params, but instantiated with %v+%v args", sym, nimplicits, nexplicits, len(implicits), len(explicits))
}
dict.targs = append(implicits[:nimplicits:nimplicits], explicits...)
@@ -972,7 +1002,7 @@ func (pr *pkgReader) objDictIdx(sym *types.Sym, idx pkgbits.Index, implicits, ex
dict.itabs[i] = itabInfo{typ: r.typInfo(), iface: r.typInfo()}
}
- return &dict
+ return &dict, nil
}
func (r *reader) typeParamNames() {
@@ -2517,7 +2547,10 @@ func (pr *pkgReader) objDictName(idx pkgbits.Index, implicits, explicits []*type
base.Fatalf("unresolved stub: %v", sym)
}
- dict := pr.objDictIdx(sym, idx, implicits, explicits, false)
+ dict, err := pr.objDictIdx(sym, idx, implicits, explicits, false)
+ if err != nil {
+ base.Fatalf("%v", err)
+ }
return pr.dictNameOf(dict)
}
@@ -3587,6 +3620,57 @@ func usedLocals(body []ir.Node) ir.NameSet {
}
// @@@ Method wrappers
+//
+// Here we handle constructing "method wrappers," alternative entry
+// points that adapt methods to different calling conventions. Given a
+// user-declared method "func (T) M(i int) bool { ... }", there are a
+// few wrappers we may need to construct:
+//
+// - Implicit dereferencing. Methods declared with a value receiver T
+// are also included in the method set of the pointer type *T, so
+// we need to construct a wrapper like "func (recv *T) M(i int)
+// bool { return (*recv).M(i) }".
+//
+// - Promoted methods. If struct type U contains an embedded field of
+// type T or *T, we need to construct a wrapper like "func (recv U)
+// M(i int) bool { return recv.T.M(i) }".
+//
+// - Method values. If x is an expression of type T, then "x.M" is
+// roughly "tmp := x; func(i int) bool { return tmp.M(i) }".
+//
+// At call sites, we always prefer to call the user-declared method
+// directly, if known, so wrappers are only needed for indirect calls
+// (for example, interface method calls that can't be devirtualized).
+// Consequently, we can save some compile time by skipping
+// construction of wrappers that are never needed.
+//
+// Alternatively, because the linker doesn't care which compilation
+// unit constructed a particular wrapper, we can instead construct
+// them as needed. However, if a wrapper is needed in multiple
+// downstream packages, we may end up needing to compile it multiple
+// times, costing us more compile time and object file size. (We mark
+// the wrappers as DUPOK, so the linker doesn't complain about the
+// duplicate symbols.)
+//
+// The current heuristics we use to balance these trade offs are:
+//
+// - For a (non-parameterized) defined type T, we construct wrappers
+// for *T and any promoted methods on T (and *T) in the same
+// compilation unit as the type declaration.
+//
+// - For a parameterized defined type, we construct wrappers in the
+// compilation units in which the type is instantiated. We
+// similarly handle wrappers for anonymous types with methods and
+// compilation units where their type literals appear in source.
+//
+// - Method value expressions are relatively uncommon, so we
+// construct their wrappers in the compilation units that they
+// appear in.
+//
+// Finally, as an opportunistic compile-time optimization, if we know
+// a wrapper was constructed in any imported package's compilation
+// unit, then we skip constructing a duplicate one. However, currently
+// this is only done on a best-effort basis.
// needWrapperTypes lists types for which we may need to generate
// method wrappers.
@@ -3610,6 +3694,8 @@ type methodValueWrapper struct {
method *types.Field
}
+// needWrapper records that wrapper methods may be needed at link
+// time.
func (r *reader) needWrapper(typ *types.Type) {
if typ.IsPtr() {
return
@@ -3643,6 +3729,8 @@ func (r *reader) importedDef() bool {
return r.p != localPkgReader && !r.hasTypeParams()
}
+// MakeWrappers constructs all wrapper methods needed for the target
+// compilation unit.
func MakeWrappers(target *ir.Package) {
// always generate a wrapper for error.Error (#29304)
needWrapperTypes = append(needWrapperTypes, types.ErrorType)
@@ -3778,7 +3866,6 @@ func wrapMethodValue(recvType *types.Type, method *types.Field, target *ir.Packa
func newWrapperFunc(pos src.XPos, sym *types.Sym, wrapper *types.Type, method *types.Field) *ir.Func {
sig := newWrapperType(wrapper, method)
-
fn := ir.NewFunc(pos, pos, sym, sig)
fn.DeclareParams(true)
fn.SetDupok(true) // TODO(mdempsky): Leave unset for local, non-generic wrappers?
@@ -3794,7 +3881,7 @@ func finishWrapperFunc(fn *ir.Func, target *ir.Package) {
// We generate wrappers after the global inlining pass,
// so we're responsible for applying inlining ourselves here.
// TODO(prattmic): plumb PGO.
- inline.InlineCalls(fn, nil)
+ interleaved.DevirtualizeAndInlineFunc(fn, nil)
// The body of wrapper function after inlining may reveal new ir.OMETHVALUE node,
// we don't know whether wrapper function has been generated for it or not, so
diff --git a/src/cmd/compile/internal/noder/unified.go b/src/cmd/compile/internal/noder/unified.go
index a803e53502..492b00d256 100644
--- a/src/cmd/compile/internal/noder/unified.go
+++ b/src/cmd/compile/internal/noder/unified.go
@@ -80,7 +80,11 @@ func lookupFunction(pkg *types.Pkg, symName string) (*ir.Func, error) {
return nil, fmt.Errorf("func sym %v missing objReader", sym)
}
- name := pri.pr.objIdx(pri.idx, nil, nil, false).(*ir.Name)
+ node, err := pri.pr.objIdxMayFail(pri.idx, nil, nil, false)
+ if err != nil {
+ return nil, fmt.Errorf("func sym %v lookup error: %w", sym, err)
+ }
+ name := node.(*ir.Name)
if name.Op() != ir.ONAME || name.Class != ir.PFUNC {
return nil, fmt.Errorf("func sym %v refers to non-function name: %v", sym, name)
}
@@ -105,7 +109,11 @@ func lookupMethod(pkg *types.Pkg, symName string) (*ir.Func, error) {
return nil, fmt.Errorf("type sym %v missing objReader", typ)
}
- name := pri.pr.objIdx(pri.idx, nil, nil, false).(*ir.Name)
+ node, err := pri.pr.objIdxMayFail(pri.idx, nil, nil, false)
+ if err != nil {
+ return nil, fmt.Errorf("func sym %v lookup error: %w", typ, err)
+ }
+ name := node.(*ir.Name)
if name.Op() != ir.OTYPE {
return nil, fmt.Errorf("type sym %v refers to non-type name: %v", typ, name)
}
@@ -280,7 +288,7 @@ func readBodies(target *ir.Package, duringInlining bool) {
oldLowerM := base.Flag.LowerM
base.Flag.LowerM = 0
- inline.InlineDecls(nil, inlDecls, false)
+ inline.CanInlineFuncs(inlDecls, nil)
base.Flag.LowerM = oldLowerM
for _, fn := range inlDecls {
diff --git a/src/cmd/compile/internal/noder/writer.go b/src/cmd/compile/internal/noder/writer.go
index 46d5213694..c57ccdf36d 100644
--- a/src/cmd/compile/internal/noder/writer.go
+++ b/src/cmd/compile/internal/noder/writer.go
@@ -189,7 +189,9 @@ type writer struct {
// A writerDict tracks types and objects that are used by a declaration.
type writerDict struct {
- implicits []*types2.TypeName
+ // implicits is a slice of type parameters from the enclosing
+ // declarations.
+ implicits []*types2.TypeParam
// derived is a slice of type indices for computing derived types
// (i.e., types that depend on the declaration's type parameters).
@@ -217,7 +219,7 @@ type itabInfo struct {
// generic function or method.
func (dict *writerDict) typeParamIndex(typ *types2.TypeParam) int {
for idx, implicit := range dict.implicits {
- if implicit.Type().(*types2.TypeParam) == typ {
+ if implicit == typ {
return idx
}
}
@@ -513,24 +515,20 @@ func (pw *pkgWriter) typIdx(typ types2.Type, dict *writerDict) typeInfo {
default:
// Handle "byte" and "rune" as references to their TypeNames.
- obj := types2.Universe.Lookup(typ.Name())
+ obj := types2.Universe.Lookup(typ.Name()).(*types2.TypeName)
assert(obj.Type() == typ)
w.Code(pkgbits.TypeNamed)
- w.obj(obj, nil)
+ w.namedType(obj, nil)
}
case *types2.Named:
- obj, targs := splitNamed(typ)
-
- // Defined types that are declared within a generic function (and
- // thus have implicit type parameters) are always derived types.
- if w.p.hasImplicitTypeParams(obj) {
- w.derived = true
- }
-
w.Code(pkgbits.TypeNamed)
- w.obj(obj, targs)
+ w.namedType(splitNamed(typ))
+
+ case *types2.Alias:
+ w.Code(pkgbits.TypeNamed)
+ w.namedType(typ.Obj(), nil)
case *types2.TypeParam:
w.derived = true
@@ -596,6 +594,17 @@ func (pw *pkgWriter) typIdx(typ types2.Type, dict *writerDict) typeInfo {
return typeInfo{idx: w.Flush(), derived: false}
}
+// namedType writes a use of the given named type into the bitstream.
+func (w *writer) namedType(obj *types2.TypeName, targs *types2.TypeList) {
+ // Named types that are declared within a generic function (and
+ // thus have implicit type parameters) are always derived types.
+ if w.p.hasImplicitTypeParams(obj) {
+ w.derived = true
+ }
+
+ w.obj(obj, targs)
+}
+
func (w *writer) structType(typ *types2.Struct) {
w.Len(typ.NumFields())
for i := 0; i < typ.NumFields(); i++ {
@@ -822,7 +831,7 @@ func (w *writer) doObj(wext *writer, obj types2.Object) pkgbits.CodeObj {
case *types2.TypeName:
if obj.IsAlias() {
w.pos(obj)
- w.typ(obj.Type())
+ w.typ(types2.Unalias(obj.Type()))
return pkgbits.ObjAlias
}
@@ -889,8 +898,7 @@ func (w *writer) objDict(obj types2.Object, dict *writerDict) {
// parameter is constrained to `int | uint` but then never used in
// arithmetic/conversions/etc, we could shape those together.
for _, implicit := range dict.implicits {
- tparam := implicit.Type().(*types2.TypeParam)
- w.Bool(tparam.Underlying().(*types2.Interface).IsMethodSet())
+ w.Bool(implicit.Underlying().(*types2.Interface).IsMethodSet())
}
for i := 0; i < ntparams; i++ {
tparam := tparams.At(i)
@@ -1209,10 +1217,17 @@ func (w *writer) stmt(stmt syntax.Stmt) {
func (w *writer) stmts(stmts []syntax.Stmt) {
dead := false
w.Sync(pkgbits.SyncStmts)
- for _, stmt := range stmts {
- if dead {
- // Any statements after a terminating statement are safe to
- // omit, at least until the next labeled statement.
+ var lastLabel = -1
+ for i, stmt := range stmts {
+ if _, ok := stmt.(*syntax.LabeledStmt); ok {
+ lastLabel = i
+ }
+ }
+ for i, stmt := range stmts {
+ if dead && i > lastLabel {
+ // Any statements after a terminating and last label statement are safe to omit.
+ // Otherwise, code after label statement may refer to dead stmts between terminating
+ // and label statement, see issue #65593.
if _, ok := stmt.(*syntax.LabeledStmt); !ok {
continue
}
@@ -2124,7 +2139,7 @@ func (w *writer) methodExpr(expr *syntax.SelectorExpr, recv types2.Type, sel *ty
// Method on a type parameter. These require an indirect call
// through the current function's runtime dictionary.
- if typeParam, ok := recv.(*types2.TypeParam); w.Bool(ok) {
+ if typeParam, ok := types2.Unalias(recv).(*types2.TypeParam); w.Bool(ok) {
typeParamIdx := w.dict.typeParamIndex(typeParam)
methodInfo := w.p.selectorIdx(fun)
@@ -2137,7 +2152,7 @@ func (w *writer) methodExpr(expr *syntax.SelectorExpr, recv types2.Type, sel *ty
}
if !isInterface(recv) {
- if named, ok := deref2(recv).(*types2.Named); ok {
+ if named, ok := types2.Unalias(deref2(recv)).(*types2.Named); ok {
obj, targs := splitNamed(named)
info := w.p.objInstIdx(obj, targs, w.dict)
@@ -2362,12 +2377,16 @@ func (w *writer) varDictIndex(obj *types2.Var) {
}
}
+// isUntyped reports whether typ is an untyped type.
func isUntyped(typ types2.Type) bool {
+ // Note: types2.Unalias is unnecessary here, since untyped types can't be aliased.
basic, ok := typ.(*types2.Basic)
return ok && basic.Info()&types2.IsUntyped != 0
}
+// isTuple reports whether typ is a tuple type.
func isTuple(typ types2.Type) bool {
+ // Note: types2.Unalias is unnecessary here, since tuple types can't be aliased.
_, ok := typ.(*types2.Tuple)
return ok
}
@@ -2416,7 +2435,7 @@ func (w *writer) exprType(iface types2.Type, typ syntax.Expr) {
// If typ is a type parameter, then isInterface reports an internal
// compiler error instead.
func isInterface(typ types2.Type) bool {
- if _, ok := typ.(*types2.TypeParam); ok {
+ if _, ok := types2.Unalias(typ).(*types2.TypeParam); ok {
// typ is a type parameter and may be instantiated as either a
// concrete or interface type, so the writer can't depend on
// knowing this.
@@ -2447,7 +2466,7 @@ type typeDeclGen struct {
gen int
// Implicit type parameters in scope at this type declaration.
- implicits []*types2.TypeName
+ implicits []*types2.TypeParam
}
type fileImports struct {
@@ -2465,7 +2484,7 @@ type declCollector struct {
typegen *int
file *fileImports
withinFunc bool
- implicits []*types2.TypeName
+ implicits []*types2.TypeParam
}
func (c *declCollector) withTParams(obj types2.Object) *declCollector {
@@ -2478,7 +2497,7 @@ func (c *declCollector) withTParams(obj types2.Object) *declCollector {
copy := *c
copy.implicits = copy.implicits[:len(copy.implicits):len(copy.implicits)]
for i := 0; i < n; i++ {
- copy.implicits = append(copy.implicits, tparams.At(i).Obj())
+ copy.implicits = append(copy.implicits, tparams.At(i))
}
return ©
}
@@ -2867,9 +2886,9 @@ func (pw *pkgWriter) isBuiltin(expr syntax.Expr, builtin string) bool {
// recvBase returns the base type for the given receiver parameter.
func recvBase(recv *types2.Var) *types2.Named {
- typ := recv.Type()
+ typ := types2.Unalias(recv.Type())
if ptr, ok := typ.(*types2.Pointer); ok {
- typ = ptr.Elem()
+ typ = types2.Unalias(ptr.Elem())
}
return typ.(*types2.Named)
}
@@ -2945,7 +2964,7 @@ func asWasmImport(p syntax.Pragma) *WasmImport {
// isPtrTo reports whether from is the type *to.
func isPtrTo(from, to types2.Type) bool {
- ptr, ok := from.(*types2.Pointer)
+ ptr, ok := types2.Unalias(from).(*types2.Pointer)
return ok && types2.Identical(ptr.Elem(), to)
}
diff --git a/src/cmd/compile/internal/pgo/irgraph.go b/src/cmd/compile/internal/pgo/irgraph.go
index 7a7cd20f2b..9ed16d224b 100644
--- a/src/cmd/compile/internal/pgo/irgraph.go
+++ b/src/cmd/compile/internal/pgo/irgraph.go
@@ -41,15 +41,19 @@
package pgo
import (
+ "bufio"
"cmd/compile/internal/base"
"cmd/compile/internal/ir"
- "cmd/compile/internal/pgo/internal/graph"
"cmd/compile/internal/typecheck"
"cmd/compile/internal/types"
+ "errors"
"fmt"
"internal/profile"
+ "io"
"os"
"sort"
+ "strconv"
+ "strings"
)
// IRGraph is a call graph with nodes pointing to IRs of functions and edges
@@ -129,7 +133,7 @@ type Profile struct {
// the percentage threshold for hot/cold partitioning.
TotalWeight int64
- // EdgeMap contains all unique call edges in the profile and their
+ // NamedEdgeMap contains all unique call edges in the profile and their
// edge weight.
NamedEdgeMap NamedEdgeMap
@@ -138,25 +142,69 @@ type Profile struct {
WeightedCG *IRGraph
}
-// New generates a profile-graph from the profile.
+var wantHdr = "GO PREPROFILE V1\n"
+
+func isPreProfileFile(r *bufio.Reader) (bool, error) {
+ hdr, err := r.Peek(len(wantHdr))
+ if err == io.EOF {
+ // Empty file.
+ return false, nil
+ } else if err != nil {
+ return false, fmt.Errorf("error reading profile header: %w", err)
+ }
+
+ return string(hdr) == wantHdr, nil
+}
+
+// New generates a profile-graph from the profile or pre-processed profile.
func New(profileFile string) (*Profile, error) {
f, err := os.Open(profileFile)
if err != nil {
return nil, fmt.Errorf("error opening profile: %w", err)
}
defer f.Close()
- profile, err := profile.Parse(f)
+
+ r := bufio.NewReader(f)
+
+ isPreProf, err := isPreProfileFile(r)
if err != nil {
+ return nil, fmt.Errorf("error processing profile header: %w", err)
+ }
+
+ if isPreProf {
+ profile, err := processPreprof(r)
+ if err != nil {
+ return nil, fmt.Errorf("error processing preprocessed PGO profile: %w", err)
+ }
+ return profile, nil
+ }
+
+ profile, err := processProto(r)
+ if err != nil {
+ return nil, fmt.Errorf("error processing pprof PGO profile: %w", err)
+ }
+ return profile, nil
+
+}
+
+// processProto generates a profile-graph from the profile.
+func processProto(r io.Reader) (*Profile, error) {
+ p, err := profile.Parse(r)
+ if errors.Is(err, profile.ErrNoData) {
+ // Treat a completely empty file the same as a profile with no
+ // samples: nothing to do.
+ return nil, nil
+ } else if err != nil {
return nil, fmt.Errorf("error parsing profile: %w", err)
}
- if len(profile.Sample) == 0 {
+ if len(p.Sample) == 0 {
// We accept empty profiles, but there is nothing to do.
return nil, nil
}
valueIndex := -1
- for i, s := range profile.SampleType {
+ for i, s := range p.SampleType {
// Samples count is the raw data collected, and CPU nanoseconds is just
// a scaled version of it, so either one we can find is fine.
if (s.Type == "samples" && s.Unit == "count") ||
@@ -170,7 +218,7 @@ func New(profileFile string) (*Profile, error) {
return nil, fmt.Errorf(`profile does not contain a sample index with value/type "samples/count" or cpu/nanoseconds"`)
}
- g := graph.NewGraph(profile, &graph.Options{
+ g := profile.NewGraph(p, &profile.Options{
SampleValue: func(v []int64) int64 { return v[valueIndex] },
})
@@ -193,45 +241,31 @@ func New(profileFile string) (*Profile, error) {
}, nil
}
-// createNamedEdgeMap builds a map of callsite-callee edge weights from the
-// profile-graph.
-//
-// Caller should ignore the profile if totalWeight == 0.
-func createNamedEdgeMap(g *graph.Graph) (edgeMap NamedEdgeMap, totalWeight int64, err error) {
- seenStartLine := false
-
- // Process graph and build various node and edge maps which will
- // be consumed by AST walk.
- weight := make(map[NamedCallEdge]int64)
- for _, n := range g.Nodes {
- seenStartLine = seenStartLine || n.Info.StartLine != 0
-
- canonicalName := n.Info.Name
- // Create the key to the nodeMapKey.
- namedEdge := NamedCallEdge{
- CallerName: canonicalName,
- CallSiteOffset: n.Info.Lineno - n.Info.StartLine,
- }
-
- for _, e := range n.Out {
- totalWeight += e.WeightValue()
- namedEdge.CalleeName = e.Dest.Info.Name
- // Create new entry or increment existing entry.
- weight[namedEdge] += e.WeightValue()
- }
+// processPreprof generates a profile-graph from the pre-procesed profile.
+func processPreprof(r io.Reader) (*Profile, error) {
+ namedEdgeMap, totalWeight, err := createNamedEdgeMapFromPreprocess(r)
+ if err != nil {
+ return nil, err
}
if totalWeight == 0 {
+ return nil, nil // accept but ignore profile with no samples.
+ }
+
+ // Create package-level call graph with weights from profile and IR.
+ wg := createIRGraph(namedEdgeMap)
+
+ return &Profile{
+ TotalWeight: totalWeight,
+ NamedEdgeMap: namedEdgeMap,
+ WeightedCG: wg,
+ }, nil
+}
+
+func postProcessNamedEdgeMap(weight map[NamedCallEdge]int64, weightVal int64) (edgeMap NamedEdgeMap, totalWeight int64, err error) {
+ if weightVal == 0 {
return NamedEdgeMap{}, 0, nil // accept but ignore profile with no samples.
}
-
- if !seenStartLine {
- // TODO(prattmic): If Function.start_line is missing we could
- // fall back to using absolute line numbers, which is better
- // than nothing.
- return NamedEdgeMap{}, 0, fmt.Errorf("profile missing Function.start_line data (Go version of profiled application too old? Go 1.20+ automatically adds this to profiles)")
- }
-
byWeight := make([]NamedCallEdge, 0, len(weight))
for namedEdge := range weight {
byWeight = append(byWeight, namedEdge)
@@ -256,9 +290,110 @@ func createNamedEdgeMap(g *graph.Graph) (edgeMap NamedEdgeMap, totalWeight int64
ByWeight: byWeight,
}
+ totalWeight = weightVal
+
return edgeMap, totalWeight, nil
}
+// restore NodeMap information from a preprocessed profile.
+// The reader can refer to the format of preprocessed profile in cmd/preprofile/main.go.
+func createNamedEdgeMapFromPreprocess(r io.Reader) (edgeMap NamedEdgeMap, totalWeight int64, err error) {
+ fileScanner := bufio.NewScanner(r)
+ fileScanner.Split(bufio.ScanLines)
+ weight := make(map[NamedCallEdge]int64)
+
+ if !fileScanner.Scan() {
+ if err := fileScanner.Err(); err != nil {
+ return NamedEdgeMap{}, 0, fmt.Errorf("error reading preprocessed profile: %w", err)
+ }
+ return NamedEdgeMap{}, 0, fmt.Errorf("preprocessed profile missing header")
+ }
+ if gotHdr := fileScanner.Text() + "\n"; gotHdr != wantHdr {
+ return NamedEdgeMap{}, 0, fmt.Errorf("preprocessed profile malformed header; got %q want %q", gotHdr, wantHdr)
+ }
+
+ for fileScanner.Scan() {
+ readStr := fileScanner.Text()
+
+ callerName := readStr
+
+ if !fileScanner.Scan() {
+ if err := fileScanner.Err(); err != nil {
+ return NamedEdgeMap{}, 0, fmt.Errorf("error reading preprocessed profile: %w", err)
+ }
+ return NamedEdgeMap{}, 0, fmt.Errorf("preprocessed profile entry missing callee")
+ }
+ calleeName := fileScanner.Text()
+
+ if !fileScanner.Scan() {
+ if err := fileScanner.Err(); err != nil {
+ return NamedEdgeMap{}, 0, fmt.Errorf("error reading preprocessed profile: %w", err)
+ }
+ return NamedEdgeMap{}, 0, fmt.Errorf("preprocessed profile entry missing weight")
+ }
+ readStr = fileScanner.Text()
+
+ split := strings.Split(readStr, " ")
+
+ if len(split) != 2 {
+ return NamedEdgeMap{}, 0, fmt.Errorf("preprocessed profile entry got %v want 2 fields", split)
+ }
+
+ co, _ := strconv.Atoi(split[0])
+
+ namedEdge := NamedCallEdge{
+ CallerName: callerName,
+ CalleeName: calleeName,
+ CallSiteOffset: co,
+ }
+
+ EWeight, _ := strconv.ParseInt(split[1], 10, 64)
+
+ weight[namedEdge] += EWeight
+ totalWeight += EWeight
+ }
+
+ return postProcessNamedEdgeMap(weight, totalWeight)
+
+}
+
+// createNamedEdgeMap builds a map of callsite-callee edge weights from the
+// profile-graph.
+//
+// Caller should ignore the profile if totalWeight == 0.
+func createNamedEdgeMap(g *profile.Graph) (edgeMap NamedEdgeMap, totalWeight int64, err error) {
+ seenStartLine := false
+
+ // Process graph and build various node and edge maps which will
+ // be consumed by AST walk.
+ weight := make(map[NamedCallEdge]int64)
+ for _, n := range g.Nodes {
+ seenStartLine = seenStartLine || n.Info.StartLine != 0
+
+ canonicalName := n.Info.Name
+ // Create the key to the nodeMapKey.
+ namedEdge := NamedCallEdge{
+ CallerName: canonicalName,
+ CallSiteOffset: n.Info.Lineno - n.Info.StartLine,
+ }
+
+ for _, e := range n.Out {
+ totalWeight += e.WeightValue()
+ namedEdge.CalleeName = e.Dest.Info.Name
+ // Create new entry or increment existing entry.
+ weight[namedEdge] += e.WeightValue()
+ }
+ }
+
+ if !seenStartLine {
+ // TODO(prattmic): If Function.start_line is missing we could
+ // fall back to using absolute line numbers, which is better
+ // than nothing.
+ return NamedEdgeMap{}, 0, fmt.Errorf("profile missing Function.start_line data (Go version of profiled application too old? Go 1.20+ automatically adds this to profiles)")
+ }
+ return postProcessNamedEdgeMap(weight, totalWeight)
+}
+
// initializeIRGraph builds the IRGraph by visiting all the ir.Func in decl list
// of a package.
func createIRGraph(namedEdgeMap NamedEdgeMap) *IRGraph {
diff --git a/src/cmd/compile/internal/rangefunc/rewrite.go b/src/cmd/compile/internal/rangefunc/rewrite.go
index 7475c570aa..d439412ea8 100644
--- a/src/cmd/compile/internal/rangefunc/rewrite.go
+++ b/src/cmd/compile/internal/rangefunc/rewrite.go
@@ -934,7 +934,7 @@ func (r *rewriter) endLoop(loop *forLoop) {
if rfunc.Params().Len() != 1 {
base.Fatalf("invalid typecheck of range func")
}
- ftyp := rfunc.Params().At(0).Type().(*types2.Signature) // func(...) bool
+ ftyp := types2.CoreType(rfunc.Params().At(0).Type()).(*types2.Signature) // func(...) bool
if ftyp.Results().Len() != 1 {
base.Fatalf("invalid typecheck of range func")
}
diff --git a/src/cmd/compile/internal/reflectdata/reflect.go b/src/cmd/compile/internal/reflectdata/reflect.go
index c2407af017..185be4dd51 100644
--- a/src/cmd/compile/internal/reflectdata/reflect.go
+++ b/src/cmd/compile/internal/reflectdata/reflect.go
@@ -55,24 +55,6 @@ type typeSig struct {
mtype *types.Type
}
-// Builds a type representing a Bucket structure for
-// the given map type. This type is not visible to users -
-// we include only enough information to generate a correct GC
-// program for it.
-// Make sure this stays in sync with runtime/map.go.
-//
-// A "bucket" is a "struct" {
-// tophash [BUCKETSIZE]uint8
-// keys [BUCKETSIZE]keyType
-// elems [BUCKETSIZE]elemType
-// overflow *bucket
-// }
-const (
- BUCKETSIZE = abi.MapBucketCount
- MAXKEYSIZE = abi.MapMaxKeyBytes
- MAXELEMSIZE = abi.MapMaxElemBytes
-)
-
func commonSize() int { return int(rttype.Type.Size()) } // Sizeof(runtime._type{})
func uncommonSize(t *types.Type) int { // Sizeof(runtime.uncommontype{})
@@ -89,6 +71,18 @@ func makefield(name string, t *types.Type) *types.Field {
// MapBucketType makes the map bucket type given the type of the map.
func MapBucketType(t *types.Type) *types.Type {
+ // Builds a type representing a Bucket structure for
+ // the given map type. This type is not visible to users -
+ // we include only enough information to generate a correct GC
+ // program for it.
+ // Make sure this stays in sync with runtime/map.go.
+ //
+ // A "bucket" is a "struct" {
+ // tophash [abi.MapBucketCount]uint8
+ // keys [abi.MapBucketCount]keyType
+ // elems [abi.MapBucketCount]elemType
+ // overflow *bucket
+ // }
if t.MapType().Bucket != nil {
return t.MapType().Bucket
}
@@ -97,25 +91,25 @@ func MapBucketType(t *types.Type) *types.Type {
elemtype := t.Elem()
types.CalcSize(keytype)
types.CalcSize(elemtype)
- if keytype.Size() > MAXKEYSIZE {
+ if keytype.Size() > abi.MapMaxKeyBytes {
keytype = types.NewPtr(keytype)
}
- if elemtype.Size() > MAXELEMSIZE {
+ if elemtype.Size() > abi.MapMaxElemBytes {
elemtype = types.NewPtr(elemtype)
}
field := make([]*types.Field, 0, 5)
// The first field is: uint8 topbits[BUCKETSIZE].
- arr := types.NewArray(types.Types[types.TUINT8], BUCKETSIZE)
+ arr := types.NewArray(types.Types[types.TUINT8], abi.MapBucketCount)
field = append(field, makefield("topbits", arr))
- arr = types.NewArray(keytype, BUCKETSIZE)
+ arr = types.NewArray(keytype, abi.MapBucketCount)
arr.SetNoalg(true)
keys := makefield("keys", arr)
field = append(field, keys)
- arr = types.NewArray(elemtype, BUCKETSIZE)
+ arr = types.NewArray(elemtype, abi.MapBucketCount)
arr.SetNoalg(true)
elems := makefield("elems", arr)
field = append(field, elems)
@@ -142,25 +136,25 @@ func MapBucketType(t *types.Type) *types.Type {
if !types.IsComparable(t.Key()) {
base.Fatalf("unsupported map key type for %v", t)
}
- if BUCKETSIZE < 8 {
- base.Fatalf("bucket size %d too small for proper alignment %d", BUCKETSIZE, 8)
+ if abi.MapBucketCount < 8 {
+ base.Fatalf("bucket size %d too small for proper alignment %d", abi.MapBucketCount, 8)
}
- if uint8(keytype.Alignment()) > BUCKETSIZE {
+ if uint8(keytype.Alignment()) > abi.MapBucketCount {
base.Fatalf("key align too big for %v", t)
}
- if uint8(elemtype.Alignment()) > BUCKETSIZE {
- base.Fatalf("elem align %d too big for %v, BUCKETSIZE=%d", elemtype.Alignment(), t, BUCKETSIZE)
+ if uint8(elemtype.Alignment()) > abi.MapBucketCount {
+ base.Fatalf("elem align %d too big for %v, BUCKETSIZE=%d", elemtype.Alignment(), t, abi.MapBucketCount)
}
- if keytype.Size() > MAXKEYSIZE {
+ if keytype.Size() > abi.MapMaxKeyBytes {
base.Fatalf("key size too large for %v", t)
}
- if elemtype.Size() > MAXELEMSIZE {
+ if elemtype.Size() > abi.MapMaxElemBytes {
base.Fatalf("elem size too large for %v", t)
}
- if t.Key().Size() > MAXKEYSIZE && !keytype.IsPtr() {
+ if t.Key().Size() > abi.MapMaxKeyBytes && !keytype.IsPtr() {
base.Fatalf("key indirect incorrect for %v", t)
}
- if t.Elem().Size() > MAXELEMSIZE && !elemtype.IsPtr() {
+ if t.Elem().Size() > abi.MapMaxElemBytes && !elemtype.IsPtr() {
base.Fatalf("elem indirect incorrect for %v", t)
}
if keytype.Size()%keytype.Alignment() != 0 {
@@ -1124,14 +1118,14 @@ func writeType(t *types.Type) *obj.LSym {
var flags uint32
// Note: flags must match maptype accessors in ../../../../runtime/type.go
// and maptype builder in ../../../../reflect/type.go:MapOf.
- if t.Key().Size() > MAXKEYSIZE {
+ if t.Key().Size() > abi.MapMaxKeyBytes {
c.Field("KeySize").WriteUint8(uint8(types.PtrSize))
flags |= 1 // indirect key
} else {
c.Field("KeySize").WriteUint8(uint8(t.Key().Size()))
}
- if t.Elem().Size() > MAXELEMSIZE {
+ if t.Elem().Size() > abi.MapMaxElemBytes {
c.Field("ValueSize").WriteUint8(uint8(types.PtrSize))
flags |= 2 // indirect value
} else {
@@ -1337,20 +1331,25 @@ func writeITab(lsym *obj.LSym, typ, iface *types.Type, allowNonImplement bool) {
// _ [4]byte
// fun [1]uintptr // variable sized. fun[0]==0 means _type does not implement inter.
// }
- o := objw.SymPtr(lsym, 0, writeType(iface), 0)
- o = objw.SymPtr(lsym, o, writeType(typ), 0)
- o = objw.Uint32(lsym, o, types.TypeHash(typ)) // copy of type hash
- o += 4 // skip unused field
+ c := rttype.NewCursor(lsym, 0, rttype.ITab)
+ c.Field("Inter").WritePtr(writeType(iface))
+ c.Field("Type").WritePtr(writeType(typ))
+ c.Field("Hash").WriteUint32(types.TypeHash(typ)) // copy of type hash
+
+ var delta int64
+ c = c.Field("Fun")
if !completeItab {
// If typ doesn't implement iface, make method entries be zero.
- o = objw.Uintptr(lsym, o, 0)
- entries = entries[:0]
- }
- for _, fn := range entries {
- o = objw.SymPtrWeak(lsym, o, fn, 0) // method pointer for each method
+ c.Elem(0).WriteUintptr(0)
+ } else {
+ var a rttype.ArrayCursor
+ a, delta = c.ModifyArray(len(entries))
+ for i, fn := range entries {
+ a.Elem(i).WritePtrWeak(fn) // method pointer for each method
+ }
}
// Nothing writes static itabs, so they are read only.
- objw.Global(lsym, int32(o), int16(obj.DUPOK|obj.RODATA))
+ objw.Global(lsym, int32(rttype.ITab.Size()+delta), int16(obj.DUPOK|obj.RODATA))
lsym.Set(obj.AttrContentAddressable, true)
}
@@ -1499,39 +1498,6 @@ func (a typesByString) Less(i, j int) bool {
}
func (a typesByString) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
-// maxPtrmaskBytes is the maximum length of a GC ptrmask bitmap,
-// which holds 1-bit entries describing where pointers are in a given type.
-// Above this length, the GC information is recorded as a GC program,
-// which can express repetition compactly. In either form, the
-// information is used by the runtime to initialize the heap bitmap,
-// and for large types (like 128 or more words), they are roughly the
-// same speed. GC programs are never much larger and often more
-// compact. (If large arrays are involved, they can be arbitrarily
-// more compact.)
-//
-// The cutoff must be large enough that any allocation large enough to
-// use a GC program is large enough that it does not share heap bitmap
-// bytes with any other objects, allowing the GC program execution to
-// assume an aligned start and not use atomic operations. In the current
-// runtime, this means all malloc size classes larger than the cutoff must
-// be multiples of four words. On 32-bit systems that's 16 bytes, and
-// all size classes >= 16 bytes are 16-byte aligned, so no real constraint.
-// On 64-bit systems, that's 32 bytes, and 32-byte alignment is guaranteed
-// for size classes >= 256 bytes. On a 64-bit system, 256 bytes allocated
-// is 32 pointers, the bits for which fit in 4 bytes. So maxPtrmaskBytes
-// must be >= 4.
-//
-// We used to use 16 because the GC programs do have some constant overhead
-// to get started, and processing 128 pointers seems to be enough to
-// amortize that overhead well.
-//
-// To make sure that the runtime's chansend can call typeBitsBulkBarrier,
-// we raised the limit to 2048, so that even 32-bit systems are guaranteed to
-// use bitmaps for objects up to 64 kB in size.
-//
-// Also known to reflect/type.go.
-const maxPtrmaskBytes = 2048
-
// GCSym returns a data symbol containing GC information for type t, along
// with a boolean reporting whether the UseGCProg bit should be set in the
// type kind, and the ptrdata field to record in the reflect type information.
@@ -1554,7 +1520,7 @@ func GCSym(t *types.Type) (lsym *obj.LSym, useGCProg bool, ptrdata int64) {
// When write is true, it writes the symbol data.
func dgcsym(t *types.Type, write bool) (lsym *obj.LSym, useGCProg bool, ptrdata int64) {
ptrdata = types.PtrDataSize(t)
- if ptrdata/int64(types.PtrSize) <= maxPtrmaskBytes*8 {
+ if ptrdata/int64(types.PtrSize) <= abi.MaxPtrmaskBytes*8 {
lsym = dgcptrmask(t, write)
return
}
diff --git a/src/cmd/compile/internal/riscv64/ssa.go b/src/cmd/compile/internal/riscv64/ssa.go
index 22338188e5..17f0d98532 100644
--- a/src/cmd/compile/internal/riscv64/ssa.go
+++ b/src/cmd/compile/internal/riscv64/ssa.go
@@ -278,7 +278,7 @@ func ssaGenValue(s *ssagen.State, v *ssa.Value) {
p.To.Type = obj.TYPE_REG
p.To.Reg = rd
case ssa.OpRISCV64ADD, ssa.OpRISCV64SUB, ssa.OpRISCV64SUBW, ssa.OpRISCV64XOR, ssa.OpRISCV64OR, ssa.OpRISCV64AND,
- ssa.OpRISCV64SLL, ssa.OpRISCV64SRA, ssa.OpRISCV64SRAW, ssa.OpRISCV64SRL, ssa.OpRISCV64SRLW,
+ ssa.OpRISCV64SLL, ssa.OpRISCV64SLLW, ssa.OpRISCV64SRA, ssa.OpRISCV64SRAW, ssa.OpRISCV64SRL, ssa.OpRISCV64SRLW,
ssa.OpRISCV64SLT, ssa.OpRISCV64SLTU, ssa.OpRISCV64MUL, ssa.OpRISCV64MULW, ssa.OpRISCV64MULH,
ssa.OpRISCV64MULHU, ssa.OpRISCV64DIV, ssa.OpRISCV64DIVU, ssa.OpRISCV64DIVW,
ssa.OpRISCV64DIVUW, ssa.OpRISCV64REM, ssa.OpRISCV64REMU, ssa.OpRISCV64REMW,
@@ -297,6 +297,72 @@ func ssaGenValue(s *ssagen.State, v *ssa.Value) {
p.Reg = r1
p.To.Type = obj.TYPE_REG
p.To.Reg = r
+
+ case ssa.OpRISCV64LoweredFMAXD, ssa.OpRISCV64LoweredFMIND, ssa.OpRISCV64LoweredFMAXS, ssa.OpRISCV64LoweredFMINS:
+ // Most of FMIN/FMAX result match Go's required behaviour, unless one of the
+ // inputs is a NaN. As such, we need to explicitly test for NaN
+ // before using FMIN/FMAX.
+
+ // FADD Rarg0, Rarg1, Rout // FADD is used to propagate a NaN to the result in these cases.
+ // FEQ Rarg0, Rarg0, Rtmp
+ // BEQZ Rtmp, end
+ // FEQ Rarg1, Rarg1, Rtmp
+ // BEQZ Rtmp, end
+ // F(MIN | MAX)
+
+ r0 := v.Args[0].Reg()
+ r1 := v.Args[1].Reg()
+ out := v.Reg()
+ add, feq := riscv.AFADDD, riscv.AFEQD
+ if v.Op == ssa.OpRISCV64LoweredFMAXS || v.Op == ssa.OpRISCV64LoweredFMINS {
+ add = riscv.AFADDS
+ feq = riscv.AFEQS
+ }
+
+ p1 := s.Prog(add)
+ p1.From.Type = obj.TYPE_REG
+ p1.From.Reg = r0
+ p1.Reg = r1
+ p1.To.Type = obj.TYPE_REG
+ p1.To.Reg = out
+
+ p2 := s.Prog(feq)
+ p2.From.Type = obj.TYPE_REG
+ p2.From.Reg = r0
+ p2.Reg = r0
+ p2.To.Type = obj.TYPE_REG
+ p2.To.Reg = riscv.REG_TMP
+
+ p3 := s.Prog(riscv.ABEQ)
+ p3.From.Type = obj.TYPE_REG
+ p3.From.Reg = riscv.REG_ZERO
+ p3.Reg = riscv.REG_TMP
+ p3.To.Type = obj.TYPE_BRANCH
+
+ p4 := s.Prog(feq)
+ p4.From.Type = obj.TYPE_REG
+ p4.From.Reg = r1
+ p4.Reg = r1
+ p4.To.Type = obj.TYPE_REG
+ p4.To.Reg = riscv.REG_TMP
+
+ p5 := s.Prog(riscv.ABEQ)
+ p5.From.Type = obj.TYPE_REG
+ p5.From.Reg = riscv.REG_ZERO
+ p5.Reg = riscv.REG_TMP
+ p5.To.Type = obj.TYPE_BRANCH
+
+ p6 := s.Prog(v.Op.Asm())
+ p6.From.Type = obj.TYPE_REG
+ p6.From.Reg = r1
+ p6.Reg = r0
+ p6.To.Type = obj.TYPE_REG
+ p6.To.Reg = out
+
+ nop := s.Prog(obj.ANOP)
+ p3.To.SetTarget(nop)
+ p5.To.SetTarget(nop)
+
case ssa.OpRISCV64LoweredMuluhilo:
r0 := v.Args[0].Reg()
r1 := v.Args[1].Reg()
@@ -356,8 +422,8 @@ func ssaGenValue(s *ssagen.State, v *ssa.Value) {
p.To.Type = obj.TYPE_REG
p.To.Reg = v.Reg()
case ssa.OpRISCV64ADDI, ssa.OpRISCV64ADDIW, ssa.OpRISCV64XORI, ssa.OpRISCV64ORI, ssa.OpRISCV64ANDI,
- ssa.OpRISCV64SLLI, ssa.OpRISCV64SRAI, ssa.OpRISCV64SRAIW, ssa.OpRISCV64SRLI, ssa.OpRISCV64SRLIW, ssa.OpRISCV64SLTI,
- ssa.OpRISCV64SLTIU:
+ ssa.OpRISCV64SLLI, ssa.OpRISCV64SLLIW, ssa.OpRISCV64SRAI, ssa.OpRISCV64SRAIW,
+ ssa.OpRISCV64SRLI, ssa.OpRISCV64SRLIW, ssa.OpRISCV64SLTI, ssa.OpRISCV64SLTIU:
p := s.Prog(v.Op.Asm())
p.From.Type = obj.TYPE_CONST
p.From.Offset = v.AuxInt
diff --git a/src/cmd/compile/internal/rttype/rttype.go b/src/cmd/compile/internal/rttype/rttype.go
index cdc399d9cf..b90e23dc5b 100644
--- a/src/cmd/compile/internal/rttype/rttype.go
+++ b/src/cmd/compile/internal/rttype/rttype.go
@@ -42,6 +42,9 @@ var UncommonType *types.Type
var InterfaceSwitch *types.Type
var TypeAssert *types.Type
+// Interface tables (itabs)
+var ITab *types.Type
+
func Init() {
// Note: this has to be called explicitly instead of being
// an init function so it runs after the types package has
@@ -64,6 +67,8 @@ func Init() {
InterfaceSwitch = fromReflect(reflect.TypeOf(abi.InterfaceSwitch{}))
TypeAssert = fromReflect(reflect.TypeOf(abi.TypeAssert{}))
+ ITab = fromReflect(reflect.TypeOf(abi.ITab{}))
+
// Make sure abi functions are correct. These functions are used
// by the linker which doesn't have the ability to do type layout,
// so we check the functions it uses here.
@@ -80,6 +85,9 @@ func Init() {
if got, want := int64(abi.TFlagOff(ptrSize)), Type.OffsetOf("TFlag"); got != want {
base.Fatalf("abi.TFlagOff() == %d, want %d", got, want)
}
+ if got, want := int64(abi.ITabTypeOff(ptrSize)), ITab.OffsetOf("Type"); got != want {
+ base.Fatalf("abi.ITabTypeOff() == %d, want %d", got, want)
+ }
}
// fromReflect translates from a host type to the equivalent target type.
@@ -154,6 +162,12 @@ func (c Cursor) WritePtr(target *obj.LSym) {
objw.SymPtr(c.lsym, int(c.offset), target, 0)
}
}
+func (c Cursor) WritePtrWeak(target *obj.LSym) {
+ if c.typ.Kind() != types.TUINTPTR {
+ base.Fatalf("can't write ptr, it has kind %s", c.typ.Kind())
+ }
+ objw.SymPtrWeak(c.lsym, int(c.offset), target, 0)
+}
func (c Cursor) WriteUintptr(val uint64) {
if c.typ.Kind() != types.TUINTPTR {
base.Fatalf("can't write uintptr, it has kind %s", c.typ.Kind())
@@ -250,6 +264,17 @@ func (c Cursor) Field(name string) Cursor {
return Cursor{}
}
+func (c Cursor) Elem(i int64) Cursor {
+ if c.typ.Kind() != types.TARRAY {
+ base.Fatalf("can't call Elem on non-array %v", c.typ)
+ }
+ if i < 0 || i >= c.typ.NumElem() {
+ base.Fatalf("element access out of bounds [%d] in [0:%d]", i, c.typ.NumElem())
+ }
+ elem := c.typ.Elem()
+ return Cursor{lsym: c.lsym, offset: c.offset + i*elem.Size(), typ: elem}
+}
+
type ArrayCursor struct {
c Cursor // cursor pointing at first element
n int // number of elements
diff --git a/src/cmd/compile/internal/ssa/_gen/ARM.rules b/src/cmd/compile/internal/ssa/_gen/ARM.rules
index a60afb000a..ed0ed80afa 100644
--- a/src/cmd/compile/internal/ssa/_gen/ARM.rules
+++ b/src/cmd/compile/internal/ssa/_gen/ARM.rules
@@ -66,17 +66,17 @@
// count trailing zero for ARMv5 and ARMv6
// 32 - CLZ(x&-x - 1)
-(Ctz32 x) && buildcfg.GOARM<=6 =>
+(Ctz32 x) && buildcfg.GOARM.Version<=6 =>
(RSBconst [32] (CLZ (SUBconst (AND x (RSBconst [0] x)) [1])))
-(Ctz16 x) && buildcfg.GOARM<=6 =>
+(Ctz16 x) && buildcfg.GOARM.Version<=6 =>
(RSBconst [32] (CLZ (SUBconst (AND (ORconst [0x10000] x) (RSBconst [0] (ORconst [0x10000] x))) [1])))
-(Ctz8 x) && buildcfg.GOARM<=6 =>
+(Ctz8 x) && buildcfg.GOARM.Version<=6 =>
(RSBconst [32] (CLZ (SUBconst (AND (ORconst [0x100] x) (RSBconst [0] (ORconst [0x100] x))) [1])))
// count trailing zero for ARMv7
-(Ctz32 x) && buildcfg.GOARM==7 => (CLZ (RBIT x))
-(Ctz16 x) && buildcfg.GOARM==7 => (CLZ (RBIT (ORconst [0x10000] x)))
-(Ctz8 x) && buildcfg.GOARM==7 => (CLZ (RBIT (ORconst [0x100] x)))
+(Ctz32 x) && buildcfg.GOARM.Version==7 => (CLZ (RBIT x))
+(Ctz16 x) && buildcfg.GOARM.Version==7 => (CLZ (RBIT (ORconst [0x10000] x)))
+(Ctz8 x) && buildcfg.GOARM.Version==7 => (CLZ (RBIT (ORconst [0x100] x)))
// bit length
(BitLen32 x) => (RSBconst [32] (CLZ x))
@@ -90,13 +90,13 @@
// t5 = x right rotate 8 bits -- (d, a, b, c )
// result = t4 ^ t5 -- (d, c, b, a )
// using shifted ops this can be done in 4 instructions.
-(Bswap32 x) && buildcfg.GOARM==5 =>
+(Bswap32 x) && buildcfg.GOARM.Version==5 =>
(XOR
(SRLconst (BICconst (XOR x (SRRconst [16] x)) [0xff0000]) [8])
(SRRconst x [8]))
// byte swap for ARMv6 and above
-(Bswap32 x) && buildcfg.GOARM>=6 => (REV x)
+(Bswap32 x) && buildcfg.GOARM.Version>=6 => (REV x)
// boolean ops -- booleans are represented with 0=false, 1=true
(AndB ...) => (AND ...)
@@ -741,10 +741,10 @@
(SUBconst [c] x) && !isARMImmRot(uint32(c)) && isARMImmRot(uint32(-c)) => (ADDconst [-c] x)
(ANDconst [c] x) && !isARMImmRot(uint32(c)) && isARMImmRot(^uint32(c)) => (BICconst [int32(^uint32(c))] x)
(BICconst [c] x) && !isARMImmRot(uint32(c)) && isARMImmRot(^uint32(c)) => (ANDconst [int32(^uint32(c))] x)
-(ADDconst [c] x) && buildcfg.GOARM==7 && !isARMImmRot(uint32(c)) && uint32(c)>0xffff && uint32(-c)<=0xffff => (SUBconst [-c] x)
-(SUBconst [c] x) && buildcfg.GOARM==7 && !isARMImmRot(uint32(c)) && uint32(c)>0xffff && uint32(-c)<=0xffff => (ADDconst [-c] x)
-(ANDconst [c] x) && buildcfg.GOARM==7 && !isARMImmRot(uint32(c)) && uint32(c)>0xffff && ^uint32(c)<=0xffff => (BICconst [int32(^uint32(c))] x)
-(BICconst [c] x) && buildcfg.GOARM==7 && !isARMImmRot(uint32(c)) && uint32(c)>0xffff && ^uint32(c)<=0xffff => (ANDconst [int32(^uint32(c))] x)
+(ADDconst [c] x) && buildcfg.GOARM.Version==7 && !isARMImmRot(uint32(c)) && uint32(c)>0xffff && uint32(-c)<=0xffff => (SUBconst [-c] x)
+(SUBconst [c] x) && buildcfg.GOARM.Version==7 && !isARMImmRot(uint32(c)) && uint32(c)>0xffff && uint32(-c)<=0xffff => (ADDconst [-c] x)
+(ANDconst [c] x) && buildcfg.GOARM.Version==7 && !isARMImmRot(uint32(c)) && uint32(c)>0xffff && ^uint32(c)<=0xffff => (BICconst [int32(^uint32(c))] x)
+(BICconst [c] x) && buildcfg.GOARM.Version==7 && !isARMImmRot(uint32(c)) && uint32(c)>0xffff && ^uint32(c)<=0xffff => (ANDconst [int32(^uint32(c))] x)
(ADDconst [c] (MOVWconst [d])) => (MOVWconst [c+d])
(ADDconst [c] (ADDconst [d] x)) => (ADDconst [c+d] x)
(ADDconst [c] (SUBconst [d] x)) => (ADDconst [c-d] x)
@@ -1139,7 +1139,7 @@
// UBFX instruction is supported by ARMv6T2, ARMv7 and above versions, REV16 is supported by
// ARMv6 and above versions. So for ARMv6, we need to match SLLconst, SRLconst and ORshiftLL.
((ADDshiftLL|ORshiftLL|XORshiftLL) [8] (BFXU [int32(armBFAuxInt(8, 8))] x) x) => (REV16 x)
-((ADDshiftLL|ORshiftLL|XORshiftLL) [8] (SRLconst [24] (SLLconst [16] x)) x) && buildcfg.GOARM>=6 => (REV16 x)
+((ADDshiftLL|ORshiftLL|XORshiftLL) [8] (SRLconst [24] (SLLconst [16] x)) x) && buildcfg.GOARM.Version>=6 => (REV16 x)
// use indexed loads and stores
(MOVWload [0] {sym} (ADD ptr idx) mem) && sym == nil => (MOVWloadidx ptr idx mem)
@@ -1209,25 +1209,25 @@
(BIC x x) => (MOVWconst [0])
(ADD (MUL x y) a) => (MULA x y a)
-(SUB a (MUL x y)) && buildcfg.GOARM == 7 => (MULS x y a)
-(RSB (MUL x y) a) && buildcfg.GOARM == 7 => (MULS x y a)
+(SUB a (MUL x y)) && buildcfg.GOARM.Version == 7 => (MULS x y a)
+(RSB (MUL x y) a) && buildcfg.GOARM.Version == 7 => (MULS x y a)
-(NEGF (MULF x y)) && buildcfg.GOARM >= 6 => (NMULF x y)
-(NEGD (MULD x y)) && buildcfg.GOARM >= 6 => (NMULD x y)
-(MULF (NEGF x) y) && buildcfg.GOARM >= 6 => (NMULF x y)
-(MULD (NEGD x) y) && buildcfg.GOARM >= 6 => (NMULD x y)
+(NEGF (MULF x y)) && buildcfg.GOARM.Version >= 6 => (NMULF x y)
+(NEGD (MULD x y)) && buildcfg.GOARM.Version >= 6 => (NMULD x y)
+(MULF (NEGF x) y) && buildcfg.GOARM.Version >= 6 => (NMULF x y)
+(MULD (NEGD x) y) && buildcfg.GOARM.Version >= 6 => (NMULD x y)
(NMULF (NEGF x) y) => (MULF x y)
(NMULD (NEGD x) y) => (MULD x y)
// the result will overwrite the addend, since they are in the same register
-(ADDF a (MULF x y)) && a.Uses == 1 && buildcfg.GOARM >= 6 => (MULAF a x y)
-(ADDF a (NMULF x y)) && a.Uses == 1 && buildcfg.GOARM >= 6 => (MULSF a x y)
-(ADDD a (MULD x y)) && a.Uses == 1 && buildcfg.GOARM >= 6 => (MULAD a x y)
-(ADDD a (NMULD x y)) && a.Uses == 1 && buildcfg.GOARM >= 6 => (MULSD a x y)
-(SUBF a (MULF x y)) && a.Uses == 1 && buildcfg.GOARM >= 6 => (MULSF a x y)
-(SUBF a (NMULF x y)) && a.Uses == 1 && buildcfg.GOARM >= 6 => (MULAF a x y)
-(SUBD a (MULD x y)) && a.Uses == 1 && buildcfg.GOARM >= 6 => (MULSD a x y)
-(SUBD a (NMULD x y)) && a.Uses == 1 && buildcfg.GOARM >= 6 => (MULAD a x y)
+(ADDF a (MULF x y)) && a.Uses == 1 && buildcfg.GOARM.Version >= 6 => (MULAF a x y)
+(ADDF a (NMULF x y)) && a.Uses == 1 && buildcfg.GOARM.Version >= 6 => (MULSF a x y)
+(ADDD a (MULD x y)) && a.Uses == 1 && buildcfg.GOARM.Version >= 6 => (MULAD a x y)
+(ADDD a (NMULD x y)) && a.Uses == 1 && buildcfg.GOARM.Version >= 6 => (MULSD a x y)
+(SUBF a (MULF x y)) && a.Uses == 1 && buildcfg.GOARM.Version >= 6 => (MULSF a x y)
+(SUBF a (NMULF x y)) && a.Uses == 1 && buildcfg.GOARM.Version >= 6 => (MULAF a x y)
+(SUBD a (MULD x y)) && a.Uses == 1 && buildcfg.GOARM.Version >= 6 => (MULSD a x y)
+(SUBD a (NMULD x y)) && a.Uses == 1 && buildcfg.GOARM.Version >= 6 => (MULAD a x y)
(AND x (MVN y)) => (BIC x y)
@@ -1259,8 +1259,8 @@
(CMPD x (MOVDconst [0])) => (CMPD0 x)
// bit extraction
-(SRAconst (SLLconst x [c]) [d]) && buildcfg.GOARM==7 && uint64(d)>=uint64(c) && uint64(d)<=31 => (BFX [(d-c)|(32-d)<<8] x)
-(SRLconst (SLLconst x [c]) [d]) && buildcfg.GOARM==7 && uint64(d)>=uint64(c) && uint64(d)<=31 => (BFXU [(d-c)|(32-d)<<8] x)
+(SRAconst (SLLconst x [c]) [d]) && buildcfg.GOARM.Version==7 && uint64(d)>=uint64(c) && uint64(d)<=31 => (BFX [(d-c)|(32-d)<<8] x)
+(SRLconst (SLLconst x [c]) [d]) && buildcfg.GOARM.Version==7 && uint64(d)>=uint64(c) && uint64(d)<=31 => (BFXU [(d-c)|(32-d)<<8] x)
// comparison simplification
((EQ|NE) (CMP x (RSBconst [0] y))) => ((EQ|NE) (CMN x y)) // sense of carry bit not preserved; see also #50854
diff --git a/src/cmd/compile/internal/ssa/_gen/LOONG64.rules b/src/cmd/compile/internal/ssa/_gen/LOONG64.rules
index 4a47c4cd47..2af9519113 100644
--- a/src/cmd/compile/internal/ssa/_gen/LOONG64.rules
+++ b/src/cmd/compile/internal/ssa/_gen/LOONG64.rules
@@ -416,7 +416,7 @@
(GetCallerSP ...) => (LoweredGetCallerSP ...)
(GetCallerPC ...) => (LoweredGetCallerPC ...)
-(If cond yes no) => (NE cond yes no)
+(If cond yes no) => (NE (MOVBUreg cond) yes no)
// Write barrier.
(WB ...) => (LoweredWB ...)
@@ -450,71 +450,37 @@
(EQ (SGTconst [0] x) yes no) => (GEZ x yes no)
(NE (SGT x (MOVVconst [0])) yes no) => (GTZ x yes no)
(EQ (SGT x (MOVVconst [0])) yes no) => (LEZ x yes no)
+(MOVBUreg x:((SGT|SGTU) _ _)) => x
// fold offset into address
(ADDVconst [off1] (MOVVaddr [off2] {sym} ptr)) && is32Bit(off1+int64(off2)) => (MOVVaddr [int32(off1)+int32(off2)] {sym} ptr)
// fold address into load/store
-(MOVBload [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVBload [off1+int32(off2)] {sym} ptr mem)
-(MOVBUload [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVBUload [off1+int32(off2)] {sym} ptr mem)
-(MOVHload [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVHload [off1+int32(off2)] {sym} ptr mem)
-(MOVHUload [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVHUload [off1+int32(off2)] {sym} ptr mem)
-(MOVWload [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVWload [off1+int32(off2)] {sym} ptr mem)
-(MOVWUload [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVWUload [off1+int32(off2)] {sym} ptr mem)
-(MOVVload [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVVload [off1+int32(off2)] {sym} ptr mem)
-(MOVFload [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVFload [off1+int32(off2)] {sym} ptr mem)
-(MOVDload [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVDload [off1+int32(off2)] {sym} ptr mem)
+// Do not fold global variable access in -dynlink mode, where it will be rewritten
+// to use the GOT via REGTMP, which currently cannot handle large offset.
+(MOV(B|BU|H|HU|W|WU|V|F|D)load [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2)
+ && (ptr.Op != OpSB || !config.ctxt.Flag_dynlink) =>
+ (MOV(B|BU|H|HU|W|WU|V|F|D)load [off1+int32(off2)] {sym} ptr mem)
-(MOVBstore [off1] {sym} (ADDVconst [off2] ptr) val mem) && is32Bit(int64(off1)+off2) => (MOVBstore [off1+int32(off2)] {sym} ptr val mem)
-(MOVHstore [off1] {sym} (ADDVconst [off2] ptr) val mem) && is32Bit(int64(off1)+off2) => (MOVHstore [off1+int32(off2)] {sym} ptr val mem)
-(MOVWstore [off1] {sym} (ADDVconst [off2] ptr) val mem) && is32Bit(int64(off1)+off2) => (MOVWstore [off1+int32(off2)] {sym} ptr val mem)
-(MOVVstore [off1] {sym} (ADDVconst [off2] ptr) val mem) && is32Bit(int64(off1)+off2) => (MOVVstore [off1+int32(off2)] {sym} ptr val mem)
-(MOVFstore [off1] {sym} (ADDVconst [off2] ptr) val mem) && is32Bit(int64(off1)+off2) => (MOVFstore [off1+int32(off2)] {sym} ptr val mem)
-(MOVDstore [off1] {sym} (ADDVconst [off2] ptr) val mem) && is32Bit(int64(off1)+off2) => (MOVDstore [off1+int32(off2)] {sym} ptr val mem)
-(MOVBstorezero [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVBstorezero [off1+int32(off2)] {sym} ptr mem)
-(MOVHstorezero [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVHstorezero [off1+int32(off2)] {sym} ptr mem)
-(MOVWstorezero [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVWstorezero [off1+int32(off2)] {sym} ptr mem)
-(MOVVstorezero [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2) => (MOVVstorezero [off1+int32(off2)] {sym} ptr mem)
+(MOV(B|H|W|V|F|D)store [off1] {sym} (ADDVconst [off2] ptr) val mem) && is32Bit(int64(off1)+off2)
+ && (ptr.Op != OpSB || !config.ctxt.Flag_dynlink) =>
+ (MOV(B|H|W|V|F|D)store [off1+int32(off2)] {sym} ptr val mem)
-(MOVBload [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVBload [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVBUload [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVBUload [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVHload [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVHload [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVHUload [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVHUload [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVWload [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVWload [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVWUload [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVWUload [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVVload [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVVload [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVFload [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVFload [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVDload [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVDload [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
+(MOV(B|H|W|V)storezero [off1] {sym} (ADDVconst [off2] ptr) mem) && is32Bit(int64(off1)+off2)
+ && (ptr.Op != OpSB || !config.ctxt.Flag_dynlink) =>
+ (MOV(B|H|W|V)storezero [off1+int32(off2)] {sym} ptr mem)
-(MOVBstore [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) val mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVBstore [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr val mem)
-(MOVHstore [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) val mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVHstore [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr val mem)
-(MOVWstore [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) val mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVWstore [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr val mem)
-(MOVVstore [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) val mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVVstore [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr val mem)
-(MOVFstore [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) val mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVFstore [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr val mem)
-(MOVDstore [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) val mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVDstore [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr val mem)
-(MOVBstorezero [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVBstorezero [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVHstorezero [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVHstorezero [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVWstorezero [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVWstorezero [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
-(MOVVstorezero [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2) && is32Bit(int64(off1)+int64(off2)) =>
- (MOVVstorezero [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
+(MOV(B|BU|H|HU|W|WU|V|F|D)load [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2)
+ && is32Bit(int64(off1)+int64(off2)) && (ptr.Op != OpSB || !config.ctxt.Flag_dynlink) =>
+ (MOV(B|BU|H|HU|W|WU|V|F|D)load [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
+
+(MOV(B|H|W|V|F|D)store [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) val mem) && canMergeSym(sym1,sym2)
+ && is32Bit(int64(off1)+int64(off2)) && (ptr.Op != OpSB || !config.ctxt.Flag_dynlink) =>
+ (MOV(B|H|W|V|F|D)store [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr val mem)
+
+(MOV(B|H|W|V)storezero [off1] {sym1} (MOVVaddr [off2] {sym2} ptr) mem) && canMergeSym(sym1,sym2)
+ && is32Bit(int64(off1)+int64(off2)) && (ptr.Op != OpSB || !config.ctxt.Flag_dynlink) =>
+ (MOV(B|H|W|V)storezero [off1+int32(off2)] {mergeSym(sym1,sym2)} ptr mem)
(LoweredAtomicStore(32|64) ptr (MOVVconst [0]) mem) => (LoweredAtomicStorezero(32|64) ptr mem)
(LoweredAtomicAdd32 ptr (MOVVconst [c]) mem) && is32Bit(c) => (LoweredAtomicAddconst32 [int32(c)] ptr mem)
diff --git a/src/cmd/compile/internal/ssa/_gen/LOONG64Ops.go b/src/cmd/compile/internal/ssa/_gen/LOONG64Ops.go
index 3442fc8d7c..3fbf5be499 100644
--- a/src/cmd/compile/internal/ssa/_gen/LOONG64Ops.go
+++ b/src/cmd/compile/internal/ssa/_gen/LOONG64Ops.go
@@ -123,17 +123,17 @@ func init() {
// Common individual register masks
var (
- gp = buildReg("R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R23 R24 R25 R26 R27 R28 R29 R31") // R1 is LR, R2 is thread pointer, R3 is stack pointer, R21-unused, R22 is g, R30 is REGTMP
+ gp = buildReg("R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19 R20 R21 R23 R24 R25 R26 R27 R28 R29 R31") // R1 is LR, R2 is thread pointer, R3 is stack pointer, R22 is g, R30 is REGTMP
gpg = gp | buildReg("g")
gpsp = gp | buildReg("SP")
gpspg = gpg | buildReg("SP")
gpspsbg = gpspg | buildReg("SB")
fp = buildReg("F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 F15 F16 F17 F18 F19 F20 F21 F22 F23 F24 F25 F26 F27 F28 F29 F30 F31")
callerSave = gp | fp | buildReg("g") // runtime.setg (and anything calling it) may clobber g
- r1 = buildReg("R19")
- r2 = buildReg("R18")
- r3 = buildReg("R17")
- r4 = buildReg("R4")
+ r1 = buildReg("R20")
+ r2 = buildReg("R21")
+ r3 = buildReg("R23")
+ r4 = buildReg("R24")
)
// Common regInfo
var (
@@ -273,32 +273,32 @@ func init() {
{name: "MOVDF", argLength: 1, reg: fp11, asm: "MOVDF"}, // float64 -> float32
// function calls
- {name: "CALLstatic", argLength: 1, reg: regInfo{clobbers: callerSave}, aux: "CallOff", clobberFlags: true, call: true}, // call static function aux.(*obj.LSym). arg0=mem, auxint=argsize, returns mem
- {name: "CALLtail", argLength: 1, reg: regInfo{clobbers: callerSave}, aux: "CallOff", clobberFlags: true, call: true, tailCall: true}, // tail call static function aux.(*obj.LSym). arg0=mem, auxint=argsize, returns mem
- {name: "CALLclosure", argLength: 3, reg: regInfo{inputs: []regMask{gpsp, buildReg("R29"), 0}, clobbers: callerSave}, aux: "CallOff", clobberFlags: true, call: true}, // call function via closure. arg0=codeptr, arg1=closure, arg2=mem, auxint=argsize, returns mem
- {name: "CALLinter", argLength: 2, reg: regInfo{inputs: []regMask{gp}, clobbers: callerSave}, aux: "CallOff", clobberFlags: true, call: true}, // call fn by pointer. arg0=codeptr, arg1=mem, auxint=argsize, returns mem
+ {name: "CALLstatic", argLength: -1, reg: regInfo{clobbers: callerSave}, aux: "CallOff", clobberFlags: true, call: true}, // call static function aux.(*obj.LSym). last arg=mem, auxint=argsize, returns mem
+ {name: "CALLtail", argLength: -1, reg: regInfo{clobbers: callerSave}, aux: "CallOff", clobberFlags: true, call: true, tailCall: true}, // tail call static function aux.(*obj.LSym). last arg=mem, auxint=argsize, returns mem
+ {name: "CALLclosure", argLength: -1, reg: regInfo{inputs: []regMask{gpsp, buildReg("R29"), 0}, clobbers: callerSave}, aux: "CallOff", clobberFlags: true, call: true}, // call function via closure. arg0=codeptr, arg1=closure, last arg=mem, auxint=argsize, returns mem
+ {name: "CALLinter", argLength: -1, reg: regInfo{inputs: []regMask{gp}, clobbers: callerSave}, aux: "CallOff", clobberFlags: true, call: true}, // call fn by pointer. arg0=codeptr, last arg=mem, auxint=argsize, returns mem
// duffzero
// arg0 = address of memory to zero
// arg1 = mem
// auxint = offset into duffzero code to start executing
// returns mem
- // R19 aka loong64.REGRT1 changed as side effect
+ // R20 aka loong64.REGRT1 changed as side effect
{
name: "DUFFZERO",
aux: "Int64",
argLength: 2,
reg: regInfo{
- inputs: []regMask{buildReg("R19")},
- clobbers: buildReg("R19 R1"),
+ inputs: []regMask{buildReg("R20")},
+ clobbers: buildReg("R20 R1"),
},
typ: "Mem",
faultOnNilArg0: true,
},
// duffcopy
- // arg0 = address of dst memory (in R20, changed as side effect) REGRT2
- // arg1 = address of src memory (in R19, changed as side effect) REGRT1
+ // arg0 = address of dst memory (in R21, changed as side effect)
+ // arg1 = address of src memory (in R20, changed as side effect)
// arg2 = mem
// auxint = offset into duffcopy code to start executing
// returns mem
@@ -307,8 +307,8 @@ func init() {
aux: "Int64",
argLength: 3,
reg: regInfo{
- inputs: []regMask{buildReg("R20"), buildReg("R19")},
- clobbers: buildReg("R19 R20 R1"),
+ inputs: []regMask{buildReg("R21"), buildReg("R20")},
+ clobbers: buildReg("R20 R21 R1"),
},
typ: "Mem",
faultOnNilArg0: true,
@@ -316,45 +316,45 @@ func init() {
},
// large or unaligned zeroing
- // arg0 = address of memory to zero (in R19, changed as side effect)
+ // arg0 = address of memory to zero (in R20, changed as side effect)
// arg1 = address of the last element to zero
// arg2 = mem
// auxint = alignment
// returns mem
- // MOVx R0, (R19)
- // ADDV $sz, R19
- // BGEU Rarg1, R19, -2(PC)
+ // MOVx R0, (R20)
+ // ADDV $sz, R20
+ // BGEU Rarg1, R20, -2(PC)
{
name: "LoweredZero",
aux: "Int64",
argLength: 3,
reg: regInfo{
- inputs: []regMask{buildReg("R19"), gp},
- clobbers: buildReg("R19"),
+ inputs: []regMask{buildReg("R20"), gp},
+ clobbers: buildReg("R20"),
},
typ: "Mem",
faultOnNilArg0: true,
},
// large or unaligned move
- // arg0 = address of dst memory (in R20, changed as side effect)
- // arg1 = address of src memory (in R19, changed as side effect)
+ // arg0 = address of dst memory (in R21, changed as side effect)
+ // arg1 = address of src memory (in R20, changed as side effect)
// arg2 = address of the last element of src
// arg3 = mem
// auxint = alignment
// returns mem
- // MOVx (R19), Rtmp
- // MOVx Rtmp, (R20)
- // ADDV $sz, R19
+ // MOVx (R20), Rtmp
+ // MOVx Rtmp, (R21)
// ADDV $sz, R20
- // BGEU Rarg2, R19, -4(PC)
+ // ADDV $sz, R21
+ // BGEU Rarg2, R20, -4(PC)
{
name: "LoweredMove",
aux: "Int64",
argLength: 4,
reg: regInfo{
- inputs: []regMask{buildReg("R20"), buildReg("R19"), gp},
- clobbers: buildReg("R19 R20"),
+ inputs: []regMask{buildReg("R21"), buildReg("R20"), gp},
+ clobbers: buildReg("R20 R21"),
},
typ: "Mem",
faultOnNilArg0: true,
@@ -476,8 +476,8 @@ func init() {
blocks: blocks,
regnames: regNamesLOONG64,
// TODO: support register ABI on loong64
- ParamIntRegNames: "R4 R5 R6 R7 R8 R9 R10 R11",
- ParamFloatRegNames: "F0 F1 F2 F3 F4 F5 F6 F7",
+ ParamIntRegNames: "R4 R5 R6 R7 R8 R9 R10 R11 R12 R13 R14 R15 R16 R17 R18 R19",
+ ParamFloatRegNames: "F0 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 F15",
gpregmask: gp,
fpregmask: fp,
framepointerreg: -1, // not used
diff --git a/src/cmd/compile/internal/ssa/_gen/RISCV64.rules b/src/cmd/compile/internal/ssa/_gen/RISCV64.rules
index 9afe5995ae..135d70bc47 100644
--- a/src/cmd/compile/internal/ssa/_gen/RISCV64.rules
+++ b/src/cmd/compile/internal/ssa/_gen/RISCV64.rules
@@ -72,6 +72,9 @@
(FMA ...) => (FMADDD ...)
+(Min(64|32)F ...) => (LoweredFMIN(D|S) ...)
+(Max(64|32)F ...) => (LoweredFMAX(D|S) ...)
+
// Sign and zero extension.
(SignExt8to16 ...) => (MOVBreg ...)
@@ -153,27 +156,27 @@
// SRL only considers the bottom 6 bits of y, similarly SRLW only considers the
// bottom 5 bits of y. Ensure that the result is always zero if the shift exceeds
// the maximum value. See Lsh above for a detailed description.
-(Rsh8Ux8 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt8to64 x) y) (Neg8 (SLTIU [64] (ZeroExt8to64 y))))
-(Rsh8Ux16 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt8to64 x) y) (Neg8 (SLTIU [64] (ZeroExt16to64 y))))
-(Rsh8Ux32 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt8to64 x) y) (Neg8 (SLTIU [64] (ZeroExt32to64 y))))
-(Rsh8Ux64 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt8to64 x) y) (Neg8 (SLTIU [64] y)))
-(Rsh16Ux8 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt16to64 x) y) (Neg16 (SLTIU [64] (ZeroExt8to64 y))))
-(Rsh16Ux16 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt16to64 x) y) (Neg16 (SLTIU [64] (ZeroExt16to64 y))))
-(Rsh16Ux32 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt16to64 x) y) (Neg16 (SLTIU [64] (ZeroExt32to64 y))))
-(Rsh16Ux64 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt16to64 x) y) (Neg16 (SLTIU [64] y)))
-(Rsh32Ux8 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt32to64 x) y) (Neg32 (SLTIU [32] (ZeroExt8to64 y))))
-(Rsh32Ux16 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt32to64 x) y) (Neg32 (SLTIU [32] (ZeroExt16to64 y))))
-(Rsh32Ux32 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt32to64 x) y) (Neg32 (SLTIU [32] (ZeroExt32to64 y))))
-(Rsh32Ux64 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt32to64 x) y) (Neg32 (SLTIU [32] y)))
-(Rsh64Ux8 x y) && !shiftIsBounded(v) => (AND (SRL x y) (Neg64 (SLTIU [64] (ZeroExt8to64 y))))
-(Rsh64Ux16 x y) && !shiftIsBounded(v) => (AND (SRL x y) (Neg64 (SLTIU [64] (ZeroExt16to64 y))))
-(Rsh64Ux32 x y) && !shiftIsBounded(v) => (AND (SRL x y) (Neg64 (SLTIU [64] (ZeroExt32to64 y))))
-(Rsh64Ux64 x y) && !shiftIsBounded(v) => (AND (SRL x y) (Neg64 (SLTIU [64] y)))
+(Rsh8Ux8 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt8to64 x) y) (Neg8 (SLTIU [64] (ZeroExt8to64 y))))
+(Rsh8Ux16 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt8to64 x) y) (Neg8 (SLTIU [64] (ZeroExt16to64 y))))
+(Rsh8Ux32 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt8to64 x) y) (Neg8 (SLTIU [64] (ZeroExt32to64 y))))
+(Rsh8Ux64 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt8to64 x) y) (Neg8 (SLTIU [64] y)))
+(Rsh16Ux8 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt16to64 x) y) (Neg16 (SLTIU [64] (ZeroExt8to64 y))))
+(Rsh16Ux16 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt16to64 x) y) (Neg16 (SLTIU [64] (ZeroExt16to64 y))))
+(Rsh16Ux32 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt16to64 x) y) (Neg16 (SLTIU [64] (ZeroExt32to64 y))))
+(Rsh16Ux64 x y) && !shiftIsBounded(v) => (AND (SRL (ZeroExt16to64 x) y) (Neg16 (SLTIU [64] y)))
+(Rsh32Ux8 x y) && !shiftIsBounded(v) => (AND (SRLW x y) (Neg32 (SLTIU [32] (ZeroExt8to64 y))))
+(Rsh32Ux16 x y) && !shiftIsBounded(v) => (AND (SRLW x y) (Neg32 (SLTIU [32] (ZeroExt16to64 y))))
+(Rsh32Ux32 x y) && !shiftIsBounded(v) => (AND (SRLW x y) (Neg32 (SLTIU [32] (ZeroExt32to64 y))))
+(Rsh32Ux64 x y) && !shiftIsBounded(v) => (AND (SRLW x y) (Neg32 (SLTIU [32] y)))
+(Rsh64Ux8 x y) && !shiftIsBounded(v) => (AND (SRL x y) (Neg64 (SLTIU [64] (ZeroExt8to64 y))))
+(Rsh64Ux16 x y) && !shiftIsBounded(v) => (AND (SRL x y) (Neg64 (SLTIU [64] (ZeroExt16to64 y))))
+(Rsh64Ux32 x y) && !shiftIsBounded(v) => (AND (SRL x y) (Neg64 (SLTIU [64] (ZeroExt32to64 y))))
+(Rsh64Ux64 x y) && !shiftIsBounded(v) => (AND (SRL x y) (Neg64 (SLTIU [64] y)))
-(Rsh8Ux(64|32|16|8) x y) && shiftIsBounded(v) => (SRL (ZeroExt8to64 x) y)
-(Rsh16Ux(64|32|16|8) x y) && shiftIsBounded(v) => (SRL (ZeroExt16to64 x) y)
-(Rsh32Ux(64|32|16|8) x y) && shiftIsBounded(v) => (SRL (ZeroExt32to64 x) y)
-(Rsh64Ux(64|32|16|8) x y) && shiftIsBounded(v) => (SRL x y)
+(Rsh8Ux(64|32|16|8) x y) && shiftIsBounded(v) => (SRL (ZeroExt8to64 x) y)
+(Rsh16Ux(64|32|16|8) x y) && shiftIsBounded(v) => (SRL (ZeroExt16to64 x) y)
+(Rsh32Ux(64|32|16|8) x y) && shiftIsBounded(v) => (SRLW x y)
+(Rsh64Ux(64|32|16|8) x y) && shiftIsBounded(v) => (SRL x y)
// SRA only considers the bottom 6 bits of y, similarly SRAW only considers the
// bottom 5 bits. If y is greater than the maximum value (either 63 or 31
@@ -188,33 +191,33 @@
//
// We don't need to sign-extend the OR result, as it will be at minimum 8 bits,
// more than the 5 or 6 bits SRAW and SRA care about.
-(Rsh8x8 x y) && !shiftIsBounded(v) => (SRA (SignExt8to64 x) (OR y (ADDI [-1] (SLTIU [64] (ZeroExt8to64 y)))))
-(Rsh8x16 x y) && !shiftIsBounded(v) => (SRA (SignExt8to64 x) (OR y (ADDI [-1] (SLTIU [64] (ZeroExt16to64 y)))))
-(Rsh8x32 x y) && !shiftIsBounded(v) => (SRA (SignExt8to64 x) (OR y (ADDI [-1] (SLTIU [64] (ZeroExt32to64 y)))))
-(Rsh8x64 x y) && !shiftIsBounded(v) => (SRA (SignExt8to64 x) (OR y (ADDI [-1] (SLTIU [64] y))))
-(Rsh16x8 x y) && !shiftIsBounded(v) => (SRA (SignExt16to64 x) (OR y (ADDI [-1] (SLTIU [64] (ZeroExt8to64 y)))))
-(Rsh16x16 x y) && !shiftIsBounded(v) => (SRA (SignExt16to64 x) (OR y (ADDI [-1] (SLTIU [64] (ZeroExt16to64 y)))))
-(Rsh16x32 x y) && !shiftIsBounded(v) => (SRA (SignExt16to64 x) (OR y (ADDI [-1] (SLTIU [64] (ZeroExt32to64 y)))))
-(Rsh16x64 x y) && !shiftIsBounded(v) => (SRA (SignExt16to64 x) (OR y (ADDI [-1] (SLTIU [64] y))))
-(Rsh32x8 x y) && !shiftIsBounded(v) => (SRA (SignExt32to64 x) (OR y (ADDI [-1] (SLTIU [32] (ZeroExt8to64 y)))))
-(Rsh32x16 x y) && !shiftIsBounded(v) => (SRA (SignExt32to64 x) (OR y (ADDI [-1] (SLTIU [32] (ZeroExt16to64 y)))))
-(Rsh32x32 x y) && !shiftIsBounded(v) => (SRA (SignExt32to64 x) (OR y (ADDI [-1] (SLTIU [32] (ZeroExt32to64 y)))))
-(Rsh32x64 x y) && !shiftIsBounded(v) => (SRA (SignExt32to64 x) (OR y (ADDI [-1] (SLTIU [32] y))))
-(Rsh64x8