108 Commits

Author SHA1 Message Date
Robert Griesemer
0775730180 cmd/compile/internal/syntax: better errors for syntax errors in lists
For syntax errors in various (syntactic) lists, instead of reporting
a set of "expected" tokens (which may be incomplete), provide context
and mention "possibly missing" tokens. The result is a friendlier and
more accurate error message.

Fixes #49205.

Change-Id: I38ae7bf62febfe790075e62deb33ec8c17d64476
Reviewed-on: https://go-review.googlesource.com/c/go/+/396914
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2022-03-31 00:26:58 +00:00
Robert Griesemer
a7e76b8e80 cmd/compile/internal/syntax: remove code dealing with multiple method names
When parsing method declarations in an interface, the parser has
for historic reasons gracefully handled a list of method names with
a single (common) signature, and then reported an error. For example

        interface {
                m1, m2, m3 (x int)
        }

This code originally came from the very first parser for Go which
initially permitted such declarations (or at least assumed that
people would write such declarations). Nobody is doing this at this
point, so there's no need for being extra careful here. Remove the
respective code and adjust the corresponding test.

Change-Id: If6f9b398bbc9e425dcd4328a80d8bf77c37fe8b6
Reviewed-on: https://go-review.googlesource.com/c/go/+/396654
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2022-03-30 18:02:38 +00:00
Robert Griesemer
bf408e741c cmd/compile/internal/syntax: remove generic-specific parse modes
Generics have landed; we cannot revert the syntax anymore. Remove
ability to choose between non-generic and generic code. Also remove
mode to enable method type parameters. Adjust code accordingly.

Also remove a couple of TODOs that are not relevant anymore.

Remove tests from types2 which were focussed on method type parameters,
make types2 and go/types tests match up where there was a difference in
this regard.

Change-Id: I989bdcb19eea7414214af739187fa013a044295d
Reviewed-on: https://go-review.googlesource.com/c/go/+/396634
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2022-03-30 18:02:31 +00:00
Robert Griesemer
042548b1fd cmd/compile: report type parameter error for methods only once
Move switch to enable method type parameters entirely
to the parser, by adding the mode AllowMethodTypeParams.
Ensure that the error messages are consistent.
Remove unnecessary code in the type checker.

Fixes #50317.

Change-Id: I4f3958722400bdb919efa4c494b85cf62f4002bb
Reviewed-on: https://go-review.googlesource.com/c/go/+/376054
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-01-06 23:39:43 +00:00
Robert Griesemer
33a1a93a92 cmd/compile/internal/syntax: fix parsing of type parameter lists
The parser cannot distinguish a type parameter list of the form

[P *T ] or
[P (T)]

where T is not a type literal from an array length specification
P*T (product) or P(T) (constant-valued function call) and thus
interprets these forms as the start of array types.

This ambiguity must be resolved explicitly by placing *T inside
an interface, adding a trailing comma, or by leaving parentheses
away where possible.

This CL adjusts the parser such that these forms are
interpreted as (the beginning) of type parameter lists
if the token after P*T or P(T) is a comma, or if T is
a type literal.

This CL also adjusts the printer to print a comma if
necessary to avoid this ambiguity, and adds additional
printer tests.

Fixes #49482

Change-Id: I36328e2a7d9439c39ba0349837c445542549e84e
Reviewed-on: https://go-review.googlesource.com/c/go/+/370774
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2021-12-17 00:14:49 +00:00
Robert Griesemer
a419f2f5c3 cmd/compile/internal/syntax: better error message when type parameters are not permitted
Fixes #48382.

Change-Id: I215896a4429839c41c9136b6922b1b748ed47734
Reviewed-on: https://go-review.googlesource.com/c/go/+/361259
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-11-04 00:29:42 +00:00
Robert Griesemer
0bef30d28a cmd/compile/internal/syntax: fix parsing of array or slice constraint types
This is a port of the idea used in CL 359134 from go/parser to syntax,
with adjustments due to the slightly different structure of the two
parsers, and some refactoring to simplify the logic.

Fixes #49175.

Change-Id: Ib4955bde708f2b08345f35523e6094c03ab3076c
Reviewed-on: https://go-review.googlesource.com/c/go/+/360135
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-10-31 02:24:29 +00:00
Robert Griesemer
bb49eb3e6a cmd/compile/internal/syntax: fix constraint literal parsing for generic functions
Fixes #49174.

Change-Id: I943c370f7abd5f50a541e682f130b3526c3b5bdb
Reviewed-on: https://go-review.googlesource.com/c/go/+/359014
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-10-27 18:08:49 +00:00
Robert Griesemer
276fb279d1 cmd/compile/internal/syntax: remove AllowTypeSets mode
The respective issue has been accepted, so we can always
accept constraint literals with omitted interfaces.

For #48424.

Change-Id: Ia3d325401252a5a22d5ffa98d2ae6af73178dec0
Reviewed-on: https://go-review.googlesource.com/c/go/+/355709
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-10-14 01:51:22 +00:00
Robert Griesemer
f05c67f182 cmd/compile/internal/syntax, types2: remove ability to handle type lists
The type set notation has been accepted a while ago.
We're not going back to supporting the original
type list notation. Remove support for it in the
parser and type checker.

Change-Id: I860651f80b89fa43a3a5a2a02cf823ec0dae583c
Reviewed-on: https://go-review.googlesource.com/c/go/+/354131
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-10-06 19:44:05 +00:00
Robert Griesemer
5279e534b5 cmd/compile/internal/syntax: allow eliding interface in constraint literals
This CL permits an arbitrary type as well as the type sets  ~T and A|B
in constraint position, without the need of a surrrounding interface.
For instance, the type parameter list

	[P interface{ ~map[K]V }, K comparable, V interface{ ~string }]

may be written as

	[P ~map[K]V, K comparable, V ~string]

The feature must be enabled explicitly with the AllowTypeSets mode
and is only available if AllowGenerics is set as well.

For #48424.

Change-Id: Ic70bb97a49ff75e67e040853eac10e6aed0fef1a
Reviewed-on: https://go-review.googlesource.com/c/go/+/353133
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-10-01 17:18:20 +00:00
Robert Griesemer
da1aa65053 cmd/compile/internal/syntax: correct follow token for type parameter lists
When parsing a type parameter declaration, parts of the code still
expected a ) as closing token. Use the correct follow token ) or ]
depending on parameter list kind.

Also, consistently use tokstring (not tok.String()) for user-facing
(error) messages.

Follow-up on comment in CL 348730.

For #43527.

Change-Id: Ib1d4feb526771a1668a54c3bb7a671f6c8a65940
Reviewed-on: https://go-review.googlesource.com/c/go/+/348742
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-09-10 16:08:21 +00:00
Robert Griesemer
73483df406 cmd/compile/internal/syntax: better error message for missing type constraint
For #43527.

Change-Id: I8c706e68572286d5675383eb2dfd75b5618b646b
Reviewed-on: https://go-review.googlesource.com/c/go/+/348730
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-09-09 22:17:54 +00:00
Robert Griesemer
df4c625d88 cmd/compile: disable type list syntax for the compiler
Add (temporary) syntax.AllowTypeLists mode to control the
acceptance of type lists; the compiler doesn't set it,
but existing syntax and types2 tests do so that the code
remains exercised while it exists.

Adjust various tests to use the type set notation.

Change-Id: I798e607912552db6bfe38a7cd4324b74c6bf4d95
Reviewed-on: https://go-review.googlesource.com/c/go/+/347249
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-09-02 01:06:17 +00:00
Robert Griesemer
2c60a99f72 cmd/compile/internal/syntax: make valid type parameter list in presence of errors
Make sure the parser fills in names and types for type parameter
lists, even in the case of errors.

While at it, adjust some of the test functions to accept generic
code and report all syntax errors.

Added offending source as test for types2.

Fixes #47996.

Change-Id: I449bcf5e2cb80fa2a24cdd3945f484bfca218a06
Reviewed-on: https://go-review.googlesource.com/c/go/+/345476
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2021-08-27 05:13:44 +00:00
Robert Griesemer
4158e88f64 cmd/compile/internal/syntax: fix position of type parameter field
Change-Id: I8bca01b935301e7bd4efa55ed21921dbf31a75b9
Reviewed-on: https://go-review.googlesource.com/c/go/+/344575
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2021-08-25 23:43:41 +00:00
Matthew Dempsky
ab9aaf46ee cmd/compile/internal/syntax: add PosBase.Trimmed
With types2, some syntax.PosBases need to be constructed from export
data, which must only contain "trimmed" filenames (i.e., that they've
already been made absolute and undergone -trimpath processing).
However, it's not safe to apply trimming to a filename multiple times,
and in general we can't distinguish trimmed from untrimmed filenames.

This CL resolves this by adding a PosBase.Trimmed boolean so we can
distinguish whether the associated filename has been trimmed yet. This
is a bit hacky, but is the least bad solution I've come up with so
far.

This unblocks enabling -G=3 by default.

Change-Id: I7383becfb704680a36f7603e3246af38b21f100b
Reviewed-on: https://go-review.googlesource.com/c/go/+/343731
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Matthew Dempsky <mdempsky@google.com>
Trust: Dan Scales <danscales@google.com>
Reviewed-by: Robert Griesemer <gri@golang.org>
2021-08-20 19:44:02 +00:00
Robert Griesemer
717894cf80 cmd/compile/internal/types2: better error message for index syntax error (follow-up)
For #47704.

Change-Id: I09e6f638df0cd456a20a3b68ab55c47bb5b1f555
Reviewed-on: https://go-review.googlesource.com/c/go/+/342370
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
2021-08-15 19:56:18 +00:00
Robert Griesemer
6ed9463133 cmd/compile/internal/syntax: better error message for index syntax error
Fixes #47704.

Change-Id: I1de9fd00baaa4b534c23f011ade54120f5153a9d
Reviewed-on: https://go-review.googlesource.com/c/go/+/342369
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-08-15 19:08:56 +00:00
Robert Griesemer
3a9fd99849 [dev.typeparams] cmd/compile/internal/syntax: cleanup panic calls
End-users are not expected to deal with the details of panics,
so providing extra information such as an "internal error" prefix
is not helpful.

Matches the types2 changes made in https://golang.org/cl/339969 .

Change-Id: Icb34a9daab981a84f41f8ae7ae5dc1b85b2d2c81
Reviewed-on: https://go-review.googlesource.com/c/go/+/339904
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-08-06 20:34:48 +00:00
Robert Griesemer
692399fbaa [dev.typeparams] cmd/compile/internal/syntax: not all index expressions can be instantiated types
An index expression followed by an opening "{" may indicate
a composite literal but only if the index expression can be
a type. Exclude cases where the index expression cannot be
a type (e.g. s[0], a[i+j], etc.).

This leads to a better error message in code that is erroneous.

Fixes #46558.

Change-Id: Ida9291ca30683c211812dfb95abe4969f44c474f
Reviewed-on: https://go-review.googlesource.com/c/go/+/325009
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-06-05 01:52:24 +00:00
Robert Griesemer
81b22480cf [dev.typeparams] cmd/compile/internal/syntax: accept embedded type literals
The parser accepted embedded elements but the first term
of an element had to be a ~-term or a type name. This CL
fixes that.

Change-Id: I013b6cdc5963fb228867ca6597f9139db2be7ec5
Reviewed-on: https://go-review.googlesource.com/c/go/+/321109
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-05-19 04:19:24 +00:00
Robert Griesemer
ea65a12f89 cmd/compile/internal/types2: catch unexpected expression lists
This is a modified port of the https://golang.org/cl/313909
change for go/types.

- add catch-all cases for unexpected expression lists
- add Checker.singleIndex function to check single indices
- better syntax error handling in parser for invalid type
  instantiations that are missing a type argument

Change-Id: I6f0f396d637ad66b79f803d886fdc20ee55a98b3
Reviewed-on: https://go-review.googlesource.com/c/go/+/314409
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-04-28 18:50:41 +00:00
Robert Griesemer
a63ff398d5 cmd/compile/internal/syntax: fix error message for ... without type
Only complain about missing type; leave it to type-checking
to decide whether "..." is permitted in the first place.

Fixes #43674.

Change-Id: Icbc8f084e364fe3ac16076406a134354219c08d0
Reviewed-on: https://go-review.googlesource.com/c/go/+/310209
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-04-15 19:41:38 +00:00
Robert Griesemer
4638545d85 cmd/compile/internal/syntax: accept "~" and "|" interface elements
Type lists continue to be accepted as before.

While at it, print missing filenames in error tests
(which uses an ad-hoc position representation).

Change-Id: I933b3acbc9cf1985ad8f70f6b206e3a1dbd64d1e
Reviewed-on: https://go-review.googlesource.com/c/go/+/307371
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2021-04-10 19:02:03 +00:00
Robert Griesemer
18bd7aa625 [dev.typeparams] cmd/compile: use nil instead of syntax.ImplicitOne
Represent x++/-- as x +=/-= with the RHS of the assignment being nil
rather than syntax.ImplicitOne.

Dependent code already had to check for syntax.ImplicitOne, but
then shared some existing code for regular assignment operations.
Now always handle this case fully explicit, which simplifies the
code.

Change-Id: I28c7918153c27cbbf97b041d0c85ff027c58687c
Reviewed-on: https://go-review.googlesource.com/c/go/+/285172
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2021-01-21 06:55:47 +00:00
Robert Griesemer
7909d6ec28 [dev.typeparams] cmd/compile/internal/syntax: type parameters must always be named
Report an error otherwise.

Change-Id: Ia76ea03a3f26b13dd9bca49f7bd42101d1ff1f9e
Reviewed-on: https://go-review.googlesource.com/c/go/+/278475
Trust: Robert Griesemer <gri@golang.org>
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2020-12-16 18:41:36 +00:00
Robert Griesemer
6b18081d01 [dev.typeparams] cmd/compile/internal/types2: don't crash if import path is missing
In package syntax:
- fix parser appendGroup to not add nil entries
- non-string paths are syntax errors per the spec; report in parser
- document ImportDecl.Path invariants

In package types2:
- guard against absent paths

In package gc:
- guard against absent paths

Fixes #43190.

Change-Id: Ic6a06f6a96b7f519feaa1ceaf4376fc5ab0f0129
Reviewed-on: https://go-review.googlesource.com/c/go/+/278114
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2020-12-15 19:44:38 +00:00
Robert Griesemer
3a912f279f [dev.typeparams] cmd/compile/internal/syntax: export NewName and use it
Most syntax.Nodes are allocated in one place and there didn't
seem a need to provide factory methods - so as a matter of
API design, all nodes are "naked", without any constructors.

However, Name nodes are frequently used/replaced and also
are created as helper nodes in clients (types2). Make an
exception and export NewName.

Change-Id: I4b5c499d65bba74592dea68b0936c118b3edaca7
Reviewed-on: https://go-review.googlesource.com/c/go/+/277572
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
2020-12-14 21:41:25 +00:00
Robert Griesemer
73f529845c [dev.typeparams] cmd/compile/internal/syntax: always use IndexExpr node for type instantiation
Per @mdempsky's suggestion: Instead of representing a type instantiation T[P]
by an IndexExpr node, and a type instantiation with multiple type arguments
T[P1, P2] by a CallExpr node with special Brackets flag, always use an IndexExpr.
Use a ListExpr as index in the (less common) case of multiple type arguments.

This removes the need for the CallExpr.Brackets field and cleans up the parser
code around type instantiations.

Backport of syntax package changes from https://golang.org/cl/262020.

Change-Id: I32e8bc4eafac5b3ef2e7eb40fa8c790a5a905b69
Reviewed-on: https://go-review.googlesource.com/c/go/+/262137
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2020-10-14 21:37:37 +00:00
Robert Griesemer
b627988b0c [dev.typeparams] cmd/compile/internal/syntax: implement parsing of type parameters
Port from dev.go2go prototype branch. The compiler doesn't yet set the
syntax.AllowGenerics mode, so parsing of generic code remains disabled.

Known issue: The doc strings documenting the specific syntax accepted
by parser methods are not all up-to-date.

Change-Id: I13d134289fd9330fd0ed7f97c997cca6f23466fd
Reviewed-on: https://go-review.googlesource.com/c/go/+/261658
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2020-10-13 04:38:33 +00:00
Robert Griesemer
617b633917 [dev.typeparams] cmd/compile/internal/syntax: prepare syntax nodes for type parameters
- add TParamList fields to TypeDecl, FuncDecl
- also: change File.Lines to File.EOF so we have the actual file end position

Change-Id: Ia345f888080a884f7ac5cefd8bff3d80e4a59cdc
Reviewed-on: https://go-review.googlesource.com/c/go/+/261657
Trust: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2020-10-13 00:22:03 +00:00
Robert Griesemer
e0cb80e4d1 cmd/compile/internal/syntax: provide BadExpr where needed, call correct error handler
- For "if" statements without a condition, provide a BadExpr rather than nil
  (clients expect IfStmt.Cond != nil since the parser is taking care of
  reporting a missing condition).

- For 3-index slice expressions, also provide BadExpr where an index is
  required but missing.

- Declare a parser-local error method to hide the embedded error method
  so we don't use it by mistake.

Accidentally found while adjusting prototype parser to work for generics.

Change-Id: Iacc211cc60869be05efe9ae630d65dff1dac00a0
Reviewed-on: https://go-review.googlesource.com/c/go/+/261218
Trust: Robert Griesemer <gri@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2020-10-12 18:01:22 +00:00
Russ Cox
768201729d cmd/compile: detect and diagnose invalid //go: directive placement
Thie CL changes cmd/compile/internal/syntax to give the gc half of
the compiler more control over pragma handling, so that it can prepare
better errors, diagnose misuse, and so on. Before, the API between
the two was hard-coded as a uint16. Now it is an interface{}.
This should set us up better for future directives.

In addition to the split, this CL emits a "misplaced compiler directive"
error for any directive that is in a place where it has no effect.
I've certainly been confused in the past by adding comments
that were doing nothing and not realizing it. This should help
avoid that kind of confusion.

The rule, now applied consistently, is that a //go: directive
must appear on a line by itself immediately before the declaration
specifier it means to apply to. See cmd/compile/doc.go for
precise text and test/directive.go for examples.

This may cause some code to stop compiling, but that code
was broken. For example, this code formerly applied the
//go:noinline to f (not c) but now will fail to compile:

	//go:noinline
	const c = 1

	func f() {}

Change-Id: Ieba9b8d90a27cfab25de79d2790a895cefe5296f
Reviewed-on: https://go-review.googlesource.com/c/go/+/228578
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
2020-04-21 16:47:01 +00:00
Robert Griesemer
4de606b55f cmd/compile/internal/syntax: faster and simpler source reader
This is one of several changes that were part of a larger rewrite
which I made in early 2019 after switching to the new number literal
syntax implementation. The purpose of the rewrite was to simplify
reading of source code (Unicode character by character) and speed up
the scanner but was never submitted for review due to other priorities.

Part 3 of 3:

This change contains a complete rewrite of source.go, the file that
implements reading individual Unicode characters from the source.
The new implementation is easier to use and has simpler literal
buffer management, resulting in faster scanner and thus parser
performance.

Thew new source.go (internal) API is centered around nextch() which
advances the scanner by one character. The scanner has been adjusted
around nextch() and now consistently does one character look-ahead
(there's no need for complicated ungetr-ing anymore). Only in one
case backtrack is needed (when finding '..' rather than '...') and
that case is now more cleanly solved with the new reset() function.

Measuring line/s parsing peformance by running

go test -run StdLib -fast -skip "syntax/(scanner|source)\.go"

(best of 5 runs on "quiet" MacBook Pro, 3.3GHz Dual-Core i7, 16GB RAM,
OS X 10.15.3) before and after shows consistently 3-5% improvement of
line parsing speed:

old: parsed 1788155 lines (3969 files) in 1.255520307s (1424234 lines/s)
new: parsed 1788155 lines (3969 files) in 1.213197037s (1473919 lines/s)

(scanner.go and parser.go are skipped because this CL changed those files.)

Change-Id: Ida947f4b538d42eb2d2349062c69edb6c9e5ca66
Reviewed-on: https://go-review.googlesource.com/c/go/+/221603
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2020-03-05 19:25:46 +00:00
Robert Griesemer
ffc0573b85 cmd/compile/internal/syntax: better error when an assignment is used in value context
The error message is now positioned at the statement position (which is
an identifing token, such as the '=' for assignments); and in case of
assignments it emphasizes the assignment by putting the Lhs and Rhs
in parentheses. Finally, the wording is changed from "use of * as value"
to the stronger "cannot use * as value" (for which there is precedent
elsewhere in the parser).

Fixes #36858.

Change-Id: Ic3f101bba50f58e3a1d9b29645066634631f2d61
Reviewed-on: https://go-review.googlesource.com/c/go/+/218337
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2020-02-21 22:57:52 +00:00
Robert Griesemer
117400ec09 cmd/compile/internal/syntax: add BasicLit.Bad field for lexical errors
The new (internal) field scanner.bad indicates whether a syntax error
occurred while scanning a literal; the corresponding scanner.lit
string may be syntactically incorrect in that case. Store the value
of scanner.bad together with the scanner.lit in BasicLit.

Clean up error handling so that all syntactic errors use one of the
scanner's error reporting methods which also set scanner.bad. Make
use of the new field in a few places where we used to track a prior
error separately.

Preliminary step towards fixing #32133 in a comprehensive manner.

Change-Id: I4d79ad6e3b50632dd5fb3fc32ca3df0598ee77b4
Reviewed-on: https://go-review.googlesource.com/c/go/+/192278
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2019-08-29 23:37:01 +00:00
Robert Griesemer
dca0d03b9c cmd/compile/internal/syntax: better error recovery after missing expression
Don't skip closing parentheses of any kind after a missing
expression. They are likely part of the lexical construct
enclosing the expression.

Fixes #33386.

Change-Id: Ic0abc2037ec339a345ec357ccc724b7ad2a64c00
Reviewed-on: https://go-review.googlesource.com/c/go/+/188502
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2019-08-27 15:46:41 +00:00
Robert Griesemer
4145c5da1f cmd/compile: better recovery after := (rather than =) in declarations
Before this fix, a mistaken := in a (const/type/var) declaration
ended that declaration with an error from which the parser didn't
recover well. This low-cost change will provide a better error
message and lets the parser recover perfectly.

Fixes #31092.

Change-Id: Ic4f94dc5e29dd00b7ef6d53a80dded638e3cea80
Reviewed-on: https://go-review.googlesource.com/c/go/+/169958
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2019-04-03 23:01:13 +00:00
Robert Griesemer
4637699e92 cmd/compile/internal/syntax: better error message for incorrect if/switch header
Fixes #23664.

Change-Id: Ic0637e9f896b2fc6502dfbab2d1c4de3c62c0bd2
Reviewed-on: https://go-review.googlesource.com/104616
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
Run-TryBot: Robert Griesemer <gri@golang.org>
2018-04-03 21:57:37 +00:00
Robert Griesemer
c65a2781be cmd/compile: better handling of incorrect type switches
Don't report errors if we don't have a correct type switch
guard; instead ignore it and leave it to the type-checker
to report the error. This leads to better error messages
concentrating on the type switch guard rather than errors
around (confusing) syntactic details.

Also clean up some code setting up AssertExpr (they never
have a nil Type field) and remove some incorrect TODOs.

Fixes #24470.

Change-Id: I69512f36e0417e3b5ea9c8856768e04b19d654a8
Reviewed-on: https://go-review.googlesource.com/103615
Run-TryBot: Robert Griesemer <gri@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-04-03 05:34:20 +00:00
Robert Griesemer
0c884d0810 cmd/compile, cmd/compile/internal/syntax: print relative column info
This change enables printing of relative column information if a
prior line directive specified a valid column. If there was no
line directive, or the line directive didn't specify a column
(or the -C flag is specified), no column information is shown in
file positions.

Implementation: Column values (and line values, for that matter)
that are zero are interpreted as "unknown". A line directive that
doesn't specify a column records that as a zero column in the
respective PosBase data structure. When computing relative columns,
a relative value is zero of the base's column value is zero.
When formatting a position, a zero column value is not printed.

To make this work without special cases, the PosBase for a file
is given a concrete (non-0:0) position 1:1 with the PosBase's
line and column also being 1:1. In other words, at the position
1:1 of a file, it's relative positions are starting with 1:1 as
one would expect.

In the package syntax, this requires self-recursive PosBases for
file bases, matching what cmd/internal/src.PosBase was already
doing. In src.PosBase, file and inlining bases also need to be
based at 1:1 to indicate "known" positions.

This change completes the cmd/compiler part of the issue below.

Fixes #22662.

Change-Id: I6c3d2dee26709581fba0d0261b1d12e93f1cba1a
Reviewed-on: https://go-review.googlesource.com/97375
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-02-28 03:51:23 +00:00
Robert Griesemer
6fa6bde924 cmd/compile/internal/syntax: implement //line :line:col handling
For line directives which have a line and a column number,
an omitted filename means that the filename has not changed
(per the issue below).

For line directives w/o a column number, an omitted filename
means the empty filename (to preserve the existing behavior).

For #22662.

Change-Id: I32cd9037550485da5445a34bb104706eccce1df1
Reviewed-on: https://go-review.googlesource.com/96476
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-02-26 18:27:44 +00:00
Robert Griesemer
5c08b9e8bd cmd/compile/internal/syntax: remove dependency on cmd/internal/src
For dependency reasons, the data structure implementing source
positions in the compiler is in cmd/internal/src. It contains
highly compiler specific details (e.g. inlining index).

This change introduces a parallel but simpler position
representation, defined in the syntax package, which removes
that package's dependency on cmd/internal/src, and also removes
the need to deal with certain filename-specific operations
(defined by the needs of the compiler) in the syntax package.
As a result, the syntax package becomes again a compiler-
independent, stand-alone package that at some point might
replace (or augment) the existing top-level go/* syntax-related
packages.

Additionally, line directives that update column numbers
are now correctly tracked through the syntax package, with
additional tests added. (The respective changes also need to
be made in cmd/internal/src; i.e., the compiler accepts but
still ignores column numbers in line directives.)

This change comes at the cost of a new position translation
step, but that step is cheap because it only needs to do real
work if the position base changed (i.e., if there is a new file,
or new line directive).

There is no noticeable impact on overall compiler performance
measured with `compilebench -count 5 -alloc`:

name       old time/op       new time/op       delta
Template         220ms ± 8%        228ms ±18%    ~     (p=0.548 n=5+5)
Unicode          119ms ±11%        113ms ± 5%    ~     (p=0.056 n=5+5)
GoTypes          684ms ± 6%        677ms ± 3%    ~     (p=0.841 n=5+5)
Compiler         3.19s ± 7%        3.01s ± 1%    ~     (p=0.095 n=5+5)
SSA              7.92s ± 8%        7.79s ± 1%    ~     (p=0.690 n=5+5)
Flate            141ms ± 7%        139ms ± 4%    ~     (p=0.548 n=5+5)
GoParser         173ms ±12%        171ms ± 4%    ~     (p=1.000 n=5+5)
Reflect          417ms ± 5%        411ms ± 3%    ~     (p=0.548 n=5+5)
Tar              205ms ± 5%        198ms ± 2%    ~     (p=0.690 n=5+5)
XML              232ms ± 4%        229ms ± 4%    ~     (p=0.690 n=5+5)
StdCmd           28.7s ± 5%        28.2s ± 2%    ~     (p=0.421 n=5+5)

name       old user-time/op  new user-time/op  delta
Template         269ms ± 4%        265ms ± 5%    ~     (p=0.421 n=5+5)
Unicode          153ms ± 7%        149ms ± 3%    ~     (p=0.841 n=5+5)
GoTypes          850ms ± 7%        862ms ± 4%    ~     (p=0.841 n=5+5)
Compiler         4.01s ± 5%        3.86s ± 0%    ~     (p=0.190 n=5+4)
SSA              10.9s ± 4%        10.8s ± 2%    ~     (p=0.548 n=5+5)
Flate            166ms ± 7%        167ms ± 6%    ~     (p=1.000 n=5+5)
GoParser         204ms ± 8%        206ms ± 7%    ~     (p=0.841 n=5+5)
Reflect          514ms ± 5%        508ms ± 4%    ~     (p=0.548 n=5+5)
Tar              245ms ± 6%        244ms ± 3%    ~     (p=0.690 n=5+5)
XML              280ms ± 4%        278ms ± 4%    ~     (p=0.841 n=5+5)

name       old alloc/op      new alloc/op      delta
Template        37.9MB ± 0%       37.9MB ± 0%    ~     (p=0.841 n=5+5)
Unicode         28.8MB ± 0%       28.8MB ± 0%    ~     (p=0.841 n=5+5)
GoTypes          113MB ± 0%        113MB ± 0%    ~     (p=0.151 n=5+5)
Compiler         468MB ± 0%        468MB ± 0%  -0.01%  (p=0.032 n=5+5)
SSA             1.50GB ± 0%       1.50GB ± 0%    ~     (p=0.548 n=5+5)
Flate           24.4MB ± 0%       24.4MB ± 0%    ~     (p=1.000 n=5+5)
GoParser        30.7MB ± 0%       30.7MB ± 0%    ~     (p=1.000 n=5+5)
Reflect         76.5MB ± 0%       76.5MB ± 0%    ~     (p=0.548 n=5+5)
Tar             38.9MB ± 0%       38.9MB ± 0%    ~     (p=0.222 n=5+5)
XML             41.6MB ± 0%       41.6MB ± 0%    ~     (p=0.548 n=5+5)

name       old allocs/op     new allocs/op     delta
Template          382k ± 0%         382k ± 0%  +0.01%  (p=0.008 n=5+5)
Unicode           343k ± 0%         343k ± 0%    ~     (p=0.841 n=5+5)
GoTypes          1.19M ± 0%        1.19M ± 0%  +0.01%  (p=0.008 n=5+5)
Compiler         4.53M ± 0%        4.53M ± 0%  +0.03%  (p=0.008 n=5+5)
SSA              12.4M ± 0%        12.4M ± 0%  +0.00%  (p=0.008 n=5+5)
Flate             235k ± 0%         235k ± 0%    ~     (p=0.079 n=5+5)
GoParser          318k ± 0%         318k ± 0%    ~     (p=0.730 n=5+5)
Reflect           978k ± 0%         978k ± 0%    ~     (p=1.000 n=5+5)
Tar               393k ± 0%         393k ± 0%    ~     (p=0.056 n=5+5)
XML               405k ± 0%         405k ± 0%    ~     (p=0.548 n=5+5)

name       old text-bytes    new text-bytes    delta
HelloSize        672kB ± 0%        672kB ± 0%    ~     (all equal)
CmdGoSize       7.12MB ± 0%       7.12MB ± 0%    ~     (all equal)

name       old data-bytes    new data-bytes    delta
HelloSize        133kB ± 0%        133kB ± 0%    ~     (all equal)
CmdGoSize        390kB ± 0%        390kB ± 0%    ~     (all equal)

name       old exe-bytes     new exe-bytes     delta
HelloSize       1.07MB ± 0%       1.07MB ± 0%    ~     (all equal)
CmdGoSize       11.2MB ± 0%       11.2MB ± 0%    ~     (all equal)

Passes toolstash compare.

For #22662.

Change-Id: I19edb53dd9675af57f7122cb7dba2a6d8bdcc3da
Reviewed-on: https://go-review.googlesource.com/94515
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-02-26 18:27:15 +00:00
Robert Griesemer
e2a86b6bd9 cmd/compile/internal/syntax: simpler position base update for line directives (cleanup)
The existing code was somewhat convoluted and made several assumptions
about the encoding of position bases:

1) The position's base for a file contained a position whose base
   pointed to itself (which is true but an implementation detail
   of src.Pos).

2) Updating the position base for a line directive required finding
   the base of the most recent's base position.

This change simply stores the file's position base and keeps using it
directly for each line directive (instead of getting it from the most
recently updated base).

Change-Id: I4d80da513bededb636eab0ce53257fda73f0dbc0
Reviewed-on: https://go-review.googlesource.com/95736
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-02-21 21:06:36 +00:00
Robert Griesemer
1a22738749 cmd/compile/internal/syntax: more tolerant handling of missing function invocation in go/defer
Assume that an expression that is not a function call in a defer/go
statement is indeed a function that is just missing its invocation.
Report the error but continue with a sane syntax tree.

Fixes #23586.

Change-Id: Ib45ebac57c83b3e39ae4a1b137ffa291dec5b50d
Reviewed-on: https://go-review.googlesource.com/94156
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-02-15 01:48:16 +00:00
Robert Griesemer
f04eebfdf5 cmd/compile/internal/syntax: follow Go naming conventions for error methods
Also, remove parser.error method (in favor of parser.errorAt) as it's only
used twice.

This is a purely cosmetic change.

Change-Id: Idb3b8b50f1c2e4d10de2ffb1c1184ceba8f7de8a
Reviewed-on: https://go-review.googlesource.com/94030
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-02-14 23:31:06 +00:00
Robert Griesemer
b890688986 cmd/compile/internal/syntax: implement comment reporting in scanner
R=go1.11

In order to collect comments in the AST and for error testing purposes,
the scanner needs to not only recognize and skip comments, but also be
able to report them if so desired. This change adds a mode flag to the
scanner's init function which controls the scanner behavior around
comments.

In the common case where comments are not needed, there must be no
significant overhead. Thus, comments are reported via a handler upcall
rather than being returned as a _Comment token (which the parser would
have to filter out with every scanner.next() call).

Because the handlers for error messages, directives, and comments all
look the same (they take a position and text), and because directives
look like comments, and errors never start with a '/', this change
simplifies the scanner's init call to only take one (error) handler
instead of 2 or 3 different handlers with identical signature. It is
trivial in the handler to determine if we have an error, directive,
or general comment.

Finally, because directives are comments, when reporting directives
the full comment text is returned now rather than just the directive
text. This simplifies the implementation and makes the scanner API
more regular. Furthermore, it provides important information about
the comment style used by a directive, which may matter eventually
when we fully implement /*line file:line:col*/ directives.

Change-Id: I2adbfcebecd615e4237ed3a832b6ceb9518bf09c
Reviewed-on: https://go-review.googlesource.com/88215
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-02-12 22:57:57 +00:00
Robert Griesemer
670494827c cmd/compile/internal/syntax: better error recovery after missing type
R=go1.11.

This is just a copy of the fix for #23434:
https://go-review.googlesource.com/c/go/+/87898.

Test pending test harness for the syntax package.

Change-Id: I52409aebe13ec784ddd7e41190a81c7e126bdd0c
Reviewed-on: https://go-review.googlesource.com/87901
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-02-12 22:57:56 +00:00
Robert Griesemer
be9a1774f2 cmd/compile/internal/syntax: better error msg for some 'if' statements
R=go1.11

A common error is to write '=' instead of '==' inside the condition
of a simple 'if' statement:

	if x = 0 { ... }

Highlight the fact that we have an assignment in the error message
to prevent further confusion.

Fixes #23385.

Change-Id: I1552050fd6da927bd12a1be0977bd2e98eca5885
Reviewed-on: https://go-review.googlesource.com/87316
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2018-02-12 22:57:54 +00:00