Semi-regular merge from tip to dev.ssa.
Two fixes:
1) Mark selectgo as not returning. This caused problems
because there are no VARKILL ops on the selectgo path,
causing things to be marked live that shouldn't be.
2) Tell the amd64 assembler that addressing modes like
name(SP)(AX*4) are ok.
Change-Id: I9ca81c76391b1a65cc47edc8610c70ff1a621913
Panic doesn't return, so record that we immediately exit after a panic
call. This will help code analysis.
Change-Id: I4d1f67494f97b6aee130c43ff4e44307b2b0f149
Reviewed-on: https://go-review.googlesource.com/19303
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
The frontend does this for 32 bits and below, but SSA needs
to do it for 64 bits. The algorithms are all copied from
cgen.go:cgen_div.
Speeds up TimeFormat substantially: ~40% slower to ~10% slower.
Change-Id: I023ea2eb6040df98ccd9105e15ca6ea695610a7a
Reviewed-on: https://go-review.googlesource.com/19302
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Todd Neal <todd@tneal.org>
Use just a single write barrier flag test, even if there
are multiple pointer fields in a struct.
This helps move more of the wb-specific code (like the LEA
needed to materialize the write address) into the unlikely path.
Change-Id: Ic7a67145904369c4ff031e464d51267d71281c8f
Reviewed-on: https://go-review.googlesource.com/19085
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
If the output register is one of the input registers,
we can use a real add instead of LEA.
Change-Id: Ide58f1536afb077c0b939d3a8c7555807fd1c5e3
Reviewed-on: https://go-review.googlesource.com/19234
Reviewed-by: Alexandru Moșoi <alexandru@mosoi.ro>
Mostly indexed loads. A few more LEA cases.
Change-Id: Idc1d447ed0dd6e906cd48e70307a95e77f61cf5f
Reviewed-on: https://go-review.googlesource.com/19172
Reviewed-by: Todd Neal <todd@tneal.org>
Run-TryBot: Keith Randall <khr@golang.org>
Add the aux type to opcodes.
Add rematerializeable as a flag.
Change-Id: I906e19281498f3ee51bb136299bf26e13a54b2ec
Reviewed-on: https://go-review.googlesource.com/19088
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Todd Neal <todd@tneal.org>
Small optimizations I noticed while looking at Giovanni's test cases.
More shifts by constants.
Indexed stores for smaller types.
Fold LEA into loads/stores.
More extending loads.
CMP $0 of AND -> TEST
Fix order of TEST ops.
Giovanni's test cases at https://gist.github.com/rasky/62fba94e3a20d1b05b2a
Change-Id: I7077bc0b5319bf05767eeb39f401f4bb4b39f635
Reviewed-on: https://go-review.googlesource.com/19086
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: Todd Neal <todd@tneal.org>
Reviewed-by: David Chase <drchase@google.com>
From memory profiling, about 3% reduction in allocation count.
Change-Id: I4b662d55b8a94fe724759a2b22f05a08d0bf40f8
Reviewed-on: https://go-review.googlesource.com/19103
Reviewed-by: Keith Randall <khr@golang.org>
Rename StoreConst to ValAndOff so we can use it for other ops.
Make ValAndOff print nicely.
Add some notes & checks related to my aborted attempt to
implement combined CMP+load ops.
Change-Id: I2f901d12d42bc5a82879af0334806aa184a97e27
Reviewed-on: https://go-review.googlesource.com/18947
Run-TryBot: David Chase <drchase@google.com>
Reviewed-by: David Chase <drchase@google.com>
The old write barriers used _nostore versions, which
don't work for Ian's cgo checker. Instead, we adopt the
same write barrier pattern as the default compiler.
It's a bit trickier to code up but should be more efficient.
Change-Id: I6696c3656cf179e28f800b0e096b7259bd5f3bb7
Reviewed-on: https://go-review.googlesource.com/18941
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Some tests make multiple Funcs per Config at once.
With value & block caching, we can't do that any more.
Change-Id: Ibdb60aa2fcf478f1726b3be0fcaa06b04433eb67
Reviewed-on: https://go-review.googlesource.com/19081
Reviewed-by: Keith Randall <khr@golang.org>
It is one of the slowest compiler phases right now, and we
run two of them.
Instead of using a map to make the initial partition, use a sort.
It is much less memory intensive.
Do a few optimizations to avoid work for size-1 equivalence classes.
Implement -N.
Change-Id: I1d2d85d3771abc918db4dd7cc30b0b2d854b15e1
Reviewed-on: https://go-review.googlesource.com/19024
Reviewed-by: David Chase <drchase@google.com>
The x86 backend automatically rewrites MOV $0, AX to
XOR AX, AX. That rewrite isn't ok when the flags register
is live across the MOV. Keep track of which moves care
about preserving flags, then disable this rewrite for them.
On x86, Prog.Mark was being used to hold the length of the
instruction. We already store that in Prog.Isize, so no
need to store it in Prog.Mark also. This frees up Prog.Mark
to hold a bitmask on x86 just like all the other architectures.
Update #12405
Change-Id: Ibad8a8f41fc6222bec1e4904221887d3cc3ca029
Reviewed-on: https://go-review.googlesource.com/18861
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
The conversion from -0.0 to +0.0 happens inside mpgetflt now.
The SSA code doesn't need this fix any more.
Change-Id: I6cd4f4a4e75b13cf284ebbb95b08af050ed9891c
Reviewed-on: https://go-review.googlesource.com/18942
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Keith Randall <khr@golang.org>
Empty blocks are introduced to remove critical edges.
After regalloc, we can remove any of the added blocks
that are still empty.
Change-Id: I0b40e95ac3a6cc1e632a479443479532b6c5ccd9
Reviewed-on: https://go-review.googlesource.com/18833
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Break small structs up into their components so they
can be registerized.
Change StructSelect to use field indexes instead of
field offsets, as field offsets aren't unique in the
presence of zero-sized fields.
Change-Id: I2f1dc89f7fa58e1cf58aa1a32b238959d53f62e4
Reviewed-on: https://go-review.googlesource.com/18570
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Redo how we keep track of forward references when building SSA.
When the forward reference is resolved, update the Value node
in place.
Improve the phi elimination pass so it can simplify phis of phis.
Give SSA package access to decoded line numbers. Fix line numbers
for constant booleans.
Change-Id: I3dc9896148d260be2f3dd14cbe5db639ec9fa6b7
Reviewed-on: https://go-review.googlesource.com/18674
Reviewed-by: David Chase <drchase@google.com>
Run-TryBot: Keith Randall <khr@golang.org>
Distinguish move/load/store ops. Unify some of this code a bit.
Reduces Mandelbrot slowdown with SSA from 58% to 12%.
Change-Id: I3276eaebcbcdd9de3f8299c79b5f25c0429194c4
Reviewed-on: https://go-review.googlesource.com/18677
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
If a failure occurs in SSA processing, we always report the
last line of the function we're compiling. Modify the callbacks
from SSA to the GC compiler so we can pass a line number back
and use it in Fatalf.
Change-Id: Ifbfad50d5e167e997e0a96f0775bcc369f5c397e
Reviewed-on: https://go-review.googlesource.com/18599
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Add new constant-flags opcodes. These can be generated from
comparisons that we know the result of, like x&31 < 32.
Constant-fold the constant-flags opcodes into all flag users.
Reorder some CMPxconst args so they read in the comparison direction.
Reorg deadcode removal a bit - it needs to remove the OpCopy ops it
generates when strength-reducing Phi ops. So it needs to splice out all
the dead blocks and do a copy elimination before it computes live
values.
Change-Id: Ie922602033592ad8212efe4345394973d3b94d9f
Reviewed-on: https://go-review.googlesource.com/18267
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
In code that does:
var x, z int32
var y int64
z = phi(x, int32(y))
We silently drop the int32 cast because truncation is a no-op.
The phi operation needs to make sure it uses the size of the
phi, not the size of its arguments, when generating spills.
Change-Id: I1f7baf44f019256977a46fdd3dad1972be209042
Reviewed-on: https://go-review.googlesource.com/18390
Reviewed-by: David Chase <drchase@google.com>
We used to compile everything with SSA and then decide whether
to use the result or not. It was useful when we were working
on coverage without much regard for correctness, but not so much now.
Instead, let's decide what we're going to compile and go through
the SSA compiler for only those functions.
TODO: next CL: get rid of all the UnimplementedF stuff.
Change-Id: If629addd8b62cd38ef553fd5d835114137885ce0
Reviewed-on: https://go-review.googlesource.com/17763
Reviewed-by: David Chase <drchase@google.com>
Make sure that when a pointer value is live across a function
call, we save it as a pointer. (And similarly a uintptr
live across a function call should not be saved as a pointer.)
Add a nasty test case.
This is probably what is preventing the merge from master
to dev.ssa. Signs point to something like this bug happening
in mallocgc.
Change-Id: Ib23fa1251b8d1c50d82c6a448cb4a4fc28219029
Reviewed-on: https://go-review.googlesource.com/16830
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Panics are only distinguished by their type and line number, so
if we can trigger two of those panics in the same line, use the
same panic call. For example, in a[i]+b[j] we need only one
panicindex call that both bounds checks can use.
Change-Id: Ia2b6d3b1a67f2775df05fb72b8a1b149833572b7
Reviewed-on: https://go-review.googlesource.com/16772
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Might as well, for a && !(b && c) and the like.
Change-Id: I2548b6e6ee5870e074bcef6edd56a7db6e81d70f
Reviewed-on: https://go-review.googlesource.com/16600
Reviewed-by: David Chase <drchase@google.com>
Run-TryBot: David Chase <drchase@google.com>
Declare a function's arguments as having already been
spilled so their use just requires a restore.
Allow spill locations to be portions of larger objects the stack.
Required to load portions of compound input arguments.
Rename the memory input to InputMem. Use Arg for the
pre-spilled argument values.
Change-Id: I8fe2a03ffbba1022d98bfae2052b376b96d32dda
Reviewed-on: https://go-review.googlesource.com/16536
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Be more consistent about this. There's no reason to do the
pointer arithmetic on a different type, as sizeof(int) >=
sizeof(ptr) on all of our platforms. It simplifies our
rewrite rules also, except for a few that need duplication.
Add some more constant folding to get constant indexing and
slicing to fold down to nothing.
Change-Id: I3e56cdb14b3dc1a6a0514f0333e883f92c19e3c7
Reviewed-on: https://go-review.googlesource.com/16586
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
For the statement
if a && b { target }
the old code allocated a new variable v and did:
v = a
if a {
v = b
}
if v { goto target }
The new code does:
if a {
if b { goto target }
}
The new arrangement tends to generate much more efficient code. In
particular, there is no temporary variable and there is only one join
point instead of two.
The old code is still used for ANDAND and OROR which are not
direct descendents of IF or FOR statements.
Change-Id: I082f246d27c823c6f32d1287300e4b0911607507
Reviewed-on: https://go-review.googlesource.com/16584
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Some tests disabled, some bifurcated into _ssa and not,
with appropriate logging added to compiler.
"tests/live.go" in particular needs attention.
SSA-specific testing removed, since it's all SSA now.
Added "-run_skips" option to tests/run.go to simplify
checking whether a test still fails (or how it fails)
on a skipped platform.
The compiler now compiles with SSA by default.
If you don't want SSA, specify GOSSAHASH=n (or N) as
an environment variable. Function names ending in "_ssa"
are always SSA-compiled.
GOSSAFUNC=fname retains its "SSA for fname, log to ssa.html"
GOSSAPKG=pkg only has an effect when GOSSAHASH=n
GOSSAHASH=10101 etc retains its name-hash-matching behavior
for purposes of debugging.
See #13068
Change-Id: I8217bfeb34173533eaeb391b5f6935483c7d6b43
Reviewed-on: https://go-review.googlesource.com/16299
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: David Chase <drchase@google.com>
This improves cse and works correctly now that divide by zero is checked
explicitly.
Change-Id: If54fbe403ed5230b897afc5def644ba9f0056dfd
Reviewed-on: https://go-review.googlesource.com/16454
Run-TryBot: Todd Neal <todd@tneal.org>
Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Added an explicit compare-zero and branch-to-panic for
integer division and mod so that other optimizations will
not be fooled by their implicit panics.
Change-Id: Ibf96f636b541c0088861907c537a6beb4b99fa4c
Reviewed-on: https://go-review.googlesource.com/16450
Reviewed-by: Keith Randall <khr@golang.org>
For debugging, spill values to named variables instead of autotmp_
variables if possible. We do this by keeping a name -> value map
for each function, keep it up-to-date during deadcode elim, and use
it to override spill decisions in stackalloc.
It might even make stack frames a bit smaller, as it makes it easy
to identify a set of spills which are likely not to interfere.
This just works for one-word variables for now. Strings/slices
will be a separate CL.
Change-Id: Ie89eba8cab16bcd41b311c479ec46dd7e64cdb67
Reviewed-on: https://go-review.googlesource.com/16336
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Introduce opcodes that store a constant value.
AuxInt now needs to hold both the value to be stored and the
constant offset at which to store it. Introduce a StoreConst
type to help encode/decode these parts to/from an AuxInt.
Change-Id: I1631883abe035cff4b16368683e1eb3d2ccb674d
Reviewed-on: https://go-review.googlesource.com/16170
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
If the closure pointer gets spilled, we need to spill it with
pointer type to make stack copy and GC happy.
Change-Id: Ic108748e6b9caecd45522141f02c9422567376e3
Reviewed-on: https://go-review.googlesource.com/16363
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Replace REP MOVSB with all the copying techniques used by the
old compiler. Copy in chunks, DUFFCOPY, etc.
Introduces MOVO opcodes and an Int128 type to move around
16 bytes at a time.
Change-Id: I1e73e68ca1d8b3dd58bb4af2f4c9e5d9bf13a502
Reviewed-on: https://go-review.googlesource.com/16174
Reviewed-by: Todd Neal <todd@tneal.org>
Run-TryBot: Keith Randall <khr@golang.org>
Use faulting loads instead of test/jeq to do nil checks.
Fold nil checks into a following load/store if possible.
Makes binaries about 2% smaller.
Change-Id: I54af0f0a93c853f37e34e0ce7e3f01dd2ac87f64
Reviewed-on: https://go-review.googlesource.com/16287
Reviewed-by: David Chase <drchase@google.com>
Modified GOSSA{HASH.PKG} environment variable filters to
make it easier to make/run with all SSA for testing.
Disable attempts at SSA for architectures that are not
amd64 (avoid spurious errors/unimplementeds.)
Removed easy out for unimplemented features.
Add convert op for proper liveness in presence of uintptr
to/from unsafe.Pointer conversions.
Tweaked stack sizes to get a pass on windows;
1024 instead 768, was observed to pass at least once.
Change-Id: Ida3800afcda67d529e3b1cf48ca4a3f0fa48b2c5
Reviewed-on: https://go-review.googlesource.com/16201
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: David Chase <drchase@google.com>
getg reads from memory, so it should really have a
memory arg. It is critical in functions which call setg
to make sure getg gets ordered correctly with setg.
Change-Id: Ief4875421f741fc49c07b0e1f065ce2535232341
Reviewed-on: https://go-review.googlesource.com/16100
Run-TryBot: Keith Randall <khr@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Run-TryBot: David Chase <drchase@google.com>
Reuse the ArgSymbol for nodes so that the Aux values will be equal for cse.
Change-Id: Iaae80bd19ff2d3f51b6c9049fd860e04baa6f175
Reviewed-on: https://go-review.googlesource.com/15930
Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Todd Neal <todd@tneal.org>
Changed racewalk/race detector to use FP in a more
sensible way.
Relaxed checks for CONVNOP when race detecting.
Modified tighten to ensure that GetClosurePtr cannot float
out of entry block (turns out this cannot be relaxed, DX is
sometimes stomped by other code accompanying race detection).
Added case for addr(CONVNOP)
Modified addr to take "bounded" flag to suppress nilchecks
where it is set (usually, by race detector).
Cannot leave unimplemented-complainer enabled because it
turns out we are optimistically running SSA on every platform.
Change-Id: Ife021654ee4065b3ffac62326d09b4b317b9f2e0
Reviewed-on: https://go-review.googlesource.com/15710
Reviewed-by: Keith Randall <khr@golang.org>
