fuzz.CoordinateFuzzing and RunFuzzWorker now accept a context.Context
parameter. They should terminate gracefully when the context is
cancelled. The worker should exit quickly without processing more
inputs. The coordinator should save interesting inputs to the cache.
The testing package can't import context directly, so it provides a
timeout argument to testdeps.CoordinateFuzzing instead. The testdeps
wrapper sets the timeout and installs an interrupt handler (for SIGINT
on POSIX and the equivalent on Windows) that cancels the context when
^C is pressed.
Note that on POSIX platforms, pressing ^C causes the shell to deliver
SIGINT to all processes in the active group: so 'go test', the
coordinator, and the workers should all react to that. On Windows,
pressing ^C only interrupts 'go test'. We may want to look at that
separately.
Change-Id: I924d3be2905f9685dae82ff3c047ca3d6b5e2357
Reviewed-on: https://go-review.googlesource.com/c/go/+/279487
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
'go test -fuzz' may now read and write interesting fuzzing values to
directories in $GOCACHE/fuzz. Files in this directory are named
$pkg/$test/$hash where $pkg is the package path containing the fuzz
target, $test is the target name, and $hash is the SHA-256 sum of the
data in the file.
Note that different versions of the same package or packages with the
same path from different modules may share the same directory.
Although files are written into a subdirectory of GOCACHE, they are
not removed automatically, nor are they removed by 'go clean -cache'.
Instead, they may be removed with 'go clean -fuzzcache'. We chose to
nest the fuzzing directory inside GOCACHE to avoid introducing a new
environment variable, since there's no real need for users to specify
a separate directory.
Change-Id: I2032cf8e6c92f715cf36a9fc6a550acf666d2382
Reviewed-on: https://go-review.googlesource.com/c/go/+/275534
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
As part of #42026, these helpers from io/ioutil were moved to os.
(ioutil.TempFile and TempDir became os.CreateTemp and MkdirTemp.)
Update the Go tree to use the preferred names.
As usual, code compiled with the Go 1.4 bootstrap toolchain
and code vendored from other sources is excluded.
ReadDir changes are in a separate CL, because they are not a
simple search and replace.
For #42026.
Change-Id: If318df0216d57e95ea0c4093b89f65e5b0ababb3
Reviewed-on: https://go-review.googlesource.com/c/go/+/266365
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The workers were printing PASS/FAIL logs and
various others things, when that should be
the sole responsibility of the coordinator
process, which will have the aggregated data.
Change-Id: I7ac9883db62f0fe79ba1799cb88773c542a2a948
Reviewed-on: https://go-review.googlesource.com/c/go/+/274652
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
This change also includes a small cleanup of the run()
function and additional tests for error conditions
in fuzz targets.
Change-Id: I2b7722b25a0d071182a84f1dc4b92e82a7ea34d9
Reviewed-on: https://go-review.googlesource.com/c/go/+/256978
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Package fuzz provides common fuzzing functionality for tests built
with "go test" and for programs that use fuzzing functionality in the
testing package.
Change-Id: I3901c6a993a9adb8a93733ae1838b86dd78c7036
Reviewed-on: https://go-review.googlesource.com/c/go/+/259259
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
This change adds support for a -fuzz flag in the go command, and sets up
the groundwork for native fuzzing support. These functions are no-ops
for now, but will be built out and tested in future PRs.
Change-Id: I58e78eceada5799bcb73acc4ae5a20372badbf40
Reviewed-on: https://go-review.googlesource.com/c/go/+/251441
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Store the PC instead of the string name of the function, and defer
that conversion until we need it.
Helper is still relatively expensive in CPU time (few hundred ns),
but memory allocation is now constant for a test rather than linear in
the number of times Helper is called.
benchstat:
name old time/op new time/op delta
TBHelper-4 1.30µs ±27% 0.53µs ± 1% -59.03% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
TBHelper-4 216B ± 0% 0B -100.00% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
TBHelper-4 2.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
Change-Id: I6565feb491513815e1058637d086b0374fa94e19
GitHub-Last-Rev: c2329cf225dab6de7309af3583daa011bafb9a62
GitHub-Pull-Request: golang/go#38834
Reviewed-on: https://go-review.googlesource.com/c/go/+/231717
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
If a signal is sent to t.signal before the panic is triggered,
a panicking test may end up with "warning: no tests to run" because
the tRunner that invokes the test in t.Run calls runtime.Goexit on
panic, which causes the panicking test not be recorded in runTests.
Send the signal if and only if there is no panic.
Fixes#41479
Change-Id: I812f1303bfe02c443a1902732e68d21620d6672e
Reviewed-on: https://go-review.googlesource.com/c/go/+/256098
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
Trust: Bryan C. Mills <bcmills@google.com>
Previously, it was impossible to call FailNow in a Cleanup.
Because it can terminate a panicking goroutine and cause its
parent hangs on t.signal channel. This CL sends the signal
in a deferred call to prevent the hang.
Fixes#41355
Change-Id: I4552d3a7ea763ef86817bf9b50c0e37fb34bf20f
Reviewed-on: https://go-review.googlesource.com/c/go/+/254637
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
Ensures that calling TempDir() in either of Cleanup or Benchmark
doesn't cause test failures which were previously caused by the
created directory having been deleted after the first run, yet
we prevented the recreation of the directory due to our selection
of concurrency primitive sync.Once. This change recreates the
temporary directory if it doesn't exist, regardless of how
many times Cleanup and Benchmark are invoked.
Fixes#41062
Change-Id: I925d9f7207d7c369a193d1e17da7a59a586244a7
Reviewed-on: https://go-review.googlesource.com/c/go/+/251297
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
While debugging #40771, I realized that the chatty printer should only
ever print to a single io.Writer (normally os.Stdout). The other
Writer implementations in the chain write to local buffers, but if we
wrote a test's output to a local buffer, then we did *not* write it to
stdout and we should not store it as the most recently logged test.
Because the chatty printer should only ever print to one place, it
shouldn't receive an io.Writer as an argument — rather, it shouldn't
be used at all for destinations other than the main output stream.
On the other hand, when we flush the output buffer to stdout in the
top-level flushToParent call, it is important that we not allow some
other test's output to intrude between the test summary header and the
remainder of the test's output. cmd/test2json doesn't know how to
parse such an intrusion, and it's confusing to humans too.
No test because I couldn't reproduce the user-reported error without
modifying the testing package. (This behavior seems to be very
sensitive to output size and/or goroutine scheduling.)
Fixes#40771
Updates #38458
Change-Id: Ic19bf1d535672b096ba1c8583a3b74aab6d6d766
Reviewed-on: https://go-review.googlesource.com/c/go/+/249026
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
cmd/go.TestScript/test_main_twice demonstrates a program that invokes
(*M).Run twice in a row. If we only restore os.Exit(0) in m.afterOnce,
we will fail to restore it after the second run and fail the test
process despite both runs passing.
Updates #29062
Updates #23129
Change-Id: Id22ec68f1708e4583c8dda14a8ba0efae7178b85
Reviewed-on: https://go-review.googlesource.com/c/go/+/251262
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This catches cases where a test calls code that calls os.Exit(0),
thereby skipping all subsequent tests.
Fixes#29062
Change-Id: If9478972f40189e27623557e7141469ca4234d89
Reviewed-on: https://go-review.googlesource.com/c/go/+/250977
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
We could instead fix cmd/test2json to treat PAUSE lines as *not*
changing the active test name, but that seems like it would be more
confusing to humans, and also wouldn't fix tools that parse output
using existing builds of cmd/test2json.
Fixes#40657
Change-Id: I937611778f5b1e7dd1d6e9f44424d7e725a589ed
Reviewed-on: https://go-review.googlesource.com/c/go/+/248727
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Jean de Klerk <deklerk@google.com>
In #24929, we decided to stream chatty test output. It looks like,
foo_test.go:138: TestFoo/sub-1: hello from subtest 1
foo_test.go:138: TestFoo/sub-2: hello from subtest 2
In this CL, we refactor the output to be grouped by === CONT lines, preserving
the old test-file-before-log-line behavior:
=== CONT TestFoo/sub-1
foo_test.go:138 hello from subtest 1
=== CONT TestFoo/sub-2
foo_test.go:138 hello from subtest 2
This should remove a layer of verbosity from tests, and make it easier to group
together related lines. It also returns to a more familiar format (the
pre-streaming format), whilst still preserving the streaming feature.
Fixes#38458
Change-Id: Iaef94c580d69cdd541b2ef055aa004f50d72d078
Reviewed-on: https://go-review.googlesource.com/c/go/+/229085
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
We use a single parent directory for all temporary directories
created by a test so they're all kept together.
Fixes#38850
Change-Id: If8edae10c5136efcbcf6fd632487d198b9e3a868
Reviewed-on: https://go-review.googlesource.com/c/go/+/231958
Reviewed-by: Russ Cox <rsc@golang.org>
testing.M.Run has this bit of code:
if !flag.Parsed() {
flag.Parse()
}
It makes sense, and it's common knowledge for many Go developers that
test flags are automatically parsed by the time tests and benchmarks are
run. However, the docs didn't clarify that. The previous wording only
mentioned that flag.Parse isn't run before TestMain, which doesn't
necessarily mean that it's run afterwards.
Fixes#38952.
Change-Id: I85f7a9dce637a23c5cb9abc485d47415c1a1ca27
Reviewed-on: https://go-review.googlesource.com/c/go/+/232806
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Record the caller when Cleanup is called to report it with t.Log
instead of unhelpful line in testing.go.
Fixes#38800
Change-Id: I3136f5d92a0e5a48f8b32a2e13b2521bc91d72d1
Reviewed-on: https://go-review.googlesource.com/c/go/+/232237
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
For GOOS=windows the path separator characters '\' and ':' also need be
replaced.
Updates #38465
Change-Id: If7c8cf93058c87d7df6cda140e82fd76578fe699
Reviewed-on: https://go-review.googlesource.com/c/go/+/229837
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
ioutil.TempDir doesn't like path separators in its pattern. Modify
(*common).TempDir to replace path separators with underscores before
using the test name as a pattern for ioutil.TempDir.
Fixes#38465.
Change-Id: I9e8ae48b99648b2bf9f561762e845165aff01972
Reviewed-on: https://go-review.googlesource.com/c/go/+/229399
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This CL fixes a race condition if there are two subtests, and
one finishing but the other is panicking.
Fixes#37551
Change-Id: Ic33963eb338aec228964b95f7c34a0d207b91e00
Reviewed-on: https://go-review.googlesource.com/c/go/+/221322
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The documentation for m.Run says it returns an "exit code" to pass to
os.Exit. The argument to os.Exit is named "code".
While "exit code", "exit status" and "exit status code" are all valid ways
to refer to the same concept, prefer to stick to one form for consistency
and to avoid confusing users.
Change-Id: If76ee3fab5cc99c79e05ac1a4e413790a9c93d60
GitHub-Last-Rev: 85a081d2f03c2cf9e8e519916986c59c86aebf57
GitHub-Pull-Request: golang/go#37899
Reviewed-on: https://go-review.googlesource.com/c/go/+/223778
Reviewed-by: Gabriel Aszalos <gabriel.aszalos@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Gabriel Aszalos <gabriel.aszalos@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
If TestMain reports a wrong exit code to os.Exit, the test will be
exited with exist code inconsist with test results.
This CL eliminates the requirement of calling os.Exit in TestMain.
Now, m.Run records the execution status of its test, the outer
main func will call os.Exit with that exit code if TestMain does
not call os.Exit.
If TestMain does not call m.Run, the outer main func remain calls
os.Exit(0) as before.
Fixes#34129
Change-Id: I9598023e03b0a6260f0217f34df41c231c7d6489
Reviewed-on: https://go-review.googlesource.com/c/go/+/219639
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Change to rand.Int, a function that the compiler cannot reliably eliminate.
Fix output to actual benchmark values.
Fixes#37341
Change-Id: Ifb5bf49b826ae0bdb4bf9de5a472ad0eaa54569c
Reviewed-on: https://go-review.googlesource.com/c/go/+/220397
Reviewed-by: Ian Lance Taylor <iant@golang.org>
We now only accumulate logs when not using -v. Just drop the sentence
entirely rather than try to describe the current situation.
Updates #24929
Updates #37203
Change-Id: Ie3bf37894ab68b5b129eff54637893c7a129da03
Reviewed-on: https://go-review.googlesource.com/c/go/+/219540
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
It's good to be explicit, as it's not necessarily obvious (and indeed
the behavior has changed recently with https://go-review.googlesource.com/c/go/+/214822)
without an associated doc comment change).
Change-Id: I99d6398bf15b404b1b1b196e712e926e363251e9
Reviewed-on: https://go-review.googlesource.com/c/go/+/215217
Reviewed-by: Paul Jolly <paul@myitcv.org.uk>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The Perm function return 0 or 1 or 2 or 3. 4 is not returned,
so that changed the argument to 5.
Change-Id: Ic980c71a9f29f522bdeef4fce70a6c2dd136d791
Reviewed-on: https://go-review.googlesource.com/c/go/+/209777
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Flush the output log up to the root when a test panics. Prior to
this change, only the current test's output log was flushed to its
parent, resulting in no output when a subtest panics.
For the following test function:
func Test(t *testing.T) {
for i, test := range []int{1, 0, 2} {
t.Run(fmt.Sprintf("%v/%v", i, test), func(t *testing.T) {
_ = 1 / test
})
}
}
Output before this change:
panic: runtime error: integer divide by zero [recovered]
panic: runtime error: integer divide by zero
(stack trace follows)
Output after this change:
--- FAIL: Test (0.00s)
--- FAIL: Test/1/0 (0.00s)
panic: runtime error: integer divide by zero [recovered]
(stack trace follows)
Fixes#32121
Change-Id: Ifee07ccc005f0493a902190a8be734943123b6b7
Reviewed-on: https://go-review.googlesource.com/c/go/+/179599
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Fixes#24929
Change-Id: Icc426068cd73b75b78001f55e1e5d81ccebbe854
Reviewed-on: https://go-review.googlesource.com/c/go/+/127120
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This reverts CL 176098.
Reason for revert: added complexity, but did not completely fix the
underlying problem. A complete solution would not be worth the
complexity, and as a partial solution this is probably not worth the
complexity either.
Updates #31859
Change-Id: Ifd34c292fd1b811c60afe3c339e5edd3f37190c8
Reviewed-on: https://go-review.googlesource.com/c/go/+/186817
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Caleb Spare <cespare@gmail.com>
callerName requested 2 PCs from Callers, and that causes
both to be looked up in the file/line mapping.
We really only need to do the work for one PC.
(And in fact the caller doesn't need file/line at all, but
the Callers API can't express that.)
We used to request 2 PCs because in 1.11 and earlier we
stored an inline skip count in the second entry.
That's not necessary any more (as of 1.12).
Fixes#32093
Change-Id: I7b272626ef6496e848ee8af388cdaafd2556857b
Reviewed-on: https://go-review.googlesource.com/c/go/+/177858
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Caleb Spare <cespare@gmail.com>
CL 121936 added this diagnostic to avoid a panic accessing *short.
(Hence the "This shouldn't really be a panic" comment.)
That CL was right to produce a clearer error than a plain memory fault,
but I think wrong to print+exit instead of panicking. I just ran into
one of these in a real program, and there is no indication anywhere
of how the program reached this point. The panic will show that.
So change print+exit to a panic with a helpful message, in contrast
to the original panic with an unhelpful message and the current
helpful message without stack trace.
Change-Id: Ib2bae1dead4ccde92f00fa3a34c05241ff7690c6
Reviewed-on: https://go-review.googlesource.com/c/go/+/177419
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
In CL 173722, we moved the flag registration in the testing package into
an Init function. In order to avoid needing changes to user code, we
called Init automatically as part of testing.MainStart.
However, that isn't early enough if flag.Parse is called before the
tests run, as part of package initialization.
Fix this by injecting a bit of code to call testing.Init into test
packages. This runs before any other initialization code in the user's
test package, so testing.Init will be called before any user code can
call flag.Parse.
Fixes#31859
Change-Id: Ib42cd8d3819150c49a3cecf7eef2472319d0c7e9
Reviewed-on: https://go-review.googlesource.com/c/go/+/176098
Run-TryBot: Caleb Spare <cespare@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Any code that imports the testing package forces the testing flags to be
defined, even in non-test binaries. People work around this today by
defining a copy of the testing.TB interface just to avoid importing
testing.
Fix this by moving flag registration into a new function, testing.Init.
Delay calling Init until the testing binary begins to run, in
testing.MainStart.
Init is exported for cases where users need the testing flags to be
defined outside of a "go test" context. In particular, this may be
needed where testing.Benchmark is called outside of a test.
Fixes#21051
Change-Id: Ib7e02459e693c26ae1ba71bbae7d455a91118ee3
Reviewed-on: https://go-review.googlesource.com/c/go/+/173722
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
