In TimeoutHandler, use a request whose context has been configured with
the handler's timeout
Fixes#20712
Change-Id: Ie670148f85fdad46841ff29232042309e15665ae
Reviewed-on: https://go-review.googlesource.com/46412
Run-TryBot: Tom Bergan <tombergan@google.com>
Reviewed-by: Tom Bergan <tombergan@google.com>
This is another attempt at the change attempted in
https://golang.org/cl/27117 and rolled back in https://golang.org/cl/34134
The difference between this and the previous attempt is that this version only
retries if the new field GetBody is set on the Request.
Additionally, this allows retries of requests with idempotent methods even if
they have bodies, as long as GetBody is defined.
This also fixes an existing bug where readLoop could make a redundant call to
setReqCanceler for DELETE/POST/PUT/etc requests with no body with zero bytes
written.
This clarifies the existing TestRetryIdempotentRequestsOnError test (and changes
it into a test with 4 subtests). When that test was written, it was in fact
testing "retry idempotent requests" logic, but the logic had changed since then,
and it was actually testing "retry requests with no body when no bytes have been
written". (You can confirm this by changing the existing test from a GET to a
DELETE; it passes without the changes in this CL.) We now test for the no-Body
and GetBody cases for both idempotent and nothing-written-non-idempotent
requests.
Fixes#18241Fixes#17844
Change-Id: I69a48691796f6dc08c31f7aa7887b7dfd67e278a
Reviewed-on: https://go-review.googlesource.com/42142
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
It used to be simple, and then it got complicated for speed (to reduce
allocations, mostly), but that involved a mutex and hurt multi-core
performance, contending on the mutex.
A change was sent to try to improve that mutex contention in
https://go-review.googlesource.com/c/42110/2/src/net/http/server.go
but that introduced its own allocations (the string->interface{}
boxing for the sync.Map key), which runs counter to the whole point of
that statusLine function: to remove allocations.
Instead, make the code simple again and not have a mutex. It's a bit
slower for the single-core case, but nobody with a single-user HTTP
server cares about 50 nanoseconds:
name old time/op new time/op delta
ResponseStatusLine 37.5ns ± 2% 87.1ns ± 2% +132.42% (p=0.029 n=4+4)
ResponseStatusLine-2 63.1ns ± 1% 43.1ns ±12% -31.67% (p=0.029 n=4+4)
ResponseStatusLine-4 53.8ns ± 8% 40.2ns ± 2% -25.29% (p=0.029 n=4+4)
name old alloc/op new alloc/op delta
ResponseStatusLine 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
ResponseStatusLine-2 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
ResponseStatusLine-4 0.00B ±NaN% 0.00B ±NaN% ~ (all samples are equal)
name old allocs/op new allocs/op delta
ResponseStatusLine 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
ResponseStatusLine-2 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
ResponseStatusLine-4 0.00 ±NaN% 0.00 ±NaN% ~ (all samples are equal)
(Note the code could be even simpler with fmt.Fprintf, but that is
relatively slow and involves a bunch of allocations getting arguments
into interface{} for the call)
Change-Id: I1fa119132dbbf97a8e7204ce3e0707d433060da2
Reviewed-on: https://go-review.googlesource.com/42133
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
If I put a 10 millisecond sleep at testHookWaitResLoop, before the big
select in (*persistConn).roundTrip, two flakes immediately started
happening, TestTransportBodyReadError (#19231) and
TestTransportPersistConnReadLoopEOF.
The problem was that there are many ways for a RoundTrip call to fail
(errors reading from Request.Body while writing the response, errors
writing the response, errors reading the response due to server
closes, errors due to servers sending malformed responses,
cancelations, timeouts, etc.), and many of those failures then tear
down the TCP connection, causing more failures, since there are always
at least three goroutines involved (reading, writing, RoundTripping).
Because the errors were communicated over buffered channels to a giant
select, the error returned to the caller was a function of which
random select case was called, which was why a 10ms delay before the
select brought out so many bugs. (several fixed in my previous CLs the past
few days).
Instead, track the error explicitly in the transportRequest, guarded
by a mutex.
In addition, this CL now:
* differentiates between the two ways writing a request can fail: the
io.Copy reading from the Request.Body or the io.Copy writing to the
network. A new io.Reader type notes read errors from the
Request.Body. The read-from-body vs write-to-network errors are now
prioritized differently.
* unifies the two mapRoundTripErrorFromXXX methods into one
mapRoundTripError method since their logic is now the same.
* adds a (*Request).WithT(*testing.T) method in export_test.go, usable
by tests, to call t.Logf at points during RoundTrip. This is disabled
behind a constant except when debugging.
* documents and deflakes TestClientRedirectContext
I've tested this CL with high -count values, with/without -race,
with/without delays before the select, etc. So far it seems robust.
Fixes#19231 (TestTransportBodyReadError flake)
Updates #14203 (source of errors unclear; they're now tracked more)
Updates #15935 (document Transport errors more; at least understood more now)
Change-Id: I3cccc3607f369724b5344763e35ad2b7ea415738
Reviewed-on: https://go-review.googlesource.com/37495
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
- Added support for If-Match and If-Unmodified-Since.
- Precondition checks now more strictly follow RFC 7232 section 6, which
affects precedence when multiple condition headers are present.
- When serving a 304, Last-Modified header is now removed when no ETag is
present (as suggested by RFC 7232 section 4.1).
- If-None-Match supports multiple ETags.
- ETag comparison now correctly handles weak ETags.
Fixes#17572
Change-Id: I35039dea6811480ccf2889f8ed9c6a39ce34bfff
Reviewed-on: https://go-review.googlesource.com/32014
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Also updates x/net/http2 to git rev 541150 for:
http2: add support for graceful shutdown of Server
https://golang.org/cl/32412
http2: make http2.Server access http1's Server via an interface check
https://golang.org/cl/32417Fixes#4674Fixes#9478
Change-Id: I8021a18dee0ef2fe3946ac1776d2b10d3d429052
Reviewed-on: https://go-review.googlesource.com/32329
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Updates bundled http2 to x/net git rev a333c53 for:
http2: add Transport support for IdleConnTimeout
https://golang.org/cl/30075
And add tests.
The bundled http2 also includes a change adding a Ping method to
http2.ClientConn, but that type isn't exposed in the standard
library. Nevertheless, the code gets compiled and adds a dependency on
"crypto/rand", requiring an update to go/build's dependency
test. Because net/http already depends on crypto/tls, which uses
crypto/rand, it's not really a new dependency.
Fixes#16808
Change-Id: I1ec8666ea74762f27c70a6f30a366a6647f923f7
Reviewed-on: https://go-review.googlesource.com/30078
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Copy all of the original request's headers on redirect, unless they're
sensitive. Only send sensitive ones to the same origin, or subdomains
thereof.
Fixes#4800
Change-Id: Ie9fa75265c9d5e4c1012c028d31fd1fd74465712
Reviewed-on: https://go-review.googlesource.com/28930
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Reviewed-by: Francesc Campoy Flores <campoy@golang.org>
Reviewed-by: Ross Light <light@google.com>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Don't keep idle HTTP client connections open forever. Add a new knob,
Transport.IdleConnTimeout, and make the default be 90 seconds. I
figure 90 seconds is more than a minute, and less than infinite, and I
figure enough code has things waking up once a minute polling APIs.
This also removes the Transport's idleCount field which was unused and
redundant with the size of the idleLRU map (which was actually used).
Change-Id: Ibb698a9a9a26f28e00a20fe7ed23f4afb20c2322
Reviewed-on: https://go-review.googlesource.com/22670
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The HTTP client had a limit for the maximum number of idle connections
per-host, but not a global limit.
This CLs adds a global idle connection limit too,
Transport.MaxIdleConns.
All idle conns are now also stored in a doubly-linked list. When there
are too many, the oldest one is closed.
Fixes#15461
Change-Id: I72abbc28d140c73cf50f278fa70088b45ae0deef
Reviewed-on: https://go-review.googlesource.com/22655
Reviewed-by: Andrew Gerrand <adg@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
TimeoutHandler was starting the Timer when the handler was created,
instead of when serving a request. It also was sharing it between
multiple requests, which is incorrect, as the requests might start
at different times.
Store the timeout duration and create the Timer when ServeHTTP is
called. Different requests will have different timers.
The testing plumbing was simplified to store the channel used to
control when timeout happens. It overrides the regular timer.
Fixes#14568.
Change-Id: I4bd51a83f412396f208682d3ae5e382db5f8dc81
Reviewed-on: https://go-review.googlesource.com/20046
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This is a subset of https://golang.org/cl/20022 with only the copyright
header lines, so the next CL will be smaller and more reviewable.
Go policy has been single space after periods in comments for some time.
The copyright header template at:
https://golang.org/doc/contribute.html#copyright
also uses a single space.
Make them all consistent.
Change-Id: Icc26c6b8495c3820da6b171ca96a74701b4a01b0
Reviewed-on: https://go-review.googlesource.com/20111
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When the Transport was creating an bound HTTP connection (protocol
unknown initially) and then ends up deciding it doesn't need it, a
goroutine sits around to clean up whatever the result was. That
goroutine made the false assumption that the result was always an
HTTP/1 connection or an error. It may also be an alternate protocol
in which case the *persistConn.conn net.Conn field is nil, and the
alt field is non-nil.
Fixes#13839
Change-Id: Ia4972e5eb1ad53fa00410b3466d4129c753e0871
Reviewed-on: https://go-review.googlesource.com/18573
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
In debugging the flaky test in #13825, I discovered that my previous
change to tighten and simplify the communication protocol between
Transport.roundTrip and persistConn.readLoop in
https://golang.org/cl/17890 wasn't complete.
This change simplifies it further: the buffered-vs-unbuffered
complexity goes away, and we no longer need to re-try channel reads in
the select case. It was trying to prioritize channels in the case that
two were readable in the select. (it was only failing in the race builder
because the race builds randomize select scheduling)
The problem was that in the bodyless response case we had to return
the idle connection before replying to roundTrip. But putIdleConn
previously both added it to the free list (which we wanted), but also
closed the connection, which made the caller goroutine
(Transport.roundTrip) have two readable cases: pc.closech, and the
response. We guarded against similar conditions in the caller's select
for two readable channels, but such a fix wasn't possible here, and would
be overly complicated.
Instead, switch to unbuffered channels. The unbuffered channels were only
to prevent goroutine leaks, so address that differently: add a "callerGone"
channel closed by the caller on exit, and select on that during any unbuffered
sends.
As part of the fix, split putIdleConn into two halves: a part that
just returns to the freelist, and a part that also closes. Update the
four callers to the variants each wanted.
Incidentally, the connections were closing on return to the pool due
to MaxIdleConnsPerHost (somewhat related: #13801), but this bug
could've manifested for plenty of other reasons.
Fixes#13825
Change-Id: I6fa7136e2c52909d57a22ea4b74d0155fdf0e6fa
Reviewed-on: https://go-review.googlesource.com/18282
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>
The Transport had a delicate protocol between its readLoop goroutine
and the goroutine calling RoundTrip. The basic concern is that the
caller's RoundTrip goroutine wants to wait for either a
connection-level error (the conn dying) or the response. But sometimes
both happen: there's a valid response (without a body), but the conn
is also going away. Both goroutines' logic dealing with this had grown
large and complicated with hard-to-follow comments over the years.
Simplify and document. Pull some bits into functions and do all
bodyless stuff in one place (it's special enough), rather than having
a bunch of conditionals scattered everywhere. One test is no longer
even applicable since the race it tested is no longer possible (the
code doesn't exist).
The bug that this fixes is that when the Transport reads a bodyless
response from a server, it was returning that response before
returning the persistent connection to the idle pool. As a result,
~1/1000 of serial requests would end up creating a new connection
rather than re-using the just-used connection due to goroutine
scheduling chance. Instead, this now adds bodyless responses'
connections back to the idle pool first, then sends the response to
the RoundTrip goroutine, but making sure that the RoundTrip goroutine
is outside of its select on the connection dying.
There's a new buffered channel involved now, which is a minor
complication, but it's much more self-contained and well-documented
than the previous complexity. (The alternative of making the
responseAndError channel itself unbuffered is too invasive and risky
at this point; it would require a number of changes to avoid
deadlocked goroutines in error cases)
In any case, flakes look to be gone now. We'll see if trybots agree.
Fixes#13633
Change-Id: I95a22942b2aa334ae7c87331fddd751d4cdfdffc
Reviewed-on: https://go-review.googlesource.com/17890
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
New implementation of TimeoutHandler: buffer everything to memory.
All or nothing: either the handler finishes completely within the
timeout (in which case the wrapper writes it all), or it misses the
timeout and none of it gets written, in which case handler wrapper can
reliably print the error response without fear that some of the
wrapped Handler's code already wrote to the output.
Now the goroutine running the wrapped Handler has its own write buffer
and Header copy.
Document the limitations.
Fixes#9162
Change-Id: Ia058c1d62cefd11843e7a2fc1ae1609d75de2441
Reviewed-on: https://go-review.googlesource.com/17752
Reviewed-by: David Symonds <dsymonds@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
This makes TestTransportResponseCloseRace much faster and no longer
flaky.
In the process it also cleans up test hooks in net/http which were
inconsistent and scattered.
Change-Id: Ifd0b11dbc7e8915c24eb5bdc36731ed6751dd7ec
Reviewed-on: https://go-review.googlesource.com/17316
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
If we try to reuse a connection that the server is in the process of
closing, we may end up successfully writing out our request (or a
portion of our request) only to find a connection error when we try to
read from (or finish writing to) the socket. This manifests as an EOF
returned from the Transport's RoundTrip.
The issue, among others, is described in #4677.
This change follows some of the Chromium guidelines for retrying
idempotent requests only when the connection has been already been used
successfully and no header data has yet been received for the response.
As part of this change, an unexported error was defined for
errMissingHost, which was previously defined inline. errMissingHost is
the only non-network error returned from a Request's Write() method.
Additionally, this breaks TestLinuxSendfile because its test server
explicitly triggers the type of scenario this change is meant to retry
on. Because that test server stops accepting conns on the test listener
before the retry, the test would time out. To fix this, the test was
altered to use a non-idempotent test type (POST).
Change-Id: I1ca630b944f0ed7ec1d3d46056a50fb959481a16
Reviewed-on: https://go-review.googlesource.com/3210
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
This CL adds skipped failing tests, showing differences between HTTP/1
and HTTP/2 behavior. They'll be fixed in later commits.
Only a tiny fraction of the net/http tests have been split into their
"_h1" and "_h2" variants. That will also continue. (help welcome)
Updates #6891
Updates #13315
Updates #13316
Updates #13317
Change-Id: I16c3c381dbe267a3098fb266ab0d804c36473a64
Reviewed-on: https://go-review.googlesource.com/17046
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
It was generating the wrong error message, always defaulting to "500
Internal Server Error", since the err variable used was always nil.
Fixes#12991
Change-Id: I94b0e516409c131ff3b878bcb91e65f0259ff077
Reviewed-on: https://go-review.googlesource.com/16060
Reviewed-by: David Crawshaw <crawshaw@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The test was measuring something, assuming other goroutines had
already scheduled.
Fixes#10427
Change-Id: I2a4d3906f9d4b5ea44b57d972e303bbe2b0b1cde
Reviewed-on: https://go-review.googlesource.com/9561
Reviewed-by: David Crawshaw <crawshaw@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
In the brief window between getConn and persistConn.roundTrip,
a cancel could end up going missing.
Fix by making it possible to inspect if a cancel function was cleared
and checking if we were canceled before entering roundTrip.
Fixes#10511
Change-Id: If6513e63fbc2edb703e36d6356ccc95a1dc33144
Reviewed-on: https://go-review.googlesource.com/9181
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
There was a logical race in Transport.RoundTrip where a roundtrip with
a pending response would race with the channel for the connection
closing. This usually happened for responses with connection: close
and no body.
We handled this race by reading the close channel, setting a timer
for 100ms and if no response was returned before then, we would then
return an error.
This put a lower bound on how fast a connection could fail. We couldn't
fail a request faster than 100ms.
Reordering the channel operations gets rid of the logical race. If
the readLoop causes the connection to be closed, it would have put
its response into the return channel already and we can fetch it with
a non-blocking receive.
Change-Id: Idf09e48d7a0453d7de0120d3055d0ce5893a5428
Reviewed-on: https://go-review.googlesource.com/1787
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
http.Client calls URL.String() to fill in the Referer header, which may
contain authentication info. This patch removes authentication info from
the Referer header without introducing any API changes.
A new test for net/http is also provided.
This is the polished version of Alberto García Hierro's
https://golang.org/cl/9766046/
It should handle https Referer right.
Fixes#8417
LGTM=bradfitz
R=golang-codereviews, gobot, bradfitz, mikioh.mikioh
CC=golang-codereviews
https://golang.org/cl/151430043
See comment 4 of https://code.google.com/p/go/issues/detail?id=8483#c4:
"So if a user creates a http.Client, issues a bunch of
requests and then wants to shutdown it and all opened connections;
what is she intended to do? The report suggests that just waiting for
all pending requests and calling CloseIdleConnections won't do, as
there can be new racing connections. Obviously she can't do what
you've done in the test, as it uses the unexported function. If this
happens periodically, it can lead to serious resource leaks (the
transport is also preserved alive). Am I missing something?"
This CL tracks the user's intention to close all idle
connections (CloseIdleConnections sets it true; and making a
new request sets it false). If a pending dial finishes and
nobody wants it, before it's retained for a future caller, the
"wantIdle" bool is checked and it's closed if the user has
called CloseIdleConnections without a later call to make a new
request.
Fixes#8483
LGTM=adg
R=golang-codereviews, dvyukov, adg
CC=golang-codereviews, rsc
https://golang.org/cl/148970043
