251 Commits

Author SHA1 Message Date
Carl Johnson
55e6e825d4 net/http: add MaxBytesHandler
Fixes #39567

Change-Id: I226089b678a6a13d7ce69f360a23fc5bd297d550
GitHub-Last-Rev: 6435fd5881fc70a276d04df5a60440e365924b49
GitHub-Pull-Request: golang/go#48104
Reviewed-on: https://go-review.googlesource.com/c/go/+/346569
Trust: Damien Neil <dneil@google.com>
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
2021-11-09 18:23:16 +00:00
Damien Neil
ccea0b2fbe net/http: deflake TestTimeoutHandlerContextCanceled
Fixes #49448

Change-Id: Ie2acff7dedbca9bd1cc0b1b3dd0a01573c7befee
Reviewed-on: https://go-review.googlesource.com/c/go/+/361920
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2021-11-08 21:28:37 +00:00
jiahua wang
9e6ad46bcc net/http: fix spelling in documentation
Change-Id: I8b0924300eafe27de98975512a78a6527a92e446
Reviewed-on: https://go-review.googlesource.com/c/go/+/354729
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Damien Neil <dneil@google.com>
2021-11-07 04:57:22 +00:00
Charlie Getzen
4c7cafdd03 net/http: distinguish between timeouts and client hangups in TimeoutHandler
Fixes #48948

Change-Id: I411e3be99c7979ae289fd937388aae63d81adb59
GitHub-Last-Rev: 14abd7e4d774ed5ef63aa0a69e80fbc8b5a5af26
GitHub-Pull-Request: golang/go#48993
Reviewed-on: https://go-review.googlesource.com/c/go/+/356009
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Damien Neil <dneil@google.com>
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
2021-11-05 21:18:28 +00:00
Damien Neil
1011e26b9c net/http: deflake TestServerKeepAlivesEnabled_h{1,2}
This test assumes that two successive TCP connections will use different
source ports. This does not appear to be a universally safe assumption.

Rewrite the test to use httptrace to detect connection reuse instead.

Fixes #46707

Change-Id: Iebfbdfdeb77a1e6663a0c654dc847cc270c5d54d
Reviewed-on: https://go-review.googlesource.com/c/go/+/360854
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2021-11-02 19:37:42 +00:00
Damien Neil
770f1de8c5 net/http: remove test-only private key from production binaries
The net/http/internal package contains a PEM-encoded private key used in
tests. This key is initialized at init time, which prevents it from
being stripped by the linker in non-test binaries.

Move the certificate and key to a new net/http/internal/testcert
package to ensure it is only included in binaries that reference it.

Fixes #46677.

Change-Id: Ie98bda529169314cc791063e7ce4d99ef99113c8
Reviewed-on: https://go-review.googlesource.com/c/go/+/326771
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2021-06-10 20:20:58 +00:00
Filippo Valsorda
e4e7807d24 net/http: add AllowQuerySemicolons
Fixes #45973

Change-Id: I6cbe05f5d1d3c324900c74314b0ea0e12524d7f2
Reviewed-on: https://go-review.googlesource.com/c/go/+/326309
Run-TryBot: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
2021-06-09 16:59:02 +00:00
Roland Shoemaker
b584230889 net/http: use relative path in Location redirect
If the cleaned path did not match the requested path, ServeMux.Handler
would return a Location header which reflected the hostname in the
request, possibly leading to an incorrect redirect. Instead the
Location header should be relative, like the other cases in
ServeMux.Handler.

Change-Id: I2c220d925e708061bc128f0bdc96cca7a32764d3
Reviewed-on: https://go-review.googlesource.com/c/go/+/313950
Trust: Roland Shoemaker <roland@golang.org>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Roland Shoemaker <roland@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
2021-05-03 17:34:10 +00:00
Damien Neil
7ece3a7b17 net/http: fix flaky TestDisableKeepAliveUpgrade
This test hijacks a connection. It was reading from the net.Conn
returned by Hijack, not the bufio.ReadWriter, causing flaky failures
when a read-ahead byte was held in the read buffer.

Fixes #43073.

Change-Id: Ic3e7f704fba9635fd851cb3c0c0c74e312b75f6e
Reviewed-on: https://go-review.googlesource.com/c/go/+/285596
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Anmol Sethi <nhooyr@gmail.com>
2021-01-22 16:23:59 +00:00
Anmol Sethi
c81343ce3a net/http: attempt deadlock fix in TestDisableKeepAliveUpgrade
1. The test now checks the response status code.
2. The transport has been changed to not set "Connection: Close" if
   DisableKeepAlive is set and the request is a HTTP/1.1 protocol
   upgrade.

Updates #43073

Change-Id: I9977a18b33b8747ef847a8d11bb7b4f2d8053b8c
GitHub-Last-Rev: f809cebb139df4f5560a8456973351c95a3dfa97
GitHub-Pull-Request: golang/go#43086
Reviewed-on: https://go-review.googlesource.com/c/go/+/276375
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
2020-12-14 19:19:09 +00:00
Anmol Sethi
50b16f9de5 net/http: allow upgrading non keepalive connections
If one was using http.Transport with DisableKeepAlives and trying
to upgrade a connection against net/http's Server, the Server
would not allow a "Connection: Upgrade" header to be written
and instead override it to "Connection: Close" which would
break the handshake.

This change ensures net/http's Server does not override the
connection header for successful protocol switch responses.

Fixes #36381.

Change-Id: I882aad8539e6c87ff5f37c20e20b3a7fa1a30357
GitHub-Last-Rev: dc0de83201dc26236527b68bd49dffc53dd0389b
GitHub-Pull-Request: golang/go#36382
Reviewed-on: https://go-review.googlesource.com/c/go/+/213277
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
Trust: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-12-01 19:47:12 +00:00
Russ Cox
1b09d43067 all: update references to symbols moved from io/ioutil to io
The old ioutil references are still valid, but update our code
to reflect best practices and get used to the new locations.

Code compiled with the bootstrap toolchain
(cmd/asm, cmd/dist, cmd/compile, debug/elf)
must remain Go 1.4-compatible and is excluded.
Also excluded vendored code.

For #41190.

Change-Id: I6d86f2bf7bc37a9d904b6cee3fe0c7af6d94d5b1
Reviewed-on: https://go-review.googlesource.com/c/go/+/263142
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2020-10-20 18:41:18 +00:00
Russ Cox
59202c4204 net/http: deflake TestServerEmptyBodyRace_h1, or at least try
Fixes #22540.
For #33585.

Change-Id: I504b5a91ce1a39cd4ffd2380178a1b8f82f49dd3
Reviewed-on: https://go-review.googlesource.com/c/go/+/261698
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-10-16 00:59:55 +00:00
Damien Neil
9c56300e62 net/http: return 505 status for rejected protocol version
When rejecting a request with an unsupported HTTP protocol version,
return a 505 error ("HTTP Version Not Supported") instead of 400.

Fixes #40454.

Change-Id: I0269f0f5755d90d1b772ba0094a6bb24b5eb4701
Reviewed-on: https://go-review.googlesource.com/c/go/+/261977
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Urban Ishimwe <urbainishimwe@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-10-14 18:05:16 +00:00
Michael Fraenkel
617f2c3e35 net/http: mark http/2 connections active
On Server.Shutdown, all idle connections are closed.
A caveat for new connections is that they are marked idle
after 5 seconds.
Previously new HTTP/2 connections were marked New, and after 5 seconds,
they would then become idle. With this change, we now mark HTTP/2
connections as Active to allow the proper shutdown sequence to occur.

Fixes #36946
Fixes #39776

Change-Id: I31efbf64b9a2850ca544da797f86d7e1b3378e8b
Reviewed-on: https://go-review.googlesource.com/c/go/+/240278
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-09-06 17:26:55 +00:00
Andrew Gerrand
bb54a855a9 net/http: handle Request.URL.RawPath in StripPrefix
The StripPrefix wrapper strips a prefix string from the request's
URL.Path field, but doesn't touch the RawPath field. This leads to the
confusing situation when StripPrefix handles a request with URL.RawPath
populated (due to some escaped characters in the request path) and the
wrapped request's RawPath contains the prefix but Path does not.

This change modifies StripPrefix to strip the prefix from both Path and
RawPath. If there are escaped characters in the prefix part of the
request URL the stripped handler serves a 404 instead of invoking the
underlying handler with a mismatched Path/RawPath pair.

This is a backward incompatible change for a very small minority of
requests; I would be surprised if anyone is depending on this behavior,
but it is possible. If that's the case, we could make a more
conservative change where the RawPath is trimmed if possible, but when
the prefix contains escaped characters then we don't 404 but rather send
through the invalid Path/RawPath pair as before.

Fixes #24366

Change-Id: I7030b8c183a3dfce307bc0272bba9a18df4cfe08
Reviewed-on: https://go-review.googlesource.com/c/go/+/233637
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2020-08-25 06:01:11 +00:00
Filippo Valsorda
d5734d4f2d net/http: only support "chunked" in inbound Transfer-Encoding headers
This is a security hardening measure against HTTP request smuggling.
Thank you to ZeddYu for reporting this issue.

We weren't parsing things correctly anyway, allowing "identity" to be
combined with "chunked", and ignoring any Transfer-Encoding header past
the first. This is a delicate security surface that already broke
before, just be strict and don't add complexity to support cases not
observed in the wild (nginx removed "identity" support [1] and multiple
TE header support [2]) and removed by RFC 7230 (see page 81).

It'd probably be good to also drop support for anything other than
"chunked" in outbound TE headers, as "identity" is not a thing anymore,
and we are probably off-spec for anything other than "chunked", but it
should not be a security concern, so leaving it for now. See #38867.

[1]: https://hg.nginx.org/nginx/rev/fe5976aae0e3
[2]: https://hg.nginx.org/nginx/rev/aca005d232ff

Change-Id: If17d0827f9c6167a0b19a158e2bc5844ec803288
Reviewed-on: https://go-review.googlesource.com/c/go/+/231418
Reviewed-by: Katie Hockman <katie@golang.org>
2020-05-06 16:25:30 +00:00
Brad Fitzpatrick
5a75f7c0b0 net/http: fix Server.Shutdown race where it could miss an active connection
Wait for Listeners to drop to zero too, not just conns.

Fixes #33313

Change-Id: I09350ae38087990d368dcf9302fbde3e95c02fcd
Reviewed-on: https://go-review.googlesource.com/c/go/+/213442
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hasit Bhatt <hasit.p.bhatt@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-04-21 23:23:30 +00:00
Bryan C. Mills
a5d1a9df81 net/http: remove arbitrary timeouts from TestIdentityResponse and TestTLSHandshakeTimeout
These hard-coded timeouts make the tests flaky on slow builders (such
as solaris-amd64-oraclerel), and make test failures harder to diagnose
anyway (by replacing dumps of the stuck goroutine stacks with failure
messages that do not describe the stuck goroutines). Eliminate them
and simplify the tests.

Fixes #37327
Fixes #38112

Change-Id: Id40febe349d134ef53c702e36199bfbf2b6468ff
Reviewed-on: https://go-review.googlesource.com/c/go/+/225977
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2020-03-27 18:16:46 +00:00
Bryan C. Mills
0e3ace42f5 net/http: use t.Deadline instead of an arbitrary timeout in TestServerConnState
Updates #37322

Change-Id: I3b8369cd9e0ed5e4b3136cedaa2f70698ead2270
Reviewed-on: https://go-review.googlesource.com/c/go/+/222957
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
2020-03-11 15:55:54 +00:00
Ziheng Liu
42f8199290 all: fix incorrect channel and API usage in some unit tests
This CL changes some unit test functions, making sure that these tests (and goroutines spawned during test) won't block.
Since they are just test functions, I use one CL to fix them all. I hope this won't cause trouble to reviewers and can save time for us.
There are three main categories of incorrect logic fixed by this CL:
1. Use testing.Fatal()/Fatalf() in spawned goroutines, which is forbidden by Go's document.
2. Channels are used in such a way that, when errors or timeout happen, the test will be blocked and never return.
3. Channels are used in such a way that, when errors or timeout happen, the test can return but some spawned goroutines will be leaked, occupying resource until all other tests return and the process is killed.

Change-Id: I3df931ec380794a0cf1404e632c1dd57c65d63e8
Reviewed-on: https://go-review.googlesource.com/c/go/+/219380
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-02-27 19:04:17 +00:00
Bryan C. Mills
931fe39400 net/http: await state traces earlier in TestServerConnState
This approach attempts to ensure that the log for each connection is
complete before the next sequence of states begins.

Updates #32329

Change-Id: I25150d3ceab6568af56a40d2b14b5f544dc87f61
Reviewed-on: https://go-review.googlesource.com/c/go/+/210717
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-12-10 18:44:22 +00:00
Emmanuel T Odeke
bff4ebb7aa net/http: unflake TestTimeoutHandlerSuperfluousLogs
Uses 2 channels to synchronize that test, because
relying on sleeps creates flaky behavior, thus:

a) 1 buffered channel to send back the last spurious line
without having to reason about "happens before" behavior
a) 1 buffered channel at the end of the handler; it'll
be controlled by whether we expect to timeout or not,
but will always be closed when the test ends

Fixes #35051

Change-Id: Iff735aa8d1ed9de8d92b792374ec161cc0a72798
Reviewed-on: https://go-review.googlesource.com/c/go/+/208477
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-07 00:46:38 +00:00
Dmitri Shuralyov
6ba1bf393a net/http: rename tests for Redirect and StripPrefix
Before May 2018, I mistakenly thought the _suffix naming convention¹
used by examples also applied to tests. Thanks to a code review comment²
from Ian Lance Taylor, I have since learned that is not true.
This trivial change fixes some collateral damage from my earlier
misunderstanding, resulting in improved test naming consistency.

¹ https://golang.org/pkg/testing/#hdr-Examples
² https://go-review.googlesource.com/c/go/+/112935/1/src/path/filepath/path_test.go#1075

Change-Id: I555f60719629eb64bf2f096aa3dd5e00851827cd
Reviewed-on: https://go-review.googlesource.com/c/go/+/207446
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-22 23:23:13 +00:00
Roman Kollár
bbbc6589df net/http: fix Server.ConnContext modifying context for all new connections
Fixes #35750

Change-Id: I65d38cfc5ddd66131777e104c269cc3559b2471d
GitHub-Last-Rev: 953fdfd49b2be665be43f8148d2a6180dae3b91c
GitHub-Pull-Request: golang/go#35751
Reviewed-on: https://go-review.googlesource.com/c/go/+/208318
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-11-21 23:15:28 +00:00
Ville Skyttä
440f7d6404 all: fix a bunch of misspellings
Change-Id: I5b909df0fd048cd66c5a27fca1b06466d3bcaac7
GitHub-Last-Rev: 778c5d21311abee09a5fbda2e4005a5fd4cc3f9f
GitHub-Pull-Request: golang/go#35624
Reviewed-on: https://go-review.googlesource.com/c/go/+/207421
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-11-15 21:04:43 +00:00
Brad Fitzpatrick
2566e21f24 net/http: support disabling built-in HTTP/2 with a new build tag
Fixes #35082
Updates #6853

Change-Id: I4eeb0e15f534cff57fefb6039cd33fadf15b946e
Reviewed-on: https://go-review.googlesource.com/c/go/+/205139
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: David Crawshaw <crawshaw@golang.org>
2019-11-04 23:16:09 +00:00
Emmanuel T Odeke
ff4e0e42d8 net/http: make TimeoutHandler log spurious WriteHeader calls
Makes TimeoutHandler consistent with other handlers, by
logging any spurious WriteHeader calls.

Fixes #30803

Change-Id: I693fbdf8378f31bca13d579eece8e8e00eb175bf
Reviewed-on: https://go-review.googlesource.com/c/go/+/200518
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-21 18:38:15 +00:00
Emmanuel T Odeke
829fae3b5e net/http: update bundled x/net/http2
Updates x/net/http2 to git rev d66e71096ffb9f08f36d9aefcae80ce319de6d68

    http2: end stream eagerly after sending the request body
    https://golang.org/cl/181157 (fixes #32254)

    all: fix typos
    https://golang.org/cl/193799

    http2: fix memory leak in random write scheduler
    https://golang.org/cl/198462 (fixes #33812)

    http2: do not sniff body if Content-Encoding is set
    https://golang.org/cl/199841 (updates #31753)

Also unskips tests from CL 199799.

Change-Id: I241c0b1cd18cad5041485be92809137a973e33bd
Reviewed-on: https://go-review.googlesource.com/c/go/+/200102
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-10-09 20:09:29 +00:00
Emmanuel T Odeke
e24a628ab1 net/http: do not sniff response if Content-Encoding header is set
Fixes #31753

Change-Id: I32ec5906ef6714e19b094f67cb0f10a211a9c500
Reviewed-on: https://go-review.googlesource.com/c/go/+/199799
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-10-09 19:49:39 +00:00
Emmanuel T Odeke
55738850c4 net/http: remove TestTimeoutHandlerAndFlusher due to flakes
Removes TestTimeoutHandlerAndFlusher due to flakes on
one of the builders due to timing issues.

Perhaps later, we might need to bring it back when we've
figured out the timing issues.

Fixes #34573.

Change-Id: Ia88d4da31fb228296144dc31f9a4288167fb4a53
Reviewed-on: https://go-review.googlesource.com/c/go/+/197757
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-27 15:38:16 +00:00
Emmanuel T Odeke
4faf8a8dc4 net/http, doc/go1.13.html: revert TimeoutHandler.Flush
Also added a test to ensure that any interactions
between TimeoutHandler and Flusher result in the
correct status code and body, but also that we don't
get superfluous logs from stray writes as was seen
in the bug report.

Fixes #34439.

Change-Id: I4af62db256742326f9353f98a2fcb5f71d2a5fd9
Reviewed-on: https://go-review.googlesource.com/c/go/+/197659
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-09-26 21:12:34 +00:00
Filippo Valsorda
41b1f88efa net/textproto: don't normalize headers with spaces before the colon
RFC 7230 is clear about headers with a space before the colon, like

X-Answer : 42

being invalid, but we've been accepting and normalizing them for compatibility
purposes since CL 5690059 in 2012.

On the client side, this is harmless and indeed most browsers behave the same
to this day. On the server side, this becomes a security issue when the
behavior doesn't match that of a reverse proxy sitting in front of the server.

For example, if a WAF accepts them without normalizing them, it might be
possible to bypass its filters, because the Go server would interpret the
header differently. Worse, if the reverse proxy coalesces requests onto a
single HTTP/1.1 connection to a Go server, the understanding of the request
boundaries can get out of sync between them, allowing an attacker to tack an
arbitrary method and path onto a request by other clients, including
authentication headers unknown to the attacker.

This was recently presented at multiple security conferences:
https://portswigger.net/blog/http-desync-attacks-request-smuggling-reborn

net/http servers already reject header keys with invalid characters.
Simply stop normalizing extra spaces in net/textproto, let it return them
unchanged like it does for other invalid headers, and let net/http enforce
RFC 7230, which is HTTP specific. This loses us normalization on the client
side, but there's no right answer on the client side anyway, and hiding the
issue sounds worse than letting the application decide.

Fixes CVE-2019-16276
Fixes #34540

Change-Id: I6d272de827e0870da85d93df770d6a0e161bbcf1
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/549719
Reviewed-by: Brad Fitzpatrick <bradfitz@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/197503
Run-TryBot: Filippo Valsorda <filippo@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-09-26 16:43:06 +00:00
Russ Cox
fbaf881cc6 net/http: fix Transport.MaxConnsPerHost limits & idle pool races
There were at least three races in the implementation of the pool of
idle HTTP connections before this CL.

The first race is that HTTP/2 connections can be shared for many
requests, but each requesting goroutine would take the connection out
of the pool and then immediately return it before using it; this
created unnecessary, tiny little race windows during which another
goroutine might dial a second connection instead of reusing the first.
This CL changes the idle pool to just leave the HTTP/2 connection in
the pool permanently (until there is reason to close it), instead of
doing the take-it-out-put-it-back dance race.

The second race is that “is there an idle connection?” and
“register to wait for an idle connection” were implemented as two
separate steps, in different critical sections. So a client could end
up registered to wait for an idle connection and be waiting or perhaps
dialing, not having noticed the idle connection sitting in the pool
that arrived between the two steps.

The third race is that t.getIdleConnCh assumes that the inability to
send on the channel means the client doesn't need the result, when it
could mean that the client has not yet entered the select.
That is, the main dial does:

	idleConnCh := t.getIdleConnCh(cm)
	select {
	case v := <-dialc:
		...
	case pc := <-idleConnCh
		...
	...
	}

But then tryPutIdleConn does:

	waitingDialer := t.idleConnCh[key] // what getIdleConnCh(cm) returned
	select {
	case waitingDialer <- pconn:
		// We're done ...
		return nil
	default:
		if waitingDialer != nil {
			// They had populated this, but their dial won
			// first, so we can clean up this map entry.
			delete(t.idleConnCh, key)
		}
	}

If the client has returned from getIdleConnCh but not yet reached the
select, tryPutIdleConn will be unable to do the send, incorrectly
conclude that the client does not care anymore, and put the connection
in the idle pool instead, again leaving the client dialing unnecessarily
while a connection sits in the idle pool.

(It's also odd that the success case does not clean up the map entry,
and also that the map has room for only a single waiting goroutine for
a given host.)

None of these races mattered too much before Go 1.11: at most they
meant that connections were not reused quite as promptly as possible,
or a few more than necessary would be created. But Go 1.11 added
Transport.MaxConnsPerHost, which limited the number of connections
created for a given host. The default is 0 (unlimited), but if a user
did explicitly impose a low limit (2 is common), all these misplaced
conns could easily add up to the entire limit, causing a deadlock.
This was causing intermittent timeouts in TestTransportMaxConnsPerHost.

The addition of the MaxConnsPerHost support added its own races.

For example, here t.incHostConnCount could increment the count
and return a channel ready for receiving, and then the client would
not receive from it nor ever issue the decrement, because the select
need not evaluate these two cases in order:

	select {
	case <-t.incHostConnCount(cmKey):
		// count below conn per host limit; proceed
	case pc := <-t.getIdleConnCh(cm):
		if trace != nil && trace.GotConn != nil {
			trace.GotConn(httptrace.GotConnInfo{Conn: pc.conn, Reused: pc.isReused()})
		}
		return pc, nil
	...
	}

Obviously, unmatched increments are another way to get to a deadlock.
TestTransportMaxConnsPerHost deadlocked approximately 100% of
the time with a small random sleep added between incHostConnCount
and the select:

	ch := t.incHostConnCount(cmKey):
	time.Sleep(time.Duration(rand.Intn(10))*time.Millisecond)
	select {
	case <-ch
		// count below conn per host limit; proceed
	case pc := <-t.getIdleConnCh(cm):
		...
	}

The limit also did not properly apply to HTTP/2, because of the
decrement being attached to the underlying net.Conn.Close
and net/http not having access to the underlying HTTP/2 conn.
The alternate decrements for HTTP/2 may also have introduced
spurious decrements (discussion in #29889). Perhaps those
spurious decrements or other races caused the other intermittent
non-deadlock failures in TestTransportMaxConnsPerHost,
in which the HTTP/2 phase created too many connections (#31982).

This CL replaces the buggy, racy code with new code that is hopefully
neither buggy nor racy.

Fixes #29889.
Fixes #31982.
Fixes #32336.

Change-Id: I0dfac3a6fe8a6cdf5f0853722781fe2ec071ac97
Reviewed-on: https://go-review.googlesource.com/c/go/+/184262
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-07-08 14:31:42 +00:00
Brad Fitzpatrick
4c84d87813 net/http: support BaseContext & ConnContext for http2 Server
This is the net/http half of #32476. This supplies the method needed
by the other half in x/net/http2 in the already-submitted CL 181259,
which this CL also bundles in h2_bundle.go.

Thanks to Tom Thorogood (@tmthrgd) for the bug report and test.

Fixes #32476
Updates #30694

Change-Id: I79d2a280e486fbf75d116f6695fd3abb61278765
Reviewed-on: https://go-review.googlesource.com/c/go/+/181260
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-06-07 18:48:19 +00:00
Dmitri Shuralyov
003dbc4cda net/http: roll back "clean the path of the stripped URL by StripPrefix"
Roll back CL 161738. That fix changed StripPrefix behavior in the
general case, not just in the situation where where stripping the
prefix from path resulted in the empty string, causing issue #31622.

That kind of change to StripPrefix behavior is not backwards compatible,
and there can be a smaller, more targeted fix for the original issue.

Fixes #31622
Updates #30165

Change-Id: Ie2fcfe6787a32e44f71d564d8f9c9d580fc6f704
Reviewed-on: https://go-review.googlesource.com/c/go/+/180498
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-06-04 22:36:51 +00:00
Brad Fitzpatrick
770746af1c net/http: quiet some log spam in tests
One of these tests creates a bunch of connections concurrently, then
discovers it doesn't need them all, which then makes the server log
that the client went away midway through the TLS handshake. Perhaps
the server should recognize that as a case not worthy of logging
about, but this is a safer way to eliminate the stderr spam during go
test for now.

The other test's client gives up on its connection and closes it,
similarly confusing the server.

Change-Id: I49ce442c9a63fc437e58ca79f044aa76e8c317b5
Reviewed-on: https://go-review.googlesource.com/c/go/+/179179
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2019-05-28 18:17:57 +00:00
Russ Cox
06b0babf31 all: shorten some tests
Shorten some of the longest tests that run during all.bash.
Removes 7r 50u 21s from all.bash.

After this change, all.bash is under 5 minutes again on my laptop.

For #26473.

Change-Id: Ie0460aa935808d65460408feaed210fbaa1d5d79
Reviewed-on: https://go-review.googlesource.com/c/go/+/177559
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-05-22 12:54:00 +00:00
Emmanuel T Odeke
88548d0211 net/http: make Server return 501 for unsupported transfer-encodings
Ensures that our HTTP/1.X Server properly responds
with a 501 Unimplemented as mandated by the spec at
RFC 7230 Section 3.3.1, which says:
    A server that receives a request message with a
    transfer coding it does not understand SHOULD
    respond with 501 (Unimplemented).

Fixes #30710

Change-Id: I096904e6df053cd1e4b551774cc27523ff3d09f6
Reviewed-on: https://go-review.googlesource.com/c/go/+/167017
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-04-30 19:23:58 +00:00
Brad Fitzpatrick
2c802e9980 net/http: add Server BaseContext & ConnContext fields to control early context
Fixes golang/go#30694

Change-Id: I12a0a870e4aee6576e879d88a4868666ef448298
Reviewed-on: https://go-review.googlesource.com/c/go/+/167681
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: JP Sugarbroad <jpsugar@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-04-15 19:15:11 +00:00
Leon Klingele
62bfa69e6e net/http: add missing error checks in tests
Change-Id: I73441ba2eb349f0e0f25068e6b24c74dd33f1456
GitHub-Last-Rev: b9e6705962b94af3b1b720cc9ad6d33d7d3f1425
GitHub-Pull-Request: golang/go#30017
Reviewed-on: https://go-review.googlesource.com/c/go/+/160441
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-03-11 21:40:21 +00:00
Ggicci
ea65d015b8 net/http: clean the path of the stripped URL by StripPrefix
The path of the new stripped URL should also be cleaned. Since an empty path
may cause unexpected errors in some HTTP handlers, e.g. http.ServeFile.

Fixes #30165

Change-Id: Ib44fdce6388b5d62ffbcab5266925ef8f13f26e2
Reviewed-on: https://go-review.googlesource.com/c/161738
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-02-26 22:55:59 +00:00
Brad Fitzpatrick
c942191c20 crypto/tls, net/http: reject HTTP requests to HTTPS server
This adds a crypto/tls.RecordHeaderError.Conn field containing the TLS
underlying net.Conn for non-TLS handshake errors, and then uses it in
the net/http Server to return plaintext HTTP 400 errors when a client
mistakenly sends a plaintext HTTP request to an HTTPS server. This is the
same behavior as Apache.

Also in crypto/tls: swap two error paths to not use a value before
it's valid, and don't send a alert record when a handshake contains a
bogus TLS record (a TLS record in response won't help a non-TLS
client).

Fixes #23689

Change-Id: Ife774b1e3886beb66f25ae4587c62123ccefe847
Reviewed-on: https://go-review.googlesource.com/c/143177
Reviewed-by: Filippo Valsorda <filippo@golang.org>
2018-10-24 22:49:50 +00:00
Daniel Martí
112f28defc io: export StringWriter
And start using it elsewhere in the standard library, removing the
copies in the process.

While at it, rewrite the io.WriteString godoc to be more clear, since it
can now make reference to the defined interface.

Fixes #27946.

Change-Id: Id5ba223c09c19e5fb49815bd3b1bd3254fc786f3
Reviewed-on: https://go-review.googlesource.com/c/139457
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-10-03 20:13:35 +00:00
Brad Fitzpatrick
da0d1a44ba all: use strings.ReplaceAll and bytes.ReplaceAll where applicable
I omitted vendor directories and anything necessary for bootstrapping.
(Tested by bootstrapping with Go 1.4)

Updates #27864

Change-Id: I7d9b68d0372d3a34dee22966cca323513ece7e8a
Reviewed-on: https://go-review.googlesource.com/137856
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-09-26 22:14:25 +00:00
Daniel Martí
9e4d87d115 all: update stale test skips
Issues #10043, #15405, and #22660 appear to have been fixed, and
whatever tests I could run locally do succeed, so remove the skips.

Issue #7237 was closed in favor of #17906, so update its skip line.

Issue #7634 was closed as it had not appeared for over three years.
Re-enable it for now. An issue should be open if the test starts being
skipped again.

Change-Id: I67daade906744ed49223291035baddaad9f56dca
Reviewed-on: https://go-review.googlesource.com/121735
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-08-20 13:10:24 +00:00
Brad Fitzpatrick
d3c3aaa61f net/http: revert CL 89275 (don't sniff Content-Type when nosniff set)
Also updates the bundled http2 to x/net/http2 git rev 49c15d80 for:

   http2: revert CL 107295 (don't sniff Content-type in Server when nosniff)
   https://golang.org/cl/126895

Fixes #24795

Change-Id: I6ae1a21c919947089274e816eb628d20490f83ce
Reviewed-on: https://go-review.googlesource.com/126896
Reviewed-by: Damien Neil <dneil@google.com>
2018-07-31 17:29:58 +00:00
Brad Fitzpatrick
d10ab13c18 net/http: expand a TimeoutHandler test a bit
Updates #22821

Change-Id: I2d0d483538174a90f56c26d99bea89fe9ce4d144
Reviewed-on: https://go-review.googlesource.com/125855
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
2018-07-31 15:07:39 +00:00
Brad Fitzpatrick
00ebfcde7f net/http: don't cancel Request.Context on pipelined Server requests
See big comment in code.

Fixes #23921

Change-Id: I2dbd1acc2e9da07a71f9e0640aafe0c59a335627
Reviewed-on: https://go-review.googlesource.com/123875
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-07-13 23:44:42 +00:00
Daniel Martí
96186a58e3 net/http: deflake TestServerShutdownStateNew
This function tests that calling Shutdown on a Server that has a "new"
connection yet to write any bytes, in which case it should wait for five
seconds until considering the connection as "idle".

However, the test was flaky. If Shutdown happened to run before the
server accepted the connection, the connection would immediately be
rejected as the server is already closed, as opposed to being accepted
in the "new" state. Then, Shutdown would return almost immediately, as
it had no connections to wait for:

	--- FAIL: TestServerShutdownStateNew (2.00s)
	    serve_test.go:5603: shutdown too soon after 49.41µs
	    serve_test.go:5617: timeout waiting for Read to unblock

Fix this by making sure that the connection has been accepted before
calling Shutdown. Verified that the flake is gone after 50k concurrent
runs of the test with no failures, whereas the test used to fail around
10% of the time on my laptop:

	go test -c && stress -p 256 ./http.test -test.run TestServerShutdownStateNew

Fixes #26233.

Change-Id: I819d7eedb67c48839313427675facb39d9c17257
Reviewed-on: https://go-review.googlesource.com/122355
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-07-06 02:13:27 +00:00