It is better to have a single implementation of IsAbs, which is quite
tricky to get right on Windows.
Change-Id: I45933b0ceff2920d9eddb61e62aacb2602c3dc8c
Reviewed-on: https://go-review.googlesource.com/c/go/+/582498
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Before, all methods of File (including Close) were
safe for concurrent use (I checked), except the three
variants of ReadDir.
This change makes the ReadDir operations
atomic too, and documents explicitly that all methods
of File have this property, which was already implied
by the package documentation.
Fixes#66498
Change-Id: I05c88b4e60b44c702062e99ed8f4a32e7945927a
Reviewed-on: https://go-review.googlesource.com/c/go/+/578322
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
(This CL takes the tests and some ideas from the abandoned CL 263538).
fixLongPath is used on Windows to process all path names
before syscalls to switch them to extended-length format
(with prefix \\?\) to workaround a historical limit
of 260-ish characters.
This CL updates fixLongPath to convert relative paths to absolute
paths if the working directory plus the relative path exceeds
MAX_PATH. This is necessary because the Windows API does not
support extended-length paths for relative paths.
This CL also adds support for fixing device paths (\\.\-prefixed),
which were not previously normalized.
Fixes#41734Fixes#21782Fixes#36375
Cq-Include-Trybots: luci.golang.try:gotip-windows-amd64-longtest,gotip-windows-amd64-race,gotip-windows-arm64
Co-authored-by: Giovanni Bajo <rasky@develer.com>
Change-Id: I63cfb79f3ae6b9d42e07deac435b730d97a6f492
Reviewed-on: https://go-review.googlesource.com/c/go/+/574695
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
This will fail because epoll_ctl() fails on directory FDs, so we
end up issuing unnecessary syscalls. My test program that calls
filepath.WalkDir on a large directory tree runs 1.23 ± 0.04 times
faster than with the original implementation.
Change-Id: Ie33d798c48057a7b2d0bacac80fcdde5b5a8bb1b
Reviewed-on: https://go-review.googlesource.com/c/go/+/570877
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Fixes#66215
Change-Id: Id7de15feabe08f66c048dc114c09494813c9febc
Reviewed-on: https://go-review.googlesource.com/c/go/+/570695
Reviewed-by: Cherry Mui <cherryyz@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
This CL updates os.Readlink so it no longer tries to normalize volumes
to drive letters, which was not always even possible.
This behavior is controlled by the `winreadlinkvolume` setting.
For Go 1.23, it defaults to `winreadlinkvolume=1`.
Previous versions default to `winreadlinkvolume=0`.
Fixes#63703.
Cq-Include-Trybots: luci.golang.try:gotip-windows-amd64-longtest,gotip-windows-arm64
Change-Id: Icd6fabbc8f0b78e23a82eef8db89940e89e9222d
Reviewed-on: https://go-review.googlesource.com/c/go/+/567735
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Change-Id: I430c9a7c4936d7a8c8c787aa63de9a796d20fdf3
Reviewed-on: https://go-review.googlesource.com/c/go/+/539597
Reviewed-by: Carlos Amedee <carlos@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Run-TryBot: shuang cui <imcusg@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Also provide a runnable example to illustrate that behavior.
This should help users to avoid the common mistake of expecting
os.Readlink to return an absolute path.
Fixes#57766.
Change-Id: I8f60aa111ebda0cae985758615019aaf26d5cb41
Reviewed-on: https://go-review.googlesource.com/c/go/+/546995
Auto-Submit: Bryan Mills <bcmills@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
GetFinalPathNameByHandle exists since Windows Vista, which we no longer
support, so we don't need to prove that it exists before using it.
Updates #57003
Change-Id: Iff2bbe51d3baa3aabcaacf39ea3cbeda0088b9d7
Reviewed-on: https://go-review.googlesource.com/c/go/+/522195
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
This edge case was accidentally broken by CL 219638.
Change-Id: I673b3b580fbe379a04f8650cf5969fe9bce83691
Reviewed-on: https://go-review.googlesource.com/c/go/+/495036
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Windows UTF-16 strings can contain unpaired surrogates, which can't be
decoded into a valid UTF-8 string. This file defines a set of functions
that can be used to encode and decode potentially ill-formed UTF-16
strings by using the
[the WTF-8 encoding](https://simonsapin.github.io/wtf-8/).
WTF-8 is a strict superset of UTF-8, i.e. any string that is
well-formed in UTF-8 is also well-formed in WTF-8 and the content
is unchanged. Also, the conversion never fails and is lossless.
The benefit of using WTF-8 instead of UTF-8 when decoding a UTF-16
string is that the conversion is lossless even for ill-formed
UTF-16 strings. This property allows to read an ill-formed UTF-16
string, convert it to a Go string, and convert it back to the same
original UTF-16 string.
Fixes#59971
Change-Id: Id6007f6e537844913402b233e73d698688cd5ba6
Reviewed-on: https://go-review.googlesource.com/c/go/+/493036
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Paul Hampson <Paul.Hampson@Pobox.com>
Empty time value time.Time{} leaves the corresponding time of the file
unchanged.
Fixes#32558
Change-Id: I1aff42f30668ff505ecec2e9509d8f2b8e4b1b6a
Reviewed-on: https://go-review.googlesource.com/c/go/+/219638
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Truncate() a non existent file on Windows currently creates a new blank
file. This behavior is not consistent with other OSes where a file not
found error would instead be returned. This change makes Truncate on
Windows return a file-not-found error when the specified file doesn't
exist, bringing the behavior consistent.
New test cases have been added to prevent a regression.
Fixes#58977
Change-Id: Iaf7b41fc4ea86a2b2ccc59f8be81be42ed211b5c
GitHub-Last-Rev: 636b6c37c1685096281ad506f3cfe35fd5810cb2
GitHub-Pull-Request: golang/go#59085
Reviewed-on: https://go-review.googlesource.com/c/go/+/477215
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Quim Muntal <quimmuntal@gmail.com>
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
This generates GetTempPath2. Go now tries to determine if the windows it runs on has GetTempPath2 by finding it only once at the loading time. If GetTempPath2 exists, it sets the flag so that any calls to tempDir will use it. If it doesn't exist, Go then uses GetTempPath.
GetTempPath2 was generated into internal/syscall/windows since syscall is locked down.
Fixes#56899
Change-Id: Iff08502aebc787fde802ee9496c070c982fbdc08
GitHub-Last-Rev: b77938953404b4e8e11f829c742e3eb109580c5e
GitHub-Pull-Request: golang/go#57980
Reviewed-on: https://go-review.googlesource.com/c/go/+/463219
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Quim Muntal <quimmuntal@gmail.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
This CL updates File.readdir() on windows so it uses
GetFileInformationByHandleEx with FILE_ID_BOTH_DIR_INFO
instead of Find* APIs. The former is more performant because
it allows us to buffer IO calls and reduces the number of system calls,
passing from 1 per file to 1 every ~100 files
(depending on the size of the file name and the size of the buffer).
This change improve performance of File.ReadDir by 20-30%.
name old time/op new time/op delta
ReadDir-12 562µs ±14% 385µs ± 9% -31.60% (p=0.000 n=9+9)
name old alloc/op new alloc/op delta
ReadDir-12 29.7kB ± 0% 29.5kB ± 0% -0.88% (p=0.000 n=8+10)
name old allocs/op new allocs/op delta
ReadDir-12 399 ± 0% 397 ± 0% -0.50% (p=0.000 n=10+10)
This change also speeds up calls to os.SameFile when using FileStats
returned from File.readdir(), as their file ID can be inferred while
reading the directory.
Change-Id: Id56a338ee66c39656b564105cac131099218fb5d
Reviewed-on: https://go-review.googlesource.com/c/go/+/452995
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Updates syscall.Open to support opening directories via CreateFileW.
CreateFileW handles are more versatile than FindFirstFile handles.
They can be used in Win32 APIs like GetFileInformationByHandle and
SetFilePointerEx, which are needed by some Go APIs.
Fixes#52747Fixes#36019
Change-Id: I26a00cef9844fb4abeeb18d2f9d854162a146651
Reviewed-on: https://go-review.googlesource.com/c/go/+/405275
Reviewed-by: Roland Shoemaker <roland@golang.org>
Reviewed-by: Patrik Nyblom <pnyb@google.com>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Quim Muntal <quimmuntal@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
[This CL is part of a sequence implementing the proposal #51082.
The design doc is at https://go.dev/s/godocfmt-design.]
Run the updated gofmt, which reformats doc comments,
on the main repository. Vendored files are excluded.
For #51082.
Change-Id: I7332f099b60f716295fb34719c98c04eb1a85407
Reviewed-on: https://go-review.googlesource.com/c/go/+/384268
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
go/doc in all its forms applies this replacement when rendering
the comments. We are considering formatting doc comments,
including doing this replacement as part of the formatting.
Apply it to our source files ahead of time.
For #51082.
Change-Id: Ifcc1f5861abb57c5d14e7d8c2102dfb31b7a3a19
Reviewed-on: https://go-review.googlesource.com/c/go/+/384262
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
A run of lines that are indented with any number of spaces or tabs
format as a <pre> block. This commit fixes various doc comments
that format badly according to that (standard) rule.
For example, consider:
// - List item.
// Second line.
// - Another item.
Because the - lines are unindented, this is actually two paragraphs
separated by a one-line <pre> block. This CL rewrites it to:
// - List item.
// Second line.
// - Another item.
Today, that will format as a single <pre> block.
In a future release, we hope to format it as a bulleted list.
Various other minor fixes as well, all in preparation for reformatting.
For #51082.
Change-Id: I95cf06040d4186830e571cd50148be3bf8daf189
Reviewed-on: https://go-review.googlesource.com/c/go/+/384257
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Change-Id: I398d324723025b559fdca783fc334de9be68f2d1
Reviewed-on: https://go-review.googlesource.com/c/go/+/314030
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Now that we don't automatically pass all inheritable handles to new
processes, we can make pipes returned by os.Pipe() inheritable, just
like they are on Unix. This then allows them to be passed through the
SysProcAttr.AdditionalInheritedHandles parameter simply.
Updates #44011.
Fixes#21085.
Change-Id: I8eae329fbc74f9dc7962136fa9aae8fb66879751
Reviewed-on: https://go-review.googlesource.com/c/go/+/288299
Trust: Jason A. Donenfeld <Jason@zx2c4.com>
Trust: Alex Brainman <alex.brainman@gmail.com>
Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Necessary to move PathError to io/fs.
For #41190.
Change-Id: I05e87675f38a22f0570d4366b751b6169f7a1b13
Reviewed-on: https://go-review.googlesource.com/c/go/+/243900
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This CL revises the document of File.Fd that explicitly points
its user to runtime.SetFinalizer where contains the information
that a file descriptor could be closed in a finalizer and therefore
causes a failure in syscall.Write if runtime.KeepAlive is not invoked.
The CL also suggests an alternative of File.Fd towards File.SyscallConn.
Fixes#41505
Change-Id: I6816f0157add48b649bf1fb793cf19dcea6894b5
Reviewed-on: https://go-review.googlesource.com/c/go/+/256899
Reviewed-by: Rob Pike <r@golang.org>
Trust: Ian Lance Taylor <iant@golang.org>
Previously, when the target (“old”) path passed to os.Symlink was a
“root-relative” Windows path,¹ we would erroneously prepend
destination (“new”) path when determining which path to Stat,
resulting in an invalid path which was then masked by the lack of
error propagation for the Stat call (#39183).
If the link target is a directory (rather than a file), that would
result in the symlink being created without the
SYMBOLIC_LINK_FLAG_DIRECTORY flag, which then fails in os.Open.
¹https://docs.microsoft.com/en-us/windows/win32/fileio/creating-symbolic-links
Updates #39183
Change-Id: I04f179cd2b0c44f984f34ec330acad2408aa3a20
Reviewed-on: https://go-review.googlesource.com/c/go/+/235317
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Several method implementations were identical in file_unix.go and
file_windows.go. Merge them into file_posix.go.
Change-Id: I8bcfad468829530f81f52fe426b3a8c042e7bbd6
Reviewed-on: https://go-review.googlesource.com/c/go/+/224138
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Fixes#35492
Change-Id: I00dce8fd1228f809e0c61013ac4de7a5953cbbf9
Reviewed-on: https://go-review.googlesource.com/c/go/+/206997
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
WriteAt use pwrite syscall on *nix or WriteFile on Windows.
On Linux/Windows, these system calls always write to end of file in
append mode, regardless of offset parameter.
It is hard (maybe impossible) to make WriteAt work portably.
Making WriteAt returns an error if file is opened in append mode, we
guarantee to get consistent behavior between platforms, also prevent
user from accidently corrupting their data.
Fixes#30716
Change-Id: If83d935a22a29eed2ff8fe53d13d0b4798aa2b81
Reviewed-on: https://go-review.googlesource.com/c/go/+/166578
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When closing a pipe, use CancelIoEx to cancel pending I/O.
This makes concurrent Read and Write calls return os.ErrClosed.
This change also enables some pipe tests on Windows.
Fixes#28477Fixes#25835
Change-Id: If52bb7d80895763488a61632e4682a78336e8420
Reviewed-on: https://go-review.googlesource.com/c/go/+/164721
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
windows-arm TMP directory live inside such link (see
https://github.com/golang/go/issues/29746#issuecomment-456526811 for
details), so symlinks like that will be common at least on windows-arm.
This CL builds on current syscall.Readlink implementation. Main
difference between the two is how new code handles symlink targets,
like \??\Volume{ABCD}\.
New implementation uses Windows CreateFile API with
FILE_FLAG_OPEN_REPARSE_POINT flag to get \??\Volume{ABCD}\ file handle.
And then it uses Windows GetFinalPathNameByHandle with VOLUME_NAME_DOS
flag to convert that handle into standard Windows path.
FILE_FLAG_OPEN_REPARSE_POINT flag ensures that symlink is not followed
when CreateFile opens the file.
Fixes#30463
Change-Id: I33b18227ce36144caed694169ef2e429fd995fb4
Reviewed-on: https://go-review.googlesource.com/c/164201
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
If TMP environment variable is set to Z:\, TempDir returns Z:.
But Z: refers to current directory on Z:, while Z:\ refers to root
directory on Z:. Adjust TempDir to return Z:\.
Fixes#29291
Change-Id: If04d0c7977a8ac2d9d558307502e81beb68776ef
Reviewed-on: https://go-review.googlesource.com/c/154384
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Windows implementation of Symlink uses CreateSymbolicLink Windows
API. The API requires to identify the target type: file or
directory. Current Symlink implementation uses Lstat to determine
symlink type, but Lstat will not be able to determine correct
result if destination is symlink. Replace Lstat call with Stat.
Fixes#28432
Change-Id: Ibee6d8ac21e2246bf8d0a019c4c66d38b09887d4
Reviewed-on: https://go-review.googlesource.com/c/145217
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
DevNull is documented on darwin, dragonfly, freebsd, linux,
nacl, netbsd, openbsd, solaris and plan9, but not on windows.
Add missing documentation.
Change-Id: Icdbded0dd5e322ed4360cbce6bee4cdca5cfbe72
Reviewed-on: https://go-review.googlesource.com/102456
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Per the notice in the Go 1.10 release notes, this change drops the
support for Windows Vista or below (including Windows XP) and
simplifies the code for the sake of maintenance.
There is one exception to the above. The code related to DLL and
system calls still remains in the runtime package. The remaining code
will be refined and used for supporting upcoming Windows versions in
future.
Updates #17245Fixes#23072
Change-Id: I9e2821721f25ef9b83dfbf85be2b7ee5d9023aa5
Reviewed-on: https://go-review.googlesource.com/94255
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Otherwise, on systems for which syscall does not implement Getwd,
a lot of unnecessary files and directories get added to the testlog,
right up the root directory. This was causing tests on such systems
to fail to cache in practice.
Updates #22593
Change-Id: Ic8cb3450ea62aa0ca8eeb15754349f151cd76f85
Reviewed-on: https://go-review.googlesource.com/83455
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When we write a cached test result, we now also write a log of the
environment variables and files inspected by the test run,
along with a hash of their content. Before reusing a cached test result,
we recompute the hash of the content specified by the log, and only
use the result if that content has not changed.
This makes test caching behave correctly for tests that consult
environment variables or stat or read files or directories.
Fixes#22593.
Change-Id: I8608798e73c90e0c1911a38bf7e03e1232d784dc
Reviewed-on: https://go-review.googlesource.com/81895
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The full truth seems too complicated to write in this method's doc, so
I'm going with a simple half truth.
The full truth is that Fd returns the descriptor in blocking mode,
because that is historically how it worked, and existing programs
would be surprised if the descriptor is suddenly non-blocking. On Unix
systems whether a file is non-blocking or not is a property of the
underlying file description, not of a particular file descriptor, so
changing the returned descriptor to blocking mode also changes the
existing File to blocking mode. Blocking mode works fine, althoug I/O
operations now take up a thread. SetDeadline and friends rely on the
runtime poller, and the runtime poller only works if the descriptor is
non-blocking. So it's correct that calling Fd disables SetDeadline.
The other half of the truth is that if the program is willing to work
with a non-blocking descriptor, it could call
syscall.SetNonblock(descriptor, true) to change the descriptor, and
the original File, to non-blocking mode. At that point SetDeadline
would start working again. I tried to write that in a way that is
short and comprehensible but failed. Since deadlines mostly work on
pipes, and there isn't much reason to call Fd on a pipe, and few
people use SetDeadline, I decided to punt.
Fixes#22934
Change-Id: I2e49e036f0bcf71f5365193831696f9e4120527c
Reviewed-on: https://go-review.googlesource.com/81636
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
windows version of Pipe function is implemented by calling
syscall.Pipe which returns handles inheritable by client process,
and then adjusting returned handles with syscall.CloseOnExec.
Just create non-inheritable handles in the first place.
Now that we don't have a race window in the code, drop use
of syscall.ForkLock.
Change-Id: Ie325da7c2397b5995db4a5ddb0117e2ce1745187
Reviewed-on: https://go-review.googlesource.com/72010
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
internal/poll package assumes that only net sockets use runtime
netpoller on windows. We get memory corruption if other file
handles are passed into runtime poller. Make FD.Init receive
and use useNetpoller argument, so FD.Init caller is explicit
about using runtime netpoller.
Fixes#21172
Change-Id: I60e2bfedf9dda9b341eb7a3e5221035db29f5739
Reviewed-on: https://go-review.googlesource.com/65810
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Catch all the cases where a file operation might return ErrFileClosing,
and convert to ErrClosed. Use a new method for the conversion, which
permits us to remove some KeepAlive calls.
Change-Id: I584178f297efe6cb86f3090b2341091b412f1041
Reviewed-on: https://go-review.googlesource.com/41793
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Now that the os package uses internal/poll on Unix and Windows systems,
it can rely on internal/poll reference counting to ensure that the
file descriptor is not closed until all I/O is complete.
That was already working. This CL completes the job by not trying to
modify the Sysfd field when it might still be used by the I/O routines.
Fixes#7970
Change-Id: I7a3daa1a6b07b7345bdce6f0cd7164bd4eaee952
Reviewed-on: https://go-review.googlesource.com/41674
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This changes the os package to use the runtime poller for file I/O
where possible. When a system call blocks on a pollable descriptor,
the goroutine will be blocked on the poller but the thread will be
released to run other goroutines. When using a non-pollable
descriptor, the os package will continue to use thread-blocking system
calls as before.
For example, on GNU/Linux, the runtime poller uses epoll. epoll does
not support ordinary disk files, so they will continue to use blocking
I/O as before. The poller will be used for pipes.
Since this means that the poller is used for many more programs, this
modifies the runtime to only block waiting for the poller if there is
some goroutine that is waiting on the poller. Otherwise, there is no
point, as the poller will never make any goroutine ready. This
preserves the runtime's current simple deadlock detection.
This seems to crash FreeBSD systems, so it is disabled on FreeBSD.
This is issue 19093.
Using the poller on Windows requires opening the file with
FILE_FLAG_OVERLAPPED. We should only do that if we can remove that
flag if the program calls the Fd method. This is issue 19098.
Update #6817.
Update #7903.
Update #15021.
Update #18507.
Update #19093.
Update #19098.
Change-Id: Ia5197dcefa7c6fbcca97d19a6f8621b2abcbb1fe
Reviewed-on: https://go-review.googlesource.com/36800
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Go 1.5 worked with Unicode console input but not ^Z.
Go 1.6 did not work with Unicode console input but did handle one ^Z case.
Go 1.7 did not work with Unicode console input but did handle one ^Z case.
The intent of this CL is for Go 1.8 to work with Unicode console input
and also handle all ^Z cases.
Here's a simple test program for reading from the console.
It prints a "> " prompt, calls read, prints what it gets, and repeats.
package main
import (
"fmt"
"os"
)
func main() {
p := make([]byte, 100)
fmt.Printf("> ")
for {
n, err := os.Stdin.Read(p)
fmt.Printf("[%d %q %v]\n> ", n, p[:n], err)
}
}
On Unix, typing a ^D produces a break in the input stream.
If the ^D is at the beginning of a line, then the 0 bytes returned
appear as an io.EOF:
$ go run /tmp/x.go
> hello
[6 "hello\n" <nil>]
> hello^D[5 "hello" <nil>]
> ^D[0 "" EOF]
> ^D[0 "" EOF]
> hello^Dworld
[5 "hello" <nil>]
> [6 "world\n" <nil>]
>
On Windows, the EOF character is ^Z, not ^D, and there has
been a long-standing problem that in Go programs, ^Z on Windows
does not behave in the expected way, namely like ^D on Unix.
Instead, the ^Z come through as literal ^Z characters:
C:\>c:\go1.5.4\bin\go run x.go
> ^Z
[3 "\x1a\r\n" <nil>]
> hello^Zworld
[13 "hello\x1aworld\r\n" <nil>]
>
CL 4310 attempted to fix this bug, then known as #6303,
by changing the use of ReadConsole to ReadFile.
This CL was released as part of Go 1.6 and did fix the case
of a ^Z by itself, but not as part of a larger input:
C:\>c:\go1.6.3\bin\go run x.go
> ^Z
[0 "" EOF]
> hello^Zworld
[13 "hello\x1aworld\r\n" <nil>]
>
So the fix was incomplete.
Worse, the fix broke Unicode console input.
ReadFile does not handle Unicode console input correctly.
To handle Unicode correctly, programs must use ReadConsole.
Early versions of Go used ReadFile to read the console,
leading to incorrect Unicode handling, which was filed as #4760
and fixed in CL 7312053, which switched to ReadConsole
and was released as part of Go 1.1 and still worked as of Go 1.5:
C:\>c:\go1.5.4\bin\go run x.go
> hello
[7 "hello\r\n" <nil>]
> hello world™
[16 "hello world™\r\n" <nil>]
>
But in Go 1.6:
C:\>c:\go1.6.3\bin\go run x.go
> hello
[7 "hello\r\n" <nil>]
> hello world™
[0 "" EOF]
>
That is, changing back to ReadFile in Go 1.6 reintroduced #4760,
which has been refiled as #17097. (We have no automated test
for this because we don't know how to simulate console input
in a test: it appears that one must actually type at a keyboard
to use the real APIs. This CL at least adds a comment warning
not to reintroduce ReadFile again.)
CL 29493 attempted to fix#17097, but it was not a complete fix:
the hello world™ example above still fails, as does Shift-JIS input,
which was filed as #17939.
CL 29493 also broke ^Z handling, which was filed as #17427.
This CL attempts the never before successfully performed trick
of simultaneously fixing Unicode console input and ^Z handling.
It changes the console input to use ReadConsole again,
as in Go 1.5, which seemed to work for all known Unicode input.
Then it adds explicit handling of ^Z in the input stream.
(In the case where standard input is a redirected file, ^Z processing
should not happen, and it does not, because this code path is only
invoked when standard input is the console.)
With this CL:
C:\>go run x.go
> hello
[7 "hello\r\n" <nil>]
> hello world™
[16 "hello world™\r\n" <nil>]
> ^Z
[0 "" EOF]
> [2 "\r\n" <nil>]
> hello^Zworld
[5 "hello" <nil>]
> [0 "" EOF]
> [7 "world\r\n" <nil>]
This almost matches Unix:
$ go run /tmp/x.go
> hello
[6 "hello\n" <nil>]
> hello world™
[15 "hello world™\n" <nil>]
> ^D
[0 "" EOF]
> [1 "\n" <nil>]
> hello^Dworld
[5 "hello" <nil>]
> [6 "world\n" <nil>]
>
The difference is in the handling of hello^Dworld / hello^Zworld.
On Unix, hello^Dworld terminates the read of hello but does not
result in a zero-length read between reading hello and world.
This is dictated by the tty driver, not any special Go code.
On Windows, in this CL, hello^Zworld inserts a zero length read
result between hello and world, which is treated as an interior EOF.
This is implemented by the Go code in this CL, but it matches the
handling of ^Z on the console in other programs:
C:\>copy con x.txt
hello^Zworld
1 file(s) copied.
C:\>type x.txt
hello
C:\>
A natural question is how to test all this. As noted above, we don't
know how to write automated tests using the actual Windows console.
CL 29493 introduced the idea of substituting a different syscall.ReadFile
implementation for testing; this CL continues that idea but substituting
for syscall.ReadConsole instead. To avoid the regression of putting
ReadFile back, this CL adds a comment warning against that.
Fixes#17427.
Fixes#17939.
Change-Id: Ibaabd0ceb2d7af501d44ac66d53f64aba3944142
Reviewed-on: https://go-review.googlesource.com/33451
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Quentin Smith <quentin@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
