This requires some discussion of how we will handle errors due to network problems
and after further changes in that direction some more stress testing. There is also an
upcomming commit implementing zero fill on WriteStream when offset is greater than
the current size of the file.
Requires all paths in the inmemory and filesystem drivers to begin with
a slash, and then contain only valid path components (2+ alphanumeric
characters with optional period, hyphen, and underscore separators)
delimited by slashes.
Also updates the storage driver test suites to construct paths of this
format, and causes the suite to abort if files are not cleaned up after
the test run.
Also makes this test respect the Short flag, reducing the number of
threads by a factor of 4 and space usage by a factor of 16
Note: this test is probably unreasonable to run on the inmemory driver
without the Short flag
Creates trees instead of flat files for TestConcurrentFileStreams
Adds TestConcurrentStreamReads, which writes a large file (smaller in
Short mode), and then ensures that several concurrent readers properly
read their portions of the file with random offsets
Ensures that Move will properly overwrite the file at the destination
location.
Also checks that Move of a nonexistent source file will NOT delete the
file at the destination.
This replaces only using flat filenames, to better test nested file
behaviors.
Fixed inmemory/mfs.go and filesystem/driver.go after finding bugs with
the new tests and test behavior.
The packages causing build errors are being disabled for now to let us split up
the work in the different driver implementations without blocking integration
into the main branch. The s3 and azure implementations need some effort to add
Stat support. The ipc package needs that work plus some care around hanging
send calls.
Several checks for ReadStream with offset around boundary conditions were
missing. The new checks ensure negative offsets are detected and io.EOF is
returned properly when trying to read past the end of a file. The filesystem
and inmemory driver have been updated accordingly.
An outline of missing checks for List are also part of this commit. Action will
be taken here based on discussion in issue #819.
This change updates the testsuite to migrate to the new driver interface. This
includes the new Stat call, changes to int64 over uint64 and the changes to the
WriteStream signature. Several test cases have been added to vet
implementations against various assumptions.
This change brings the storagedriver API in line with the Go standard library's
use of int64 for offsets. The main benefit is simplicity in interfacing with
the io library reducing the number of type conversions in simple code.
Fixes/tests listing for keys beginning with "/"
No longer extraneously wraps Closers in ioutil.NopClosers
Uses omitempty for all ipc struct type fields
This only works for a specific whitelist of error types, which is
currently all errors in the storagedriver package.
Also improves storagedriver tests to enforce proper error types are
returned
Custom storage drivers can register a factory to create the driver by
name, similar to the database/sql package's Register and Open
factory.Create returns an in-process driver if registered or an IPC
driver if one can be found, erroring otherwise
This standardizes parameter passing for creation of storage drivers
Also adds documentation for storagedriver package and children
