Commit Graph

107 Commits

Author SHA1 Message Date
Sylvain Baubeau
d91c4cb694 Improve 404 errors handling
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
9ab55eae39 Use only one Swift container for both files and manifests
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
a1ae7f7122 Increase default chunk size to 20M
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
davidli
062d6266cf Add support for Openstack Identity v3 API
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
3f9e7ed169 Use 'prefix' parameter instead of 'path' when listing files
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
1d46bb2bcc Create full folder hierarchy instead of just the top level folder
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
4e619bc9b1 Remove one level of indentation in swift path handling code
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
75ce67c469 Use mitchellh/mapstructure library to parse Swift parameters
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
ea81e208a4 Move Dynamic Large Object handling to dedicated methods
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
8a22c0f4e1 Simplify code that handles non existing manifests
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
16a49ade16 Handle error during copy of original content
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:09 +02:00
nevermosby
9f7f23e373 Update the import path for swift driver test
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
davidli
1f4eb7b735 Use gofmt to format the code of swift driver.
Signed-off-by: Li Wenquan <wenquan.li@hp.com>
2015-07-21 23:55:09 +02:00
Sylvain Baubeau
cce4956131 Add Openstack Swift storage driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-07-21 23:55:08 +02:00
yuzou
feebd69d26 Close reader after the test is finished.
Signed-off-by: yuzou <zouyu7@huawei.com>
2015-07-17 14:55:31 +08:00
Stephen Day
940b865bc0 Merge pull request #670 from stevvooe/remove-ipc
Remove half-baked Storage Driver IPC support
2015-06-30 19:21:54 -07:00
Richard Scothern
fa17f9254f Pass correct context into tracer
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-06-30 10:28:14 -07:00
Stephen J Day
6167220cdd Remove half-baked Storage Driver IPC support
This removes documentation and code related to IPC based storage driver
plugins. The existence of this functionality was an original feature goal but
is now not maintained and actively confusing incoming contributions. We will
likely explore some driver plugin mechanism in the future but we don't need
this laying around in the meantime.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-29 16:52:12 -07:00
Stephen J Day
f9e152d912 Ensure that rados is disabled without build tag
This ensures that rados is not required when building the registry. This was
slightly tricky in that when the flags were applied, the rados package was
completely missing. This led to a problem where rados was basically unlistable
and untestable as a package. This was fixed by simply adding a doc.go file that
is included whether rados is built or not.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-06-16 20:00:02 -07:00
Stephen Day
d796729b6b Merge pull request #623 from ahmetalpbalkan/azure-vendor
storage/driver/azure: Update vendored Azure SDK
2015-06-16 17:41:38 -07:00
Ahmet Alp Balkan
5c372ded1b storage/driver/azure: Update vendored Azure SDK
This change refreshes the updated version of Azure SDK
for Go that has the latest changes.

I manually vendored the new SDK (github.com/Azure/azure-sdk-for-go)
and I removed `management/` `core/` packages manually simply because
they're not used here and they have a fork of `net/http` and `crypto/tls`
for a particular reason. It was introducing a 44k SLOC change otherwise...

This also undoes the `include_azure` flag (actually Steven removed the
driver from imports but forgot to add the build flag apparently, so the
flag wasn't really including azure. 😄 ). This also must be obsolete
now.

Fixes #620, #175.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-06-16 17:13:44 -07:00
Olivier Gambier
280b9c50ac Saner default data location
Signed-off-by: Olivier Gambier <olivier@docker.com>
2015-06-10 20:54:24 -07:00
Vincent Giersch
4bc53818cb Fix rados build, remove uuid dependency
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-06-01 09:57:40 +00:00
Stephen Day
af987fd3dc Merge pull request #443 from gierschv/driver-rados
Storage Driver: Ceph Object Storage (RADOS)
2015-05-29 18:10:49 -07:00
Stephen J Day
d4c50637f9 Better error message when failing to get AWS auth
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-22 16:45:45 -07:00
Vincent Giersch
2c1a83f940 Storage Driver: Ceph Object Storage (RADOS)
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.

Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:

* All the objects data are stored with opaque UUID names prefixed
  (e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
  omap object identifier is the virtual directory name, the keys in
  a specific are the relative filenames and the values the blob
  object identifier (or empty value for a sub directory).

  e.g. For the following hierarchy:

     /directory1
     /directory1/object1
     /directory1/object2
     /directory1/directory2/object3

  The omap "/directory1" will contains the following key / values:
    - "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
    - "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
    - "directory2" ""

  The omap "/directory1/directory2" will contains:
    - "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"

* The MOVE is implemented by changing the reference to a specific
  blob in its parent virtual directory omap.

This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.

Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
2015-05-20 01:44:34 +00:00
Stephen Day
3a0d6faba1 Merge pull request #410 from RichardScothern/driver-context
Add golang/x/net/context.Context to storage driver method calls
2015-05-04 19:18:30 -07:00
Stephen J Day
80abf9fce0 Use done channel to avoid goroutine leak
This deals with a memory leak, caused by goroutines, experienced when using the
s3 driver. Unfortunately, this section of the code leaks goroutines like a
sieve. There is probably some refactoring that could be done to avoid this but
instead, we have a done channel that will cause waiting goroutines to exit.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-28 14:40:20 -07:00
Richard
5d9105bd25 Make Storage Driver API calls context aware.
- Change driver interface to take a context as its first argument
     - Make newFileReader take a context as its first argument
     - Make newFileWriter take a context as its first argument
     - Make blobstore exists and delete take a context as a first argument
     - Pass the layerreader's context to the storage layer
     - Pass the app's context to purgeuploads
     - Store the app's context into the blobstore (was previously null)
     - Pass the trace'd context to the storage drivers

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 15:58:58 -07:00
Olivier Gambier
ecdf1f6daa Merge pull request #430 from stevvooe/address-s3-timeout-error
Attempt to address intermittent s3 RequestTimeout error
2015-04-24 11:20:30 -07:00
Stephen Day
12a8fd49c7 Merge pull request #420 from stevvooe/expose-drivernames
Expose storage driver names for tracing
2015-04-24 10:53:40 -07:00
Stephen J Day
0f897aea8f Attempt to address intermittent s3 RequestTimeout error
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 20:07:32 -07:00
Stephen J Day
0d8cb4dca8 Correctly check s3 chunksize parameter
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-23 16:31:41 -07:00
Stephen J Day
ecda1f4eff Include driver name in trace messsages
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:31 -07:00
Stephen J Day
b361b4811b Require storage drivers to report their name
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:01 -07:00
Stephen Day
0281f4dce5 Merge pull request #419 from stevvooe/pool-buffers-s3
Pool buffers used in S3.WriteStream
2015-04-22 16:41:28 -07:00
Stephen J Day
c49f7cd015 Pool buffers used in S3.WriteStream
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 16:25:53 -07:00
Stephen J Day
e4794ff73d Allow configuration of chunksize parameter
The code using values from the yaml package wasn't careful enought with the
possible incoming types. Turns out, it is just an int but we've made this
section somewhat bulletproof in case that package changes the behavior.

This code likely never worked. The configuration system should be decoupled
from the object instantiation.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 14:31:34 -07:00
Stephen Day
3802701cdc Merge pull request #375 from RichardScothern/path-logging
registry/storage/driver: add path and other info to filesytem trace methods.
2015-04-17 14:09:23 -07:00
Richard
bc2b6efaa6 Add path and other info to filesytem trace methods.
Also fix Delete (was 'Move').
2015-04-17 13:55:18 -07:00
bin liu
f3f46307f2 fix some typos in source comments
Signed-off-by: bin liu <liubin0329@gmail.com>
2015-04-17 12:39:52 +00:00
Stephen J Day
12bf470b2f Trace function calls to Base storage driver
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-10 16:42:36 -07:00
Stephen J Day
2b4ad94cee Defer case-sensitive support to storage backend
Rather than enforce lowercase paths for all drivers, support for
case-sensitivity has been deferred to the driver. There are a few caveats to
this approach:

1. There are possible security implications for tags that only differ in their
case. For instance, a tag "A" may be equivalent to tag "a" on certain file
system backends.
2. All system paths should not use case-sensitive identifiers where possible.
This might be problematic in a blob store that uses case-sensitive ids. For
now, since digest hex ids are all case-insensitive, this will not be an issue.

The recommend workaround is to not run the registry on a case-insensitive
filesystem driver in security sensitive applications.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-07 14:14:45 -07:00
Josh Hawn
8c0859e39c Handle cloudFront bucket prefix issue
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-04-06 19:46:05 -07:00
Stephen J Day
06acde06cb Avoid crash on invalid Move arguments
This chnage prevents a crash when moving from a non-existent directory that has
a file as a parent. To prevent this, we simply check that the node is a
directory and throws an error if it is not.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-01 19:03:02 -07:00
Stephen Day
9ee35877e3 Merge pull request #294 from fkautz/pr_out_updating_msopentech_azure_sdk_for_go_to_latest_master
Updating MSOpenTech/azure-sdk-for-go to latest master
2015-03-24 13:27:19 -07:00
Frederick F. Kautz IV
dffd1babd2 Updating MSOpenTech/azure-sdk-for-go to latest master
Signed-off-by: Frederick F. Kautz IV <fkautz@alumni.cmu.edu>
2015-03-23 21:59:21 -07:00
Ahmet Alp Balkan
594f733e03 storage/driver/azure: Allow non-default realms
This enables Azure storage driver to be used with non-default
cloud endpoints like Azure China or Azure Government that does
not use `.blob.core.windows.net` FQDN suffix.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-03-23 20:41:16 -07:00
Josh Hawn
6d14019368 Refactor Layer interface to return a Handler
... Rather than ServeHTTP directly.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-12 21:59:07 -07:00
Andy Goldstein
5c3f53b70f Fix Godoc typos
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:29 -07:00
Andy Goldstein
30bcc17b85 Middleware!
Convert middleware in the config to be a map of type->[]Middleware

Add support for registry & repository middleware.

Some naming updates as well.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
2015-03-11 08:56:28 -07:00
David Lawrence
952f39edff Refactoring cloudfactory layer handler into a more generic storage
middleware concept.

This also breaks the dependency the storage package had on goamz
Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-11 08:56:28 -07:00
Stephen J Day
19061f347e doc: move storage driver readmes into docs
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-03-05 17:23:33 -08:00
Andrey Kostov
58269e73fc Fix S3 driver's list when the root directory is either "" or "/" 2015-02-19 16:31:34 -08:00
Andrey Kostov
02718ee277 Add an empty root directory s3 driver specific test 2015-02-19 16:28:32 -08:00
Ahmet Alp Balkan
8728074d65 storagedriver/azure: Add README
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-02-11 14:40:51 -08:00
Stephen J Day
6e4f9a2e3e Move storagedriver package to registry/storage/driver
This change is slightly more complex than previous package maves in that the
package name changed. To address this, we simply always reference the package
driver as storagedriver to avoid compatbility issues with existing code. While
unfortunate, this can be cleaned up over time.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00