Commit Graph

1375 Commits

Author SHA1 Message Date
Stephen J Day
1089cae282 Separate request data from actor in Event
To clarify the role of actor, the request data that initiates an event has been
separated. The ActorRecord is pared down to just the username. This eliminates
confusion about where event related data should be added.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:32:37 -08:00
Stephen J Day
2aed7c2d0c Webhook notification support in registry webapp
Endpoints are now created at applications startup time, using notification
configuration. The instances are then added to a Broadcaster instance, which
becomes the main event sink for the application. At request time, an event
bridge is configured to listen to repository method calls. The actor and source
of the eventBridge are created from the requeest context and application,
respectively. The result is notifications are dispatched with calls to the
context's Repository instance and are queued to each endpoint via the
broadcaster.

This commit also adds the concept of a RequestID and App.InstanceID. The
request id uniquely identifies each request and the InstanceID uniquely
identifies a run of the registry. These identifiers can be used in the future
to correlate log messages with generated events to support rich debugging.

The fields of the app were slightly reorganized for clarity and a few horrid
util functions have been removed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:32:37 -08:00
Alexander Morozov
3790b5d6b4 Fix some go vet errors
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
2015-02-02 14:53:20 -08:00
Stephen J Day
b6270d9c14 Handle empty blob files more appropriately
Several API tests were added to ensure correct acceptance of zero-size and
empty tar files. This led to several changes in the storage backend around the
guarantees of remote file reading, which backs the layer and layer upload type.

In support of these changes, zero-length and empty checks have been added to
the digest package. These provide a sanity check against upstream tarsum
changes. The fileReader has been modified to be more robust when reading and
seeking on zero-length or non-existent files. The file no longer needs to exist
for the reader to be created. Seeks can now move beyond the end of the file,
causing reads to issue an io.EOF. This eliminates errors during certain race
conditions for reading files which should be detected by stat calls. As a part
of this, a few error types were factored out and the read buffer size was
increased to something more reasonable.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 13:01:49 -08:00
Arnaud Porterie
bdbb90e01f Merge pull request #10472 from jlhawn/handle_mux_route_url_bug
Handle gorilla/mux route url bug
2015-01-31 19:52:51 -08:00
Tibor Vass
cbda817ec3 Merge pull request #10471 from jlhawn/token_auth_header
Fix token basic auth header issue
2015-01-31 19:51:50 -05:00
Josh Hawn
63af81b883 Fix token basic auth header issue
When requesting a token, the basic auth header is always being set even
if there is no username value. This patch corrects this and does not set
the basic auth header if the username is empty.

Also fixes an issue where pulling all tags from a v2 registry succeeds
when the image does not actually exist on the registry.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-01-31 15:12:34 -08:00
Josh Hawn
050337b257 Handle gorilla/mux route url bug
When getting the URL from a v2 registry url builder, it does not
honor the scheme from the endpoint object and will cause an https
endpoint to return urls starting with http.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-01-31 12:54:57 -08:00
Liu Hua
5589ce8b8a delete duplicated word in registry/session.go
Signed-off-by: Liu Hua <sdu.liu@huawei.com>
2015-01-31 20:15:59 +08:00
Stephen J Day
33a1f4ef7d Address server errors received during layer upload
This changeset addresses intermittent internal server errors encountered during
pushes.  The root cause has been isolated to layers that result in identical,
empty filesystems but may have some path declarations (imaginge "./"),
resulting in different tarsums. The main error message reported during these
upload problems was a 500 error, which was not correct.  Further investigation
showed the errors to be rooted in digest verification when finishing uploads.

Inspection of the surrounding code also identified a few issues. PutLayerChunk
was slightly refactered into PutLayerUploadComplete. Helper methods were
avoided to make handler less confusing. This simplification leveraged an
earlier change in the spec that moved non-complete chunk uploads to the PATCH
method. Simple logging was also added in the unknown error case that should
help to avoid mysterious 500 errors in the future.

At the same time, the glaring omission of a proper layer upload cancel method
was rectified. This has been added in this change so it is not missed in the
future.

In the future, we may want to refactor the handler code to be more
straightforward, hopefully letting us avoid these problems in the future.

Added test cases that reproduce these errors and drove these changes include
the following:

1. Push a layer with an empty body results in invalid blob upload.
2. Push a layer with a different tarsum (in this case, empty tar)
3. Deleting a layer upload works.
4. Getting status on a deleted layer upload returns 404.

Common functionality was grouped into shared functions to remove repitition.
The API tests will still require future love.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-29 21:26:35 -08:00
Derek McGowan
9dc3529dfe Add distribution maintainers to maintainers files
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-28 16:30:00 -08:00
Stephen J Day
0818476cb1 Open up v2 http status code checks for put and head checks
Under certain cases, such as when putting a manifest or check for the existence
of a layer, the status code checks in session_v2.go were too narrow for their
purpose. In the case of putting a manifest, the handler only cares that an
error is not returned. Whether it is a 304 or 202 does not matter, as long as
the server reports success. Having the client only accept specific http codes
inhibits future protocol evolution.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-27 18:09:53 -08:00
Jessie Frazelle
b7db855fc6 Merge pull request #10249 from jlhawn/distribution_version_header_10247
Split API Version header when checking for v2
2015-01-27 11:48:42 -08:00
Derek McGowan
d96d4aa9f0 Better error messaging and logging for v2 registry requests
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-26 14:00:51 -08:00
Derek McGowan
41703e2bb7 Fix write after close on http response
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-21 15:14:01 -08:00
Josh Hawn
6a736c20f0 Split API Version header when checking for v2
Since the Docker-Distribution-API-Version header value may contain multiple
space delimited versions as well as many instances of the header key, the
header value is now split on whitespace characters to iterate over all versions
that may be listed in one instance of the header.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-01-21 15:08:32 -08:00
Tibor Vass
b2830a0786 Merge pull request #10227 from jlhawn/v1_v2_login_patch
Resolve ambiguity on registry v2 ping
2015-01-21 02:10:32 -05:00
Josh Hawn
1c7271129b Resolve ambiguity on registry v2 ping
v2 ping now checks for a Docker-Distribution-API-Version
header that identifies the endpoint as "registry/2.0"

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-01-20 19:52:08 -08:00
Stephen Day
cd647beb1d Merge pull request #89 from jlhawn/version_header
Add Docker Distribution API Version header
2015-01-20 13:42:12 -08:00
Josh Hawn
acfcc955de Add Docker Distribution API Version header
Setting a header for all responses can help clients better determine
if the server speaks the legacy v1 API or the v2 API. It is important
that the header be set *BEFORE* routing the request.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-01-20 13:26:05 -08:00
Jessie Frazelle
35bb812cee Merge pull request #9784 from dmcgowan/v2-registry
Client Support for Docker Registry HTTP API V2
2015-01-19 10:46:38 -08:00
Jessie Frazelle
693c9d4120 Merge pull request #10143 from cpuguy83/10129_marshal-indent_dockercfg
Make .dockercfg with json.MarshallIndent
2015-01-17 14:18:45 -08:00
Brian Goff
4eaf644321 Make .dockercfg with json.MarshallIndent
Fixes #10129
Makes the .dockercfg more human parsable.

Also cleaned up the (technically) racey login test.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2015-01-17 07:05:56 -05:00
Stephen J Day
825da388a4 Update the registry app to use the new storage interfaces
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-16 18:33:28 -08:00
Derek McGowan
9c24fc93ad Add token cache
Token cache prevents the need to get a new token for every registry interaction.
Since the tokens are short lived, the cache expires after only a minute.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-15 14:05:06 -08:00
Derek McGowan
5bf94a6438 Cleanup v2 session to require endpoint
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-15 14:05:06 -08:00
Derek McGowan
735a112415 Fix list tags
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-15 14:05:06 -08:00
Derek McGowan
e5744a3bad Refactor from feedback
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-15 14:05:06 -08:00
Josh Hawn
826bde851b Add Tarsum Calculation during v2 Pull operation
While the v2 pull operation is writing the body of the layer blob to disk
it now computes the tarsum checksum of the archive before extracting it to
the backend storage driver. If the checksum does not match that from the
image manifest an error is raised.

Also adds more debug logging to the pull operation and fixes existing test
cases which were failing. Adds a reverse lookup constructor to the tarsum
package so that you can get a tarsum object using a checksum label.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-01-15 14:05:05 -08:00
Stephen J Day
6f09abd5c9 Correctly check and propagate errors in v2 session
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-15 14:05:05 -08:00
Derek McGowan
22c7328529 Get token on each request
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2015-01-15 14:05:05 -08:00
Derek McGowan
6f36ce3a01 Allow private V2 registry endpoints
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2015-01-15 14:05:05 -08:00
Derek McGowan
751a1a8dd0 Update push and pull to registry 2.1 specification
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2015-01-15 14:05:05 -08:00
Stephen J Day
ee1e1abb15 Remove dependencies on registry packages
Because docker core cannot vendor non-master Go dependencies, we need to remove
dependencies on registry package. The definition of digest.Digest has been
changed to a string and the regular expressions have been ported from
docker-registry/common library.

We'll likely change this be dependent on the registry in the future when the
API stabilizies and use of the master branch becomes the norm.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-15 14:05:05 -08:00
Stephen J Day
2fcad2a10f Registry V2 HTTP route and error code definitions
This package, ported from next-generation docker regsitry, includes route and
error definitions. These facilitate compliant V2 client implementation. The
portions of the HTTP API that are included in this package are considered to be
locked down and should only be changed through a careful change proposal.
Descriptor definitions package layout may change without affecting API behavior
until the exported Go API is ready to be locked down.

When the new registry stabilizes and becomes the master branch, this package
can be vendored from the registry.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-15 14:05:05 -08:00
Josh Hawn
e256a0e0bc Update token response handling
Registry authorization token is now taken from the response body rather than
the repsonse header.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-01-15 14:05:05 -08:00
Derek McGowan
24895820bd Update push to use mount blob endpoint
Using mount blob prevents repushing images which have already been uploaded

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-15 14:05:05 -08:00
Derek McGowan
06d0ef4179 Push flow
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-15 14:05:05 -08:00
Josh Hawn
6b400cd63c Adds support for v2 registry login
summary of changes:

registry/auth.go
  - More logging around the login functions
  - split Login() out to handle different code paths for v1 (unchanged logic)
    and v2 (does not currently do account creation)
  - handling for either basic or token based login attempts
registry/authchallenge.go
  - New File
  - credit to Brian Bland <brian.bland@docker.com> (github: BrianBland)
  - handles parsing of WWW-Authenticate response headers
registry/endpoint.go
  - EVEN MOAR LOGGING
  - Many edits throught to make the coad less dense. Sparse code is more
    readable code.
  - slit Ping() out to handle different code paths for v1 (unchanged logic)
    and v2.
  - Updated Endpoint struct type to include an entry for authorization
    challenges discovered during ping of a v2 registry.
  - If registry endpoint version is unknown, v2 code path is first attempted,
    then fallback to v1 upon failure.
registry/service.go
  - STILL MOAR LOGGING
  - simplified the logic around starting the 'auth' job.
registry/session.go
  - updated use of a registry.Endpoint struct field.
registry/token.go
  - New File
  - Handles getting token from the parameters of a token auth challenge.
  - Modified from function written by Brian Bland (see above credit).
registry/types.go
  - Removed 'DefaultAPIVersion' in lieu of 'APIVersionUnknown = 0'`

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-01-15 14:04:14 -08:00
Alexander Morozov
1f98347924 Fix format calls as suggested by vet
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
2015-01-14 14:12:03 -08:00
Stephen J Day
594263a3f5 Correctly handle missing layer upload
Because we guarded the error check, nil Upload on the handler was getting
through to unexpected branches. This directly handles the missing upload
ensuring its set as expected.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 16:09:45 -08:00
Stephen Day
887b940ec1 Merge pull request #53 from stevvooe/spool-uploads-remotely
Spool uploads remotely
2015-01-09 15:40:22 -08:00
Stephen J Day
cd92071caa Directly manage layerUploadState in webapp
Most of this change follows from the modifications to the storage api. The
driving factor is the separation of layerUploadState from the storage backend,
leaving it to the web application to store and update it. As part of the
updates to meet changes in the storage api, support for the size parameter has
been completely removed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:51:13 -08:00
Stephen J Day
fdcfc56f7b Refactor handling of hmac state packing
This refactors the hmac state token to take control of the layerUploadState
json message, which has been removed from the storage backend. It also moves
away from the concept of a LayerUploadStateStore callback object, which was
short-lived. This allows for upload offset to be managed by the web application
logic in the face of an inconsistent backend. By controlling the upload offset
externally, we reduce the possibility of misreporting upload state to a client.

We may still want to modify the way this works after getting production
experience.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-09 14:50:39 -08:00
Stephen Day
8f57e05016 Merge pull request #55 from BrianBland/layerhandler
Adds support for content redirects for layer downloads
2015-01-09 14:44:54 -08:00
Brian Bland
9d3436c18e Fixes tests, moves layerhandler in config file 2015-01-08 17:29:22 -08:00
Qiang Huang
23f9f8c3f4 registry: fix minor type
Signed-off-by: Qiang Huang <h.huangqiang@huawei.com>
2015-01-09 09:06:27 +08:00
Brian Bland
b11d549fd0 Adds support for content redirects for layer downloads
Includes a delegate implementation which redirects to the URL generated
by the storagedriver, and a cloudfront implementation.
Satisfies proposal #49
2015-01-08 17:01:28 -08:00
Don Kjer
c899a49a95 Moving NewIndexInfo, NewRepositoryInfo and associated helpers into config.go
Signed-off-by: Don Kjer <don.kjer@gmail.com>
2015-01-08 20:15:00 +00:00
Don Kjer
64b000c3ea Deprecating ResolveRepositoryName
Passing RepositoryInfo to ResolveAuthConfig, pullRepository, and pushRepository

Moving --registry-mirror configuration to registry config

Created resolve_repository job

Repo names with 'index.docker.io' or 'docker.io' are now synonymous with omitting an index name.

Adding test for RepositoryInfo

Adding tests for opts.StringSetOpts and registry.ValidateMirror

Fixing search term use of repoInfo

Adding integration tests for registry mirror configuration

Normalizing LookupImage image name to match LocalName parsing rules

Normalizing repository LocalName to avoid multiple references to an official image

Removing errorOut use in tests

Removing TODO comment

gofmt changes

golint comments cleanup.  renaming RegistryOptions => registry.Options, and RegistryServiceConfig => registry.ServiceConfig

Splitting out builtins.Registry and registry.NewService calls

Stray whitespace cleanup

Moving integration tests for Mirrors and InsecureRegistries into TestNewIndexInfo unit test

Factoring out ValidateRepositoryName from NewRepositoryInfo

Removing unused IndexServerURL

Allowing json marshaling of ServiceConfig.  Exposing ServiceConfig in /info

Switching to CamelCase for json marshaling

PR cleanup; removing 'Is' prefix from boolean members.  Removing unneeded json tags.

Removing non-cleanup related fix for 'localhost:[port]' in splitReposName

Merge fixes for gh9735

Fixing integration test

Reapplying #9754

Adding comment on config.IndexConfigs use from isSecureIndex

Remove unused error return value from isSecureIndex

Signed-off-by: Don Kjer <don.kjer@gmail.com>

Adding back comment in isSecureIndex

Signed-off-by: Don Kjer <don.kjer@gmail.com>
2015-01-08 20:14:58 +00:00