Back in the before time, the best practices surrounding usage of Context
weren't quite worked out. We defined our own type to make usage easier.
As this packaged was used elsewhere, it make it more and more
challenging to integrate with the forked `Context` type. Now that it is
available in the standard library, we can just use that one directly.
To make usage more consistent, we now use `dcontext` when referring to
the distribution context package.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Remove EndpointConfig.Transport from the return value of the
registry.notifications.endpoints expvar.Func. It results in an empty
value for that expvar variable under Go 1.7 because it is a non-nil
*http.Transport, which Go 1.7 can no longer encode as JSON.
Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
The registry uses partial Named values which the named parsers
no longer support. To allow the registry service to continue
to operate without canonicalization, switch to use WithName.
In the future, the registry should start using fully canonical
values on the backend and WithName should no longer support
creating partial values.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
If running test behide a proxy, we may get the error code
403 Forbidden which will fail line 135 for the last testcase.
Detail:
```
metrics not as expected: notifications.EndpointMetrics{Pending:0,
Events:0, Successes:4, Failures:0, Errors:0,
Statuses:map[string]int{"307 Temporary Redirect":0, "400 Bad Request":0,
"403 Forbidden":0, "200 OK":4}} !=
notifications.EndpointMetrics{Pending:0, Events:0, Successes:4,
Failures:0, Errors:0, Statuses:map[string]int{"400 Bad Request":0, "200
OK":4, "307 Temporary Redirect":0}}
```
Immediate close will fix that
Signed-off-by: Hu Keping <hukeping@huawei.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
The Hub registry generates a large volume of notifications, many of
which are uninteresting based on target media type. Discarding them
within the notification endpoint consumes considerable resources that
could be saved by discarding them within the registry. To that end,
this change adds registry configuration options to restrict the
notifications sent to an endpoint based on target media type.
Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
The EndpointConfig struct in the notifications package has some config
fields for a notification endpoint. This commit adds the ability to pass
in an *http.Transport to use when notifying that endpoint of an event.
This is especially useful for endpoints that use self-signed CAs.
Signed-off-by: Josh Chorlton <josh.chorlton@docker.com>
Most places in the registry were using string types to refer to
repository names. This changes them to use reference.Named, so the type
system can enforce validation of the naming rules.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
The Payload function for schema1 currently returns a signed manifest,
but indicates the content type is that of a manifest that isn't signed.
Note that this breaks compatibility with Registry 2.3 alpha 1 and
Docker 1.10-rc1, because they use the incorrect content type.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Removes the Mount operation and instead implements this behavior as part
of Create a From option is provided, which in turn returns a rich
ErrBlobMounted indicating that a blob upload session was not initiated,
but instead the blob was mounted from another repository
Signed-off-by: Brian Bland <brian.bland@docker.com>
Add a generic Manifest interface to represent manifests in the registry and
remove references to schema specific manifests.
Add a ManifestBuilder to construct Manifest objects. Concrete manifest builders
will exist for each manifest type and implementations will contain manifest
specific data used to build a manifest.
Remove Signatures() from Repository interface.
Signatures are relevant only to schema1 manifests. Move access to the signature
store inside the schema1 manifestStore. Add some API tests to verify
signature roundtripping.
schema1
-------
Change the way data is stored in schema1.Manifest to enable Payload() to be used
to return complete Manifest JSON from the HTTP handler without knowledge of the
schema1 protocol.
tags
----
Move tag functionality to a seperate TagService and update ManifestService
to use the new interfaces. Implement a driver based tagService to be backward
compatible with the current tag service.
Add a proxyTagService to enable the registry to get a digest for remote manifests
from a tag.
manifest store
--------------
Remove revision store and move all signing functionality into the signed manifeststore.
manifest registration
---------------------
Add a mechanism to register manifest media types and to allow different manifest
types to be Unmarshalled correctly.
client
------
Add ManifestServiceOptions to client functions to allow tags to be passed into Put and
Get for building correct registry URLs. Change functional arguments to be an interface type
to allow passing data without mutating shared state.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
tarsum is not actually used by the registry. Remove support for it.
Convert numerous uses in unit tests to SHA256.
Update docs to remove mentions of tarsums (which were often inaccurate).
Remove tarsum dependency.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
The current implementation of digest.FromBytes returns an error. This
error can never be non-nil, but its presence in the function signature
means each call site needs error handling code for an error that is
always nil.
I verified that none of the hash.Hash implementations in the standard
library can return an error on Write. Nor can any of the hash.Hash
implementations vendored in distribution.
This commit changes digest.FromBytes not to return an error. If Write
returns an error, it will panic, but as discussed above, this should
never happen.
This commit also avoids using a bytes.Reader to feed data into the hash
function in FromBytes. This makes the hypothetical case that would panic
a bit more explicit, and should also be more performant.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
contains equal length History and FSLayer arrays.
This is required to prevent malformed manifests being put to the registry and
failing external verification checks.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
As we begin our march towards multi-arch, we must prepare for the reality of
multiple manifest schemas. This is the beginning of a set of changes to
facilitate this. We are both moving this package into its target position where
it may live peacefully next to other manfiest versions.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Clean up calling convention for NewRegistryWithDriver to use functional
arguments.
This is a first step towards the refactor described in #215. I plan to
add additional options in the process of moving configurable items from
the App structure to the registry structure.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
with a new `proxy` section in the configuration file.
Create a new registry type which delegates storage to a proxyBlobStore
and proxyManifestStore. These stores will pull through data if not present
locally. proxyBlobStore takes care not to write duplicate data to disk.
Add a scheduler to cleanup expired content. The scheduler runs as a background
goroutine. When a blob or manifest is pulled through from the remote registry,
an entry is added to the scheduler with a TTL. When the TTL expires the
scheduler calls a pre-specified function to remove the fetched resource.
Add token authentication to the registry middleware. Get a token at startup
and preload the credential store with the username and password supplied in the
config file.
Allow resumable digest functionality to be disabled at runtime and disable
it when the registry is a pull through cache.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Storage drivers can implement a method called URLFor which can return a direct
url for a given path. The functionality allows the registry to direct clients
to download content directly from the backend storage. This is commonly used
with s3 and cloudfront. Under certain conditions, such as when the registry is
not local to the backend, these redirects can hurt performance and waste
incoming bandwidth on pulls. This feature addition allows one to disable this
feature, if required.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Conflicts:
configuration/configuration.go
registry/handlers/app.go
registry/storage/catalog_test.go
registry/storage/manifeststore_test.go
registry/storage/registry.go
Implement the delete API by implementing soft delete for layers
and blobs by removing link files and updating the blob descriptor
cache. Deletion is configurable - if it is disabled API calls
will return an unsupported error.
We invalidate the blob descriptor cache by changing the linkedBlobStore's
blobStatter to a blobDescriptorService and naming it blobAccessController.
Delete() is added throughout the relevant API to support this functionality.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
After consideration, we've changed the main descriptor field name to for number
of bytes to "size" to match convention. While this may be a subjective
argument, commonly we refer to files by their "size" rather than their
"length". This will match other conventions, like `(FileInfo).Size()` and
methods on `io.SizeReaderAt`. Under more broad analysis, this argument doesn't
necessarily hold up. If anything, "size" is shorter than "length".
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Add a functional argument to pass a digest to (ManifestService).GetByTag().
If the digest matches an empty manifest and nil error are returned.
See 1bc740b0d5 for server implementation.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Previously, the most accurate reference for a manifest was the tag url. After
adding pull by digest, all event notifications should refer directly to the
digest url. This ensures that event uniquely identifies the target of the
notification. Testing has been added for manifest pull events to check that
this doesn't change.
In addition, the listener interface has been refactored to only use the
repository name, rather than the full repository object.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Ensure that clients can use the blob descriptor cache provider without needing
the redis package.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.
The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.
"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.
Recommend Review Approach
-------------------------
This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.
To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.
Storage
-------
The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links. Eventually, we
will fully consolidate this storage.
The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.
The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.
Cache
-----
Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.
One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.
Handlers
--------
The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.
Configuration
-------------
One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.
Notifications
-------------
Changes the `notification` package are simply to support the interface
changes.
Context
-------
A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
- Change driver interface to take a context as its first argument
- Make newFileReader take a context as its first argument
- Make newFileWriter take a context as its first argument
- Make blobstore exists and delete take a context as a first argument
- Pass the layerreader's context to the storage layer
- Pass the app's context to purgeuploads
- Store the app's context into the blobstore (was previously null)
- Pass the trace'd context to the storage drivers
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
This changeset integrates the layer info cache with the registry webapp and
storage backend. The main benefit is to cache immutable layer meta data,
reducing backend roundtrips. The cache can be configured to use either redis or
an inmemory cache.
This provides massive performance benefits for HEAD http checks on layer blobs
and manifest verification.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
through proxies.
Add a function to examine X-Forward-For and X-Real-Ip headers for
originating IP addresses. Use RemoteAddr for notification request
record and HTTP request context.
