ba6b774aea
To smooth initial implementation, uploads were spooled to local file storage, validated, then pushed to remote storage. That approach was flawed in that it present easy clustering of registry services that share a remote storage backend. The original plan was to implement resumable hashes then implement remote upload storage. After some thought, it was found to be better to get remote spooling working, then optimize with resumable hashes. Moving to this approach has tradeoffs: after storing the complete upload remotely, the node must fetch the content and validate it before moving it to the final location. This can double bandwidth usage to the remote backend. Modifying the verification and upload code to store intermediate hashes should be trivial once the layer digest format has settled. The largest changes for users of the storage package (mostly the registry app) are the LayerService interface and the LayerUpload interface. The LayerService now takes qualified repository names to start and resume uploads. In corallry, the concept of LayerUploadState has been complete removed, exposing all aspects of that state as part of the LayerUpload object. The LayerUpload object has been modified to work as an io.WriteSeeker and includes a StartedAt time, to allow for upload timeout policies. Finish now only requires a digest, eliding the requirement for a size parameter. Resource cleanup has taken a turn for the better. Resources are cleaned up after successful uploads and during a cancel call. Admittedly, this is probably not completely where we want to be. It's recommend that we bolster this with a periodic driver utility script that scans for partial uploads and deletes the underlying data. As a small benefit, we can leave these around to better understand how and why these uploads are failing, at the cost of some extra disk space. Many other changes follow from the changes above. The webapp needs to be updated to meet the new interface requirements. Signed-off-by: Stephen J Day <stephen.day@docker.com>
Distribution
The Docker toolset to pack, ship, store, and deliver content.
Planned content for this repository:
- Distribution related specifications
- Image format
- JSON registry API
- Registry implementation: a Golang implementation of the JSON API
- Client libraries to consume conforming implementations of the JSON API
Ongoing open sprint
What is an open sprint?
The open sprint is a focused effort of a small group of people to kick-off a new project, while commiting to becoming maintainers of the resulting work.
Having a dedicated team work on the subject doesn't mean that you, the community, cannot contribute! We need your input to make the best use of the sprint, and focus our work on what matters for you. For this particular topic:
- Come discuss on IRC: #docker-distribution on FreeNode
- Submit your ideas, and upvote those you think matter the most on Google Moderator
Goal of the distribution sprint
Design a professional grade and extensible content distribution system, that allow users to:
- Enjoy an efficient, secured and reliable way to store, manage, package and exchange content
- Hack/roll their own on top of healthy open-source components
- Implement their own home made solution through good specs, and solid extensions mechanism.
Schedule and expected output
The Open Sprint will start on Monday December 29th, and end on Friday January 16th.
What we want to achieve as a result is:
-
Tactical fixes of today's frustrations in the existing Docker codebase
- This includes a throrough review of docker/docker#9784 by core maintainers
-
Laying the base of a new distribution subsystem, living independently, and with a well defined group of maintainers. This is the purpose of this repository, which aims at hosting:
- A specification of the v2 image format
- A specification of the JSON/HTTP protocol
- Server-side Go implementation of the v2 registry
- Client-side Go packages to consume this new API
- Standalone binaries providing content distribution functionalities outside of Docker
-
Constraints for interface provided by Distribution to Core:
- The distribution repository is a collection of tools for packaging and shipping content with Docker
- All tools are usable primarily as standalone command-line tools. They may also expose bindings in one or more programming languages. Typically the first available language is Go, but that is not automatically true and more languages will be supported over time
- The distribution repository is still under incubation, any code layout, interface and name may change before it gets included in a stable release of Docker
How will this integrate with Docker engine?
Building awesome, independent, and well maintained distribution tools should give Docker core maintainers enough incentive to switch to the newly develop subsystem. We make no assumptions on a given date or milestone as urgency should be fixed through docker/docker#9784, and in order to maintain focus on producing a top quality alternative.