fcrozat/distribution
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md and documentation

Browse Source 
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>
This commit is contained in:
Arnaud Porterie 2014-12-23 17:48:14 -08:00
parent 83b810104c
commit e1eeec3e2f
2 changed files with 211 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
README.md
View File

@ -1,4 +1,55 @@
distribution
Distribution
============
The Docker toolset to pack, ship, store, and deliver content
The Docker toolset to pack, ship, store, and deliver content.
Planned content for this repository:
* Distribution related specifications
- Image format
- JSON registry API
* Registry implementation: a Golang implementation of the JSON API
* Client libraries to consume conforming implementations of the JSON API
# Ongoing open sprint
### What is an open sprint?
The open sprint is a focused effort of a small group of people to kick-off a new project, while commiting to becoming maintainers of the resulting work.
**Having a dedicated team work on the subject doesn't mean that you, the community, cannot contribute!** We need your input to make the best use of the sprint, and focus our work on what matters for you. For this particular topic:
* Come discuss on IRC: #docker-distribution on FreeNode
* Submit your ideas, and upvote those you think matter the most on [Google Moderator](https://www.google.com/moderator/?authuser=1#16/e=2165c3)
### Goal of the distribution sprint
Design a professional grade and extensible content distribution system, that allow users to:
* Enjoy an efficient, secured and reliable way to store, manage, package and exchange content
* Hack/roll their own on top of healthy open-source components
* Implement their own home made solution through good specs, and solid extensions mechanism.
### Schedule and expected output
The Open Sprint will start on **Monday December 29th**, and end on **Friday January 16th**.
What we want to achieve as a result is:
* Tactical fixes of today's frustrations in the existing Docker codebase
- This includes a throrough review of [docker/docker#9784](https://github.com/docker/docker/pull/9784) by core maintainers
* Laying the base of a new distribution subsystem, living independently, and with a well defined group of maintainers. This is the purpose of this repository, which aims at hosting:
- A specification of the v2 image format
- A specification of the JSON/HTTP protocol
- Server-side Go implementation of the v2 registry
- Client-side Go packages to consume this new API
- Standalone binaries providing content distribution functionalities outside of Docker
### How will this integrate with Docker engine?
Building awesome, independent, and well maintained distribution tools should give Docker core maintainers enough incentive to switch to the newly develop subsystem. We make no assumptions on a given date or milestone as urgency should be fixed through [docker/docker#9784](https://github.com/docker/docker/pull/9784), and in order to maintain focus on producing a top quality alternative.
### Relevant documents
* [Analysis of current state and goals](docs/opensprint/kickoff.md)

158
doc/opensprint/kickoff.md Normal file
View File

@ -0,0 +1,158 @@
Distribution
=========================
## Project intentions
**Problem statement and requirements**
* What is the exact scope of the problem?
Design a professional grade and extensible content distribution system, that allows docker users to:
... by default enjoy:
* an efficient, secured and reliable way to store, manage, package and exchange content
... optionally:
* can hack/roll their own on top of healthy open-source components
... with the liberty to:
* implement their own home made solution through good specs, and solid extensions mechanism
* Who will the result be useful to?
* users
* ISV (who distribute images or develop image distribution solutions)
* docker
* What are the use cases (distinguish dev & ops population where applicable)?
* Everyone (... uses docker push/pull).
* Why does it matter that we build this now?
* Shortcomings of the existing codebase are the #1 pain point (by large) for users, partners and ISV, hence the most urgent thing to address (?)
* That situation is getting worse everyday and killer competitors are going/have emerged.
* Who are the competitors?
* existing artifact storage solutions (eg: artifactory).
* emerging products that aim at handling pull/push in place of docker.
* ISV that are looking for alternatives to workaround this situation
**Current state: what do we have today?**
Problems of the existing system:
1. not reliable
* registry goes down whenever the hub goes down
* failing push result in broken repositories
* concurrent push is not handled
* python boto and gevent have a terrible history
* organically grown, under-designed features are in a bad shape (search)
2. inconsistent
* discrepancies between duplicated API (and *duplicated APIs*)
* unused features
* missing essential features (proper SSL support)
3. not reusable
* tightly entangled with hub component makes it very difficult to use outside of docker
* proper access-control is almost impossible to do right
* not easily extensible
4. not efficient
* no parallel operations (by design)
* sluggish client-side processing / bad pipeline design
* poor reusability of content (random ids)
* scalability issues (tags)
* too many useless requests (protocol)
* too much local space consumed (local garbage collection: broken + not efficient)
* no squashing
5. not resilient to errors
* no resume
* error handling is obscure or inexistent
6. security
* content is not verified
* current tarsum is broken
* random ids are a headache
7. confusing
* registry vs. registry.hub?
* layer vs. image?
8. broken features
* mirroring is not done correctly (too complex, bug-laden, caching is hard)
9. poor integration with the rest of the project
* technology discrepancy (python vs. go)
* poor testability
* poor separation (API in the engine is not defined enough)
10. missing features / prevents future
* trust / image signing
* naming / transport separation
* discovery / layer federation
* architecture + os support (eg: arm/windows)
* quotas
* alternative distribution methods (transport plugins)
**Future state: where do we want to get?**
* Deliverable
* new JSON/HTTP protocol specification
* new image format specification
* (new image store in the engine)
* new transport API between the engine and the distribution client code / new library
* new registry in go
* new authentication service on top of the trust graph in go
* What are the interactions with other components of the project?
* critical interactions with docker push/pull mechanism
* critical interactions with the way docker stores images locally
* In what way will the result be customizable?
* transport plugins allowing for radically different transport methods (bittorent, direct S3 access, etc)
* extensibility design for the registry allowing for complex integrations with other systems
* backend storage drivers API
## Kick-off output
**What is the expected output of the kick-off session?**
* draft specifications
* separate binary tool for demo purpose
* a mergeable PR that fixes 90% of the listed issues
* agree on a vision that allows solving all that are deemed worthy
* propose a long term battle plan with clear milestones that encompass all these
* define a first milestone that is compatible with the future and does already deliver some of the solutions
* deliver the specifications for image manifest format and transport API
* deliver a working implementation that can be used as a drop-in replacement for the existing v1 with an equivalent feature-set
**How is the output going to be demoed?**
docker pull
docker push
**Once demoed, what will be the path to shipping?**
A minimal PR that include the first subset of features to make docker work well with the new server side components.
## Pressing matters
* need a codename (ship, distribute)
* new repository
* new domains
* architecture / OS
* persistent ids
* registries discovery
* naming (quay.io/foo/bar)
* mirroring
## Assorted issues
* some devops want a docker engine that cannot do push/pull