Slide 1

Slide 1 text

Custom Volume Plugins @agonzalezro

Slide 2

Slide 2 text

Just one thing

Slide 3

Slide 3 text

It's easy!

Slide 4

Slide 4 text

Easier than you might think

Slide 5

Slide 5 text

What's a volume?

Slide 6

Slide 6 text

Persistent vs Non persistent

Slide 7

Slide 7 text

User POV

Slide 8

Slide 8 text

spec: containers: - name: web image: nginx ports: - name: web containerPort: 80 volumeMounts: - name: www-root mountPath: "/usr/share/nginx/html" volumes: - name: www-root flocker: datasetName: my-flocker-vol

Slide 9

Slide 9 text

... volumeMounts: - name: www-root mountPath: "/usr/share/nginx/html" volumes: - name: www-root flocker: datasetName: my-flocker-vol

Slide 10

Slide 10 text

Available 4 AWS EBS, GCE PD, Azure File 4 Git Repo 4 NFS 4 GlusterFS, Cinder 4 Flocker 4 Secrets

Slide 11

Slide 11 text

Is this new?

Slide 12

Slide 12 text

HTTP API

Slide 13

Slide 13 text

/VolumeDriver.Create .Mount .Path .Unmount .Remove

Slide 14

Slide 14 text

Go interface{}

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

1. Add it to the API (pkg/api/{,v1/}types.go): type VolumeSource struct { EmptyDir *EmptyDirVolumeSource `json:"emptyDir,omitempty"` Flocker *FlockerVolumeSource `json:"flocker,omitempty"` ...

Slide 17

Slide 17 text

2. Add it to the Kubelet (cmd/kubelet/app/plugins.go): func ProbeVolumePlugins(pluginDir string) []volume.VolumePlugin { allPlugins := []volume.VolumePlugin{} allPlugins = append(allPlugins, empty_dir.ProbeVolumePlugins()...) allPlugins = append(allPlugins, git_repo.ProbeVolumePlugins()...) ...

Slide 18

Slide 18 text

3. Implement the volume plugin (pkg/volume/...): type VolumePlugin interface { Init(host VolumeHost) error Name() string CanSupport(spec *Spec) bool NewBuilder(spec *Spec, podRef *api.Pod, opts VolumeOptions) (Builder, error) NewCleaner(name string, podUID types.UID) (Cleaner, error) }

Slide 19

Slide 19 text

type VolumePlugin interface { Init(host VolumeHost) error Name() string CanSupport(spec *Spec) bool ...

Slide 20

Slide 20 text

... NewBuilder( spec *Spec, podRef *api.Pod, opts VolumeOptions, ) (Builder, error) NewCleaner( name string, podUID types.UID, ) (Cleaner, error) }

Slide 21

Slide 21 text

Simplified example of NewBuilder func (p *plg) NewBuilder(spec, pod, opts) (volume.Builder, error) { source, _ := p.getFlockerVolumeSource(spec) builder := flockerBuilder{ flocker: &flocker{ datasetName: source.DatasetName, pod: pod, ... }, ... } return &builder, nil }

Slide 22

Slide 22 text

4. Implement the builder type Builder interface { Volume SetUp(fsGroup *int64) error SetUpAt(dir string, fsGroup *int64) error GetAttributes() Attributes }

Slide 23

Slide 23 text

Simplified example of SetUpAt for git repo

Slide 24

Slide 24 text

I lied

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

func (b *gitRepoVolumeBuilder) SetUpAt(dir string, fsGroup *int64) error { if volumeutil.IsReady(b.getMetaDir()) { return nil } wrapped, err := b.plugin.host.NewWrapperBuilder(b.volName, wrappedVolumeSpec, &b.pod, b.opts) if err != nil { return err } if err := wrapped.SetUpAt(dir, fsGroup); err != nil { return err } args := []string{"clone", b.source} if len(b.target) != 0 { args = append(args, b.target) } if output, err := b.execCommand("git", args, dir); err != nil { return fmt.Errorf("failed to exec 'git %s': %s: %v", strings.Join(args, " "), output, err) } files, err := ioutil.ReadDir(dir) if err != nil { return err } if len(b.revision) == 0 { // Done! volumeutil.SetReady(b.getMetaDir()) return nil } var subdir string switch { case b.target == ".": // if target dir is '.', use the current dir subdir = path.Join(dir) case len(files) == 1: // if target is not '.', use the generated folder subdir = path.Join(dir, files[0].Name()) default: // if target is not '.', but generated many files, it's wrong return fmt.Errorf("unexpected directory contents: %v", files) } if output, err := b.execCommand("git", []string{"checkout", b.revision}, subdir); err != nil { return fmt.Errorf("failed to exec 'git checkout %s': %s: %v", b.revision, output, err) } if output, err := b.execCommand("git", []string{"reset", "--hard"}, subdir); err != nil { return fmt.Errorf("failed to exec 'git reset --hard': %s: %v", output, err) } volumeutil.SetReady(b.getMetaDir()) return nil }

Slide 27

Slide 27 text

1. Check meta: was it called before? Is it ready? 2. Use empty dir to prepare the path 3. git clone it 4. Checkout the revision sent on the pod definition 5. Job done? Mark as ready

Slide 28

Slide 28 text

Important bit of empty_dir switch ed.medium { case api.StorageMediumDefault: err = ed.setupDir(dir) case api.StorageMediumMemory: err = ed.setupTmpfs(dir, securityContext) default: err = fmt.Errorf("unknown storage medium %q", ed.medium) } volume.SetVolumeOwnership(ed, fsGroup)

Slide 29

Slide 29 text

Another simplified example for Flocker

Slide 30

Slide 30 text

I lied again

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

func (b flockerBuilder) SetUpAt(dir string, fsGroup *int64) error { if volumeutil.IsReady(b.getMetaDir()) { return nil } if b.client == nil { c, err := b.newFlockerClient() if err != nil { return err } b.client = c } datasetID, err := b.client.GetDatasetID(dir) if err != nil { return err } s, err := b.client.GetDatasetState(datasetID) if err != nil { return fmt.Errorf("The volume '%s' is not available in Flocker. You need to create this manually with Flocker CLI before using it.", dir) } primaryUUID, err := b.client.GetPrimaryUUID() if err != nil { return err } if s.Primary != primaryUUID { if err := b.updateDatasetPrimary(datasetID, primaryUUID); err != nil { return err } } b.flocker.path = s.Path volumeutil.SetReady(b.getMetaDir()) return nil }

Slide 33

Slide 33 text

1. Was it called? Is it ready? 2. Is the dataset ready? If not, not my problem mate 3. If primary UUIDs doesn't match (rescheduled pod), update them 4. Wait until ready 5. Mark as ready

Slide 34

Slide 34 text

5. Implement the Persistent Volume Plugin type PersistentVolumePlugin interface { VolumePlugin GetAccessModes() []api.PersistentVolumeAccessMode }

Slide 35

Slide 35 text

func (p *plg) GetAccessModes() []api.PersistentVolumeAccessMode { return []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, } }

Slide 36

Slide 36 text

http://bit.ly/kubecon

Slide 37

Slide 37 text

"Problems"

Slide 38

Slide 38 text

4 CLAs 4 Shippable & Jenkins 4 hack/ scripts 4 PR

Slide 39

Slide 39 text

Summary 1. Add API 2. Enable the plugin on the Kubelet 3. Implement VolumePlugin interface: NewBuilder & NewCleaner 4. Implement the builder itself: SetUpAt 5. Persistent? 6. ❤

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Just one (more) thing

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

We are hiring at

Slide 44

Slide 44 text

Thanks! @agonzalezro