Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Custom Volume Plugins

Custom Volume Plugins

This talk will introduce you to the concept of Kubernetes Volume plugins. We will not only help you understand the basic concepts, but more importantly, using practical examples, we will show how you can develop your own volume plugins and contribute them back to the community of the OSS project as large as Kubernetes.

We will conclude the talk by discussing various challenges one can come across when contributing to a high velocity OSS project of Kubernetes' size which can help you avoid the pain and enjoy the path.

Alexandre González

March 10, 2016
Tweet

More Decks by Alexandre González

Other Decks in Technology

Transcript

  1. spec: containers: - name: web image: nginx ports: - name:

    web containerPort: 80 volumeMounts: - name: www-root mountPath: "/usr/share/nginx/html" volumes: - name: www-root flocker: datasetName: my-flocker-vol
  2. Available 4 AWS EBS, GCE PD, Azure File 4 Git

    Repo 4 NFS 4 GlusterFS, Cinder 4 Flocker 4 Secrets
  3. 1. Add it to the API (pkg/api/{,v1/}types.go): type VolumeSource struct

    { EmptyDir *EmptyDirVolumeSource `json:"emptyDir,omitempty"` Flocker *FlockerVolumeSource `json:"flocker,omitempty"` ...
  4. 2. Add it to the Kubelet (cmd/kubelet/app/plugins.go): func ProbeVolumePlugins(pluginDir string)

    []volume.VolumePlugin { allPlugins := []volume.VolumePlugin{} allPlugins = append(allPlugins, empty_dir.ProbeVolumePlugins()...) allPlugins = append(allPlugins, git_repo.ProbeVolumePlugins()...) ...
  5. 3. Implement the volume plugin (pkg/volume/...): type VolumePlugin interface {

    Init(host VolumeHost) error Name() string CanSupport(spec *Spec) bool NewBuilder(spec *Spec, podRef *api.Pod, opts VolumeOptions) (Builder, error) NewCleaner(name string, podUID types.UID) (Cleaner, error) }
  6. ... NewBuilder( spec *Spec, podRef *api.Pod, opts VolumeOptions, ) (Builder,

    error) NewCleaner( name string, podUID types.UID, ) (Cleaner, error) }
  7. Simplified example of NewBuilder func (p *plg) NewBuilder(spec, pod, opts)

    (volume.Builder, error) { source, _ := p.getFlockerVolumeSource(spec) builder := flockerBuilder{ flocker: &flocker{ datasetName: source.DatasetName, pod: pod, ... }, ... } return &builder, nil }
  8. 4. Implement the builder type Builder interface { Volume SetUp(fsGroup

    *int64) error SetUpAt(dir string, fsGroup *int64) error GetAttributes() Attributes }
  9. func (b *gitRepoVolumeBuilder) SetUpAt(dir string, fsGroup *int64) error { if

    volumeutil.IsReady(b.getMetaDir()) { return nil } wrapped, err := b.plugin.host.NewWrapperBuilder(b.volName, wrappedVolumeSpec, &b.pod, b.opts) if err != nil { return err } if err := wrapped.SetUpAt(dir, fsGroup); err != nil { return err } args := []string{"clone", b.source} if len(b.target) != 0 { args = append(args, b.target) } if output, err := b.execCommand("git", args, dir); err != nil { return fmt.Errorf("failed to exec 'git %s': %s: %v", strings.Join(args, " "), output, err) } files, err := ioutil.ReadDir(dir) if err != nil { return err } if len(b.revision) == 0 { // Done! volumeutil.SetReady(b.getMetaDir()) return nil } var subdir string switch { case b.target == ".": // if target dir is '.', use the current dir subdir = path.Join(dir) case len(files) == 1: // if target is not '.', use the generated folder subdir = path.Join(dir, files[0].Name()) default: // if target is not '.', but generated many files, it's wrong return fmt.Errorf("unexpected directory contents: %v", files) } if output, err := b.execCommand("git", []string{"checkout", b.revision}, subdir); err != nil { return fmt.Errorf("failed to exec 'git checkout %s': %s: %v", b.revision, output, err) } if output, err := b.execCommand("git", []string{"reset", "--hard"}, subdir); err != nil { return fmt.Errorf("failed to exec 'git reset --hard': %s: %v", output, err) } volumeutil.SetReady(b.getMetaDir()) return nil }
  10. 1. Check meta: was it called before? Is it ready?

    2. Use empty dir to prepare the path 3. git clone it 4. Checkout the revision sent on the pod definition 5. Job done? Mark as ready
  11. Important bit of empty_dir switch ed.medium { case api.StorageMediumDefault: err

    = ed.setupDir(dir) case api.StorageMediumMemory: err = ed.setupTmpfs(dir, securityContext) default: err = fmt.Errorf("unknown storage medium %q", ed.medium) } volume.SetVolumeOwnership(ed, fsGroup)
  12. func (b flockerBuilder) SetUpAt(dir string, fsGroup *int64) error { if

    volumeutil.IsReady(b.getMetaDir()) { return nil } if b.client == nil { c, err := b.newFlockerClient() if err != nil { return err } b.client = c } datasetID, err := b.client.GetDatasetID(dir) if err != nil { return err } s, err := b.client.GetDatasetState(datasetID) if err != nil { return fmt.Errorf("The volume '%s' is not available in Flocker. You need to create this manually with Flocker CLI before using it.", dir) } primaryUUID, err := b.client.GetPrimaryUUID() if err != nil { return err } if s.Primary != primaryUUID { if err := b.updateDatasetPrimary(datasetID, primaryUUID); err != nil { return err } } b.flocker.path = s.Path volumeutil.SetReady(b.getMetaDir()) return nil }
  13. 1. Was it called? Is it ready? 2. Is the

    dataset ready? If not, not my problem mate 3. If primary UUIDs doesn't match (rescheduled pod), update them 4. Wait until ready 5. Mark as ready
  14. 5. Implement the Persistent Volume Plugin type PersistentVolumePlugin interface {

    VolumePlugin GetAccessModes() []api.PersistentVolumeAccessMode }
  15. Summary 1. Add API 2. Enable the plugin on the

    Kubelet 3. Implement VolumePlugin interface: NewBuilder & NewCleaner 4. Implement the builder itself: SetUpAt 5. Persistent? 6. ❤