Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Snickers: Open Source HTTP API for Media Encoding

Snickers: Open Source HTTP API for Media Encoding

Snickers, an open source alternative to encoding providers, presented at GoLab.io in Florence, Italy.

5dc4abe3dc14c25021ff2295fc380d73?s=128

Flávio Ribeiro

January 20, 2017
Tweet

Transcript

  1. Open Source HTTP API for Media Encoding Snickers GoLab 2017.

    Florence, Italy
  2. /flavioribeiro /flavioribeiro flavioribeiro.com senior engineer @ the new york times

  3. context

  4. context motivation

  5. context motivation how it works?

  6. context motivation how it works? future

  7. context

  8. context media factory client acquistion API media factory API transcoding

    API distribution API CDN Storage database http://nyt.ms/mediafactory
  9. transcoding api transcoding API http://github.com/nytimes/video-transcoding-api

  10. {      "providers":  ["elastictranscoder",  "elementalconductor",  "encodingcom",  "zencoder"],
    "preset":

     {
        "name":  "sample_preset",
        "description":  "This  is  an  example  preset",
        "container":  "mp4",
        "rateControl":  "VBR",
        "video":  {                  "profile":  "Main",  "profileLevel":  "3.1",
                "height":  "720",  "width":  "1080",
                "codec":  "h264",  "bitrate":  "1000000",
                "gopSize":  "90",  "gopMode":  "fixed",
                "interlaceMode":  "progressive"
        },
        "audio":  {
                "codec":  "aac",
                "bitrate":  "64000"
        }
 } transcoding api - presets
  11. transcoding api - jobs {      "provider":  "elastictranscoder",
  

     "source":  “ftp://nytimes:isg00d@ftp.nytimes.com/folder/my_video_source.mov”,      “destination”:  “ftp://nytimes:isg00d@ftp.nytimes.com/outputs”,
    "outputs":  [
        {"preset":  "720p_mp4",  "fileName":  "my_video_720p.mp4"},
        {"preset":  "1080p_mp4",  "fileName":  "my_video_1080p.mp4"},
        {"preset":  "256p_hls",  "fileName":  "hls/my_video_480p.m3u8"},
        {"preset":  "480p_hls",  "fileName":  "hls/my_video_480p.m3u8"},
        {"preset":  "720p_hls",  "fileName":  "hls/my_video_720p.m3u8"},
        {"preset":  "1080p_hls",  "fileName":  "hls/my_video_1080p.m3u8"},
        {"preset":  "2160p_hls",  "fileName":  "hls/my_video_2160p.m3u8"}
    ],      "streamingParams":  {
        "segmentDuration":  5,
        "protocol":  "hls"
    }
 }  
  12. • old system worked well for ~10 years, not anymore

    • new system is a set of microservices in Go • fast encoding, scalability, reliability • transcoding API is a wrapper for encoding services • presets, encoding jobs context recap
  13. motivation

  14. motivation • team fluent in Go — except me

  15. motivation • team fluent in Go — except me •

    I wanted to test different approaches
  16. motivation • team fluent in Go — except me •

    I wanted to test different approaches • What if I create another encoding service in Go?
  17. • open source alternative to encoding providers • deploy &

    run everywhere • add features any time • compatible with transcoding api jobs and presets What if I create another encoding service in Go?
  18. None
  19. how it works?

  20. RESTful API how it works?

  21. var  Routes  =  map[Route]RouterArguments{      //Job  routes    

     CreateJob:          RouterArguments{Path:  "/jobs",  Method:  http.MethodPost},      ListJobs:            RouterArguments{Path:  "/jobs",  Method:  http.MethodGet},      GetJobDetails:  RouterArguments{Path:  "/jobs/{jobID}",  Method:  http.MethodGet},      StartJob:            RouterArguments{Path:  "/jobs/{jobID}/start",  Method:  http.MethodPost},      //Preset  routes      CreatePreset:          RouterArguments{Path:  "/presets",  Method:  http.MethodPost},      UpdatePreset:          RouterArguments{Path:  "/presets",  Method:  http.MethodPut},      ListPresets:            RouterArguments{Path:  "/presets",  Method:  http.MethodGet},      GetPresetDetails:  RouterArguments{Path:  "/presets/{presetName}",  Method:  http.MethodGet},      DeletePreset:          RouterArguments{Path:  "/presets/{presetName}",  Method:  http.MethodDelete},   } how it works? server/routes.go
  22. RESTful API database how it works?

  23. RESTful API database pipeline downloaders uploaders encoding engines how it

    works?
  24. func  StartJob(cfg  gonfig.Gonfig,  dbInstance  db.Storage,  job  types.Job)  {    

     newJob,  err  :=  SetupJob(job.ID,  dbInstance,  cfg)      job  =  *newJob      if  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return    }      downloadFunc  :=  downloaders.GetDownloadFunc(job.Source)      if  err  :=  downloadFunc(log,  cfg,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return    }      encodeFunc  :=  encoders.GetEncodeFunc(job)      if  err  :=  encodeFunc(logger,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return      }      uploadFunc  :=  uploaders.GetUploadFunc(job.Destination)      if  err  :=  uploadFunc(logger,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return      }      CleanSwap(dbInstance,  job.ID);  err  !=  nil      job.Status  =  types.JobFinished      dbInstance.UpdateJob(job.ID,  job)   }   how it works? pipeline/pipeline.go
  25. RESTful API database pipeline downloaders uploaders encoding engines ffmpeg binding

    how it works?
  26. how it works? • encoding engines • Cgo wrapper for

    FFmpeg functions • https://github.com/3d0c/gmf
  27. how it works? package  segmenter   /*   #include  <stdio.h>

      #include  "libavformat/avformat.h"   #include  <libavdevice/avdevice.h>   #include  "c/segmenter.h"   #include  "c/util.h"   #cgo  LDFLAGS:  -­‐L${SRCDIR}/../build  -­‐lsegmenter  -­‐lavcodec  -­‐lavformat  -­‐lavutil   */   import  "C"   import  (     "fmt"     "os"     "unsafe"     "github.com/3d0c/gmf"   )  
  28. DEMO!

  29. None
  30. future

  31. future transcoding API

  32. future • multibitrate HLS • go client (WIP) • add

    another encoding engine using GStreamer
  33. We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com

  34. Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com

  35. Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com

  36. thank you! { , } /flavioribeiro