Snickers: Open Source HTTP API for Media Encoding

Snickers, an open source alternative to encoding providers, presented at GoLab.io in Florence, Italy.

1. Open Source HTTP API for Media Encoding Snickers GoLab 2017. Florence, Italy
2. /flavioribeiro /flavioribeiro flavioribeiro.com senior engineer @ the new york times
3. context
4. context motivation
5. context motivation how it works?
6. context motivation how it works? future
7. context
8. context media factory client acquistion API media factory API transcoding API distribution API CDN Storage database http://nyt.ms/mediafactory
9. transcoding api transcoding API http://github.com/nytimes/video-transcoding-api
10. {      "providers":  ["elastictranscoder",  "elementalconductor",  "encodingcom",  "zencoder"],
    "preset":  {
        "name":  "sample_preset",
        "description":  "This  is  an  example  preset",
        "container":  "mp4",
        "rateControl":  "VBR",
        "video":  {                  "profile":  "Main",  "profileLevel":  "3.1",
                "height":  "720",  "width":  "1080",
                "codec":  "h264",  "bitrate":  "1000000",
                "gopSize":  "90",  "gopMode":  "fixed",
                "interlaceMode":  "progressive"
        },
        "audio":  {
                "codec":  "aac",
                "bitrate":  "64000"
        }
 } transcoding api - presets
11. transcoding api - jobs {      "provider":  "elastictranscoder",
    "source":  “ftp://nytimes:isg00d@ftp.nytimes.com/folder/my_video_source.mov”,      “destination”:  “ftp://nytimes:isg00d@ftp.nytimes.com/outputs”,
    "outputs":  [
        {"preset":  "720p_mp4",  "fileName":  "my_video_720p.mp4"},
        {"preset":  "1080p_mp4",  "fileName":  "my_video_1080p.mp4"},
        {"preset":  "256p_hls",  "fileName":  "hls/my_video_480p.m3u8"},
        {"preset":  "480p_hls",  "fileName":  "hls/my_video_480p.m3u8"},
        {"preset":  "720p_hls",  "fileName":  "hls/my_video_720p.m3u8"},
        {"preset":  "1080p_hls",  "fileName":  "hls/my_video_1080p.m3u8"},
        {"preset":  "2160p_hls",  "fileName":  "hls/my_video_2160p.m3u8"}
    ],      "streamingParams":  {
        "segmentDuration":  5,
        "protocol":  "hls"
    }
 }  
12. • old system worked well for ~10 years, not anymore • new system is a set of microservices in Go • fast encoding, scalability, reliability • transcoding API is a wrapper for encoding services • presets, encoding jobs context recap
13. motivation
14. motivation • team fluent in Go — except me
15. motivation • team fluent in Go — except me • I wanted to test different approaches
16. motivation • team fluent in Go — except me • I wanted to test different approaches • What if I create another encoding service in Go?
17. • open source alternative to encoding providers • deploy & run everywhere • add features any time • compatible with transcoding api jobs and presets What if I create another encoding service in Go?
19. how it works?
20. RESTful API how it works?
21. var  Routes  =  map[Route]RouterArguments{      //Job  routes      CreateJob:          RouterArguments{Path:  "/jobs",  Method:  http.MethodPost},      ListJobs:            RouterArguments{Path:  "/jobs",  Method:  http.MethodGet},      GetJobDetails:  RouterArguments{Path:  "/jobs/{jobID}",  Method:  http.MethodGet},      StartJob:            RouterArguments{Path:  "/jobs/{jobID}/start",  Method:  http.MethodPost},      //Preset  routes      CreatePreset:          RouterArguments{Path:  "/presets",  Method:  http.MethodPost},      UpdatePreset:          RouterArguments{Path:  "/presets",  Method:  http.MethodPut},      ListPresets:            RouterArguments{Path:  "/presets",  Method:  http.MethodGet},      GetPresetDetails:  RouterArguments{Path:  "/presets/{presetName}",  Method:  http.MethodGet},      DeletePreset:          RouterArguments{Path:  "/presets/{presetName}",  Method:  http.MethodDelete},   } how it works? server/routes.go
22. RESTful API database how it works?
23. RESTful API database pipeline downloaders uploaders encoding engines how it works?
24. func  StartJob(cfg  gonfig.Gonfig,  dbInstance  db.Storage,  job  types.Job)  {      newJob,  err  :=  SetupJob(job.ID,  dbInstance,  cfg)      job  =  *newJob      if  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return    }      downloadFunc  :=  downloaders.GetDownloadFunc(job.Source)      if  err  :=  downloadFunc(log,  cfg,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return    }      encodeFunc  :=  encoders.GetEncodeFunc(job)      if  err  :=  encodeFunc(logger,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return      }      uploadFunc  :=  uploaders.GetUploadFunc(job.Destination)      if  err  :=  uploadFunc(logger,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return      }      CleanSwap(dbInstance,  job.ID);  err  !=  nil      job.Status  =  types.JobFinished      dbInstance.UpdateJob(job.ID,  job)   }   how it works? pipeline/pipeline.go
25. RESTful API database pipeline downloaders uploaders encoding engines ffmpeg binding how it works?
26. how it works? • encoding engines • Cgo wrapper for FFmpeg functions • https://github.com/3d0c/gmf
27. how it works? package  segmenter   /*   #include  <stdio.h>   #include  "libavformat/avformat.h"   #include  <libavdevice/avdevice.h>   #include  "c/segmenter.h"   #include  "c/util.h"   #cgo  LDFLAGS:  -­‐L${SRCDIR}/../build  -­‐lsegmenter  -­‐lavcodec  -­‐lavformat  -­‐lavutil   */   import  "C"   import  (     "fmt"     "os"     "unsafe"     "github.com/3d0c/gmf"   )  
28. DEMO!
30. future
31. future transcoding API
32. future • multibitrate HLS • go client (WIP) • add another encoding engine using GStreamer
33. We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com
34. Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com
35. Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com
36. thank you! { , } /flavioribeiro