Open Source HTTP API for Media Encoding Snickers GoLab 2017. Florence, Italy

/flavioribeiro /flavioribeiro flavioribeiro.com senior engineer @ the new york times

context

context motivation

context motivation how it works?

context motivation how it works? future

context

context media factory client acquistion API media factory API transcoding API distribution API CDN Storage database http://nyt.ms/mediafactory

transcoding api transcoding API http://github.com/nytimes/video-transcoding-api

{      "providers":  ["elastictranscoder",  "elementalconductor",  "encodingcom",  "zencoder"],
    "preset":  {
        "name":  "sample_preset",
        "description":  "This  is  an  example  preset",
        "container":  "mp4",
        "rateControl":  "VBR",
        "video":  {                  "profile":  "Main",  "profileLevel":  "3.1",
                "height":  "720",  "width":  "1080",
                "codec":  "h264",  "bitrate":  "1000000",
                "gopSize":  "90",  "gopMode":  "fixed",
                "interlaceMode":  "progressive"
        },
        "audio":  {
                "codec":  "aac",
                "bitrate":  "64000"
        }
 } transcoding api - presets

transcoding api - jobs {      "provider":  "elastictranscoder",
    "source":  “ftp://nytimes:isg00d@ftp.nytimes.com/folder/my_video_source.mov”,      “destination”:  “ftp://nytimes:isg00d@ftp.nytimes.com/outputs”,
    "outputs":  [
        {"preset":  "720p_mp4",  "fileName":  "my_video_720p.mp4"},
        {"preset":  "1080p_mp4",  "fileName":  "my_video_1080p.mp4"},
        {"preset":  "256p_hls",  "fileName":  "hls/my_video_480p.m3u8"},
        {"preset":  "480p_hls",  "fileName":  "hls/my_video_480p.m3u8"},
        {"preset":  "720p_hls",  "fileName":  "hls/my_video_720p.m3u8"},
        {"preset":  "1080p_hls",  "fileName":  "hls/my_video_1080p.m3u8"},
        {"preset":  "2160p_hls",  "fileName":  "hls/my_video_2160p.m3u8"}
    ],      "streamingParams":  {
        "segmentDuration":  5,
        "protocol":  "hls"
    }
 }  

• old system worked well for ~10 years, not anymore • new system is a set of microservices in Go • fast encoding, scalability, reliability • transcoding API is a wrapper for encoding services • presets, encoding jobs context recap

motivation

motivation • team fluent in Go — except me

motivation • team fluent in Go — except me • I wanted to test different approaches

motivation • team fluent in Go — except me • I wanted to test different approaches • What if I create another encoding service in Go?

• open source alternative to encoding providers • deploy & run everywhere • add features any time • compatible with transcoding api jobs and presets What if I create another encoding service in Go?

No content

how it works?

RESTful API how it works?

var  Routes  =  map[Route]RouterArguments{      //Job  routes      CreateJob:          RouterArguments{Path:  "/jobs",  Method:  http.MethodPost},      ListJobs:            RouterArguments{Path:  "/jobs",  Method:  http.MethodGet},      GetJobDetails:  RouterArguments{Path:  "/jobs/{jobID}",  Method:  http.MethodGet},      StartJob:            RouterArguments{Path:  "/jobs/{jobID}/start",  Method:  http.MethodPost},      //Preset  routes      CreatePreset:          RouterArguments{Path:  "/presets",  Method:  http.MethodPost},      UpdatePreset:          RouterArguments{Path:  "/presets",  Method:  http.MethodPut},      ListPresets:            RouterArguments{Path:  "/presets",  Method:  http.MethodGet},      GetPresetDetails:  RouterArguments{Path:  "/presets/{presetName}",  Method:  http.MethodGet},      DeletePreset:          RouterArguments{Path:  "/presets/{presetName}",  Method:  http.MethodDelete},   } how it works? server/routes.go

RESTful API database how it works?

RESTful API database pipeline downloaders uploaders encoding engines how it works?

func  StartJob(cfg  gonfig.Gonfig,  dbInstance  db.Storage,  job  types.Job)  {      newJob,  err  :=  SetupJob(job.ID,  dbInstance,  cfg)      job  =  *newJob      if  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return    }      downloadFunc  :=  downloaders.GetDownloadFunc(job.Source)      if  err  :=  downloadFunc(log,  cfg,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return    }      encodeFunc  :=  encoders.GetEncodeFunc(job)      if  err  :=  encodeFunc(logger,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return      }      uploadFunc  :=  uploaders.GetUploadFunc(job.Destination)      if  err  :=  uploadFunc(logger,  dbInstance,  job.ID);  err  !=  nil  {          updateJobWithError(dbInstance,  job,  err.Error())          return      }      CleanSwap(dbInstance,  job.ID);  err  !=  nil      job.Status  =  types.JobFinished      dbInstance.UpdateJob(job.ID,  job)   }   how it works? pipeline/pipeline.go

RESTful API database pipeline downloaders uploaders encoding engines ffmpeg binding how it works?

how it works? • encoding engines • Cgo wrapper for FFmpeg functions • https://github.com/3d0c/gmf

how it works? package  segmenter   /*   #include     #include  "libavformat/avformat.h"   #include     #include  "c/segmenter.h"   #include  "c/util.h"   #cgo  LDFLAGS:  -­‐L${SRCDIR}/../build  -­‐lsegmenter  -­‐lavcodec  -­‐lavformat  -­‐lavutil   */   import  "C"   import  (     "fmt"     "os"     "unsafe"     "github.com/3d0c/gmf"   )  

DEMO!

No content

future

future transcoding API

future • multibitrate HLS • go client (WIP) • add another encoding engine using GStreamer

We’re hiring nyti.ms/technology @NYTDevs | developers.nytimes.com

Stay updated open.blogs.nytimes.com @NYTDevs | developers.nytimes.com

Connect with us on Slack! http://video-dev.org @NYTDevs | developers.nytimes.com

thank you! { , } /flavioribeiro