52 weeks after Digdag operation

Self-introduction 4 id: @amesho 4 role: Operation,Development 4 product: Quant

Quant 4 constitution 4 rails 4 redshift 4 aurora 4 backend 4 Dependency complexity

Agenda 4 How has batch operation changed? 4 How has the situation changed? 4 Problems with operation.

How has batch operation changed?

Before

Cron

Problems of CRON 4 It is hard to understand which server is running 4 It is hard to understand startup time 4 Difficult to understand dependency 4 Elapsed time is difficult to understand

Problem solved by introducing digdag

Which server is running

Start-up time

Dependency

Elapsed time

How has the situation changed 4 It became easy to match recognition with your boss and colleagues 4 People who can not write programs can join 4 Safe to understand what you are doing

Operation 4 pullreq 4 checkout with digdag server 4 Processing in combination with embulk

Note 4 If you look closely at the document you will find detailed notes etc 4 Some things are not written in the document

API 4 It exists under digdag-server Ϋϥε ϝιου API ֓ཁ SessionResource GET /api/sessions List sessions from recent to old GET /api/sessions/{id} Get a session by id GET /api/sessions/{id}/attempts List attempts of a session AdminResource GET /api/admin/attempts/{id}/userinfo AttemptResource GET /api/attempts list attempts from recent to old GET /api/attempts?include_retried=1 list attempts from recent to old GET /api/attempts?project= list attempts that belong to a particular project GET /api/attempts?project=&workflow= list attempts that belong to a particular workflow GET /api/attempts/{id} show a session GET /api/attempts/{id}/tasks list tasks of a session GET /api/attempts/{id}/retries list retried attempts of this session PUT /api/attempts starts a new session POST /api/attempts/{id}/kill kill a session

Sample API access #!/bin/env ruby require 'net/http' require 'uri' require 'json' require 'time' url = URI.parse('http://localhost:65432/') res = Net::HTTP.start(url.host, url.port) {|http| http.get('/api/schedules') } schedules = JSON.parse(res.body) schedules['schedules'].sort_by { |s| Time.strptime(s['nextRunTime'],'%Y-%m-%dT%H:%M:%S%z').to_i }.each do |row| next if row['disabledAt'] printf("%s\t%s/%s\n",Time.strptime(row['nextRunTime'],'%Y-%m-%dT%H:%M:%S%z').localtime,row['project']['name'],row['workflow']['name']) end

storelastresults of operator

storelastresults: BOOLEAN 4 Whether to store the query results to redshift.last_results parameter. Default: false. 4 Setting first stores the first row to the parameter as an object (e.g. ${redshift.last_results.count}). 4 Setting all stores all rows to the parameter as an array of objects (e.g. $ {redshift.last_results[0].name}). If number of rows exceeds limit, task fails.

td operator

redshift operators

redshift operator is not implemented It is described in comments // TODO store_last_results should be io.digdag.standards.operator.jdbc.StoreLastResultsOption // instead of boolean to be consistent with pg> and redshift> operators but not implemented yet. this.storeLastResults = params.get("store_last_results", boolean.class, false);

Summary

end