2019/06/27 Data Pipeline Casual Talk #3
Argo Workflow ʹΑΔػցֶशϫʔΫϑϩʔཧShotaro Tanaka / @yubessy / ϦϒηϯεData Pipeline Casual Talk #3
View Slide
͢͜ͱͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ• ϦϒηϯεͷαʔϏεͱMLγεςϜ• MLγεςϜͷ։ൃɾӡ༻ࣄ• MLγεςϜͷίϯϙʔωϯτׂͱίϯςφԽArgo Workflow ΛͲ͏͍ͬͯΔ͔• Argo Workflow ͷجຊػೳ• ϦϒηϯεͰͷ Argo Workflow ӡ༻※ Kubernetes ͷجૅࣝΛલఏͱ͍ͯ͠·͢
αʔϏεͱMLγεςϜ
ϦϒηϯεͷαʔϏε
ϦϒηϯεͰͷMLར༻αʔϏεͱML• ٻਓɾෆಈ࢈ྖҬͰෳͷWebαʔϏεΛӡӦ• MLγεςϜͷ։ൃɾӡ༻νʔϜԣஅ৫ͱͯ͠αʔϏε͔Βಠཱ• ֤αʔϏεʹϨίϝϯυޮՌ༧ଌϞσϧͳͲෳͷMLγεςϜΛఏڙओͳMLγεςϜ• ٻਓϨίϝϯυΤϯδϯ• Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ• A/BςετɾόϯσΟοτπʔϧ
ٻਓϨίϝϯυΤϯδϯ• ϚοϋόΠτɾస৬φϏͳͲͷϢʔβʹٻਓΛਪન• ϝʔϧɾWebαΠτɾωΠςΟϒΞϓϦͳͲ༷ʑͳॴͰಋೖ
ٻਓϨίϝϯυΤϯδϯͷ෦• ධՁɾίϯςϯπΛͱʹ MF, FM ͳͲͷΞϧΰϦζϜΛద༻͠είΞΛࢉग़• user-item item-item ͷϦετΛόονॲཧͰੜ֤͠αʔϏεʹఏڙ
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ• ϚοϋόΠτɾస৬φϏͳͲͷٻਓͷCVRԠื୯ՁΛࢉग़• ݕࡧ݁ՌͷॱҐ੍ޚࠂग़ߘͷ࠷దԽʹ׆༻
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧͷ෦• ϩάΛ༻͍ͯϕΠζਪఆɾϩδεςΟοΫճؼͰ༧ଌɾਪఆ• σΟϨΫλʔ͚ʹ؆қతͳϏϡʔϫΛWebΞϓϦͱͯ͠։ൃ
A/BςετɾόϯσΟοτπʔϧ• A/Bςετͷύλʔϯ৴ൺΛόϯσΟοτΞϧΰϦζϜͰࣗಈௐ• WebαΠτɾωΠςΟϒΞϓϦͷ࠷దԽΛޮԽ
A/BςετɾόϯσΟοτπʔϧͷ෦• ཧը໘͔ΒύλʔϯΛొ͠ɺWeb APIͰϥϯμϜʹ৴• CVϩάΛੳج൫Ͱूܭ͠ɺύλʔϯ৴ൺΛࣗಈߋ৽
MLγεςϜͷ։ൃɾӡ༻ࣄ
ϦϒηϯεͷMLγεςϜͷಛֶशɾ༧ଌͱେ෦͕όονॲཧ• ֶश: CVR༧ଌϞσϧͷֶशϨίϝϯυͷҼࢠղ• ༧ଌ: ݕࡧɾϨίϝϯυ༻ͷείΞΛࣄલʹҰׅܭࢉόονॲཧͷߏ͕ෳࡶ• ୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ• ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍• తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ
୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ• ϨίϝϯυΤϯδϯͰෳͷΞϧΰϦζϜΛΈ߹Θͤͯ͏• ϑΟϧλϦϯάϦετͷϚʔδΛߦͬͯϨίϝϯυϦετΛੜ
ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍• ಉαʔϏεͰA/BςετͷͨΊΞϧΰϦζϜ͚ͩมߋ• ผαʔϏεͷԣల։ͷࡍʹΞϧΰϦζϜΛ࠶ར༻
తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚ΔٻਓαʔϏεࠂαʔϏεͳͲͱൺ୯Ձ͕େ͖͘CVR͕খ͍͞→ ࠷ਪఆϕʔεͷҰൠతͳMLϥΠϒϥϦ͕ద͠ͳ͍͜ͱ→ ϞσϧɾΞϧΰϦζϜͷࣗલ࣮ͷͨΊݴޠɾϥΠϒϥϦΛ͍͚Δ• ϨίϝϯυΞϧΰϦζϜΛ Julia Ͱ࣮• Alternating Least SquaresʹΑΔFactorization Machinesͷύϥϝʔλਪఆ• Factorization MachinesΛϨίϝϯσʔγϣϯͰ͏ͱ͖ͷධՁਪఆܭࢉ• ਪఆɾ༧ଌϞσϧͰ Stan Λར༻• ֊ϕΠζʹΑΔখඪຊσʔλͷൺͷਪఆ
ෳࡶԽ͢ΔߏͷରॲҎલ֤γεςϜ͕୯ҰϨϙδτϦͰཧ͞ΕΔϞϊϦγοΫͳߏ→ ߏͷෳࡶԽͰ։ൃɾӡ༻͕·ΘΒͳ͘ͳ͖ͬͯͨ• MLͷίΞ෦ͱDBIO͕ີ݁߹͠ݸผ࣮ߦͰ͖ͳ͍• γεςϜؒͰڞ௨͢ΔΞϧΰϦζϜ͕ίϐϖ͞ΕΔ• ಉҰͷόονॲཧͰεςοϓ͝ͱʹݴޠΛม͑ʹ͍͘→ γεςϜΛػೳ͝ͱʹׂɾ࠶ߏங͢Δ͜ͱʹ
ίϯϙʔωϯτׂͱίϯςφԽ
ίϯϙʔωϯτͷׂ·ͣγεςϜΛ࣍ͷΑ͏ͳ୯ػೳίϯϙʔωϯτʹׂͨ͠• ֤ίϯϙʔωϯτ CLI Ͱ୯ಠ࣮ߦͰ͖Δ• ίϯϙʔωϯτؒͷೖग़ྗͯ͢ϑΝΠϧΛհ͢Δname role input file output filesqlkit DBIO SQL CSVnlpkit ࣗવݴޠॲཧ ςΩετ BoWϕΫτϧrecommender Ϩίϝϯυ ධՁ ਪનείΞ
ίϯϙʔωϯτͷίϯςφԽ͞Βʹ֤ίϯϙʔωϯτΛ୯ҰͷίϯςφΠϝʔδʹͨ͠• ֤ίϯςφίϯϙʔωϯτ docker run kubectl run Ͱ࣮ߦͰ͖Δ• γεςϜ͝ͱͷࠩ΄΅ઃఆϑΝΠϧSQL͚ͩͰදݱ# load datasetdocker run -v $(pwd):/workdir sqlkit select ratings.sql /workdir/ratings.csvdocker run -v $(pwd):/workdir sqlkit select content.sql /workdir/content.csv# preprocessdocker run -v $(pwd):/workdir nlpkit vectorize /workdir/content.csv /workdir/features.csv# run recommenderdocker run -v $(pwd):/workdir recommender predict config.yaml /workdir
ίϯϙʔωϯτͷׂͱίϯςφԽ
ϫʔΫϑϩʔΛͲ͏࣮ݱ͢Δ͔ʁίϯϙʔωϯτͷׂͱίϯςφԽʹΑΓෳͷ՝ΛղܾͰ͖ͨ• ີ݁߹ͷղফɾεςοϓ࣮ߦͷՄೳԽ• ڞ௨෦ͷ࠶ར༻ՄೳԽ• ݴޠɾϥΠϒϥϦͷ͍͚ͷ༰қԽ͔͠͠ɺෳࡶͳϫʔΫϑϩʔΛͲ͏ߏஙɾཧ͢Δ͔ͷ՝Δ• ୯७ͳόονॲཧͳΒ docker run kubectl run Λஞ࣮࣍ߦ͢Δ͚ͩ• ࣮ࡍʹ͜ͷํࣜͰຊ൪Քಇ͍ͯ͠ΔγεςϜଘࡏ• ฒྻԽɾϦτϥΠͳͲͷߴͳϫʔΫϑϩʔΛ࣮ݱ͍ͨ͠߹ʁ
ͦΜͳ͋Δ (2017)
Argo Workflow Λൃݟhttps://argoproj.github.io/
Argo Workflow"Container native workflow engine for Kubernetes"Kubernetes ্Ͱෳͷίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ࣮ߦͰ͖Δͻͱ͜ͱͰݴ͏ͱʮߴػೳͳ k8s Jobʯ• ෳίϯςφͷྻɾฒྻɾDAG࣮ߦ• ذɾϧʔϓɾϑοΫͷ੍ޚϑϩʔ• ϦτϥΠɾλΠϜΞτɾϫʔΧʔϊʔυબ• ϞχλϦϯά༻ Web UI
Argo Workflow ͷಛCRD controller ͱ࣮ͯ͠͞Ε͍ͯΔ• argo submit Ͱ࡞͞Εͨ Workflow ϦιʔεΛ controller ͕࣮ߦ• ϫʔΫϑϩʔͷ֤εςοϓ Pod ͱͯ͠ಈ࡞ϫʔΫϑϩʔͷ࣮ߦʹઐ೦͠ɺτϦΨʔఆظ࣮ߦͷػೳͨͳ͍• ͍উख Airflow, Digdag ΑΓ Luigi ʹ͍ۙ• Argo Events ͱ͍͏ผπʔϧͰ༷ʑͳτϦΨʔΛఏڙ
Argo Workflow ͷجຊػೳ
୯ҰίϯςφΛ࣮ߦ͢Δ࠷؆୯ͳϫʔΫϑϩʔapiVersion: argoproj.io/v1alpha1kind: Workflowmetadata:generateName: hello-world-spec:entrypoint: entrypoint # ࠷ॳʹ࣮ߦ͢ΔίϯςφςϯϓϨʔτΛࢦఆtemplates: # ̍ͭҎ্ͷίϯςφςϯϓϨʔτΛఆٛ- name: entrypointcontainer:image: alpine:latestcommand: ["echo", "hello world"]※Ҏ߱ͷྫ spec ԼͷΈهࡌ
ϫʔΫϑϩʔʹύϥϝʔλΛ͢entrypoint: entrypointarguments:# ϫʔΫϑϩʔ࣮ߦ࣌ʹ argo submit -p message=hello ͷΑ͏ʹͤΔparameters:- name: messagetemplates:- name: entrypointcontainer:image: alpine:latestcommand: ["echo", "{{workflow.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
εςοϓʹύϥϝʔλΛ͢entrypoint: entrypointtemplates:- name: entrypointinputs:# ޙड़ͷ steps, dag ͳͲ͔Β͢parameters:- name: messagevalue: hellocontainer:image: alpine:latestcommand: ["echo", "{{inputs.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
steps: εςοϓͷྻɾฒྻ࣮ߦtemplates:- name: entrypointsteps:- - name: hello1template: echo # ίϯςφςϯϓϨʔτΛࢦఆarguments: {parameters: [{name: "message", value: "hello1"}]}- - name: hello2a # hello1 ͷ࣍ʹ hello2a, hello2b Λ࣮ߦtemplate: echoarguments: {parameters: [{name: "message", value: "hello2a"}]}- name: hello2b # hello2a, hello2b ฒྻ࣮ߦtemplate: echoarguments: {parameters: [{name: "message", value: "hello2b"}]}- name: echoinputs: {parameters: [{name: "message"}]}container:image: alpine:latestcommand: ["echo", "{{inputs.parameters.message}}"]
dag: ͰλεΫͷDAG࣮ߦtemplates:- name: entrypointdag:tasks:- name: Atemplate: echoarguments: {parameters: [{name: message, value: A}]}- name: Bdependencies: [A] # ґଘλεΫΛࢦఆtemplate: echoarguments: {parameters: [{name: message, value: B}]}- name: Cdependencies: [A]template: echoarguments: {parameters: [{name: message, value: C}]}- name: Ddependencies: [B, C] # ґଘλεΫΛෳࢦఆtemplate: echoarguments: {parameters: [{name: message, value: D}]}
artifact: εςοϓؒͰϑΝΠϧΛड͚͠templates:- name: entrypointsteps:- - {name: generate-artifact, template: generate-artifact}- - {name: consume-artifact, template: consume-artifact}- name: generate-artifactcontainer:image: alpine:latestcommand: ["sh", "-c", "echo hello > /tmp/output.txt"]outputs:artifacts:- {name: "result", path: "/tmp/output.txt"}- name: consume-artifactcontainer:image: alpine:latestcommand: ["sh", "-c", "cat /tmp/input.txt"]inputs:artifacts:- {name: "result", path: "/tmp/input.txt"}
when: ϫʔΫϑϩʔͷذtemplates:- name: entrypointsteps:- - name: flip-cointemplate: flip-coin# when Ͱશεςοϓͷ݁ՌΛͱʹذ- - when: "{{steps.flip-coin.outputs.result}} == heads"name: heads- when: "{{steps.flip-coin.outputs.result}} == tails"name: tails- name: flip-coinscript:image: python:latestcommand: [python]source: "import random; print(random.choice(['heads', 'tails']))"
withItems, withParams: εςοϓͷ܁Γฦ͠templates:- name: entrypointsteps:# withItems Ͱͨ͠ item ͷ͚ͩεςοϓΛฒྻ࣮ߦ- - withItems: ["hello world", "goodbye world", "ok world"]name: eachtemplate: echoarguments: {parameters: [{name: "message", value: "{{item}}"}]}# withParams ʹ ["hello world", "goodbye world"] ͷΑ͏ͳ JSON Λ͢͜ͱՄೳ- - withParams: "{{workflow.parameters.params}}"name: eachtemplate: echoarguments: {parameters: [{name: "message", value: "{{item}}"}]}
exitHandler: ϫʔΫϑϩʔͷޭɾࣦഊ࣌ͷϋϯυϦϯάonExit: exit-handlertemplates:- name: entrypointcontainer:image: alpine:latestcommand: ["exit", "1"]- name: exit-handlersteps:# workflow.status Λͱʹذ- - when: "{{workflow.status}} == Succeeded"template: echoarguments: {parameters: [{name: "message", value: "SUCCESS"}]}- when: "{{workflow.status}} != Succeeded"template: echoarguments: {parameters: [{name: "message", value: "ERROR!"}]}
ϦτϥΠɾλΠϜΞτͳͲtemplates:- name: entrypoint# ϦτϥΠճͳͲΛઃఆretryStrategy:limit: 2# λΠϜΞτΛઃఆ (Pod ͷه๏ͱಉ͡)activeDeadlineSeconds: 28800# ϊʔυͷࢦఆ (Pod ͷه๏ͱಉ͡)nodeSelector:cloud.google.com/gke-nodepool: highmem-pool# Ϧιʔε੍ݶ (Pod ͷه๏ͱಉ͡)container:resources:limits:memory: "32Gi"
ͦͷଞ• ฒྻ࣮ߦ࣌ͷฒྻ্ݶΛઃఆ• ϘϦϡʔϜʹΑΔσʔλͷड͚͠• ิॿίϯςφͷར༻ (Sidecar, Daemon, ...)• ֎෦ετϨʔδͷར༻• etc.ৄ͘͠ެࣜͷ example Λࢀরhttps://github.com/argoproj/argo/tree/master/examples
ϦϒηϯεͰͷ Argo Workflow ӡ༻
MLγεςϜͷ࣮ߦج൫GCP্ͰGKE Λத৺ͱ͢Δػցֶशج൫Λߏங• ෳͷMLγεςϜΛ୯ҰͷGKEΫϥελʹू• όονॲཧ͚ͩͰͳ͘WebΞϓϦಉ͡ΫϥελͰӡ༻Argo Workflow ͷར༻• ίϯςφίϯϙʔωϯτGCBͰϏϧυ͠GCR ʹొ• ϫʔΫϑϩʔఆٛଞͷ manifest ͱಉ͡ϨϙδτϦͰཧ• ఆظ࣮ߦ͢ΔϫʔΫϑϩʔ CronJob Ͱ argo submit
GCP, GKE, Argo Workflow ͷߏਤ
ӡ༻ࢦόονॲཧͱΓ͋͑ͣ Workflow ͱͯ͠ఆٛ• खݩͰ docker run ͚ͩͰࢼݧ࣮ߦͰ͖ΔΑ͏γεςϜΛ࣮• ·ͣ୯Ұεςοϓͷ Workflow ͱͯ͠ӡ༻ʹࡌͤΔӡ༻͠ͳ͕ΒίϯϙʔωϯτԽΛਐΊͯຊମΛεϦϜԽ• DBIO௨ͳͲͷڞ௨ॲཧΛஈ֊తʹΓग़͍ͯ͘͠• ฒྻԽɾϦτϥΠͳͲͳΔ͘ Workflow ଆͷػೳͰ࣮ݱҎԼɺࣄྫͱӡ༻ϊϋΛհ
CASE: ίϯϙʔωϯτͷΈ߹Θͤ• ϨίϝϯυΤϯδϯಛʹίϯϙʔωϯτԽ͕ਐΜͰ͍Δ• SQLઃఆϑΝΠϧͻͱͭͷίϯςφʹ·ͱΊͯ࠷ॳʹల։templates:- name: entrypointsteps:- - name: load-config- - name: sqlkitwithItems:- sqlfile: /workspace/sql/ratings.sql- sqlfile: /workspace/sql/contents.sql- - name: nlpkit- - name: recommender
CASE: ϝΠϯͷόονॲཧͷεϦϜԽ• ਪఆɾ༧ଌϞσϧDBIO௨ͳͲΛΓग़ͯ͠ϝΠϯͷόονॲཧΛεϦϜԽ• MLΤϯδχΞɾMLج൫ΤϯδχΞͰͷ୲Λ͍ͯ͘͢͠͠ΔonExit: exit-handlertemplates:- name: entrypointsteps:- - name: train-predict # MLΤϯδχΞ͕࣮ (ग़ྗCSV)- - name: import-to-db # MLج൫ΤϯδχΞ͕࣮- name: exit-handler # MLج൫ΤϯδχΞ͕࣮steps:- - when: "{{workflow.status}} != Succeeded"name: notify-error
CASE: MLϞσϧͷ؆қతͳCD• ਪఆɾ༧ଌϞσϧͷ݁ՌϏϡʔϫ Deployment ͱͯ͠ӡ༻• ਪఆॲཧྃ࣌ʹ kubectl set env ͰϏϡʔϫʹ৽͍͠ϞσϧΛಡΈࠐ·ͤΔ• Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳtemplates:- name: entrypointsteps:- - name: train-predict- - name: import-to-db- - name: update-viewer- name: update-viewercontainer:image: kubectlcommand: ["sh", "-c"]args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
CASE: ॏ͍ɾෆ҆ఆͳMLॲཧΛѻ͏• ਪఆɾ༧ଌϞσϧͳͲͰ Stan Λଟ༻• ϝϞϦɾCPUΛେྔʹফඅ͢Δ߹ઐ༻ͷϊʔυͰ࣮ߦ• αϯϓϦϯά͕֬తʹࣦഊ͢ΔͷͰϦτϥΠɾλΠϜΞτ͕ඞཁ- name: train-predictactiveDeadlineSeconds: 28800 # 8hretryStrategy:limit: 2nodeSelector:cloud.google.com/gke-nodepool: highmem-poolcontainer:resources:limits:memory: "32Gi"
CASE: Ϟσϧਪఆͷಈతͳฒྻ࣮ߦ• όϯσΟοτπʔϧͰ࣮ࢪதͷςετ͝ͱʹਪఆॲཧ͕ඞཁ• ֤ςετͷਪఆॲཧΛಈతʹฒྻ࣮ߦtemplates:- name: entrypointsteps:# ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ- - name: list-experiments # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ# લͷεςοϓͷग़ྗ͔ΒύϥϝʔλͷϦετΛಡΈࠐΈ- - withParams: "{{steps.list-experiments.outputs.parameters.experiments}}"# Ϧετͷཁૉ͝ͱʹޙଓͷεςοϓΛ࣮ߦname: calc-weightsarguments:parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
ӡ༻ TIPSArgo Workflow ͷ Web UI ͷΞΫηε• σϑΥϧτͰ kubectl port-forward ͰΞΫηε͢Δඞཁ͕͋Δ• ΠϯλʔωοτΞΫηεΛՄೳʹ͢Δʹ Ingress ͰϩʔυόϥϯαΛཱͯΔ• GCP ͷ Identity-Aware Proxy Λ͏ͱϩʔυόϥϯαଆͰೝূΛ͔͚ΒΕΔݹ͍ϫʔΫϑϩʔͷΫϦʔϯΞοϓ• ࣮ߦࡁΈͷ Workflow ͱͦͷཧ͢Δ Pod Successful ͷ··Γଓ͚Δ• ఆظతʹݹ͍ Workflow Λআ͢Δ CronJob Λཱ͍ͯͯΔ• argo delete --older Φϓγϣϯ͕ศར
Pros, Cons, ·ͱΊ
Argo Workflow - ProsଞͷϫʔΫϑϩʔΤϯδϯͱൺϩοΫΠϯ͞Εʹ͍͘• ίϯςφԽ͞Ε͍ͯΕԿͰಈ͔ͤΔ• ࠓޙଞͷϫʔΫϑϩʔΤϯδϯ͕ग़͖ͯͯΓ͍͑͢όονॲཧͱWebΞϓϦΛಉ͡ΫϥελͰཧͰ͖Δ• σϓϩΠɾϩΪϯάɾϞχλϦϯάɾΤϥʔϨϙʔτͳͲΛҰݩԽ• ΦʔτεέʔϦϯάͳͲͱΈ߹ΘͤͯϦιʔεར༻ΛޮԽ
Argo Workflow - ConsଞͷϫʔΫϑϩʔΤϯδϯ΄ͲϓϩάϥϚϒϧͰͳ͍• Airflow, Luigi ͷΑ͏ʹ Python DSL ͕ॻ͚ͨΓ͠ͳ͍• ֤ΫϥυαʔϏεઐ༻ͷΦϖϨʔλ༻ҙ͞Ε͍ͯͳ͍࡞͞ΕͨϫʔΫϑϩʔΛଈ࣮࣌ߦ͢ΔҎ֎ͷػೳͨͳ͍• ఆظ࣮ߦʹ CronJob ͳͲΛ͏ඞཁ͕͋Δ• Web UI ϞχλϦϯάͷΈͰϦτϥΠͳͲͷૢ࡞Ͱ͖ͳ͍• ϫʔΫϑϩʔࣗମͷςϯϓϨʔτԽɾ࠶ར༻͕͠ʹ͍͘• WorkflowTemplate ͕ఏҊ͞Ε͍ͯΔͷͰظ
·ͱΊͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ• ෳͷαʔϏεͰMLγεςϜΛར༻• ଟ͘ͷεςοϓ͔ΒͳΔόονॲཧ͕ෳଘࡏ• ։ൃɾӡ༻ΛޮԽ͢ΔͨΊίϯϙʔωϯτΛׂͯ͠ίϯςφԽArgo Workflow ΛͲ͏͍ͬͯΔ͔• ίϯςφίϯϙʔωϯτΛΈ߹ΘͤͯϫʔΫϑϩʔΛߏங• MLγεςϜͷ։ൃɾӡ༻্ͷ߹ʹ߹Θ֤ͤͯछػೳΛ׆༻
livesense
yoshiitaka
minorun365
nikkei_engineer_recruiting
ufried
takanorig
stefafafan
s2terminal
sat
yamamuteki
yukimatsuyoshi
smt7174
cmusudakeisuke
dougneiner
mojombo
geeforr
mza
keithpitt
holman
marktimemedia
akmur
jcasabona
rmw
trishagee
cassininazir