Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Introduction of AnnoOps for mass production wit...

Yachi
April 30, 2020

Introduction of AnnoOps for mass production with multi-task and large-volume / annops_english

Yachi

April 30, 2020
Tweet

More Decks by Yachi

Other Decks in Technology

Transcript

  1. Introduction of AnnoOps for mass production with multi-task and large-volume

    Toyota Research Institute Advanced Development (TRI-AD) Yusuke Yachide 2020.04.30
  2. Introduction • Yachi (Yusuke Yachide, Ph.D ◦ Software Platform/ MLTools

    ▪ MLOps development ▪ AnnoOps (Annotation platform & Service) ▪ Arene development @yachide yachide-yusuke-23a2 7035/ Arene
  3. ?

  4. Our goal is to make the world’s safest car. James

    Kuffner (CEO) The future of car is software. Nikos Michalakis (VP of software platform)
  5. What are annotation requirements? Volume: 1,000 ~ XX0,000 Accuracy: Several

    percent Data: Image, video, LiDAR etc Task: 20prj (Point, box, seg etc) Delivery data: 1, 2 month ~ a half year
  6. A review on annotation process ① Rule making… Defining annotation

    rule for annotators ② Project creation… Creating a project for annotation ④ Inspection… Inspecting annotation data ③ Annotation… Annotating data ⑤ Delivery… Releasing annotation data
  7. Issues in vendor agnostic approach (Today’s scope) ① Rule making…

    Defining annotation rule for annotators ② Project creation… Creating a project for annotation ④ Inspection… Inspecting annotation data ③ Annotation… Annotating data ⑤ Delivery… Releasing annotation data 3. Easy to deliver and to share annotation data… Unifying annotation format 1. Standardize annotation quality Making standard rule by analyzing vendor’s feature 2. Vendor agnostic platform... Handling different annotation format
  8. 0. Premise : two sides of the same coin for

    rule and inspection Rule Inspection Bad rule -> Defective product Defective product -> reflecting a rule
  9. 0. Premise : Know annotation vendors Vendor A : Good

    productivity / Excellent accuracy Vendor A : Excellent productivity / Good accuracy NG rate NG rate #annotated data #annotated data Date Date Start annotation with a new rule Start annotation with a new rule Both vendors has more than 100 annotator, then both of standardizing a rule and respecting unique case study is important
  10. 1. Standardize a rule → Use of flowchart Ministry of

    Land, Infrastructure and Transport Example: Traffic sign annotation Language agnostic standard rule • Impossible to unify subjectivity → Clarify subjectivity part and not • ML eng has thought of annotation → Need to pull out procedure from ML eng Road surface? Statistic object? Shape? Rectangle? Triangle? Circle? Yes Yes Yes No annotation Annotation
  11. 1. Case study: dig in vendor response Vendor A :

    Good productivity, Low NG rate → Many questions in the background! #Question #annotated data Date Normally, put NG case in the case study but for Vendor A, put Q/A in it. NG rate #annotated data Date
  12. 1. Effective of rule making efforts #Question #annotated data Date

    Vendor A : Improve productivityベンダーA:生産性が向上 Venor B : Decrease NG rate NG rate #annotated data Date NG 30% decrease Productivity 10% increase
  13. 2. Vendor agnostic annotation platform Unlabeled data format converter Project

    generation & submission Annotation Downloader Labeled data format converter
  14. 2. Vendor agnostic annotation platform Unlabeled data format converter Project

    generation & submission Annotation Downloader Labeled data format converter Tool #2 Tool #3 • Each vendor has own tool (Multiple in/out format) • Different annotation unit • (A project of multiple annotation task at a time is allowed or not) Each tool is different
  15. Example : Tool difference Vendor B : need to make

    different project in a pipe. (Bbox project → Segmentation) Vendor A : simultaneous BBox and segmentation annotation
  16. 2. Vendor agnostic annotation platform Unlabeled data format converter Project

    generation & submission Annotation Downloader Labeled data format converter Simultaneous annotation Nonsimultaneous with multiple annotation tasks (Process multiple annotation projects in series) task task Tool in Vendor A Tool Vendor B Control
  17. 3. Easy delivery and sharing dataset Unlabeled data format converter

    Project generation & submission Annotation Downloader Labeled data format converter MLOps Dataset DB Easy delivery with unified annotation format Standardize dataset loader
  18. Summary At TRI-AD, for multi-task and large volume annotation requirements

    like ➔ Volume: 1,000 ~ XX0,000 ➔ Accuracy: Several percent ➔ Data: Image, video, LiDAR etc ➔ Task: 20prj (Point, box, seg etc) ➔ Delivery data: 1, 2 month ~ a half year Build vendor agnostic annotation platform to use multiple vendors easily 7~8 people are handling above massible annotation project