Upgrade to Pro — share decks privately, control downloads, hide ads and more …

"OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments" (PerCom2024 - Presentation)

"OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments" (PerCom2024 - Presentation)

This is a slide for the presentation at PerCom 2024 (March 13).

== Citation ==
Naoya Yoshimura, Jaime Morales, Takuya Maekawa, Takahiro Hara, “OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments”. Proceedings of IEEE International Conference on Pervasive Computing and Communications (2024).

== Links ==
PerCom2024: https://www.percom.org/full-program/
OpenPack HP: https://open-pack.github.io/

Yoshimura Naoya

March 17, 2024

More Decks by Yoshimura Naoya

Other Decks in Research


  1. N.Yoshimura, J. Morales, T. Maekawa, T.Hara (Osaka University, Japan) [PerCom2024,

    March 13] OpenPack: A Large-scale Dataset for Recognizing Packaging Works in IoT-enabled Logistic Environments
  2. OpenPack Dataset Background CPS / Digital Twin Manager Factory ML

    ML’ Worker’s Activity Data IoT Device Sensor Readings In Smart Factories • Many sensors and IoT devices are installed. • Some works mainly depend on human workers. Supportive techniques to track worker activities are required. ⇒ Work Activity Recognition 2
  3. OpenPack Dataset Challenges of Work Activity Recognition Dataset • Most

    of the public dataset focuses on the ADL (activity of daily living). • Public datasets in industrial domains containing complex activities is limited. Lack of datasets for industrial domains
 • Datasets for manual task provide only vision-related modalities. Limited Modality
 Lack of Io Data & Metadata
 • A lot of information related to the work activity is available in the system in the factory. ◦ e.g.) Order Management System → Items to be packed, • Existing activity recognition datasets do not provide these data. 3
  4. OpenPack Dataset OpenPack Dataset • New large-scale multimodal activity recognition

    dataset in industrial domain. • Designed for developing activity recognition models with IoT-enabled devices. 0100
 elocate Item Label 
 Assemble Box
 Insert Items
 Close Box
 Attach Box Label
 can Label
 Attach hipping Label 
 Put on Back able
 Fill out Order
 Work Operation & Action
 emove Item Label
 Write Check mark
 4 https://youtu.be/RiZ7kVpIHwU?si=DipjAt2s0Kp8KN6q
  5. OpenPack Dataset OpenPack Dataset Recordings 53.8H Subjects 16 Annotations Work

    Operation Action 20K 53K 5 Modality 9+IoT Metadata
  6. OpenPack Dataset Public Datasets for Manual Works
 Subjects 16* 14

    Work Periods 2048 (~100 /person) ~30 trial/person Recording Length 53.5h 12.5h Activity Class 10 / 32* 8* / 19 Modality Depth + Keypoints +Acc +Gyro +Ori +LiDAR +EDA + BVP +Tmp Keypoints +Acc +Gyro Metadata / Subject Available Available ー OpenPack LARa [2020] 12 5 times / person 25h 4 / 17 Acc+Gyro+Mag +Mic+Loc +Object+Ambient - Opportunity [2010] ** work operation / action 9 (No Repetition) 10h 18 Acc+Gyro +Mag+Ori +HR+Temp Available PAMAP [2012] Metadata / System Available No No No IoT Data Available No (Object Sensor) No 6 Annotation Labels 20,161 / 53,286** N/A 2640 / 7153 * N/A *Not work related classes. * 12 subjects have work experience. *Locomotion/Hand interaction occurrence
  7. OpenPack Dataset Data Collection Scenarios • 5 sessions/subjects, 1 session

    = 20 periods • The difficulties in packaging work recognition depend on the various factors. ⇒ Four scenarios are prepared to incrementally solve challenges. # of Items 54 Work Procedure Follow the work instruction as much as possible Scenario 1 Irregulars No Alarm Sound No 75 Able to alter the work procedure at subjects decision. Scenario 2 No No 75 Able to alter the work procedure at subjects decision. Scenario 3 Yes No 75 Able to alter the work procedure at subjects decision. Scenario 4 Yes Yes Ideal Wild 7
  8. OpenPack Dataset Environment & Sensors Front-view • Kinect (Depth +

    Keypoints) • LiDAR Top-view • RealSense (Depth) Vision Modality Wearable ensors • IMU x 4 (Acc + Gyro + Quaternion) • Empatica E4 x 2 (Acc + BVP + EDA + Temp) Io Devices • Handheld Scanner • Label Printer 1 1 2 3 2 3 8
  9. OpenPack Dataset Metadata Two types of metadata are available. ubject-related

 Order Sheet Order-related Metadata
 1 2 • Work Experience • Dominant Hand • Gender, Age Online order management system can provide information of a set of items to pack in the order. Item List Subject Data 9 Metadata can be used for recognition. ⇒ Useful for estimating the compositions of the work. (e.g. # of items = repetition of “remove labels” action.)
  10. OpenPack Dataset IoT Data Advantage • Strong connection between devices

    in use and worker’s activity. ⇒ High-confidence source IoT Data can
 Item Label
 Acc can Label Operation Disadvantage • Data is generated only when a worker operates the devices. ⇒ Sparse data Existing sensor fusion techniques assume normal sensor data such as acceleration. ⇒ Method to make the best use of this high-confidence but sparse data source can enhance work activity recognition performance. 10
  11. OpenPack Dataset Analysis: Factors Impacting Processing Time Length of Work

    Operation • The # actions performed in one period differs for each order. ◦ “Relocate item label” ⇒ 1+ action/item ◦ “Attach Box label” ⇒ 1 action/period # of Item, Box Size • The more items, the longer it takes. • The bigger the box, the longer it takes. OpenPack contains a huge variation of data! 11
  12. OpenPack Dataset Benchmark • Task: 10 work operation recognition at

    1 Hz • Input: Acceleration data from left wrist (window size = 60s) • Metrics: F1-measure (macro average) (Note: Protocols differs for each benchmark scenario.) Evaluation Protocol
 Baseline Models
 • CNN [F. Ordonez 2016] • U-Net [Y. Zhang 2019] … CNN-based segmentation model • DeepConvLSTM (DCL) [F. Ordonez 2016] … CNN + LSTM • DCL + Self-attention [S.P. Singh 2021] … Self-attention • ConformerHAR [ Y.-W. Kim 2022] … Transformer-based model • LOS-Net(-R) [N. Yoshimura 2022] … Designed for manual work recognition model 12
  13. OpenPack Dataset Benchmark 1: Data-rich Setting Activity recognition with the

    enough amount of training data. • Objective: Confirm the upper bounds of recognition performance. • Protocol: Leave-one-subject-out CV Results • LOS-Net(-R) achieved 0.83 in Scenario 1. • Scores for Scenario 4 (Rushed) was lower than others. ⇒ Models are not speed invariant. • Contents of order have impact on performance. ⇒ Models are not robust to changing orders. Development of model that is robust to the work speed and orders are necessary. 13
  14. OpenPack Dataset Benchmark 2: Data-scarce Setting Activity recognition with the

    limited amount of training data • Objective: More realistic than Scenario.1 • Training: Data from the 3rd session only (= 20 periods ~= 5h annotation.) • Test: Data from the remaining session. Results • Benchmark 1 ⇒ Benchmark 2 @ Scenario 1 ◦ LOS-Net(-R): 0.83 ⇒ 0.67 (- 0.16pt↓) ◦ Conformer : 0.78 ⇒ 0.53 (- 0.24pt↓) • CNN outperformed U-Net. • DCL outperformed DCL with Self-attention. 14 Technique to train the SOTA models with the limited training data is necessary.
  15. OpenPack Dataset Research Directions with OpenPack Dataset • Metadata-aided activity

    recognition • Fusion with high-confidence data source. • Speed-invariant activity recognition Others Topics • Transfer learning across subjects, across modalities. • Skill assessment using sensor data and metadata • Counting the number of necessary actions / packed items. • Estimating worker’s level of fatigue using sensor and physiological data. • Detecting mistakes and accidents in the work process. Metadata IoT 15 Metadata IoT 4 Scenarios 16 Subjects 9 Moldaity 16 Subjects Metadata Action Label EDA+BVP Action Label+ Related Freatures Related Freatures
  16. OpenPack Dataset Summary • The largest multimodal work activity dataset

    of packaging work. ◦ 53H+ Recording, 16 Subjects, IoT + Metadata ◦ 20K Work Operation Labels, 53K Action Labels • For more information ⇒ Visit our Website ! • Check sample data ⇒ GitHub • Try it out ⇒ Preprocessed Data at Zenodo. OpenPack Dataset
 Dataset is Available Now! 
 16 Preprocessed Data Label (Work Operation) IMU (Acc) Website