Slide 1

Slide 1 text

Sergey Senozhatsky, Fei Shao, Yunke Cao, Tomasz Figa, Ricardo Ribalda Kernel CAM Rethinking the kernel camera framework Paris, 1st June 2022

Slide 2

Slide 2 text

Proprietary + Confidential Once upon a time

Slide 3

Slide 3 text

Operating system (for driver developers) Program My framework Driver Hardware open(/dev/my_device) read(fd) ioctl(fd,IOC_,... ) my_init() my_alloc() my_read_from() inb() outb()

Slide 4

Slide 4 text

Operating system (for driver developers) Joystick interface xpad gamecon

Slide 5

Slide 5 text

Cameras are special

Slide 6

Slide 6 text

Great diversity of output ● There are more than 200 video formats ○ grep -c "^#define V4L2_PIX_FMT" include/uapi/linux/videodev2.h -> 202! Userspace Kernel Hardware Userspace Kernel Hardware Userspace Kernel Hardware

Slide 7

Slide 7 text

Multiple ways to do the same thing ● Do you want a 640x480 image from a 24Mpix (6000*400)?: ○ Cropping ○ Binning ○ Interpolate ● Image is too dark? ○ Exposure time ○ Analog Gain ○ Digital Gain

Slide 8

Slide 8 text

Do it fast or don’t do it ● Big data rates ○ 1080p RGB32@60fps -> 474 MiB/sec! ● Low latency ● Most of the time handled by other hardware ○ GPU ○ TPU ○ CODEC

Slide 9

Slide 9 text

Cameras drive the consumer market ● Work from home/anywhere ● E-learning ● Deciding factor for purchase

Slide 10

Slide 10 text

Cameras in Linux today

Slide 11

Slide 11 text

Video4Linux 2 (non media controller) Program Video Buffer 2 Driver Hardware Control Framework Video Helpers Media Subsystem libv4l2 ● Output can be converted via software to 4 standard formats. ● The driver makes most of the decisions for us. ● Hardware produces decent images by default.

Slide 12

Slide 12 text

Video4Linux 2 (media controller) Program Video Buffer 2 Driver Hardware Control Framework Video Helpers Media Subsystem Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers

Slide 13

Slide 13 text

Video4Linux 2 (post media controller) Program Video Buffer 2 Driver Hardware Control Framework Video Helpers Media Subsystem Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers Video Buffer 2 Driver Hardware Control Framework Video Helpers ● Software stack needs to know about the internals of the hardware. ● All the decisions are taken by the software stack. ● A lot of Image tuning before we can use the images.

Slide 14

Slide 14 text

Libcamera

Slide 15

Slide 15 text

Libcamera

Slide 16

Slide 16 text

Secret Sauce

Slide 17

Slide 17 text

Secret Sauce

Slide 18

Slide 18 text

Secret Sauce

Slide 19

Slide 19 text

Secret Sauce

Slide 20

Slide 20 text

Recap Aware Userspace Kernel Hardware Agnostic Userspace Kernel Hardware V4L2 Media Controller

Slide 21

Slide 21 text

Proprietary + Confidential Kernel CAM

Slide 22

Slide 22 text

What is CAM? Userspace Kernel Hardware ● New kernel subsystem. ● No media abstractions. ● Fluid collaboration between userspace and other subsystems. ● Two components: ○ Entities ○ Operations

Slide 23

Slide 23 text

KCAM components: Entities Root MIPI Codec Sensor ISP DMA VCM ● Organized in a tree, based on the hardware architecture. ● Single register-set. ● Can throw events.

Slide 24

Slide 24 text

KCAM components: Entities Root MIPI Codec Sensor ISP DMA VCM ● Organized in a tree, based on the hardware architecture. ● Single register-set. ● Can throw events.

Slide 25

Slide 25 text

KCAM components: Operations ● Read and/or write to an entity ○ regmap ○ parameter buffers ● Can depend on: ○ Events ○ Other operation ○ Fence ● Can create a fence http://www.paella10.com/calculadora/paellas/paella-valenciana/

Slide 26

Slide 26 text

KCAM components: Operations Operation List Configure Sensor Trigger image Init ISP Start DMA ioctl ready Circular buffer fence_out fence_in signal

Slide 27

Slide 27 text

Show me the code!! #define CAM_MAX_DEPENDENCIES 8 struct cam_operation_add { __u32 id; __u32 fence_out; __u32 flags; /* * Pre-execution dependencies list and dependency execution mode */ __u32 mode; struct cam_dependency deps[CAM_MAX_DEPENDENCIES]; /* * Execution context specific data (if any) */ __u64 delay_ns; __u64 rd_wr_list; __u32 entity; } __attribute__((packed)); enum cam_dependency_type { CAM_DEPENDENCY_NONE, CAM_DEPENDENCY_OP, CAM_DEPENDENCY_EVENT, CAM_DEPENDENCY_FENCE_IN, }; struct cam_dependency { __u32 type; __u32 id; } __attribute__((packed)); enum cam_dependency_mode { CAM_DEPENDENCY_WEAK_ORDER, CAM_DEPENDENCY_STRICT_ORDER, };

Slide 28

Slide 28 text

Testing (lots of) ● Software testing: ○ kunit ○ libkc ○ vcam ○ error injection ● Hardware testing ○ ChromeOS test infra ○ human test

Slide 29

Slide 29 text

Comparison Userspace Kernel Hardware libcamera ● Limitless functionality ● Fast upstream ● Small drivers ● Operations vs Streams Kernel CAM Userspace Kernel Hardware libcamera Media Controller

Slide 30

Slide 30 text

Governance ● There must be an open source stack before a driver is merged. ○ Similar to DRM model. ● Compliance-test with minimum requirements.

Slide 31

Slide 31 text

Proprietary + Confidential Future

Slide 32

Slide 32 text

Benchmark the stack Camera Service Kernel Userspace Hardware Clients Android Camera Framework Sors Svers SoC ISP Drivers h/EEPROM Sensor/Actuator/Flash/EEP ROM SoC VMs So SoC ISP Chrome Video Capture Device VMs Generic V4L2 Frameworks Sensor/Actuator/Fl ash/ EEPROM Drivers Sensor/Actuator/Fl ash/ EEPROM Drivers Sensor/Actuator/Flash/ EEPROM Drivers Camera Service Clients Android Camera Framework Sors Svers CAM drivers h/EEPROM Sensor/Actuator/Flash/EEP ROM SoC VMs So SoC ISP Chrome Video Capture Device VMs CAM Sensor/Actuator/Fl ash/ EEPROM Drivers Sensor/Actuator/Fl ash/ EEPROM Drivers Sensor/Actuator/Flash/ EEPROM Drivers Libcamera - cam Secret Sauce Secret Sauce

Slide 33

Slide 33 text

Land upstream

Slide 34

Slide 34 text

Contributing

Slide 35

Slide 35 text

How to contribute ● Show me the code! ○ https://chromium.googlesource.com/chromiumos/third_party/kernel/+/ref s/heads/kcam ● Mailing List ○ [email protected] ● Bugs: ○ WIP ● Join the team ○ https://crosjobs.page.link/event

Slide 36

Slide 36 text

Thank you! [email protected] https://crosjobs.page.link/event

Slide 37

Slide 37 text

Thank you! [email protected] https://crosjobs.page.link/event 37

Slide 38

Slide 38 text

Thank you! [email protected] https://crosjobs.page.link/event 38

Slide 39

Slide 39 text

Thank you! [email protected] https://crosjobs.page.link/event 39

Slide 40

Slide 40 text

Thank you! [email protected] https://crosjobs.page.link/event 40

Slide 41

Slide 41 text

Thank you! [email protected] https://crosjobs.page.link/event 41

Slide 42

Slide 42 text

Thank you! [email protected] https://crosjobs.page.link/event 42

Slide 43

Slide 43 text

Thank you! [email protected] https://crosjobs.page.link/event 43

Slide 44

Slide 44 text

Thank you! [email protected] https://crosjobs.page.link/event 44

Slide 45

Slide 45 text

Thank you! [email protected] https://crosjobs.page.link/event 45

Slide 46

Slide 46 text

Thank you! [email protected] https://crosjobs.page.link/event 46

Slide 47

Slide 47 text

Thank you! [email protected] https://crosjobs.page.link/event 47

Slide 48

Slide 48 text

Thank you! [email protected] https://crosjobs.page.link/event 48

Slide 49

Slide 49 text

Thank you! [email protected] https://crosjobs.page.link/event 49

Slide 50

Slide 50 text

Thank you! [email protected] https://crosjobs.page.link/event 50

Slide 51

Slide 51 text

Thank you! [email protected] https://crosjobs.page.link/event 51

Slide 52

Slide 52 text

Thank you! [email protected] https://crosjobs.page.link/event 46

Slide 53

Slide 53 text

Thank you! [email protected] https://crosjobs.page.link/event 53

Slide 54

Slide 54 text

Thank you! [email protected] https://crosjobs.page.link/event 54

Slide 55

Slide 55 text

Thank you! [email protected] https://crosjobs.page.link/event 55

Slide 56

Slide 56 text

Thank you! [email protected] https://crosjobs.page.link/event 56

Slide 57

Slide 57 text

Thank you! [email protected] https://crosjobs.page.link/event 57

Slide 58

Slide 58 text

Thank you! [email protected] https://crosjobs.page.link/event 58

Slide 59

Slide 59 text

Thank you! [email protected] https://crosjobs.page.link/event 0xffffffff