Go言語での排他制御/Mutual exclusion in Golang

Geeks Who Drink in Fukuoka -Go Go Golang Edition!-
https://nulab.connpass.com/event/57238/

Transcript

1. GoݴޠͰͷഉଞ੍ޚ
   monochromegane / Pepabo R&D Institute, GMO Pepabo, Inc.
   2017.06.28 Geeks Who Drink in Fukuoka -Go Go Golang Edition!-
   Mutual exclusion in Golang.
   

   View Slide

2. ϓϦϯγύϧΤϯδχΞ
   ࡾ୐ ༔հ / @monochromegane
   2
   http://blog.monochromegane.com
   Yusuke Miyake
   ϖύϘݚڀॴ ݚڀһ
   

   View Slide

3. 1. Motivation
   2. Mutual exclusion
   3. Resource locking
   3
   Contents
   

   View Slide

4. 1.
   Motivation
   

   View Slide

5. Gannoy
   Approximate nearest neighbor search server
   and dynamic index written in Golang.
   https://github.com/monochromegane/gannoy
   

   View Slide

6. Similar images search system using Gannoy
   6
   Features Similar items
   Gannoy
   [2048]float64
   query
   by http
   find similar features
   mapping
   similar features
   to items
   response
   Deep CNN
   index
   Features
   [2048]float64
   Deep CNN
   register
   by http
   

   View Slide

7. Many goroutines access one resource
   concurrently.
   

   View Slide

8. 2.
   Mutual exclusion
   

   View Slide

9. Many goroutines access one variable
   concurrently.
   

   View Slide

10. • sync.Mutex
    • A Mutex is a mutual exclusion lock.
    • sync.RWMutex
    • An RWMutex is a reader/writer mutual exclusion lock. The lock can be held
    by an arbitrary number of readers or a single writer.
    10
    Mutual exclusion in Golang
    

    View Slide

11. View Slide

12. 12
    type Free struct {
    mu sync.Mutex
    free []int
    }
    func (f *Free) push(id int) {
    f.mu.Lock()
    defer f.mu.Unlock()
    f.free = append(f.free, id)
    }
    func (f *Free) pop() (int, error) {
    f.mu.Lock()
    defer f.mu.Unlock()
    if len(f.free) == 0 {
    return -1, fmt.Errorf("empty")
    }
    x, newFree := f.free[len(f.free)-1], f.free[:len(f.free)-1]
    f.free = newFree
    return x, nil
    }
    Free nodes stack using sync.Mutex
    Exclusive lock
    Exclusive lock
    

    View Slide

13. 13
    type Maps struct {
    mu *sync.RWMutex
    keyToId map[int]int
    }
    func (m *Maps) add(id, key int) {
    m.mu.Lock()
    defer m.mu.Unlock()
    m.keyToId[key] = id
    }
    func (m Maps) getId(key int) (int, error) {
    m.mu.RLock()
    defer m.mu.RUnlock()
    if id, ok := m.keyToId[key]; !ok {
    return -1, fmt.Errorf("not found")
    } else {
    return id, nil
    }
    }
    Mapping id and key using sync.RWMutex
    Exclusive lock
    Shared lock
    

    View Slide

14. • Mutex and RWMutex must not be copied after first use.
    14
    Caution
    type Maps struct {
    mu sync.RWMutex
    keyToId map[int]int
    }
    func (m Maps) getId(key int) (int, error) {
    fmt.Printf("mu address in getId: %p\n", &m.mu)
    m.mu.RLock()
    defer m.mu.RUnlock()
    if id, ok := m.keyToId[key]; !ok {
    return -1, fmt.Errorf("not found")
    } else {
    return id, nil
    }
    }
    If type of member isn’t a pointer
    receiver isn’t a pointer
    $ go run -race main.go
    mu address in getId: 0xc420010110
    mu address in getId: 0xc4200781c0
    ==================
    WARNING: DATA RACE
    wg.Add(4)
    for i := 0; i < 2; i++ {
    go func(i int) {
    maps.add(i, i)
    wg.Done()
    }(i)
    go func(i int) {
    maps.getId(i)
    wg.Done()
    }(i)
    }
    wg.Wait()
    and
    

    View Slide

15. 3.
    Resource locking
    

    View Slide

16. Many goroutines access one file
    concurrently.
    

    View Slide

17. Data structure of Gannoy
    17
    Hyperplane
    Logical data structure Data structure on file system
    Bucket Bucket
    /PEF
    'FBUVSFT
    
    /PEF
    'FBUVSFT
    
    /PEF
    'FBUVSFT
    
    /PEF
    'FBUVSFT
    
    Node
    (Features)
    /PEF
    'FBUVSFT
    
    /PEF
    'FBUVSFT
    
    /PEF
    'FBUVSFT
    
    /PEF
    'FBUVSFT
    
    Node
    (Features)
    query
    features
    cosine similarity > 0
    <= 0
    B N
    H N
    free (1byte)
    number of descendants (4byte)
    key (4byte)
    parents (4byte * tree)
    children (4byte * K)
    or
    hyperplane / features (8byte* dim)
    …
    

    View Slide

18. • syscall.Flock
    • apply or remove an advisory lock on an open file.
    • syscall.FcntlFlock
    • file control. F_SETLK, F_SETLKW, and F_GETLK are used to acquire,
    release, and test for the existence of record locks.
    18
    Resource locking in Golang (using system call)
    locks associated with
    Flock file open file description
    Fcntl byte range process ID and i-node
    

    View Slide

19. 19
    // Flock
    func flock(fd uintptr) {
    syscall.Flock(int(fd), syscall.LOCK_EX) // or LOCK_SH
    defer syscall.Flock(int(fd), syscall.LOCK_UN)
    }
    // Fcntl + F_SETLKW
    func fcntl(fd uintptr, start, len int64) {
    syscall.FcntlFlock(fd, syscall.F_SETLKW, &syscall.Flock_t{
    Start: start,
    Len: len,
    Type: syscall.F_WRLCK, // or syscall.F_RDLCK
    Whence: io.SeekStart,
    })
    defer syscall.FcntlFlock(fd, syscall.F_SETLKW, &syscall.Flock_t{
    Start: start,
    Len: len,
    Type: syscall.F_UNLCK,
    Whence: io.SeekStart,
    })
    }
    Flock
    Shared or Exclusive lock
    Fcntl + F_SETLKW
    Shared or Exclusive lock
    byte range
    

    View Slide

20. 20
    file descriptor
    fd0
    fd1
    fd2
    Open file description
    0
    1
    2
    10
    20
    i-node
    1000
    1001
    1002
    2000
    3000
    file descriptor
    fd0
    fd1
    fd2
    fd10
    dup
    fork
    Process System System
    Goroutine
    Goroutine
    OFD
    0
    2
    ProcessID i-node
    100 1000
    101 1001
    Flock Fcntl + F_SETLKW
    Locks that associated with same
    one don’t block each other.
    

    View Slide

21. 21
    file1, _ := os.OpenFile("dummy.txt", os.O_RDWR, 0)
    file2, _ := os.OpenFile("dummy.txt", os.O_RDWR, 0)
    fd1 := file1.Fd()
    fd2 := file2.Fd()
    var wg sync.WaitGroup
    wg.Add(2)
    locker := gannoy.Flock{}
    go lockAndProcess("fd1", fd1, locker, &wg)
    go lockAndProcess("fd2", fd2, locker, &wg)
    wg.Wait()
    fd2 processed something (1sec).
    fd2 processed something (2sec).
    fd2 processed something (3sec).
    fd1 processed something (1sec).
    fd1 processed something (2sec).
    fd1 processed something (3sec).
    locker := gannoy.Fcntl{}
    fd2 processed something (1sec).
    fd1 processed something (1sec).
    fd2 processed something (2sec).
    fd1 processed something (2sec).
    fd2 processed something (3sec).
    fd1 processed something (3sec).
    No blocking
    Blocking
    Flock
    Fcntl
    +
    F_SETLKW
    

    View Slide

22. 22
    func lockAndProcess(name string, fd uintptr, locker gannoy.Locker, wg *sync.WaitGroup) {
    locker.WriteLock(fd, 0, 4)
    defer locker.UnLock(fd, 0, 4)
    t := time.NewTicker(1 * time.Second)
    count := 1
    loop:
    for {
    select {
    case <-t.C:
    fmt.Printf("%s processed something (%d sec).\n", name, count)
    count++
    if count > 3 {
    break loop
    }
    }
    }
    t.Stop()
    wg.Done()
    }
    

    View Slide

23. Open file description lock
    open file description locks are associated with the open
    file description on which they are acquired,
    http://man7.org/linux/man-pages/man2/fcntl.2.html
    ❝
    https://www.oreilly.co.jp/community/blog/2014/11/file-lock-and-new-ofd-lock.html
    

    View Slide

24. 24
    type Fcntl struct {
    }
    const F_OFD_SETLKW = 38
    func (f Fcntl) ReadLock(fd uintptr, start, len int64) error {
    return f.fcntl(syscall.F_RDLCK, fd, start, len)
    }
    func (f Fcntl) WriteLock(fd uintptr, start, len int64) error {
    return f.fcntl(syscall.F_WRLCK, fd, start, len)
    }
    func (f Fcntl) UnLock(fd uintptr, start, len int64) error {
    return f.fcntl(syscall.F_UNLCK, fd, start, len)
    }
    func (f Fcntl) fcntl(typ int16, fd uintptr, start, len int64) error {
    return syscall.FcntlFlock(fd, F_OFD_SETLKW, &syscall.Flock_t{
    Start: start,
    Len: len,
    Type: typ,
    Whence: io.SeekStart,
    })
    }
    Command for working with
    open file description locks
    Exclusive lock
    Shared lock
    

    View Slide

25. 25
    file1, _ := os.OpenFile("dummy.txt", os.O_RDWR, 0)
    file2, _ := os.OpenFile("dummy.txt", os.O_RDWR, 0)
    fd1 := file1.Fd()
    fd2 := file2.Fd()
    var wg sync.WaitGroup
    wg.Add(2)
    locker := gannoy.Flock{}
    go lockAndProcess("fd1", fd1, locker, &wg)
    go lockAndProcess("fd2", fd2, locker, &wg)
    wg.Wait()
    fd2 processed something (1sec).
    fd2 processed something (2sec).
    fd2 processed something (3sec).
    fd1 processed something (1sec).
    fd1 processed something (2sec).
    fd1 processed something (3sec).
    locker := gannoy.Fcntl{}
    Blocking!!
    Blocking
    Flock
    Fcntl
    +
    F_ODF_SETLKW
    fd2 processed something (1sec).
    fd2 processed something (2sec).
    fd2 processed something (3sec).
    fd1 processed something (1sec).
    fd1 processed something (2sec).
    fd1 processed something (3sec).
    

    View Slide

26. 1. non-POSIX (Only Linux)
    2. kernel >= 3.15.0 (CentOS7.2 is 3.10.0…)
    26
    Caution
    func newLocker() Locker {
    bytes, err := exec.Command("uname", "-sr").Output()
    if err != nil {
    return Flock{}
    }
    kernel := strings.Split(strings.TrimRight(string(bytes), "\n"), " ")
    if len(kernel) != 2 {
    return Flock{}
    }
    if kernel[0] == "Linux" && !semver.New(kernel[1]).LessThan(*semver.New("3.15.0")) {
    return Fcntl{}
    }
    return Flock{}
    }
    kernel >= 3.15.0
    

    View Slide

27. ݚڀһɺੵۃతʹืूதʂ
    http://rand.pepabo.com/
    

    View Slide