Create libcsv based ruby/csv compatible CSV library

RubyKaigi 2018 - Create libcsv based ruby/csv compatible CSV library

Transcript

1. Create libcsv based
   ruby/csv compatible
   CSV library
   Asakusa.rb @284km
   

   View Slide

2. http://rubykaigi.org/2018/parties
   Talked in the title:
   “Super fast CSV parser”
   - libcsv based
   - ruby/csv compatible
   

   View Slide

3. # libcsv based
   - fast
   - standard
   

   View Slide

4. # ruby/csv compatible
   - small && easy to read
   - Backward compatibility
   

   View Slide

5. ## Motivation
   - CSV is often used
   - Sometimes I use a
   large CSV
   

   View Slide

6. ## 284km/csv/tree/libcsv
   ```
   # 100,000 lines
   Comparison:
   (libcsv) quoted: 81022184 allocated
   (libcsv) unquoted: 82071384 allocated - 1.01x more
   unquoted: 110535101 allocated - 1.36x more
   quoted: 133536546 allocated - 1.65x more
   ```
   

   View Slide

7. ## libcsv
   - columns↑ performance↓
   - rows↑ performance➘
   

   View Slide

8. ## memory usage
   ```
   # 5 columns, 10,000 lines
   Comparison:
   (libcsv) quoted: 8090304 allocated
   (libcsv) unquoted: 9139504 allocated - 1.13x more
   unquoted: 11053221 allocated - 1.37x more
   quoted: 13354666 allocated - 1.65x more
   ```
   

   View Slide

9. ## memory usage
   ```
   # 5 columns, 100,000 lines
   Comparison:
   (libcsv) quoted: 81022184 allocated
   (libcsv) unquoted: 82071384 allocated - 1.01x more
   unquoted: 110535101 allocated - 1.36x more
   quoted: 133536546 allocated - 1.65x more
   ```
   

   View Slide

10. ## memory usage
    ```
    # 10 columns, 10,000 lines
    Comparison:
    unquoted: 13954361 allocated
    (libcsv) quoted: 14090304 allocated - 1.01x more
    (libcsv) unquoted: 15139504 allocated - 1.08x more
    quoted: 18555121 allocated - 1.33x more
    ```
    

    View Slide

11. ## memory usage
    ```
    # 10 columns, 1,000,000 lines
    Comparison:
    unquoted: 1396650961 allocated
    (libcsv) quoted: 1411636904 allocated - 1.01x more
    (libcsv) unquoted: 1412686104 allocated - 1.01x
    more
    quoted: 1856651721 allocated - 1.33x more
    ```
    

    View Slide

12. # What/How to do after this
    talk
    ## Continue development
    - libcsv multibyte support
    - column↑ performance→
    - ffi gem replace to fiddle
    

    View Slide

13. RubyKaigi 2018
    Thank you
    

    View Slide