$30 off During Our Annual Pro Sale. View Details »

Create libcsv based ruby/csv compatible CSV library

Create libcsv based ruby/csv compatible CSV library

RubyKaigi 2018 - Create libcsv based ruby/csv compatible CSV library

秒速284km

May 31, 2018
Tweet

More Decks by 秒速284km

Other Decks in Programming

Transcript

  1. Create libcsv based
    ruby/csv compatible
    CSV library
    Asakusa.rb @284km

    View Slide

  2. http://rubykaigi.org/2018/parties
    Talked in the title:
    “Super fast CSV parser”
    - libcsv based
    - ruby/csv compatible

    View Slide

  3. # libcsv based
    - fast
    - standard

    View Slide

  4. # ruby/csv compatible
    - small && easy to read
    - Backward compatibility

    View Slide

  5. ## Motivation
    - CSV is often used
    - Sometimes I use a
    large CSV

    View Slide

  6. ## 284km/csv/tree/libcsv
    ```
    # 100,000 lines
    Comparison:
    (libcsv) quoted: 81022184 allocated
    (libcsv) unquoted: 82071384 allocated - 1.01x more
    unquoted: 110535101 allocated - 1.36x more
    quoted: 133536546 allocated - 1.65x more
    ```

    View Slide

  7. ## libcsv
    - columns↑ performance↓
    - rows↑ performance➘

    View Slide

  8. ## memory usage
    ```
    # 5 columns, 10,000 lines
    Comparison:
    (libcsv) quoted: 8090304 allocated
    (libcsv) unquoted: 9139504 allocated - 1.13x more
    unquoted: 11053221 allocated - 1.37x more
    quoted: 13354666 allocated - 1.65x more
    ```

    View Slide

  9. ## memory usage
    ```
    # 5 columns, 100,000 lines
    Comparison:
    (libcsv) quoted: 81022184 allocated
    (libcsv) unquoted: 82071384 allocated - 1.01x more
    unquoted: 110535101 allocated - 1.36x more
    quoted: 133536546 allocated - 1.65x more
    ```

    View Slide

  10. ## memory usage
    ```
    # 10 columns, 10,000 lines
    Comparison:
    unquoted: 13954361 allocated
    (libcsv) quoted: 14090304 allocated - 1.01x more
    (libcsv) unquoted: 15139504 allocated - 1.08x more
    quoted: 18555121 allocated - 1.33x more
    ```

    View Slide

  11. ## memory usage
    ```
    # 10 columns, 1,000,000 lines
    Comparison:
    unquoted: 1396650961 allocated
    (libcsv) quoted: 1411636904 allocated - 1.01x more
    (libcsv) unquoted: 1412686104 allocated - 1.01x
    more
    quoted: 1856651721 allocated - 1.33x more
    ```

    View Slide

  12. # What/How to do after this
    talk
    ## Continue development
    - libcsv multibyte support
    - column↑ performance→
    - ffi gem replace to fiddle

    View Slide

  13. RubyKaigi 2018
    Thank you

    View Slide