Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Revolution in Storage (RICON East 2013)

Revolution in Storage (RICON East 2013)

Presented by James Hughes at RICON East 2013

The trends of technology are rocking the storage industry. Fundamental changes in basic technology, combined with massive scale, new paradigms, and fundamental economics leads to predictions of a new storage programming paradigm. The growth of low cost/GB disk is continuing with technologies such as Shingled Magnetic Recording. Flash and RAM are continuing to scale with roadmaps, some argue, down to atom scale. These technologies do not come without a cost. It is time to reevaluate the interface that we use to all kinds of storage, RAM, Flash and Disk. The discussion starts with the unique economics of storage (as compared to processing and networking), discusses technology changes, posits a set of open questions and ends with predictions of fundamental shifts across the entire storage hierarchy.

About James

James Hughes is a Principal Technologist at Seagate Technology. Formerly with Huawei, and Sun Microsystems where he was a Sun Fellow, VP and the Solaris Chief Technologist. James is a recognized expert in the area of storage, networking, and information security. Before Sun, James worked at StorageTek, Network Systems, and Control Data Corp. He has over 40 years experience in OS, storage, networking, information security, and cryptography and is the holder of 30 patents with many more pending.

Basho Technologies

May 14, 2013
Tweet

More Decks by Basho Technologies

Other Decks in Technology

Transcript

  1. Revolution in Storage
    James Hughes

    View full-size slide

  2. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Agenda
    Economics
    Technology Shifts
    Open Questions
    Predictions
    2

    View full-size slide

  3. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Agenda
    Economics
    Technology Shifts
    Open Questions
    Predictions
    Prediction is very difficult, especially if it's about the future.
    Niels Bohr
    2

    View full-size slide

  4. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Demand for Storage
    The demand to store more data is not slowing down
    • Enabling new applications
    –Recording internet traffic
    –All CCTV surveillance for years
    • All human experience of 7B people is 1,000 EB
    Recording less valuable information “just in case”
    • The future value of information is not known
    All predictions that demand for computing or storage will
    be satisfied have all failed over the years
    3

    View full-size slide

  5. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Storage is a Price Elastic Market
    4
    http://en.wikipedia.org/wiki/Alfred_Marshall

    View full-size slide

  6. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Storage is a Price Elastic Market
    Price elasticity of demand
    • Alfred Marshall (1890)
    4
    http://en.wikipedia.org/wiki/Alfred_Marshall

    View full-size slide

  7. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Storage is a Price Elastic Market
    Price elasticity of demand
    • Alfred Marshall (1890)
    As the price of Storage approaches
    $0
    • Demands for storage will approach
    infinity
    4
    http://en.wikipedia.org/wiki/Alfred_Marshall

    View full-size slide

  8. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Storage is a Price Elastic Market
    Price elasticity of demand
    • Alfred Marshall (1890)
    As the price of Storage approaches
    $0
    • Demands for storage will approach
    infinity
    If the price of a Cisco router
    approaches $0
    • Demands for routers will not approach
    infinity
    4
    http://en.wikipedia.org/wiki/Alfred_Marshall

    View full-size slide

  9. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Jevons Paradox
    • Cloud Computing increases the efficiency of
    computing....
    Cloud Computing will increase this trend
    5
    http://en.wikipedia.org/wiki/Jevons_paradox

    View full-size slide

  10. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Jevons Paradox
    • Cloud Computing increases the efficiency of
    computing....
    Cloud Computing will increase this trend
    Improved technology doubles
    the amount of Information produced
    with a given amount of Storage
    Demand for Storage rises
    5
    http://en.wikipedia.org/wiki/Jevons_paradox

    View full-size slide

  11. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Storage Technology
    Flash Devices
    HDD Futures
    Shingled Disks
    Log Structure
    Distributed Hash Tables
    Metadata Servers (not)
    Object Storage
    6

    View full-size slide

  12. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Moore’s Law for Flash Scaling
    http://upload.wikimedia.org/wikipedia/commons/6/64/NAND_scaling_timeline.png
    7

    View full-size slide

  13. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    http://cseweb.ucsd.edu/users/swanson/papers/FAST2012BleakFlash.pdf
    Flash Drive Density Forecast
    8

    View full-size slide

  14. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    http://cseweb.ucsd.edu/users/swanson/papers/FAST2012BleakFlash.pdf
    2010 2013
    Flash Drive Density Forecast
    8

    View full-size slide

  15. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Disk Performance
    Factor of 10x performance in 30 years
    • Processors are 1,000,000x in 30 years
    http://www.cs.princeton.edu/courses/archive/spr05/cos598E/bib/grochowski.pdf
    9

    View full-size slide

  16. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    10
    Areal Density Growth
    0.1
    1
    10
    100
    1000
    10000
    100000
    1989
    1991
    1993
    1995
    1997
    1999
    2001
    2003
    2005
    2007
    2009
    2011
    2013
    2015
    2017
    2019
    year
    gigabit / in2
    Single particle
    superparamagnetic limit
    (estimated)
    Charap’s limit
    (broken)
    •  Late 1990s – super paramagnetic
    limit demonstrated through
    modeling
    •  Perpendicular expected to extend
    to 0.5-1 Tb/in2
    •  Additional innovations required
    at that point
    •  heat-assisted
    recording
    •  bit patterned
    media recording
    •  Areal Density CAGR 40%
    •  Transfer Rate CAGR 20%
    Perpendicular Writing & GMR
    HAMR
    HAMR
    +BPM
    •  Inductive Writing & Reading
    •  Inductive Writing/ MR reading
    •  Inductive Writing/ GMR reading
    29%
    100%
    40%

    View full-size slide

  17. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Shingled Disks
    Write head larger than read head
    • Turns Disk into a sequential media
    All updates to data and metadata are
    written sequentially to a
    continuous stream, called a log
    Disk API of sectors is no longer
    “natural”
    • One read may require several seeks
    http://www.ssrc.ucsc.edu/Papers/amer-ieeetm11.pdf
    11

    View full-size slide

  18. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Log Structured Storage
    How much is erased on a reposition?
    • Tape - the remainder of the tape
    • Singled disk - the remainder of the track group
    • Flash - the entire page
    All persistent Storage systems do/will implement log structure
    • e.g. “NoSQL Database of sectors”
    Does it make sense to layer a database on top of a
    database?
    • Could we use the log structure of the media to provide a more
    natural storage systems, not mimicking an antique paradigm?
    12

    View full-size slide

  19. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Leading to disaggregation of servers
    Single System Performance Trend
    http://web.eecs.umich.edu/~twenisch/papers/isca09-disaggregate.pdf
    13

    View full-size slide

  20. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Scaling Storage
    Distributed Hash Table
    • Key/Value Store RAM Memcached
    Flash Voldemort
    Disk Cassandra
    http://en.wikipedia.org/wiki/Distributed_hash_table
    14

    View full-size slide

  21. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Metadata Servers
    Required by traditional file systems (POSIX) to translate
    names to sectors
    • Hard to scale, heavy HA requirements, expensive
    Can we use a name as a key?
    • Place the data into a scaled key value store?
    • Eliminate costly metadata servers?
    15

    View full-size slide

  22. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Object Storage
    A storage system where objects (files) are read, written,
    replaced, but never changed.
    • e.g. Amazon S3
    Allows log structure with a minimum of garbage collection
    New tier of storage
    • Lowest cost for online storage (not tape)
    • Huge aggregate performance (High throughput, OK latency)
    16

    View full-size slide

  23. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Open Questions
    Should system-level tiering decisions be automated or
    better left as an economic decision?
    • Is the complexity worth it
    Can Hadoop clusters be general purpose?
    • Amdahl’s Law
    Is there a general paradigm for turning drives off?
    • given complexity and access time
    17

    View full-size slide

  24. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Predictions
    Tape ➜ Log Structured B-Tree
    • Can allow multiple people to be streaming to the same tape
    creating independent file systems?
    • To aggregate key/value pairs?
    Distributed File System ➜ Scaled Object Store
    • Lower cost, higher performance
    RAM ➜ Remotely accessed as Key/Value Store
    • Implemented in hardware
    18

    View full-size slide

  25. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Conclusion
    Storage devices are continuing to get denser
    • Flash will not take over capacity disks for quite a while
    Object Stores are replacing Distributed File Systems
    • Success of S3
    A Key/Value API for Storage
    • Reduces or eliminates the metadata server
    • More natural for the log structure of storage devices
    19

    View full-size slide

  26. Seagate Confidential: Subject to NDA
    No. 77103, effective Jan. 18, 2009,
    and all applicable supplements
    Conclusion
    Storage devices are continuing to get denser
    • Flash will not take over capacity disks for quite a while
    Object Stores are replacing Distributed File Systems
    • Success of S3
    A Key/Value API for Storage
    • Reduces or eliminates the metadata server
    • More natural for the log structure of storage devices
    19
    • Common API for disk, ram, flash, tape?

    View full-size slide