Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SMARTIAN: Enhancing Smart Contract Fuzzing with Data-Flow Analyses

SMARTIAN: Enhancing Smart Contract Fuzzing with Data-Flow Analyses

LINE DEVDAY 2021

November 11, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. Who Am I? - I joined here in January of

    this year. - My research interests include software security, software engineering, and machine learning. - I am a security researcher at Security R&D team in LINE.
  2. Introduction - We protect smart contract which is the basic

    component of dApp. - dApp is decentralized application running on blockchain. Making these Services Safe Smart Contract - A smart contract is a program stored inside a blockchain. - How does a smart contract work? Various Blockchain Business in LINE - BITFRONT, LINE BITMAX, LINK - LINE Blockchain Wallet , NFT Market β
  3. Crowdfunding - Creator - Person who proposes the project to

    be funded - Investor - Person who supports the project - Crowdfunding platform - Moderating organization between creators and investors. Three types of actors Crowdfunding - Crowdfunding is funding a project by raising small amounts of money from a large number of people.
  4. Crowdfunding Example - The platform mediates between the investors and

    the creators who want at least $10,000 to get investment from investors. - If the platform success to get at least $10,000 for creators from investors, it transfers all money to creators. If not, it re-transfers funds to each investors. Crowdfunding Platform Investors - They can invest to the creators if they want to support through the crowdfunding platform. Creators - They want at least $10,000 to make a novel product, but they have no money. - They go to a crowdfunding platform to get investment. Three Types of Actors
  5. Smart Contract - A smart contract can be totally replaced

    with previous crowdfunding platform on crowdfunding example. - It is possible to exist a bug in a smart contract because the smart contract is implemented by developers who can make a mistake during development. - Remind that a smart contract is a computer program implemented by developers and is stored inside a blockchain.
  6. - To prevent such a disaster, finding vulnerabilities in smart

    contracts before deployment on a blockchain is really important. - Bugs in smart contracts can cause catastrophic failures because smart contracts often handle digital assets worth millions of dollars. - There has been surging research interest in automatically finding bugs in smart contracts. Importance of Secure Smart Contract
  7. Many Bug-Finding Tools - sFuzz - SASC - Security -

    Slither - ILF - SmartCheck - Remix - MadMax - Verismart - Harvey - Echidna - Osiris - Zeus - Maian - Manticore - Oyente - sCompile - Mythril - ContractFuzzer - Reguard - ContraMaster - Vandal - teEther
  8. - sFuzz - SASC - Security - Slither - ILF

    - SmartCheck - Remix - MadMax - Verismart - Harvey - Echidna - Osiris - Zeus - Maian - Manticore - Oyente - sCompile - Mythril - ContractFuzzer - Reguard - ContraMaster - Vandal - teEther Fuzzers for Smart Contract
  9. Fuzz? Fuzzing? Fuzzer? - Fuzzing is the execution of the

    PUT (Program Under Test) using input(s) sampled from an input space (the “fuzz input space”) that protrudes the expected input space of the PUT. Fuzzing Fuzzer - A fuzzer is a program that performs fuzz testing on a PUT. - Fuzz testing is the use of fuzzing to test if a PUT violates a security policy. - It sometimes uses feedback from PUT to do effective fuzzing. Fuzz - Fuzz is generating a stream of random characters to be consumed by a target program. - The term was originated by Miller et al. in 1990. Definition
  10. Fuzzer Categorization - It observes whole information on internal of

    the target program. - Its overhead is higher than block-box fuzzer due to dynamic instrumentation and SMT solving. White-box fuzzer Grey-box fuzzer - It observes some information on internal of the target program. - Overhead is lower than white-box fuzzer because it performs lightweight analysis. Black-box fuzzer - It does not see the internal of the target program. - It only observes the input/output behavior of PUT.
  11. Many Bug-Finding Tools Suffer from Issues Doesn’t report where the

    bug is triggered. Doesn’t find various bugs in the smart contract. Not open-sourced.
  12. One More Critical Technical Challenge - A smart contract takes

    in a sequence of transactions as input while maintaining a persistent state. - We need to find a transaction sequence that can change the persistent state in a critical way. Main Challenge in Handling Stateful Transaction Current Fuzzers for Smart Contract - They handle it either by randomly varying transaction orders or by resorting to machine learning. - However, all of them is not deterministic, thus, all fails in detecting crucial transaction sequences. Code Coverage Is Not Enough - A traditional fuzzer for binary program usually uses code coverage as its feedback. - However, it is not enough because a smart contract has its own feature, persistent state.
  13. Open = False, Funds = 0, Creator = Creator’s account

    // State function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example (Code)
  14. Open = False, Funds = 0, Creator = Creator’s account

    // State function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example (Code) function SendAllFundsToCreator() { msg.sender.transfer(Funds); } Bug Example For Broken Access Control
  15. Open = False, Funds = 0, Creator = Creator’s account

    // State function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example (Code) function SendAllFundsToCreator() { msg.sender.transfer(Funds); } function SendAllFundsToCreator() { require(msg.sender == Creator); // Fix msg.sender.transfer(Funds); } Bug Example For Broken Access Control (Patch) Fix Bug!
  16. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example (Code) OpenCrowdFunding() Sender: Creator Invest($10,000) Sender: Attacker CloseCrowdFunding() Sender: Attacker Transaction Sequence to Trigger a Bug
  17. Crowdfunding Example (Code) - It is hard to automatically generate

    such a transaction sequence because below three elements must be satisfied at the same time. - Transaction Order - OpenCrowdFunding → Invest → CloseCrowdFunding - Transaction Arguments - value of Invest function must be at least $10,000. - Sender of Transaction - Sender OpenCrowdFunding must be Creator.
  18. SMARTIAN Our Contribution - Propose a novel static analysis technique

    for generating meaningful transaction sequences. - Present fuzzing for smart contracts with dynamic analysis technique. - Make our benchmark public, which includes 500 non-trivial, real-world smart contracts. What Is SMARTIAN? - A grey-box fuzzer for smart contracts supporting that - Reports where the bug is triggered, - Is open-sourced, - Finds various bugs in the smart contracts, - Is able to systemically generate critical transaction sequences with static and dynamic analyses.
  19. SMARTIAN Test Case = Transaction Sequence = Seed Dynamic Analysis

    5FTU$BTF1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Static Analysis EVM Bytecode 010101010011 101001000101 100100101011 101011010010 Bugs Architecture
  20. SMARTIAN Compiling Smart Contract Ethereum Virtual Machine (EVM) is an

    environment to execute a smart contract. Dynamic Analysis 5FTU$BTF1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Static Analysis EVM Bytecode 010101010011 101001000101 100100101011 101011010010 Bugs
  21. SMARTIAN Taking EVM Bytecode As an Input Dynamic Analysis 5FTU$BTF1PPM

    *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Static Analysis EVM Bytecode 010101010011 101001000101 100100101011 101011010010 Bugs
  22. SMARTIAN Dynamic Analysis 5FTU$BTF1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Static Analysis

    EVM Bytecode 010101010011 101001000101 100100101011 101011010010 Bugs Information Gathering with Static Analysis Static analysis is an analysis before the execution of target smart contract.
  23. SMARTIAN Dynamic Analysis 5FTU$BTF1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Static Analysis

    EVM Bytecode 010101010011 101001000101 100100101011 101011010010 Bugs Static analysis is an analysis before the execution of target smart contract. Taking Gathered Information as Input
  24. SMARTIAN Dynamic Analysis 5FTU$BTF1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Static Analysis

    EVM Bytecode 010101010011 101001000101 100100101011 101011010010 Bugs Test Case Pool Initialization with Static Analysis Static analysis is an analysis before the execution of target smart contract.
  25. SMARTIAN Dynamic Analysis 5FTU$BTF1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Static Analysis

    EVM Bytecode 010101010011 101001000101 100100101011 101011010010 Bugs Taking Test Case Pool as Input
  26. SMARTIAN Dynamic Analysis 5FTU$BTF1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Static Analysis

    EVM Bytecode 010101010011 101001000101 100100101011 101011010010 Bugs Fuzzing Dynamic analysis is an analysis during the execution of target smart contract.
  27. SMARTIAN Dynamic Analysis 5FTU$BTF1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Static Analysis

    EVM Bytecode 010101010011 101001000101 100100101011 101011010010 Bugs Getting Feedback and Storing Test Cases during Fuzzing Dynamic analysis is an analysis during the execution of target smart contract.
  28. SMARTIAN - Predict which transaction sequences are meaningful based on

    the gathered information and consider them as initial test cases. - Set the arguments of each function as default value. Test Case Pool Initialization Fuzzing - Find concrete values that can penetrate branches with grey-box testing. - Get feedback if there is new data-flow facts during a dynamic analysis. - Detect various bugs with help of our own bug oracles. Information Gathering - Run a static analysis to collect useful data-flow facts. - Gather which state variables are defined and used by the function, and facts related with sender. Summary
  29. Crowdfunding Example 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Open =

    False, Funds = 0, Creator = Creator’s account function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False
  30. Crowdfunding Example Information Gathering 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH

    Open = False, Funds = 0, Creator = Creator’s account function OpenCrowdFunding(): if sender is Creator: Open = True // Define function Invest(value): if Open: // Use Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False
  31. Crowdfunding Example Information Gathering 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH

    Open = False, Funds = 0, Creator = Creator’s account function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value // Define function CloseCrowdFunding(): if Funds >= $10,000: // Use SendAllFundsToCreator() Open = False
  32. Crowdfunding Example Information Gathering 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH

    Open = False, Funds = 0, Creator = Creator’s account function OpenCrowdFunding(): if sender is Creator: // Sender must be a creator. Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False
  33. Crowdfunding Example Information Gathering - Open is defined in OpenCrowdFunding.

    - Open is used in Invest. - Funds is defined in Invest. - Funds is used in CloseCrowdFunding. - To define Open, sender of OpenCrowdFunding must be Creator. 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH
  34. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: Open = True // Define function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Test Case Pool Initialization
  35. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: // Use Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Test Case Pool Initialization
  36. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value // Define function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Test Case Pool Initialization
  37. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: // Use SendAllFundsToCreator() Open = False Crowdfunding Example 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Test Case Pool Initialization
  38. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Test Case Pool Initialization OpenCrowdFunding() Sender: Attacker Invest(0) Sender: Attacker CloseCrowdFunding() Sender: Attacker
  39. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: // Sender must be Creator Open = True // from gathered information. function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example Fuzzing with Dynamic Analysis OpenCrowdFunding() Sender: Creator Invest(0) Sender: Attacker CloseCrowdFunding() Sender: Attacker 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH
  40. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: // Funds is 0, so SendAllFundsToCreator() // can’t pass this code. Open = False Crowdfunding Example Fuzzing with Dynamic Analysis 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH OpenCrowdFunding() Sender: Creator Invest(0) Sender: Attacker CloseCrowdFunding() Sender: Attacker
  41. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000 SendAllFundsToCreator() Open = False Crowdfunding Example Fuzzing with Dynamic Analysis OpenCrowdFunding() Sender: Creator Invest(0) Sender: Attacker CloseCrowdFunding() Sender: Attacker 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH Define-Use Chain Define-Use Chain
  42. Crowdfunding Example Fuzzing with Dynamic Analysis - SMARTIAN knew that

    there are new def-use chains in the transaction sequence, so it marks the sequence as meaningful, and will re-test the sequence. - SMARTIAN knew that if it pass ‘if Funds >= $10,000’, the argument in the Invest function should be at least $10,000. 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH
  43. Open = False, Funds = 0, Creator = Creator’s account

    function OpenCrowdFunding(): if sender is Creator: Open = True function Invest(value): if Open: Funds += value function CloseCrowdFunding(): if Funds >= $10,000: SendAllFundsToCreator() Open = False Crowdfunding Example Re-Fuzzing with Dynamic Analysis 5FTU$BTF 1PPM *OJUJBMJ[BUJPO 'V[[JOH *OGPSNBUJPO (BUIFSJOH OpenCrowdFunding() Sender: Creator Invest($10,000) Sender: Attacker CloseCrowdFunding() Sender: Attacker
  44. Evaluation Our Environment - Used one container to run a

    tool on a single contract - Spawned at most 72 containers in parallel - Assigned a single CPU core and 6GB of memory to each container. Docker 20.10.3 Solidity Compiler - solc-0.4.25 Machine - An Ubuntu 18.04 server machine - Two Intel E5-2699v4 (2.2GHz) CPUs - 512 GB of main memory.
  45. Evaluation Comparison Target - ILF stands for imitation learning based

    fuzzer. - It is published in CCS’19. - It doesn’t support integer bug. ILF - It is a grey-box fuzzer. - It is published in ICSE’20. sFuzz - It is a symbolic executor. - It is published in ASE’19. - It is made by Trail of Bits. Manticore - It is a symbolic executor. - It supports most bug oracles among comparison target. - It is made by ConsenSys. Mythril
  46. Evaluation Benchmarks SLoC: Source Line of Code SD: Standard Deviation

    ID Source Average SLoC SD SLoC Number of Contracts B1 Verismart 136 48 58 B2 SmartBug 51 75 72 B3 Ethescan 331 277 500
  47. Evaluation Impact of Data-Flow Analyses on B1 - Found most

    CVEs with static and dynamic analyses. - Increased 1% instruction coverage with static and dynamic analyses. - Incurred only 2.7% overhead in the execution time. 0 10 20 30 40 50 60 Total # of CVEs found w/ static & dynamic analyses w/ static analysis only w/ dynamic analysis only w/o any analyses
  48. Evaluation Bugs Found Comparison against State-Of-The-Art Tools on Subset of

    B1 0 5 10 15 20 25 30 35 Total # of CVEs found SMARTIAN Mythril Manticore sFuzz - Some tools didn’t work on several smart contracts of B1. - We compared with subset of B1 that all tools can work. - ILF is excluded because it doesn’t support Integer Bug oracle. - SMARTIAN found most CVEs compared with other tools.
  49. Evaluation Coverage Comparison against State-Of-The-Art Tools on Subset of B1

    0 20000 40000 60000 80000 100000 120000 Instruction Coverage SMARTIAN Mythril Manticore sFuzz - Some tools didn’t work on several smart contracts of B1. - We compared with subset of B1 that all tools can work. - ILF is excluded because it doesn’t support integer bug oracle. - SMARTIAN covered most instructions compared with other tools.
  50. Evaluation Bugs Found Comparison against State-Of-The-Art Tools on B2 0

    10 20 30 40 50 60 70 80 90 Total # of Bugs Found SMARTIAN Mythril ILF Manticore sFuzz - SMARTIAN found most bugs compared with other tools. - All bugs found by SMARTIAN are true positive. - On the other hand, other tools alarmed many false positive.
  51. Evaluation Coverage Comparison against State-Of-The-Art Tools on B2 0 10000

    20000 30000 40000 50000 60000 70000 Instruction Coverage SMARTIAN Mythril ILF Manticore sFuzz - SMARTIAN covered most instructions compared with other tools.
  52. Evaluation Number of Bugs Found by SMARTIAN on B3. Bug

    Name # of Bugs Reported # of TP # of FP Arbitrary Write 0 0 0 Block State Dependency 26 20 6 Control-Flow Hijack 0 0 0 Ether Leak 5 3 2 Integer Bug 170 170 0 Mishandled Exception 2 2 0 Multiple Send 5 5 0 Reentrancy 0 0 0 Suicidal Contract 0 0 0 Transaction Origin Use 11 11 0 Total 219 211 8 TP stands for True Positive. FP stands for False Positive.
  53. Evaluation Number of Bugs Found by SMARTIAN on B3. -

    Most pattern of bugs are similar with B1 that means many developers do copy-and-paste vulnerable codes. - 8 false positives are due to the imperfect of bug oracles. - Found 219 bugs reported in real-world smart contracts including 211 true positives and 8 false positives.
  54. Conclusion - We implemented SMARTIAN, an effective grey-box fuzzer with

    static and dynamic data-flow analyses for finding bugs in smart contract. - SMARTIAN shows best results compared with other state-of-the-art tools. - Secure smart contract is really important.
  55. SMARTIAN - https://github.com/SoftSec-KAIST/Smartian SMARTIAN Source Code SMARTIAN Benchmark - https://github.com/SoftSec-KAIST/Smartian-Artifact

    SMARTIAN Paper - https://conf.researchr.org/details/ase-2021/ase-2021-papers/66/SMARTIAN-Enhancing-Smart- Contract-Fuzzing-with-Static-and-Dynamic-Data-Flow-Analyse
  56. Reference Crowdfunding - https://en.wikipedia.org/wiki/Crowdfunding Fuzz? Fuzzing? Fuzzer? & Fuzzer Categorization

    - https://www.computer.org/csdl/journal/ts/5555/01/08863940/1e0YnO3GyJO And… - Please see the reference section on SMARTIAN paper.