Testing smart contracts
Testing smart contracts is one of the most important measures for improving smart contract security. Unlike traditional software, smart contracts cannot typically be updated after launching, making it imperative to test rigorously before deploying contracts on the Ethereum network.
What is smart contract testing?
Smart contract testing means performing detailed analysis and evaluation of a smart contract to assess the quality of its source code during the development cycle. Testing a smart contract makes it easier to identify bugs and vulnerabilities and reduces the possibility of software errors that could lead to costly exploits.
Smart contract testing takes many forms, with different methods offering benefits. Strategies for testing Ethereum smart contracts can be classified into two broad categories: automated testing and manual testing.
Automated testing involves using automated tools to carry out scripted testing of smart contracts. This technique relies on automated software that can execute repeated tests to find defects in smart contracts.
Automated testing is efficient, uses fewer resources, and promises higher levels of coverage than manual analysis. Automated testing tools can also be configured with test data, allowing them to compare predicted behaviors with actual results.
Manual testing is human-aided and involves an individual who executes testing steps manually. Code audits, where developers and/or auditors, go over every line of contract code, are an example of manual testing for smart contracts.
Manual testing of smart contracts requires considerable skill and a considerable investment of time, money, and effort. Moreover, manual testing can sometimes be susceptible to the problems of human error.
However, applying manual testing to smart contracts can also be beneficial. Code audits harness human intelligence to find defects in contract code that might go undetected during automated testing.
Manual-testing your smart contracts can also reveal vulnerabilities that exist outside the code, but can still affect it. For example, a smart contract audit can discover vulnerabilities arising from flawed interaction with off-chain components.
Why is it important to test smart contracts?
Testing smart contracts is important for the following reasons:
1. Smart contracts are high-value applications
Smart contracts often deal with high-value financial assets, especially in industries like decentralized finance (DeFi), and valuable items, such as non-fungible tokens (NFTs). As such, minor vulnerabilities in smart contracts can and often lead to massive, irrecoverable losses for users. Comprehensive testing can, however, expose errors in smart contract code and reduce security risks before deployment.
2. Smart contracts are immutable
Smart contracts deployed in the Ethereum Virtual Machine (EVM) are immutable by default. While traditional developers may be used to fixing software bugs after launching, Ethereum development leaves little room for patching security flaws once a smart contract is live on the blockchain.
While upgradeability mechanisms for smart contracts exist, such as proxy patterns, these can be difficult to implement. Besides reducing immutability and introducing complexity, upgrades often demand complex governance processes.
For the most part, upgrades should be considered a last resort and avoided unless necessary. Detecting potential vulnerabilities and flaws in your smart contract during the pre-launch phase reduces the need for a logic upgrade.
Automated testing for smart contracts
1. Functional testing
Functional testing verifies the functionality of a smart contract and provides assurance that each function in the code works as expected. Functional testing requires understanding how your smart contract should behave in certain conditions. Then you can test each function by running computations with selected values and comparing the returned output with the expected output.
Functional testing covers three methods: unit testing, integration testing, and system testing.
Unit testing involves testing individual components in a smart contract for correctness. A unit test is simple, quick to run, and provides a clear idea of what went wrong if the test fails.
Unit tests are crucial for smart contract development, especially if you need to add new logic to the code. You can verify the behavior of each function and confirm that it executes as intended.
Running a unit test often requires creating assertions—simple, informal statements specifying requirements for a smart contract. Unit testing can then be used to test each assertion and see if it holds true under execution.
Examples of contract-related assertions include:
i. "Only the admin can pause the contract"
ii. "Non-admins cannot mint new tokens"
iii. "The contract reverts on errors"
Integration testing is a level higher than unit testing on the testing hierarchy. In integration testing, individual components of the smart contract are tested together.
This approach detects errors arising from interactions between different components of a contract or across multiple contracts. You should use this method if you have a complex contract with multiple functions or one that interfaces with other contracts.
Integration testing can be useful for ensuring that things like inheritance(opens in a new tab) and dependency injection work properly.
System testing is the final phase of functional testing for smart contracts. A system evaluates the smart contract as one fully integrated product to see if it performs as specified in the technical requirements.
You can think of this stage as checking the end-to-end flow of your smart contract from a user’s point of view. A good way to perform system testing on a smart contract is to deploy it on a production-like environment, such as a testnet or development network.
Here, end-users can perform trial runs and report any issues with the contract’s business logic and overall functionality. System testing is important because you cannot change code once the contract is deployed in the main EVM environment.
2. Static/dynamic analysis
Static analysis and dynamic analysis are two automated testing methods for evaluating the security qualities of smart contracts. Both techniques, however, use different approaches for finding defects in contract code.
Static analysis examines the source code or bytecode of a smart contract before execution. This means you can debug contract code without actually running the program. Static analyzers can detect common vulnerabilities in Ethereum smart contracts and aid compliance with best practices.
Dynamic analysis techniques require executing the smart contract in a runtime environment to identify issues in your code. Dynamic code analyzers observe contract behaviors during execution and generate a detailed report of identified vulnerabilities and property violations.
Fuzzing is an example of a dynamic analysis technique for testing contracts. During fuzz testing, a fuzzer feeds your smart contract with malformed and invalid data and monitors how the contract responds to those inputs.
Like any program, smart contracts rely on inputs provided by users to execute functions. And, while we assume users will provide correct inputs, this may not always be the case.
In some cases, sending incorrect input values to a smart contract can cause resource leaks, crashes, or worse, lead to unintended code execution. Fuzzing campaigns identify such problems beforehand, allowing you to eliminate the vulnerability.
Manual testing for smart contracts
1. Code audits
A code audit is a detailed evaluation of a smart contract's source code to uncover possible failure-points, security flaws, and poor development practices. While code audits can be automated, we refer to human-aided code analysis here.
Code audits require an attacker mindset to map out possible attack vectors in smart contracts. Even if you run automated audits, analyzing every line of source code is a minimum requirement for writing secure smart contracts.
You can also commission a security audit to give users higher assurances of smart contract safety. Audits benefit from extensive analysis performed by cybersecurity professionals and detect potential vulnerabilities or bugs that could break the smart contract functionality.
2. Bug bounties
A bug bounty is a financial reward given to an individual who discovers a vulnerability or bug in a program's code and reports to developers. Bug bounties are similar to audits since it involves asking others to help find defects in smart contracts. The major difference is that bug bounty programs are open to the wider developer/hacker community.
Bug bounty programs often attract a broad class of ethical hackers and independent security professionals with unique skills and experience. This may be an advantage over smart contract audits that mainly rely on teams who may possess limited or narrow expertise.
Testing vs. formal verification
While testing helps confirm that a contract returns the expected results for some data inputs, it cannot conclusively prove the same for inputs not used during tests. Testing a smart contract cannot guarantee "functional correctness", meaning it cannot show that a program behaves as required for all sets of input values and conditions.
As such, developers are encouraged to incorporate formal verification into their approach for assessing the correctness of smart contracts. Formal verification uses formal methods(opens in a new tab)—mathematically rigorous techniques for specifying and verifying software.
Formal verification is considered important for smart contracts because it helps developers formally test assumptions relating to smart contracts. This is done by creating formal specifications that describe a smart contract's properties and verifying that a formal model of the smart contract matches the specification. This approach increases confidence that a smart contract will only execute functions as defined in its business logic and nothing else.
More on formal verification for smart contracts
Testing tools and libraries
Unit testing tools
Solidity-Coverage - Solidity code coverage tool useful for testing smart contracts.
Waffle - Framework for advanced smart contract development and testing (based on ethers.js).
Remix Tests - Tool for testing Solidity smart contracts. Works underneath Remix IDE "Solidity Unit Testing" plugin which is used to write and run test cases for a contract.
OpenZeppelin Test Helpers - Assertion library for Ethereum smart contract testing. Make sure your contracts behave as expected!
Truffle smart contract test framework - Automated testing framework to make testing your contracts a breeze.
Brownie unit testing framework - Brownie utilizes Pytest, a feature-rich test framework that lets you write small tests with minimal code, scales well for large projects, and is highly extendable.
Foundry Tests - Foundry offers Forge, a fast and flexible Ethereum testing framework capable of executing simple unit tests, gas optimization checks, and contract fuzzing.
Etheno - All-in-one Ethereum testing tool comprising a JSON RPC multiplexer, analysis tool wrapper, and test integration tool. Etheno eliminates the complexity of setting up analysis tools like Manticore and Echidna on large, multi-contract projects.
Woke development and testing framework - Test and deployment scripts in Python with type hints, fuzzer, debugging support, code coverage, cross-chain testing.
Static analysis tools
Mythril - EVM bytecode assessment tool for detecting contract vulnerabilities using taint analysis, concolic analysis, and control flow checking.
Slither - Python-based Solidity static analysis framework for finding vulnerabilities, enhancing code comprehension, and writing custom analyses for smart contracts.
Rattle - EVM bytecode static analysis framework designed to work on deployed smart contracts.
Dynamic analysis tools
Echidna - Fast contract fuzzer for detecting vulnerabilities in smart contracts through property-based testing.
Harvey - Automated fuzzing tool useful for detecting property violations in smart contract code.
Manticore - Dynamic symbolic execution framework for analyzing EVM bytecode.
Smart contract auditing services
ConsenSys Diligence - Smart contract auditing service helping projects across the blockchain ecosystem ensure their protocols are ready for launch and built to protect users.
CertiK - Blockchain security firm pioneering the use of cutting-edge formal Verification technology on smart contracts and blockchain networks.
Trail of Bits - Cybersecurity company that combines security research with an attacker mentality to reduce risk and fortify code.
PeckShield - Blockchain security company offering products and services for the security, privacy, and usability of the entire blockchain ecosystem.
QuantStamp - Auditing service facilitating the mainstream adoption of blockchain technology through security and risk assessment services.
OpenZeppelin - Smart contract security company providing security audits for distributed systems.
Nethermind - Solidity and Cairo auditing services, ensuring the integrity of smart contracts and the safety of users across Ethereum and Starknet.
Bug bounty platforms
Immunefi - Bug bounty platform for smart contracts and DeFi projects, where security researchers review code, disclose vulnerabilities, get paid, and make crypto safer.
HackerOne - Vulnerability coordination and bug bounty platform that connects businesses with penetration testers and cybersecurity researchers.
- Solidity and Truffle Continuous Integration Setup – How to setup Travis or Circle CI for Truffle testing along with useful plugins.
- Testing products overview – An overview and comparison of different testing products.
- How to use Echidna to test smart contracts
- How to use Manticore to find smart contract bugs
- How to use Slither to find smart contract bugs
- How to mock Solidity contracts for testing
- How to migrate from Truffle Tests to OpenZeppelin Test Environment(opens in a new tab)
- How to test contracts after they have been deployed on a network(opens in a new tab)
- Solidity, Blockchain, and Smart Contract Course (YouTube)(opens in a new tab)
- An In-Depth Guide to Testing Ethereum Smart Contracts(opens in a new tab) - Ben Hauser
- How to Test Ethereum Smart Contracts(opens in a new tab) - Alex Roan