Problem
In a new project assignment, I was made responsible for the pre-silicon verification of a moderately complex ASIC IP block. It was a newly designed, entirely digital IP block meant to do firmware-guided processing of data packets. The firmware instruction set and packet formats were proprietary. The IP block was to become part of a large wireless communications system. It was written by a single RTL designer. I had to build a testbench and write and run all the acceptance tests. A recurring problem for me (and others in my position) is that a lot of my effort is usually spent debugging my own testbench code. Debug always seems to become a huge time suck for ASIC projects of all sizes. For this particular project, I intended to use TDD to improve the quality of the code I write to drive down the amount of time lost to code debugging.
Action
I’d experimented with TDD before this point. For this project, however, I used it in the strictest sense possible in that every line of code I wrote was written test-first (with very few exceptions). As a verification engineer, that meant dynamically building a suite of unit tests for each of the individual pieces in my testbench as each piece was developed. This included all the bus functional models (BFMs), transactions, transaction monitors, and the behavioral model as well as the integrated testbench. Unit tests focused on verifying pin-level protocol, firmware instruction handling, packet format/content handling, and testbench integration. I used SVUnit as the unit test framework (SVUnit is an open-source unit test framework for hardware developers that use SystemVerilog).
Result
I followed the test-first approach strictly for roughly seven weeks (my project was placed on hold at the seven-week mark). During this seven-week period, I had the following observations:
- It took me roughly 2 weeks to find a comfortable rhythm within the TDD cycle (i.e. write a test, confirm the test fails, code the implementation, confirm the test passes). Beyond the 2 week mark, TDD felt very natural.
- Given that I had found a natural rhythm after 2 weeks, test-first development was not as difficult to stick to as I thought it would be; even considering my long history of writing code test-last.
- The constant confirmation (i.e. passing feedback from my unit tests) was very comforting and built a lot of confidence. In terms of code quality relative to past experience, I feel that the code I produced in this project with TDD was of far higher quality.
- In roughly 2000 lines of code, I produced a total of 2 defects in my testbench. These were defects that were not trapped by my unit test suite and were instead found by the designer of the IP block.
- I had a design-to-testbench defect ratio of 15:1. This is substantial. From my past, I would guestimate a ratio nearer 1:>1 (i.e. testbench defect rates slightly higher than design defect rates). This, to me, is the clearest sign that TDD helped me write higher-quality code, especially considering the designer for the project was experienced and very capable.
- I wrote over 100 unit tests during the development of my testbench; the ratio of unit test code to testbench code was almost 2:1. This is a lot of extra code when compared to test-last, however, I didn’t find the extra code consumed as much time as I expected it to.
- Because of the test-first approach, I found myself spending very little time debugging my own code. More significantly, the designer of the IP block wasted very little of his time debugging my code and was able to focus on defects in the design.
