Case Studies

Acceptance Test-Driven Development / Behavior Driven Development can increase the effectiveness of teams. This has been demonstrated numerous times. Here are some case studies.

Critical Success Factor

At a financial firm where I taught and coached ATDD and TDD, here's the results as reported by one team:
One of the critical success factors for our project was our adoption of ATTD and TDD, including frequent test collaboration, engaging the business in writing acceptance test criteria, and robust test automation. Collaborating on writing tests ensured that the entire team understood all stories, which facilitated swarming and cross-skill development. We got questions answered real-time from the business that used to occur in one-off conversations and caused system test failure rework. We saw dramatically reduced failures during exploratory tests which further stabilized velocity. We added 0 manual test cases to our regression bed, and had high quality in production.

ATDD Saves a Release

Here is a report from a team at a leading financial company:

Our project team was formed to deliver a business-critical function with a fixed delivery date. Many of the team members were new to line of business that we were asked to support. Early in the project, Acceptance Test-Driven Development, and more specifically Cucumber, was a useful tool for capturing and documenting the expectations of the system. The business team was able to communicate requirements in familiar terms, even if Gherkin was a new syntax. The IT team was able to understand table-driven examples, while learning the business terminology. The two collaborating sides of our project were never completely out of their comfort zones.

Having an automated test suite based on Cucumber gives both our IT team and business team the confidence to release our software to our production environment more frequently than other project teams and without long, manual regression test cycles typical of a company of our size. Our project team knows immediately when a software change unintentionally impacts another part of the system because our automated test suite is integrated into our continuous build environment. This point was clearly on display during a Sprint Review meeting with senior leadership when a series of tests failed during a demo of our application, caused by a pre-demo check-in from a project team member. In turn, we modified the demo to show the failing tests, and how the software would be corrected to prevent defects from leaking into our production baseline.

It's worth calling out one specific example of a Cucumber success story for our project. An entire software release was saved because we were able to debug a troublesome piece of business logic through the creation of a comprehensive, production-like integration test using Cucumber. In this case, we had a series of JUnit tests that were passing, but an integration scenario was failing. The JUnit tests were not comprehensive enough with their use of mock objects and data to uncover a hidden bug in the logic. Through the creation and debug of a real world example using Cucumber, we were able to find the flaw in our logic, correct the software defect, and elevate on schedule.


Totally Worth It

Here's a report from one team:
The product owner was originally skeptical going into training, but she is clearly a fan now, stating that it is "totally worth it" to do ATDD. Her thoughts were mostly about the effects of the workflow changes. Automated testing was more of an after-thought. She could clearly see and articulate the benefits of simply implementing the workflow change associated with ATDD. She said that one day of ATDD specific training, plus a half day of coaching, plus a committed team is all you need to get going. Benefits she sited included:
  • Team "happiness factor" increased
  • Specifically, the lead developer and tester are much happier
  • Less stress on testers, more distributed testing effort across the sprint
  • Helped to create/enhance the "we are a team" feeling
  • Fewer production defects
  • Fewer test environment defects

What ATDD/BDD Solves

I've been teaching Acceptance Test-Driven Development (ATDD) / Behavior Driven Development (BDD) for many years. ATDD/BDD involves the triad (customer, developer, and tester) collaborating together on defining the details of a requirement or user story in the form of tests. These tests are written prior to coding and the developer uses these tests to check that his or her code passes those tests. The tests may be run manually or they may be automated. At the start of every course, I ask the attendees for issues they have with their development processes. After they have experienced the ATDD/BDD process, including creation of acceptance tests, I review their issues to see whether they feel that ATDD/BDD will help, hurt, or be neutral in respect to those issues. Here's a sampling of the issues in the attendees' own words. According to them, ATDD will help in solving:
  • Unclear requirements
  • Missed requirements
  • No detailed requirements
  • Development without requirements
  • Unclear business rules
  • Poorly defined acceptance criteria


  • Huge user stories
  • Smaller stories leave gaps
  • Not enough detail in stories


  • Not enough testers
  • Missing test cases
  • Not enough time for testing
  • Stories not verifiable until end of the sprint
  • Test data not available
  • Issues with integration tests


  • Coding starts before tests are written
  • Development team not understanding business process
  • Acceptance tests written for coding, not testing perspective
  • Communication challenges - IT is technical, business is not technical

If you have any of these issues, there's a great chance that adopting ATDD will help in solving it.


 

Class Reviews

ATDD Malvern, PA

  • Ken was great about addressing the specific needs of our team. He gave great examples and helped ensure understanding through interaction.
  • Ken did an excellent job of providing quality information as to how our technology driven projects team could best apply the concepts of the course.

    • ATDD + Cucumber Malvern, PA

  • Ken is knowledgeable and kept the class moving and engaged. I liked the part where he coached us about how we could utilize ATDD for our current projects.
  • The course helped me think differently about a problem. I loved the analogy to "when you get a compile error, you fix it don't you?" Write a test first, if the test doesn't pass, fix it.

    • ATDD, Malvern PA

  • Ken was very interested in our specific questions and not just delivering the content of the course; this use of examples and writing down a lot of what he said was effective.
  • Looking forward to tomorrow where Ken will give advice on our specific project/stories and how we can apply the new concepts.

    • ATDD, Charlotte, NC

  • Ken is extremely knowledgeable and was a great consultant.
  • I liked taking the course with your team so you can use real and relevant examples. Ken was able to apply company examples.
  • Ken presented in a way that was thorough and easy to understand. I appreciate the time he took to answer all kinds of questions about how we can improve our process.
  • Ken kept interest and used some of our real life issues. I want more practice with him.

    • ATDD Charlotte, NC

  • I liked lots of examples and exercises, not the usual ATM example. Ken was awesome and made the course better. He was full of energy and knowledge.

    • ATDD, Malvern, PA

  • Ken was engaging. I liked his real-world examples. It was easy to pay attention. I liked what a quick study he was on our actual work.
  • Ken is very knowledgeable and was able to steer our conversations in productive directions.

    • ATDD, Malvern PA

  • I like the open discussion and working with project team members on a current story to deepen the understanding and application.

    • TDD, Malvern, PA

  • Ken was good at taking our questions about our own issues and helping us work through them.

    •