[Testing] What's the difference between a mock & stub?


Answers

Foreword

There are several definitions of objects, that are not real. The general term is test double. This term encompasses: dummy, fake, stub, mock.

Reference

According to Martin Fowler's article:

  • Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
  • Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
  • Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'.
  • Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive.

Style

Mocks vs Stubs = Behavioral testing vs State testing

Principle

According to the principle of Test only one thing per test, there may be several stubs in one test, but generally there is only one mock.

Lifecycle

Test lifecycle with stubs:

  1. Setup - Prepare object that is being tested and its stubs collaborators.
  2. Exercise - Test the functionality.
  3. Verify state - Use asserts to check object's state.
  4. Teardown - Clean up resources.

Test lifecycle with mocks:

  1. Setup data - Prepare object that is being tested.
  2. Setup expectations - Prepare expectations in mock that is being used by primary object.
  3. Exercise - Test the functionality.
  4. Verify expectations - Verify that correct methods has been invoked in mock.
  5. Verify state - Use asserts to check object's state.
  6. Teardown - Clean up resources.

Summary

Both mocks and stubs testing give an answer for the question: What is the result?

Testing with mocks are also interested in: How the result has been achieved?

Question

I've read various articles about mocking vs stubbing in testing, including Martin Fowler's Mocks Aren't Stubs, but still don't understand the difference.




I think the most important difference between them is their intentions.

Let me try to explain it in WHY stub vs. WHY mock

Suppose I'm writing test code for my mac twitter client's public timeline controller

Here is test sample code

twitter_api.stub(:public_timeline).and_return(public_timeline_array)
client_ui.should_receive(:insert_timeline_above).with(public_timeline_array)
controller.refresh_public_timeline
  • STUB: The network connection to twitter API is very slow, which make my test slow. I know it will return timelines, so I made a stub simulating HTTP twitter API, so that my test will run it very fast, and I can running the test even I'm offline.
  • MOCK: I haven't written any of my UI methods yet, and I'm not sure what methods I need to write for my ui object. I hope to know how my controller will collaborate with my ui object by writing the test code.

By writing mock, you discover the objects collaboration relationship by verifying the expectation are met, while stub only simulate the object's behavior.

I suggest to read this article if you're trying to know more about mocks: http://jmock.org/oopsla2004.pdf




This slide explain the main differences very good.

*From CSE 403 Lecture 16 , University of Washington (slide created by "Marty Stepp")







Stubs don't fail your tests, mock can.




I like the explanantion put out by Roy Osherove [video link].

Every class or object created is a Fake. It is a Mock if you verify calls against it. Otherwise its a stub.




Stubs are used on methods with an expected return value which you setup in your test. Mocks are used on void methods which are verified in the Assert that they are called.




Here's a description of each one followed by with real world sample.

  • Dummy - just bogus values to satisfy the API.

    Example: If you're testing a method of a class which requires many mandatory parameters in a constructor which have no effect on your test, then you may create dummy objects for the purpose of creating new instances of a class.

  • Fake - create a test implementation of a class which may have a dependency on some external infrastructure. (It's good practice that your unit test does NOT actually interact with external infrastructure.)

    Example: Create fake implementation for accessing a database, replace it with in-memory collection.

  • Stub - override methods to return hard-coded values, also referred to as state-based.

    Example: Your test class depends on a method Calculate() taking 5 minutes to complete. Rather than wait for 5 minutes you can replace its real implementation with stub that returns hard-coded values; taking only a small fraction of the time.

  • Mock - very similar to Stub but interaction-based rather than state-based. This means you don't expect from Mock to return some value, but to assume that specific order of method calls are made.

    Example: You're testing a user registration class. After calling Save, it should call SendConfirmationEmail.

Stubs and Mocks are actually sub types of Mock, both swap real implementation with test implementation, but for different, specific reasons.




If you compare it to debugging:

Stub is like making sure a method returns the correct value

Mock is like actually stepping into the method and making sure everything inside is correct before returning the correct value.




I have used python examples in my answer to illustrate the differences.

Stub - Stubbing is a software development technique used to implement methods of classes early in the development life-cycle. They are used commonly as placeholders for implementation of a known interface, where the interface is finalized or known but the implementation is not yet known or finalized. You begin with stubs, which simply means that you only write the definition of a function down and leave the actual code for later. The advantage is that you won't forget methods and you can continue to think about your design while seeing it in code. You can also have your stub return a static response so that the response can be used by other parts of your code immediately. Stub objects provide a valid response, but it's static no matter what input you pass in, you'll always get the same response:

class Foo(object):
    def bar1(self):
        pass

    def bar2(self):
        #or ...
        raise NotImplementedError

    def bar3(self):
        #or return dummy data
        return "Dummy Data"

Mock objects are used in mock test cases they validate that certain methods are called on those objects. Mock objects are simulated objects that mimic the behaviour of real objects in controlled ways. You typically creates a mock object to test the behaviour of some other object. Mocks let us simulate resources that are either unavailable or too unwieldy for unit testing.

mymodule.py:

import os
import os.path

def rm(filename):
    if os.path.isfile(filename):
        os.remove(filename)

test.py:

from mymodule import rm
import mock
import unittest

class RmTestCase(unittest.TestCase):
    @mock.patch('mymodule.os')
    def test_rm(self, mock_os):
        rm("any path")
        # test that rm called os.remove with the right parameters
        mock_os.remove.assert_called_with("any path")

if __name__ == '__main__':
    unittest.main()

This is a very basic example that just runs rm and asserts the parameter it was called with. You can use mock with objects not just functions as shown here, and you can also return a value so a mock object can be used to replace a stub for testing.

More on unittest.mock, note in python 2.x mock is not included in unittest but is a downloadable module that can be downloaded via pip (pip install mock).

I have also read "The Art of Unit Testing" by Roy Osherove and I think it would be great if a similar book was written using Python and Python examples. If anyone knows of such a book please do share. Cheers :)




  • Stubs vs. Mocks
    • Stubs
      • provide specific answers to methods calls
      • used by code under test to isolate it
      • cannot fail test
      • often implement abstract methods
    • Mocks
      • "superset" of stubs; can assert that certain methods are called
      • used to test behavour of code under test
      • can fail test
      • often mocks interfaces



I came across this interesting article by UncleBob The Little Mocker. It explains all the terminology in a very easy to understand manner, so its useful for beginners. Martin Fowlers article is a hard read especially for beginners like me.




Right from the paper Mock Roles, not Objects, by the developers of jMock :

Stubs are dummy implementations of production code that return canned results. Mock Objects act as stubs, but also include assertions to instrument the interactions of the target object with its neighbours.

So, the main differences are:

  • expectations set on stubs are usually generic, while expectations set on mocks can be more "clever" (e.g. return this on the first call, this on the second etc.).
  • stubs are mainly used to setup indirect inputs of the SUT, while mocks can be used to test both indirect inputs and indirect outputs of the SUT.

To sum up, while also trying to disperse the confusion from Fowler's article title: mocks are stubs, but they are not only stubs.




A stub is an empty function which is used to avoid unhandled exceptions during tests:

function foo(){}

A mock is an artificial function which is used to avoid OS, environment or hardware dependencies during tests:

function foo(bar){ window = this; return window.toString(bar); }

In terms of assertions and state:

  • Mocks are asserted before an event or state change
  • Stubs are not asserted, they provide state before an event to avoid executing code from unrelated units
  • Spies are setup like stubs, then asserted after an event or state change
  • Fakes are not asserted, they run after an event with hardcoded dependencies to avoid state

References