testing - mocks - what is the difference between mock and stub for microservice tests




What's the difference between a mock & stub? (20)

Foreword

There are several definitions of objects, that are not real. The general term is test double. This term encompasses: dummy, fake, stub, mock.

Reference

According to Martin Fowler's article:

  • Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
  • Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
  • Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'.
  • Mocks are what we are talking about here: objects pre-programmed with expectations which form a specification of the calls they are expected to receive.

Style

Mocks vs Stubs = Behavioral testing vs State testing

Principle

According to the principle of Test only one thing per test, there may be several stubs in one test, but generally there is only one mock.

Lifecycle

Test lifecycle with stubs:

  1. Setup - Prepare object that is being tested and its stubs collaborators.
  2. Exercise - Test the functionality.
  3. Verify state - Use asserts to check object's state.
  4. Teardown - Clean up resources.

Test lifecycle with mocks:

  1. Setup data - Prepare object that is being tested.
  2. Setup expectations - Prepare expectations in mock that is being used by primary object.
  3. Exercise - Test the functionality.
  4. Verify expectations - Verify that correct methods has been invoked in mock.
  5. Verify state - Use asserts to check object's state.
  6. Teardown - Clean up resources.

Summary

Both mocks and stubs testing give an answer for the question: What is the result?

Testing with mocks are also interested in: How the result has been achieved?

I've read various articles about mocking vs stubbing in testing, including Martin Fowler's Mocks Aren't Stubs, but still don't understand the difference.


A Stub is an object that implements an interface of a component, but instead of returning what the component would return when called, the stub can be configured to return a value that suits the test. Using stubs a unit test can test if a unit can handle various return values from its collaborator. Using a stub instead of a real collaborator in a unit test could be expressed like this:

unit test --> stub

unit test --> unit --> stub

unit test asserts on results and state of unit

First the unit test creates the stub and configures its return values. Then the unit test creates the unit and sets the stub on it. Now the unit test calls the unit which in turn calls the stub. Finally the unit test makes assertions about the results of the method calls on the unit.

A Mock is like a stub, only it also has methods that make it possible determine what methods where called on the Mock. Using a mock it is thus possible to both test if the unit can handle various return values correctly, and also if the unit uses the collaborator correctly. For instance, you cannot see by the value returned from a dao object whether the data was read from the database using a Statement or a PreparedStatement. Nor can you see if the connection.close() method was called before returning the value. This is possible with mocks. In other words, mocks makes it possible to test a units complete interaction with a collaborator. Not just the collaborator methods that return values used by the unit. Using a mock in a unit test could be expressed like this:

unit test --> mock

unit test --> unit --> mock

unit test asserts on result and state of unit

unit test asserts on the methods called on mock

More Detail >> Here


Stub helps us to run test. How? It gives values which helps to run test. These values are itself not real and we created these values just to run the test. For example we create a HashMap to give us values which are similar to values in database table. So instead of directly interacting with database we interact with Hashmap.

Mock is an fake object which runs the test. where we put assert.


Stub

I believe the biggest distinction is that a stub you have already written with predetermined behavior. So you would have a class that implements the dependency (abstract class or interface most likely) you are faking for testing purposes and the methods would just be stubbed out with set responses. They would not do anything fancy and you would have already written the stubbed code for it outside of your test.

Mock

A mock is something that as part of your test you have to setup with your expectations. A mock is not setup in a predetermined way so you have code that does it in your test. Mocks in a way are determined at runtime since the code that sets the expectations has to run before they do anything.

Difference

Tests written with mocks usually follow an initialize -> set expectations -> exercise -> verify pattern to testing. While the pre-written stub would follow an initialize -> exercise -> verify.

Similarity

The purpose of both is to eliminate testing all the dependencies of a class or function so your tests are more focused and simpler in what they are trying to prove.


A Mock is just testing behaviour, making sure certain methods are called. A Stub is a testable version (per se) of a particular object.

What do you mean an Apple way?


A stub is a fake object built for test purposes. A mock is a stub that records whether expected calls effectively occurred.


Here's a description of each one followed by with real world sample.

  • Dummy - just bogus values to satisfy the API.

    Example: If you're testing a method of a class which requires many mandatory parameters in a constructor which have no effect on your test, then you may create dummy objects for the purpose of creating new instances of a class.

  • Fake - create a test implementation of a class which may have a dependency on some external infrastructure. (It's good practice that your unit test does NOT actually interact with external infrastructure.)

    Example: Create fake implementation for accessing a database, replace it with in-memory collection.

  • Stub - override methods to return hard-coded values, also referred to as state-based.

    Example: Your test class depends on a method Calculate() taking 5 minutes to complete. Rather than wait for 5 minutes you can replace its real implementation with stub that returns hard-coded values; taking only a small fraction of the time.

  • Mock - very similar to Stub but interaction-based rather than state-based. This means you don't expect from Mock to return some value, but to assume that specific order of method calls are made.

    Example: You're testing a user registration class. After calling Save, it should call SendConfirmationEmail.

Stubs and Mocks are actually sub types of Mock, both swap real implementation with test implementation, but for different, specific reasons.


I came across this interesting article by UncleBob The Little Mocker. It explains all the terminology in a very easy to understand manner, so its useful for beginners. Martin Fowlers article is a hard read especially for beginners like me.


I like the explanantion put out by Roy Osherove [video link].

Every class or object created is a Fake. It is a Mock if you verify calls against it. Otherwise its a stub.


I think the most important difference between them is their intentions.

Let me try to explain it in WHY stub vs. WHY mock

Suppose I'm writing test code for my mac twitter client's public timeline controller

Here is test sample code

twitter_api.stub(:public_timeline).and_return(public_timeline_array)
client_ui.should_receive(:insert_timeline_above).with(public_timeline_array)
controller.refresh_public_timeline
  • STUB: The network connection to twitter API is very slow, which make my test slow. I know it will return timelines, so I made a stub simulating HTTP twitter API, so that my test will run it very fast, and I can running the test even I'm offline.
  • MOCK: I haven't written any of my UI methods yet, and I'm not sure what methods I need to write for my ui object. I hope to know how my controller will collaborate with my ui object by writing the test code.

By writing mock, you discover the objects collaboration relationship by verifying the expectation are met, while stub only simulate the object's behavior.

I suggest to read this article if you're trying to know more about mocks: http://jmock.org/oopsla2004.pdf


If you compare it to debugging:

Stub is like making sure a method returns the correct value

Mock is like actually stepping into the method and making sure everything inside is correct before returning the correct value.


In the codeschool.com course, Rails Testing for Zombies, they give this definition of the terms:

Stub

For replacing a method with code that returns a specified result.

Mock

A stub with an assertion that the method gets called.

So as Sean Copenhaver described in his answer, the difference is that mocks set expectations (i.e. make assertions, about whether or how they get called).


Right from the paper Mock Roles, not Objects, by the developers of jMock :

Stubs are dummy implementations of production code that return canned results. Mock Objects act as stubs, but also include assertions to instrument the interactions of the target object with its neighbours.

So, the main differences are:

  • expectations set on stubs are usually generic, while expectations set on mocks can be more "clever" (e.g. return this on the first call, this on the second etc.).
  • stubs are mainly used to setup indirect inputs of the SUT, while mocks can be used to test both indirect inputs and indirect outputs of the SUT.

To sum up, while also trying to disperse the confusion from Fowler's article title: mocks are stubs, but they are not only stubs.


See below example of mocks vs stubs using C# and Moq framework. Moq doesn't have a special keyword for Stub but you can use Mock object to create stubs too.

namespace UnitTestProject2
{
    using Microsoft.VisualStudio.TestTools.UnitTesting;
    using Moq;
    [TestClass]
    public class UnitTest1
    {
        /// <summary>
        /// Test using Mock to Verify that GetNameWithPrefix method calls Repository GetName method "once" when Id is greater than Zero
        /// </summary>
        [TestMethod]
        public void GetNameWithPrefix_IdIsTwelve_GetNameCalledOnce()
        {
            // Arrange 
            var mockEntityRepository = new Mock<IEntityRepository>();
            mockEntityRepository.Setup(m => m.GetName(It.IsAny<int>()));

            var entity = new EntityClass(mockEntityRepository.Object);
            // Act 
            var name = entity.GetNameWithPrefix(12);
            // Assert
            mockEntityRepository.Verify(m => m.GetName(It.IsAny<int>()), Times.Once);
        }
        /// <summary>
        /// Test using Mock to Verify that GetNameWithPrefix method doesn't call Repository GetName method when Id is Zero
        /// </summary>
        [TestMethod]
        public void GetNameWithPrefix_IdIsZero_GetNameNeverCalled()
        {
            // Arrange 
            var mockEntityRepository = new Mock<IEntityRepository>();
            mockEntityRepository.Setup(m => m.GetName(It.IsAny<int>()));
            var entity = new EntityClass(mockEntityRepository.Object);
            // Act 
            var name = entity.GetNameWithPrefix(0);
            // Assert
            mockEntityRepository.Verify(m => m.GetName(It.IsAny<int>()), Times.Never);
        }
        /// <summary>
        /// Test using Stub to Verify that GetNameWithPrefix method returns Name with a Prefix
        /// </summary>
        [TestMethod]
        public void GetNameWithPrefix_IdIsTwelve_ReturnsNameWithPrefix()
        {
            // Arrange 
            var stubEntityRepository = new Mock<IEntityRepository>();
            stubEntityRepository.Setup(m => m.GetName(It.IsAny<int>()))
                .Returns("Stub");
            const string EXPECTED_NAME_WITH_PREFIX = "Mr. Stub";
            var entity = new EntityClass(stubEntityRepository.Object);
            // Act 
            var name = entity.GetNameWithPrefix(12);
            // Assert
            Assert.AreEqual(EXPECTED_NAME_WITH_PREFIX, name);
        }
    }
    public class EntityClass
    {
        private IEntityRepository _entityRepository;
        public EntityClass(IEntityRepository entityRepository)
        {
            this._entityRepository = entityRepository;
        }
        public string Name { get; set; }
        public string GetNameWithPrefix(int id)
        {
            string name = string.Empty;
            if (id > 0)
            {
                name = this._entityRepository.GetName(id);
            }
            return "Mr. " + name;
        }
    }
    public interface IEntityRepository
    {
        string GetName(int id);
    }
    public class EntityRepository:IEntityRepository
    {
        public string GetName(int id)
        {
            // Code to connect to DB and get name based on Id
            return "NameFromDb";
        }
    }
}

Stub is simple fake object. It just makes sure test runs smoothly.
Mock is smarter stub. You verify Your test passes through it.


Stubs are used on methods with an expected return value which you setup in your test. Mocks are used on void methods which are verified in the Assert that they are called.


This slide explain the main differences very good.

*From CSE 403 Lecture 16 , University of Washington (slide created by "Marty Stepp")


To be very clear and practical:

Stub: A class or object that implements the methods of the class/object to be faked and returns always what you want.

Example in JavaScript:

var Stub = {
   method_a: function(param_a, param_b){
      return 'This is an static result';
   }
}

Mock: The same of stub, but it adds some logic that "verifies" when a method is called so you can be sure some implementation is calling that method.

As @mLevan says imagine as an example that you're testing a user registration class. After calling Save, it should call SendConfirmationEmail.

A very stupid code Example:

var Mock = {
   calls: {
      method_a: 0
   }

   method_a: function(param_a, param_b){
     this.method_a++; 
     console.log('Mock.method_a its been called!');
   }
}

following is my understanding...

  • if you create test objects locally and feed your local service with that, you are using mock object. this will give a test for the method you implemented in your local service. it is used to verify behaviors

  • when you get the test data from the real service provider, though from a test version of interface and get a test version of the object, you are working with stubs the stub can have logic to accept certain input and give corresponding output to help you perform state verification...


let see Test Doubles:

  • Fake: Fakes are objects that have working implementations, but not the same as production one. Such as: in-memory implementation of Data Access Object or Repository.
  • Stub: Stub is an object that holds predefined data and uses it to answer calls during tests. Such as: an object that needs to grab some data from the database to respond to a method call.

  • Mocks: Mocks are objects that register calls they receive. In test assertion, we can verify on Mocks that all expected actions were performed. Such as: a functionality that calls e-mail sending service. for more just check this.





stub