ribbon

Dru is Data Reconstruction Utility which helps to create and maintain test data based on real-life production data as it is for example often easier to grab production data of web application as JSON than trying to create selective export from one or more data stores.

The quality of tests depends on the quality of the test data being used. It is important to keep the test data aligned with the production data as much as it is possible. This was relatively easy in the time of relational databases' dominance as test data can be set up with database dump but now, when the data required for the test can be stored in multiple data the safest way to load test data is to use your own data persistence layer or underlying framework. Dru comes with out of box support for Plain Old Java Objects (POJOs), GORM and AWS DynamoDB. It can consume JSON or YAML files as data sources.

Dru is designed to load complex data models. References by ids are translated into associations even the identity in newly created data store is not the same as original one. For example if you have entity Item with id 5 and entity ItemComment with property itemId with value 5 then the loaded ItemComment entity will have itemId property set to the actual id of the loaded Item e.g. 1.

Installation

Dru is available in JCenter. At the moment, you can use any of POJO, GORM or DynamoDB modules your project.

Gradle Installation
repositories {
    jcenter()
}

dependencies {
    // load just simple implementation with POJO client and reflection based parser
    testCompile "com.agorapulse:dru:0.1.0"

    // and pick any client
    testCompile "com.agorapulse:dru-client-gorm:0.1.0"
    testCompile "com.agorapulse:dru-client-dynamodb:0.1.0"

    // and pick any parser
    testCompile "com.agorapulse:dru-parser-json:0.1.0"
    testCompile "com.agorapulse:dru-parser-yaml:0.1.0"
}

Setup

Dru is a JUnit rule which ensures each of your test methods gets fresh data loaded.

Simple Specification
package avl

import com.agorapulse.dru.Dru
import grails.testing.gorm.DataTest
import org.junit.Rule
import spock.lang.Specification

/**
 * Test loading item.
 */
class ItemSpec extends Specification {


    @Rule Dru dru = Dru.plan {                                                          (1)
        from ('item.json') {                                                            (2)
            map { to Item }                                                             (3)
        }
    }

    void 'entities can be access from the data set'() {
        expect:
            dru.findAllByType(Item).size() == 1                                         (4)
        when:
            Item item = dru.findByTypeAndOriginalId(Item, ID)                           (5)
        then:
            item
            item.name == 'PX-41'                                                        (6)
            item.description == "The PX-41 is a very dangerous mutator engineered in the top secret PX-Labs, located in the Arctic Circle. It is capable of turning any living things in the world into a purple, furry, indestructible, mindless, killing machine that is so dangerous that it can destroy anything in its path."
            item.tags.contains('superpowers')
    }

    void 'all data are used'() {
        expect:
            dru.report.empty                                                            (7)
    }

    private static final String ID = '050e4fcf-158d-4f44-9b8b-a6ba6809982e:PX-41'
}
1 Prepare the data loading plan
2 Load the content of items.json
3 Map the root element to Item entity
4 Loaded entity is available by its type
5 Entity can be loaded by its original id
6 Properties are loaded as expected
7 Check whether all properties from the source has been used

You can take a look at the item.json file containing the test data:

item.json
{
  "id": "050e4fcf-158d-4f44-9b8b-a6ba6809982e",
  "name": "PX-41",
  "description": "The PX-41 is a very dangerous mutator engineered in the top secret PX-Labs, located in the Arctic Circle. It is capable of turning any living things in the world into a purple, furry, indestructible, mindless, killing machine that is so dangerous that it can destroy anything in its path.",
  "tags": [
    "mutator",
    "monsters",
    "superpowers"
  ]
}

The file must be located inside a folder of same name as the class where the source was defined i.g avl/ItemSpec/item.json, resp. src/test/resources/avl/ItemSpec/item.json for Gradle project.

Source Mapping

You can map directly to the root object or array or to any path inside the source you need:

Complex Path
@Rule Dru dru = Dru.plan {
    from ('items.json') {
        map ('mission.items') {
            to Item
        }
    }
}
items.json
{
  "mission": {
    "items": [
      {
        "id": "050e4fcf-158d-4f44-9b8b-a6ba6809982e",
        "name": "PX-41",
        "description": "The PX-41 is a very dangerous mutator engineered in the top secret PX-Labs, located in the Arctic Circle. It is capable of turning any living things in the world into a purple, furry, indestructible, mindless, killing machine that is so dangerous that it can destroy anything in its path.",
        "tags": [
          "mutator",
          "monsters",
          "superpowers"
        ]
      }
    ]
  }
}

Property and Type Mapping

For basic use cases when the source exactly fits the entity properties there is no need for additional mappings.

Default Values

You can set a default value for a property. The object passed as argument to the closure is the map obtained from the source.

Default Value
@Rule Dru dru = Dru.plan {
    from ('item.json') {
        map {
            to (Item) {
                defaults {
                    description = "Description for $it.name"
                }
            }
        }
    }
}

void 'entities can be access from the data set'() {
    when:
        Item item = dru.findByTypeAndOriginalId(Item, ID)
    then:
        item
        item.name == 'PX-41'
        item.description == 'Description for PX-41'
}
item.json
{
  "id": "050e4fcf-158d-4f44-9b8b-a6ba6809982e",
  "name": "PX-41",
  "tags": [
    "mutator",
    "monsters",
    "superpowers"
  ]
}

Overriding Properties

You can override any value coming from the source. The object passed as argument to the closure is the map obtained from the source. Contrary to defaults, the value is set to overridden value even it is present in the source.

Overriding Properties
@Rule Dru dru = Dru.plan {
    from ('item.json') {
        map {
            to (Item) {
                overrides {
                    description = "Description for $it.name"
                }
            }
        }
    }
}

void 'entities can be access from the data set'() {
    when:
        Item item = dru.findByTypeAndOriginalId(Item, ID)
    then:
        item
        item.name == 'PX-41'
        item.description == 'Description for PX-41'
}
item.json
{
  "id": "050e4fcf-158d-4f44-9b8b-a6ba6809982e",
  "name": "PX-41",
  "description": "The PX-41 is a very dangerous mutator engineered in the top secret PX-Labs, located in the Arctic Circle. It is capable of turning any living things in the world into a purple, furry, indestructible, mindless, killing machine that is so dangerous that it can destroy anything in its path.",
  "tags": [
    "mutator",
    "monsters",
    "superpowers"
  ]
}

Aliasing Properties

You can alias properties with different names in the source and in the entity.

Aliasing Properties
@Rule Dru dru = Dru.plan {
    from ('item.json') {
        map {
            to (Item) {
                map('desc') {
                    to (description: String)
                }
            }
        }
    }
}

void 'entities can be access from the data set'() {
    when:
        Item item = dru.findByTypeAndOriginalId(Item, ID)
    then:
        item
        item.name == 'PX-41'
        item.description == "The PX-41 is a very dangerous mutator engineered in the top secret PX-Labs, located in the Arctic Circle. It is capable of turning any living things in the world into a purple, furry, indestructible, mindless, killing machine that is so dangerous that it can destroy anything in its path."
        item.tags.contains('superpowers')
}
item.json
{
  "id": "050e4fcf-158d-4f44-9b8b-a6ba6809982e",
  "name": "PX-41",
  "desc": "The PX-41 is a very dangerous mutator engineered in the top secret PX-Labs, located in the Arctic Circle. It is capable of turning any living things in the world into a purple, furry, indestructible, mindless, killing machine that is so dangerous that it can destroy anything in its path.",
  "tags": [
    "mutator",
    "monsters",
    "superpowers"
  ]
}

Ignoring Properties

If you want to be sure that every information from the source is persisted you can access MissingPropertiesReport object from Dru instance. The report contains list of properties which hasn’t been matched. If you explicitly ignore a property for example because it is derived it will not appear in the report.

Ignoring Properties
@Rule Dru dru = Dru.plan {
    from ('item.json') {
        map {
            to (Item) {
                ignore 'owner'
            }
        }
    }
}

void 'owner does is not present in the report'() {
    when:
        dru.load()
    then:
        dru.report.empty
}
item.json
{
  "id": "050e4fcf-158d-4f44-9b8b-a6ba6809982e",
  "name": "PX-41",
  "description": "The PX-41 is a very dangerous mutator engineered in the top secret PX-Labs, located in the Arctic Circle. It is capable of turning any living things in the world into a purple, furry, indestructible, mindless, killing machine that is so dangerous that it can destroy anything in its path.",
  "owner": "Unknown",
  "tags": [
    "mutator",
    "monsters",
    "superpowers"
  ]
}

Conditional Type Mapping

You can add condition to type mappings to map to different entities based on source properties.

Conditional Mapping
@Rule Dru dru = Dru.plan {
    from ('persons.json') {
        map {
            to (Agent) {
                when { it.type == 'agent' }
                defaults { securityLevel = 1 }
            }
            to (Villain) {
                when { it.type == 'villain' }
            }
        }
    }
}

void 'entities are mapped to proper types'() {
    expect:
        dru.findAllByType(Agent).size() == 1
        dru.findAllByType(Villain).size() == 1
}
persons.json
[
  {
    "id": 12345,
    "name": "Felonius Gru",
    "bio": "Born from the family with long line of villainy and formerly the world's greatest villain.",
    "type": "agent"
  },
  {
    "id": 247,
    "name": "El Macho",
    "bio": "A former renowned, nearly superhuman-level strong bank robber who now wants to dominate the world",
    "type": "villain"
  }
]

Nested Type Mapping

You can nest type mapping to maps complex hierarchical structures.

Nested Mapping
@Rule Dru dru = Dru.plan {
    from ('agents.json') {
        map {
            to (Agent) {
                map ('manager') {
                    to (Agent) {
                        defaults { securityLevel = 1 }
                    }
                }
            }
        }
    }
}

void 'nested properties are mapped'() {
    expect:
        dru.findAllByType(Agent).size() == 2
        dru.findByTypeAndOriginalId(Agent, 12345).manager.name == 'Silas Ramsbottom'
}
agents.json
[
  {
    "id": 12345,
    "name": "Felonius Gru",
    "bio": "Born from the family with long line of villainy and formerly the world's greatest villain.",
    "securityLevel": 2,
    "manager": {
      "id": 101,
      "name": "Silas Ramsbottom"
    }
  }
]

You can declare the type mapping at top level so it applies to every occurrence of given type wherever in the tree:

Top Level Mapping
@Rule Dru reuse = Dru.plan {
    from ('agents.json') {
        map {
            to (Agent) {
                map ('manager') {
                    to (Agent)
                }
            }
        }
    }

    any (Agent) {
        defaults { securityLevel = 1 }
    }
}

Partial Retrieval

You can assign just a particular property of the loaded entity, usually an id.

Partial Retrieval
@Rule Dru dru = Dru.plan {
    from ('missionLogEntry.json') {
        map {
            to (MissionLogEntry) {
                map ('agent') {
                    to (agentId: Agent) {
                        just { id }
                        defaults {
                            securityLevel = 1
                        }
                    }
                }
            }
        }
    }
}

void 'mission log entry has agent id assigned'() {
    expect:
        dru.findByType(Agent)
        dru.findByType(MissionLogEntry).agentId == 1
}
missionLogEntry.json
{
  "mission": 7,
  "date": "2013-07-05T01:23:22Z",
  "type": "started",
  "description": "Mission started by Silas Ramsbottom",
  "agent": {
    "id": 101,
    "name": "Silas Ramsbottom"
  }
}

Data Sets

Data set is unit of reuse in Dru. Data set can contain multiple sources and mappings. The sources are evaluated relatively to the class in which the data set is defined. You usually defined one data set for mapping an entity and other to load the source to maximise reuse.

Agents Data Set
package avl

import com.agorapulse.dru.Dru
import com.agorapulse.dru.PreparedDataSet

/**
 * Agents data set.
 */
class AgentsDataSet {
    public static final PreparedDataSet agentsMapping = Dru.prepare {                   (1)
        any (Agent) {
            map ('manager') {
                to (Agent)
            }
            defaults {
                securityLevel = 1
            }
        }
    }

    public static final PreparedDataSet agents = Dru.prepare {                          (2)
        include agentsMapping                                                           (3)
        from ('agents.json') {
            map {
                to (Agent)
            }
        }
    }
}
1 Define data set for agents mapping
2 Define data set for agents data
3 Include data set for agents mapping

You can use method include to include any existing data set or you can use method load to load data set into existing data set.

Using Data Set
@Rule Dru dru = Dru.plan {
    include AgentsDataSet.agents
}

void 'agents get loaded from data set using prepare and include'() {
    expect:
        dru.findAllByType(Agent).size() == 2
        dru.findByTypeAndOriginalId(Agent, 12345).manager.name == 'Silas Ramsbottom'
}

void 'agents get loaded from data set using load'() {
    given:
        DataSet dataSet = Dru.steal(this).load(AgentsDataSet.agents)
    expect:
        dataSet.findAllByType(Agent).size() == 2
        dataSet.findByTypeAndOriginalId(Agent, 12345).manager.name == 'Silas Ramsbottom'
}

If additional logic needs to be executed when the data set is loaded or changed significantly then you can use whenLoaded hook. You can trigger the hooks manually using loaded method of the data set.

Using Data Set Hooks
@Rule Dru dru = Dru.plan {
    from ('AGENTS') {
        map {
            to (Agent) {
                map ('manager') {
                    to (Agent) {
                        defaults { securityLevel = 1 }
                    }
                }
            }
        }
    }
}

void 'calling when loaded hook'() {
    when:
        int count = 0
        dru.load {
            whenLoaded {
                count++
            }
        }
    then:
        count == 1                                                                  (1)
    when:
        dru.loaded()
    then:
        count == 2                                                                  (2)
}
1 First call to the hook is triggered immediately as we are defining the hook inside load method
2 Second call to the hook is triggered manually using loaded method

Parsers

Dru loads all parsers available on the classpath automatically. Which client is used is determined by the name of the source.

Reflection

Reflection parser is the simples parser. It searches for property of given name in the class where the data set is defined. This is a default parser if any other does not support given name.

Using Reflection Parser
private static final List<Map<String, Object>> AGENTS = [
    [
        id           : 12345,
        name         : 'Felonius Gru',
        bio          : 'Born from the family with long line of villainy and formerly the world\'s greatest villain.',
        securityLevel: 2,
        manager      : [
            id  : 101,
            name: 'Silas Ramsbottom'
        ]
    ]
]

@Rule Dru dru = Dru.plan {
    from ('AGENTS') {
        map {
            to (Agent) {
                map ('manager') {
                    to (Agent) {
                        defaults { securityLevel = 1 }
                    }
                }
            }
        }
    }
}

void 'nested properties are mapped'() {
    expect:
        dru.findAllByType(Agent).size() == 2
        dru.findByTypeAndOriginalId(Agent, 12345).manager.name == 'Silas Ramsbottom'
}

JSON

JSON parser parses JSON files to combination of maps and lists. The source files must end with .json to get parsed and they must be contained in directory with the same name as the reference class (unit test or data set)

Using JSON Parser
@Rule Dru dru = Dru.plan {
    from ('agents.json') {
        map {
            to (Agent) {
                map ('manager') {
                    to (Agent) {
                        defaults { securityLevel = 1 }
                    }
                }
            }
        }
    }
}

void 'nested properties are mapped'() {
    expect:
        dru.findAllByType(Agent).size() == 2
        dru.findByTypeAndOriginalId(Agent, 12345).manager.name == 'Silas Ramsbottom'
}
agents.json
[
  {
    "id": 12345,
    "name": "Felonius Gru",
    "bio": "Born from the family with long line of villainy and formerly the world's greatest villain.",
    "securityLevel": 2,
    "manager": {
      "id": 101,
      "name": "Silas Ramsbottom"
    }
  }
]

YAML

YAML parser parses YAML files to combination of maps and lists. The source files must end with .yml or .yaml to get parsed and they must be contained in directory with the same name as the reference class (unit test or data set)

Using YAML Parser
@Rule Dru dru = Dru.plan {
    from ('agents.yml') {
        map {
            to (Agent) {
                map ('manager') {
                    to (Agent) {
                        defaults { securityLevel = 1 }
                    }
                }
            }
        }
    }
}

void 'nested properties are mapped'() {
    expect:
        dru.findAllByType(Agent).size() == 2
        dru.findByTypeAndOriginalId(Agent, 12345).manager.name == 'Silas Ramsbottom'
}
agents.yml
- id: 12345
  name: Felonius Gru
  bio: Born from the family with long line of villainy and formerly the world's greatest
    villain.
  securityLevel: 2
  manager:
    id: 101
    name: Silas Ramsbottom

Clients

Dru loads all clients available on the classpath automatically if they support the unit test where Dru instance is defined.

POJO

POJO client is default fallback client which loads data into Plain Old Java Objects. POJO client is able to recognize associations but it is unable to load other sides of bidirectional relations.

Using POJO Client
@Rule Dru dru = Dru.plan {
    from ('library.json') {
        map {
            to (Library)
        }
    }
}

void 'library is loaded'() {
    expect:
        dru.findAllByType(Library).size() == 1
        dru.findAllByType(Book).size() == 2
}
library.json
{
  "name": "National Library",
  "books": [
    {
      "title": "It",
      "author": "Stephen King"
    },
    {
      "title": "Leviathan Wakes",
      "author": "James S. A. Corey"
    }
  ]
}

DynamoDB

DynamoDB client is extension to POJO client which understands DynamoDB data mapping annotations (see bellow). The client is used if @DynamoDBTable annotation is present on the class.

Table 1. DynamoDB Annotations
Annotation Effect

@DynamoDBTable

DynamoDB client is used for given class

@DynamoDBHashKey

Property is used as hash key part of the id

@DynamoDBRangeKey

Property is used as hash range part of the id

@DynamoDBIgnore

Property is ignored

@DynamoDBMarshalling

Property is marked as embedded

DynamoDB client determines the hash and range properly from the class so you can later retrieve the entity from the data set.

Using DynamoDB Client
@Rule Dru dru = Dru.plan {
    from ('missionLogEntry.json') {
        map {
            to MissionLogEntry
        }
    }
}

void 'mission log entry has agent id assigned'() {
    given:
        String id = DynamoDB.getOriginalId(7, '2013-07-05T01:23:22Z')
    expect:
        dru.findByType(MissionLogEntry)
        dru.findByTypeAndOriginalId(MissionLogEntry, id)
}
library.json
{
  "missionId": 7,
  "date": "2013-07-05T01:23:22Z",
  "type": "started",
  "description": "Mission started by Silas Ramsbottom",
  "agentId": 101
}
MissionLogEntry.groovy
@DynamoDBTable(tableName = "MissionLogEntry")
class MissionLogEntry {

    @DynamoDBHashKey
    Long missionId

    @DynamoDBRangeKey
    Date date

    MissionLogEntryType type

    String description

    @DynamoDBMarshalling(marshallerClass = ExtMarshaller)
    Map<String, Object> ext
}

You can create DynamoDBMapper based on the data in the data set using DynamoDB.createMapper(dataSet).

Using DynamoDBMapper
void 'use dynamodb mapper'() {
    when: "DynamoDB mapper is created from data set"
        DynamoDBMapper mapper = DynamoDB.createMapper(dru)
        Date date = new DateTime('2013-07-05T01:23:22Z').toDate()
        Long missionId = 7

    then: "loaded entities can be queried by this mapper"
        mapper.load(MissionLogEntry, missionId, date)
        mapper.load(new MissionLogEntry(missionId: missionId, date: date))
        mapper.query(MissionLogEntry,
            new DynamoDBQueryExpression<MissionLogEntry>().withHashKeyValues(new MissionLogEntry(missionId: missionId))
        ).size() == 1

    and: "the can be also deleted using this mapper"
        mapper.delete(mapper.load(new MissionLogEntry(missionId: missionId, date: date)))
        mapper.query(MissionLogEntry,
            new DynamoDBQueryExpression<MissionLogEntry>().withHashKeyValues(new MissionLogEntry(missionId: missionId))
        ).size() == 0

    when: "new entities are saved using this mapper"
        Date now = new Date()
        mapper.save(new MissionLogEntry(missionId: 7, date: now))

    then: "they are available in the data set"
        dru.findAllByType(MissionLogEntry).find { it.missionId == 7 && it.date == now}

}

If you are using Grails AWS SDK DynamoDB Plugin you can inject such DynamoDBMapper into AbstractDBService to get instance of the service working against the data set.

Using DynamoDBMapper with Grails Plugin
void 'use grails service'() {
    when:
        MissionLogEntryDBService service = new MissionLogEntryDBService()
        service.mapper = DynamoDB.createMapper(dru)
    then:
        service.query(7).count == 1
}

The Dru’s implementation of DynamoDBMapper provides limited query and scan capabilities. You can query by hash keys and range keys and you can scan with filter. For additional more complex queries you need to implement your own logic using DruDynamoDBMapper callback onQuery and onScan.

Using Avanced Queries and Scans
void 'advanced dynamodb mapper'() {
    when: "DynamoDB mapper is created from data set"
        DruDynamoDBMapper mapper = DynamoDB.createMapper(dru)
        mapper.onQuery(MissionLogEntry) { MissionLogEntry entry, DynamoDBQueryExpression<MissionLogEntry> query, DynamoDBMapperConfig config ->
            return entry.agentId == 101
        }
    then:
        mapper.query(MissionLogEntry, buildCompexQuery()).size() == 1

}

GORM

GORM uses the Grails Object Relational Mapping to import entities into test in-memory storage. It automatically mocks all the entities involved so there is no need to call to mockDomains method. Your unit tests must implement DataTest trait if you want to take advantage of using Dru with GORM.

Using GORM Client
@Rule Dru dru = Dru.plan {
    from ('agents.json') {
        map {
            to (Agent) {
                map ('manager') {
                    to (Agent)
                }
            }
        }
    }

    any (Agent) {
        defaults { securityLevel = 1 }
    }
}

void 'entities can be accessed from data set and using GORM methods'() {
    expect:
        dru.findAllByType(Agent).size() == 2
        dru.findByTypeAndOriginalId(Agent, 12345).manager.name == 'Silas Ramsbottom'
    and:
        Agent.count() == 2
        Agent.findByName('Silas Ramsbottom').id != 12345
}
agents.json
[
  {
    "id": 12345,
    "name": "Felonius Gru",
    "bio": "Born from the family with long line of villainy and formerly the world's greatest villain.",
    "securityLevel": 2,
    "manager": {
      "id": 101,
      "name": "Silas Ramsbottom"
    }
  }
]
Agent.groovy
class Agent extends Person implements WithSecurityLevel {
    String name
    String bio

    Long securityLevel

    static hasOne = [manager: Agent]

    static constraints = {
        securityLevel nullable: false
        bio nullable: true
    }
}
GORM client is unable to set the id of the entities to the original value. The original value is replaced wherever it is obvious from the mapping to the actual generated id.