to language specification ‣ Natural language ‣ Formal semantics ‣ Reference implementations ‣ Test suites ‣ Syntax specified by context-free grammars and regular expressions 5
to language specification ‣ Specification documents, e.g., ECMA-262 (JavaScript) ‣ Readily understood by most ‣ Free choice of content and format ‣ Cannot be automatically verified 6 Natural language
to language specification ‣ Can be difficult to read and write ‣ Rarely used for consumer software ‣ Most precise, unambiguous, provable ‣ Tool support, e.g, VDM-SL 7 Formal semantics
self-sufficient ‣ Written in a programming language ‣ Possible bugs & unexpected behavior ‣ Dependency on implementation details 8 Reference implementations Existing approaches to language specification
consists of tests written in the specified language ‣ Can define positive and negative test cases ‣ Should be executable by any implementation ‣ Nondeterministic behavior difficult to specify 9 Test suites Existing approaches to language specification
with greater abstraction ‣ No tool support for specification ‣ Visual tests not feasible 10 Visual programming languages Existing approaches to language specification
Example 12 From business goals to executable specification ‣ Introduced by Gojko Adzic ‣ Agile methodology ‣ Acceptance Test-Driven Development ‣ Behavior-Driven Development
Product Built Right Business Failure Success Useless Software Maintenance Difficulties test-driven development programming languages frameworks & tools specification by example project management Success of a software product, adapted from Adzic [1] business intent
Tests (or Acceptance Tests) Business Intent Usability Testing Is it pleasurable? Component Tests Architect Intent Exploratory Testing Is it self-consistent? Unit Tests Developer Intent Property Testing Is it responsive, secure, scalable? Per Functionality Cross-Functional Business Facing Technology Facing The testing matrix, adapted from Meszaros [2]
a ubiquitous language? A common language for a shared understanding of the domain. 17 E. Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software [3]
(one line describing the story) ! Narrative: As a [role] I want [feature] So that [benefit] ! Acceptance Criteria: (presented as Scenarios) Scenario: Title Given [context] And [some more context]… When [event] Then [outcome] And [another outcome]… D. North. What’s in a story? [4]
Calculator with Gherkin Feature: Addition Scenario: Adding two numbers Given I enter 1 into the calculator And I enter 2 into the calculator When I press add Then I should see 3 on the display 20
enter (d+) into the calculator$/ do |n| @stack.push n.to_i end 21 Then /^I should see (d+) on the display$/ do |n| @result.should == @result end When /^I press add^$/ do while not @stack.emtpy? do @result += @stack.pop end end
Features Scenarios Steps Step Definitions Support Code Automation Library System Business Facing Technology Facing The Cucumber testing stack, adapted from Hellesoy and Wynne [5]
Gherkin is simple but effective ‣ Steps are platform independent ‣ Definition methods are reusable ‣ Redundant code is reduced 23 Advantages of steps & step definitions
to Cucumber on mobile ‣ Open-source software by Xamarin Inc. ‣ Extends Ruby variant of Cucumber ‣ Supports Google Android & Apple iOS 25 E.g., Calabash
computer or build server Ruby client library & step definitions Cucumber features device or emulator Calabash HTTP server & native test runner Mobile Application Calabash system architecture
to Cucumber on mobile ‣ Ruby installation required ‣ Only pre-defined operations available ‣ Step-definitions are written in Ruby ‣ No integration with native libraries 27 Disadvantages
of the Android test framework application package (.apk) InstrumentationTestRunner test package (.apk) test tools monkey runner test case classes Instrumentation JUnit mock objects process
on Android ‣ Still no solution for Windows Phone and iOS ‣ Best option is to improve Cucumber ‣ Gherkin parser is based on Ragel ‣ Cucumber runtime can be ported to C 31 Remaining issue
to finished product ‣ Stories are imprecise ‣ Story cards are archived or thrown away ‣ No automatic validation ‣ Android implementation serves as the model for other teams 36 Problems
to finished product ‣ Translate stories into features and scenarios ‣ Share feature files with everybody ‣ Each team implements their own step- definitions independently 37 With Cucumber
to finished product ‣ Features and scenarios are precise ‣ Feature files are stored in SCM system (Git) ‣ Automated validation of specifications ‣ Implementation-agnostic model for all teams ‣ Better synchronization between teams 39 Advantages
has an Object ‘(\\w+)’$”) public void program_has_object(String name) { ProjectManager pm = ProjectManager.getInstance(); Sprite sprite = new Sprite(name); pm.getCurrentProject().addNewObject(sprite); Cucumber.put(Cucumber.KEY_CURRENT_OBJECT, sprite); } 43 Step definitions can require a mechanism to share state
loop (Repeat brick) Feature: Repeat brick A Repeat Brick should repeat another set of bricks a given number of times. ! Background: ... ! Scenario: Increment variable inside loop Given ‘Object’ has a Start script And this script has a set ‘i’ to 0 brick And this script has a Repeat 8 times brick And this script has a change ‘i’ by 1 brick And this script has a Repeat end brick When I start the program And I wait until the program has stopped Then the variable ‘i’ should be equal 8 46
loop (Repeat brick) Feature: Repeat brick A Repeat Brick should repeat another set of bricks a given number of times. ! Background: ... ! Scenario: Increment variable inside loop Given ‘Object’ has a Start script And this script has a set ‘i’ to 0 brick And this script has a Repeat 8 times brick And this script has a change ‘i’ by 1 brick And this script has a Repeat end brick When I start the program And I wait until the program has stopped Then the variable ‘i’ should be equal 8 47 Add script to Object for 8 times do i = i + 1; end
of a loop Scenario: No more than 100 iterations in 2 seconds Given ‘Object’ has a Start script And this script has a set ‘i’ to 0 brick And this script has a set ‘k’ to 0 brick And this script has a Wait 2 seconds brick And this script has a set ‘k’ to ‘i’ brick ! Given ‘Object’ has a Start script And this script has a Repeat 400 times brick And this script has a change ‘i’ by 1 brick And this script has a Repeat end brick ! When I start the program And I wait until the program has stopped Then the variable ‘k’ should be less than or equal 100 52
upper bound for iterations per second Scenario: No more than 100 iterations in 2 seconds Given ‘Object’ has a Start script And this script has a set ‘i’ to 0 brick And this script has a set ‘k’ to 0 brick And this script has a Wait 2 seconds brick And this script has a set ‘k’ to ‘i’ brick ! Given ‘Object’ has a Start script And this script has a Repeat 400 times brick And this script has a change ‘i’ by 1 brick And this script has a Repeat end brick ! When I start the program And I wait until the program has stopped Then the variable ‘k’ should be less than or equal 100 53 first Start script second Start script store value of i after 2 s. increment i by 1 1 loop should take at least 20 ms
control (Broadcast Wait) Scenario: Broadcast Wait brick sends message in program with two When scripts Given ‘Object’ has a Start script And this script has a BroadcastWait ‘hello’ brick And this script has a Print brick with ‘-S1-’ Given ‘Object’ has a When ‘hello’ script And this script has a Wait 100 milliseconds brick And this script has a Print brick with ‘-W1-’ Given ‘Object’ has a When ‘hello’ script And this script has a Wait 200 milliseconds brick And this script has a Print brick with ‘-W2-’ When I start the program And I wait until the program has stopped Then I should see the printed output ‘-W1--W2--S1-’ 54
control (Broadcast Wait) Scenario: Broadcast Wait brick sends message in program with two When scripts Given ‘Object’ has a Start script And this script has a BroadcastWait ‘hello’ brick And this script has a Print brick with ‘-S1-’ Given ‘Object‘ has a When ‘hello’ script And this script has a Wait 100 milliseconds brick And this script has a Print brick with ‘-W1-’ Given ‘Object’ has a When ‘hello’ script And this script has a Wait 200 milliseconds brick And this script has a Print brick with ‘-W2-’ When I start the program And I wait until the program has stopped Then I should see the printed output ‘-W1--W2--S1-’ 55 Start script initiates two When scripts first When script runs ≥ 100 ms second When script runs ≥ 200 ms output shows execution order of scripts
‣ Expected output of a program ‣ Object-oriented ‣ Properties of objects (attributes & methods) ‣ Hardware-oriented ‣ Comparable performance ‣ Measured values from sensors 56 Specifying semantics of a visual language
extends present test-driven approach ‣ New features can be developed inside existing project workflow ‣ Features and steps can be shared between teams working on different platforms 58 Advantages Conclusion
and scenarios need to be well- formulated ‣ When does a certain property correspond to a feature and when to a scenario? ‣ Step definitions need to be implemented per-platform and depend on existing code 59 Issues with Cucumber features Conclusion
Cucumber throughout all Catrobat sub-teams ‣ Port Cucumber to C or C++ ‣ Enhance development process with continuous integration & cucumber.pro 60 Future work Conclusion
1. G. Adzic. Specification by Example: How Successful Teams Deliver the Right Software. Manning Pubs Co Series. Manning, 2011. isbn: 9781617290084. 2. G. Meszaros. XUnit Test Patterns: Refactoring Test Code. A Martin Fowler signature book. Addison Wesley Professional, 2007. isbn: 9780131495050. 3. E. Evans. Domain-Driven Design: Tackling Complexity in the Heart of Software. Addison Wesley Professional, 2004. isbn: 9780321125217. 4. D. North. What’s in a story? Online; accessed 2013-10-05. url: http: //dannorth.net/whats-in-a-story. 5. A. Hellesoy and M. Wynne. The Cucumber Book: Behaviour-Driven Development for Testers and Developers. Pragmatic Programmers. Pragmatic Book- shelf, 2012. isbn: 9781934356807.
mfellner
tsoshiro
tsucchinoko
cer
qnighy
junk0612
harunatsujita
euxn23
masuda220
izumin5210
ivargrimstad
mikik0
riofujimon
danielanewman
aarron
jeffersonlam
deanohume
trallard
tammyeverts
panda_program
thoeni
moore
jlugia
shpigford
