Tweet
Tweet

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 . Magento Test Automation: My Journey @fschmengler

Slide 3

Slide 3 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 . Magento Test Automation: My Journey @fschmengler

Slide 4

Slide 4 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 .. Magento Test Automation: My Journey @fschmengler

Slide 5

Slide 5 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ... Magento Test Automation: My Journey @fschmengler

Slide 6

Slide 6 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 .... SimpleTest? PHPUnit? Ibuildings MageTest? EcomDev PHPUnit? Magento Test Automation: My Journey @fschmengler

Slide 7

Slide 7 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ..... Magento Test Automation: My Journey @fschmengler

Slide 8

Slide 8 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ...... foo_bar? Foo_Bar? fooBar? foobar? bar? foo/bar? Magento Test Automation: My Journey @fschmengler

Slide 9

Slide 9 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ....... TEST ALL THE THINGS! public function testObserver() { $this->assertEventObserverDefined( 'adminhtml', 'core_block_abstract_prepare_layout_before', SGH_OrderPrintPopups_Model_Observer_Block::MODEL, 'onCoreBlockAbstractPrepareLayoutBefore', 'orderprintpopups' ); } Magento Test Automation: My Journey @fschmengler

Slide 10

Slide 10 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ........ TEST ALL THE THINGS! /** * @param Mage_Core_Block_Abstract $block * @return Varien_Event_Observer */ protected function _mockEventObserver($block, $otherArgs = array()) { $args = array_merge(array('block' => $block), $otherArgs); $observer = new Varien_Event_Observer(); $observer->setEvent(new Varien_Event($args)); return $observer; } Magento Test Automation: My Journey @fschmengler

Slide 11

Slide 11 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ......... TEST ALL THE THINGS! $this->assertLayoutHandleLoaded( 'orderprintpopups_warensendung_index', 'module layout handle loaded' ); $this->assertLayoutBlockRendered( SGH_OrderPrintPopups_Block_Popup_Warensendung::ALIAS, 'popup block rendered' ); Magento Test Automation: My Journey @fschmengler

Slide 12

Slide 12 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 .......... TEST ALL THE THINGS! $route = 'orderprintpopups/warensendung/barcodestatic'; $this->assertResponseBodyQueryRegex( '#sgh_warensendung', '#]*src="[^"]+' . $route . '#' ); Magento Test Automation: My Journey @fschmengler

Slide 13

Slide 13 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 .......... TEST ALL THE THINGS! /** * getter tests */ test("getter", function() { equal(this.block.getName(), "foo", "getName()"); equal(this.block.getContent(), "lorem ipsum falleri fallera", "getContent()"); equal(this.block.getPlaceholder(), "{foo}", "getPlaceholder()"); }); Magento Test Automation: My Journey @fschmengler

Slide 14

Slide 14 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............ What lies behind the code Hours of debugging Hidden depenencies to global state Bugs of the testing frameworks Magento Test Automation: My Journey @fschmengler

Slide 15

Slide 15 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); Magento Test Automation: My Journey @fschmengler

Slide 16

Slide 16 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); $this->markTestSkipped('conflict with fixture and existing db'); Magento Test Automation: My Journey @fschmengler

Slide 17

Slide 17 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); $this->markTestSkipped('conflict with fixture and existing db'); $this->markTestSkipped('"out of stock", although fixture seems ok'); //FIXME Magento Test Automation: My Journey @fschmengler

Slide 18

Slide 18 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); $this->markTestSkipped('conflict with fixture and existing db'); $this->markTestSkipped('"out of stock", although fixture seems ok'); //FIXME $this->markTestSkipped( 'The new comparator classes in PHPUnit 3.6 cannot handle recursive comparision with $maxDepth parameter YET.' ); Magento Test Automation: My Journey @fschmengler

Slide 19

Slide 19 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); $this->markTestSkipped('conflict with fixture and existing db'); $this->markTestSkipped('"out of stock", although fixture seems ok'); //FIXME $this->markTestSkipped( 'The new comparator classes in PHPUnit 3.6 cannot handle recursive comparision with $maxDepth parameter YET.' ); $this->markTestSkipped( 'expectation + fixture changed. reason for missing prefix unknown '. 'but only occurs in test environment!' ); Magento Test Automation: My Journey @fschmengler

Slide 20

Slide 20 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SS A story of skipped tests $this->markTestSkipped( 'Das klappt schon wieder alles nicht mit Adminlogin+Dispatch' ); Magento Test Automation: My Journey @fschmengler

Slide 21

Slide 21 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SS A story of skipped tests $this->markTestSkipped( 'Das klappt schon wieder alles nicht mit Adminlogin+Dispatch' ); $this->markTestSkipped('Untestbar in der Form. Mglw. ein Bug in EcomDev_PHPUnit der zu leerem orig_data führt. Leider funktioniert @runInSeparateProcess nicht mit Magento.') Magento Test Automation: My Journey @fschmengler

Slide 22

Slide 22 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSS Magento Test Automation: My Journey @fschmengler

Slide 23

Slide 23 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSF Magento 1.6 Update Magento Test Automation: My Journey @fschmengler

Slide 24

Slide 24 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF Magento Test Automation: My Journey @fschmengler

Slide 25

Slide 25 text

06/2011 12/2011 06/2012 12/2012 06/2013 12/2013 06/2014 0 50 100 150 200 250 300 350 400 450 Product Description Blocks one of the first modules LLOC LLOC (Tests) 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF. Magento Test Automation: My Journey @fschmengler

Slide 26

Slide 26 text

01/2012 07/2012 12/2012 0 50 100 150 200 250 300 350 Payment one year into the project... LLOC LLOC (Tests) 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S Magento Test Automation: My Journey @fschmengler

Slide 27

Slide 27 text

01/2012 07/2012 12/2012 0 500 1000 1500 2000 2500 3000 3500 Payment Symfony Service LLOC LLOC (Tests) 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S. Magento Test Automation: My Journey @fschmengler

Slide 28

Slide 28 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.. Diploma Thesis on Magento Development Magento Test Automation: My Journey @fschmengler

Slide 29

Slide 29 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S... Research & Experiments Built-in tests Magento TAF for Smoke Test Suite Abstraction of automated acceptance tests + ZiBreve Magento Test Automation: My Journey @fschmengler

Slide 30

Slide 30 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.... Selenium IDE > FitNesse Magento Test Automation: My Journey @fschmengler

Slide 31

Slide 31 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S..... Agency project with Selenium Automated Acceptance Tests Selenium IDE Tests for every feature Partially written by QA Magento Test Automation: My Journey @fschmengler

Slide 32

Slide 32 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S Magento Test Automation: My Journey @fschmengler

Slide 33

Slide 33 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S. Agency project with EcomDev_PHPUnit Automated Unit/Integration T ests Tests mandatory, coverage monitored Sophisticated setup Still typical problems Magento Test Automation: My Journey @fschmengler

Slide 34

Slide 34 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.F DEADLINE IS COMING Magento Test Automation: My Journey @fschmengler

Slide 35

Slide 35 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF M1 Integration Tests: Typical Problems Tests altering global state (singletons, registry) Magento Test Automation: My Journey @fschmengler

Slide 36

Slide 36 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF M1 Integration Tests: Typical Problems Tests altering global state (singletons, registry) Tests depending on existing data, others delete it and use their own Magento Test Automation: My Journey @fschmengler

Slide 37

Slide 37 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF M1 Integration Tests: Typical Problems Tests altering global state (singletons, registry) Tests depending on existing data, others delete it and use their own Tests not well isolated, not running in DB transaction Magento Test Automation: My Journey @fschmengler

Slide 38

Slide 38 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF M1 Integration Tests: Typical Problems Tests altering global state (singletons, registry) Tests depending on existing data, others delete it and use their own Tests not well isolated, not running in DB transaction Complicated fixture setup Magento Test Automation: My Journey @fschmengler

Slide 39

Slide 39 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF. THIS WEIRD UNIT TESTING TRICK BLEW MY MIND! Decouple business logic from Magento Use plain PHPUnit Magento Test Automation: My Journey @fschmengler

Slide 40

Slide 40 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF.. Magento Test Automation: My Journey @fschmengler

Slide 41

Slide 41 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF... Bad news we still need higher level tests Magento Test Automation: My Journey @fschmengler

Slide 42

Slide 42 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF... Bad news we still need higher level tests Good news less of them Magento Test Automation: My Journey @fschmengler

Slide 43

Slide 43 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF.... Magento Test Automation: My Journey @fschmengler

Slide 44

Slide 44 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF..... Experiments Acceptance and integration test framework Magento Test Automation: My Journey @fschmengler

Slide 45

Slide 45 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF..... Experiments Acceptance and integration test framework Acceptance test framework "THE SELENIUM-BASED TESTING FRAMEWORK FOR PEOPLE WHO HATE TESTING" Magento Test Automation: My Journey @fschmengler

Slide 46

Slide 46 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF..... Experiments Acceptance and integration test framework Acceptance test framework "THE SELENIUM-BASED TESTING FRAMEWORK FOR PEOPLE WHO HATE TESTING" Test framework Codeception Magento Test Automation: My Journey @fschmengler

Slide 47

Slide 47 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF...... A new era: Magento 2 Magento Test Automation: My Journey @fschmengler

Slide 48

Slide 48 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF....... Magento 2 Extension Reused refactored lib from Magento 1 extension Integration tests for Magento 2 specific code Magento Test Automation: My Journey @fschmengler

Slide 49

Slide 49 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........ Magento 2 Projects Great integration test framework Magento Test Automation: My Journey @fschmengler

Slide 50

Slide 50 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........ Magento 2 Projects Great integration test framework Easier to start with Magento Test Automation: My Journey @fschmengler

Slide 51

Slide 51 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........ Magento 2 Projects Great integration test framework Easier to start with More reliable Magento Test Automation: My Journey @fschmengler

Slide 52

Slide 52 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S Magento 2 Projects Still frustrating sometimes Magento Test Automation: My Journey @fschmengler

Slide 53

Slide 53 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S Magento 2 Projects Still frustrating sometimes Failures after minor version update Magento Test Automation: My Journey @fschmengler

Slide 54

Slide 54 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S Magento 2 Projects Still frustrating sometimes Failures after minor version update Test runs not completely isolated Magento Test Automation: My Journey @fschmengler

Slide 55

Slide 55 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S Magento 2 Projects Still frustrating sometimes Failures after minor version update Test runs not completely isolated Decoupled approach still works best Magento Test Automation: My Journey @fschmengler

Slide 56

Slide 56 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S. TDD Katas: Back to the roots Regular practice Small tasks, no real project Goal: Make TDD my default habit https://www.schmengler-se.de/katas Magento Test Automation: My Journey @fschmengler

Slide 57

Slide 57 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S.. How I would start today / Recommendations 1. For a quick win, create smoke test suite with Codeception Magento Test Automation: My Journey @fschmengler

Slide 58

Slide 58 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S.. How I would start today / Recommendations 1. For a quick win, create smoke test suite with Codeception 2. Katas / small self contained projects to learn TDD Magento Test Automation: My Journey @fschmengler

Slide 59

Slide 59 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S.. How I would start today / Recommendations 1. For a quick win, create smoke test suite with Codeception 2. Katas / small self contained projects to learn TDD 3. Start to build (small) parts of your code independent from Magento - "TDD" those parts Magento Test Automation: My Journey @fschmengler

Slide 60

Slide 60 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S.. How I would start today / Recommendations 1. For a quick win, create smoke test suite with Codeception 2. Katas / small self contained projects to learn TDD 3. Start to build (small) parts of your code independent from Magento - "TDD" those parts 4. Try to add simple integration tests (in M1 with Xtest) but skip if it's getting complicated Magento Test Automation: My Journey @fschmengler

Slide 61

Slide 61 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S... Mistakes I won't make again Try to sell tests as separate part of project quote or estimation Magento Test Automation: My Journey @fschmengler

Slide 62

Slide 62 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S... Mistakes I won't make again Try to sell tests as separate part of project quote or estimation Try to teach "unit testing" and "testing Magento" at once Magento Test Automation: My Journey @fschmengler

Slide 63

Slide 63 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S... Time: 40 minutes OK (42 slides, 0 cat pictures) Magento Test Automation: My Journey @fschmengler

Slide 64

Slide 64 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S... Time: 40 minutes OK (42 slides, 0 cat pictures) Thank you! Magento Test Automation: My Journey @fschmengler

Slide 65

Slide 65 text

2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S.....S.FF........S... Time: 40 minutes OK (42 slides, 0 cat pictures) Thank you! Questions? Magento Test Automation: My Journey @fschmengler

Slide 66

Slide 66 text

Magento Test Automation: My Journey @fschmengler

Slide 67

Slide 67 text

Announcement: MageTestFest NOVEMBER 17TH 2017 MAGENTO. SOFTWARE TESTING. PARTY. https://magetestfest.nl/ by Yireo. Magento Test Automation: My Journey @fschmengler

Slide 68

Slide 68 text

Links http://xtest-mage.com/ https://magiumlib.com/ http://codeception.com/ Image sources Marvin: The Hitchhiker's Guide to the Galaxy (2005 movie) Angry German: https://www.youtube.com/watch?v=-_xUIDRxdmc Black metal cat: https://twitter.com/evilbmcats Agile testing pyramid: http://www.agilecoachjournal.com Magento Test Automation: My Journey @fschmengler