Testing Fiber Optics with Ruby - a use case

A real use case of using Ruby, Cucumber, JIRA+Xray, for implementing tests for fiber optical equipments

Transcript

1. TESTING FIBER OPTICS WITH RUBY Sergio Freire
2. What?  GPON in 30 seconds  SUT  Challenges  The tools  Testing approach  Test scenario  Code (really?)  Results & Learned lessons 2
3. Ruby and Fiber Optics? Why not? 3
4. Terminology  GPON - Gigabit Passive Optical Network  OLT - Optical Line Terminal  ONT - Optical Network Terminal 4
5. GPON in 30 seconds Source: AlticeLabs 5
6. GPON in 30 seconds 6
7. The BIG challenge  Automate testing… for everything! 7
8. SUT – System Under Test  OLT + ONTs + … + … + ??? We need to simplify & focus! 8
9. SUT – System Under Test  SUT  OLT OS, running in a OLT But… there are several OLT types…. OLT1T3 OLT1T0 9
10. OLT - quick overview Allows up to 16K clients! OLT1T3 10
11. SUT – kind of  OLT is not just A hardware  Each card has its own hardware revision  OLT OS is a “mix” We need to get back to reality! 11
12. Some other challenges  “Resistence”  Manual Tests  QA vs Devel  Waterfall  Tools & Methodologies 12
13. Implementation Challenges  Build process not fully automated  Firmware uploading takes too much  OLT: up to 15min  ONTs: up to 15min, each  Scenario initialization just takes too loooong  We really needed Continuous Integration… 13
14. HOW WE DID IT
15. OLT 1T3 hardware OLT OS ONT SFU ONT RGW Traffic Generator “APIs”: • SNMP • CLI (telnet/ssh/rs232) • Web • XML (http) GBe Gbe Firmware repository (OLT, ONT) ONT xxx GBe … optical (OMCI) GBe FTP ethernet switch uplink LOGIC DIAGRAM
16. The Tools - before  HP Quality Center  JIRA (partially) 16
17. The Tools - after  JIRA + Xray add-on  Ruby & Cucumber  Jenkins 17
18. Ruby - a perfect choice  Simple yet powerfull  Many libraries available for free  snmp  TCL  SSH, Telnet, (S)FTP, …  HTTP  Logging  Etc, etc … 18
19. Approach  Focus on Regression Testing! 1. Start from the start 2. Share documentation & know-how 3. Identify core features/requirements 4. Make some ruby gems 5. Implement some automatic tests 6. Measure and *share* progress 7. Iterate Share and involve everyone! Testing makes everyone win. 19
20. Why regression?  Increasing complexity, short release cycles  Ensure *no surprises*  Easy to sell! Sometimes there is no time… 20
21. Time to get hands dirty  Use semi-automated prebuilt builds  Scenario initialization optimization  Don’t upload firmware if already there  Quick reset configuration  Not perfect but it works most times!  OLT equipment type and scenario completely configurable in YAML file  21
22. Ruby gems  Generic support functions  OLT basic interface  Files (ftp, …)  CLI (telnet/ssh)  Traffic Generator  TCL  => make some wrapping, pre-record traffic 22
23. Testing scenario Test Execution (Jenkins) OLT1T3 Many ONT RGW’s Traffic Generator SVN Build repository Firmware OLT, ONTs Tests source code •Execution evidences •Test Management •Requirement Coverage Many ONT SFU’s 23
24. Workflow Implement tests in Ruby + Cucumber Validate locally and commit to SVN Run the tests using Jenkins Import the results to JIRA + Xray Create and link tests + requirements 24
25. And now…  Where to start from?  Implement protocols from scratch? ? 25
26. Build an API in “minutes”  OLT CLI  typical commands: > /equipment/boards/show > /backup-manager/show > /backup-manager/import --param1=value1 … --paramN=valueN Ruby Metaprogramming to the rescue! 26
27. Build an API in “minutes”  Typical OLT CLI commands > /equipment/boards/show > /backup-manager/show > /backup-manager/import --param1=value1 … --paramN=valueN .show_equipment_boards .show_backupmanager .import_backupmanager({:param1 => value1, ..., :paramN => valueN}) 27
28. Build an API in “minutes” 28
29. Build an API in “minutes” {:title=>"onusprofiles", :data=> {"ID"=> {"1"=>{"name"=>"sfu", "admin"=>"enable", "vendor"=>"ptin", "model"=>"sfu", "pon"=>"1", "eth"=>"1", "rf"=>"1", "voip"=>"0", "e1"=>"0", "veip"=>"0"}, "2"=>{"name"=>"4ge-2fxs", "admin"=>"enable", "vendor"=>"ptin", "model"=>"4ge-2fxs", "pon"=>"1", "eth"=>"4", "rf"=>"1", "voip"=>"2", "e1"=>"0", "veip"=>"0"}, … }}}} Parsing answers 29
30. Cucumber step examples Given ONT in slot "<slot>" and pon "<port>" with id "<onuid>" is upgraded with version "<file_fw_version>“ And the OLT equipment does not have any services configured And a ONU profile exists with |state |name | vendor |model |pon|eth|rf|voip|e1|veip| |enabled |RGWs/rf |PTIN |RGWs/rf |1 |4 |0 |2 |0 |1 | |enabled |SFU_PTIN |PTIN |SFU |1 |1 |1 |0 |0 |0 | |enabled |teste |PTIN |teste |1 |1 |0 |0 |0 |0 | Then ONU present in slot "<slot>" and pon "<port>" with id "<onuid>" should have the version "<version_to_test>" commited and activated 30
31. Cucumber step examples (2) Given the OLT equipment does not have any services configured And an Ethernet interface exists in slot "<slot>", port "<port>" with mtu "<mtu>" and state "enabled“ When the traffic with the following characteristics is created in generator |stream_name|svlan |cvlan |src_MAC |dest_MAC |src_IP |dest_IP |rate |frame_size|downstream_port|upstream_port| |downstream1 |100 | |00:00:08:00:AA:01|00:00:08:00:AB:01|192.0.0.1 |192.0.1.1 |1000000|1000|1|2| |upstream1 |3100 |100 |00:00:08:00:AB:01|00:00:08:00:AA:01|192.0.1.1 |192.0.0.1 |1000000|1000|2|1| And the traffic configured in generator runs for "5" seconds Then the traffic stream named "downstream1" should have no losses 31
32. Results - Key numbers After 3 months 2 years later… Requirements validated 13 189 Automatic Tests 63 5143 Does it still seems impossible? No! 32
33. Results – Before and After 0 5 10 15 20 25 30 35 Time to execute regression tests Days Before After Average distinct tests executed per build Before After + 744 % Total regression tests executed Before After + 811 % 0 5 10 15 20 25 30 35 Effort need to execute regression testing Man Days Before After 33
34. Requirement coverage evolution 34
35. Results & Learned Lessons  Ruby is perfect! Good side effects:  Hardware stress tests  Testing team now mainly writes code  Brake human barriers, use APIs   LISTEN and involve everyone  Show how to fish  Make things that matter => real benefits!  Start simple, not perfect  Use the right tools (Ruby, Cucumber, Jenkins, JIRA, Xray add-on) 35
36. Ruby and Fiber Optics? Of course! 36
37. Q/A Thanks! Sergio Freire Xray Product Manager sergio.freire@xpand-it.com @darktelecom 37