Introduction to API Testing Automation by Postman

ソフトウェアテスト自動化カンファレンス2021
https://testautomationresearch.connpass.com/event/228204/

Transcript

1. Introduction to API Testing Automation by Postman STAR 2021 12/11 (Sat) @konifar
2. Agenda 1. Postman as a testing tool 2. CLI tool: Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool
3. Agenda 1. Postman as a testing tool 2. CLI tool: Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool • What are Collection and Request? • How to write test code?
4. Agenda 1. Postman as a testing tool 2. CLI tool: Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool • What is Newman? • How to run Newman?
5. Agenda 1. Postman as a testing tool 2. CLI tool: Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool • Switching API endpoint by environment • Preparing before request • Solving API dependencies
6. Agenda 1. Postman as a testing tool 2. CLI tool: Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool • Pros and cons of Postman • Introduction to Scenarigo
7. Sample API: Pixela https://pixe.la/ Thanks for awesome API! @a_know
8. 1. Postman as a testing tool
9. Sending HTTP Request
10. Sending HTTP Request Test cases Request Details Request Result
11. Collection and Request Test cases Collection Folder Request
12. Building Request Request Details Method, URL Headers, Body…
13. Checking response Request Result
14. Verifying response
15. Test code snippets Verify Status code Verify JSON response Code snippets
16. Test results
17. 2. CLI tool: Newman & GitHub Actions
18. Newman https://github.com/postmanlabs/newman ❯ npm install -g newman ❯ newman run https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924 newman E2E Test ❏ POST /v1/users ↳ 200_success POST https://pixe.la/v1/users [200 OK, 518B, 118ms] ✓ Status code is 200 ✓ message is correct ✓ isSuccess is correct
19. Install Newman ❯ npm install -g newman ❯ newman run https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924 newman E2E Test ❏ POST /v1/users ↳ 200_success POST https://pixe.la/v1/users [200 OK, 518B, 118ms] ✓ Status code is 200 ✓ message is correct ✓ isSuccess is correct Install via npm
20. Run Newman ❯ npm install -g newman ❯ newman run https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924 newman E2E Test ❏ POST /v1/users ↳ 200_success POST https://pixe.la/v1/users [200 OK, 518B, 118ms] ✓ Status code is 200 ✓ message is correct ✓ isSuccess is correct Run requests in specific collection
21. Run Newman ❯ npm install -g newman ❯ newman run https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924 newman E2E Test ❏ POST /v1/users ↳ 200_success POST https://pixe.la/v1/users [200 OK, 518B, 118ms] ✓ Status code is 200 ✓ message is correct ✓ isSuccess is correct Show results. All tests are passed!
22. # With API Key ❯ newman run https://www.getpostman.com/collections/{collection_id}?apikey={api_key} # Specify folder ❯ newman run https://www.getpostman.com/collections/{collection_id} --folder '{folder_name}' With API Key
23. # With API Key ❯ newman run https://www.getpostman.com/collections/{collection_id}?apikey={api_key} # Specify folder ❯ newman run https://www.getpostman.com/collections/{collection_id} --folder '{folder_name}' Specify folder
24. GitHub Actions workflow jobs: run: runs-on: ubuntu-latest steps: - uses: actions/checkout@v1 - uses: actions/setup-node@v1 - run: sudo npm install -g newman - name: Run newman run: newman run https://www.getpostman.com/collections/$POSTMAN_COLLECTION_ID env: POSTMAN_COLLECTION_ID: ${{ secrets.POSTMAN_COLLECTION_ID }} https://github.com/konifar/stac2021-newman-example/pull/1
25. Using newman-action jobs: run: runs-on: ubuntu-latest steps: - uses: actions/checkout@v1 - uses: matt-ball/newman-action@master with: apiKey: ${{ secrets.POSTMAN_API_KEY }} collection: ${{ secrets.POSTMAN_COLLECTION_ID }} environment: ${{ secrets.POSTMAN_ENVIRONMENT_ID }} https://github.com/matt-ball/newman-action
26. 3. Automation practices
27. 3 Practices 1. Preparing to run repeatedly 2. Generating data before request 3. Switching API URL by environment
28. 1. Preparing to run repeatedly
29. 1. Preparing to run repeatedly Failed when the request runs again because the user already exists.
30. Solution1 Solution1: Create different user every time.
31. Use dynamic variables https://learning.postman.com/docs/writing-scripts/script-references/variables-list/ Set unique username by using timestamp.
32. Solution1 cons https://learning.postman.com/docs/writing-scripts/script-references/variables-list/ Many test users are created😰
33. Solution2 Create another request which delete user in same Collection and run it before the same user is created.
34. Run all requests in Collection
35. Solution2 cons Depends on run order😰
36. Solution3 Call user delete API in Pre-request Script I prefer this way.
37. 2. Generating data before request
38. 2. Generating data before request • Base64 encoding • Create random parameter • etc…
39. Base64 encoding in Pre-request Script // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID'); const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded);
40. Get environment variables // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID'); const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded); Defined env variables in Postman
41. Base64 encoding // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID'); const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded);
42. Collection variables // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID'); const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded);
43. Collection variables // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID'); const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded); Check collection variable is set
44. Collection variables // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID'); const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded); Use variable in request
45. 3. Switching API URL by environment
46. Depends on test environment Different URLs for staging and prod Different usernames and tokens for each environment
47. Define Environment Define variables for each environments. I prefer upper snake case style. https://learning.postman.com/docs/sending-requests/managing-environments/
48. Use Environment in request
49. Use Environment in script
50. 4. Cons of Postman and alternative tool
51. Pros of Postman 1. Simple GUI. No coding skills required. 2. Easy to setup CI by Newman. 3. Apply JavaScript code.
52. Cons of Postman 1. Can not reuse code in Pre-request scripts. 2. Difficult to maintain with several members. 3. Pricing.
53. Cons of Postman 1. Can not reuse code in Pre-request scripts. 2. Difficult to maintain with several members. 3. Pricing. We may have to write many same code for authentication request in Pre-requests
54. Cons of Postman 1. Can not reuse code in Pre-request scripts. 2. Difficult to maintain with several members. 3. Pricing. • Postman has some features like Fork, GitHub integration. • But it's hard to setup and difficult to review each other. • I prefer GitHub Pull Request driven development 🙂
55. Cons of Postman 1. Can not reuse code in Pre-request scripts. 2. Difficult to maintain with several members. 3. Pricing. • free for up to 3 users. • $12 per user/month on Basic plan. • Not so expensive but it depends on the budget. https://www.postman.com/pricing/
56. Scenarigo • A scenario based API testing tool written in Go. • Easy to write test scenarios as YAML file. • Enable to customize by writing Go plugin. https://github.com/zoncoen/scenarigo Thanks for awesome tool! @zoncoen
57. Create User scenario yaml steps: - include: '../include/delete_user.yml' - protocol: http request: method: POST url: '{{env.SCENARIGO_BASE_URL}}/v1/users' body: token: '{{env.SCENARIGO_TEST_USER_TOKEN}}' username: '{{env.SCENARIGO_TEST_USER_NAME}}' agreeTermsOfService: 'yes' notMinor: 'yes' expect: code: 200 body: message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!" isSuccess: true
58. Create User scenario yaml steps: - include: '../include/delete_user.yml' - protocol: http request: method: POST url: '{{env.SCENARIGO_BASE_URL}}/v1/users' body: token: '{{env.SCENARIGO_TEST_USER_TOKEN}}' username: '{{env.SCENARIGO_TEST_USER_NAME}}' agreeTermsOfService: 'yes' notMinor: 'yes' expect: code: 200 body: message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!" isSuccess: true Define delete user request in other yaml file. Same feature to Postman Pre-request Script.
59. Create User scenario yaml steps: - include: '../include/delete_user.yml' - protocol: http request: method: POST url: '{{env.SCENARIGO_BASE_URL}}/v1/users' body: token: '{{env.SCENARIGO_TEST_USER_TOKEN}}' username: '{{env.SCENARIGO_TEST_USER_NAME}}' agreeTermsOfService: 'yes' notMinor: 'yes' expect: code: 200 body: message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!" isSuccess: true Define create user request. Same feature to Postman request. Environment variables are available.
60. Create User scenario yaml steps: - include: '../include/delete_user.yml' - protocol: http request: method: POST url: '{{env.SCENARIGO_BASE_URL}}/v1/users' body: token: '{{env.SCENARIGO_TEST_USER_TOKEN}}' username: '{{env.SCENARIGO_TEST_USER_NAME}}' agreeTermsOfService: 'yes' notMinor: 'yes' expect: code: 200 body: message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!" isSuccess: true Verify response values here. Same feature to Postman tests.
61. Run scenarigo ❯ scenarigo run ok scenarios/include/delete_user.yml 0.415s --- FAIL: scenarios/post_v1_users/200_success.yml (0.19s) … 10 | agreeTermsOfService: 'yes' 11 | notMinor: 'yes' 12 | expect: > 13 | code: 500 ^ … expected 500 but got OK FAIL FAIL scenarios/post_v1_users/200_success.yml 0.201s
62. Postman & Scenarigo Postman Scenarigo Manage scenarios in… Postman GUI GitHub, GitLab, BitBucket… We can review in Pull Request! Define Request in… Postman GUI request block in YAML file Define Environment in… Postman Environment System Environment Generate data before request Pre-request Script Go Plugin Request before request as... Pre-request Script Include another YAML file We can reuse it! Define Test in… Tests expect block in YAML file https://github.com/konifar/stac2021-newman-example/pull/5/files
63. Postman & Scenarigo Postman Scenarigo Manage scenarios in… Postman GUI GitHub, GitLab, BitBucket… We can review in Pull Request! Define Request in… Postman GUI request block in YAML file Define Environment in… Postman Environment System Environment Generate data before request Pre-request Script Go Plugin Request before request as... Pre-request Script Include another YAML file We can reuse it! Define Test in… Tests expect block in YAML file https://github.com/konifar/stac2021-newman-example/pull/5/files QA Team needs to learn YAML, Go, Git and GitHub. (We did it !)
64. Summary
65. Postman is awesome! • Simple GUI makes easy to set up API tests for especially non-developers. • Newman makes easy to run tests on CI.
66. But Scenarigo is better for us! • We prefer Pull Request based development style • So we decided to migrate Postman to Scenarigo • Scenarigo covers major features of Postman
67. Thanks for watching!
68. Appendix • Sample code to run Newman and Scenarigo on GitHub Actions ◦ https://github.com/konifar/stac2021-newman-example • Sample API: Pixela ◦ https://pixe.la/ • Postman official docs ◦ https://learning.postman.com/docs • Newman ◦ https://github.com/postmanlabs/newman • Scenarigo ◦ https://github.com/zoncoen/scenarigo