$30 off During Our Annual Pro Sale. View Details »

Introduction to API Testing Automation by Postman

konifar
December 11, 2021

Introduction to API Testing Automation by Postman

ソフトウェアテスト自動化カンファレンス2021
https://testautomationresearch.connpass.com/event/228204/

konifar

December 11, 2021
Tweet

More Decks by konifar

Other Decks in Programming

Transcript

  1. Introduction to API Testing Automation by Postman STAR 2021 12/11

    (Sat) @konifar
  2. Agenda 1. Postman as a testing tool 2. CLI tool:

    Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool
  3. Agenda 1. Postman as a testing tool 2. CLI tool:

    Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool • What are Collection and Request? • How to write test code?
  4. Agenda 1. Postman as a testing tool 2. CLI tool:

    Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool • What is Newman? • How to run Newman?
  5. Agenda 1. Postman as a testing tool 2. CLI tool:

    Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool • Switching API endpoint by environment • Preparing before request • Solving API dependencies
  6. Agenda 1. Postman as a testing tool 2. CLI tool:

    Newman & GitHub Actions 3. Automation practices 4. Cons of Postman and alternative tool • Pros and cons of Postman • Introduction to Scenarigo
  7. Sample API: Pixela https://pixe.la/ Thanks for awesome API! @a_know

  8. 1. Postman as a testing tool

  9. Sending HTTP Request

  10. Sending HTTP Request Test cases Request Details Request Result

  11. Collection and Request Test cases Collection Folder Request

  12. Building Request Request Details Method, URL Headers, Body…

  13. Checking response Request Result

  14. Verifying response

  15. Test code snippets Verify Status code Verify JSON response Code

    snippets
  16. Test results

  17. 2. CLI tool: Newman & GitHub Actions

  18. Newman https://github.com/postmanlabs/newman ❯ npm install -g newman ❯ newman run

    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924 newman E2E Test ❏ POST /v1/users ↳ 200_success POST https://pixe.la/v1/users [200 OK, 518B, 118ms] ✓ Status code is 200 ✓ message is correct ✓ isSuccess is correct
  19. Install Newman ❯ npm install -g newman ❯ newman run

    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924 newman E2E Test ❏ POST /v1/users ↳ 200_success POST https://pixe.la/v1/users [200 OK, 518B, 118ms] ✓ Status code is 200 ✓ message is correct ✓ isSuccess is correct Install via npm
  20. Run Newman ❯ npm install -g newman ❯ newman run

    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924 newman E2E Test ❏ POST /v1/users ↳ 200_success POST https://pixe.la/v1/users [200 OK, 518B, 118ms] ✓ Status code is 200 ✓ message is correct ✓ isSuccess is correct Run requests in specific collection
  21. Run Newman ❯ npm install -g newman ❯ newman run

    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924 newman E2E Test ❏ POST /v1/users ↳ 200_success POST https://pixe.la/v1/users [200 OK, 518B, 118ms] ✓ Status code is 200 ✓ message is correct ✓ isSuccess is correct Show results. All tests are passed!
  22. # With API Key ❯ newman run https://www.getpostman.com/collections/{collection_id}?apikey={api_key} # Specify

    folder ❯ newman run https://www.getpostman.com/collections/{collection_id} --folder '{folder_name}' With API Key
  23. # With API Key ❯ newman run https://www.getpostman.com/collections/{collection_id}?apikey={api_key} # Specify

    folder ❯ newman run https://www.getpostman.com/collections/{collection_id} --folder '{folder_name}' Specify folder
  24. GitHub Actions workflow jobs: run: runs-on: ubuntu-latest steps: - uses:

    actions/checkout@v1 - uses: actions/setup-node@v1 - run: sudo npm install -g newman - name: Run newman run: newman run https://www.getpostman.com/collections/$POSTMAN_COLLECTION_ID env: POSTMAN_COLLECTION_ID: ${{ secrets.POSTMAN_COLLECTION_ID }} https://github.com/konifar/stac2021-newman-example/pull/1
  25. Using newman-action jobs: run: runs-on: ubuntu-latest steps: - uses: actions/checkout@v1

    - uses: matt-ball/newman-action@master with: apiKey: ${{ secrets.POSTMAN_API_KEY }} collection: ${{ secrets.POSTMAN_COLLECTION_ID }} environment: ${{ secrets.POSTMAN_ENVIRONMENT_ID }} https://github.com/matt-ball/newman-action
  26. 3. Automation practices

  27. 3 Practices 1. Preparing to run repeatedly 2. Generating data

    before request 3. Switching API URL by environment
  28. 1. Preparing to run repeatedly

  29. 1. Preparing to run repeatedly Failed when the request runs

    again because the user already exists.
  30. Solution1 Solution1: Create different user every time.

  31. Use dynamic variables https://learning.postman.com/docs/writing-scripts/script-references/variables-list/ Set unique username by using timestamp.

  32. Solution1 cons https://learning.postman.com/docs/writing-scripts/script-references/variables-list/ Many test users are created😰

  33. Solution2 Create another request which delete user in same Collection

    and run it before the same user is created.
  34. Run all requests in Collection

  35. Solution2 cons Depends on run order😰

  36. Solution3 Call user delete API in Pre-request Script I prefer

    this way.
  37. 2. Generating data before request

  38. 2. Generating data before request • Base64 encoding • Create

    random parameter • etc…
  39. Base64 encoding in Pre-request Script // Get env variables const

    clientId = pm.environment.get('VALID_CLIENT_ID'); const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded);
  40. Get environment variables // Get env variables const clientId =

    pm.environment.get('VALID_CLIENT_ID'); const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded); Defined env variables in Postman
  41. Base64 encoding // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID');

    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded);
  42. Collection variables // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID');

    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded);
  43. Collection variables // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID');

    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded); Check collection variable is set
  44. Collection variables // Get env variables const clientId = pm.environment.get('VALID_CLIENT_ID');

    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET'); // Base64 encoding const base64encoded = CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`)); console.log(`base64encoded: ${base64encoded}`); // Set value to collection variables pm.collectionVariables.set("base64encoded", base64encoded); Use variable in request
  45. 3. Switching API URL by environment

  46. Depends on test environment Different URLs for staging and prod

    Different usernames and tokens for each environment
  47. Define Environment Define variables for each environments. I prefer upper

    snake case style. https://learning.postman.com/docs/sending-requests/managing-environments/
  48. Use Environment in request

  49. Use Environment in script

  50. 4. Cons of Postman and alternative tool

  51. Pros of Postman 1. Simple GUI. No coding skills required.

    2. Easy to setup CI by Newman. 3. Apply JavaScript code.
  52. Cons of Postman 1. Can not reuse code in Pre-request

    scripts. 2. Difficult to maintain with several members. 3. Pricing.
  53. Cons of Postman 1. Can not reuse code in Pre-request

    scripts. 2. Difficult to maintain with several members. 3. Pricing. We may have to write many same code for authentication request in Pre-requests
  54. Cons of Postman 1. Can not reuse code in Pre-request

    scripts. 2. Difficult to maintain with several members. 3. Pricing. • Postman has some features like Fork, GitHub integration. • But it's hard to setup and difficult to review each other. • I prefer GitHub Pull Request driven development 🙂
  55. Cons of Postman 1. Can not reuse code in Pre-request

    scripts. 2. Difficult to maintain with several members. 3. Pricing. • free for up to 3 users. • $12 per user/month on Basic plan. • Not so expensive but it depends on the budget. https://www.postman.com/pricing/
  56. Scenarigo • A scenario based API testing tool written in

    Go. • Easy to write test scenarios as YAML file. • Enable to customize by writing Go plugin. https://github.com/zoncoen/scenarigo Thanks for awesome tool! @zoncoen
  57. Create User scenario yaml steps: - include: '../include/delete_user.yml' - protocol:

    http request: method: POST url: '{{env.SCENARIGO_BASE_URL}}/v1/users' body: token: '{{env.SCENARIGO_TEST_USER_TOKEN}}' username: '{{env.SCENARIGO_TEST_USER_NAME}}' agreeTermsOfService: 'yes' notMinor: 'yes' expect: code: 200 body: message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!" isSuccess: true
  58. Create User scenario yaml steps: - include: '../include/delete_user.yml' - protocol:

    http request: method: POST url: '{{env.SCENARIGO_BASE_URL}}/v1/users' body: token: '{{env.SCENARIGO_TEST_USER_TOKEN}}' username: '{{env.SCENARIGO_TEST_USER_NAME}}' agreeTermsOfService: 'yes' notMinor: 'yes' expect: code: 200 body: message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!" isSuccess: true Define delete user request in other yaml file. Same feature to Postman Pre-request Script.
  59. Create User scenario yaml steps: - include: '../include/delete_user.yml' - protocol:

    http request: method: POST url: '{{env.SCENARIGO_BASE_URL}}/v1/users' body: token: '{{env.SCENARIGO_TEST_USER_TOKEN}}' username: '{{env.SCENARIGO_TEST_USER_NAME}}' agreeTermsOfService: 'yes' notMinor: 'yes' expect: code: 200 body: message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!" isSuccess: true Define create user request. Same feature to Postman request. Environment variables are available.
  60. Create User scenario yaml steps: - include: '../include/delete_user.yml' - protocol:

    http request: method: POST url: '{{env.SCENARIGO_BASE_URL}}/v1/users' body: token: '{{env.SCENARIGO_TEST_USER_TOKEN}}' username: '{{env.SCENARIGO_TEST_USER_NAME}}' agreeTermsOfService: 'yes' notMinor: 'yes' expect: code: 200 body: message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!" isSuccess: true Verify response values here. Same feature to Postman tests.
  61. Run scenarigo ❯ scenarigo run ok scenarios/include/delete_user.yml 0.415s --- FAIL:

    scenarios/post_v1_users/200_success.yml (0.19s) … 10 | agreeTermsOfService: 'yes' 11 | notMinor: 'yes' 12 | expect: > 13 | code: 500 ^ … expected 500 but got OK FAIL FAIL scenarios/post_v1_users/200_success.yml 0.201s
  62. Postman & Scenarigo Postman Scenarigo Manage scenarios in… Postman GUI

    GitHub, GitLab, BitBucket… We can review in Pull Request! Define Request in… Postman GUI request block in YAML file Define Environment in… Postman Environment System Environment Generate data before request Pre-request Script Go Plugin Request before request as... Pre-request Script Include another YAML file We can reuse it! Define Test in… Tests expect block in YAML file https://github.com/konifar/stac2021-newman-example/pull/5/files
  63. Postman & Scenarigo Postman Scenarigo Manage scenarios in… Postman GUI

    GitHub, GitLab, BitBucket… We can review in Pull Request! Define Request in… Postman GUI request block in YAML file Define Environment in… Postman Environment System Environment Generate data before request Pre-request Script Go Plugin Request before request as... Pre-request Script Include another YAML file We can reuse it! Define Test in… Tests expect block in YAML file https://github.com/konifar/stac2021-newman-example/pull/5/files QA Team needs to learn YAML, Go, Git and GitHub. (We did it !)
  64. Summary

  65. Postman is awesome! • Simple GUI makes easy to set

    up API tests for especially non-developers. • Newman makes easy to run tests on CI.
  66. But Scenarigo is better for us! • We prefer Pull

    Request based development style • So we decided to migrate Postman to Scenarigo • Scenarigo covers major features of Postman
  67. Thanks for watching!

  68. Appendix • Sample code to run Newman and Scenarigo on

    GitHub Actions ◦ https://github.com/konifar/stac2021-newman-example • Sample API: Pixela ◦ https://pixe.la/ • Postman official docs ◦ https://learning.postman.com/docs • Newman ◦ https://github.com/postmanlabs/newman • Scenarigo ◦ https://github.com/zoncoen/scenarigo