Introduction to API Testing Automation by Postman

ソフトウェアテスト自動化カンファレンス2021
https://testautomationresearch.connpass.com/event/228204/

Transcript

1. Introduction to
   API Testing Automation
   by Postman
   STAR 2021 12/11 (Sat)
   @konifar
   

   View Slide

2. Agenda
   1. Postman as a testing tool
   2. CLI tool: Newman & GitHub Actions
   3. Automation practices
   4. Cons of Postman and alternative tool
   

   View Slide

3. Agenda
   1. Postman as a testing tool
   2. CLI tool: Newman & GitHub Actions
   3. Automation practices
   4. Cons of Postman and alternative tool
   ● What are Collection and Request?
   ● How to write test code?
   

   View Slide

4. Agenda
   1. Postman as a testing tool
   2. CLI tool: Newman & GitHub Actions
   3. Automation practices
   4. Cons of Postman and alternative tool
   ● What is Newman?
   ● How to run Newman?
   

   View Slide

5. Agenda
   1. Postman as a testing tool
   2. CLI tool: Newman & GitHub Actions
   3. Automation practices
   4. Cons of Postman and alternative tool
   ● Switching API endpoint by environment
   ● Preparing before request
   ● Solving API dependencies
   

   View Slide

6. Agenda
   1. Postman as a testing tool
   2. CLI tool: Newman & GitHub Actions
   3. Automation practices
   4. Cons of Postman and alternative tool
   ● Pros and cons of Postman
   ● Introduction to Scenarigo
   

   View Slide

7. Sample API: Pixela
   https://pixe.la/
   Thanks for awesome API! @a_know
   

   View Slide

8. 1. Postman as a testing tool
   

   View Slide

9. Sending HTTP Request
   

   View Slide

10. Sending HTTP Request
    Test cases Request Details
    Request Result
    

    View Slide

11. Collection and Request
    Test cases
    Collection
    Folder
    Request
    

    View Slide

12. Building Request
    Request Details
    Method, URL
    Headers, Body…
    

    View Slide

13. Checking response
    Request Result
    

    View Slide

14. Verifying response
    

    View Slide

15. Test code snippets
    Verify Status
    code
    Verify JSON
    response
    Code snippets
    

    View Slide

16. Test results
    

    View Slide

17. 2. CLI tool: Newman & GitHub Actions
    

    View Slide

18. Newman
    https://github.com/postmanlabs/newman
    ❯ npm install -g newman
    ❯ newman run
    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924
    newman
    E2E Test
    ❏ POST /v1/users
    ↳ 200_success
    POST https://pixe.la/v1/users [200 OK, 518B, 118ms]
    ✓ Status code is 200
    ✓ message is correct
    ✓ isSuccess is correct
    

    View Slide

19. Install Newman
    ❯ npm install -g newman
    ❯ newman run
    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924
    newman
    E2E Test
    ❏ POST /v1/users
    ↳ 200_success
    POST https://pixe.la/v1/users [200 OK, 518B, 118ms]
    ✓ Status code is 200
    ✓ message is correct
    ✓ isSuccess is correct
    Install via npm
    

    View Slide

20. Run Newman
    ❯ npm install -g newman
    ❯ newman run
    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924
    newman
    E2E Test
    ❏ POST /v1/users
    ↳ 200_success
    POST https://pixe.la/v1/users [200 OK, 518B, 118ms]
    ✓ Status code is 200
    ✓ message is correct
    ✓ isSuccess is correct
    Run requests in specific collection
    

    View Slide

21. Run Newman
    ❯ npm install -g newman
    ❯ newman run
    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924
    newman
    E2E Test
    ❏ POST /v1/users
    ↳ 200_success
    POST https://pixe.la/v1/users [200 OK, 518B, 118ms]
    ✓ Status code is 200
    ✓ message is correct
    ✓ isSuccess is correct
    Show results. All tests are passed!
    

    View Slide

22. # With API Key
    ❯ newman run https://www.getpostman.com/collections/{collection_id}?apikey={api_key}
    # Specify folder
    ❯ newman run https://www.getpostman.com/collections/{collection_id} --folder '{folder_name}'
    With API Key
    

    View Slide

23. # With API Key
    ❯ newman run https://www.getpostman.com/collections/{collection_id}?apikey={api_key}
    # Specify folder
    ❯ newman run https://www.getpostman.com/collections/{collection_id} --folder '{folder_name}'
    Specify folder
    

    View Slide

24. GitHub Actions workflow
    jobs:
    run:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - uses: actions/setup-node@v1
    - run: sudo npm install -g newman
    - name: Run newman
    run: newman run https://www.getpostman.com/collections/$POSTMAN_COLLECTION_ID
    env:
    POSTMAN_COLLECTION_ID: ${{ secrets.POSTMAN_COLLECTION_ID }}
    https://github.com/konifar/stac2021-newman-example/pull/1
    

    View Slide

25. Using newman-action
    jobs:
    run:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - uses: matt-ball/newman-action@master
    with:
    apiKey: ${{ secrets.POSTMAN_API_KEY }}
    collection: ${{ secrets.POSTMAN_COLLECTION_ID }}
    environment: ${{ secrets.POSTMAN_ENVIRONMENT_ID }}
    https://github.com/matt-ball/newman-action
    

    View Slide

26. 3. Automation practices
    

    View Slide

27. 3 Practices
    1. Preparing to run repeatedly
    2. Generating data before request
    3. Switching API URL by environment
    

    View Slide

28. 1. Preparing to run repeatedly
    

    View Slide

29. 1. Preparing to run repeatedly
    Failed when the request runs again
    because the user already exists.
    

    View Slide

30. Solution1
    Solution1:
    Create different user every time.
    

    View Slide

31. Use dynamic variables
    https://learning.postman.com/docs/writing-scripts/script-references/variables-list/
    Set unique username by using timestamp.
    

    View Slide

32. Solution1 cons
    https://learning.postman.com/docs/writing-scripts/script-references/variables-list/
    Many test users are created😰
    

    View Slide

33. Solution2
    Create another request which delete user
    in same Collection and run it before the
    same user is created.
    

    View Slide

34. Run all requests in Collection
    

    View Slide

35. Solution2 cons
    Depends on run order😰
    

    View Slide

36. Solution3
    Call user delete API in Pre-request Script
    I prefer this way.
    

    View Slide

37. 2. Generating data before request
    

    View Slide

38. 2. Generating data before request
    ● Base64 encoding
    ● Create random parameter
    ● etc…
    

    View Slide

39. Base64 encoding in Pre-request Script
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);
    

    View Slide

40. Get environment variables
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);
    Defined env variables in Postman
    

    View Slide

41. Base64 encoding
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);
    

    View Slide

42. Collection variables
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);
    

    View Slide

43. Collection variables
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);
    Check collection variable is set
    

    View Slide

44. Collection variables
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);
    Use variable in request
    

    View Slide

45. 3. Switching API URL by environment
    

    View Slide

46. Depends on test environment
    Different URLs for staging and prod
    Different usernames and tokens for each environment
    

    View Slide

47. Define Environment
    Define variables for each environments.
    I prefer upper snake case style.
    https://learning.postman.com/docs/sending-requests/managing-environments/
    

    View Slide

48. Use Environment in request
    

    View Slide

49. Use Environment in script
    

    View Slide

50. 4. Cons of Postman and alternative tool
    

    View Slide

51. Pros of Postman
    1. Simple GUI. No coding skills required.
    2. Easy to setup CI by Newman.
    3. Apply JavaScript code.
    

    View Slide

52. Cons of Postman
    1. Can not reuse code in Pre-request scripts.
    2. Difficult to maintain with several members.
    3. Pricing.
    

    View Slide

53. Cons of Postman
    1. Can not reuse code in Pre-request scripts.
    2. Difficult to maintain with several members.
    3. Pricing.
    We may have to write many same code
    for authentication request in Pre-requests
    

    View Slide

54. Cons of Postman
    1. Can not reuse code in Pre-request scripts.
    2. Difficult to maintain with several members.
    3. Pricing.
    ● Postman has some features like Fork, GitHub integration.
    ● But it's hard to setup and difficult to review each other.
    ● I prefer GitHub Pull Request driven development 🙂
    

    View Slide

55. Cons of Postman
    1. Can not reuse code in Pre-request scripts.
    2. Difficult to maintain with several members.
    3. Pricing.
    ● free for up to 3 users.
    ● $12 per user/month on Basic plan.
    ● Not so expensive but it depends on the budget.
    https://www.postman.com/pricing/
    

    View Slide

56. Scenarigo
    ● A scenario based API testing tool written in Go.
    ● Easy to write test scenarios as YAML file.
    ● Enable to customize by writing Go plugin.
    https://github.com/zoncoen/scenarigo
    Thanks for awesome tool! @zoncoen
    

    View Slide

57. Create User scenario yaml
    steps:
    - include: '../include/delete_user.yml'
    - protocol: http
    request:
    method: POST
    url: '{{env.SCENARIGO_BASE_URL}}/v1/users'
    body:
    token: '{{env.SCENARIGO_TEST_USER_TOKEN}}'
    username: '{{env.SCENARIGO_TEST_USER_NAME}}'
    agreeTermsOfService: 'yes'
    notMinor: 'yes'
    expect:
    code: 200
    body:
    message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!"
    isSuccess: true
    

    View Slide

58. Create User scenario yaml
    steps:
    - include: '../include/delete_user.yml'
    - protocol: http
    request:
    method: POST
    url: '{{env.SCENARIGO_BASE_URL}}/v1/users'
    body:
    token: '{{env.SCENARIGO_TEST_USER_TOKEN}}'
    username: '{{env.SCENARIGO_TEST_USER_NAME}}'
    agreeTermsOfService: 'yes'
    notMinor: 'yes'
    expect:
    code: 200
    body:
    message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!"
    isSuccess: true
    Define delete user request in other yaml file.
    Same feature to Postman Pre-request Script.
    

    View Slide

59. Create User scenario yaml
    steps:
    - include: '../include/delete_user.yml'
    - protocol: http
    request:
    method: POST
    url: '{{env.SCENARIGO_BASE_URL}}/v1/users'
    body:
    token: '{{env.SCENARIGO_TEST_USER_TOKEN}}'
    username: '{{env.SCENARIGO_TEST_USER_NAME}}'
    agreeTermsOfService: 'yes'
    notMinor: 'yes'
    expect:
    code: 200
    body:
    message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!"
    isSuccess: true
    Define create user request.
    Same feature to Postman request.
    Environment variables are available.
    

    View Slide

60. Create User scenario yaml
    steps:
    - include: '../include/delete_user.yml'
    - protocol: http
    request:
    method: POST
    url: '{{env.SCENARIGO_BASE_URL}}/v1/users'
    body:
    token: '{{env.SCENARIGO_TEST_USER_TOKEN}}'
    username: '{{env.SCENARIGO_TEST_USER_NAME}}'
    agreeTermsOfService: 'yes'
    notMinor: 'yes'
    expect:
    code: 200
    body:
    message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!"
    isSuccess: true
    Verify response values here.
    Same feature to Postman tests.
    

    View Slide

61. Run scenarigo
    ❯ scenarigo run
    ok scenarios/include/delete_user.yml 0.415s
    --- FAIL: scenarios/post_v1_users/200_success.yml (0.19s)
    …
    10 | agreeTermsOfService: 'yes'
    11 | notMinor: 'yes'
    12 | expect:
    > 13 | code: 500
    ^
    …
    expected 500 but got OK
    FAIL
    FAIL scenarios/post_v1_users/200_success.yml 0.201s
    

    View Slide

62. Postman & Scenarigo
    Postman Scenarigo
    Manage scenarios in… Postman GUI
    GitHub, GitLab, BitBucket…
    We can review in Pull Request!
    Define Request in… Postman GUI request block in YAML file
    Define Environment in… Postman Environment System Environment
    Generate data before request Pre-request Script Go Plugin
    Request before request as... Pre-request Script
    Include another YAML file
    We can reuse it!
    Define Test in… Tests expect block in YAML file
    https://github.com/konifar/stac2021-newman-example/pull/5/files
    

    View Slide

63. Postman & Scenarigo
    Postman Scenarigo
    Manage scenarios in… Postman GUI
    GitHub, GitLab, BitBucket…
    We can review in Pull Request!
    Define Request in… Postman GUI request block in YAML file
    Define Environment in… Postman Environment System Environment
    Generate data before request Pre-request Script Go Plugin
    Request before request as... Pre-request Script
    Include another YAML file
    We can reuse it!
    Define Test in… Tests expect block in YAML file
    https://github.com/konifar/stac2021-newman-example/pull/5/files
    QA Team needs to learn YAML,
    Go, Git and GitHub. (We did it !)
    

    View Slide

64. Summary
    

    View Slide

65. Postman is awesome!
    ● Simple GUI makes easy to set up API tests
    for especially non-developers.
    ● Newman makes easy to run tests on CI.
    

    View Slide

66. But Scenarigo is better for us!
    ● We prefer Pull Request based development style
    ● So we decided to migrate Postman to Scenarigo
    ● Scenarigo covers major features of Postman
    

    View Slide

67. Thanks for watching!
    

    View Slide

68. Appendix
    ● Sample code to run Newman and Scenarigo on GitHub Actions
    ○ https://github.com/konifar/stac2021-newman-example
    ● Sample API: Pixela
    ○ https://pixe.la/
    ● Postman official docs
    ○ https://learning.postman.com/docs
    ● Newman
    ○ https://github.com/postmanlabs/newman
    ● Scenarigo
    ○ https://github.com/zoncoen/scenarigo
    

    View Slide