Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to API Testing Automation by Postman

konifar
December 11, 2021

Introduction to API Testing Automation by Postman

ソフトウェアテスト自動化カンファレンス2021
https://testautomationresearch.connpass.com/event/228204/

konifar

December 11, 2021
Tweet

More Decks by konifar

Other Decks in Programming

Transcript

  1. Introduction to
    API Testing Automation
    by Postman
    STAR 2021 12/11 (Sat)
    @konifar

    View full-size slide

  2. Agenda
    1. Postman as a testing tool
    2. CLI tool: Newman & GitHub Actions
    3. Automation practices
    4. Cons of Postman and alternative tool

    View full-size slide

  3. Agenda
    1. Postman as a testing tool
    2. CLI tool: Newman & GitHub Actions
    3. Automation practices
    4. Cons of Postman and alternative tool
    ● What are Collection and Request?
    ● How to write test code?

    View full-size slide

  4. Agenda
    1. Postman as a testing tool
    2. CLI tool: Newman & GitHub Actions
    3. Automation practices
    4. Cons of Postman and alternative tool
    ● What is Newman?
    ● How to run Newman?

    View full-size slide

  5. Agenda
    1. Postman as a testing tool
    2. CLI tool: Newman & GitHub Actions
    3. Automation practices
    4. Cons of Postman and alternative tool
    ● Switching API endpoint by environment
    ● Preparing before request
    ● Solving API dependencies

    View full-size slide

  6. Agenda
    1. Postman as a testing tool
    2. CLI tool: Newman & GitHub Actions
    3. Automation practices
    4. Cons of Postman and alternative tool
    ● Pros and cons of Postman
    ● Introduction to Scenarigo

    View full-size slide

  7. Sample API: Pixela
    https://pixe.la/
    Thanks for awesome API! @a_know

    View full-size slide

  8. 1. Postman as a testing tool

    View full-size slide

  9. Sending HTTP Request

    View full-size slide

  10. Sending HTTP Request
    Test cases Request Details
    Request Result

    View full-size slide

  11. Collection and Request
    Test cases
    Collection
    Folder
    Request

    View full-size slide

  12. Building Request
    Request Details
    Method, URL
    Headers, Body…

    View full-size slide

  13. Checking response
    Request Result

    View full-size slide

  14. Verifying response

    View full-size slide

  15. Test code snippets
    Verify Status
    code
    Verify JSON
    response
    Code snippets

    View full-size slide

  16. Test results

    View full-size slide

  17. 2. CLI tool: Newman & GitHub Actions

    View full-size slide

  18. Newman
    https://github.com/postmanlabs/newman
    ❯ npm install -g newman
    ❯ newman run
    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924
    newman
    E2E Test
    ❏ POST /v1/users
    ↳ 200_success
    POST https://pixe.la/v1/users [200 OK, 518B, 118ms]
    ✓ Status code is 200
    ✓ message is correct
    ✓ isSuccess is correct

    View full-size slide

  19. Install Newman
    ❯ npm install -g newman
    ❯ newman run
    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924
    newman
    E2E Test
    ❏ POST /v1/users
    ↳ 200_success
    POST https://pixe.la/v1/users [200 OK, 518B, 118ms]
    ✓ Status code is 200
    ✓ message is correct
    ✓ isSuccess is correct
    Install via npm

    View full-size slide

  20. Run Newman
    ❯ npm install -g newman
    ❯ newman run
    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924
    newman
    E2E Test
    ❏ POST /v1/users
    ↳ 200_success
    POST https://pixe.la/v1/users [200 OK, 518B, 118ms]
    ✓ Status code is 200
    ✓ message is correct
    ✓ isSuccess is correct
    Run requests in specific collection

    View full-size slide

  21. Run Newman
    ❯ npm install -g newman
    ❯ newman run
    https://www.getpostman.com/collections/18507875-41095035-c383-49ab-8553-a353132a1924
    newman
    E2E Test
    ❏ POST /v1/users
    ↳ 200_success
    POST https://pixe.la/v1/users [200 OK, 518B, 118ms]
    ✓ Status code is 200
    ✓ message is correct
    ✓ isSuccess is correct
    Show results. All tests are passed!

    View full-size slide

  22. # With API Key
    ❯ newman run https://www.getpostman.com/collections/{collection_id}?apikey={api_key}
    # Specify folder
    ❯ newman run https://www.getpostman.com/collections/{collection_id} --folder '{folder_name}'
    With API Key

    View full-size slide

  23. # With API Key
    ❯ newman run https://www.getpostman.com/collections/{collection_id}?apikey={api_key}
    # Specify folder
    ❯ newman run https://www.getpostman.com/collections/{collection_id} --folder '{folder_name}'
    Specify folder

    View full-size slide

  24. GitHub Actions workflow
    jobs:
    run:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - uses: actions/setup-node@v1
    - run: sudo npm install -g newman
    - name: Run newman
    run: newman run https://www.getpostman.com/collections/$POSTMAN_COLLECTION_ID
    env:
    POSTMAN_COLLECTION_ID: ${{ secrets.POSTMAN_COLLECTION_ID }}
    https://github.com/konifar/stac2021-newman-example/pull/1

    View full-size slide

  25. Using newman-action
    jobs:
    run:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - uses: matt-ball/newman-action@master
    with:
    apiKey: ${{ secrets.POSTMAN_API_KEY }}
    collection: ${{ secrets.POSTMAN_COLLECTION_ID }}
    environment: ${{ secrets.POSTMAN_ENVIRONMENT_ID }}
    https://github.com/matt-ball/newman-action

    View full-size slide

  26. 3. Automation practices

    View full-size slide

  27. 3 Practices
    1. Preparing to run repeatedly
    2. Generating data before request
    3. Switching API URL by environment

    View full-size slide

  28. 1. Preparing to run repeatedly

    View full-size slide

  29. 1. Preparing to run repeatedly
    Failed when the request runs again
    because the user already exists.

    View full-size slide

  30. Solution1
    Solution1:
    Create different user every time.

    View full-size slide

  31. Use dynamic variables
    https://learning.postman.com/docs/writing-scripts/script-references/variables-list/
    Set unique username by using timestamp.

    View full-size slide

  32. Solution1 cons
    https://learning.postman.com/docs/writing-scripts/script-references/variables-list/
    Many test users are created😰

    View full-size slide

  33. Solution2
    Create another request which delete user
    in same Collection and run it before the
    same user is created.

    View full-size slide

  34. Run all requests in Collection

    View full-size slide

  35. Solution2 cons
    Depends on run order😰

    View full-size slide

  36. Solution3
    Call user delete API in Pre-request Script
    I prefer this way.

    View full-size slide

  37. 2. Generating data before request

    View full-size slide

  38. 2. Generating data before request
    ● Base64 encoding
    ● Create random parameter
    ● etc…

    View full-size slide

  39. Base64 encoding in Pre-request Script
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);

    View full-size slide

  40. Get environment variables
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);
    Defined env variables in Postman

    View full-size slide

  41. Base64 encoding
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);

    View full-size slide

  42. Collection variables
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);

    View full-size slide

  43. Collection variables
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);
    Check collection variable is set

    View full-size slide

  44. Collection variables
    // Get env variables
    const clientId = pm.environment.get('VALID_CLIENT_ID');
    const clientSecret = pm.environment.get('VALID_CLIENT_SECRET');
    // Base64 encoding
    const base64encoded =
    CryptoJS.enc.Base64.stringify(CryptoJS.enc.Utf8.parse(`${clientId}:${clientSecret}`));
    console.log(`base64encoded: ${base64encoded}`);
    // Set value to collection variables
    pm.collectionVariables.set("base64encoded", base64encoded);
    Use variable in request

    View full-size slide

  45. 3. Switching API URL by environment

    View full-size slide

  46. Depends on test environment
    Different URLs for staging and prod
    Different usernames and tokens for each environment

    View full-size slide

  47. Define Environment
    Define variables for each environments.
    I prefer upper snake case style.
    https://learning.postman.com/docs/sending-requests/managing-environments/

    View full-size slide

  48. Use Environment in request

    View full-size slide

  49. Use Environment in script

    View full-size slide

  50. 4. Cons of Postman and alternative tool

    View full-size slide

  51. Pros of Postman
    1. Simple GUI. No coding skills required.
    2. Easy to setup CI by Newman.
    3. Apply JavaScript code.

    View full-size slide

  52. Cons of Postman
    1. Can not reuse code in Pre-request scripts.
    2. Difficult to maintain with several members.
    3. Pricing.

    View full-size slide

  53. Cons of Postman
    1. Can not reuse code in Pre-request scripts.
    2. Difficult to maintain with several members.
    3. Pricing.
    We may have to write many same code
    for authentication request in Pre-requests

    View full-size slide

  54. Cons of Postman
    1. Can not reuse code in Pre-request scripts.
    2. Difficult to maintain with several members.
    3. Pricing.
    ● Postman has some features like Fork, GitHub integration.
    ● But it's hard to setup and difficult to review each other.
    ● I prefer GitHub Pull Request driven development 🙂

    View full-size slide

  55. Cons of Postman
    1. Can not reuse code in Pre-request scripts.
    2. Difficult to maintain with several members.
    3. Pricing.
    ● free for up to 3 users.
    ● $12 per user/month on Basic plan.
    ● Not so expensive but it depends on the budget.
    https://www.postman.com/pricing/

    View full-size slide

  56. Scenarigo
    ● A scenario based API testing tool written in Go.
    ● Easy to write test scenarios as YAML file.
    ● Enable to customize by writing Go plugin.
    https://github.com/zoncoen/scenarigo
    Thanks for awesome tool! @zoncoen

    View full-size slide

  57. Create User scenario yaml
    steps:
    - include: '../include/delete_user.yml'
    - protocol: http
    request:
    method: POST
    url: '{{env.SCENARIGO_BASE_URL}}/v1/users'
    body:
    token: '{{env.SCENARIGO_TEST_USER_TOKEN}}'
    username: '{{env.SCENARIGO_TEST_USER_NAME}}'
    agreeTermsOfService: 'yes'
    notMinor: 'yes'
    expect:
    code: 200
    body:
    message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!"
    isSuccess: true

    View full-size slide

  58. Create User scenario yaml
    steps:
    - include: '../include/delete_user.yml'
    - protocol: http
    request:
    method: POST
    url: '{{env.SCENARIGO_BASE_URL}}/v1/users'
    body:
    token: '{{env.SCENARIGO_TEST_USER_TOKEN}}'
    username: '{{env.SCENARIGO_TEST_USER_NAME}}'
    agreeTermsOfService: 'yes'
    notMinor: 'yes'
    expect:
    code: 200
    body:
    message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!"
    isSuccess: true
    Define delete user request in other yaml file.
    Same feature to Postman Pre-request Script.

    View full-size slide

  59. Create User scenario yaml
    steps:
    - include: '../include/delete_user.yml'
    - protocol: http
    request:
    method: POST
    url: '{{env.SCENARIGO_BASE_URL}}/v1/users'
    body:
    token: '{{env.SCENARIGO_TEST_USER_TOKEN}}'
    username: '{{env.SCENARIGO_TEST_USER_NAME}}'
    agreeTermsOfService: 'yes'
    notMinor: 'yes'
    expect:
    code: 200
    body:
    message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!"
    isSuccess: true
    Define create user request.
    Same feature to Postman request.
    Environment variables are available.

    View full-size slide

  60. Create User scenario yaml
    steps:
    - include: '../include/delete_user.yml'
    - protocol: http
    request:
    method: POST
    url: '{{env.SCENARIGO_BASE_URL}}/v1/users'
    body:
    token: '{{env.SCENARIGO_TEST_USER_TOKEN}}'
    username: '{{env.SCENARIGO_TEST_USER_NAME}}'
    agreeTermsOfService: 'yes'
    notMinor: 'yes'
    expect:
    code: 200
    body:
    message: "Success. Let's visit {{env.SCENARIGO_BASE_URL}}/@{{env.SCENARIGO_TEST_USER_NAME}} , it is your profile page!"
    isSuccess: true
    Verify response values here.
    Same feature to Postman tests.

    View full-size slide

  61. Run scenarigo
    ❯ scenarigo run
    ok scenarios/include/delete_user.yml 0.415s
    --- FAIL: scenarios/post_v1_users/200_success.yml (0.19s)

    10 | agreeTermsOfService: 'yes'
    11 | notMinor: 'yes'
    12 | expect:
    > 13 | code: 500
    ^

    expected 500 but got OK
    FAIL
    FAIL scenarios/post_v1_users/200_success.yml 0.201s

    View full-size slide

  62. Postman & Scenarigo
    Postman Scenarigo
    Manage scenarios in… Postman GUI
    GitHub, GitLab, BitBucket…
    We can review in Pull Request!
    Define Request in… Postman GUI request block in YAML file
    Define Environment in… Postman Environment System Environment
    Generate data before request Pre-request Script Go Plugin
    Request before request as... Pre-request Script
    Include another YAML file
    We can reuse it!
    Define Test in… Tests expect block in YAML file
    https://github.com/konifar/stac2021-newman-example/pull/5/files

    View full-size slide

  63. Postman & Scenarigo
    Postman Scenarigo
    Manage scenarios in… Postman GUI
    GitHub, GitLab, BitBucket…
    We can review in Pull Request!
    Define Request in… Postman GUI request block in YAML file
    Define Environment in… Postman Environment System Environment
    Generate data before request Pre-request Script Go Plugin
    Request before request as... Pre-request Script
    Include another YAML file
    We can reuse it!
    Define Test in… Tests expect block in YAML file
    https://github.com/konifar/stac2021-newman-example/pull/5/files
    QA Team needs to learn YAML,
    Go, Git and GitHub. (We did it !)

    View full-size slide

  64. Postman is awesome!
    ● Simple GUI makes easy to set up API tests
    for especially non-developers.
    ● Newman makes easy to run tests on CI.

    View full-size slide

  65. But Scenarigo is better for us!
    ● We prefer Pull Request based development style
    ● So we decided to migrate Postman to Scenarigo
    ● Scenarigo covers major features of Postman

    View full-size slide

  66. Thanks for watching!

    View full-size slide

  67. Appendix
    ● Sample code to run Newman and Scenarigo on GitHub Actions
    ○ https://github.com/konifar/stac2021-newman-example
    ● Sample API: Pixela
    ○ https://pixe.la/
    ● Postman official docs
    ○ https://learning.postman.com/docs
    ● Newman
    ○ https://github.com/postmanlabs/newman
    ● Scenarigo
    ○ https://github.com/zoncoen/scenarigo

    View full-size slide