1 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks BUILD SOFTWARE TO TEST SOFTWARE exactpro.com Negative Testing Julia Emelianova PhD, Researcher, Exactpro 22 DECEMBER | 11:30 AM SLST

2 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Negative Testing is a type of software testing used to check software applications for unexpected input data and conditions, that might range from wrong data type to a strong hacking attack. The purpose of negative testing is to prevent software systems from crashing due to negative inputs and improve the quality and stability, to ensure the software is built in such a way that it will not crash or perform abnormally in negative scenarios. When performing negative testing, exceptions and appropriate errors are expected. This shows that the application is able to handle improper user behaviour. During negative testing, users input values that do not work in the system to test its ability to handle incorrect values or system failure. https://www.guru99.com/negative-testing.html https://en.wikipedia.org/wiki/Negative_testing https://www.mstsolutions.com/technical/why-testing-process-needs-negative-testing/ https://www.javatpoint.com/negative-testing Synonyms for ”Negative Testing” ● Error Path Testing ● Failure Testing ● Dirty Testing ● Invalid Testing ● Destructive Testing Invalid Input Error Negative Test System 1. What is Negative Testing?

3 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks ● is used to analyse the stability of an application or a software product against failed conditions, invalid input data or input values, outside the scope or limit; ● is executed to break the system and achieve failure in the operation of a software product by feeding harmful data; ● is performed to identify the defects which can result in major failures or outages, data corruption or security violations; ● allows to evaluate the potential security breaches and special handling procedures; ● is implemented to find the critical loopholes or bugs and weak areas resulting in failures; ● usually, is performed by a test engineer. 2. Characteristics of Negative Testing

4 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks To improve the testing coverage of the application. To improve the quality of the application by detecting defects. To make the application more stable and reliable. To prevent the application from crashing. 3. Purposes of Negative Testing

5 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks https://www.javatpoint.com/negative-testing What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing 4. Negative Testing Scenarios

6 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Some software programmes include input fields that only receive data that falls under a particular range of numbers or text. Negative Testing Scenario 1 – Allowed data bounds and limits. Data Bounds Test – Test all the lower and upper bounds for data fields, and also limits of possible data values.

7 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Test Case Examples: 1. Enter “Birth Year” equal to 1899. 2. Enter “Birth Year” equal to 2017. 3. Enter “Birth Year” equal to -2000. 4. Enter “Your Name” equal to “John123”. Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Only numbers, no letters or special characters, values from 1900 to 2016) Birth Year* Comments Save Some software programmes include input fields that only receive data that falls under a particular range of numbers or text. Negative Testing Scenario 1 – Allowed data bounds and limits. Data Bounds Test – Test all the lower and upper bounds for data fields, and also limits of possible data values.

8 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Examples: 1. Enter “Birth Year” equal to 1899. 2. Enter “Birth Year” equal to 2017. 3. Enter “Birth Year” equal to -2000. 4. Enter “Your Name” equal to “John123”. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing An error occurred. Please enter valid data. Retry Your Name* Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Only numbers, no letters or special characters, values from 1900 to 2016) Birth Year* Comments Save Some software programmes include input fields that only receive data that falls under a particular range of numbers or text. Negative Testing Scenario 1 – Allowed data bounds and limits. Data Bounds Test – Test all the lower and upper bounds for data fields, and also limits of possible data values.

9 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks There are web pages and applications with fields that only permit a limited number of characters to be entered by the user. Negative Testing Scenario 2 – Allowed number of characters. Field Size Test – Test the number of characters one can enter into a field which exceeds the limit specified in the functional specifications. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing

10 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Example: Enter “Your Name” equal to “Maximilianus” (12 characters long). 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Only numbers, no letters or special characters, values from 1900 to 2016) Birth Year* Comments Save There are web pages and applications with fields that only permit a limited number of characters to be entered by the user. Negative Testing Scenario 2 – Allowed number of characters. Field Size Test – Test the number of characters one can enter into a field which exceeds the limit specified in the functional specifications.

11 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Example: Enter “Your Name” equal to “Maximilianus” (12 characters long). 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing An error occurred. The number of characters is too large. Retry Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Only numbers, no letters or special characters, values from 1900 to 2016) Birth Year* Comments Save There are web pages and applications with fields that only permit a limited number of characters to be entered by the user. Negative Testing Scenario 2 – Allowed number of characters. Field Size Test – Test the number of characters one can enter into a field which exceeds the limit specified in the functional specifications.

12 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks There are certain software and web pages which showcase fields that are required to be filled up by the user (marked with “*”). Negative Testing Scenario 3 – Populating required fields. Necessary Data Entry Test – Skip the required data entry and try to proceed. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing

13 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Examples: 1. Leave the field “Birth Year” blank. Enter valid “Your Name” and some text in “Comments”. 2. Leave the fields “Your Name” and “Comments” blank. Enter valid “Birth Year”. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Your Name* Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Only numbers, no letters or special characters, values from 1900 to 2016) Birth Year* Comments Save 3. Leave the fields “Your Name” and “Birth Year” blank. Enter some text in “Comments”. There are certain software and web pages which showcase fields that are required to be filled up by the user (marked with “*”). Negative Testing Scenario 3 – Populating required fields. Necessary Data Entry Test – Skip the required data entry and try to proceed.

14 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Examples: 1. Leave the field “Birth Year” blank. Enter valid “Your Name” and some text in “Comments”. 2. Leave the fields “Your Name” and “Comments” blank. Enter valid “Birth Year”. 3. Leave the fields “Your Name” and “Birth Year” blank. Enter some text in “Comments”. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing An error occurred. Not all the required fields are populated. Retry Your Name* Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Only numbers, no letters or special characters, values from 1900 to 2016) Birth Year* Comments Save There are certain software and web pages which showcase fields that are required to be filled up by the user (marked with “*”). Negative Testing Scenario 3 – Populating required fields. Necessary Data Entry Test – Skip the required data entry and try to proceed.

15 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Most digital forms and dialog boxes are capable of receiving data in a certain form, such as text, number, date, and time. Negative Testing Scenario 4 – Correspondence between data and field types. Data Format/Type Test – Verify each field type, date and time format and its validity. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing

16 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Examples: 1. Enter “1987/02/17” in the field “Birth Date”. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Use the date format YYYY-MM-DD) Birth Date* Comments Save Most digital forms and dialog boxes are capable of receiving data in a certain form, such as text, number, date, and time. Negative Testing Scenario 4 – Correspondence between data and field types. Data Format/Type Test – Verify each field type, date and time format and its validity.

17 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Examples: 1. Enter “1987/02/17” in the field “Birth Date”. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing An error occurred. The date format is incorrect. Retry Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Use the date format YYYY-MM-DD) Birth Date* Comments Save Most digital forms and dialog boxes are capable of receiving data in a certain form, such as text, number, date, and time. Negative Testing Scenario 4 – Correspondence between data and field types. Data Format/Type Test – Verify each field type, date and time format and its validity.

18 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Examples: 1. Enter “1987/02/17” in the field “Birth Date”. 2. Enter “1987-02-45” in the field “Birth Date”. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing An error occurred. The date format is incorrect. Retry Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Use the date format YYYY-MM-DD) Birth Date* Comments Save Most digital forms and dialog boxes are capable of receiving data in a certain form, such as text, number, date, and time. Negative Testing Scenario 4 – Correspondence between data and field types. Data Format/Type Test – Verify each field type, date and time format and its validity.

19 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Examples: 1. Enter “1987/02/17” in the field “Birth Date”. 2. Enter “1987-02-45” in the field “Birth Date”. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing An error occurred. The date format is incorrect. Retry An error occurred. The specified day is invalid. Retry Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Use the date format YYYY-MM-DD) Birth Date* Comments Save Most digital forms and dialog boxes are capable of receiving data in a certain form, such as text, number, date, and time. Negative Testing Scenario 4 – Correspondence between data and field types. Data Format/Type Test – Verify each field type, date and time format and its validity.

20 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Some fields are required to receive alphanumeric data entry containing a single quote. Negative Testing Scenario 5 – Single Quote Testing. Implanted Single Quote Test – Try to store information containing a single quote: embed a single quote on URL when it tries to query the database or enter text involving one or more single quotes in the text fields. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing

21 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Example: Enter “It’s my name in the field ‘Your Name’ above” in the field “Comments” and click the “Save” button. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Use the date format YYYY-MM-DD) Birth Date* Comments Save Some fields are required to receive alphanumeric data entry containing a single quote. Negative Testing Scenario 5 – Single Quote Testing. Implanted Single Quote Test – Try to store information containing a single quote: embed a single quote on URL when it tries to query the database or enter text involving one or more single quotes in the text fields.

22 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Example: Enter “It’s my name in the field ‘Your Name’ above” in the field “Comments” and click the “Save” button. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Use the date format YYYY-MM-DD) Birth Date* Comments Save The following comment is saved: Its my name in the field Your Name above Ok ! Some fields are required to receive alphanumeric data entry containing a single quote. Negative Testing Scenario 5 – Single Quote Testing. Implanted Single Quote Test – Try to store information containing a single quote: embed a single quote on URL when it tries to query the database or enter text involving one or more single quotes in the text fields.

23 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Example: Enter “It’s my name in the field ‘Your Name’ above” in the field “Comments” and click the “Save” button. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Your Name* (Only a-z and A-Z characters, no numbers or special characters, maximum 10 characters long) (Use the date format YYYY-MM-DD) Birth Date* Comments Save The following comment is saved: Its my name in the field Your Name above Ok ! = = = Some fields are required to receive alphanumeric data entry containing a single quote. Negative Testing Scenario 5 – Single Quote Testing. Implanted Single Quote Test – Try to store information containing a single quote: embed a single quote on URL when it tries to query the database or enter text involving one or more single quotes in the text fields.

24 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Negative Testing Scenario 6 – Previous and current release performance. Performance Modification Test – The test suite should contain test cases that compare previous and current release performance statistics, which could help in categorising potential performance problems. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing

25 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Example: Compare round-trip latency distribution for the previous and current releases. Previous release Current release Latency is the time required for a data packet to travel from the sending endpoint to the receiving endpoint. The two maximums correspond to the two types of messages sent. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Negative Testing Scenario 6 – Previous and current release performance. Performance Modification Test – The test suite should contain test cases that compare previous and current release performance statistics, which could help in categorising potential performance problems.

26 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Example: Compare round-trip latency distribution for the previous and current releases. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Previous release Current release Latency is the time required for a data packet to travel from the sending endpoint to the receiving endpoint. The two maximums correspond to the two types of messages sent. The following problem is observed: Round-trip latency gets globally worse. Ok ! Negative Testing Scenario 6 – Previous and current release performance. Performance Modification Test – The test suite should contain test cases that compare previous and current release performance statistics, which could help in categorising potential performance problems.

27 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Certain web browsers require the user to log in first before it loads web pages. Negative Testing Scenario 7 – Web Session Testing. Web Session Test – Attempt to open web pages in the application without the user logging in beforehand. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing

28 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Test Case Example: Try to download boards for the user “johnsmith123” in the Trello application. 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Certain web browsers require the user to log in first before it loads web pages. Negative Testing Scenario 7 – Web Session Testing. Web Session Test – Attempt to open web pages in the application without the user logging in beforehand.

29 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks ! 4. Negative Testing Scenarios What are Negative Test Cases? Data Bounds Test Field Size Test Necessary Data Entry Data Format/ Type Test Implanted Single Quote Performance Modification Web Session Testing Certain web browsers require the user to log in first before it loads web pages. Negative Testing Scenario 7 – Web Session Testing. Web Session Test – Attempt to open web pages in the application without the user logging in beforehand. Test Case Example: Try to download boards for the user “johnsmith123” in the Trello application.

30 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques Testers need: ● to try every input which is invalid; ● check the behaviour of the application against it. If an application crashes or shuts down without any warning or message, it needs: ● to be fixed; ● to give a proper message or warning. So that the user would know what not to do while using the application. Consider some of the smart techniques that you can utilise to perform negative testing using an AI chatbot as an example.

31 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.1. Boundary Value Analysis This method involves writing the Test Cases for values that are outside the boundary limits.

32 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.1. Boundary Value Analysis This method involves writing the Test Cases for values that are outside the boundary limits. Test Case Example: Boundary values on the character length of a question for the chatbot.

33 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.1. Boundary Value Analysis This method involves writing the Test Cases for values that are outside the boundary limits. Test Case Example: Boundary values on the character length of a question for the chatbot. Short question: Julia: Do you love cats? ChatBot: Yes.

34 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.1. Boundary Value Analysis This method involves writing the Test Cases for values that are outside the boundary limits. Test Case Example: Boundary values on the character length of a question for the chatbot. Short question: Long question: Julia: The cat is similar in anatomy to the other felid species: it has a strong flexible body, quick reflexes, sharp teeth, and retractable claws adapted to killing small prey. Its night vision and sense of smell are well developed. Cat communication includes vocalisations like meowing, purring, trilling, hissing, growling, and grunting as well as cat-specific body language. Do you love cats? ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Julia: Do you love cats? ChatBot: Yes.

35 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.1. Boundary Value Analysis This method involves writing the Test Cases for values that are outside the boundary limits. Test Case Example: Boundary values on the character length of a question for the chatbot. Long question: Julia: The cat is similar in anatomy to the other field species: it has a strong flexible body, quick reflexes, sharp teeth, and retractable claws adapted to killing small prey. Its night vision and sense of smell are well developed. Cat communication includes vocalizations like meowing, purring, trilling, hissing, growling, and grunting as well as cat-specific body language. Do you love cats? ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Short question: Julia: Do you love cats? ChatBot: Yes. Negative testing

36 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.2. Equivalence Partitioning This is a software testing technique which divides the input data into many partitions. Values from each partition must be tested at least once. Partitions with valid values are used for positive testing. Partitions with invalid values are used for negative testing.

37 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.2. Equivalence Partitioning This is a software testing technique which divides the input data into many partitions. Values from each partition must be tested at least once. Partitions with valid values are used for positive testing. Partitions with invalid values are used for negative testing. Test Case Example: Two types of languages used for dialog with the chatbot = Two equivalence partitions ● Supported languages (English, German, French) ● Unsupported languages (Korean, Chinese, Japan, etc.)

38 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.2. Equivalence Partitioning This is a software testing technique which divides the input data into many partitions. Values from each partition must be tested at least once. Partitions with valid values are used for positive testing. Partitions with invalid values are used for negative testing. Test Case Example: Two types of languages used for dialog with the chatbot = Two equivalence partitions ● Supported languages (English, German, French) ● Unsupported languages (Korean, Chinese, Japan, etc.) Julia: 잘 지내고 있나요? ChatBot: I do not even know what to answer... 잘 지내고 있나요? Means “How are you?” in Korean.

39 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.3. Error Guessing This procedure pinpoints the specific conditions which trigger failure or error messages. If possible, try to identify and correct the issue without a system crash. In some cases, an incorrect result may lead to an extreme “downstream” failure if the error condition is not immediately noticed and addressed. https://botsurfer.com/learn/8-ways-to-break-chatbot

40 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.3. Error Guessing This procedure pinpoints the specific conditions which trigger failure or error messages. If possible, try to identify and correct the issue without a system crash. In some cases, an incorrect result may lead to an extreme “downstream” failure if the error condition is not immediately noticed and addressed. Test Case Examples: There are several known issues which may break a chatbot. So if you need to test the chatbot, you may guess that your chatbot can also be broken in similar ways. https://botsurfer.com/learn/8-ways-to-break-chatbot

41 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.3. Error Guessing This procedure pinpoints the specific conditions which trigger failure or error messages. If possible, try to identify and correct the issue without a system crash. In some cases, an incorrect result may lead to an extreme “downstream” failure if the error condition is not immediately noticed and addressed. Test Case Examples: There are several known issues which may break a chatbot. So if you need to test the chatbot, you may guess that your chatbot can also be broken in similar ways. Julia: Reset ChatBot: I do not know what you want. Do you have any other questions? 1. Tell the chatbot to reset or start over when you came to the end of your line of questioning and you wanted to look at other information. https://botsurfer.com/learn/8-ways-to-break-chatbot

42 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.3. Error Guessing This procedure pinpoints the specific conditions which trigger failure or error messages. If possible, try to identify and correct the issue without a system crash. In some cases, an incorrect result may lead to an extreme “downstream” failure if the error condition is not immediately noticed and addressed. Test Case Examples: There are several known issues which may break a chatbot. So if you need to test the chatbot, you may guess that your chatbot can also be broken in similar ways. Julia: What can I do? ChatBot: You have to do what you are good at. What can you do? 2. Ask for Help or Assistance Many chatbots don’t know what to do when you ask them for assistance or help. https://botsurfer.com/learn/8-ways-to-break-chatbot

43 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.4. Checklist It is a basic method that documents error conditions that you are planning to test. A software testing checklist is commonly used in tandem with error guessing. https://www.qable.io/the-chatbot-testing-checklist-top-tools-techniques/

44 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.4. Checklist It is a basic method that documents error conditions that you are planning to test. A software testing checklist is commonly used in tandem with error guessing. Example: https://www.qable.io/the-chatbot-testing-checklist-top-tools-techniques/ ❏ Slow or bad navigation ❏ Poor User-Interface ❏ Lack of word knowledge ❏ Lack of intelligence ❏ Take too much time to respond ❏ Same or wrong answers ❏ Poor connection timing

45 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.5. Exploratory Testing This technique is used to increase your knowledge about the application while the test is ongoing. It may be conducted simultaneously with other tests. Exploratory testing can show you a clear picture of which aspects of the application work and which ones do not.

46 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.5. Exploratory Testing Test Case Examples: 1. If a chatbot shares links to other resources such as websites, email ids, images, etc., make sure all these entities are functional. This technique is used to increase your knowledge about the application while the test is ongoing. It may be conducted simultaneously with other tests. Exploratory testing can show you a clear picture of which aspects of the application work and which ones do not.

47 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.5. Exploratory Testing Test Case Examples: Julia: 000111000011001010000111 ChatBot: Unfortunately, I do not know what to answer... 2. Enter the incorrect question as a sequence of numbers and check the result. 1. If a chatbot shares links to other resources such as websites, email ids, images, etc., make sure all these entities are functional. This technique is used to increase your knowledge about the application while the test is ongoing. It may be conducted simultaneously with other tests. Exploratory testing can show you a clear picture of which aspects of the application work and which ones do not.

48 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.6. Small Scale Test Automation It is a method that uncovers: ● memory errors, ● code defects, ● other issues that are occasionally revealed in production use.

49 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.6. Small Scale Test Automation It is a method that uncovers: ● memory errors, ● code defects, ● other issues that are occasionally revealed in production use. It entails performing the same action thousands of times to see its effect on the program.

50 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.7. Fuzz Testing This procedure involves the input of: ● invalid data, ● random data, ● semi random data which may cause unexpected failures, crashes, and issues. In this case, there are no preset expected results, unlike other negative test cases. It is simply an observation of what may potentially occur because of arbitrary inputs. Fuzzing looks only for crashes or some other undesirable behaviour, and millions of test cases might be executed before a crash. Moreover, many bugs such as logic errors do not crash the software system but instead produce incorrect output – a failure type that is much more difficult to detect.

51 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.7. Fuzz Testing Test Case Example: Julia: fjdhdsjkfsdkj ChatBot: thinking… The chatbot is not active for a long time Step 1

52 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.7. Fuzz Testing Test Case Example: Julia: fjdhdsjkfsdkj ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Step 1

53 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.7. Fuzz Testing Test Case Example: Julia: fjdhdsjkfsdkj ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Step 1 Julia: 4357845hjfd ??? sd iufd @@^%? ChatBot: thinking… Step 2 The chatbot is not active for a long time

54 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.7. Fuzz Testing Test Case Example: Julia: fjdhdsjkfsdkj ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Step 1 Julia: 4357845hjfd ??? sd iufd @@^%? ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Step 2

55 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.7. Fuzz Testing Test Case Example: Julia: fjdhdsjkfsdkj ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Step 1 Julia: 4357845hjfd ??? sd iufd @@^%? ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Step 2 Julia: What oh yes ring do help horse? ChatBot: thinking… Step N … The chatbot is not active for a long time

56 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Sorry, the ChatBot just crashed! Ok ! 5. Negative Testing Techniques 5.7. Fuzz Testing Test Case Example: Julia: fjdhdsjkfsdkj ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Step 1 Julia: 4357845hjfd ??? sd iufd @@^%? ChatBot: Sorry, friend. Something went wrong and now I can not answer you. Step 2 Step N …

57 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.8. State Transition Testing It originates from the concept that software can only exist in one particular state at one point in time. For instance, a program is considered to be in its normal state until an issue reveals itself. Test Case Examples: State transition diagram for logging into the chatbot application. https://www.guru99.com/state-transition-testing.html START 3rd Try ACCESS GRANTED 2nd Try 1st Try Correct Password Correct Password Correct Password ACCOUNT BLOCKED Incorrect Password Incorrect Password Incorrect Password

58 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks 5. Negative Testing Techniques 5.8. State Transition Testing It originates from the concept that software can only exist in one particular state at one point in time. For instance, a program is considered to be in its normal state until an issue reveals itself. Test Case Examples: State transition diagram for logging into the chatbot application. If you enter the incorrect password three times, your account will be blocked with a warning message. https://www.guru99.com/state-transition-testing.html Sorry, your account is blocked! Ok ! START 3rd Try ACCESS GRANTED 2nd Try 1st Try Correct Password Correct Password Correct Password ACCOUNT BLOCKED Incorrect Password Incorrect Password Incorrect Password

59 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks ● Increasing software application protection from cybercrime and unethical hacks. ● Increasing guarantee of a product's quality, stability and pleasant to use. https://www.javatpoint.com/negative-testing https://www.educba.com/negative-testing/ 6. Benefits of Negative Testing

60 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks ● Increasing software application protection from cybercrime and unethical hacks. ● Increasing guarantee of a product's quality, stability and pleasant to use. https://www.javatpoint.com/negative-testing https://www.educba.com/negative-testing/ ● Enhance the possibility of covering all bases and each type of error, which can arise because of inconsistent human behaviour. ● The user will know the limitations beforehand. Knowledge of the application limitations allows us to use it accordingly. 6. Benefits of Negative Testing

61 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks ● Increasing software application protection from cybercrime and unethical hacks. ● Increasing guarantee of a product's quality, stability and pleasant to use. https://www.javatpoint.com/negative-testing https://www.educba.com/negative-testing/ ● Enhance the possibility of covering all bases and each type of error, which can arise because of inconsistent human behaviour. ● The user will know the limitations beforehand. Knowledge of the application limitations allows us to use it accordingly. ● Give more confidence to the client before going live. 6. Benefits of Negative Testing

62 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks ● The execution of negative testing needs extra time, money, and effort. https://www.javatpoint.com/negative-testing https://www.educba.com/negative-testing/ 7. Disadvantages of Negative Testing

63 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks ● The execution of negative testing needs extra time, money, and effort. https://www.javatpoint.com/negative-testing https://www.educba.com/negative-testing/ 7. Disadvantages of Negative Testing ● For the customer, it causes extreme delays in a software product or an application release.

64 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks ● The execution of negative testing needs extra time, money, and effort. https://www.javatpoint.com/negative-testing https://www.educba.com/negative-testing/ 7. Disadvantages of Negative Testing ● For the customer, it causes extreme delays in a software product or an application release. ● The implementation of negative testing needs a trained and experienced test engineer to develop negative test cases. Highly experienced people are required for negative testing.

65 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Which test cases are related to negative testing? ❏ Upload invalid files like XML, SQL, etc. files in image upload field. ❏ Try to enter a valid email id in the email field, a valid password in the password field and try to login. ❏ Verify whether payment is done for an invalid member, provider or broker ID by creating a respective record in the feed. ❏ Test system with expired card details. ❏ Test with a valid PIN for the credit card. Question

66 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Please scan the QR code and choose the right answer(s): Join at slido.com #7237 487

67 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Which test cases are related to negative testing? ❏ Upload invalid files like XML, SQL, etc. files in image upload field. ❏ Try to enter a valid email id in the email field, a valid password in the password field and try to login. ❏ Verify whether payment is done for an invalid member, provider or broker ID by creating a respective record in the feed. ❏ Test system with expired card details. ❏ Test with a valid PIN for the credit card. Question

68 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks AI Testing Talks Thank You! Questions?