Build Software to Test Software exactpro.com Elena Treshcheva, Program Manager, Exactpro Automated Ambiguity Detection in Software Specifications

2 Build Software to Test Software exactpro.com Overview Introduction Software requirements specifications: what they are and who they are for Types of ambiguity in software requirements specifications Manual and automated ambiguity detection on different linguistic levels Rule-based vs ML-based automated ambiguity detection Software requirements defects: is it just about ambiguity? References

3 Build Software to Test Software exactpro.com About Exactpro A specialist firm focused on functional and non-functional testing services for financial market infrastructures, financial and non-financial institutions. Headquartered in the UK with operations in the US and Eastern Europe. Most of our clients are regulated market infrastructures. Exactpro provides independent software testing services for mission critical technology that underpins global financial markets. The firm is experienced with trading and clearing & settlement platforms, risk management systems, central data warehouses, accounting and treasury systems, regulatory reporting. Incorporated in 2009 with 10 people, Exactpro is now employing over 600 specialists. Part of the London Stock Exchange Group (LSEG) from May 2015 until January 2018, when the Exactpro management proceeded through the buyout of the company from LSEG

4 Build Software to Test Software exactpro.com About me 2005 - 2018 – General and applied linguistics (Saratov State University): - linguistic database engineer - head of the Laboratory of Applied Linguistics - PhD in general linguistics (2012) - associate professor in computer linguistics 2017 – Software testing, QA analyst (Exactpro, Saratov) 2018 – Business Development and Research (Exactpro, USA) https://www.linkedin.com/in/treshcheva/

5 Build Software to Test Software exactpro.com Software requirements specifications (SRS): definition The SRS is a specification for a particular software product, program, or set of programs that performs certain functions in a specific environment. "IEEE Recommended Practice for Software Requirements Specifications," in IEEE Std 830-1998, pp.1-40, 20 Oct. 1998, doi: 10.1109/IEEESTD.1998.88286.

6 Build Software to Test Software exactpro.com Role of SRS in the software engineering project success

7 Build Software to Test Software exactpro.com Two types of specs Human readable Machine readable Specifications

8 Build Software to Test Software exactpro.com Two types of specs Human readable Machine readable + unambiguous - many nontechnical users find them unintelligible + but those who understand them - do it the same way - require specialized knowledge to create them - target specific systems, FL can influence requirements in subtle ways - innately ambiguous + easy to understand for any reader - but no guarantee that the understanding will be the same + easier to find people able to write them + more universal -> do not impose anything on the requirements Stark reality: NLs are inevitable, even if it is only for the initial conception. (Unless the client is some really weird mathtype nerd who thinks in first-order predicate calculus.) © D. Berry, International Workshop on Requirements Engineering 2001

9 Build Software to Test Software exactpro.com Ambiguity in Requirements Engineering https://conf.researchr.org/home/RE-2021 https://requirements-engineering.org/ (papers)

10 Build Software to Test Software exactpro.com Ambiguity in “ambiguity” 1) “Ambiguity — where something can be interpreted in more than one way” (Chantree et. al. 2006) 2) “An SRS is unambiguous if, and only if, every requirement stated therein has only one interpretation” (IEEE Standard, 1998) → more than one + no interpretation at all Vague or incomplete statements may not have a valid interpretation → ambiguous because they are not unambiguous

11 Build Software to Test Software exactpro.com Types of ambiguity 1) Language ambiguities vs. conceptual ambiguities 2) Nocuous vs. innocuous 3) Acknowledged vs. unacknowledged

12 Build Software to Test Software exactpro.com Types of ambiguity (Berry et al. 2003) Linguistic ambiguity Generality and vagueness Syntactic ambiguity Semantic ambiguity Pragmatic ambiguity Lexical ambiguity Language error

13 Build Software to Test Software exactpro.com Ambiguity Taxonomy by Massey et al. (2014)

14 Build Software to Test Software exactpro.com Ambiguity detection types Automated Manual Ambiguity detection - Checklists - Modeling - Scenario-based reading (Kamsties et al. 2001) - Rule-based - AI-based (Berry 2001) “We prefer naturally dumb clericality to artificial intelligence. Stupidity is preferred to intelligence if the latter can lose information as a result of it not ever being perfect.” © D. Berry, International Workshop on Requirements Engineering 2001

15 Build Software to Test Software exactpro.com Manual ambiguity detection: Challenges - Dependency on the reader’s having an ear for the language - Dependency on the reader’s competence and field of work - Subjectivity - Intentionality of some ambiguities (e.g. to avoid frequent document updates) - Workload

16 Build Software to Test Software exactpro.com Automated ambiguity detection - Helps reduce human effort - Reduces subjectivity (to a certain degree) - Identifies unacknowledged ambiguity

17 Build Software to Test Software exactpro.com Automated ambiguity detection: Rule-based - Searching for an exact phrase - Using tokenizer + POS-tagger and searching for particular tags - Searching using regular expressions on both words and tags - Using heuristics about words in texts of requirements specifications: e.g. collocation frequency, distributional similarity, semantic similarity, etc. - Combination of the above

18 Build Software to Test Software exactpro.com Exact phrases: ambiguous, vague, and uncertain words and, any, or after, before, next, previous, etc. minimum and maximum a large number of adjectives and adverbs: acceptable, accurate, appropriate, easy, efficient, essential, immediately, minimum, maximum, periodically, sufficient, user-friendly, etc. also verbs and nouns can be vague: support, handle, process, reject, use, etc. and user. uncertainty is expressed using words such as not limited to, etc., can, may, probably, possibly, usually, etc.

19 Build Software to Test Software exactpro.com Exact phrases: Quantification all, each, every a, all, any, each one, some, and the as quantifiers many and few only, also, others even

20 Build Software to Test Software exactpro.com Patterns: Structural ambiguity ● Adjectives and other modifiers: English grammar teacher English, i.e., British, teacher of grammar, not necessarily English grammar teacher of English grammar, with the teacher’s nationality left unspecified ● Pronoun references: Bob said to Joe that he must leave ● This and Whole Ideas: this that refers to a whole idea vs. this referring to a specific noun ● Otherwise ● Not ● Not and because ● And and or in the same sentence

21 Build Software to Test Software exactpro.com Rule-based ambiguity detection: Example 1 Using NLP to detect requirements defects: An industrial experience in the railway domain (B. Rosadini et al., 2017)

22 Build Software to Test Software exactpro.com Rule-based ambiguity detection: Example 2 Using Domain-specific Corpora for Improved Handling of Ambiguity in Requirements (S. Ezzini et al., 2021)

23 Build Software to Test Software exactpro.com Automated ambiguity detection: ML-based More advanced techniques allowing to tackle higher linguistic levels: - Word embeddings / vector representations, often involving domain-specific text corpora - Bayesian algorithms trained on lexical and syntactic features / anaphoric pronouns - Decision-tree based text classifiers

24 Build Software to Test Software exactpro.com ML-based ambiguity detection: Example Detecting domain-specific ambiguities: An NLP approach based on wikipedia crawling and word embeddings (A. Ferrari et al., 2017)

25 Build Software to Test Software exactpro.com Specifications written in natural language: What else can go wrong Unreadable texts: Difficulty and complexity measurements: - phrases and words length - phrase structure - uncommon words usage (number of rare words) - number of idioms/metaphors/metonyms - readability indices K. T. Jensen, “Indicators of text complexity,” Copenhagen Studies in Language, vol. 37, pp. 61–80, 2009.

26 Build Software to Test Software exactpro.com References IEEE Recommended Practice for Software Requirements Specifications (1998) D. Berry Natural language and requirements engineering-Nu? (2001) E. Kamsties et al. Detecting ambiguities in requirements documents using inspections (2001) D. Berry et. al. From Contract Drafting to Software Specification: Linguistic Sources of Ambiguity (2003) F. Chantree et al. Identifying nocuous ambiguities in natural language requirements (2006) K.T. Jensen Indicators of text complexity (2009) B. Rosadini et al. Using NLP to detect requirements defects: An industrial experience in the railway domain (2017) A. Ferrari et al. Detecting domain-specific ambiguities: An NLP approach based on wikipedia crawling and word embeddings (2017) S. Ezzini et al. Using Domain-specific Corpora for Improved Handling of Ambiguity in Requirements (2021)

Kari Kakkonen A Dragon Lesson of Software Testing // Software Testing Webinar 19 August 2021

THANK YOU!