Elena Treshcheva. Automated Ambiguity Detection in Software Specifications

We invite you to join the International Summer School on Data Science in Software Engineering. The summer school will be held online on 12-16 July, 2021, organized by the Laboratory of Software Testing, Tomsk Polytechnic University.
Participation is free. The official language of the school is English.

Students, young researchers and practitioners interested in applications of modern data science methods to the development and testing of complex software systems are invited to join. Follow the link to learn the full program and register your participation: https://itr-tpu.timepad.ru/event/1629835/
____
To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. Build Software to Test Software
   exactpro.com
   Elena Treshcheva, Program Manager,
   Exactpro
   Automated Ambiguity Detection
   in Software Specifications
   

   View Slide

2. 2 Build Software to Test Software
   exactpro.com
   Overview
   Introduction
   Software requirements specifications: what they are and who they are for
   Types of ambiguity in software requirements specifications
   Manual and automated ambiguity detection on different linguistic levels
   Rule-based vs ML-based automated ambiguity detection
   Software requirements defects: is it just about ambiguity?
   References
   

   View Slide

3. 3 Build Software to Test Software
   exactpro.com
   About Exactpro
   A specialist firm focused on
   functional and non-functional
   testing services for financial
   market infrastructures, financial
   and non-financial institutions.
   Headquartered in the UK with
   operations in the US and
   Eastern Europe. Most of our
   clients are regulated market
   infrastructures.
   Exactpro provides independent
   software testing services for
   mission critical technology that
   underpins global financial
   markets. The firm is experienced
   with trading and clearing &
   settlement platforms, risk
   management systems, central
   data warehouses, accounting
   and treasury systems, regulatory
   reporting.
   Incorporated in 2009 with 10
   people, Exactpro is now
   employing over 600 specialists.
   Part of the London Stock
   Exchange Group (LSEG) from May
   2015 until January 2018, when the
   Exactpro management proceeded
   through the buyout of the
   company from LSEG
   

   View Slide

4. 4 Build Software to Test Software
   exactpro.com
   About me
   2005 - 2018 – General and applied linguistics (Saratov State University):
   - linguistic database engineer
   - head of the Laboratory of Applied Linguistics
   - PhD in general linguistics (2012)
   - associate professor in computer linguistics
   2017 – Software testing, QA analyst (Exactpro, Saratov)
   2018 – Business Development and Research (Exactpro, USA)
   https://www.linkedin.com/in/treshcheva/
   

   View Slide

5. 5 Build Software to Test Software
   exactpro.com
   Software requirements specifications (SRS): definition
   The SRS is a specification for a
   particular software product, program,
   or set of programs that performs
   certain functions in a specific
   environment.
   "IEEE Recommended Practice for Software Requirements
   Specifications," in IEEE Std 830-1998, pp.1-40, 20 Oct. 1998,
   doi: 10.1109/IEEESTD.1998.88286.
   

   View Slide

6. 6 Build Software to Test Software
   exactpro.com
   Role of SRS in the software engineering project success
   

   View Slide

7. 7 Build Software to Test Software
   exactpro.com
   Two types of specs
   Human readable
   Machine readable
   Specifications
   
   
   
   
   
   
   
   

   View Slide

8. 8 Build Software to Test Software
   exactpro.com
   Two types of specs
   Human readable
   Machine readable
   + unambiguous
   - many nontechnical users find them
   unintelligible
   + but those who understand them - do it the
   same way
   - require specialized knowledge to create them
   - target specific systems, FL can influence
   requirements in subtle ways
   - innately ambiguous
   + easy to understand for any reader
   - but no guarantee that the
   understanding will be the same
   + easier to find people able to write them
   + more universal -> do not impose
   anything on the requirements
   Stark reality: NLs are inevitable, even if it is only for the initial conception.
   (Unless the client is some really weird mathtype nerd who thinks in first-order
   predicate calculus.)
   © D. Berry, International Workshop on Requirements Engineering 2001
   

   View Slide

9. 9 Build Software to Test Software
   exactpro.com
   Ambiguity in Requirements Engineering
   https://conf.researchr.org/home/RE-2021
   https://requirements-engineering.org/ (papers)
   

   View Slide

10. 10 Build Software to Test Software
    exactpro.com
    Ambiguity in “ambiguity”
    1) “Ambiguity — where something can be interpreted in more than one
    way” (Chantree et. al. 2006)
    2) “An SRS is unambiguous if, and only if, every requirement stated therein
    has only one interpretation” (IEEE Standard, 1998) → more than one +
    no interpretation at all
    Vague or incomplete statements may not have a valid interpretation → ambiguous
    because they are not unambiguous
    

    View Slide

11. 11 Build Software to Test Software
    exactpro.com
    Types of ambiguity
    1) Language ambiguities vs. conceptual ambiguities
    2) Nocuous vs. innocuous
    3) Acknowledged vs. unacknowledged
    

    View Slide

12. 12 Build Software to Test Software
    exactpro.com
    Types of ambiguity (Berry et al. 2003)
    Linguistic ambiguity
    Generality and
    vagueness
    Syntactic
    ambiguity
    Semantic
    ambiguity
    Pragmatic
    ambiguity
    Lexical
    ambiguity
    Language error
    

    View Slide

13. 13 Build Software to Test Software
    exactpro.com
    Ambiguity Taxonomy by Massey et al. (2014)
    

    View Slide

14. 14 Build Software to Test Software
    exactpro.com
    Ambiguity detection types
    Automated
    Manual
    Ambiguity detection
    - Checklists
    - Modeling
    - Scenario-based reading
    (Kamsties et al. 2001)
    - Rule-based
    - AI-based
    (Berry 2001)
    “We prefer naturally dumb clericality to artificial intelligence. Stupidity is preferred to
    intelligence if the latter can lose information as a result of it not ever being perfect.”
    © D. Berry, International Workshop on Requirements Engineering 2001
    

    View Slide

15. 15 Build Software to Test Software
    exactpro.com
    Manual ambiguity detection: Challenges
    - Dependency on the reader’s having an ear for the language
    - Dependency on the reader’s competence and field of work
    - Subjectivity
    - Intentionality of some ambiguities (e.g. to avoid frequent document updates)
    - Workload
    

    View Slide

16. 16 Build Software to Test Software
    exactpro.com
    Automated ambiguity detection
    - Helps reduce human effort
    - Reduces subjectivity (to a certain degree)
    - Identifies unacknowledged ambiguity
    

    View Slide

17. 17 Build Software to Test Software
    exactpro.com
    Automated ambiguity detection: Rule-based
    - Searching for an exact phrase
    - Using tokenizer + POS-tagger and searching for particular tags
    - Searching using regular expressions on both words and tags
    - Using heuristics about words in texts of requirements specifications: e.g. collocation
    frequency, distributional similarity, semantic similarity, etc.
    - Combination of the above
    

    View Slide

18. 18 Build Software to Test Software
    exactpro.com
    Exact phrases: ambiguous, vague, and uncertain words
    and, any, or
    after, before, next, previous, etc.
    minimum and maximum
    a large number of adjectives and adverbs: acceptable, accurate, appropriate, easy,
    efficient, essential, immediately, minimum, maximum, periodically, sufficient, user-friendly,
    etc.
    also verbs and nouns can be vague: support, handle, process, reject, use, etc. and user.
    uncertainty is expressed using words such as not limited to, etc., can, may, probably,
    possibly, usually, etc.
    

    View Slide

19. 19 Build Software to Test Software
    exactpro.com
    Exact phrases: Quantification
    all, each, every
    a, all, any, each one, some, and the as quantifiers
    many and few
    only, also, others
    even
    

    View Slide

20. 20 Build Software to Test Software
    exactpro.com
    Patterns: Structural ambiguity
    ● Adjectives and other modifiers: English grammar teacher
    English, i.e., British, teacher of grammar, not necessarily English grammar
    teacher of English grammar, with the teacher’s nationality left unspecified
    ● Pronoun references: Bob said to Joe that he must leave
    ● This and Whole Ideas: this that refers to a whole idea vs. this referring to a specific
    noun
    ● Otherwise
    ● Not
    ● Not and because
    ● And and or in the same sentence
    

    View Slide

21. 21 Build Software to Test Software
    exactpro.com
    Rule-based ambiguity detection: Example 1
    Using NLP to detect requirements defects: An industrial experience in the railway domain
    (B. Rosadini et al., 2017)
    

    View Slide

22. 22 Build Software to Test Software
    exactpro.com
    Rule-based ambiguity detection: Example 2
    Using Domain-specific Corpora for Improved Handling of Ambiguity in Requirements
    (S. Ezzini et al., 2021)
    

    View Slide

23. 23 Build Software to Test Software
    exactpro.com
    Automated ambiguity detection: ML-based
    More advanced techniques allowing to tackle higher linguistic levels:
    - Word embeddings / vector representations, often involving domain-specific text
    corpora
    - Bayesian algorithms trained on lexical and syntactic features / anaphoric pronouns
    - Decision-tree based text classifiers
    

    View Slide

24. 24 Build Software to Test Software
    exactpro.com
    ML-based ambiguity detection: Example
    Detecting domain-specific ambiguities: An NLP approach based on wikipedia crawling and word
    embeddings (A. Ferrari et al., 2017)
    

    View Slide

25. 25 Build Software to Test Software
    exactpro.com
    Specifications written in natural language:
    What else can go wrong
    Unreadable texts: Difficulty and complexity measurements:
    - phrases and words length
    - phrase structure
    - uncommon words usage (number of rare words)
    - number of idioms/metaphors/metonyms
    - readability indices
    K. T. Jensen, “Indicators of text complexity,” Copenhagen Studies in Language, vol. 37, pp.
    61–80, 2009.
    

    View Slide

26. 26 Build Software to Test Software
    exactpro.com
    References
    IEEE Recommended Practice for Software Requirements Specifications (1998)
    D. Berry Natural language and requirements engineering-Nu? (2001)
    E. Kamsties et al. Detecting ambiguities in requirements documents using inspections (2001)
    D. Berry et. al. From Contract Drafting to Software Specification: Linguistic Sources of Ambiguity (2003)
    F. Chantree et al. Identifying nocuous ambiguities in natural language requirements (2006)
    K.T. Jensen Indicators of text complexity (2009)
    B. Rosadini et al. Using NLP to detect requirements defects: An industrial experience in the railway domain
    (2017)
    A. Ferrari et al. Detecting domain-specific ambiguities: An NLP approach based on wikipedia crawling and
    word embeddings (2017)
    S. Ezzini et al. Using Domain-specific Corpora for Improved Handling of Ambiguity in Requirements (2021)
    

    View Slide

27. Kari Kakkonen
    A Dragon Lesson of
    Software Testing
    // Software Testing Webinar
    19 August 2021
    

    View Slide

28. THANK YOU!
    

    View Slide