Easy Development of Your Own Chatbot with Flask and Fine-Tuned GPT Model - PyConThailand2023

Easy Development of Your Own Chatbot with Flask and Fine-Tuned GPT Model - PyConThailand2023

Transcript

1. Easy Development of Your Own Chatbot with Flask and Fine-Tuned GPT Model Tetsuya Hirata - Jesse 2023/12/16 #PyConTH #PyConTH2023
2. #PyConTH #PyConTH2023 Target Audience Personas Ignorant in AI / ML products Novice in AI / ML developments An Expert AI / ML Engineer A Professional Researcher Not Target Audience Personas
3. #PyConTH #PyConTH2023 Agenda 1. Overview of Recent Trends 2. Understanding LLM Product Development Workflow 3. Developing ChatBot Application Using Flask and GPT Model 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 5. Wrap-Up Evaluation: GPT Model Before and After Fine-Tuning, and Memory Effects
4. #PyConTH #PyConTH2023 Agenda 1. Overview of Recent Trends 2. Understanding LLM Product Development Workflow 3. Developing ChatBot Application Using Flask and GPT Model 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 5. Wrap-Up Evaluation: GPT Model Before and After Fine-Tuning, and Memory Effects
5. #PyConTH #PyConTH2023 1. Overview of Recent Trends 1.1 ChatBot AI bloom is yet to arrive. Tasks Conventional Chatbots Excel At: Chatbots are highly effective in automating routine queries, providing instant responses, handling high-volume transactional tasks, gathering user data, and offering 24/7 customer service. Tasks Conventional Chatbots Struggle With: They often struggle with understanding complex, nuanced queries, interpreting human emotions, providing personalized advice, and handling situations that require empathy or advanced problem-solving. => Generative AI is progressively overcoming these shortcomings. ChatBot is Rapidly Evolving. Goldman Sachs Global Investment Research. (2023). "Exhibit 12: Previous Milestone Technologies Have Led to Labor Productivity Booms, but the Timing Is Hard to Predict." Retrieved from https://www.gspublishing.com/content/research/en/reports/2023/03/27/d64e052b-0f 6e-45d7-967b-d7be35fabd16.html.
6. #PyConTH #PyConTH2023 1. Overview of Recent Trends 1.2 Flask Flask 3.0.0 (Released 2023-09-30) Flask 2.3.3 (Released 2023-08-21) 1. Removed previously deprecated code. 2. Deprecated the __version__ attribute in favor of feature detection or importlib.metadata.version("flask"). 3. Restructured the code for Flask (app) and Blueprint classes to have Sans-IO bases. 4. Allowed 'self' as an argument to url_for. 5. Required Werkzeug version >= 3.0.0 . 1. Added compatibility with Python 3.12. 2. Required Werkzeug version >= 2.3.7. 3. Switched from setuptools to flit_core as the build backend. 4. Refactored how an app’s root and instance paths are determined . "Flask 3.0.0 Changes," Flask Documentation (3.0.x), 2023.　 flask.palletsprojects.com/3.0.x/changes/#version-3-0-0. "Flask 2.3.0 Changes," Flask Documentation (3.0.x), 2023. flask.palletsprojects.com/3.0.x/changes/#version-2-3-0. Flask 2.3.0 (Released 2023-04-25) 1. Dropped support for Python 3.7. 2. … etc "Flask 2.3.3 Changes," Flask Documentation (3.0.x), 2023. flask.palletsprojects.com/3.0.x/changes/#version-2-3-3.
7. #PyConTH #PyConTH2023 1. Overview of Recent Trends 1.3 GPT Model [2023/11/30] Chat GPT vs Open AI API (GPT Model) RAG vs Fine Tuned GPT Model ChatGPT: A web application offering partial OpenAI API access via a browser GUI, with conversation history and multimodal data support. It enables GPT model fine-tuning, with free access to older versions like GPT-3.5 and a subscription for the latest models. ✩Open AI API (GPT Model): Similar in functionality to Chat GPT, part of OpenAI's various APIs including Whisper and DALL·E 3. Unlike ChatGPT, it doesn't retain conversation history and charges per token, with costs varying by model. RAG (Retrieval-Augmented Generation): Combines information retrieval with text generation, pulling data from external sources to provide informed responses. Ideal for tasks requiring up-to-date or specific knowledge not contained within the training data. Developers need to implement RAG by themselves. ✩Fine-Tuned GPT Model: Customized versions of GPT models tailored to specific tasks or datasets. These models are trained on top of pre-trained GPT models with additional data, enhancing their performance for particular applications or industries. Fine-tuning incurs costs according to token usage.
8. #PyConTH #PyConTH2023 Agenda 1. Overview of Recent Trends 2. Understanding LLM Product Development Workflow 3. Developing ChatBot Application Using Flask and GPT Model 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 5. Wrap-Up Evaluation: GPT Model Before and After Fine-Tuning, and Memory Effects
9. #PyConTH #PyConTH2023 2. Understanding LLM Product Development Workflow Select Foundation Model Prompt Engineering Evaluate Results Room for improve ment ? Need to specify use case ? Fine-Tune Model Web Application Code A / B Testing Canary Blue Green Deploy Model to Production No Yes Yes Prepare dataset for training and evaluating model "LLMOps," Databricks Glossary. Available at [https://www.databricks.com/glossary/llmops](https://www.databricks.com/glossary/llmops). "An Introduction to LLMOPs: Operationalizing and Managing Large Language Models," Microsoft Tech Community. Available at [https://techcommunity.microsoft.com/t5/ai-machine-learning-blog/an-introduction-to-llmops-operationalizing-and-managing-large/ba-p/3910996] "MLOps," Databricks Glossary. Available at [https://www.databricks.com/glossary/mlops](https://www.databricks.com/glossary/mlops).
10. #PyConTH #PyConTH2023 Agenda 1. Overview of Recent Trends 2. Understanding LLM Product Development Workflow 3. Developing ChatBot Application Using Flask and GPT Model 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 5. Wrap-Up Evaluation: GPT Model Before and After Fine-Tuning, and Memory Effects
11. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.1.1 General ChatBot Features No. Required Feature for Chatbot Description Implementation Strategy Difficulty Priority 1 Conversational Management Maintains conversation context and generates appropriate responses. Use session management to maintain the conversation state and generate responses based on context. Medium High 2 Information Retrieval Searches for information from external sources and integrates it into the conversation. This includes web pages, databases, etc. Use web scraping tools or APIs to collect and integrate information into responses. Medium Medium 3 Intent Recognition Recognizes user's intent from their input and responds accordingly. Use NLP libraries to extract intent from input and generate corresponding responses. High High 4 Response Generation Generates responses to user questions and comments. Use pre-trained language models or develop custom response logic. High High 5 User Interface Provides an interface for interaction with users, including text-based interfaces. Build a web-based interface using Flask, HTML, and CSS. Medium High 6 Multilingual Support Supports dialogue and information processing in multiple languages. Use multilingual NLP libraries to handle inputs in different languages. Medium Low 7 Database Integration Retrieves or stores data for conversations from databases. Use databases compatible with Flask to manage user data and conversation histories. Medium Medium
12. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.1.2 Flask ChatBot Features No. Required Feature for Chatbot Description Implementation Strategy Difficulty Priority 1 Conversational Management Maintains conversation context and generates appropriate responses. Use session management to maintain the conversation state and generate responses based on context. Medium High 2 Information Retrieval Searches for information from external sources and integrates it into the conversation. This includes web pages, databases, etc. Use web scraping tools or APIs to collect and integrate information into responses. Medium Medium 3 Intent Recognition Recognizes user's intent from their input and responds accordingly. Use NLP libraries to extract intent from input and generate corresponding responses. High High 4 Response Generation Generates responses to user questions and comments. Use pre-trained language models or develop custom response logic. High High 5 User Interface Provides an interface for interaction with users, including text-based interfaces. Build a web-based interface using Flask, HTML, and CSS. Medium High 6 Multilingual Support Supports dialogue and information processing in multiple languages. Use multilingual NLP libraries to handle inputs in different languages. Medium Low 7 Database Integration Retrieves or stores data for conversations from databases. Use databases compatible with Flask to manage user data and conversation histories. Medium Medium
13. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.1.3 Flask ChatBot Features with GPT Model No. Required Feature for Chatbot Description Implementation Strategy Difficulty Priority 1 Conversational Management Maintains conversation context and generates appropriate responses. Use session management to maintain the conversation state and generate responses based on context. Medium High 2 Information Retrieval Searches for information from external sources and integrates it into the conversation. This includes web pages, databases, etc. Fine Tune the GPT model by using Open AI API Use web scraping tools or APIs to collect and integrate information into responses. Easy Medium Medium 3 Intent Recognition Recognizes user's intent from their input and responds accordingly. Use the GPT Model by using Open AI API Use NLP libraries to extract intent from input and generate corresponding responses. Easy High High 4 Response Generation Generates responses to user questions and comments. Use the GPT Model by using Open AI API Use pre-trained language models or develop custom response logic. Easy High High 5 User Interface Provides an interface for interaction with users, including text-based interfaces. Build a web-based interface using Flask, HTML, and CSS. Medium High 6 Multilingual Support Supports dialogue and information processing in multiple languages. GPT Model already pre-trained information in multiple languages. Use multilingual NLP libraries to handle inputs in different languages. Easy Medium Low 7 Database Integration Retrieves or stores data for conversations from databases. Open AI hosts database, so there is nothing to implement. Use databases compatible with Flask to manage user data and conversation histories. Easy Medium Medium
14. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.1.4 Flask ChatBot Features with GPT Mode and LangChain No. Required Feature for Chatbot Description Implementation Strategy Difficult y Priority Corresponding LangChain Features 1 Conversational Management Maintains conversation context and generates appropriate responses. Use session management to maintain the conversation state and generate responses based on context. *If you use the Assistant API(beta) , there's no need to implement sessions. Read the official documentation for more insights Medium High Chains, Memory 2 Information Retrieval Searches for information from external sources and integrates it into the conversation. This includes web pages, databases, etc. Fine Tune the GPT model Use web scraping tools or APIs to collect and integrate information into responses. Easy Medium Medium Retrieval 3 Intent Recognition Recognizes user's intent from their input and responds accordingly. Use the GPT Model Use NLP libraries to extract intent from input and generate corresponding responses. Easy High High Model I/O 4 Response Generation Generates responses to user questions and comments. Use the GPT Model Use pre-trained language models or develop custom response logic. Easy High High Model I/O, Agent 5 User Interface Provides an interface for interaction with users, including text-based interfaces. Build a web-based interface using Flask, HTML, and CSS. Medium High - 6 Multilingual Support Supports dialogue and information processing in multiple languages. GPT Model already pre-trained information in multiple languages. Use multilingual NLP libraries to handle inputs in different languages. Easy Medium Low - 7 Database Integration Retrieves or stores data for conversations from databases. Open AI hosts database, so there is nothing to implement. Use databases compatible with Flask to manage user data and conversation histories. Easy Medium Medium Memory
15. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.2.1 Demo: Basic Functional Flask ChatBot
16. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.2.2 Code: Basic Functional Flask ChatBot Python Code of Basic Functional Flask ChatBot [1/2] HTML of Basic Functional Flask ChatBot [1/1]
17. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.2.3 Code: Basic Functional Flask ChatBot Python Code of Basic Functional Flask ChatBot [2/2] HTML of Basic Functional Flask ChatBot [1/1]
18. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.3.1 Demo: Flask ChatBot with Streamlined UI
19. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.3.2 Code: Flask ChatBot with Streamlined UI Python Code of Basic Functional Flask ChatBot [1/2] Python Code of Flask ChatBot with Streamlined UI [1/2]
20. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.3.3 Code: Flask ChatBot with Streamlined UI Python Code of Basic Functional Flask ChatBot [2/2] Python Code of Flask ChatBot with Streamlined UI [2/2]
21. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.4.1 Demo: Flask ChatBot with User Memory
22. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.4.2 Code: Flask ChatBot with User Memory Python Code of Flask ChatBot with Streamlined UI [1/2] Python Code of Flask ChatBot with User Memory [1/2]
23. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.4.3 Code: Flask ChatBot with User Memory Python Code of Flask ChatBot with Streamlined UI [2/2] Python Code of Flask ChatBot with User Memory [2/2]
24. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.5.1 Demo: Flask ChatBot with User Memory and Bot Memory
25. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.5.2 Code: Flask ChatBot with User Memory and Bot Memory Python Code of Flask ChatBot with User Memory and Bot Memory [1/2] Python Code of Flask ChatBot with User Memory [1/1]
26. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.5.3 Code: Flask ChatBot with User Memory and Bot Memory Python Code of Flask ChatBot with User Memory and Bot Memory [2/2] Python Code of Flask ChatBot with User Memory [1/1]
27. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.6.1 Evaluating Flask ChatBot: User Memory Only vs. Both User and Bot Memory Flask ChatBot with User Memory and Bot Memory Flask ChatBot with User Memory
28. #PyConTH #PyConTH2023 3. Developing a Chatbot Application Using Flask and GPT Model 3.6.2 Evaluating Flask ChatBot: User Memory Only vs. Both User and Bot Memory Flask ChatBot with User Memory and Bot Memory Flask ChatBot with User Memory
29. #PyConTH #PyConTH2023 Agenda 1. Overview of Recent Trends 2. Understanding LLM Product Development Workflow 3. Developing ChatBot Application Using Flask and GPT Model 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 5. Wrap-Up Evaluation: GPT Model Before and After Fine-Tuning, and Memory Effects
30. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.1 Evaluating GPT Model Version Differences in Second Language Acquisition(SLA) Contexts
31. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.2 Evaluation Metrics: Reliability and Validity in Measurement Accuracy 1. Unreliable and Unvalid: The dots are scattered and rarely hit the center, showcasing both unreliability and invalidity. 2. Unreliable, But Valid: The dots are randomly distributed, occasionally striking the center. This results in accurate group averages but not individual accuracy. 3. Reliable, Not Valid: The dots consistently group together in an area near, but not at, the center, indicating consistent positioning but a deviation from the target. 4.Both Reliable and Valid: The dots consistently align with the central point, demonstrating a measure that is both dependable and precisely on target. Ruel, E., Wagner, W., & Gillespie, B. (2016). The quality of measurement : reliability and validity. In The Practice of Survey Research: Theory and Applications (pp. 78-100). SAGE Publications, Inc, https://doi.org/10.4135/9781483391700 Trochim, W. M. K. (n.d.). Reliability & Validity. In Research Methods Knowledge Base. Conjointly. Retrieved from https://conjointly.com/kb/reliability-and-validity/
32. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.3 Evaluation Plan for Reliability and Validity in LLM-as-a-Judge Approach The below evaluation plan is inspired by the LLM-as-a-Judge approach[1], which involves using powerful LLMs to evaluate LLMs themselves, and the principles of OpenAI Evals[2], a framework recommended by OpenAI for such purposes. A) Evaluation Prompt Dataset Preparation: Utilizing gpt-4-1106-preview to generate prompts and corresponding answers. B) Prompt Dataset Content for Reliability and Validity: - Reliability: Each of the 10 sets of different questions will have answers containing same unique proper nouns (specific_words) and relevant nouns (relevant_words) for each question. - Validity: Each of the 10 sets of different questions will have answers containing different unique proper nouns (specific_words) and relevant nouns (relevant_words) for each question. C) Evaluation Process: Calculating the accuracy rates for reliability and validity by assessing how many generated keywords by gpt-4-1106-preview are included in the responses from the gpt-3.5-turbo model. A response is deemed accurate if it contains any of the keywords at least once. - Reliability: Accuracy rate for different sets of questions with the same answers. - Validity: Accuracy rate for sets of questions each with different answers. [1]: Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. *arXiv:2306.05685v3 [cs.CL]*. NeurIPS 2023 Datasets and Benchmarks Track. DOI: https://doi.org/10.48550/arXiv.2306.05685. [2]: OpenAI Evals: "OpenAI Evals." GitHub. Accessed at: https://github.com/openai/evals
33. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.4.1 Evaluation Reliability Prompt Dataset Created By GPT-4-Turbo(gpt-4-1106-preview) { "input": [ { "role": "user", "content": "How does understanding syntax contribute to language acquisition?" } ], "ideal": "Understanding syntax is crucial in language acquisition as it involves the rules that govern the structure of sentences, which is a fundamental aspect of linguistics. Syntax helps learners grasp how words combine to form phrases and sentences, which is essential for effective communication. Additionally, knowledge of syntax is complemented by phonetics, which aids in the correct pronunciation of words, and semantics, which deals with the meanings of words and sentences. Together, these elements are vital for a comprehensive understanding of any language.", "specific_words": [ "syntax", "linguistics", "phonetics", "semantics" ], "relevant_words": [ "grammar", "language structure", "sound patterns", "meaning" ] } { "input": [ { "role": "user", "content": "Can you describe how semantics is related to cultural context in language learning?" } ], "ideal": "Semantics is deeply related to cultural context in language learning because the meaning of words and phrases can vary significantly across different cultures, which is a fundamental concept in linguistics. Understanding semantics helps learners grasp these nuances and use language appropriately in various cultural settings. This understanding is supported by knowledge of syntax, which allows learners to form culturally relevant sentences, and phonetics, which helps them to pronounce words in a way that is culturally intelligible.", "specific_words": [ "syntax", "linguistics", "phonetics", "semantics" ], "relevant_words": [ "grammar", "language structure", "sound patterns", "meaning" ] } Reliability Prompt Dataset Sample 1 Reliability Prompt Dataset Sample 2
34. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.4.2 Evaluation Validity Prompt Dataset Created By GPT-4-Turbo(gpt-4-1106-preview) { "input": [ { "role": "user", "content": "How can I differentiate between the use of 'ser' and 'estar' in Spanish, and what are some common pitfalls learners might encounter with these verbs?" } ], "ideal": "The verbs 'ser' and 'estar' both translate to 'to be' in English, but they're used in different contexts. 'Ser' is used for characteristics that are permanent or long-lasting, such as identity, time, and physical attributes. For example, 'Ella es alta' (She is tall). 'Estar', on the other hand, is used for states or conditions that are temporary, like emotions or locations. For example, 'Ella está feliz' (She is happy). Common pitfalls include using 'ser' for temporary states or 'estar' for inherent qualities. Also, there are idiomatic expressions where the choice of 'ser' or 'estar' can change the meaning entirely, such as 'ser aburrido' (to be boring) versus 'estar aburrido' (to be bored).", "specific_words": [ "ser", "estar", "permanent", "temporary" ], "relevant_words": [ "verbs", "identity", "emotions", "locations", "attributes", "idiomatic expressions" ] } { "input": [ { "role": "user", "content": "What strategies can I use to expand my vocabulary in a new language, and how should I prioritize which words to learn first?" } ], "ideal": "To expand your vocabulary in a new language, you can use strategies such as spaced repetition, flashcards, and context-based learning. Prioritize learning high-frequency words that are commonly used in everyday conversations. Categorize words into themes such as food, travel, or business to make them more memorable. Engage with authentic materials like books, songs, and movies in the target language. Additionally, practice using new words in sentences to improve retention. It's also beneficial to learn cognates—words that share a similar meaning and form across languages—as they can be easily remembered and increase your vocabulary quickly.", "specific_words": [ "spaced repetition", "flashcards", "high-frequency words", "cognates" ], "relevant_words": [ "vocabulary expansion", "context-based learning", "authentic materials", "retention", "themes" ] } Validity Prompt Dataset Sample 1 Validity Prompt Dataset Sample 2
35. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.5.1 Comparative Evaluation of GPT-4-Turbo(gpt-4-1106-preview) and GPT-3-Turbo Responses on Reliability Prompt Dataset { "prompt": "How does understanding syntax contribute to language acquisition?", "response(GPT3-Turbo)": "Understanding syntax is crucial for language acquisition as it helps individuals comprehend and produce grammatically correct sentences. Syntax refers to the rules and structures that govern how words are organized and combined to form meaningful sentences in a language.Here are a few ways in which understanding syntax contributes to language acquisition:1. Comprehension: Syntax provides the framework for understanding the meaning of sentences. By understanding the order and arrangement of words, learners can decipher the intended message and extract meaning from what they hear or read.2. Production: Syntax guides learners in constructing grammatically correct sentences when speaking or writing. It helps them understand how to arrange words, use appropriate verb tenses, and apply other grammatical rules to convey their thoughts effectively.3. Sentence Expansion: Knowledge of syntax allows learners to expand their sentence structures and express more complex ideas. As they acquire more syntactic patterns, they can combine words and phrases in different ways to create longer and more sophisticated sentences.4. Error Detection: Understanding syntax helps learners identify and correct errors in their own speech or writing. By recognizing when a sentence violates the rules of syntax, they can make necessary adjustments to improve their language proficiency.5. Language Development: Syntax plays a crucial role in the development of language skills. As learners acquire more syntactic knowledge, they become better equipped to understand and produce more advanced vocabulary, idiomatic expressions, and nuanced language usage.Overall, understanding syntax is essential for language acquisition as it provides the foundation for comprehending and producing grammatically correct sentences, expanding language skills, and developing proficiency in a given language.", "ideal(GPT4-Turbo)": "Understanding syntax is crucial in language acquisition as it involves the rules that govern the structure of sentences, which is a fundamental aspect of linguistics. Syntax helps learners grasp how words combine to form phrases and sentences, which is essential for effective communication. Additionally, knowledge of syntax is complemented by phonetics, which aids in the correct pronunciation of words, and semantics, which deals with the meanings of words and sentences. Together, these elements are vital for a comprehensive understanding of any language.", "accuracy": 0.25 }, { "prompt": "Can you explain the role of phonetics in mastering a new language?", "response(GPT3-Turbo)": "Phonetics plays a crucial role in mastering a new language. It is the study of the sounds used in human speech and how they are produced, perceived, and interpreted. Here's how phonetics can help in language learning:1. Pronunciation: Phonetics helps learners understand and produce the correct sounds of a language. It teaches the specific sounds, intonation patterns, and stress patterns used in the target language. By mastering the phonetic aspects, learners can improve their pronunciation and be better understood by native speakers.2. Listening skills: Phonetics training enhances listening skills by enabling learners to recognize and differentiate between different sounds in the target language. This is particularly important in languages with distinct phonetic features, such as tonal languages or languages with unfamiliar phonemes.3. Accent reduction: Phonetics helps learners reduce their accent by identifying and addressing specific pronunciation challenges. By understanding the phonetic differences between their native language and the target language, learners can work on modifying their speech patterns to sound more native-like.4. Spelling and reading: Phonetics aids in understanding the relationship between sounds and written symbols. It helps learners decode and pronounce words correctly, even if they are unfamiliar. By understanding the phonetic rules of a language, learners can improve their reading skills and expand their vocabulary.5. Communication and fluency: Mastering phonetics allows learners to communicate more effectively and fluently. By accurately producing the sounds of a language, learners can convey their message clearly and be better understood by native speakers. It also helps in understanding native speakers' pronunciation, making conversations smoother.Overall, phonetics is essential in language learning as it provides the foundation for accurate pronunciation, listening skills, accent reduction, reading, and effective communication. By focusing on phonetics, learners can improve their overall language proficiency and feel more confident in their language abilities.", "ideal(GPT4-Turbo)": "Semantics is deeply related to cultural context in language learning because the meaning of words and phrases can vary significantly across different cultures, which is a fundamental concept in linguistics. Understanding semantics helps learners grasp these nuances and use language appropriately in various cultural settings. This understanding is supported by knowledge of syntax, which allows learners to form culturally relevant sentences, and phonetics, which helps them to pronounce words in a way that is culturally intelligible.", "accuracy": 0.125 }, … "total_average_accuracy": 0.3125 Response Reliability Prompt Dataset Sample 1 Response Reliability Prompt Dataset Sample 2
36. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.5.2 Comparative Evaluation of GPT-4-Turbo(gpt-4-1106-preview) and GPT-3-Turbo Responses on Validity Prompt Dataset { "prompt": "How can I differentiate between the use of 'ser' and 'estar' in Spanish, and what are some common pitfalls learners might encounter with these verbs?", "response(GPT3-Turbo)": "Differentiating between the use of 'ser' and 'estar' in Spanish can be challenging for learners, as both verbs can be translated as 'to be' in English. However, there are some general guidelines to help you understand when to use each verb.1. 'Ser' is used to express: - Inherent or permanent characteristics: nationality, profession, physical appearance, personality traits, etc. - Time and dates: 'Es la una' (It is one o'clock), 'Es martes' (It is Tuesday). - Origin or nationality: 'Soy de Espauñ' (I am from Spain), 'Ella es mexicana' (She is Mexican). - Possession: 'El libro es mí' (The book is mine). - Generalizations: 'El café es caliente' (Coffee is hot).2. 'Estar' is used to express: - Temporary states or conditions: emotions, physical states, locations, etc. - Actions in progress: 'Estoy estudiando' (I am studying). - Location: 'Estoy en casa' (I am at home). - Changes or transformations: 'El agua está caliente' (The water is hot).Common pitfalls learners might encounter with these verbs include:- Overgeneralizing the use of 'ser' or 'estar' based on English equivalents. Remember that Spanish has different rules for when to use each verb.- Confusing the use of 'ser' and 'estar' with adjectives. While both verbs can be followed by adjectives, the choice of verb depends on the context and the meaning you want to convey.- Forgetting that 'ser' is used for generalizations and 'estar' is used for temporary states or conditions.- Misusing 'ser' and 'estar' when talking about locations. 'Ser' is used for permanent locations (e.g., 'La tienda es en la esquina' - The store is on the corner), while 'estar' is used for temporary locations (e.g., 'Estoy en el parque' - I am at the park).To become more comfortable with the use of 'ser' and 'estar', it is important to practice and expose yourself to different contexts and examples.", "ideal(GPT4-Turbo)": "The verbs 'ser' and 'estar' both translate to 'to be' in English, but they're used in different contexts. 'Ser' is used for characteristics that are permanent or long-lasting, such as identity, time, and physical attributes. For example, 'Ella es alta' (She is tall). 'Estar', on the other hand, is used for states or conditions that are temporary, like emotions or locations. For example, 'Ella está feliz' (She is happy). Common pitfalls include using 'ser' for temporary states or 'estar' for inherent qualities. Also, there are idiomatic expressions where the choice of 'ser' or 'estar' can change the meaning entirely, such as 'ser aburrido' (to be boring) versus 'estar aburrido' (to be bored).", "accuracy": 0.7 }, { "prompt": "What strategies can I use to expand my vocabulary in a new language, and how should I prioritize which words to learn first?", "response(GPT3-Turbo)": "To expand your vocabulary in a new language, here are some strategies you can use:1. Read extensively: Read books, newspapers, magazines, and online articles in the target language. This will expose you to a wide range of vocabulary and help you understand how words are used in context.2. Use flashcards: Create flashcards with new words and their translations. Review them regularly to reinforce your memory.3. Watch movies and TV shows: Watch movies, TV shows, and documentaries in the target language with subtitles. Pay attention to new words and phrases and try to use them in your own conversations.4. Listen to podcasts and music: Listen to podcasts or music in the target language to improve your listening skills and expose yourself to new vocabulary.5. Engage in conversations: Practice speaking with native speakers or language exchange partners. This will help you learn new words and phrases and improve your pronunciation.When prioritizing which words to learn first, consider the following:1. Frequency: Focus on high-frequency words that are commonly used in everyday conversations. These words will give you a solid foundation and help you communicate more effectively.2. Relevance: Prioritize words that are relevant to your interests, needs, or profession. This will make learning more engaging and practical.3. Context: Learn words that are commonly used in specific contexts or situations that you frequently encounter. For example, if you often travel, prioritize learning words related to transportation, accommodation, and directions.4. Personal goals: Identify your personal goals for learning the language. If you have specific goals, such as passing a language proficiency exam or working in a particular field, prioritize learning words related to those goals.Remember, learning vocabulary is an ongoing process, so be consistent and practice regularly to expand your language skills.", "ideal(GPT4-Turbo)": "To expand your vocabulary in a new language, you can use strategies such as spaced repetition, flashcards, and context-based learning. Prioritize learning high-frequency words that are commonly used in everyday conversations. Categorize words into themes such as food, travel, or business to make them more memorable. Engage with authentic materials like books, songs, and movies in the target language. Additionally, practice using new words in sentences to improve retention. It's also beneficial to learn cognates—words that share a similar meaning and form across languages—as they can be easily remembered and increase your vocabulary quickly.", "accuracy": 0.2222222222222222 }, … "total_average_accuracy": 0.4677777777777778 Response Validity Prompt Dataset Sample 1 Response Validity Prompt Dataset Sample 2
37. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.6 Fine Tune the GPT-3.5-Turbo Model Using a Low Accuracy Prompt Dataset { "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "How does understanding syntax contribute to language acquisition?" }, { "role": "assistant", "content": "Understanding syntax is crucial in language acquisition as it involves the rules that govern the structure of sentences, which is a fundamental aspect of linguistics. Syntax helps learners grasp how words combine to form phrases and sentences, which is essential for effective communication. Additionally, knowledge of syntax is complemented by phonetics, which aids in the correct pronunciation of words, and semantics, which deals with the meanings of words and sentences. Together, these elements are vital for a comprehensive understanding of any language." } ] } ..… OpenAI. (n.d.). Preparing Your Dataset for Fine-tuning. Retrieved December 3, 2023, from https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset 13 Prompt Set with Accuracy under 0.5 Python Code for Fine Tuning GPT-3.5-Turbo
38. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.7.1 Comparative Evaluation of GPT-3-Turbo and Fine Tuned GPT-3-Turbo Responses on Reliability Prompt Dataset Sample 1 { "prompt": "How does understanding syntax contribute to language acquisition?", "response(GPT3-Turbo)": "Understanding syntax is crucial for language acquisition as it helps individuals comprehend and produce grammatically correct sentences. Syntax refers to the rules and structures that govern how words are organized and combined to form meaningful sentences in a language.Here are a few ways in which understanding syntax contributes to language acquisition:1. Comprehension: Syntax provides the framework for understanding the meaning of sentences. By understanding the order and arrangement of words, learners can decipher the intended message and extract meaning from what they hear or read.2. Production: Syntax guides learners in constructing grammatically correct sentences when speaking or writing. It helps them understand how to arrange words, use appropriate verb tenses, and apply other grammatical rules to convey their thoughts effectively.3. Sentence Expansion: Knowledge of syntax allows learners to expand their sentence structures and express more complex ideas. As they acquire more syntactic patterns, they can combine words and phrases in different ways to create longer and more sophisticated sentences.4. Error Detection: Understanding syntax helps learners identify and correct errors in their own speech or writing. By recognizing when a sentence violates the rules of syntax, they can make necessary adjustments to improve their language proficiency.5. Language Development: Syntax plays a crucial role in the development of language skills. As learners acquire more syntactic knowledge, they become better equipped to understand and produce more advanced vocabulary, idiomatic expressions, and nuanced language usage.Overall, understanding syntax is essential for language acquisition as it provides the foundation for comprehending and producing grammatically correct sentences, expanding language skills, and developing proficiency in a given language.", "ideal(GPT4-Turbo)": "Understanding syntax is crucial in language acquisition as it involves the rules that govern the structure of sentences, which is a fundamental aspect of linguistics. Syntax helps learners grasp how words combine to form phrases and sentences, which is essential for effective communication. Additionally, knowledge of syntax is complemented by phonetics, which aids in the correct pronunciation of words, and semantics, which deals with the meanings of words and sentences. Together, these elements are vital for a comprehensive understanding of any language.", "accuracy": 0.25 }, { "prompt": "How does understanding syntax contribute to language acquisition?", "response(Fine Tuned GPT3-Turbo)": "Understanding syntax is crucial in language acquisition as it is the set of rules that governs how words are combined to form meaningful sentences. Syntax provides the structure that allows learners to convey their thoughts and ideas effectively. It also helps in comprehending the meaning of sentences spoken or written by others. Alongside phonetics and semantics, syntax forms one of the core components of linguistics, which is the scientific study of language and its structure.", "ideal(GPT4-Turbo)": "Understanding syntax is crucial in language acquisition as it involves the rules that govern the structure of sentences, which is a fundamental aspect of linguistics. Syntax helps learners grasp how words combine to form phrases and sentences, which is essential for effective communication. Additionally, knowledge of syntax is complemented by phonetics, which aids in the correct pronunciation of words, and semantics, which deals with the meanings of words and sentences. Together, these elements are vital for a comprehensive understanding of any language." "accuracy": 0.625 }, [GPT-3.5-Turbo] Response Reliability Prompt Dataset Sample 1 [Fine Tuned GPT-3.5-Turbo] Response Reliability Prompt Dataset Sample 1
39. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.7.2 Comparative Evaluation of GPT-3-Turbo and Fine Tuned GPT-3-Turbo Responses on Reliability Prompt Dataset Sample 2 { "prompt": "Can you explain the role of phonetics in mastering a new language?", "response(Fine Tuned GPT3-Turbo)": "Phonetics plays a crucial role in mastering a new language as it is the branch of linguistics that studies the sounds of human speech. Understanding phonetics helps learners recognize and produce the distinct sounds of a language, which is essential for accurate pronunciation. This knowledge is complemented by an understanding of syntax, which allows learners to construct sentences, and semantics, which helps them grasp the meanings of words and sentences.", "ideal(GPT4-Turbo)": "Semantics is deeply related to cultural context in language learning because the meaning of words and phrases can vary significantly across different cultures, which is a fundamental concept in linguistics. Understanding semantics helps learners grasp these nuances and use language appropriately in various cultural settings. This understanding is supported by knowledge of syntax, which allows learners to form culturally relevant sentences, and phonetics, which helps them to pronounce words in a way that is culturally intelligible.", "accuracy": 0.625 }, … "total_average_accuracy": 0.525 } { "prompt": "Can you explain the role of phonetics in mastering a new language?", "response(GPT3-Turbo)": "Phonetics plays a crucial role in mastering a new language. It is the study of the sounds used in human speech and how they are produced, perceived, and interpreted. Here's how phonetics can help in language learning:1. Pronunciation: Phonetics helps learners understand and produce the correct sounds of a language. It teaches the specific sounds, intonation patterns, and stress patterns used in the target language. By mastering the phonetic aspects, learners can improve their pronunciation and be better understood by native speakers.2. Listening skills: Phonetics training enhances listening skills by enabling learners to recognize and differentiate between different sounds in the target language. This is particularly important in languages with distinct phonetic features, such as tonal languages or languages with unfamiliar phonemes.3. Accent reduction: Phonetics helps learners reduce their accent by identifying and addressing specific pronunciation challenges. By understanding the phonetic differences between their native language and the target language, learners can work on modifying their speech patterns to sound more native-like.4. Spelling and reading: Phonetics aids in understanding the relationship between sounds and written symbols. It helps learners decode and pronounce words correctly, even if they are unfamiliar. By understanding the phonetic rules of a language, learners can improve their reading skills and expand their vocabulary.5. Communication and fluency: Mastering phonetics allows learners to communicate more effectively and fluently. By accurately producing the sounds of a language, learners can convey their message clearly and be better understood by native speakers. It also helps in understanding native speakers' pronunciation, making conversations smoother.Overall, phonetics is essential in language learning as it provides the foundation for accurate pronunciation, listening skills, accent reduction, reading, and effective communication. By focusing on phonetics, learners can improve their overall language proficiency and feel more confident in their language abilities.", "ideal(GPT4-Turbo)": "Semantics is deeply related to cultural context in language learning because the meaning of words and phrases can vary significantly across different cultures, which is a fundamental concept in linguistics. Understanding semantics helps learners grasp these nuances and use language appropriately in various cultural settings. This understanding is supported by knowledge of syntax, which allows learners to form culturally relevant sentences, and phonetics, which helps them to pronounce words in a way that is culturally intelligible.", "accuracy": 0.125 }, … "total_average_accuracy": 0.3125 [GPT-3.5-Turbo] Response Reliability Prompt Dataset Sample 2 [Fine Tuned GPT-3.5-Turbo] Response Reliability Prompt Dataset Sample 2
40. #PyConTH #PyConTH2023 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.7.3 Comparative Evaluation of GPT-3-Turbo and Fine Tuned GPT-3-Turbo Responses on Validity Prompt Dataset Sample 1 { "prompt": "How can I differentiate between the use of 'ser' and 'estar' in Spanish, and what are some common pitfalls learners might encounter with these verbs?", "response(Fine Tuned GPT3-Turbo)": "In Spanish, 'ser' and 'estar' both translate to 'to be' in English, but they are used in different contexts. 'Ser' is used for more permanent characteristics, such as nationality, profession, and physical descriptions. 'Estar' is used for temporary states, locations, and emotions. Common pitfalls for learners include overgeneralizing the use of 'ser' for all situations, as it is the default verb for 'to be' in many languages. Another pitfall is not considering the nuances of 'estar' when describing conditions that may seem permanent, such as saying "estoy aburrido" (I am bored) instead of "soy aburrido" (I am boring). Practice and exposure to the language are key to mastering the correct use of these verbs.", "ideal(GPT4-Turbo)": "The verbs 'ser' and 'estar' both translate to 'to be' in English, but they're used in different contexts. 'Ser' is used for characteristics that are permanent or long-lasting, such as identity, time, and physical attributes. For example, 'Ella es alta' (She is tall). 'Estar', on the other hand, is used for states or conditions that are temporary, like emotions or locations. For example, 'Ella está feliz' (She is happy). Common pitfalls include using 'ser' for temporary states or 'estar' for inherent qualities. Also, there are idiomatic expressions where the choice of 'ser' or 'estar' can change the meaning entirely, such as 'ser aburrido' (to be boring) versus 'estar aburrido' (to be bored).", "accuracy": 0.7 }, { "prompt": "How can I differentiate between the use of 'ser' and 'estar' in Spanish, and what are some common pitfalls learners might encounter with these verbs?", "response(GPT3-Turbo)": "Differentiating between the use of 'ser' and 'estar' in Spanish can be challenging for learners, as both verbs can be translated as 'to be' in English. However, there are some general guidelines to help you understand when to use each verb.1. 'Ser' is used to express: - Inherent or permanent characteristics: nationality, profession, physical appearance, personality traits, etc. - Time and dates: 'Es la una' (It is one o'clock), 'Es martes' (It is Tuesday). - Origin or nationality: 'Soy de Espauñ' (I am from Spain), 'Ella es mexicana' (She is Mexican). - Possession: 'El libro es mí' (The book is mine). - Generalizations: 'El café es caliente' (Coffee is hot).2. 'Estar' is used to express: - Temporary states or conditions: emotions, physical states, locations, etc. - Actions in progress: 'Estoy estudiando' (I am studying). - Location: 'Estoy en casa' (I am at home). - Changes or transformations: 'El agua está caliente' (The water is hot).Common pitfalls learners might encounter with these verbs include:- Overgeneralizing the use of 'ser' or 'estar' based on English equivalents. Remember that Spanish has different rules for when to use each verb.- Confusing the use of 'ser' and 'estar' with adjectives. While both verbs can be followed by adjectives, the choice of verb depends on the context and the meaning you want to convey.- Forgetting that 'ser' is used for generalizations and 'estar' is used for temporary states or conditions.- Misusing 'ser' and 'estar' when talking about locations. 'Ser' is used for permanent locations (e.g., 'La tienda es en la esquina' - The store is on the corner), while 'estar' is used for temporary locations (e.g., 'Estoy en el parque' - I am at the park).To become more comfortable with the use of 'ser' and 'estar', it is important to practice and expose yourself to different contexts and examples.", "ideal(GPT4-Turbo)": "The verbs 'ser' and 'estar' both translate to 'to be' in English, but they're used in different contexts. 'Ser' is used for characteristics that are permanent or long-lasting, such as identity, time, and physical attributes. For example, 'Ella es alta' (She is tall). 'Estar', on the other hand, is used for states or conditions that are temporary, like emotions or locations. For example, 'Ella está feliz' (She is happy). Common pitfalls include using 'ser' for temporary states or 'estar' for inherent qualities. Also, there are idiomatic expressions where the choice of 'ser' or 'estar' can change the meaning entirely, such as 'ser aburrido' (to be boring) versus 'estar aburrido' (to be bored).", "accuracy": 0.7 }, [GPT-3.5-Turbo] Response Validity Prompt Dataset Sample 1 [Fine Tuned GPT-3.5-Turbo] Response Validity Prompt Dataset Sample 1
41. #PyConTH #PyConTH2023 { "prompt": "What strategies can I use to expand my vocabulary in a new language, and how should I prioritize which words to learn first?", "response(Fine Tuned GPT3-Turbo)": "To expand your vocabulary in a new language, you can use strategies like spaced repetition, flashcards, and context-based learning. Prioritize high-frequency words that are commonly used in everyday conversations and texts. Categorize words by themes such as food, travel, or business to make learning more efficient. Additionally, learning cognates—words are similar in both languages—can give you a head start. Engaging with native speakers and media in the target language will expose you to new words and help you understand their usage in context.", "ideal(GPT4-Turbo)": "To expand your vocabulary in a new language, you can use strategies such as spaced repetition, flashcards, and context-based learning. Prioritize learning high-frequency words that are commonly used in everyday conversations. Categorize words into themes such as food, travel, or business to make them more memorable. Engage with authentic materials like books, songs, and movies in the target language. Additionally, practice using new words in sentences to improve retention. It's also beneficial to learn cognates—words that share a similar meaning and form across languages—as they can be easily remembered and increase your vocabulary quickly.", "accuracy": 0.6666666666666666 }, … "total_average_accuracy": 0.4833333333333334 } { "prompt": "What strategies can I use to expand my vocabulary in a new language, and how should I prioritize which words to learn first?", "response(GPT3-Turbo)": "To expand your vocabulary in a new language, here are some strategies you can use:1. Read extensively: Read books, newspapers, magazines, and online articles in the target language. This will expose you to a wide range of vocabulary and help you understand how words are used in context.2. Use flashcards: Create flashcards with new words and their translations. Review them regularly to reinforce your memory.3. Watch movies and TV shows: Watch movies, TV shows, and documentaries in the target language with subtitles. Pay attention to new words and phrases and try to use them in your own conversations.4. Listen to podcasts and music: Listen to podcasts or music in the target language to improve your listening skills and expose yourself to new vocabulary.5. Engage in conversations: Practice speaking with native speakers or language exchange partners. This will help you learn new words and phrases and improve your pronunciation.When prioritizing which words to learn first, consider the following:1. Frequency: Focus on high-frequency words that are commonly used in everyday conversations. These words will give you a solid foundation and help you communicate more effectively.2. Relevance: Prioritize words that are relevant to your interests, needs, or profession. This will make learning more engaging and practical.3. Context: Learn words that are commonly used in specific contexts or situations that you frequently encounter. For example, if you often travel, prioritize learning words related to transportation, accommodation, and directions.4. Personal goals: Identify your personal goals for learning the language. If you have specific goals, such as passing a language proficiency exam or working in a particular field, prioritize learning words related to those goals.Remember, learning vocabulary is an ongoing process, so be consistent and practice regularly to expand your language skills.", "ideal(GPT4-Turbo)": "To expand your vocabulary in a new language, you can use strategies such as spaced repetition, flashcards, and context-based learning. Prioritize learning high-frequency words that are commonly used in everyday conversations. Categorize words into themes such as food, travel, or business to make them more memorable. Engage with authentic materials like books, songs, and movies in the target language. Additionally, practice using new words in sentences to improve retention. It's also beneficial to learn cognates—words that share a similar meaning and form across languages—as they can be easily remembered and increase your vocabulary quickly.", "accuracy": 0.2222222222222222 }, … "total_average_accuracy": 0.4677777777777778 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 4.7.4 Comparative Evaluation of GPT-3-Turbo and Fine Tuned GPT-3-Turbo Responses on Validity Prompt Dataset Sample 2 [GPT-3.5-Turbo] Response Validity Prompt Dataset Sample 2 [Fine Tuned GPT-3.5-Turbo] Response Validity Prompt Dataset Sample 2
42. #PyConTH #PyConTH2023 Agenda 1. Overview of Recent Trends 2. Understanding LLM Product Development Workflow 3. Developing ChatBot Application Using Flask and GPT Model 4. Evaluating ChatBot Application and Fine Tuning the GPT Model 5. Wrap-Up Evaluation: GPT Model Before and After Fine-Tuning, and Memory Effects
43. #PyConTH #PyConTH2023 5. Wrap-Up Evaluation: GPT Model Before and After Fine-Tuning, and Memory Effects