Tweet
Tweet

Slide 1

Slide 1 text

StackSpot.com @StackSpot Developer Experiences with a Contextualized AI Coding Assistant: Usability Expectations, and Outcomes Igor Steinmacher Northern Arizona U. Alberto Tavares Zup I.T. Cleidson de Souza UFPA Thayssa Rocha Zup I.T. & UFPA Gustavo Pinto Zup I.T. & UFPA Edward Monteiro Stackspot

Slide 2

Slide 2 text

StackSpot.com @StackSpot Motivation

Slide 3

Slide 3 text

StackSpot.com @StackSpot Motivation https://survey.stackoverflow.co/2023/#ai

Slide 4

Slide 4 text

StackSpot.com @StackSpot Motivation

Slide 5

Slide 5 text

StackSpot.com @StackSpot Motivation

Slide 6

Slide 6 text

StackSpot.com @StackSpot Motivation

Slide 7

Slide 7 text

StackSpot.com @StackSpot Top 10 Limitations Of ChatGPT 1. Lack of emotional intelligence 2. Lack of common sense 3. Limitations in understanding context 4. Trouble generating long-form, structured content 5. Limitations in handling multiple tasks at the same time 6. Potentially biased responses: 7. Limited knowledge 8. Accuracy problems or grammatical issues 9. Need for fine-tuning 10. Computational costs and power Motivation 1 https://www.forbes.com/sites/bernardmarr/2023/03/03/the-top-10-limitations-of-chatgpt/

Slide 8

Slide 8 text

StackSpot.com @StackSpot Motivation Top 10 Limitations Of ChatGPT1 1. Lack of emotional intelligence 2. Lack of common sense 3. Limitations in understanding context 4. Trouble generating long-form, structured content 5. Limitations in handling multiple tasks at the same time 6. Potentially biased responses: 7. Limited knowledge 8. Accuracy problems or grammatical issues 9. Need for fine-tuning 10. Computational costs and power 1 https://www.forbes.com/sites/bernardmarr/2023/03/03/the-top-10-limitations-of-chatgpt/

Slide 9

Slide 9 text

StackSpot.com @StackSpot Motivation Lack of context lead to subpar answers

Slide 10

Slide 10 text

StackSpot.com @StackSpot Motivation How can we make AI-based tools more contextualized to our problems?

Slide 11

Slide 11 text

StackSpot.com @StackSpot StackSpot AI https://ai.stackspot.com/

Slide 12

Slide 12 text

StackSpot.com @StackSpot StackSpot AI Prompt com esquema de código, Stack Al, snippets, OpenAPI e esquemas de eventos StackSpot LLM StackSpot Embeddings StackSpot UI Canais StackSpotCLI VSCode extension IntelliJ extension API StackSpot AI StackSpot Actions Agents Space Dev Assistant Agent Built-in Quick Commands Agent Custom Quick Commands Agent Code Suggestion Agent StackSpot Workspaces StackSpot Studios Knowledge Sources Stack AI Dynamic Source Code Code Snippets API Event Schemas Custom https://ai.stackspot.com/

Slide 13

Slide 13 text

StackSpot.com @StackSpot StackSpot AI Prompt com esquema de código, Stack Al, snippets, OpenAPI e esquemas de eventos StackSpot LLM StackSpot Embeddings StackSpot UI Canais StackSpotCLI VSCode extension IntelliJ extension API StackSpot AI StackSpot Actions Agents Space Dev Assistant Agent Built-in Quick Commands Agent Custom Quick Commands Agent Code Suggestion Agent StackSpot Workspaces StackSpot Studios Knowledge Sources Stack AI Dynamic Source Code Code Snippets API Event Schemas Custom https://ai.stackspot.com/ ICSE SEIP 2024

Slide 14

Slide 14 text

StackSpot.com @StackSpot Research problems ● What developers think about StackSpot AI? ● Would they think they are more productive using StackSpot AI? ● Would they trust in StackSpot AI answers?

Slide 15

Slide 15 text

StackSpot.com @StackSpot Research Approach User Study: Online workshop with 62 devs ● Hands on workshop (3h) ● Group discussion (1h)

Slide 16

Slide 16 text

StackSpot.com @StackSpot Research Approach User Study: Online workshop with 62 devs Experience with AI tools ● Hands on workshop (3h) ● Group discussion (1h)

Slide 17

Slide 17 text

StackSpot.com @StackSpot ● How did you perceive the ease of use of StackSpot AI? ● What challenges did you face when using StackSpot AI? ● How useful and accurate were the responses generated? ● Did StackSpot AI save you time during the task development? Approach: Focus Group

Slide 18

Slide 18 text

StackSpot.com @StackSpot Results: Ease of use Generating precise code for easy integration "Thus, the tool accurately grasped the context, generating a code with Spring framework — it was impressive. I did not expect it to work out so well. StackSpot AI created the class as I requested. Indeed, I am quite amazed here"

Slide 19

Slide 19 text

StackSpot.com @StackSpot Results: Ease of use Generating precise code for easy integration "Thus, the tool accurately grasped the context, generating a code with Spring framework — it was impressive. I did not expect it to work out so well. StackSpot AI created the class as I requested. Indeed, I am quite amazed here" "For me, the contextualization helped in generating accurate code. It helped a lot. I saved time; the answer was straight to the point. The contextualization was the biggest gain for me”. Responses are more context-aware compared to other tools

Slide 20

Slide 20 text

StackSpot.com @StackSpot Results: Ease of use Generating precise code for easy integration "Thus, the tool accurately grasped the context, generating a code with Spring framework — it was impressive. I did not expect it to work out so well. StackSpot AI created the class as I requested. Indeed, I am quite amazed here" "For me, the contextualization helped in generating accurate code. It helped a lot. I saved time; the answer was straight to the point. The contextualization was the biggest gain for me”. Responses are more context-aware compared to other tools Time-saving by centralizing information access "I obtained the answer from the code StackSpot AI generated; I didn’t need to go to the original knowledge source. In this case, it was accurate and helpful. [...] Even though the question I asked was a very simple example, StackSpot AI demonstrated that centralizing information in the IDE would be highly useful"

Slide 21

Slide 21 text

StackSpot.com @StackSpot Results: Perception of productivity Contextualized code snippets "Yes, because the knowledge base will become increasingly richer and will help in generating better responses"

Slide 22

Slide 22 text

StackSpot.com @StackSpot Results: Perception of productivity Contextualized code snippets "Yes, because the knowledge base will become increasingly richer and will help in generating better responses" "StackSpot AI saves time, as it’s a shortcut for accessing information. If done through conventional means, you’d have to search through a search engine, consult books, or find people with that information to assist you, which would certainly take longer." Aggregating knowledge sources within the IDE

Slide 23

Slide 23 text

StackSpot.com @StackSpot Results: Perception of productivity Contextualized code snippets "Yes, because the knowledge base will become increasingly richer and will help in generating better responses" "StackSpot AI saves time, as it’s a shortcut for accessing information. If done through conventional means, you’d have to search through a search engine, consult books, or find people with that information to assist you, which would certainly take longer." Aggregating knowledge sources within the IDE "However, if used carelessly or by less experienced people, it may result in more work for the more experienced developers." Productivity gains unlock only if users know how to use it

Slide 24

Slide 24 text

StackSpot.com @StackSpot Results: Challenges encountered "Initially, I thought using various Knowledge Sources in the same workspace might not be a good practice. [..]. I combined a postal code API and various elements from the provided knowledge source. The result wasn’t great" Knowledge Source Mixing Impact

Slide 25

Slide 25 text

StackSpot.com @StackSpot "need to understand more about it, but were able to use it in a very basic way." Results: Challenges encountered Figuring out what is a good knowledge source "Initially, I thought using various Knowledge Sources in the same workspace might not be a good practice. [..]. I combined a postal code API and various elements from the provided knowledge source. The result wasn’t great" Knowledge Source Mixing Impact

Slide 26

Slide 26 text

StackSpot.com @StackSpot "need to understand more about it, but were able to use it in a very basic way." Results: Challenges encountered Figuring out what is a good knowledge source "Initially, I thought using various Knowledge Sources in the same workspace might not be a good practice. [..]. I combined a postal code API and various elements from the provided knowledge source. The result wasn’t great" Knowledge Source Mixing Impact Initial Configuration and Onboarding Process “[the code suggestions] were better than other tools, but I believe it was because of how I configured the Knowledge Sources.”

Slide 27

Slide 27 text

StackSpot.com @StackSpot Concluding

Slide 28

Slide 28 text

StackSpot.com @StackSpot Concluding

Slide 29

Slide 29 text

StackSpot.com @StackSpot Concluding

Slide 30

Slide 30 text

StackSpot.com @StackSpot Concluding

Slide 31

Slide 31 text

StackSpot.com @StackSpot Concluding

Slide 32

Slide 32 text

StackSpot.com @StackSpot Concluding [email protected]

Slide 33

Slide 33 text

StackSpot.com @StackSpot StackSpot.com @StackSpot Obrigado! Alguma pergunta? Estamos aqui para ajudar!

Slide 34

Slide 34 text

StackSpot.com @StackSpot IA e a evolução tecnológica no desenvolvimento de software. Entendemos o quão difícil é o dia a dia do desenvolvedor. Contexto Acelere o processo de codificação com IA. Aderência às melhores práticas com o ecossistema StackSpot. Uma experiência de desenvolvimento mais eficiente e simplificada. Objetivos-chave Aumento da produtividade de codificação do desenvolvedor com sugestões de código hiper contextualizado. Geração de Código AI com a mais alta qualidade do mercado. Nossas metas

Slide 35

Slide 35 text

StackSpot.com @StackSpot AI and the Technological Evolution in Software Development We understand how hard is the developer day-by-day. Context Accelerate the coding process with AI. Adherence to best practices with the StackSpot ecosystem. A more efficient and streamlined development experience. Key objectives Increase developer's coding productivity up to 50% with hyper-contextualized code suggestions. AI Code Generation with the highest quality in the market. Our goals

Slide 36

Slide 36 text

StackSpot.com @StackSpot Software 2.0: ● LLM, deep learning, neural networks, data labeling, model fine tuning Software 3.0: ● AI interaction ● Agents With Tools ● Data ● Vector Software 1.0: ● Current modernization and digital transformation default Software 1.0, 2.0 e 3.0 Rules + Data = Respostas Respostas + Data = Rules Objetivos + Data = Actions Humanos Escrevem Software Software Escrevem Software Software que toma ações

Slide 37

Slide 37 text

StackSpot.com @StackSpot Prompt com esquema de código, Stack Al, snippets, OpenAPI e esquemas de eventos Modelo e camada de pesquisa StackSpot LLM StackSpot Embeddings StackSpot UI Canais StackSpotCLI VSCode extension IntelliJ extension API StackSpot AI StackSpot Actions Agents Space Dev Assistant Agent Built-in Quick Commands Agent Custom Quick Commands Agent Code Suggestion Agent StackSpot Workspaces StackSpot Studios Knowledge Sources Stack AI Dynamic Source Code Code Snippets API Event Schemas Custom StackSpot AI: Arquitetura

Slide 38

Slide 38 text

StackSpot.com @StackSpot Prompt with code schema, stack Al, snippets, OpenAPI and event schemas Model & Search Layer StackSpot LLM StackSpot Embeddings StackSpot UI Channels StackSpotCLI VSCode extension IntelliJ extension API StackSpot Code Buddy StackSpot Actions Agents Space Dev Assistant Agent Built-in Quick Commands Agent Custom Quick Commands Agent Code Suggestion Agent StackSpot Workspaces StackSpot Studios Knowledge Sources Stack AI Dynamic Source Code Code Snippets API Event Schemas Custom StackSpot AI: Architecture

Slide 39

Slide 39 text

StackSpot.com @StackSpot Com o poder da Inteligência Artificial (IA) combinado com o uso da Hiper Contextualização trazida pelos Workspaces, Studios, Stacks, Catálogos de API, Catálogos de Eventos (Schemas) e Custom Knowledge Objects, os desenvolvedores entregarão aplicações e códigos com maior qualidade em uma forma mais rápida e eficiente. StackSpot.com @StackSpot

Slide 40

Slide 40 text

StackSpot.com @StackSpot With the power of Artificial Intelligence (AI) combined with the use of the Hyper Contextualization brought with the Workspaces, Studios, Stacks, API Catalogs, Event Catalogs (Schemas), and Custom Knowledge Objects, developers will deliver applications and code with higher quality in a faster and more efficient way. StackSpot.com @StackSpot

Slide 41

Slide 41 text

Demo Time Software 3.0

Slide 42

Slide 42 text

StackSpot.com @StackSpot Identificar, analisar e resolver problemas Specs Code Review Writing Code Code Code Review Agent Mind Writing Specs or Use Knowledge Source Hoje Futuro Breve

Slide 43

Slide 43 text

StackSpot.com @StackSpot Identify, Analyze and Solve Problems Specs Code Review Code Code Code Review Knowledge Sources Mente dos agentes Specs

Slide 44

Slide 44 text

Demo Time Get ready to see StackSpot AI in action! Discover how it assists developers in essential tasks, offering hyper contextualized tips and code generation.

Slide 45

Slide 45 text

StackSpot.com @StackSpot Roadmap & e novos recursos ● Modelagem de contas (Freemium/Premium/Enterprise) ● Personalização de Comandos Rápidos ● Sugestões de código ● Knowledge Sources: Event and Custom types ● Histórico de bate-papo por Workspace Principais recursos a serem entregues:

Slide 46

Slide 46 text

StackSpot.com @StackSpot Roadmap & New Features ● Account modelling (Freemium/Premium/Enterprise) ● Custom Quick Commands ● Code Suggestions ● Knowledge Sources: Event and Custom types ● Chat History per Workspace Main features to be delivered:

Slide 47

Slide 47 text

StackSpot.com @StackSpot Geração e Assistência com IA: Automação avançada para codificação, economizando tempo e esforço. Comandos Rápidos: Interação eficiente e rápida em seu IDE. O poder da Hiper Contextualização: Melhoramos a partir dos dados do projeto para gerar códigos mais relevantes e precisos. Personalização com Snippets de Código: Oferecemos sugestões de código e podemos ajudar o LLM a entender as regras e o estilo de codificação da sua empresa. Múltiplas fontes de conhecimento & HUB : Aproveite APIs, esquemas de eventos e objetos personalizados. Use, Crie e Compartilhe Integração avançada com Workspaces, Studios & Stacks AI: Uma abordagem integrada para acelerar o desenvolvimento. StackSpot AI

Slide 48

Slide 48 text

StackSpot.com @StackSpot Generation and Assistance with AI: Advanced automation for coding, saving time and effort. Quick Commands: Efficient and fast interaction in your IDE. The power of the Hyper Contextualization: We improve from project data to generate more relevant and accurate codes. Customization with Code Snippets: We offer code suggestions and we can help the LLM to understand the rules and coding style of your company. Multiple Knowledge Sources: Leverage APIs, Event Schemas, and Custom Objects. Advanced Integration with Workspaces, Studios & Stacks AI: An integrated approach to accelerate development. Why StackSpot AI and Code Buddy?

Slide 49

Slide 49 text

StackSpot.com @StackSpot Quer liderar a mudança? Just Ask! Junte-se a nós em nosso Discord Acesse o Portal StackSpot AI

Slide 50

Slide 50 text

StackSpot.com @StackSpot Want to lead the change? Just ask! 10/02, exclusive for all QCon San Francisco Community Alpha launch Join us in our Discord Access the StackSpot AI Portal.

Slide 51

Slide 51 text

StackSpot.com @StackSpot StackSpot.com @StackSpot Obrigado! Alguma pergunta? Estamos aqui para ajudar!