Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Developer Experiences with a Contextualized AI ...

Gustavo Pinto
July 31, 2024
6

Developer Experiences with a Contextualized AI Coding Assistant: Usability Expectations, and Outcomes

Paper presented at CAIN'2024

Gustavo Pinto

July 31, 2024
Tweet

More Decks by Gustavo Pinto

Transcript

  1. StackSpot.com @StackSpot Developer Experiences with a Contextualized AI Coding Assistant:

    Usability Expectations, and Outcomes Igor Steinmacher Northern Arizona U. Alberto Tavares Zup I.T. Cleidson de Souza UFPA Thayssa Rocha Zup I.T. & UFPA Gustavo Pinto Zup I.T. & UFPA Edward Monteiro Stackspot
  2. StackSpot.com @StackSpot Top 10 Limitations Of ChatGPT 1. Lack of

    emotional intelligence 2. Lack of common sense 3. Limitations in understanding context 4. Trouble generating long-form, structured content 5. Limitations in handling multiple tasks at the same time 6. Potentially biased responses: 7. Limited knowledge 8. Accuracy problems or grammatical issues 9. Need for fine-tuning 10. Computational costs and power Motivation 1 https://www.forbes.com/sites/bernardmarr/2023/03/03/the-top-10-limitations-of-chatgpt/
  3. StackSpot.com @StackSpot Motivation Top 10 Limitations Of ChatGPT1 1. Lack

    of emotional intelligence 2. Lack of common sense 3. Limitations in understanding context 4. Trouble generating long-form, structured content 5. Limitations in handling multiple tasks at the same time 6. Potentially biased responses: 7. Limited knowledge 8. Accuracy problems or grammatical issues 9. Need for fine-tuning 10. Computational costs and power 1 https://www.forbes.com/sites/bernardmarr/2023/03/03/the-top-10-limitations-of-chatgpt/
  4. StackSpot.com @StackSpot StackSpot AI Prompt com esquema de código, Stack

    Al, snippets, OpenAPI e esquemas de eventos StackSpot LLM StackSpot Embeddings StackSpot UI Canais StackSpotCLI VSCode extension IntelliJ extension API StackSpot AI StackSpot Actions Agents Space Dev Assistant Agent Built-in Quick Commands Agent Custom Quick Commands Agent Code Suggestion Agent StackSpot Workspaces StackSpot Studios Knowledge Sources Stack AI Dynamic Source Code Code Snippets API Event Schemas Custom https://ai.stackspot.com/
  5. StackSpot.com @StackSpot StackSpot AI Prompt com esquema de código, Stack

    Al, snippets, OpenAPI e esquemas de eventos StackSpot LLM StackSpot Embeddings StackSpot UI Canais StackSpotCLI VSCode extension IntelliJ extension API StackSpot AI StackSpot Actions Agents Space Dev Assistant Agent Built-in Quick Commands Agent Custom Quick Commands Agent Code Suggestion Agent StackSpot Workspaces StackSpot Studios Knowledge Sources Stack AI Dynamic Source Code Code Snippets API Event Schemas Custom https://ai.stackspot.com/ ICSE SEIP 2024
  6. StackSpot.com @StackSpot Research problems • What developers think about StackSpot

    AI? • Would they think they are more productive using StackSpot AI? • Would they trust in StackSpot AI answers?
  7. StackSpot.com @StackSpot Research Approach User Study: Online workshop with 62

    devs • Hands on workshop (3h) • Group discussion (1h)
  8. StackSpot.com @StackSpot Research Approach User Study: Online workshop with 62

    devs Experience with AI tools • Hands on workshop (3h) • Group discussion (1h)
  9. StackSpot.com @StackSpot • How did you perceive the ease of

    use of StackSpot AI? • What challenges did you face when using StackSpot AI? • How useful and accurate were the responses generated? • Did StackSpot AI save you time during the task development? Approach: Focus Group
  10. StackSpot.com @StackSpot Results: Ease of use Generating precise code for

    easy integration "Thus, the tool accurately grasped the context, generating a code with Spring framework — it was impressive. I did not expect it to work out so well. StackSpot AI created the class as I requested. Indeed, I am quite amazed here"
  11. StackSpot.com @StackSpot Results: Ease of use Generating precise code for

    easy integration "Thus, the tool accurately grasped the context, generating a code with Spring framework — it was impressive. I did not expect it to work out so well. StackSpot AI created the class as I requested. Indeed, I am quite amazed here" "For me, the contextualization helped in generating accurate code. It helped a lot. I saved time; the answer was straight to the point. The contextualization was the biggest gain for me”. Responses are more context-aware compared to other tools
  12. StackSpot.com @StackSpot Results: Ease of use Generating precise code for

    easy integration "Thus, the tool accurately grasped the context, generating a code with Spring framework — it was impressive. I did not expect it to work out so well. StackSpot AI created the class as I requested. Indeed, I am quite amazed here" "For me, the contextualization helped in generating accurate code. It helped a lot. I saved time; the answer was straight to the point. The contextualization was the biggest gain for me”. Responses are more context-aware compared to other tools Time-saving by centralizing information access "I obtained the answer from the code StackSpot AI generated; I didn’t need to go to the original knowledge source. In this case, it was accurate and helpful. [...] Even though the question I asked was a very simple example, StackSpot AI demonstrated that centralizing information in the IDE would be highly useful"
  13. StackSpot.com @StackSpot Results: Perception of productivity Contextualized code snippets "Yes,

    because the knowledge base will become increasingly richer and will help in generating better responses"
  14. StackSpot.com @StackSpot Results: Perception of productivity Contextualized code snippets "Yes,

    because the knowledge base will become increasingly richer and will help in generating better responses" "StackSpot AI saves time, as it’s a shortcut for accessing information. If done through conventional means, you’d have to search through a search engine, consult books, or find people with that information to assist you, which would certainly take longer." Aggregating knowledge sources within the IDE
  15. StackSpot.com @StackSpot Results: Perception of productivity Contextualized code snippets "Yes,

    because the knowledge base will become increasingly richer and will help in generating better responses" "StackSpot AI saves time, as it’s a shortcut for accessing information. If done through conventional means, you’d have to search through a search engine, consult books, or find people with that information to assist you, which would certainly take longer." Aggregating knowledge sources within the IDE "However, if used carelessly or by less experienced people, it may result in more work for the more experienced developers." Productivity gains unlock only if users know how to use it
  16. StackSpot.com @StackSpot Results: Challenges encountered "Initially, I thought using various

    Knowledge Sources in the same workspace might not be a good practice. [..]. I combined a postal code API and various elements from the provided knowledge source. The result wasn’t great" Knowledge Source Mixing Impact
  17. StackSpot.com @StackSpot "need to understand more about it, but were

    able to use it in a very basic way." Results: Challenges encountered Figuring out what is a good knowledge source "Initially, I thought using various Knowledge Sources in the same workspace might not be a good practice. [..]. I combined a postal code API and various elements from the provided knowledge source. The result wasn’t great" Knowledge Source Mixing Impact
  18. StackSpot.com @StackSpot "need to understand more about it, but were

    able to use it in a very basic way." Results: Challenges encountered Figuring out what is a good knowledge source "Initially, I thought using various Knowledge Sources in the same workspace might not be a good practice. [..]. I combined a postal code API and various elements from the provided knowledge source. The result wasn’t great" Knowledge Source Mixing Impact Initial Configuration and Onboarding Process “[the code suggestions] were better than other tools, but I believe it was because of how I configured the Knowledge Sources.”
  19. StackSpot.com @StackSpot IA e a evolução tecnológica no desenvolvimento de

    software. Entendemos o quão difícil é o dia a dia do desenvolvedor. Contexto Acelere o processo de codificação com IA. Aderência às melhores práticas com o ecossistema StackSpot. Uma experiência de desenvolvimento mais eficiente e simplificada. Objetivos-chave Aumento da produtividade de codificação do desenvolvedor com sugestões de código hiper contextualizado. Geração de Código AI com a mais alta qualidade do mercado. Nossas metas
  20. StackSpot.com @StackSpot AI and the Technological Evolution in Software Development

    We understand how hard is the developer day-by-day. Context Accelerate the coding process with AI. Adherence to best practices with the StackSpot ecosystem. A more efficient and streamlined development experience. Key objectives Increase developer's coding productivity up to 50% with hyper-contextualized code suggestions. AI Code Generation with the highest quality in the market. Our goals
  21. StackSpot.com @StackSpot Software 2.0: • LLM, deep learning, neural networks,

    data labeling, model fine tuning Software 3.0: • AI interaction • Agents With Tools • Data • Vector Software 1.0: • Current modernization and digital transformation default Software 1.0, 2.0 e 3.0 Rules + Data = Respostas Respostas + Data = Rules Objetivos + Data = Actions Humanos Escrevem Software Software Escrevem Software Software que toma ações
  22. StackSpot.com @StackSpot Prompt com esquema de código, Stack Al, snippets,

    OpenAPI e esquemas de eventos Modelo e camada de pesquisa StackSpot LLM StackSpot Embeddings StackSpot UI Canais StackSpotCLI VSCode extension IntelliJ extension API StackSpot AI StackSpot Actions Agents Space Dev Assistant Agent Built-in Quick Commands Agent Custom Quick Commands Agent Code Suggestion Agent StackSpot Workspaces StackSpot Studios Knowledge Sources Stack AI Dynamic Source Code Code Snippets API Event Schemas Custom StackSpot AI: Arquitetura
  23. StackSpot.com @StackSpot Prompt with code schema, stack Al, snippets, OpenAPI

    and event schemas Model & Search Layer StackSpot LLM StackSpot Embeddings StackSpot UI Channels StackSpotCLI VSCode extension IntelliJ extension API StackSpot Code Buddy StackSpot Actions Agents Space Dev Assistant Agent Built-in Quick Commands Agent Custom Quick Commands Agent Code Suggestion Agent StackSpot Workspaces StackSpot Studios Knowledge Sources Stack AI Dynamic Source Code Code Snippets API Event Schemas Custom StackSpot AI: Architecture
  24. StackSpot.com @StackSpot Com o poder da Inteligência Artificial (IA) combinado

    com o uso da Hiper Contextualização trazida pelos Workspaces, Studios, Stacks, Catálogos de API, Catálogos de Eventos (Schemas) e Custom Knowledge Objects, os desenvolvedores entregarão aplicações e códigos com maior qualidade em uma forma mais rápida e eficiente. StackSpot.com @StackSpot
  25. StackSpot.com @StackSpot With the power of Artificial Intelligence (AI) combined

    with the use of the Hyper Contextualization brought with the Workspaces, Studios, Stacks, API Catalogs, Event Catalogs (Schemas), and Custom Knowledge Objects, developers will deliver applications and code with higher quality in a faster and more efficient way. StackSpot.com @StackSpot
  26. StackSpot.com @StackSpot Identificar, analisar e resolver problemas Specs Code Review

    Writing Code Code Code Review Agent Mind Writing Specs or Use Knowledge Source Hoje Futuro Breve
  27. StackSpot.com @StackSpot Identify, Analyze and Solve Problems Specs Code Review

    Code Code Code Review Knowledge Sources Mente dos agentes Specs
  28. Demo Time Get ready to see StackSpot AI in action!

    Discover how it assists developers in essential tasks, offering hyper contextualized tips and code generation.
  29. StackSpot.com @StackSpot Roadmap & e novos recursos • Modelagem de

    contas (Freemium/Premium/Enterprise) • Personalização de Comandos Rápidos • Sugestões de código • Knowledge Sources: Event and Custom types • Histórico de bate-papo por Workspace Principais recursos a serem entregues:
  30. StackSpot.com @StackSpot Roadmap & New Features • Account modelling (Freemium/Premium/Enterprise)

    • Custom Quick Commands • Code Suggestions • Knowledge Sources: Event and Custom types • Chat History per Workspace Main features to be delivered:
  31. StackSpot.com @StackSpot Geração e Assistência com IA: Automação avançada para

    codificação, economizando tempo e esforço. Comandos Rápidos: Interação eficiente e rápida em seu IDE. O poder da Hiper Contextualização: Melhoramos a partir dos dados do projeto para gerar códigos mais relevantes e precisos. Personalização com Snippets de Código: Oferecemos sugestões de código e podemos ajudar o LLM a entender as regras e o estilo de codificação da sua empresa. Múltiplas fontes de conhecimento & HUB : Aproveite APIs, esquemas de eventos e objetos personalizados. Use, Crie e Compartilhe Integração avançada com Workspaces, Studios & Stacks AI: Uma abordagem integrada para acelerar o desenvolvimento. StackSpot AI
  32. StackSpot.com @StackSpot Generation and Assistance with AI: Advanced automation for

    coding, saving time and effort. Quick Commands: Efficient and fast interaction in your IDE. The power of the Hyper Contextualization: We improve from project data to generate more relevant and accurate codes. Customization with Code Snippets: We offer code suggestions and we can help the LLM to understand the rules and coding style of your company. Multiple Knowledge Sources: Leverage APIs, Event Schemas, and Custom Objects. Advanced Integration with Workspaces, Studios & Stacks AI: An integrated approach to accelerate development. Why StackSpot AI and Code Buddy?
  33. StackSpot.com @StackSpot Quer liderar a mudança? Just Ask! Junte-se a

    nós em nosso Discord Acesse o Portal StackSpot AI
  34. StackSpot.com @StackSpot Want to lead the change? Just ask! 10/02,

    exclusive for all QCon San Francisco Community Alpha launch Join us in our Discord Access the StackSpot AI Portal.