How to enhance dev perf with GenAI?

During our webinar, we demonstrated how generative AI can enhance developer performance by increasing productivity and profitability. We formed a task force to assess its impact on business and skills while addressing ethical challenges. The use of tools has increased productivity by 15%, and 60% of users find it difficult to go back. Tabnine was chosen for its security. Deployment is done in stages, with ongoing training. It is essential to keep humans involved and support the change among developers.

Transcript

1. GROUPE SMILE How to enhance dev perf with GenAI? I.T IS OPEN
2. GROUPE SMILE 01 02 03 04 05 06 07 08 Who we are Why we explored the use of LLM for developers How we approach this experiment Which metrics we kept an eye on Results of our experiment How to choose a solution that works best for your company Demo A couple of important things to keep in mind before drawing conclusions Sommaire principal I.T IS OPEN 2
3. I.T IS OPEN Who we are 01
4. 4 Holistic expertise & convergence of businesses & technologies Open, flexible, high-performance architectures MVP to Digital Factory The essential levers for combining experience and operational performance Conseil Stratégie & Business Technologie & Innovation UX Design & UX Data Vision & Sustainability Feasibility Desirability & Uses E-Commerce experience Data & IA Mobile app. experience Content experience Cloud & Cybersecurity Activation & Loyalty XP & PERF. XP & PERF. We do IT all to make you Smile! I.T IS OPEN 02
5. TASK FORCE (MANAGEMENT / IT / HR / LEGAL) ⇾ Creation of a task force (impact on business, skills & professions, legal) ⇾ Consideration of the challenges of responsible AI (privacy, ethics, impact on employment, open source, etc.) ⇾ Definition of a strategy: human and financial resources 5 Our AI factory offer ACCULTURATION (COMEX / BUSINESS DIVISIONS) ⇾ Monitoring and deciphering issues with the innovation team ⇾ Keynote on issues, benefits, risks, limitations, costs ⇾ Workshops to experiment (tasks, text, audio, video, etc.) ⇾ Setting up a business watch (sector, projects, etc.) IDENTIFICATION OF USE CASES ⇾ Choice of a selection of professions ⇾ Audit of current practices ⇾ Sourcing of available data ⇾ ROI study & roadmap SETTING UP A POC ⇾ Setting up a private and secure "ChatGPT like ⇾ RAG & Fine tuning ⇾ Vector search TRAINING & CHANGE ⇾ Keynote on issues, benefits, risks, limits ⇾ Workshops for experimentation (tasks, text, audio, video, etc.) ⇾ Demos and training in cross-functional tools ⇾ Communication, distribution of KPIs LARGE-SCALE DEPLOYMENT ⇾ Taking costs into account (Setup, Build, Tokens, Licenses, TMA & planning) ⇾ Solution governance, monitoring of inference costs ⇾ ROI monitoring, updates and upgrades, ML OPS I.T IS OPEN 03
6. Why we did all of that? 04 I.T IS OPEN
7. 7 Our goals This experiment around AI technologies applied to the developer profession had as a goal to demonstrate that we can : ➔ Achieve productivity gains due to time saving and increase our profitability ➔ Be as competitive as our competitors if not more so by being ahead 05 I.T IS OPEN
8. 8 Our goals But it also serve as our continuous engagement to ➔ Provide quality tools, software and hardware to our employees ➔ support them in this profound transformation of their profession ➔ explore what’s the future of work could be 06 I.T IS OPEN
9. How we did it? 07 I.T IS OPEN
10. 10 We put a proper org. in place Train + advise Review + corrects Innovation director Engagement Manager Tech Leader Developer Control Tech Expert / CTO Control Monitor KPI TEAM A TEAM B I.T IS OPEN 08
11. 11 DEC 2023 SYNTHESIS Aggregate learnings of the experiment phasis. Export KPIs and draw a representative learning curve JUN 2024 CONCLUSION Tabnine have been chosen. Price’up deployed, including AI expected productivity for some technologies. Deployment strategy initiated and ready. JAN 2024 TOOL BENCHMARK Test several solutions to identify the one that best meets our needs, offers the best ROI, and ensures strong compliance for client acceptance and legal adherence. DEC 2024 DEPLOYMENT Deploy progressively the tool on eligible projects, train collaborators and monitor use, adhesion and productivity gains. 2025 SEP 2023 EXPERIMENT Validate the interest in using AI, confirm measurable time savings in coding and handling complex tasks. Study Timeline 09 I.T IS OPEN
12. But which metric to look at? 10 I.T IS OPEN
13. Check-in attendance 13 It’s super hard to gather datas by just asking developers how they feel about the tools, what they did over the last period, etc. 11 I.T IS OPEN
14. What feature they are using 14 The more the feature is automagically pushed to the developer, the more likely they will use it. 12 I.T IS OPEN
15. Quantitative datas 15 13 I.T IS OPEN
16. How much we are automating 16 14 I.T IS OPEN
17. How people are using more advance feature 17 15 I.T IS OPEN
18. What they are trying to do 18 16 I.T IS OPEN
19. Results 17 I.T IS OPEN
20. 20 Our results & learnings 80% Of the developers notices time savings using the tool 10 Of AI generated code or documentation is approved by the developer 15% Overall productivity increase on the project 60% Of collaborators can’t imagine step back on AI 80% • Higher automation potential than expected. • Tool aids developers in writing code faster without fully replacing their work. • Side effect of quality improvement by developers documenting the code and writing unit tests • Side effect on experts sollicitation, NPS, SLA on projects, HR satisfaction with new tool 39% Time improvement on new code writing 18 I.T IS OPEN
21. GROUPE SMILE Experienced gain for Drupal teams 21 19 I.T IS OPEN
22. 22 Full datas Our experiment on Drupal The New Stack & Tidelift Study McKinsey Study Writing new code 22.50% 39.00% 35.00% Code refactoring & maintenance Not enough data 22.00% 20.00% Writing & Executing Tests No data 15.00% No data Writing or updating documentation 30.00% No data 45.00% Reviewing security issues No data No data No data Meeting, management & operations No data 14.00% No data High complexity tasks 35.00% No data 10.00% Average 29.17% 22.50% 27.50% 20 I.T IS OPEN
23. How to evaluate solutions? 21 I.T IS OPEN
24. Our main criteria of evaluation 22 1 2 3 Covers relevant required IDEs & languages State of the art LLM model High security and privacy standard 4 5 6 Covers the most relevant use cases Popular, evolutive with no deprecation risk Easy adoption & use, quick go-live & revert We have a wide range of IDE used at Smile, so we needed a solution that could work not only on one solution. Some solutions offer more than just code completion or chat, but is it useful to you? Can you use all of those functions you’ll end up pay for? A lot of open-source solutions let you integrate any kind of LLM you want. However, a lot of them aren’t designed for code generation. Nothing worse than spending time and investing money on a solution that just die a couple of months after you rolled it out. Solution editor will have access to all your project data (prompt, code context, workspace, generated code). You better want to protect this. If we are doomed to fail, better fail fast, recover fast and find another solution. 24 I.T IS OPEN
25. GROUPE SMILE 23 I.T IS OPEN
26. GROUPE SMILE 24 I.T IS OPEN
27. 27 Reference tracking Why Tabnine ? Security & Compliance Generated code is provided with its source, so the developer can see it used threw an other context and qualify it for its own need. Tabnine is committed to security and compliance. They are GDPR compliant, certified ISO9001 & AICPA SOC. They are expected to meet the EUAI Act when enforceable. IP indemnification Admin tool for tracking The data used to train Tabnine comes from permissive licenses, ensuring the tool’s use does not pose legal risks related to copyright. They are offering indemnification in case of legal issues. Tabnine provide admin tools for tracking usage and user productivity Private SaaS or on Premise And many more technical reasons Tabnine models can be deployed on private cloud infrastructure in our clients if required 01 25 LLM 4DEV
28. How we’ll roll it out? 27 I.T IS OPEN
29. 30 Senior developers and AI interested people can masterize pretty fast the tool usage. Junior ones need continuous support to grow and discover the tool. Learning is flat on the first month and progressively grow after that. We designed a roll out plan 1 week Start 3 weeks Skill Development 2 months Onboarding Initial Use Open talk with Tabnine support Bi-weekly follow-up This 4 months curve is the time to get a developper ready and meet the experiment results. Onboarding session by Tabnine First use Discovery of the tool capabilities. Main usage is code autocompletion and basic features Developers starts showing productivity gains 1 month Productivity unchanged due to time losses by exploring the tool potential Internal Certification AI Beginner Internal Certification AI Explorer Internal Certification AI Developer Weekly follow-up 28 I.T IS OPEN
30. 31 Accordingly to the experiment, the deployment first wave is targeting Drupal based new engagements. Depending on project’s launches and according to the commercial pipe, the first wave could be extended to other technologies like Front-end / Javascript projects. Deployment & next step First wave Drupal new engagements 1st wave 2nd wave 3rd wave Train Tabnine on solutions Smile Private Chat LLM Bring AI tools for other tasks in a project lifecycle with a Smile secured internal LLM chat (Chat GPT like). Ongoing ßeta test at neopixl. Expected productivity increase on tasks like specifications writing, QA test cases, reporting and many more. Tabnine can be trained by our own Smile Knowledge and best practices on solutions like Adobe Commerce that are not easy to handle by a native AI model. Initial deployment on Drupal based projects and potentially extended to technologies which have proved good results during the experiment Potential deployment on specific new solutions that have been learnt by Tabnine. Deployment on specific development technologies like Symfony, Java or other eligible solutions. 29 I.T IS OPEN
31. Some important things to keep in mind 30 I.T IS OPEN
32. Our initial thoughts 31 1 2 4 AI won’t do everything alone Enhance, not replace You need people to constantly monitor it 4 5 You need to monitor the model production too Get ready to adapt your protocol all the time We have to keep humans in the loop and make clever decisions from what is proposed by AI It's also crucial to monitor quality of the output from the LLM because it can be hallucinated suggestions or garbage The goal is to make the developers' jobs easier and more efficient, not to replace them. It’s moving fast and improves everyday. Don’t fall for FOMO. Always seek feedback from the team and adjust our approach based on their experiences and needs. 33 I.T IS OPEN
33. Developers are fed by FUD from the medias. You’ll need a lot of acculturation and kindness to overcome it. What changed 32 1 2 4 AI won’t do everything alone Acknowledge the fear of change Your developers have HUGE biais 4 5 6 Code prediction isn’t the challenge You’ll be too slow for everyone It’s not just about developers We have to keep humans in the loop and make clever decisions from what is proposed by AI. Most of the model are okay~ish to predict code on the flow. But being able to handle more complex request is key. From your exco to your developers, everyone will be so eager to use that new tech they will find you slow. But that’s how you’ll be able to win the game. Asking what tool they prefer using will not be sufficient. A lot of our developers love Github Copilot while it’s not making them as much productive as other tools. That’s an obvious one, but that kind of first experiment will unlock dozen of new use cases that are affecting more jobs than just developers. Get ready to handle it. 34 I.T IS OPEN
34. How we can help 33 I.T IS OPEN
35. How we can help 10 01 03 05 Understand your context - starting at 1 day Emmitt Guideline around your need - 3 days Design a roll out plan - 2 days 02 04 06 Acculturate - starting at ½ day Audit solutions regarding your needs - 2 days Follow up with your team - starting at 3 days Our team will fully engage with your operational methods to gain a deeper understanding of your business and workflow. Give a solid base to all your employees regarding GenAI, what to expect and how to leverage on it Benchmark the solutions on the market against your needs in order to see which one would fits your needs the most. Either a smaller pilot project or a global rollout plan for your entire organization Regular follow up with your teams to ensure they are making the most of the tool, collect feedback and monitor time saving across all the tasks I.T IS OPEN 36 Consolidate your requirements and develop a criteria matrix to assess the solutions.
36. Votre contact 34 Thibault Milan Innovation Director thibault.milan@smile.fr +352 691 544 141 I.T IS OPEN 38
37. GROUPE SMILE I.T IS OPEN