Prompt Engineering for Developers @ AWS Community Day Adria 2024

Transcript

1. It's hard to give clear instructions
2. SMS Goeben August 1914, the Mediterranean
3. TL;DR, Wait until war is declared, and destroy the Goeben Winston Churchi!, head of Admiralty, July 31, 1914
4. SMS Goeben
5. SMS Goeben
6. TL;DR, Wait until war is declared, and destroy the Goeben "… do not at this stage be brought to action against superior forces." Winston Churchi!, head of Admiralty, July 31, 1914
7. SMS Goeben
8. SMS Goeben
9. For Churchi!: Superior forces = Austrian battleships
10. I am sure that your unclear instructions never caused major disruption of someone's fleet*, but you definitely have many stories * the Goeben did that to the Russian fl"t in the Black Sea during WW1
11. Let's talk about giving clear instructions!
12. Prompt Engineering , and other people who love talking to computers for developers
13. Human communication is complicated
14. Luckily, it's a bit more clear how Large Language Models (LLMs) work
23. Source: https://platform.openai.com/tokenizer
24. Prompts: - Prompts are instructions - You te! an LLM what you want, and it tries to reply based its training and your instructions - more clear instructions = better reply - LLM always answer, but not always based on truth
25. Slobodan Stojanovic cofounder/CTO @ Vacation Tracker AWS Serverless Hero @slobodan_
26. Prompts are just instructions
27. So, how LLMs work?
28. You give your instructions
30. You get some unexpected wisdom or hallucination
32. Tokens and vectors
33. @slobodan_ Source: https://cthiriet.com/blog/infinite-memory-llm
34. Anatomy of a prompt
35. The prompt is a set of textual instructions that fit LLM's context and other limitations
36. "Who is faster: Godzi!a or T-Rex?" A valid prompt
37. "Write a 500-word article about the bad influence of Amazon's RTO policy on Lambda cold starts" Also valid prompt
40. But LLMs are products that, like most other products, evolve with user needs and requests
41. System prompts
43. Some parts of your instructions might be more important than other or you might want to make them repeatable
44. Usage via the API gives you additional superpowers
45. Repeatability
46. (simulated) conversations
47. more control
49. Successful prompts require writing
50. I know you hate writing
51. But you wrote thousands of lines of JavaScript or Python last month
52. That's also just a bunch of text and instructions
53. 7 Habits of highly effective prompters
54. Five simple tricks for better prompting results
55. 1. Hint the beginning of the answer
56. We often need a response from an LLM to follow the specific structure we defined
57. User: // Some long instructions But always reply with valid JSON and nothing else! Here's your JSON: ```json { "some": "JSON", Assistant:
58. I SAID JSON ONLY!!! Works, sometimes
59. But there's something else you can do!
60. Write the beginning of the reply in the API request!
61. User: // Your instructions Answer with valid JSON and nothing else. { " Assistant: System: // Your system prompt
62. User: // Your instructions Answer with valid JSON and nothing else. { " Assistant: System: // Your system prompt
63. User: // Your instructions Answer with valid JSON and nothing else. { " Assistant: System: // Your system prompt some": "valid", "JSON": true }
64. 2. Give examples
65. We can use the same idea for one more trick
66. ChatGPT and Claude often don't do the task perfectly immediately. It takes some back and forth before you get the desired outcome.
67. But we often try to do just one direct command via the API
68. There's a fancy name for a single direct co#and: Zero-Shot Prompting
69. Because, there's another popular fancy name: Few-Shot Prompting
70. "Great product, 10/10": {"label": "positive"} "Didn't work very well": {"label": "negative"} "Super helpful, worth it":
71. "Great product, 10/10": {"label": "positive"} "Didn't work very well": {"label": "negative"} "Super helpful, worth it": {"label": "positive"}
72. Like a mini "in-prompt" training
73. But you can also use this differently (and for more complicated use cases)
74. User: Assistant: "Great product, 10/10" {"label": "positive"} User: Assistant: "Didn't work very we!" {"label": "negative"} User: Assistant: "Super helpful, worth it" System: // Your system prompt
75. User: Assistant: "Great product, 10/10" {"label": "positive"} User: Assistant: "Didn't work very we!" {"label": "negative"} User: Assistant: "Super helpful, worth it" {"label": "POSITIVE"} System: // Your system prompt
77. 3. Think step by step
78. The next trick would work good with humans, too
79. I want you to create this unreasonable feature! And I want it now!
80. Let's think step by step. How would that feature help our users?
81. But in reality, it would probably just give you a new Linkedin employment status
82. #opentowork
83. Luckily, LLMs have no emotions
84. Adding "Let's think step by step" or a similar phrase uses more tokens for the response, but it often gives you better results
85. OpenAI O1 preview: "We've developed a new series of AI models designed to spend more time thinking before they respond."
86. OpenAI O1 preview: "They can reason through complex tasks and solve harder problems than previous models in science, coding, and math."
87. 4. Use tools
88. LLMs sucks at math and some other tasks
89. Tricks above might improve it a bit but often not enough
90. But all major LLMs support tools
91. So, instead of this:
92. User: Assistant: "Do some complex calculations" Brand new ha!ucination System: // Your system prompt
93. You can do this:
94. User: Assistant: Do some complex calculations and give me an analysis. I need an answer from calc tool, here are the arguments to pass it System: // Your system prompt
95. User: Assistant: Do some complex calculations and give me an analysis. I n"d an answer from calc tool, here are the arguments to pass it User: Assistant: // Ca!s JS function and returns result Here's the analysis System: // Your system prompt
96. Tools allow an LLM to ask you for some help
97. Tools power AI Agents!
98. But, tools are just simple functions! (JavaScript, Python, PHP, or whatever you want)
99. 5. Ask an LLM to improve the prompt
100. Can an LLM write a prompt?
101. A prompt is just a textual command. It can review and improve it!
102. Guess what? It works even better if you provide clear instructions!
103. Commander's Intent
104. Commander's Intent: - Purpose: Why personnel must complete the assignment. - Task: What the objective or goal entails. - End state: How the result should look.
105. You can try something like this: Here's my prompt: ${INITIAL PROMPT } Rewrite it to fo!ow the Co#ander's Intent Statement. Ask for missing details.
106. "Considerations in Communicating intent" Sources of Power: How People Make Decisions Book by Gary Klein
107. "There are seven types of information that a person could present to help the people receiving the request to understand what to do" Sources of Power Book by Gary Klein
108. 1. The purpose of the task (the higher level goals) 2.The objective of the task (an image of desired outcome) 3.The sequence of steps in the plan 4.The rationale for the plan 5.The key decisions that may have to be made 6.Antigoals 7.Constrains and other considerations
109. "A! seven types of information are not always necessary." Sources of Power Book by Gary Klein
110. @slobodan_ Summary
111. @slobodan_ • Hint the beginning of an answer, LLM will follow • Give examples (few-shot prompting) • Ask an LLM to think step by step • Help an LLM with tools (tools are just functions) • Ask an LLM to help you improve the prompt
112. @slobodan_
113. @slobodan_