How we built an AI code reviewer with serverless and Bedrock

Slides for my talk at the Serverless London meetup on 12-Feb-2025

Transcript

1. How we built an AI Code Reviewer with Serverless and Bedrock
2. Yan Cui http://theburningmonk.com @theburningmonk AWS user since 2010
3. Yan Cui http://theburningmonk.com @theburningmonk running serverless in production since 2016
4. Developer Advocate @ Yan Cui http://theburningmonk.com @theburningmonk
5. Yan Cui http://theburningmonk.com @theburningmonk independent consultant
7. evolua.io Demo
8. Architecture
9. API Gateway EventBridge Webhook
10. API Gateway DynamoDB Bedrock EventBridge Webhook
11. API Gateway DynamoDB Bedrock EventBridge Webhook
12. API Gateway DynamoDB Bedrock EventBridge Webhook evolua.io
14. API Gateway DynamoDB Bedrock EventBridge Webhook AppSync evolua.io
15. API Gateway DynamoDB Bedrock EventBridge Webhook AppSync evolua.io
17. API Gateway DynamoDB Bedrock EventBridge Webhook AppSync evolua.io Authoriser
18. API Gateway DynamoDB Bedrock EventBridge Webhook AppSync evolua.io Authoriser
19. API Gateway DynamoDB Bedrock EventBridge Webhook AppSync evolua.io Authoriser
20. API Gateway DynamoDB Bedrock EventBridge Webhook AppSync evolua.io Authoriser
21. API Gateway DynamoDB Bedrock EventBridge Webhook AppSync evolua.io Authoriser
22. Challenges (for an AI code reviewer) Handling sensitive data for customers
23. Challenges (for an AI code reviewer) Large fi les. Large PRs with many fi les. Handling sensitive data for customers
24. Why Bedrock?
25. Security
26. Security Data is encrypted at rest.
27. www.wiz.io/blog/wiz-research-uncovers-exposed-deepseek-database-leak
28. aws.amazon.com/bedrock/faqs
29. Security Data is encrypted at rest. Inputs & Outputs are not shared with model providers. Inputs & Outputs are not used to train other models.
30. API Gateway DynamoDB Bedrock EventBridge Webhook AppSync evolua.io Authoriser Fallback Primary
31. privacy.anthropic.com/en/articles/7996885-how-do-you-use-personal-data-in-model-training
32. Serverless
33. Serverless Usage-based AND provisioned throughput pricing
36. 1M Input Tokens 1M Output Tokens $0.14 v3 r1 $0.28 $0.55 $2.19 Sonnet $3.75 $15.0 Haiku $0.80 $4.00
37. Very cost ef fi cient!
38. Very cost ef fi cient! Data is stored in China.
39. Very cost ef fi cient! Data is stored in China. Data might be used to train other models.
40. www.wiz.io/blog/wiz-research-uncovers-exposed-deepseek-database-leak
41. Very cost ef fi cient! Data is stored in China. Data might be used to train other models. Operationally immature.
43. No token-based pricing yet
44. No token-based pricing yet “GPU-based instance type like ml.p5e.48xlarge is recommended”
45. ml.p5e.48xlarge 💰💰💰💰💰💰💰💰💰💰 💰💰💰💰💰💰💰💰💰💰 💰💰💰💰💰💰💰💰💰💰 💰💰💰💰💰💰💰💰💰💰 💰💰💰💰💰💰💰💰
46. Other capabilities Guardrails Knowledge base (managed RAG) Agents Cross-region inference Model evaluations
50. API Gateway DynamoDB Bedrock EventBridge Webhook AppSync evolua.io Authoriser Fallback Primary
51. Lessons
52. Webhook
53. Webhook Analyse changes
54. Webhook Analyse changes Feedback
55. Condensed view…
57. Lambda timed out after 15 mins
58. Succeeded on automatic retry
59. Webhook Analyse changes Feedback LLM limits GitHub limits AWS limits
60. Lesson: AI is 10% of the problem
62. Reasoning ability
63. Context window Max response tokens API rate limit Reasoning ability
64. Context window Max response tokens API rate limit Reasoning ability Cost Performance
65. Context window Max response tokens API rate limit Reasoning ability Cost Performance Important selection criteria for LLMs
66. Doing cool AI stuff! Working around AI limits
67. Doing cool AI stuff! Working around AI limits Stop playing with my bowl…
68. Context window Max response tokens API rate limit Reasoning ability Cost Performance
69. Claude 3.5 Sonnet’s default throughput is 50 per minute
70. Claude 3.5 Sonnet’s default throughput is 50 per minute Can be raised to 1,000 per minute
71. Claude 3.5 Sonnet’s default throughput is 50 per minute Can be raised to 1,000 per minute Bedrock has cross- region inference
72. Mitigate API rate limit Raise account limits. Use Bedrock cross-region inference.
73. Mitigate API rate limit Raise account limits. Use Bedrock cross-region inference. Limit no. of parallel requests per PR.
74. Mitigate API rate limit Raise account limits. Use Bedrock cross-region inference. Limit no. of parallel requests per PR. Fallback to Anthropic & less powerful models (Claude 3 Sonnet, Claude 3.5 Haiku)
75. Future work: incorporate other models (Nova, DeepSeek, etc.)
76. Future work: incorporate other models (Nova, DeepSeek, etc.) Also good for cost control!
77. Lesson: LLMs are still quite expensive
79. Dif fi cult to build a sustainable and competitive business
80. Cost control Only analyse changed lines.
81. Cost control Only analyse changed lines. Good for cost control Good for UX
82. Cost control Only analyse changed lines. Limit free users to few PRs per month.
83. API Gateway DynamoDB Bedrock EventBridge Webhook
84. API Gateway DynamoDB Bedrock EventBridge Webhook Built-in retries & DLQ
85. Lambda timed out after 15 mins
86. Lambda timed out after 15 mins Reprocess fi les on retry…
87. Lambda timed out after 15 mins Reprocess fi les on retry… Duplicated side- effects (e.g. Github comments)
88. Cost control Only analyse changed lines. Limit free users to few PRs per month. Use checkpoints to avoid re-processing fi les on retries
89. const issues = await executeIdempotently( `${event-id}-${filename}-analyze`, () => analyzeFile(file) ); ... await executeIdempotently( `${event-id}-${filename}-add-gh-comment`, () => addReviewComment(filename, comment) );
90. Webhook Analyse changes Feedback Why not Step Functions?
91. Webhook Analyse changes Feedback Why not Step Functions? Checkpoints is just easier 🤷
92. Lesson: Latency is a challenge
93. Models take 10s of seconds to analyse each fi le
94. Wasted CPU cycles in Lambda
95. Future work: try other models
96. Future work: make use of these CPU cycles
97. Lesson: Be ware of hallucinations
98. “Give me JSON in this format”
100. “Give me JSON in this format” “Nope!”
102. Non-existent codes, invalid URLs
103. Non-existent line numbers
104. Future works
105. Go to evolua.io to try it out. We’d love your feedback!
106. Questions?