All-in-one Serverless mit AWS und Go

Die Trendsprache im Bereich DevOps Tooling ist Go. Ob für Service Discovery, Container-Orchestrierung, Monitoring-Lösungen oder andere Tools, Go findet man in vielen Bereichen wieder. Der andere große Trend sind Serverless Architekturen, also Code einfach in die Cloud deployen, ohne sich um Server zu kümmern. Hier hat Amazon Web Services mit Lambda den Startschuss gegeben. Aktuell lässt sich Go leider nur über Umwege mit AWS Lambda zusammenbringen. Das Go-Framework Sparta (gosparta.io) bringt die beiden Trends zusammen und bietet eine integrierte Lösung für Entwicklung und Deployment von Serverless Anwendungen. Alles aus einem Guss. Alles in Go.

Transcript

1. All-in-one Serverless mit AWS und Go Andreas Mohrhard
 andreas.mohrhard@cosee.biz @amohrhard
2. What is serverless?
3. AWS Lambda Function as a Service (FaaS)
 cost estimates: serverlesscalc.com
4. Simple Queueing Service Simple Notification Service Step Functions DynamoDB Simple Storage Service (S3) API Gateway Lambda CloudWatch Events
5. $ cat where_to_use.md Driving APIs Deployments Sending mails Transforming logs Running stress tests on APIs Running stress tests on API Automating long-running
 workflows Periodic tasks Resizing images Custom resources
 for CloudFormation Running Tests in CodePipeline Data pipelines and much more… Reacting to 
 infrastructure changes Monitoring databases
 (running SQL) Analysing CloudTrail
 logs
6. $ cat requirements.md • Fast to develop (fast builds, good IDEs) • Easy to deploy • Lightweight solution • Good performance • Easy to learn • Availability of good libraries and SDKs for the ☁
7. $ cat why.go Static type system Fast tooling, fast compiles Easy deployment (static binaries) Popular for systems programming
 (also “DevOps tools”) Easy to learn
10. Configuration Serverless Build IAM Policies / Security Unit Tests Integration Tests Versioning Rollback Stress Testing Events / Triggers Deployment Complex Infrastructure
11. […] AMIIDLookup: Type: "AWS!::Lambda!::Function" Properties: Handler: "index.handler" Role: Fn!::GetAtt: - "LambdaExecutionRole" - "Arn" Code: S3Bucket: "lambda-functions" S3Key: "amilookup.zip" Runtime: "nodejs4.3" Timeout: "25" TracingConfig: Mode: "Active" AWSTemplateFormatVersion: "2010-09-09" Resources: RootRole: Type: "AWS!::IAM!::Role" Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Principal: Service: - "ec2.amazonaws.com" Action: - "sts:AssumeRole" Path: "/" RolePolicies: Type: "AWS!::IAM!::Policy" Properties: PolicyName: "root" PolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Action: "*" Resource: "*" $ cat cloudformation.yaml
12. * At least in theory. const exec = require('child_process').exec; exports.handler = function(event, context) { const child = exec('./goLambdaGo ' + ''' + JSON.stringify(event) + ''', (error) !=> { context.done(error, 'Process complete!'); }); child.stdout.on('data', console.log); child.stderr.on('data', console.error); } $ cat goWithLambda/index.js
13. $ go get github.com/mweagle/sparta
14. Demo
15. Archive: HelloWorld.zip Length Date Time Name --------- ---------- ----- ---- 20849801 11-12-2017 09:58 bin/Sparta.lambda.amd64 11822 00-00-1980 00:00 index.js 0 00-00-1980 00:00 node_modules/google-protobuf/ [more protobuf stuff] 1553 00-00-1980 00:00 sparta_utils.js 42309 00-00-1980 00:00 proto/proxy_pb.js 1421 00-00-1980 00:00 golang-constants.json --------- ------- 21562781 24 files $ unzip -l HelloWorld.zip
16. $ open architecture.svg Go Binary NodeJS HTTP Server Invocation Response protobuf Python python module built with cgo Invocation Response python + cgo about 50% faster!
17. $ cat sparta_vision.md Everthing as Code!
18. $ cat example_iam_role.go lambdaFn.RoleDefinition.Privileges = append(lambdaFn.RoleDefinition.Privileges, sparta.IAMRolePrivilege{ Actions: []string{"s3:GetObject", "s3:HeadObject"}, Resource: "arn:aws:s3!!:::MyS3Bucket", })
19. $ cat example_vault.go package main import ( "fmt" spartaVault "github.com/mweagle/SpartaVault/encrypt" ) var testKey = &spartaVault.KMSEncryptedValue{ KMSKeyARNOrGuid: "4f2f62e1-41e0-49e2-8da4-3a7ec511f498", PropertyName: "testKey", Key: “……wfAYJKoZIhvcNAQcGoG8wbQIBADse8m/G4G6iI=“, Nonce: "VDS+3LffkcSUGEpc", Value: "U4RQWOVsYyGiaJ2VhGXeWhO5Gd3+6uhaiqcg", Created: "2016-09-20T05:57:42-07:00", } func main() { plaintextValue, _ !:= testKey.Decrypt() fmt.Printf("Decrypted: %s\n", plaintextValue) }
20. func appendS3Lambda(api *sparta.API, lambdaFunctions []*sparta.LambdaAWSInfo) []*sparta.LambdaAWSInfo { lambdaFn !:= sparta.HandleAWSLambda(sparta.LambdaName(echoS3Event), http.HandlerFunc(echoS3Event), sparta.IAMRoleDefinition{}) apiGatewayResource, _ !:= api.NewResource("/hello/world/test", lambdaFn) apiGatewayResource.NewMethod("GET", http.StatusOK) lambdaFn.Permissions = […] return append(lambdaFunctions, lambdaFn) } func main() { stage !:= sparta.NewStage("prod") apiGateway !:= sparta.NewAPIGateway("MySpartaAPI", stage) apiGateway.CORSEnabled = true stackName !:= spartaCF.UserScopedStackName("SpartaApplication") sparta.Main(stackName, "Simple Sparta application", spartaLambdaData(apiGateway), apiGateway, nil) } $ cat example_apigateway.go
21. $ cat example_s3permission.go s3Lambda !:= HandleAWSLambda(LambdaName(s3LambdaProcessor), http.HandlerFunc(s3LambdaProcessor), IAMRoleDefinition{}) s3Lambda.Permissions = append(s3Lambda.Permissions, S3Permission{ BasePermission: BasePermission{ SourceArn: s3Bucket, }, Events: []string{"s3:ObjectCreated:*", "s3:ObjectRemoved:*"}, })
22. $ cat example_stepfunc.go lambdaTaskState !:= step.NewTaskState("lambdaRollDie", lambdaFn) successState !:= step.NewSuccessState("success") delayState !:= step.NewWaitDelayState("tryAgainShortly", 3*time.Second) lambdaChoices !:= []step.ChoiceBranch{ &step.Not{ Comparison: &step.NumericGreaterThan{ Variable: "$.roll", Value: 3, }, Next: delayState, }, } choiceState !:= step.NewChoiceState("checkRoll", lambdaChoices!!...). WithDefault(successState) lambdaTaskState.Next(choiceState) delayState.Next(lambdaTaskState)
23. $ cat example_custominfra.go lambdaFn.Decorator = func(lambdaResourceName string, lambdaResource gocf.LambdaFunction, template *gocf.Template, logger *logrus.Logger) error { cfResource !:= template.AddResource(s3BucketResourceName, &gocf.S3Bucket{ AccessControl: gocf.String("PublicRead"), }) cfResource.DeletionPolicy = "Delete" return nil } go-cloudformation: type-safe CloudFormation!
24. $ go test • Integrates in go test workflow (httptest.NewServer) • Manual testing with http server on localhost (go run main.go explore)
25. $ cat ci_cd.md github.com/mweagle/SpartaCodePipeline var pipelineProvisionCommand = &cobra.Command{ Use: "provisionPipeline", Short: "Provision a CI/CD pipeline for this stack", RunE: func(cmd *cobra.Command, args []string) error { validate !:= validator.New() cliErrors !:= validate.Struct(&pipelineOptions) if cliErrors !!= nil { return cliErrors } return pipeline.Provision(&pipelineOptions) }, }
26. $ cat monitoring.md [http://gosparta.io/docs/faq/]
27. $ cat sparta_conclusion.md • Full featured solution • Everything as (Go) code: Application code, infrastructure and monitoring • Sparta helps with best practices: Testing, CI/CD, Principle of Least Privileges • Alternatives • github.com/eawsy/aws-lambda-go-shim, github.com/apex/apex
28. func main() { template !:= cloudformation.NewTemplate() ports !:= [3]int{22, 80, 443} var ingressRules []cloudformation.AWSEC2SecurityGroup_Ingress for _, port !:= range ports { ingressRules = append(ingressRules, cloudformation.AWSEC2SecurityGroup_Ingress{ CidrIp: "0.0.0.0/0", FromPort: port, ToPort: port, }) } template.Resources["SecurityGroup"] = &cloudformation.AWSEC2SecurityGroup{ GroupDescription: "Illustrating generating security group entries", GroupName: "testgroup", VpcId: "vpc-1234beef", SecurityGroupIngress: ingressRules, } y, _ !:= template.YAML() fmt.Printf("%s\n", string(y)) } $ go get github.com/awslabs/goformation
29. $ cat goformation_example.go AWSTemplateFormatVersion: 2010-09-09 Resources: SecurityGroup: Properties: GroupDescription: Illustrating generating security group entries GroupName: testgroup SecurityGroupIngress: - CidrIp: 0.0.0.0/0 FromPort: 22 ToPort: 22 - CidrIp: 0.0.0.0/0 FromPort: 80 ToPort: 80 - CidrIp: 0.0.0.0/0 FromPort: 443 ToPort: 443 VpcId: vpc-1234beef Type: AWS!::EC2!::SecurityGroup
30. $ cat conclusion.md • Easy choice for DevOps automation (response times do not matter that much) • Fast startup times, performance of a compiled language • Good tooling • Easy to use, easy to integrate, easy to develop, easy to maintain
31. $ cat future.md
32. $ cat NEWS.md • Aurora Serverless - serverless sql database • Fargate - serverless containers • …. and Go!
33. > blog.cosee.biz > talks.cosee.biz > jobs.cosee.biz Thanks to @mweagle for gosparta.io!