Golang + Cloud Vision API + GAE - GoCon Spring

Transcript

1. Golang + Google Cloud Platform
   Presented by @kaneshin
   04/23/2016 Go Conference
   

   View Slide

2. @kaneshin (Shintaro Kaneko)
   - Principal Engineering Manager at eureka, Inc.
   - Photographer when Not Coding
   - MacBook Air 11 Inch ʻ Camera+Lens
   

   View Slide

3. recruit.eure.jp
   

   View Slide

4. • Online Dating Service
   • Responsible for dates, relationships and marriages.
   • Development
   • Language: PHP -> Golang
   • AWS: EC2/Aurora /DynamoDB/ElastiCache/…
   • GCP: BigQuery
   • Microservices
   

   View Slide

5. Golang for
   • Web Application Framework
   • Revel -> Gin
   • O/R Mapper/DB
   • Xorm/Wizard (DB Shard Library)
   • Vendor Package Management
   • Glide
   • etc…
   

   View Slide

6. What is Google Cloud Platform?
   

   View Slide

7. Google Cloud Platform
   • Compute
   • Compute Engine
   • Container Engine
   • App Engine
   • Storage
   • Cloud Storage (Nearline)
   • Cloud Datastore
   • Cloud Bigtable
   • Networking
   • Cloud CDN
   • Cloud DNS
   • Big Data
   • BigQuery
   • Machine Learning
   • Cloud Machine Learning
   • Cloud Vision API
   • Operations, Tools, …
   

   View Slide

8. Google Cloud Platform
   • Compute
   • Compute Engine
   • Container Engine
   • App Engine
   • Storage
   • Cloud Storage (Nearline)
   • Cloud Datastore
   • Cloud Bigtable
   • Networking
   • Cloud CDN
   • Cloud DNS
   • Big Data
   • BigQuery
   • Machine Learning
   • Cloud Machine Learning
   • Cloud Vision API
   • Operations, Tools, …
   

   View Slide

9. Google Cloud Platform
   • Compute
   • Compute Engine
   • Container Engine
   • App Engine
   • Storage
   • Cloud Storage (Nearline)
   • Cloud Datastore
   • Cloud Bigtable
   • Networking
   • Cloud CDN
   • Cloud DNS
   • Big Data
   • BigQuery
   • Machine Learning
   • Cloud Machine Learning
   • Cloud Vision API
   • Operations, Tools, …
   

   View Slide

10. Cloud Vision API
    

    View Slide

11. • Understands the content of submitted images.
    • Encapsulating powerful machine learning models.
    • Classifies images into thousands of categories.
    • detects individual objects
    • faces within images
    • finds and reads printed words contained within images.
    • Official Documentation
    • https://cloud.google.com/vision/docs/
    Cloud Vision API
    

    View Slide

12. pigeon - Cloud Vision API on Golang
    

    View Slide

13. pigeon
    • A package for the Google Cloud Vision API on Golang.
    • It provides command and web app.
    • Repository
    • https://github.com/kaneshin/pigeon
    

    View Slide

14. Getting Started
    • Enabling the Google Cloud Vision API
    • API Manager > Overview > Cloud Vision API
    • Setting Up a Service Account
    • API Manager > Credentials > Create Credentials
    • Get service account credentials json file
    $ export GOOGLE_APPLICATION_CREDENTIALS
    =~/.credentials/service_account.json
    

    View Slide

15. Installation
    • The pigeon provides the command-line tools.
    • a
    • Make sure that pigeon was installed correctly:
    $ go get github.com/kaneshin/pigeon/tools/cmd/...
    

    View Slide

16. Usage - command
    # Default Detection is LabelDetection.
    $ pigeon assets/lenna.jpg
    $ pigeon -face gs://bucket_name/lenna.jpg
    $ pigeon -label https://httpbin.org/image/jpeg
    

    View Slide

17. Usage - command
    # Default Detection is LabelDetection.
    $ pigeon assets/lenna.jpg
    $ pigeon -face gs://bucket_name/lenna.jpg
    $ pigeon -label https://httpbin.org/image/jpeg
    This flag is specified detection of Cloud Vision API
    Given image file path (OK: GCS, Over http/s)
    

    View Slide

18. Label Detection
    

    View Slide

19. Text Detection (OCR)
    

    View Slide

20. Safe Search
    

    View Slide

21. Usage - web app
    # Default port is 8080.
    # Default Detection is LabelDetection.
    $ pigeon-app
    $ pigeon-app -port=8000 -- -face -label -safe-search
    

    View Slide

22. View Slide

23. Deploy to GAE
    

    View Slide

24. Deploy - pigeon
    # Requisite appengine go sdk and direnv
    $ cd /path/to/pigeon/tools/appengine
    $ direnv
    $ goapp deploy -application [your-project-id] ./src/app
    

    View Slide

25. How to use Cloud Vision API on GAE
    After That.
    

    View Slide

26. Customizable
    

    View Slide

27. import "github.com/kaneshin/pigeon"
    import "github.com/kaneshin/pigeon/credentials"
    func main() {
    // Initialize vision service by a credentials json.
    creds := credentials.NewApplicationCredentials("credentials.json")
    config := NewConfig().WithCredentials(creds)
    client, err := pigeon.New(config)
    if err != nil {
    panic(err)
    }
    // To call multiple image annotation requests.
    feature := pigeon.NewFeature(pigeon.LabelDetection)
    batch, err :=
    pigeon.NewBatchAnnotateImageRequest([]string{"lenna.jpg"}, feature)
    if err != nil {
    panic(err)
    http://kaneshin.github.io/2016/02/27/pigeon/
    

    View Slide

28. pigeon is good to learn GAE, API :)
    

    View Slide

29. App Engine
    

    View Slide

30. For what purpose?
    

    View Slide

31. Slack Slash Command
    

    View Slide

32. kaonashi
    

    View Slide

33. human and bot are
    

    View Slide

34. Scheduled Tasks
    

    View Slide

35. Notify Today’s Scheduled
    

    View Slide

36. Time Tone
    

    View Slide

37. Slack Integration
    • Private Repository
    • Logging (importance)
    • Clean Architecture
    • Easy to add new integration
    

    View Slide

38. How to use API on GAE
    

    View Slide

39. How to use API on GAE
    • Cloud Vision API
    • Calendar API
    • Cloud Storage API
    • Youtube API
    • …
    

    View Slide

40. Non GAE
    config, err := google.JWTConfigFromJSON(b, scope)
    if err != nil {
    return nil, err
    }
    client := config.Client(context.Background())
    srv, err := vision.New(client)
    if err != nil {
    return nil, err
    }
    

    View Slide

41. http.DefaultTransport and http.DefaultClient
    are not available in App Engine. See https://
    developers.google.com/appengine/docs/go/
    urlfetch/overview
    

    View Slide

42. http.DefaultTransport and http.DefaultClient
    are not available in App Engine. See https://
    developers.google.com/appengine/docs/go/
    urlfetch/overview
    

    View Slide

43. Use urlfetch.Client/Transport
    

    View Slide

44. Why urlfetch?
    • Using Google's network infrastructure
    • For efficiency and scaling purposes.
    

    View Slide

45. http.Client on GAE via OAuth2
    httpClient := &http.Client{
    Transport: &oauth2.Transport{
    Source: google.AppEngineTokenSource(ctx, scope),
    Base: &urlfetch.Transport{Context: ctx},
    },
    }
    

    View Slide

46. GAE Oriented
    • Unavailable packages including Sirupsen/logrus
    • PR: https://github.com/Sirupsen/logrus/pull/343
    • Some packages can’t access to http.Client to send a request.
    • Web Application Framework
    • Double Context Problem. (Complicated)
    

    View Slide

47. GAE Oriented - http.Client Best Practice
    type (
    // A Config provides service configuration for service clients.
    // all clients will use the {defaults.DefaultConfig} structure.
    Config struct {
    // The credentials object to use when signing requests.
    // Defaults to application credentials file.
    Credentials *credentials.Credentials
    // The HTTP client to use when sending requests.
    // Defaults to `http.DefaultClient`.
    HTTPClient *http.Client
    }
    )
    

    View Slide

48. recruit.eure.jp
    

    View Slide

49. Thank you
    Credit: NASA Earth Observatory/NOAA NGDC
    

    View Slide