Using transtats to track translation string change

Presented in Flock (the Fedora Project's annual contributor-focused conference) 2018 at The Radisson Blu Park Hotel & Conference Centre, Dresden Radebeul, Germany.

Transcript

1. Using transtats to track translation
   string change
   Flock 2018
   Sundeep Anand (suanand)
   Parag Nemade (pnemade)
   

   View Slide

2. Topics Why transtats
   ● Challenges
   ● Touchpoints
   How to use jobs
   ● Server
   ● Client
   What’s next
   ● Developments
   ● Integrations
   Get Involved
   ● Try & test
   ● Contribute
   

   View Slide

3. Why transtats?
   

   View Slide

4. Challenges
   ● How to track...
   ○ Strings of packages have been duly
   pushed/pulled to/from respective
   Translation Platform?
   ○ Latest strings are being packaged and
   build in koji?
   ● What is the volume of translation
   required for next fedora release?
   ● Can we fix major translation bugs
   reported after translation deadline,
   before final freeze?
   ● Which set of languages need attention?
   ● Is there any way to automate few steps?
   ● By comparing translation statistics derived
   from different places: upstream repository,
   translation platform and build system.
   ● Transtats release summary can tell us volume
   of translation required per languages
   (summing up statistics of all packages).
   ● YAML based job can parse packaged SRPM
   to check criticality of translation bugs.
   ● Transtats can answer - which packages are
   out of sync (has diff) and in which
   languages?
   ● By adding a few more tasks to YAML,
   automation can be achieved.
   

   View Slide

5. Touchpoints
   Software
   String Freeze
   Translation
   Deadline
   string
   change
   coverage
   translation
   update
   volume
   Out of Sync
   Packages
   languages
   need
   attention
   

   View Slide

6. How to use Jobs?
   

   View Slide

7. Transtats Jobs
   Create
   Run
   Save
   Schedule
   Share ...
   Predefined Jobs
   ● Sync Build Tags
   ● Sync Translation Platform
   ● Sync Release Calendars
   YML Based Jobs
   ● Sync Upstream
   ● Sync Downstream
   ● String Change
   ● Verify Translations
   Inspect Jobs
   ● List View
   ● Detail View
   

   View Slide

8. Sync YML Job Templates
   job:
   exception: raise
   execution: sequential
   name: upstream stats
   package: %PACKAGE_NAME%
   return_type: json
   tasks:
   - clone: git repo
   - filter: PO files
   - calculate: Stats
   type: syncupstream
   job:
   buildsys: %BUILD_SYSTEM%
   exception: raise
   execution: sequential
   name: downstream stats
   package: %PACKAGE_NAME%
   return_type: json
   tags: %BUILD_TAG%
   tasks:
   - get: latest build info
   - download: SRPM
   - unpack: SRPM
   - load: Spec file
   - unpack: tarball
   - apply: patch
   - filter: PO files
   - calculate: Stats
   type: syncdownstream
   

   View Slide

9. Track String Change YML Job Templates
   job:
   exception: raise
   execution: sequential
   name: string change
   package: %PACKAGE_NAME%
   release: %RELEASE_SLUG%
   return_type: json
   tasks:
   - clone:
   - name: git repo
   - branch: master
   - generate:
   - name: POT file
   - domain: %PACKAGE_NAME%
   - cmd: cd po && intltool-update
   --pot --gettext-package=%PACKAGE_NAME%
   - download: Platform POT file
   - calculate: Diff
   type: stringchange
   Under Development - verifytrans
   Steps:
   ● Download and unpack latest SRPM
   from build system.
   ● Create fresh translation template
   (POT file) from source tarball.
   ● Pull translations from translation
   platform.
   ● Merge pulled translations to newly
   created POT – set 1
   ● Apply patch and filter PO files –
   set 2
   ● Compare set 1 vs set 2 – find diff
   for each locale
   

   View Slide

10. if diff:
    # string change is detected
    if after_string_freeze_date:
    # it should be a breakage!
    send_warning_to_maintainer()
    if after_translation_deadline:
    # may be an error
    send_alert_to_quality_engineer()
    

    View Slide

11. Just four steps
    Pick
    Template
    Define
    Tasks
    Set
    Values
    Trigger
    Job
    

    View Slide

12. Transtats Demo
    Server Side - transtats.xyz
    Youtube https://youtu.be/D1sHpxfxrk4
    

    View Slide

13. Transtats CLI
    

    View Slide

14. Transtats APIs
    1. Ping Server: Returns server version
    GET /api/ping
    2. Package Status: Returns translation stats of package for enabled languages, for example abrt.
    GET /api/package/
    3. Graph Rule Coverage: Returns translation coverage according to graph rule, for example rhinstaller.
    GET /api/coverage/
    4. Release Status: Returns translation stats of packages which are being tracked for a given release, for
    example fedora-29.
    GET /api/release/
    5. Job Details: Returns job log against given job id, for example 2a6d4b23-6a6b-4d0e-b617-a0ece01d790f.
    GET /api/job//log
    

    View Slide

15. Transtats CLI
    ● Written in python
    ● Uses REST API
    ● Code hosted on github
    ● Provides output in json
    format as well as normal
    text format
    cli now supports 5
    commands
    ● coverage
    ● job
    ● package
    ● release
    ● version
    

    View Slide

16. Transtats CLI Demo
    Client Side - commands
    

    View Slide

17. What’s next?
    

    View Slide

18. Development Plans 1. Transtats deployment in fedora
    infrastructure.
    2. More templates in YML based
    Jobs like verifytrans etc.
    3. Better UI, to eliminate navigation
    confusions. (New Dashboard)
    4. Transtats jobs scheduling feature.
    To automate.
    5. Notifications about string
    breakage, translation deadlines
    etc.
    6. Better user guide. Improve
    docs.transtats.org
    

    View Slide

19. Transtats UI Demo
    New Dashboard
    

    View Slide

20. Integrations
    1. Automated test plugin in
    update systems like bodhi
    2. Message Queues like
    fedmsg
    3. Chat process like IRC bot.
    

    View Slide

21. Get Involved ...
    

    View Slide

22. Try & test: deployments
    Get docker daemon running.
    Build or pull transtats image (docker.io) and
    get started.
    Clone the repo and build the image
    $ git clone https://github.com/transtats/transtats.git
    $ cd transtats; sudo docker build -t transtats/transtats
    deploy/docker
    Or, Pull the image
    $ sudo docker pull docker.io/transtats/transtats
    Run the image
    $ sudo docker run -d --name container_name -p
    8080:8015 transtats/transtats
    Application should be accessible at
    localhost:8080
    Openshift
    ● Install PostgreSQL Server >= 9.5 and
    make it accessible over FQDN.
    ● Install minishift
    ● Modify the database-host in secret.yml
    which should point to PostgreSQL
    host.
    ● Run the following commands after
    you login to your cluster on terminal
    $ oc new-project transtats-deployment
    $ oc create -f deploy/openshift
    $ oc start-build transtats-build
    Once build/deploy finishes, application shall be
    accessible.
    

    View Slide

23. Lets
    Make
    Transtats
    Better
    Together...
    $ git clone transtats
    ...
    $ sudo vagrant up
    $ sudo vagrant ssh
    $ cd /workspace
    $ make run
    http://localhost:8080
    Transtats Server
    ● A simple django application
    ● Uses requests to talk to several services
    ● Few APIs written in Django REST
    Framework
    ● Forms are in django-crispy-forms
    ● Uses PostgreSQL database
    ● Uses fixture to load test data
    Transtats CLI
    ● Uses click framework
    ● Consume Server APIs
    ● Few commands
    Transtats Automation
    ● Selenium Project
    Transtats Ansible
    ● Ansible playbooks
    Project: http://transtats.org
    Docs: http://docs.transtats.org
    Source: https://github.com/transtats
    

    View Slide

24. Thank You
    #fedora-g11n
    #transtats
    feedback.transtats.xyz
    

    View Slide