Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using transtats to track translation string change

Using transtats to track translation string change

Presented in Flock (the Fedora Project's annual contributor-focused conference) 2018 at The Radisson Blu Park Hotel & Conference Centre, Dresden Radebeul, Germany.

Sundeep Anand

August 09, 2018

More Decks by Sundeep Anand

Other Decks in Programming


  1. Topics Why transtats • Challenges • Touchpoints How to use

    jobs • Server • Client What’s next • Developments • Integrations Get Involved • Try & test • Contribute
  2. Challenges • How to track... ◦ Strings of packages have

    been duly pushed/pulled to/from respective Translation Platform? ◦ Latest strings are being packaged and build in koji? • What is the volume of translation required for next fedora release? • Can we fix major translation bugs reported after translation deadline, before final freeze? • Which set of languages need attention? • Is there any way to automate few steps? • By comparing translation statistics derived from different places: upstream repository, translation platform and build system. • Transtats release summary can tell us volume of translation required per languages (summing up statistics of all packages). • YAML based job can parse packaged SRPM to check criticality of translation bugs. • Transtats can answer - which packages are out of sync (has diff) and in which languages? • By adding a few more tasks to YAML, automation can be achieved.
  3. Touchpoints Software String Freeze Translation Deadline string change coverage translation

    update volume Out of Sync Packages languages need attention
  4. Transtats Jobs Create Run Save Schedule Share ... Predefined Jobs

    • Sync Build Tags • Sync Translation Platform • Sync Release Calendars YML Based Jobs • Sync Upstream • Sync Downstream • String Change • Verify Translations Inspect Jobs • List View • Detail View
  5. Sync YML Job Templates job: exception: raise execution: sequential name:

    upstream stats package: %PACKAGE_NAME% return_type: json tasks: - clone: git repo - filter: PO files - calculate: Stats type: syncupstream job: buildsys: %BUILD_SYSTEM% exception: raise execution: sequential name: downstream stats package: %PACKAGE_NAME% return_type: json tags: %BUILD_TAG% tasks: - get: latest build info - download: SRPM - unpack: SRPM - load: Spec file - unpack: tarball - apply: patch - filter: PO files - calculate: Stats type: syncdownstream
  6. Track String Change YML Job Templates job: exception: raise execution:

    sequential name: string change package: %PACKAGE_NAME% release: %RELEASE_SLUG% return_type: json tasks: - clone: - name: git repo - branch: master - generate: - name: POT file - domain: %PACKAGE_NAME% - cmd: cd po && intltool-update --pot --gettext-package=%PACKAGE_NAME% - download: Platform POT file - calculate: Diff type: stringchange Under Development - verifytrans Steps: • Download and unpack latest SRPM from build system. • Create fresh translation template (POT file) from source tarball. • Pull translations from translation platform. • Merge pulled translations to newly created POT – set 1 • Apply patch and filter PO files – set 2 • Compare set 1 vs set 2 – find diff for each locale
  7. if diff: # string change is detected if after_string_freeze_date: #

    it should be a breakage! send_warning_to_maintainer() if after_translation_deadline: # may be an error send_alert_to_quality_engineer()
  8. Transtats APIs 1. Ping Server: Returns server version GET <transtats_server>/api/ping

    2. Package Status: Returns translation stats of package for enabled languages, for example abrt. GET <transtats_server>/api/package/<package_name> 3. Graph Rule Coverage: Returns translation coverage according to graph rule, for example rhinstaller. GET <transtats_server>/api/coverage/<graph_rule_name> 4. Release Status: Returns translation stats of packages which are being tracked for a given release, for example fedora-29. GET <transtats_server>/api/release/<release_branch_name> 5. Job Details: Returns job log against given job id, for example 2a6d4b23-6a6b-4d0e-b617-a0ece01d790f. GET <transtats_server>/api/job/<job-id>/log
  9. Transtats CLI • Written in python • Uses REST API

    • Code hosted on github • Provides output in json format as well as normal text format cli now supports 5 commands • coverage • job • package • release • version
  10. Development Plans 1. Transtats deployment in fedora infrastructure. 2. More

    templates in YML based Jobs like verifytrans etc. 3. Better UI, to eliminate navigation confusions. (New Dashboard) 4. Transtats jobs scheduling feature. To automate. 5. Notifications about string breakage, translation deadlines etc. 6. Better user guide. Improve docs.transtats.org
  11. Integrations 1. Automated test plugin in update systems like bodhi

    2. Message Queues like fedmsg 3. Chat process like IRC bot.
  12. Try & test: deployments Get docker daemon running. Build or

    pull transtats image (docker.io) and get started. Clone the repo and build the image $ git clone https://github.com/transtats/transtats.git $ cd transtats; sudo docker build -t transtats/transtats deploy/docker Or, Pull the image $ sudo docker pull docker.io/transtats/transtats Run the image $ sudo docker run -d --name container_name -p 8080:8015 transtats/transtats Application should be accessible at localhost:8080 Openshift • Install PostgreSQL Server >= 9.5 and make it accessible over FQDN. • Install minishift • Modify the database-host in secret.yml which should point to PostgreSQL host. • Run the following commands after you login to your cluster on terminal $ oc new-project transtats-deployment $ oc create -f deploy/openshift $ oc start-build transtats-build Once build/deploy finishes, application shall be accessible.
  13. Lets Make Transtats Better Together... $ git clone transtats ...

    $ sudo vagrant up $ sudo vagrant ssh $ cd /workspace $ make run http://localhost:8080 Transtats Server • A simple django application • Uses requests to talk to several services • Few APIs written in Django REST Framework • Forms are in django-crispy-forms • Uses PostgreSQL database • Uses fixture to load test data Transtats CLI • Uses click framework • Consume Server APIs • Few commands Transtats Automation • Selenium Project Transtats Ansible • Ansible playbooks Project: http://transtats.org Docs: http://docs.transtats.org Source: https://github.com/transtats