Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using transtats to track translation string change

Using transtats to track translation string change

Presented in Flock (the Fedora Project's annual contributor-focused conference) 2018 at The Radisson Blu Park Hotel & Conference Centre, Dresden Radebeul, Germany.

Sundeep Anand

August 09, 2018
Tweet

More Decks by Sundeep Anand

Other Decks in Programming

Transcript

  1. Using transtats to track translation
    string change
    Flock 2018
    Sundeep Anand (suanand)
    Parag Nemade (pnemade)

    View full-size slide

  2. Topics Why transtats
    ● Challenges
    ● Touchpoints
    How to use jobs
    ● Server
    ● Client
    What’s next
    ● Developments
    ● Integrations
    Get Involved
    ● Try & test
    ● Contribute

    View full-size slide

  3. Why transtats?

    View full-size slide

  4. Challenges
    ● How to track...
    ○ Strings of packages have been duly
    pushed/pulled to/from respective
    Translation Platform?
    ○ Latest strings are being packaged and
    build in koji?
    ● What is the volume of translation
    required for next fedora release?
    ● Can we fix major translation bugs
    reported after translation deadline,
    before final freeze?
    ● Which set of languages need attention?
    ● Is there any way to automate few steps?
    ● By comparing translation statistics derived
    from different places: upstream repository,
    translation platform and build system.
    ● Transtats release summary can tell us volume
    of translation required per languages
    (summing up statistics of all packages).
    ● YAML based job can parse packaged SRPM
    to check criticality of translation bugs.
    ● Transtats can answer - which packages are
    out of sync (has diff) and in which
    languages?
    ● By adding a few more tasks to YAML,
    automation can be achieved.

    View full-size slide

  5. Touchpoints
    Software
    String Freeze
    Translation
    Deadline
    string
    change
    coverage
    translation
    update
    volume
    Out of Sync
    Packages
    languages
    need
    attention

    View full-size slide

  6. How to use Jobs?

    View full-size slide

  7. Transtats Jobs
    Create
    Run
    Save
    Schedule
    Share ...
    Predefined Jobs
    ● Sync Build Tags
    ● Sync Translation Platform
    ● Sync Release Calendars
    YML Based Jobs
    ● Sync Upstream
    ● Sync Downstream
    ● String Change
    ● Verify Translations
    Inspect Jobs
    ● List View
    ● Detail View

    View full-size slide

  8. Sync YML Job Templates
    job:
    exception: raise
    execution: sequential
    name: upstream stats
    package: %PACKAGE_NAME%
    return_type: json
    tasks:
    - clone: git repo
    - filter: PO files
    - calculate: Stats
    type: syncupstream
    job:
    buildsys: %BUILD_SYSTEM%
    exception: raise
    execution: sequential
    name: downstream stats
    package: %PACKAGE_NAME%
    return_type: json
    tags: %BUILD_TAG%
    tasks:
    - get: latest build info
    - download: SRPM
    - unpack: SRPM
    - load: Spec file
    - unpack: tarball
    - apply: patch
    - filter: PO files
    - calculate: Stats
    type: syncdownstream

    View full-size slide

  9. Track String Change YML Job Templates
    job:
    exception: raise
    execution: sequential
    name: string change
    package: %PACKAGE_NAME%
    release: %RELEASE_SLUG%
    return_type: json
    tasks:
    - clone:
    - name: git repo
    - branch: master
    - generate:
    - name: POT file
    - domain: %PACKAGE_NAME%
    - cmd: cd po && intltool-update
    --pot --gettext-package=%PACKAGE_NAME%
    - download: Platform POT file
    - calculate: Diff
    type: stringchange
    Under Development - verifytrans
    Steps:
    ● Download and unpack latest SRPM
    from build system.
    ● Create fresh translation template
    (POT file) from source tarball.
    ● Pull translations from translation
    platform.
    ● Merge pulled translations to newly
    created POT – set 1
    ● Apply patch and filter PO files –
    set 2
    ● Compare set 1 vs set 2 – find diff
    for each locale

    View full-size slide

  10. if diff:
    # string change is detected
    if after_string_freeze_date:
    # it should be a breakage!
    send_warning_to_maintainer()
    if after_translation_deadline:
    # may be an error
    send_alert_to_quality_engineer()

    View full-size slide

  11. Just four steps
    Pick
    Template
    Define
    Tasks
    Set
    Values
    Trigger
    Job

    View full-size slide

  12. Transtats Demo
    Server Side - transtats.xyz
    Youtube https://youtu.be/D1sHpxfxrk4

    View full-size slide

  13. Transtats CLI

    View full-size slide

  14. Transtats APIs
    1. Ping Server: Returns server version
    GET /api/ping
    2. Package Status: Returns translation stats of package for enabled languages, for example abrt.
    GET /api/package/
    3. Graph Rule Coverage: Returns translation coverage according to graph rule, for example rhinstaller.
    GET /api/coverage/
    4. Release Status: Returns translation stats of packages which are being tracked for a given release, for
    example fedora-29.
    GET /api/release/
    5. Job Details: Returns job log against given job id, for example 2a6d4b23-6a6b-4d0e-b617-a0ece01d790f.
    GET /api/job//log

    View full-size slide

  15. Transtats CLI
    ● Written in python
    ● Uses REST API
    ● Code hosted on github
    ● Provides output in json
    format as well as normal
    text format
    cli now supports 5
    commands
    ● coverage
    ● job
    ● package
    ● release
    ● version

    View full-size slide

  16. Transtats CLI Demo
    Client Side - commands

    View full-size slide

  17. What’s next?

    View full-size slide

  18. Development Plans 1. Transtats deployment in fedora
    infrastructure.
    2. More templates in YML based
    Jobs like verifytrans etc.
    3. Better UI, to eliminate navigation
    confusions. (New Dashboard)
    4. Transtats jobs scheduling feature.
    To automate.
    5. Notifications about string
    breakage, translation deadlines
    etc.
    6. Better user guide. Improve
    docs.transtats.org

    View full-size slide

  19. Transtats UI Demo
    New Dashboard

    View full-size slide

  20. Integrations
    1. Automated test plugin in
    update systems like bodhi
    2. Message Queues like
    fedmsg
    3. Chat process like IRC bot.

    View full-size slide

  21. Get Involved ...

    View full-size slide

  22. Try & test: deployments
    Get docker daemon running.
    Build or pull transtats image (docker.io) and
    get started.
    Clone the repo and build the image
    $ git clone https://github.com/transtats/transtats.git
    $ cd transtats; sudo docker build -t transtats/transtats
    deploy/docker
    Or, Pull the image
    $ sudo docker pull docker.io/transtats/transtats
    Run the image
    $ sudo docker run -d --name container_name -p
    8080:8015 transtats/transtats
    Application should be accessible at
    localhost:8080
    Openshift
    ● Install PostgreSQL Server >= 9.5 and
    make it accessible over FQDN.
    ● Install minishift
    ● Modify the database-host in secret.yml
    which should point to PostgreSQL
    host.
    ● Run the following commands after
    you login to your cluster on terminal
    $ oc new-project transtats-deployment
    $ oc create -f deploy/openshift
    $ oc start-build transtats-build
    Once build/deploy finishes, application shall be
    accessible.

    View full-size slide

  23. Lets
    Make
    Transtats
    Better
    Together...
    $ git clone transtats
    ...
    $ sudo vagrant up
    $ sudo vagrant ssh
    $ cd /workspace
    $ make run
    http://localhost:8080
    Transtats Server
    ● A simple django application
    ● Uses requests to talk to several services
    ● Few APIs written in Django REST
    Framework
    ● Forms are in django-crispy-forms
    ● Uses PostgreSQL database
    ● Uses fixture to load test data
    Transtats CLI
    ● Uses click framework
    ● Consume Server APIs
    ● Few commands
    Transtats Automation
    ● Selenium Project
    Transtats Ansible
    ● Ansible playbooks
    Project: http://transtats.org
    Docs: http://docs.transtats.org
    Source: https://github.com/transtats

    View full-size slide

  24. Thank You
    #fedora-g11n
    #transtats
    feedback.transtats.xyz

    View full-size slide