Slide 1

Slide 1 text

Mu-Fan @ RubyConf AU 2024 Reconstructing Taiwan's Internet-Based Party Politics

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Self Introduction • Mu-Fan Teng • Ruby Evangelist in Taiwan. • Organize most Ruby Community Events in Taiwan. • CEO of 5xruby.com • Twitter / GitHub: @ryudoawaru

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

RubyConf Taiwan • 2010~ • One of Matz's "Major" Ruby Conference choices outside Japan • Featured Matz all the time after 2012

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Regular Ruby Meetups in Taiwan • Monthly Event @ PicCollage O ffi ce, Taipei. • Sharing and networking. • If you happen to be in Taiwan, please remember to join. • @rubytaiwan

Slide 8

Slide 8 text

COSCUP 2024 Ruby Track 2024/8/3~4, Taipei • Community-led agenda track at COSCUP. • Scheduled for August 3-4 at National Taiwan University of Science and Technology, Taipei. • Call for proposals now open. • CFP open now at: https:// pretalx.coscup.org/ coscup-2024/

Slide 9

Slide 9 text

AGENDA • Introduction of New Power Party and major political parties' internet participation in Taiwan. • Development history and issues of the legacy version. • Challenges and problems encountered when creating the new version.

Slide 10

Slide 10 text

New Power Party(NPP, 時代 力 量) • Was one of the major political parties in Taiwan. • Known for its internet participation. • Maximum number of active party members was around 2,000.

Slide 11

Slide 11 text

Main Features of NPP Website Membership Application Membership Due Payment Donation Internal Election CMS

Slide 12

Slide 12 text

Party Joining Process • Fill in basic information. • Pay online or O ff l ine in local branches. • Become a member after Review & Approve by Headquarter Sta ff s.

Slide 13

Slide 13 text

Membership Expiration Rules • Annual membership is similar to a subscription service. • Divided by fi rst and second half of the year. • Unpaid memberships will enter an expired state. • Membership can be restored by paying within two years. • After two years, it will change to a withdrawa state.

Slide 14

Slide 14 text

Donations Diverse Ways to Support the Party • One-time • Premium • Monthly Recurring

Slide 15

Slide 15 text

NPP’s Distinctive Approach Fully Online Elections for Core Decision-Making Bodies • Decision Making Committee (DMC) • Serves as the primary operational decision-making body. • Elects the party chairperson from among its members. • Party Representative Committee (PRC) • A statutory body required by Taiwanese party law. • Convenes semi-annually for tasks like amending the party constitution and ratifying fi nancial reports.

Slide 16

Slide 16 text

Party Election Process

Slide 17

Slide 17 text

Legacy Version Development Status • Built with Ruby on Rails from day 1. • First version completed by volunteers. • Later taken over by a full-time Junior Engineer for development. • Took over by 2021 after the last engineer quits.

Slide 18

Slide 18 text

Issues of Legacy Version—Project Management • No professional project management from the start. • Many features developed but rarely or never used. • Senior members unclear on the purpose of numerous functions. • Over-engineered features with unnecessary complexity and poor system design.

Slide 19

Slide 19 text

Issues of Legacy Version—Data Integrity • Frequent errors due to data errors. • DB Migration cannot be executed normally.

Slide 20

Slide 20 text

The History of Remaking Decision Making • Took over project maintenance in 2021, focusing on front-end and infrastructure. • End of 2021, faced challenges due to party charter amendments and dues calculation changes. • Spent considerable time on remediation and data adjustments. • Initiated project rebuild in Q3 of 2022, o ffi cially starting in October 2022.

Slide 21

Slide 21 text

Application Stack Changes Legacy New Ruby 2.6 3.1 Rails 5.2 7.0 Database PostgreSQL 9 PostgreSQL 14 Worker Sidekiq 5 + Redis GoodJob CSS Framework Bootstrap 3 TailwindCSS 3 + CSS Bundling JS Framework jQuery Stimulus JS + Importmap Deploy Tool Capistrano Docker Swarm

Slide 22

Slide 22 text

Main Improvement Goals for the New Version • Normalize the database schema. • Retain only essential functions. • Isolate election functionality, enhancing security and reliability. • Achieve >80% unit test coverage, including feature tests for critical functions. • Improve DevOps practices with CI/CD automation. • Integrate data from the legacy system into the new system.

Slide 23

Slide 23 text

Schema Improvement — User, Membership and Due

Slide 24

Slide 24 text

Use AASM to Reduce Code Coupling and Complexity

Slide 25

Slide 25 text

Before class User def set_status if ["friend", "member_unchecked", "member_unchecked_paperworked", "member"].include? (self.classi fi cation.to_s) if con fi rmed_membership_dues.any?(&:lifelong?) && !self.vip? && self.member? self.vip! self.issue_membership_number if self.membership_number.blank? update_columns membership_expires_on: LIFELONG_MEMBERSHIP_EXPIRES_ON return self.membership_number end last_due = con fi rmed_membership_dues.last # …… end end

Slide 26

Slide 26 text

After class Membership aasm column: :state do event(:pay) do before { |due| extend_by_due(due) } after_commit { |due| send_applied_mail(due) } transitions from: :applying_unpaid, to: :applying_paid transitions from: :legal, to: :legal transitions from: :expired, to: :legal end event(:cancel_applying, after_commit: :send_cancel_mailer) do transitions from: %i[applying_unpaid applying_paid], to: :applying_cancelled end end end class MembershipDue aasm column: :state do state :pending, initial: true state :paid event :pay, before: :before_pay, after: :after_pay, guard: :payable? do transitions(from: :pending, to: :paid) end end end

Slide 27

Slide 27 text

Improvement Results: Lines of Code Model Before After Di f User 1,212 190 -1,022 Membership 0 225 225 MembershipDue 369 157 -212 Total 1,581 572 -1,009 *W/O Comment, Concerns Included.

Slide 28

Slide 28 text

Issues with the Legacy Election Functions • Need to manually generate voter lists, noti fi cations, and ballots using rake tasks • Voting can be done just by accessing the ballot URL, although part of the ballot URL uses random numbers and alphabets, the identity of the voter cannot be ensured. • Ballot data is not encrypted, so risks cannot be ensured in case of data leakage.

Slide 29

Slide 29 text

Election System Refinement Plan • Separate the election functionality as an independent website. • Ensure compatibility with both new and old website logins and data. • Complete the development within one month due to unexpected political schedule changes.

Slide 30

Slide 30 text

OAuth Sign In • Use Doorkeeper Gem to establish OAuth Authorization Server on both legacy & new sites. • Respond with the same OAuth Raw Info through API.

Slide 31

Slide 31 text

Member Data Integration • Synchronize member data from the o ffi cial website for voter roll creation. • Initially used database syncing from the legacy version to the election site’s Voter Model due to tight development schedules. • Adopt RESTful integration with the new o ffi cial website for subsequent updates.

Slide 32

Slide 32 text

Encrypting Ballot Data • The ballot data stores the voter's member number. • Use ActiveRecord Encryption to encrypt the ballot data to prevent leakage of voting details.

Slide 33

Slide 33 text

Automate All Batch Processing • Write ActiveJob to do the batch tasks. • Use ActiveJob's set_wait to schedule corresponding Jobs based on the creation time of the election data.

Slide 34

Slide 34 text

The Test Coverage • Developed by 2 members in 1 month. • Achieved 81% test coverage.

Slide 35

Slide 35 text

DevOps Improvement

Slide 36

Slide 36 text

Concrete Goals • When submitting a Pull Request: • Automatically scan and check the code for syntax and security issues. • Automated testing. • Automatically generate an executable environment for User Testing (Review Apps). • When a Pull Request is merged into the main branch: • Can automatically deploy to staging environment. • Can semi-automatically deploy the program to the production through CI/CD Job.

Slide 37

Slide 37 text

Actions Needed to Achieve Goals • Dockerize the project. • Write CI/CD pipeline jobs to automate the work fl ow. • Choose an appropriate runtime stack.

Slide 38

Slide 38 text

Gems For Dockerize The Project Quickly • boxing:Generate a Docker fi le for the project. • openbox:Generates the Entrypoint for Docker. • liveness:Generates a health check endpoint.

Slide 39

Slide 39 text

boxing Gem • https://rubygems.org/gems/boxing • Multi-stage builds. • Use Alpine Linux to compact the image size. • DSL & Ruby way to set up Docker fi le. # con fi g/boxing.rb Boxing.con fi g do |c| c.revision = true c.sentry_release = true c.health_check = true c.build_packages = %w[gcompat] end

Slide 40

Slide 40 text

openbox Gem • https://github.com/elct9620/openbox • Checks DB connection automatically before starting any command • Easy to extend new commands • Use the same entrypoint with di ff erent entrypoint commands # Docker fi le ENTRYPOINT ["bin/openbox"] # docker-compose.yml version: '3.8' x-application: &application image: "${IMAGE_NAME}:$ {IMAGE_TAG}" #... services: # Default entrypoint: 'bin/openbox server' application: <<: *application good_job: <<: *application command: good_job

Slide 41

Slide 41 text

liveness Gem • The Rack middleware to provide health check endpoints • Con fi gurable by DSL • Easy to extend check provider and add endpoint security like token and IP whitelist # con fi g/routes.rb Rails.application.routes.draw do mount Liveness::Status => '/status' end # con fi g/initializers/liveness.rb Liveness.con fi g do |c| c.add :postgres, timeout: 10 c.add :redis, timeout: 10 end

Slide 42

Slide 42 text

Docker Compose File version: '3.8' x-application: &application image: "${IMAGE_NAME}:${IMAGE_TAG}" environment: - AUTO_MIGRATION=yes services: good_job: <<: *application deploy: replicas: ${WORKER_REPLICAS:-1} command: good_job healthcheck: test: ["NONE"] application: <<: *application deploy: replicas: ${APPLICATION_REPLICAS:-1} labels: - trae fi k.enable=true - trae fi k.http.routers.${DEPLOY_NAME}-https.rule=Host(`${DEPLOY_DOMAIN}`) - trae fi k.http.routers.${DEPLOY_NAME}-https.entrypoints=https - trae fi k.http.services.${DEPLOY_NAME}.loadbalancer.server.port=3000 networks: - trae fi k-public

Slide 43

Slide 43 text

Review Apps Provide an automatic live preview of changes made in a feature branch

Slide 44

Slide 44 text

Predefined CI/CD variables review:deploy: when: manual extends: .deploy stage: deploy environment: name: review/$CI_COMMIT_REF_NAME #prede fi ned variable on_stop: review:stop # Job name for stopping the app variables: # $CI_COMMIT_REF_NAME is one of prede fi ned vars DEPLOY_NAME: npp2023-$CI_ENVIRONMENT_SLUG tags: # Ensure Run on Docker Swarm - swarm-deploy only: - merge_requests needs: - docker - job: assets:precompile artifacts: true

Slide 45

Slide 45 text

Use Templates to Templatize Repeated Configurations • https://github.com/elct9620/ruby-gitlab-ci • GitLab CI templates for Ruby and Rails projects. • Contains many examples for reference. • Supports deployment via Trae fi k and Docker Swarm Mode.

Slide 46

Slide 46 text

What it Looks Like(Pull Request)

Slide 47

Slide 47 text

What it Looks Like(PR Merged)

Slide 48

Slide 48 text

Data Migration

Slide 49

Slide 49 text

Encapsulate Migration Code as a Rails Engine • The Migrator works for: 1. Migrate Database. 2. Migrate Carrierwave Attachment to Active Storage. • Developed as an independent Rails Engine Gem. gem 'legacy', git: 'https://gitlab.com/newpowerparty/legacy_migrator.git', branch: 'main'

Slide 50

Slide 50 text

Migrator Implementation: Base Model class Legacy::ApplicationRecord < ActiveRecord::Base self.abstract_class = true establish_connection ENV['LEGACY_DATABASE_URL'] scope :scope2export, -> { all } def skip?; false; end # De fi ne conditions if do want this row to be converted def migrate; raise NoMethodError; end def migrate! migrate.save!(validate: self.class.const_get(:VALIDATE_ON_MIGRATING), callbacks: self.class.const_get(:SAVE_WITH_CALLBACKS)) end class << self def after_migration; return; end def migrate! ::ApplicationRecord.transaction do scope2export. fi nd_each do |lrec| unless lrec.skip? new_record = lrec.migrate! lrec.record_migration!(new_record) end end rescue StandardError => e logger.info e.inspect end after_migration

Slide 51

Slide 51 text

Migrator Implementation: Attachments class Legacy::FileVault < ApplicationRecord mount_uploader : fi le, GeneralUploader scope :scope2export, -> { preload(:creator, :updator).all } def migrate @new_ fi le_vault = ::FileVault.new(id:, legacy_creator_name: creator&.name, legacy_updater_name: updator&.name) @new_ fi le_vault. fi le.attach(io: fi le.sanitized_ fi le. fi le, content_type: fi le.content_type, fi lename: fi le. fi le. fi lename ) @new_ fi le_vault end def skip? begin fi le.cache_stored_ fi le! # Skip the record if the fi le does not exist rescue Errno::ENOENT return true end false end

Slide 52

Slide 52 text

Use Git Token When Building Image # Docker fi le ARG APP_ROOT=/src/app ARG RUBY_VERSION=3.1.3 FROM ruby:${RUBY_VERSION}-alpine AS gem ARG APP_ROOT ARG GITLAB_TOKEN_NAME ARG GITLAB_TOKEN_TOKEN ENV BUNDLE_GITLAB__COM=$ {GITLAB_TOKEN_NAME}:${GITLAB_TOKEN_TOKEN} ...

Slide 53

Slide 53 text

On Launch Day • Migration took about 30 minutes • Including subsequent data veri fi cation, it took about 2 hours • Re-entering news and other CMS data took about 2 days • No data loss issues occurred

Slide 54

Slide 54 text

THANK YOU