Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deploying Drupal: Patterns and Antipatterns

Deploying Drupal: Patterns and Antipatterns

A session about the evolution of deployment process from FTP to config management, GIT workflows and automation from Drupal Dev Days Seville. Originally held at Drupal Mountain Camp in Davos.

Branislav Bujisic

March 22, 2017
Tweet

More Decks by Branislav Bujisic

Other Decks in Programming

Transcript

  1. Deploying Drupal: Patterns and Antipatterns

    View Slide

  2. Hi.
    Branislav.
    Drupal guy.

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. Continuous Deployment Cloud Hosting

    View Slide

  9. Always Be Compiling
    Why would I compile my PHP application?

    View Slide

  10. interpreted language
    (no compiling)

    View Slide

  11. Deployment before life
    became complicated

    View Slide

  12. • Open your
    favourite

    FTP client

    Deployment before life
    became complicated

    View Slide

  13. • Open your
    favourite

    FTP client

    • Replace files
    on server with
    your updated
    PHP
    Deployment before life
    became complicated

    View Slide

  14. • Open your
    favourite

    FTP client

    • Replace files
    on server with
    your updated
    PHP

    • Hit F5 in your
    browser

    • Life is good!
    Deployment before life
    became complicated

    View Slide

  15. http://www.mysite.com/login.php?user=admin&pass=admin
    Warning: fopen(/../users-and-passwords.txt) [function.fopen]:
    failed to open stream. No such file or directory in /home/
    my_site/public_html/login.php on line 56
    Warning: stream_get_contents() expects parameter 1 to be
    resource, boolean given in /home/my_site/public_html/
    login.php on line 57
    Warning: fclose() supplied argument is not a valid stream
    resource in /home/my_site/public_html/login.php on line 58

    View Slide

  16. ;php.ini
    display_errors = Off

    View Slide

  17. Every solution
    we introduce to solve any problem,

    no matter how trivial,
    increases development complexity.
    The curse of being a web developer

    View Slide

  18. Revision Control
    1982
    RCS
    (Revision
    Control System
    )
    1986
    CVS
    (Concurrent Version
    System
    )
    1989
    Sir Tim
    Berners-Lee
    invents W
    eb
    2005
    Git
    2011
    Git @
    drupal.org
    2001
    Drupal release

    View Slide

  19. Git
    • It replaced well established FTP workflows.
    • It introduced new and strange commands and
    concepts.
    • Due to steep learning curve, only a fraction of it was
    used.

    View Slide

  20. Early Git Workflow
    git add -A
    git commit -m “whatever”
    git pull origin master
    git push origin master
    ssh [email protected]
    cd /var/www/mysite
    git pull origin master

    View Slide

  21. Why is it an Antipattern?
    • Power of Git lies in branching
    • Publicly accessible .git directory is a serious security
    risk
    • SSH to server for every single pull?
    • Boring task that can easily be automated!

    View Slide

  22. post-receive hook
    #!/bin/sh
    GIT_WORK_TREE=/var/www/my_project git checkout -f
    Local git push Git /home/git/my_project.git
    Web root /var/www

    View Slide

  23. Things to Consider
    • Post-receive hook can do much more than simply copying
    the working tree.
    • Control access to repositories.
    • Multiple sites on a single server, all owned by git, all run by
    apache user is not a good idea.
    23

    View Slide

  24. SA-CORE-2014-005 -
    Drupal core - SQL injection

    View Slide

  25. Code review
    diff --git a/ds_bootstrap_layouts.module b/ds_bootstrap_layouts.module
    index b17b762..78dbe9d 100644
    --- a/ds_bootstrap_layouts.module
    +++ b/ds_bootstrap_layouts.module
    @@ -128,7 +128,20 @@ function ds_bootstrap_layouts_ds_layout_info() {
    'form' => TRUE,
    'image' => TRUE,
    );
    -
    +
    + $layouts['bootstrap_7_5_stacked'] = array(
    + 'label' => t('Bootstrap Two Columns: 7-5 stacked'),
    + 'path' => $twocol . '/bootstrap_7_5_stacked',
    + 'regions' => array(
    + 'top' => t('Top'),
    + 'left' => t('Left'),
    + 'right' => t('Right'),
    + 'bottom' => t('Bottom'),
    + ),
    + 'form' => TRUE,
    + 'image' => TRUE,
    + );
    +

    View Slide

  26. Code review
    git pull origin master
    git fetch origin feature-branch
    git checkout feature-branch
    git pull
    git diff master > my-diff-file.patch
    git checkout -b feature-branch
    git add -A
    git commit -m “Stuff”
    git push origin feature-branch

    View Slide

  27. View Slide

  28. Why is it an Antipattern?
    • it is just time consuming and boring!
    • The need for inline code commenting.
    • 100’s comments in issue queue.
    • There are some better tools for that, such as Gerrit,
    Github or Gitlab

    View Slide

  29. Pull Request Workflow
    be54c33 Created…
    f1a971a Fixed…
    1ca3582 Updated…
    fork
    Fork
    repository
    Main

    repository
    GitHub
    Local
    pull push
    pull request
    Local
    Local

    View Slide

  30. View Slide

  31. Manual deployment after code review
    fork
    Fork
    repository
    Main

    repository
    GitHub
    Local
    pull
    pull request
    push
    post-receive
    Web
    root
    Git
    repo
    Hosting

    Environment

    View Slide

  32. diff --git a/CHANGELOG.txt b/CHANGELOG.txt
    index 4853101..c015fb4 100644
    --- a/CHANGELOG.txt
    +++ b/CHANGELOG.txt
    @@ -1,4 +1,53 @@
    +Drupal 7.54, 2017-02-01
    +-----------------------
    +- Modules are now able to define theme engines (API ad
    + https://www.drupal.org/node/2826480).
    +- Logging of searches can now be disabled (new option
    + interface).
    +- Added menu tree render structure to (pre-)process ho
    theme_menu_tree()
    + (API addition: https://www.drupal.org/node/2827134).
    +- Added new function for determining whether an HTTPS
    + (API addition: https://www.drupal.org/node/2824590).
    +- Fixed incorrect default value for short and medium d
    + type configuration page.
    +- File validation error message is now removed after s
    valid
    + file.
    +- Numerous bug fixes.
    +- Numerous API documentation improvements.
    +- Additional performance improvements.
    +- Additional automated test coverage.
    +
    +Drupal 7.53, 2016-12-07
    +-----------------------
    +- Fixed drag and drop support on newer Chrome/IE 11+ v
    update
    + when jQuery is updated to 1.7-1.11.0.
    +
    +Drupal 7.52, 2016-11-16
    Meanwhile in Planet Drupal
    • Somebody updated core from 7.50 to
    7.54
    • 4501 lines of diff
    • Impossible to review
    • Entire git history soon becomes clogged
    and useless after a dozen of
    core+module updates.

    View Slide

  33. Meanwhile in Planet Drupal
    Patching (hacking) core and contrib
    Make a change
    in core/contrib
    Maintain a list of
    hacked files
    Never perform
    an update again
    Make a patch
    file
    Commit the
    patch file
    Maintain a list of
    patches
    Update a
    module
    git apply -v
    name_of.patch

    View Slide

  34. Drush make
    • Built to be a packaging tool
    • …but became a patching tool
    • Downloads Drupal core, modules and
    themes from d.o.
    • Downloads libraries in .tar.gz or .zip format
    • Fetches and applies patches

    View Slide

  35. The anatomy of makefile
    core = 7.x
    api = 2
    defaults[projects][subdir] = contrib
    projects[drupal][version] = “7.52"
    projects[ctools][version] = “1.10”
    ; remote patch
    projects[ctools][patch][] = “https://www.drupal.org/.../2839602_1.patch”

    View Slide

  36. The anatomy of YAML flavoured

    makefile
    core: 7.x
    api: 2
    defaults:
    projects:
    subdir: “contrib”
    projects:
    drupal:
    version: “7.52”
    ctools:
    version: “1.10”

    patch:
    - “https://www.drupal.org/.../2839602_1.patch"

    View Slide

  37. drush make project.make -y
    • Downloads Drupal
    • Downloads ctools into /sites/all/modules/contrib
    • Downloads and applies a remote patch

    View Slide

  38. Updating a module
    defaults[projects][subdir] = contrib
    projects[drupal][version] = “7.52”
    -projects[ctools][version] = “1.10”
    +projects[ctools][version] = “1.12”
    ; remote patch
    projects[ctools][patch][] = “https://www.drupal.org/.../2839602_1.patch"
    Commit the
    change
    Deploy to server drush make
    project.make -y

    View Slide

  39. Things to Consider
    • Prepare a backup strategy if things go wrong.
    • Running drush make on live server can and will result in
    site instability.
    • Drush make does not install Drupal or contrib modules.
    • Drush make does not run database update.
    • Drush make does not handle dependencies.

    View Slide

  40. Composer
    • Dependency management
    • Autoloader
    • Keeps the code updated
    • Feels nerdy!
    • Universal PHP tool

    View Slide

  41. The anatomy of composer.json file

    (Drupal-relevant parts)
    “require”: {

    "drupal/core": "~8.0"

    }
    “requre-dev”: {

    "behat/mink": "~1.7"

    }
    “scripts”: {

    "post-install-cmd": [

    "DrupalProject\\…\\ScriptHandler::createRequiredFiles"

    ]

    }
    “extra”: {

    “arbitrary”: “data”

    }

    View Slide

  42. Adding a module to project
    - Download Address module
    - Download Entity module
    - Download State Machine module
    - Download Inline Entity Form module
    - Download Profile module
    - Download needed libraries
    - Update composer.json and composer.lock
    composer require "drupal/commerce 2.x-dev"
    git add composer.json composer.lock
    git commit -m “Add Drupal Commerce”
    git push origin master

    View Slide

  43. Deployment to the web host
    - Download Address module
    - Download Entity module
    - Download State Machine module
    - Download Inline Entity Form module
    - Download Profile module
    - Download needed libraries
    composer install

    View Slide

  44. Composer isn’t Drush
    • Does not do patching.
    • Not aware of Drupal’s directory structure.
    • Drupal.org has its own package repository.
    • There’s some new syntax to be learned.
    • Not a speed champion!

    View Slide

  45. The fuss of setting up the project
    Getting the core
    composer create-project drupal/drupal my-d8-site 8.2.*
    Wait for 131 seconds in average
    0
    35
    70
    105
    140
    Drush Composer (warm caches) Composer (cold caches)
    131s
    31s
    12s

    View Slide

  46. The fuss of setting up the project
    Setting the right repository
    composer config repositories.drupal composer https://packages.drupal.org/8
    "repositories": {
    "drupal": {
    "type": "composer",
    "url": "https://packages.drupal.org/8"
    }
    }

    View Slide

  47. The fuss of setting up the project
    Making sure modules are downloaded to appropriate folders
    "extra": {

    "installer-paths": {

    "modules/contrib/{$name}": ["type:drupal-module"],

    "profiles/contrib/{$name}": ["type:drupal-profile"],

    "themes/contrib/{$name}": ["type:drupal-theme"]

    }

    }
    "require": {

    "composer/installers": "^1.0.20"

    }

    View Slide

  48. The fuss of setting up the project
    Enabling patching
    composer require cweagans/composer-patches
    "require": {

    "cweagans/composer-patches": "~1.0"

    }

    View Slide

  49. Applying patches
    “extra”: {
    "patches": {
    "drupal/admin_toolbar": {
    "Startup config": “https://www.drupal.org/...-2781745-3.patch”,
    "Local patch": "patches/my-patch.patch"
    }
    }
    }

    View Slide

  50. Deployment to the web host
    - Delete outdated code
    - Download new and updated code
    - Download dependencies
    - Apply patches
    composer install

    View Slide

  51. Why is it an antipattern?
    • It is not!
    • It is just time consuming and boring to set up the
    boilerplate
    • Somebody did it for us.

    View Slide

  52. Composer template

    for Drupal projects
    composer create-project drupal-composer/drupal-project
    • Installs Drupal in web directory
    • Uses composer-generated autoloader
    • Ships with functional .gitignore file
    • Composer-patches installed and set up
    • Composer set up to respect Drupal directory structure
    • Creates default writable versions of settings.php, services.yml and sites/
    default/files directory
    • Installs drush and drupal console locally

    View Slide

  53. Drush Make or Composer

    View Slide

  54. Deployment
    Config

    View Slide

  55. View Slide

  56. hook_update_N()
    Database
    mymodule.install
    in dev
    Git
    mymodule.install
    in live
    Database

    View Slide

  57. Features
    Database
    Feature

    in dev
    Git
    Feature

    in live
    Database

    View Slide

  58. It’s not all lollipops and unicorns!
    • Auto-increment indexes in Drupal 7
    • Crazy terminology: recreate, revert?
    • A feature is just a module, let’s build some custom
    logic in it!
    • Circular dependency between features
    • Disable a feature, see what will happen…
    • WE DO NOT HAVE ANYTHING BETTER IN D7 :(

    View Slide

  59. Configuration Management
    (CMI*)
    * Obviously wrong acronym, previously used for
    Configuration Management Initiative, but got stuck.

    View Slide

  60. Database
    YAML files

    in dev
    Git
    YAML files

    in live
    Database
    drush cex drush cim

    View Slide

  61. Front-end
    (Fastest growing source of complexity)
    ?

    View Slide

  62. The need to compile the front end
    • Growing number of tools, including CSS compiling,
    JavaScript compiling, asset management, image
    optimisation etc.
    • Setting up local development stacks and keeping the
    toolset same across the large team can be challenging.
    • Should front end be compiled locally and then committed,
    or vice versa?

    View Slide

  63. The final goal is
    automation

    View Slide

  64. drush dl admin_toolbar;

    drush en admin_toolbar
    drush make;

    drush features-revert-all
    composer install;

    drush config-import

    View Slide

  65. A really simple D7 deploy script
    #!/usr/bin/env bash
    cd /var/www
    drush -y make project.make
    drush -y updatedb
    drush cc all
    drush -y features-revert-all
    grunt build

    View Slide

  66. A really simple D8 deploy script
    #!/usr/bin/env bash
    cd /var/www
    composer install
    drush -y updatedb
    drush cache-rebuild
    drush -y config-import
    grunt build

    View Slide

  67. #!/usr/bin/env bash
    GIT_WORK_TREE=/var/www git
    checkout -f
    cd /var/www
    composer install
    drush -y updatedb
    drush cache-rebuild
    drush -y config-import
    grunt build
    git push
    Local
    Web host
    A simple deployment workflow

    View Slide

  68. git push
    Local
    Web host
    push event
    Let’s make it complicated
    cd /var/www
    composer install
    drush -y updatedb
    drush cache-rebuild
    drush -y config-import
    grunt build

    View Slide

  69. Let’s make it even more complicated
    git push
    Local
    Test host

    /var/www/test
    push event
    Live host

    /var/www/live
    tag push

    View Slide

  70. OMG, security made it even more complicated!
    git push
    Local
    push event
    tag push

    View Slide

  71. Things to consider
    • Site instability during builds
    • Matter of security
    • Matter of flexibility
    • Matter of scaling

    View Slide

  72. Git push
    Assemble container
    Run build hook
    Stop serving requests
    Replace container
    Run deploy hook
    Serve requests
    Site instability during builds
    Serve requests from the old
    container until the new one was
    built.

    View Slide

  73. Matter of security
    Strong isolation of services on
    network, read only file system for
    PHP files, regular updates,
    minimum OS, single point of
    password access

    View Slide

  74. Matter of flexibility
    A dedicated working environment
    for each git branch. CLI and UI.

    View Slide

  75. Matter of scaling
    Every environment is a cluster.
    Commit YAML files to git
    repository to add or remove
    services, or use web UI to request
    more resources

    View Slide

  76. Well… That Escalated Quickly

    View Slide

  77. Thank you.
    [email protected]
    @bbujisic
    http://drupal.org/u/bbujisic

    View Slide