DevOps - How I develop web applications

Development of software, it's just like any other profession. You learn from more experienced people, and you become a better caftsman day by day. Your best practices and automated tools make you finish the job more efficiently and fool-proof. After a while you're working autonomosly, and you are a real problem solver; fallback on experience, and use logical reasoning.

One of the things I've learned in the past few years is, that staying up to date with latest technologies is so important! I've met a lot of developers which didn't, and they are still living in a world "why change? it works!", and that's not how life works.

Choose a framework

In the past few years a lot of new frameworks arised: AngularJS, Laravel, ElectronJS, NodeJS, Django, and the list goes on. All the frameworks have one thing in common they learn from each other. Up until today, I'm still using PHP for all my webapplications. The main reason is due to the introduction of Laravel, it has boosted the community and popularity enormously. It's our new rockstar.

Laravel

Despite the programming language you prefer, you definetly find a good framework which suits you, and contain the full list of features your project requires:

  • dependency injection
  • service providers
  • module architecture
  • rendering engine
  • optimized compilation
  • ..

Continous Integration with Travis-CI

Unit-Testing, Test-Driven-Development, E2E-Testing, Functional Testing, we have a test for everything! When your company is developing a software product which is the companies core business, you definetly want to have some quality checks, so you don't break things.

Each time you develop a new (big) feature, you run the tests, and see if everything still works as expected. This will give you peace of mind and trust, especially when you arrived in the development team recently.

How I use Travis-CI


Nowadays I'm using Travis-CI for every application I develop: C++, PHP, NodeJS, etc. When creating a new project I create a .travis.yml file, which contains all the information to build the project and run the tests. You can find an example here.

Travis-CI

The idea is that each time I commit something to my Github repository, the project is compiled again, and you receive a notification if the build passed or failed. The nice thing is that you can integrate deployment steps as well. For example you can build a docker image and deploy it to the docker hub, or you can push the new code to production.


language: php

php:
  - 5.6

services:
    - docker

before_script:
  - echo "extension = fileinfo.so" >> ~/.phpenv/versions/$(phpenv version-name)/etc/php.ini
  - echo "extension = gd.so" >> ~/.phpenv/versions/$(phpenv version-name)/etc/php.ini
  - echo "extension = phar.so" >> ~/.phpenv/versions/$(phpenv version-name)/etc/php.ini
  - echo "extension = openssl.so" >> ~/.phpenv/versions/$(phpenv version-name)/etc/php.ini
  - composer self-update
  - composer install --dev

script:
  - mkdir -p build/logs
  - phpunit --coverage-clover build/logs/clover.xml

after_script:
  - php vendor/bin/coveralls -v

notifications:
    recipients:
        - [email protected]
    irc: "chat.freenode.net#kerberos.io"
    email:
        on_success: always
        on_failure: always

after_success:
    - if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then
        cd docker;
        docker login -e="$DOCKER_EMAIL" -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD";
        if [ "$TRAVIS_BRANCH" == "master" ]; then
            docker build -t kerberos/web:latest .;
            docker push kerberos/web:latest;
        else
            docker build --build-arg APP_ENV="$TRAVIS_BRANCH" -t kerberos/web:$TRAVIS_BRANCH .;
            docker push kerberos/web:$TRAVIS_BRANCH;
        fi
      fi

Development environments with Vagrant

When I started developing software, I usually spend a day of two setting up my development environment. The hard thing was that when I've joined a new team, they used all kinds of exotic tools which where introduced by the guy who just left the company. And as you may expect, nobody knew about it, no documentation available, and everybody was blaiming the guy who left. It didn't solved my problem..

Vagrant

By using Vagrant, you can solve most of these problems. Now you can fire every developer in your company; haha. When I have a new development machine or a new collegue, I simply install Vagrant, and run a single command.


vagrant up


This will process a Vagrant file, which contains the complete set of instructions to create a development environment, which will build a 100% identical setup for every developer in your team; no more it works on my machine syndrome. The second advantage of using a tool like Vagrant, is that it creates a Virtual Machine which installs all the tools inside the VM. So nothing is installed on your machine; ofcourse you'll need a IDE e.g. Sublime Text.

This makes upgrading and updating of your tools very easy. You can simply destroy and re-create the machine; e.g. if you want to upgrade from PHP5 to PHP7.


# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.box = "laravel/homestead"

  config.vm.network "private_network", ip: "10.10.10.10"

  config.vm.synced_folder "repos/", "/var/www/"

  config.vm.provision :shell, inline: <<-SHELL
     sudo apt-get install php7.0-mongodb
  SHELL

  config.vm.provision :shell, :inline => "sudo service nginx restart", run: "always"

end

Organize your filesystem

If you have created a lot of project in the past few years, you will need a way to organize them on your machine. Ofcourse you have them all stored on a version management system like Github or Bitbucket, but when working on a lot of project simultaneously you'll have them all cloned to your machine. This is how I organize my filesystem:

  • Create a Vagrantfile for my PHP applications.
  • Clone all my projects to an archive folder.
  • Create symbol links to my work-in-progress projects.

The idea is that when I create a new project, or clone an existing one. I will move it to the archive folder, in which I organize my projects by year or any other characteristic you prefer. When I'm actively working on a project, I create a symlink from the repo folder to the archive folder. The idea is that you'll only have a limited view of your complete workspace; you can ofcourse just remove the projects from your filesystem.

Filestructure

Automated deployment with Capistrano

We've already read about Travis-CI, it allows you to verify your software and deploy it to one of your production servers. However maybe you want to have this automated deployment process under control. Deploying to production should be semi-automatic, not every commit to Github should be deployed to production.

To help you with this, you can use Capistrano, another tool which helps you to automate deployment to one of your servers. Run a single command and your new version is up and running.

How I use capistrano


For my personal project Kerberos.io, I have a cloud application which makes it able to view your security cameras from anywhere in the world. Regulary we release new features or solve bugs, and we need to push this to one of our servers.

For the cloud application we have two servers:

  • A staging server for testing
  • A production server


What we do when releasing a new version is: deploy to staging, test it and if it works deploy to production. As we don't want to SSH in all those different servers, we use Capistrano.


cap staging deploy BRANCH=5.3


Or when deploying to production.


cap production deploy BRANCH=5.3


The configuration file for a Laravel application looks like this. It will install composer and bower to install all the dependencies.


# config valid only for current version of Capistrano
lock '3.4.0'

set :application, 'kerberos-cloud'
set :repo_url, [email protected]:kerberos-io/kerberos-cloud'
set :branch, ENV['BRANCH'] if ENV['BRANCH']

namespace :environment do

    desc "Copy Environment Variables"
    task :sync do
        on roles(:app), in: :sequence, wait: 5 do
            execute :echo, "-n /etc/environment", raise_on_non_zero_exit: false
            fetch(:default_environment).each do |key, value|
                execute :echo, "'#{key}=\"#{value}\"' >> /etc/environment"
            end
            execute :service, "nginx restart"
        end
    end

    desc "Restart ngin"
    task :nginx do
        on roles(:app), in: :sequence, wait: 5 do
            execute :service, "nginx restart"
        end
    end

end

namespace :composer do

    desc "Running Composer Self-Update"
    task :update do
        on roles(:app), in: :sequence, wait: 5 do
            execute :composer, "self-update"
        end
    end

    desc "Running Composer Install"
    task :install do
        on roles(:app), in: :sequence, wait: 5 do
            within release_path  do
                execute :composer, "install --no-dev"
            end
        end
    end

end

namespace :laravel do

    desc "Setup Laravel folder permissions"
    task :permissions do
        on roles(:app), in: :sequence, wait: 5 do
            within release_path  do
                execute :chmod, "u+x artisan"
                execute :chmod, "-R 777 storage/app"
                execute :chmod, "-R 777 storage/framework"
                execute :chmod, "-R 777 storage/logs"
                execute :chmod, "-R 777 bootstrap"
            end
        end
    end

    desc "Run Laravel Artisan migrate task."
    task :migrate do
        on roles(:app), in: :sequence, wait: 5 do
            within release_path  do
                execute :php, "artisan migrate"
            end
        end
    end

    desc "Run Laravel Artisan seed task."
    task :seed do
        on roles(:app), in: :sequence, wait: 5 do
            within release_path  do
                execute :php, "artisan db:seed"
            end
        end
    end

    desc "Optimize Laravel Class Loader"
    task :optimize do
        on roles(:app), in: :sequence, wait: 5 do
            within release_path  do
                execute :php, "artisan clear-compiled"
                execute :php, "artisan optimize"
            end
        end
    end

end


namespace :bower do

    desc "Install bower"
    task :install do
        on roles(:app), in: :sequence, wait: 5 do
            within release_path  do
                execute :bower, "install --allow-root"
            end
        end
    end

end

namespace :deploy do

    after :published, "composer:update"
    after :published, "composer:install"
    after :published, "laravel:permissions"
    after :published, "bower:install"
    after :published, "environment:nginx"

end


The idea is that you have a single file called deploy.rb, which contains all the build steps. Next to that you'll have a seperate file for every environment you own, which contain the IP-address of the server deploying to.

Capistrano example

Server management with Digital Ocean

Some years ago when I created my first website, I bought a subscription at one.com. This gave me a very cheap shared hosting solution where I could deploy my website to by using an FTP-client. If you are still working like this today, you should look at Digital Ocean.

When you need more flexibility, e.g. install dependencies or special configurations, you definetly need a VPS. One of the best VPS hosts I found today is Digital Ocean. They give you a very user-friendly dashboard, on which you can create a new webserver in just a few minutes. It contains a lot of features like DNS management, automated backups and live snapshots.

Digitalocean dashboard

Monitor with New Relic

Resolving issues before your users reported them to you, is the true art of supporting services. One of the most common sentences that I've heared in my carriere so far is: No news is good news. I'm really not amused when someone says something like that, and I really dislike those people.

Knowing what's happening on the battle field is the most valuable feedback you can get. Some years ago this was a very hard practice for IT applications, however that changed with New Relic. Now you can see all the exceptions that where thrown on your server.

Newrelic errors

Next to error handling it also shows you the performance of your application, and even how long it took to execute specific methods in your code.

Newrelic errors

Wrap up

So this was it, a summarize of all the tools that I'm using nowadays. Are you using a tool which isn't in this list, and is it worthwhile noting? Let us know!

Hey, I'm Cédric a software engineer, who's motivated to broaden his horizon and to discover and learn new methodologies. With a huge interest in web technologies, artificial intelligence and image processing, I try to understand our mountainous world a little bit better.