JQ provides this nifty env input that enables you to reference environment variables using the JQ style query syntax.

# prints all envs in JSON format
jq -n env

And you can use piping like you would with a normal JSON input.

# print $HOME in raw format
jq -nr 'env | .HOME'

Capture CLI to a variable in a bash script without redirecting STDOUT.

OUTPUT=$(command --foo "${BAR}" | tee >(cat - >&2))

JQ is a powerful command-line JSON processing tool. It's super fast (C), provides solid documentation, and is easy to use. In this example, given a JSON object from a file, curl response; we look through an array nested inside an object, and return a matching object within the array. I want to lookup the id of an item, given the name.

{
    "ok": true,
    "items": [
        {"id": 123, "name": "thing-1"},
        {"id": 124, "name": "thing-2"},
        {"id": 125, "name": "thing-3"}
    ]
}

We can pipe a filter result within the JQ sequence with the | character, just like a standard shell interface. The select command then searches a match pattern within an iterable array.

cat file.json | jq '.items[] | select(.name == "thing-3") | .'

The final select result is piped and formatted as needed. For example, we can use the -r (raw output) option to return unformatted (not quoted) output, and ONLY the id field.

cat file.json | jq -r '.items[] | select(.name == "thing-3") | .id'

There are numerous useful operators and functions provided in the documentation. JQ also provides a fantastic playground here.

I had a requirement for a bash script to check for required variables before running a function. Rather than creating a conditional block for each required variable (or in my case, needing to dynamically change them), let's do something that's more terse and maintainable. The example covers a few bash concepts:

• Generating an array of variable names
• Variable name as a string (indirect parameter expansion)
• Breaking a loop over an array (control flow)

#!/bin/bash

ERROR=0
declare -a REQUIRED=(
  "APP_VAR_1" "APP_VAR_2" "APP_VAR_3"
)
for REQ in "${REQUIRED[@]}"
do
  if [ -z "${!REQ}" ]; then
    ERROR=1
    printf "\n  - ERROR: ${REQ} undefined.\n\n"
    break
  fi
done

run ()
{
  printf "\n  - Validated.\n\n"
}

if [ "${ERROR}" == 0 ]; then
  run
fi

The ${REQUIRED[@]} syntax sets up the iterable REQ variable. We can access the literal name of the variable with ${REQ} syntax. Whereas, we can access the value of the variable with ${!REQ} indirect parameter expansion. If all the required variables are not -z null, the run() function will continue as expected.

Ansible provides some excellent utilities for maintaining single lines and partial blocks of text. Both modules have support for handling template and fact variables, and a variety of options to support your use case. Here are a couple of examples using each module.

Using the lineinfile module for a single line. This module will not support inserting newline \n characters:

- lineinfile: >
    dest=/etc/memcached.conf
    regexp='^-m [\d]*'
    line='-m {{ memory }}'
    state=present

Using the blockinfile module for multiple lines:

- blockinfile:
    dest: /etc/hosts
    content: |
      127.0.0.1 {{ ansible_hostname }}.local
      ::1       {{ ansible_hostname }}.local
    state: present

The blockinfile module offers several useful options such as:

• insertafter and insertbefore to manage exactly where you need the block to be inserted. Useful for structured files like XML.
• marker to customize block markers, allowing you to manage multiple blocks in the same file.

In my case, I have a much larger block I'd like to be able to maintain using a separate Jinja2 template file. For this we will need to use the more advanced lookup plugin, and capture the template content.

- set_fact:
    hosts_content: "{{ lookup('template', 'templates/etc-hosts.j2') }}"

- blockinfile:
    dest: /etc/hosts
    content: '{{ hosts_content }}'
    state: present

In this manner, you can keep your tasks configuration file concise by storing large blocks of content more appropriately in a separate file.

A Single Line Shell Solution

In some cases, Ansible might be overkill. If you only need to ensure that a single line exists, and the order within the file does not matter, you can use something like this simple one-liner. This will check if the line exists, and append it to the end of the file if it does not.

Append line to file if it doesn't exist:

LN="127.0.0.1 dev-local"
grep -q -F "${LN}" /etc/hosts || echo "${LN}" | sudo tee -a /etc/hosts

Important: this is a potentially destructive command. Please proceed cautiously and at your own risk.

I'll sometimes leave local git branches around a bit longer than I should. And, depending on your workflow, you may occasionally have a handful of ephemeral branches that need to be cleaned up.

In this example, we exclude the active branch with grep invert match (not) * and filter out any additional branches with an escaped \| pipe (master, staging); this is only necessary if your match pattern includes branches you want to keep. Then we match the prefix feature- for the branches we intend to delete.

git branch -D \
  $(git branch | grep -v '*\|master\|staging' | tr -d ' ' | grep -E '^feature-')

Modify the git branch command options as needed. For example, you can specify to only include previously merged branches, etc.

PHP 7 has been out long enough now that it has seen a couple of patch releases, which is about the time I will start evaluating an upgrade. PHP 7 can have some significant performance benefits from previous versions, so I was eager to give it a try. With the php-gearman extension now available for PHP 7, my prerequisites have been met.

Install latest PHP 7.0 on Ubuntu 14.04

sudo add-apt-repository -y ppa:ondrej/php
sudo apt-get update  
sudo apt-get -y install php7.0 php7.0-fpm php-gearman

Install additional extensions as needed

# common extensions
sudo apt-get -y install php7.0-curl php7.0-json php7.0-mcrypt php7.0-xml
# memcached and mongodb extensions
sudo apt-get -y install php-memcached php-mongodb

This PPA is maintained by Ondřej Surý. I have used his PHP Debian packages for several years without issue.

After making the jump to Ubuntu as my preferred distribution, I've admittedly become addicted to bash completions (aka autocomplete, tab completion, typeahead). Bash completions provide immediate hints for common commands, and even the options associated with them.

For example, a quick \t\t (double-tap of the tab key) after the service command will provide a quick snapshot of services available on the machine, beginning with your query.

someone@example:~$ sudo service a
acpid  anacron  apache2  atd

As with most things, seemingly small details, like typeahead completions, can make for a very useful productivity improvement.

Consul

I have been utilizing the Consul agent from HashiCorp for service discovery inside an AWS VPC. Consul is a well-thought-out, distributed, service discovery agent that accomplishes a few things quite elegantly:

• Determines location of services via CLI and DNS
• Keeps availability status up-to-date with health checks
• Stores facts that can be shared across instances
• Provides performant localhost access to everything
• Easy to setup, highly configurable, and resilient

Though individually these features seem pretty straight-forward, the combined elements of service discovery can be incredibly complex in dynamic environments, and challenging to keep synchronized. You can read more about Consul on their website.

Bash Completion

Bash completion on Ubuntu provides a configuration directory for adding your own complete handlers. In this example, you will need to have the consul agent binary accessible in your path.

someone@example:~$ consul members
Node       Address          Status  Type    Build  Protocol  DC
dev-web    10.10.0.17:8301  alive   server  0.6.4  2         dc-east
dev-mysql  10.10.0.44:8301  alive   client  0.6.4  2         dc-east
qa-web     10.10.0.63:8301  failed  client  0.6.4  2         dc-east

The consul members command will provide a list of nodes alongside a status of: alive, failed, or left. You may choose to filter out different statuses from your completion. I only filter out left because it indicates a proper shutdown and removal for my use case. The failed status nodes are something I would still want access to.

We will create our completion script here:

# /etc/bash_completion.d/ssh_consul

With the following contents:

#!/bin/bash
_consul_members()
{
  local cur
  local res
  cur=${COMP_WORDS[COMP_CWORD]}
  res=$(consul members | grep -v ' left ' | tail -n +2 | cut -f1 -d ' ' | sort -u)
  COMPREPLY=($(compgen -W "${res}" -- ${cur}))
}
complete -F _consul_members ssh

By default, Consul uses the node.consul domain to access nodes. You have some options for resolving the domain:

• Make the domain accessible using the consul local DNS
• Use your own hostnames as the full name of the node
• Or, suffix the COMPREPLY variable with an existing domain (hack)

I have setup dnsmasq to forward the .consul domain requests to the consul DNS server. And, added domain search resolution here:

# /etc/resolvconf/resolv.conf.d/base
search node.consul

This will attempt to resolve hostnames matching the name of the consul node, for example: ssh dev-web would attempt to resolve the hostname {dev-web}.node.consul through the local DNS server.

Putting It All Together

After saving the completion config, we'll need to reinitialize our bash session for it to take effect. Now our ssh command will provide immediate and accurate insights into what instances are available.

someone@example:~$ ssh dev-
dev-foo-20  dev-foo-21  dev-mysql  dev-web

Gentoo has provided some excellent documentation for bash completions here. You can write some pretty advanced completion scripts, if one were so inclined.

I highly recommend managing your Consul implementation with Ansible, or a similar configuration management tool. If you have questions or suggestions, please feel free to get in touch via the appropriate social outlet.

If you need to generate a CSR, OpenSSL has a helpful prompt interface for completing the required fields one at a time. However, if you are using automation, collecting STDIN is not always an option.

While working with Ansible, I learned that OpenSSL CSR generation allows you to pass the appropriate details using the -subj parameter like in this example:

# C:  Country Code
# ST: State (Region)
# L:  Locality (City)
# O:  Organization
# OU: Organizational Unit
# CN: Common Name

openssl req \
  -subj "/C=US/ST=Ohio/L=Columbus/O=Acme Company/OU=Acme/CN=example.com" \
  -newkey rsa:2048 -nodes \
  -keyout example_com.key \
  -out example_com.csr

Now you can continue uninterrupted without needing to manually enter details anytime you generate a CSR.

With Google sunsetting the SPDY protocol, and broad support for HTTP/2 shipping with most modern browsers, I began investigating moving our SPDY support over to HTTP/2.

Nginx recently released official support for HTTP/2 with the mainline repository version 1.9.5. While upgrading to the new release and enabling the http2 module we ran into an issue with transport security, as reported by Google Chrome:

# google chrome client error
ERR_SPDY_INADEQUATE_TRANSPORT_SECURITY
# or more generically 
INADEQUATE_SECURITY (0xc)

According to the HTTP/2 specification, over TLS 1.2 HTTP/2 SHOULD NOT use any of the cipher suites that are listed in the cipher suite black list, found here.

The following settings (with changes to the ssl_ciphers directive) addressed the transport security issue reported by the browser.

server {

    listen 9443 ssl http2 proxy_protocol;
    port_in_redirect off;
    real_ip_header proxy_protocol;
    set_real_ip_from 10.0.0.0/16;
    server_name acme.com www.acme.com;

    ssl on;
    ssl_certificate /path/to/full-chain.pem;
    ssl_certificate_key /path/to/pivate-key.pem;
    
    # disable unsupported ciphers
    ssl_ciphers AESGCM:HIGH:!aNULL:!MD5;
    
    # ssl optimizations
    ssl_session_cache shared:SSL:30m;
    ssl_session_timeout 30m;
    add_header Strict-Transport-Security "max-age=31536000";

}

In the example, we are also opting into the HTTP Strict Transport Security (HSTS) enhancement to prevent client communications from being sent over standard HTTP. The proxy_protocol configuration in use by the load balancer (ELB) continues to work as expected without any changes. You can read more about using proxy protocol with Nginx here.

Like anyone who spends a good deal of time in a terminal window, switching between machine instances is fairly commonplace. I keep a persistent tmux session open to manage a handful of connections.

Knowing which machine you're currently using is obviously imperative, but can be little challenging in highly dynamic environments like AWS.

I have implemented Ansible for managing configuration on a large number AWS instances. The following script applies a dynamic hostname to the instances based on a predefined prefix.

The playbook should look something like this:

- hosts: dev
  remote_user: user
  sudo: yes
  vars:
    env: dev
    hostname: dev-web-*
  roles:
    - common
    - supervisor

We will dynamically replace the ampersand * character with the tail of the host IP address. For our environment, hosts are commonly identified by their roles and the last /32 of the address within the current subnet.

We will add the following tasks to the common role:

# set the friendly hostname from the playbook variable
- name: set hostname
  hostname: name={{ hostname.replace('*', ansible_all_ipv4_addresses[0].split('.')[3]) }}

# add the new hostname to the hosts file
- name: add to hosts
  lineinfile: >
    dest=/etc/hosts
    regexp='^127\.0\.0\.1'
    line='127.0.0.1 localhost {{ hostname.replace("*", ansible_all_ipv4_addresses[0].split(".")[3]) }}'
    state=present

The hostname module will work for just about any distribution (in my case, Ubuntu) you might be using. Once you have run the playbook, your hostname should look something like:

ubuntu@dev-web-62:~$

The new hostname dev-web-62 is much more meaningful than the default hostname provided by EC2. You can use this same pattern to configure DNS for each instance, and update the tag Name for better readability in the AWS web console.

Update: this also works as expected with macOS Sierra.

The El Capitan release of OS X introduces a more strict security model around the concept of root level access to the underpinnings of the operating system. For a typical user, this is great news for avoiding malware, etc.

As a developer, this requires a little more thoughtfulness when trying to behave as root. Even with the sudo command, there are protected locations and processes that will no longer work as expected. To be specific /usr/bin for this example.

This command will NOT work in OS X 10.11:

curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/bin --filename=composer

Yields an error, unable to write to the global path.

Downloading...
Could not create file /usr/bin/composer: fopen(/usr/bin/composer): failed to open stream: Operation not permitted
Download failed: fopen(/usr/bin/composer): failed to open stream: Operation not permitted
fwrite() expects parameter 1 to be resource, boolean given

Instead, let's write to the /usr/local/bin path for the user:

curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer

Now we can access the composer command globally, just like before.

I previously published instructions on using a specific commit hash for Composer. This is a quick and useful way for referencing the most recent version of our work during the development process.

In some cases, specifying a commit hash as the current version may compromise the dependency tree, causing a conflict like:

Your requirements could not be resolved to an installable set of packages.

This can be caused by a conflicting version with a child dependency:

acme/acme-app @ 1.0.0
requires:
 -- guzzlehttp/guzzle @ 5.0.3
 -- acme/acme-service @ 1.0.0
    requires:
    -- guzzlehttp/guzzle @ 5.0.1

In this case we have another dependency acme/acme-service that requires the guzzlehttp/guzzle package at a different version than the repository we are working in. Let's say we need 5.0.3 in the application because it fixes an issue we are currently working on.

As long as the API for the newer version is compatible with the older one (usually the case for most minor/patch changes), we can satisfy the acme/acme-service dependency by telling the package to use version 5.0.3 as 5.0.1:

{
    "require": {
        "guzzlehttp/guzzle": "5.0.3 as 5.0.1",
        "acme/acme-service": "1.0.0"
    }
}

We can also reference a commit hash as an alias:

{
    "require": {
        "guzzlehttp/guzzle": "dev-master#d61a4539f57d65620785604b8b380890225c518e as 5.0.1",
        "acme/acme-service": "1.0.0"
    }
}

You can learn more about Composer Aliases here.

I do not generally use this type of version aliasing in a production scenario if it can be avoided. There are some obvious pitfalls with testing and general confusion. However, it can be useful for quickly resolving a conflict during the development process.

Is it possible to serve multiple domains (each with a unique SSL certificate) via HTTPS behind a single load balancer on AWS?

Yes you can; with TCP and Proxy Protocol.

Proxy Protocol allows you to safely and transparently forward TCP (layer 4) requests while attaching upstream client address information. More details are available in the HAProxy abstract about how it all actually works. As of July 2013, Amazon Web Services ELB (Elastic Load Balancer) supports distributing TCP connections with Proxy Protocol.

In a typical ELB/HTTPS setup, the SSL connection is negotiated by the load balancer and HTTP traffic is forwarded on to the web server. If you are only hosting one top-level domain, this setup works just fine. For our use case, we need to handle HTTPS connections for multiple domains on a single application stack. This means multiple certificates, which a single ELB instance does not support.

For our setup, SSL negotiation will be done by nginx on the web server, rather than by the ELB. With nginx, leveraging multiple server blocks each with its own SSL certificate is pretty straight forward. Here is what you will need:

• Nginx 1.6.2
• Ubuntu 14.04 LTS
• AWS CLI

Install Prerequisites

The nginx PPA includes the required modules, so there is no need to compile a build. Feel free to adjust to your own requirements.

# install nginx
sudo add-apt-repository -y ppa:nginx/stable
sudo apt-get update
sudo apt-get -y install nginx=1.6.*

The AWS CLI will require credentials provided by your account.

# install aws cli
sudo apt-get install awscli
aws configure

Create and Configure the Load Balancer

If you are also (likely) handling standard requests over port 80, you do not need to enable Proxy Protocol for non-secure traffic. The HTTP traffic can remain unaffected while adding HTTPS to an existing ELB.

First, we need an ELB instance. If you do not already have a load balancer, you can create one using the AWS console, or by following these instructions for AWS CLI. In the example, we use acme-balancer as the ELB name and we are forwarding to backend port 9443.

The listener port should be created using the TCP protocol for both the Load Balancer Protocol and the Instance Protocol. The application layer protocol (HTTPS) is not handled until we reach the nginx instance. In most cases, the public port should be the standard 443.

# create proxy protocol policy
aws elb create-load-balancer-policy \
  --load-balancer-name acme-balancer \
  --policy-name EnableProxyProtocol \
  --policy-type-name ProxyProtocolPolicyType \
  --policy-attributes AttributeName=ProxyProtocol,AttributeValue=True

# add policy to elb
aws elb set-load-balancer-policies-for-backend-server \
  --load-balancer-name acme-balancer \
  --instance-port 9443 \
  --policy-names EnableProxyProtocol

# results
aws elb describe-load-balancers --load-balancer-name acme-balancer

Configure Nginx with Proxy Protocol

If you have multiple server blocks running on the same port (virtual hosts), any port that includes proxy_protocol in your nginx configuration will enable proxy protocol handling for ALL traffic on this port, not just the particular server block.

# block for proxy traffic
server {

    # port elb is forwarding ssl traffic to
    listen 9443 ssl proxy_protocol;
    
    # sets the proper client ip
    real_ip_header proxy_protocol;
    
    # aws vpc subnet ip range
    set_real_ip_from 10.0.0.0/16;
    
    server_name acme.com www.acme.com;
    
    ssl on;
    ssl_certificate /etc/ssl/acme/acme.com.crt;
    ssl_certificate_key /etc/ssl/acme/acme.com.key;

}

# block for direct traffic
server {

    listen 443 ssl;
    
    server_name acme.com www.acme.com;
    
    ssl on;
    ssl_certificate /etc/ssl/acme/acme.com.crt;
    ssl_certificate_key /etc/ssl/acme/acme.com.key;

}

And that's it. If the real IP settings are working correctly, you should not need to setup a custom log format. I have chosen to forward proxy protocol requests to a non-standard port (9443 in the example). This provides me access to the standard port outside of the load balancer for troubleshooting and debugging.

Creating separate server blocks for direct and proxied traffic is more verbose, but has a few benefits. It mitigates the need for conditional blocks later down the road. I also find that it is easier for others to understand. Lastly, separate blocks allow you to maintain a distinct configuration to test direct traffic to production nodes before they are added to the load balancer.

Putting It All Together

This example is obviously a summation of the required changes, and may vary greatly depending on your AWS setup, with respect to subnets, ports, security groups, etc. If you have questions or suggestions, please feel free to get in touch via the appropriate social outlet.

JavaScript interaction is not usually an option for most modern websites (try accessing some of your favorite sites with JavaScript disabled). Given the matrix of browsers, devices, software versions and extensions, exhaustive testing can be challenging.

Capturing and logging JavaScript errors that occur on the client can be just as important as inspecting application and server logs.

The following example assumes you have access to jQuery, and adds an error event handler to the current window. In the case of an error event, we dispatch a POST request with the pertinent details.

(function() {

    var logError = function( e )
    {
        if (window.jQuery)
        {
            var data = {
                f : e.filename,
                l : e.lineno,
                m : e.message
            };
            $.post('/log/error', data);
        }
    };

    if (window.addEventListener)
    {
        window.addEventListener('error', logError, false);
    }
    else
    {
        window.attachEvent('onerror', logError);
    }

})();

Depending on your application, additional information about the error can be logged server-side: referring URL, user agent, timestamp and possibly session information. Capturing these errors should give you better insight into where visitors may be experiencing an issue.