Shane Rainville
Posted on

How to scale WordPress sites on Ubuntu using AWS S3 Storage

Scale Wordpress using S3 buckets

This tutorial will show you how to mount an S3 volume onto multiple WordPress servers. Doing so will allow you to scale your site across multiple servers, increasing the number of traffic it can receive and offering high availability.

Expanding your WordPress site to two or more servers elements the risk of there being a single-point of failure. This means your site will remain online when a server goes offline.

Amazon’s S3 storage, also known as Simple Storage Share, allows you to upload files to a single point that can be shared with multiple targets. If you are familiar with NFS or Windows Shares, it’s a similar concept.

Mounting an S3 volume to all of your WordPress servers we can ensure they all have access to the same content. No need for setting up complicated RSync scripts that could fail or be out of sync. Any media upload to S3 via WordPress is immediately available to all other WordPress servers.

Weighing the Cost-Benefit of S3

S3 isn’t free. You will be charged based on the amount of storage your or consuming and by how much data is transferred.  Thankfully, S3 is relatively inexpensive. This is especially true when compared with running your own file server to host shares.

Having to stand up another server to offer NFS or Windows File Shares would be considerably more expensive than having an S3 volume. The other downside is having to maintain the server, ensuring it has the latest patches, for example.

By standing up your own file server you add more risk and complexity to your infrastructure. You have created a file share to increase your site’s capacity and availability, but what happens when your single file server goes offline? Depending on what you place on that file server, you may lose access to just your uploads or your site will be offline from losing your theme, as well.

The Prerequisites

Before following along with this tutorial you will need the following.

  • An AWS Account
  • An S3 bucket
  • An IAM account granted permissions to the S3 Bucket
  • The access key and the secret key for the IAM account.

Mounting S3 To Your Server

We cannot simply mount an S3 volume to our server. Ubuntu or CentOS, for example, wouldn’t understand how to mount that volume type. We will first have to install a package that enables S3 support. After we install it we can begin mounting the volume across all of our servers.

Installing S3FS

The S3FS package allows us to mount S3 volumes. The following instructions will show you how to install it onto your Linux server.

Ubuntu

If you are running Ubuntu 16.04 or higher, you simply install the package from the official Ubuntu repository. Keep in mind you will not be running the latest, most bleeding edge version.

sudo apt install s3fs

If you would prefer running the latest version or running a very specific version, you can compile the package from source following these steps.

  1. Install the dependent packages to compile S3FS. 
    sudo apt-get install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
  2. Clone the S3FS repository. 
    git clone https://github.com/s3fs-fuse/s3fs-fuse.git
  3. Change into the new s3fs-fuse directory. 
    cd s3fs-fuse
  4. Run the autogen shell script. 
    ./autogen.sh
  5. Run configure to prepare the package for being compiled on your system. 
    ./configure
  6. Run the make command to compile s3fs-fuse. 
    make
  7. And finally, let’s install the newly compiled package. 
    sudo make install

CentOS

CentOS and Red Hat are a little more involved. You will have to download and compile the package from source. Don’t worry, it’s really simple to do following these instructions.

  1. Install the dependent packages to compile S3FS. 
    sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
  2. Clone the S3FS repository. 
    git clone https://github.com/s3fs-fuse/s3fs-fuse.git
  3. Change into the new s3fs-fuse directory. 
    cd s3fs-fuse
  4. Run the autogen shell script. 
    ./autogen.sh
  5. Run configure to prepare the package for being compiled on your system. 
    ./configure
  6. Run the make command to compile s3fs-fuse. 
    make
  7. And finally, let’s install the newly compiled package. 
    sudo make install

Configuring S3FS-Fuse

In order for us to connect to an S3 share we will need to configure S3FS to use our AWS credentials. Rather than using a username and password, we use the more secure approach of a secret and a key.

Do the following on each Linux server.

  1. Create a system-wide S3FS credentials file. 
    sudo touch /etc/.passwd-s3fs
  2. Modify the file permissions to allow only Root to access it. 
    sudo chmod 0600 /etc/.passwd-s3fs
  3. Open the file in a text editor, such as VI. 
    sudo vi /etc/.passwd-s3fs
  4. Add your AWS credentials to the file using the following format. 
    ACCESS_KEY:SECRET_KEY
  5. Save your change and exit the text editor.

Mounting the S3 Volume

After installing S3FS and configuring it we can begin mounting our share to each WordPress server. First we’ll test that we are able to mount the share. If successful, we will then configure each WordPress server to mount the share automatically at boot, and then mount the share.

Testing the Mount

  1. Create a temporary directory to test your S3 volume mount. 
    sudo mkdir /tmp/s3test
  2. Running the following command to mount the volume. 
    s3fs mybucket /path/to/mountpoint -o passwd_file=~/.passwd-s3fs
  3. If you encounter any errors, mount again and set the debug flag. 
    s3fs mybucket /path/to/mountpoint -o passwd_file=~/.passwd-s3fs -o dbglevel=info -f -o curldbg
  4. After successfully mounting the volume, unmount it. 
    umount /tmp/s3test

Add Mount to FSTAB and Mount the Volume

We’re finally ready to create a persistent mount of our S3 volume. Before you continue you need to decide what you want shared between WordPress servers.

  • All data under wp-content.
  • Only wp-content/uploads.

Based on your decision you will need to do the following to mount the volume. If you already have content in the WordPress directory you want shared with all servers, backup it up now. You can then restore to your S3 volume after we’ve mounted it.

  1. Open fstab in a text editor, such as VI. 
    sudo vi /etc/fstab
  2. Add the following line to mount the S3 volume in away that all wp-content is shared. 
    mybucket /path/to/wp-content fuse.s3fs _netdev,allow_other 0 0

    If your WordPress install is under /var/www/html, you would use the following fstab line.

    mybucket /var/www/html/wp-copntent fuse.s3fs _netdev,allow_other 0 0
  3. Save your changes and exit the text editor.
  4. Now mount the new volume. 
    sudo mount -a