There are a lot of redundant problems you need to solve as a web developer. Dealing with users is a common problem: storing them, authenticating them, and properly securing their data. This particular problem is what we here at Stormpath try to solve in a reusable way so that you don’t have to.

Another common problem web developers face is file storage. How do you securely store user files? Things like avatar images, PDF document receipts, stuff like that. When you’re building a web application, you have a lot of choices:

  1. Store user files in your database in a text column, or something similar
  2. Store user files directly on your web server
  3. Store user files in a file storage service like Amazon S3

Out of the above choices, I always encourage people to go with #3.

Storing files in a database directly is not very performant. Databases are not optimized for storing large blobs of content. Both retrieving and storing files from a database server is incredibly slow and will tax all other database queries.

Storing files locally on your web server is also not normally a good idea. A given web server only has so much disk space, which means you now have to deal with the very real possibility of running out of disk space. Furthermore, ensuring your user files are properly backed up and easily, consistently accessible can be a difficult task, even for experienced engineers.

Unlike the other two options, storing files in a file storage service like S3 is a great option: it’s cheap, your files are replicated and backed up transparently, and you’re also able to quickly retrieve and store files there without taxing your web servers or database servers. It even provides a fine-grained amount of control over who can access what files, which allows you to build complex authorization rules for your files if necessary.

This is why I’m excited to announce a new project I’ve been working on here at Stormpath that I hope you’ll find useful: express-stormpath-s3.

This is a new Express.js middleware library you can easily use with your existing express-stormpath web applications. It natively supports storing user files in Amazon S3, and provides several convenience methods for directly working with files in an abstract way.

Instead of rambling on about it, let’s take a look at a simple web application:

This is a bare-bones web application that uses Express.js, express-stormpath, and express-stormpath-s3 to provide file storage support using Amazon S3 transparently.

This example initialization code requires you to define several variables which are all hard-coded above. This minimal application requires you to:

  • Have a Stormpath account and have already created a Stormpath Application
  • Have an Amazon Web Services account, and to have created an S3 bucket in the US Standard region (this is where your user files will be stored)

Assuming you’ve got both of the above things, you can immediately start using this library to do some cool stuff.

Uploading User Files

First, let’s take a look at how you can store files for each of your users:

This library automatically adds a new method to all of your user Account objects: uploadFile. This method allows you to upload a file from disk to Amazon S3. By default, all files uploaded will be private so that they are not publicly accessible to anyone except you (the AWS account holder).

If you’d like to make your uploaded files publicly available or set them with a different permission scope, you can easily do so by passing an optional acl parameter like so:

The way this all works is that all user files will be stored in your specified S3 bucket, in a sub-folder based on the user’s ID.

Let’s say you have a Stormpath user who’s ID is xxx, and you then upload a file for this user called some-file.txt. This means that your S3 bucket would now have a new file that looks like this: /xxx/some-file.txt. All files are namespaced inside of a user-specific folder to make parsing these values simple.

Once the file has been uploaded to S3, the user’s Custom Data store is then updated to contain a JSON object that looks like this:

This way, you can easily see what files your user has uploaded within Stormpath, and link out to files when necessary.

The express-stormpath-s3 documentation talks more about uploading files here.

Downloading User Files

As you saw in the last section, uploading user files to Amazon S3 is a simple process. Likewise — downloading files from S3 to your local disk is also easy. Here’s an example which shows how you can easily download previously uploaded S3 files:

As you can see in the example above, you only need to specify the filename, no path information is required to download a file. This makes working with files less painful as you don’t need to traverse directory paths.

You can read more about download files in the documentation here.

Deleting User Files

To delete a previously uploaded user file, you can use the deleteFile method:

You can read more about this in the documentation here.

Syncing Files

Finally, this library provides a nice way to ensure your S3 bucket is kept in sync with your Stormpath Accounts.

Let’s say you have a large web application where you have users uploading files from many different services into S3. This might result in edge cases where files that were NOT uploaded via this library are not ‘viewable’ because the file metadata has not been persisted in the Stormpath Account.

To remedy this issue, you can call the syncFiles method before performing any mission critical tasks:

This makes building large scale service oriented applications a lot simpler.

You can read more about the sync file support here.

Wrapping Up

Right now this library is available only for Express.js developers. If you find it useful, please leave a comment below and go star it on Github! If we get enough usage from it, I’ll happily support it for the other Stormpath web frameworks as well.

If you have any questions about Stormpath, Express.js, or Amazon S3, also feel free to drop me a line!