Five million users making 40m monthly pageviews across 30 servers with 99.99% uptime? That’s what you walk into on day one. The right candidate for this job knows that when you’re proactive about managing and monitoring servers, DevOps is less about firefighting and more about empowering millions of users via a well-crafted infrastructure.

Bonanza is a fast-growing online marketplace that has proven Rails does scale, even when 90% of our content is dynamically rendered. Our infrastructure consists of more than 30 server instances, split between app, async, test, database, search, utility, and memcached-focused self-hosted Linux installs on around 25 physical servers. We make heavy use of ansible to provision servers and ensure consistency across the various server roles. 

The principles we’ve used to design our infrastructure include:

The sum of these and other measures have created an environment that has proven highly stable in spite of its many components, as listed in the “Requirements” section.

Our presumption is that our pure Devops-related needs will occupy 50-80% of a full time role; with whatever remaining time our Devops engineer may find, they will make contributions to our Ruby on Rails codebase.

Desired Skills and Experience

  • 99.99% uptime target. Our total downtime has averaged less than 5 hours per year for 8 years by being tireless about root cause analysis, and working doggedly to eliminate single points of failure. 
  • No unsolved mysteries. This is partly possible because of our consistent use of root cause analysis when problems occur. The other part is ensuring that no server setup is a one-off endeavor. All server configuration is handled via ansible, which means we can store and track server history through source control.
  • Minimize proliferation of architecture components. Developers always want to employ the latest hotness, but at Bonanza they must clear a high barrier of addressing “why?” when they seek to introduce a new dependency or failure point to the existing tech stack. That said, Bonanza and Background Burner are many-faceted organisms that inevitably require understanding a breadth of systems.
  • Ruby on Rails
  • Phusion Passenger on Apache
  • MySQL
  • Resque
  • Ubuntu
  • Redis
  • pfSense
  • HAProxy
  • Sphinx search
  • Ansible
  • Jenkins (CI server)
  • Memcached
  • Phusion Passenger on nginx (for websockets)
  • Pusher (for websockets)
  • … and a few others
  • New Relic
  • Datadog
  • VividCortex
  • Papertrail
  • Rollbar
  • Monit
  • Nagios
  • Fast growing and already-profitable company. It’s fun to work for a winner.
  • “Work from home Wednesday” - every Wednesday we get to work from home, a coffee shop, or even the office, if that’s your bag.
  • “Freedom February” - every February we retreat from Seattle and spend a month annoying our friends with selfies taken in Mexico, Belize, Europe, etc.
  • Five bonus (not counted as vacation) days off in February to enjoy your vacation or staycation
  • Twice-weekly catered lunches
  • Monthly on-site full body massage
  • Work from a waterfront office in downtown Seattle (food trucks aplenty!) convenient to many bus lines
  • Full arcade room in the office
  • Pong table on site
  • The boring stuff: fully paid health insurance coverage, vacation benefits, holidays, stock options, and more.