Dave Lester: Production-Ready Containers (Using Aurora and Mesos)
This week, we had the pleasure of hosting Dave Lester in our San Francisco office to give his talk "Production-Ready Containers: Using Aurora and Mesos".
Dave is an Open Source Developer Advocate at Twitter and gave us the rundown of how Twitter is currently using Mesos+Aurora in production.
We're particularly interested in these technologies at Bigcommerce because we're currently in the early-phases of moving pieces of our infrastructure into containers, which leads us to many questions:
- What containerization framework do you use? Docker? Rkt? OCI?
- How do you manage the container cluster? Mesos? ECS? Nomad? Fleet? Swarm?
- Service Discovery? Data Storage? Failover?
Needless to say, the industry is still in the early days of picking a clear winner or centralizing on a popular standard for productionizing containers, which is why it's nice to hear from Dave, who's able to share first-hand how Twitter is doing it.
Aurora and Mesos at Twitter
Twitter is running Apache Aurora on their Mesos cluster (they are also the original developers of the Aurora project and contribute heavily to both). Dave says that Mesos gives them four value propositions that make it worth using— Reliability, Utilization, Manageability, and Approachability.
In particular, I personally find approachability to be the most important feature of any system that you want your entire engineering team to use. If you centralize on a tool that's difficult to use, odds are that it won't be well-adopted within the company.
Although Mesos allows you to run many different scheduler frameworks simultaneously, Twitter is only running Aurora in production. The reason for "running one scheduler to rule them all" is that it gives them a single interface to manage long-running services and cron-jobs, as well as giving them a centralized priority and quota management system to kill of non-crucial tasks during high-traffic events.
Overall, I think that Dave really made a good case for Aurora, especially because it's been battle-tested in production at Twitter. I encourage you to watch the full talk, included below. Thanks again, Dave, for spending some time with the Bigcommerce team.