Do you love working with technologies like AWS and Puppet, and relish keeping a large-scale, data-intensive application running smoothly? DeepCrawl is a sophisticated web site crawling and auditing tool, and we’d love for you to join our team to develop and improve our AWS infrastructure.

Our many customers depend on DeepCrawl to improve user experience, manage change, and improve search engine performance. At DeepCrawl, you’ll have great opportunities to learn about large-scale web crawling, and how to build a complex Ruby application. You can also use 10% of your time to learn or build anything you want.

Our team is built foremost on constant learning, supporting each other, and loving working together. A great team environment supports us in building great technology. We’re always exploring new ideas and tech. Our stack now includes Ruby, PostgreSQL, MongoDB, a sophisticated API using Sinatra, and a front end in AngularJS.

Responsibilities will include:

  • Use Puppet to develop and modify server configuration across our system
  • Manage AWS services
  • Continuously improve monitoring capabilities, and engage in regular retrospectives to analyse system reliability and monitoring effectiveness
  • Automate and document manual tasks
  • Develop and maintain autoscaling systems
  • Spin up new resources as required for developers, for eg benchmarking, load testing, and
  • prototyping
  • Develop and maintain backup and disaster recovery plans
  • Respond to system performance problems and outages

Skills & Requirements


  • A positive character and an ability to collaborate well with others
  • Solid professional Ops experience
  • A deep knowledge of configuration management tools, ideally Puppet
  • Knowledge of a scripting language, ideally Ruby
  • Solid experience managing a variety of database systems, ideally including PostgreSQL,
  • MySQL, and MongoDB
  • Strong knowledge of AWS
  • Experience with monitoring tools such as DataDog or New Relic
  • Excellent spoken and written English


  • Good knowledge of Ruby
  • Experience with Java


  • Flexible Working
  • 10% time on your own projects or learning
  • Share options
  • Free fresh fruit delivery

About DeepCrawl

DeepCrawl is a sophisticated crawling service, used by the world’s largest brands and SEO agencies to find and monitor many problems with their sites — ultimately making them easier to find and use. Our team emphasises friendliness, learning, and collaboration. We’ve got a diverse range of backgrounds, and our interests include classic & independent film, life drawing, crafts & interior decoration, photography, motorcycle racing, hiking, and good Japanese whiskey.

We believe that by providing powerful tools to improve web sites, we make the web a better place for everyone. As an established but still fast-growing startup, we’ve got big ambitions to continue our growth and expansion into new markets, and be the tool of choice for SEO, digital marketing and beyond.

Tagged as: AWS, DevOps, Linux, Puppet

Cntrl/Cmd + Click to attach multiple files.
I confirm I am based in the UK and DO NOT require sponsorship.