Twitter - Murder Bittorrent Deploy System from Larry Gadea on Vimeo.
The Twitter Engineering Blog: Murder: Fast datacenter code deploys using BitTorrent
Now that's a cool idea. The code is open source and can be found at http://github.com/lg/murder.
It was time for something completely different, something decentralized, something more like.. BitTorrent..
running inside of our datacenter to quickly copy files around. Using
the file-sharing protocol, we launched a side-project called Murder and
after a few days (and especially nights) of nervous full-site
tinkering, it turned a 40 minute deploy process into one that lasted
just 12 seconds!
Murder (which by the way is the name for a flock of crows) is a combination of scripts written in Python and Ruby to easily deploy large binaries throughout your company’s datacenter(s). It takes advantage of the fact that the environment in a datacenter is somewhat different from regular internet connections: low-latency access to servers, high bandwidth, no NAT/Firewall issues, no ISP traffic shaping, only trusted peers, etc. This let us come up with a list of optimizations on top of BitTornado to make BitTorrent not only reasonable, but also effective on our internal network. Since at the time we used Capistrano for signaling our servers to perform tasks, Murder also includes a Capistrano deploy strategy to make it easy for existing users of Capistrano to convert their file distribution to be decentralized. The final component is the work Matt Freels (@mf) did in bundling everything into an easy to install ruby gem. This further helped get Murder to be usable for more deploy tasks at Twitter.
Now that's a cool idea. The code is open source and can be found at http://github.com/lg/murder.
