Twitter Command Prefix
Google is Melting

blog

HOWTO Use Amazon EC2 for Bittorrent

01.17.09 | 12 Comments

Bittorrent, for how brilliant it is, is becoming a problem. You’ve probably heard about how ISPs are choosing to throttle even legitimate usage of the protocol as it slowly eats away at their bandwidth. Additionally, Inter-office VoIP networks can become crippled by just a few employees seeding last week’s episode of Battlestar. For me, at home, trying to maintain my ratio has caused big problems for my evening Left 4 Dead sessions, and can sometimes even make day-to-day web browsing a frustration.

So then why not then move Bittorrent out of the home/office and into the cloud? This weekend I was able to do just that with great success. Using Amazon’s Elastic Compute Cloud (EC2) and TorrentFlux (a web-based Bittorrent manager which runs on top of Bit Tornado). I created a web-based, open-source Bittorrent “machine” that liberated my network and leveraged Amazon’s instead. I can access it from anywhere, uploading Torrent files from wherever, and manage them from my iPhone. I also pay only for what I use – which by my estimates will hopefully be less than $30 per month Update: (That was a quick and not-so-thorough estimate. For some silly reason I didn’t account for the fact that the instance was probably going to have to be on all of the time. Once I gather some real data on how much it costs I’ll be sure and update this post. I agree it’s probably going to be over $75/month. Real data to follow.) And the best part? My ratio is maintained while my Left 4 Dead games speed along without any annoying lag.

If you think you might be interested in doing the same thing, please read on for a thorough HOWTO.

The first thing to do is sign up for Amazon Web Services, which will give you access to Amazon’s Elastic Compute Cloud (Amazon EC2) service. Amazon EC2, as you may already know, essentially lets you “run” a computer on Amazon’s infrastructure, paying for only the processing, space, and bandwidth you use.

WARNING: When you sign up for the service, you will of course have to agree to Amazon AWS’s TOS, which includes a promise that your EC2 instance will not be one that:

Distributes, shares, or facilitates the distribution of unauthorized data, malware, viruses, Trojan horses, spyware, worms, or other malicious or harmful code (collectively, “Harmful Components”).

So piracy is bad, mmmmkay?

Once you’ve signed up, you’ll next need to log-in to The AWS Management Console. This is where you’ll be setting up and managing all the virtual “hardware” you’ll need to get this thing running.

I suggest first starting with what’s called a “Security Group.” Basically, this is the firewall for your machine. You’ll want to make a security group that allows access to the instance over SSH, Web, and of course also includes a range of ports open for Bittorrent. Click the “Security Groups” link and then the “Create Security Group” button. Name your security group something like “TorrentFlux” and give it whatever description you see fit. Then click “Create.”

security-groups

Once the security group is created, you’ll see a table appaear via which you can configure the ports for this Security Group. First, using the drop-down provided, select HTTP, and then click “Save,” followed by SSH, and then “Save” for that row as well.

Finally you’ll want to open up a range of ports for Bittorrent. Select “Custom,” “TCP” and enter 49160 for the “From Port” and 49300 for the “To Port.” Use the same “Source,” 0.0.0.0/0, as the other services, and then press save.

security-group-ports

Now to create your EC2 instance. Clicking on the “Instances” link will take you to the list of EC2 instances you have running. Click the “Launch Instances” button to add a new instance.

The dialog will present you with a variety of pre-configured instances, called Amazon Machine Images or AMIs, that you can get started with. I’m an Ubuntu guy, and at this time while there are no “official” Ubuntu AMIs, the community is hard at work on several Ubuntu and Debian AMIs for Amazon EC2. For this project, I suggest using the Ubuntu AMI provided by alestic, which while in beta now, will in the future will be recognized as the “official” AMI for Ubuntu. They also already include most everything you’ll need to get started – including plenty of hard disk space.

Switch over to the “Community AMIs” tab and filter the list by entering in the text ami-1c5db975 where it says “Viewing:” This will bring up the 32-bit base Hardy AMI recommended for use by alestic. Then, just click the “Select” button.

ami-select

Next Amazon will ask how many instances you want to run (just say “1″), and the Instance Type (”Small” will save you money and work just fine).

It will also ask you to select a key pair. Basically, this is an SSH key pair that you’ll use to log in to the machine. If you’ve never used Amazon EC before, you’ll need to press the “Create” button here, name your keypair, and then download it. Put this file somewhere safe and don’t ever share it with anyone! Just like the keys to your house, this is the key to your machine. You don’t use a password to log into EC2 instances – just this key.

You’ll also need to select a Security Group before you can launch your instance. Simply select the TorrentFlux group that you created earlier, and then click “Launch Instance.”

It might take a few minutes for your instance to initialize. Once it’s done, it’s time to login to your instance and start configuring it. If, in the Instance list, you click the “Connect” button after selecting your new instance, you’ll see a few important bits of information that you’ll want to take note of.

connect

First, take note of your instance’s public DNS. (Mine is blurred out in the screenshot above.) Not only is this the address you’re going to use to connect to your server, it’s also going to be the web address you will use to connect to TorrentFlux once it’s up and running.

You’ll also want to note the command line it provides. This is the SSH command you’ll use to connect to your sever. Note that it uses the key you downloaded previously to connect. There will be no password! You just need that key, and then you’re in as root. The details of using SSH is kind of beyond the scope of this HOWTO, but search around if you need help.

Go ahead and connect to your instance using the command provided (changing the path to your key to be wherever you put it).

Update: I’ve received a few emails regarding issues with connecting using your SSH key. If you get prompted for a password, or see other errors, chances are the permissions for your key are wrong. Basically, your SSH key needs to be read/write owner – nothing more and nothing less – or problems may occur. Here’s what the permissions should look like:

-rw------- 1 bretto bretto 1675 2009-01-12 08:33 brettokey.pem

You’ll also of course have to accept the host key before you can fully connect. Once you do though, you’re in! Welcome to your new Amazon EC2 computer. :) Time to run updates!

Issue this command to make sure you’re running with the latest-greatest (which, considering you’re starting with a very bare bones Hardy installation, won’t take long).

$apt-get update && apt-get upgrade -y

Of course, do not type the $ symbol. That is only meant to depict a shell command.

Next you’re going to want to let Ubuntu know that you’re not running anything from a front end. This will prevent dialog boxes and such from pop-ing up and preventing the installation of the additional software we need to install.

$export DEBIAN_FRONTEND=noninteractive

Now it’s time to install Apache Web Server, PHP, and MySQL – everything TorrentFlux needs to do its thang. Issue this command to start that process:

$apt-get install apache2 php5 php5-mysql mysql-server-5.0

Follow along with the install and, once it’s complete, try to access that public DNS name you noted earlier in your web browser. The URL will probably be something like http://ec2-SOME-AMOUNT-OF-NUMBERS.compute-1.amazonaws.com. If you see “It Works!” well, then, it works! ;) You now have a LAMP (Linux, Apache, MySQL PHP) server running live on Amazon’s infrastructure – only a few more steps left.

Now to download and install TorrentFlux. TorrentFlux is hosted on Sourceforge.Net, so as soon as you access the download it will start to send the file to your local computer. Cancel this download, copy the direct link to the file, and then issue this command on your new EC2 instance, substituting the link to provided with the one you copied:

$wget http://mirror.dl.sourceforge.net/sourceforge/torrentflux/torrentflux_2.4.tar.gz

Extract it:

$tar -zxf torrentflux_2.4.tar.gz

Remove the tar file if you like and then rename the directory to “torrentflux:”

$rm torrentflux_2.4.tar.gz
$mv torrentflux_2.4 torrentflux

NOTE: The file names shown above may not be exactly like yours.

Your EC2 instance will include 120 GiB of space mounted at /mnt, which should be enough for a good amount of Bittorrent fun. You’ll want to create your data directory there, and assign it the correct permissions as well so that TorrentFlux can access it.

$mkdir /mnt/torrentflux-data
$chmod 777 /mnt/torrentflux-data

Now move into your torrentflux install:

$cd torrentflux

Create the MySQL database that TorrentFlux will use:

$mysqladmin create torrentflux

And import the initial tables and data provided by TorrentFlux:

$mysql torrentflux < sql/mysql_torrentflux.sql

The final step is configuring Apache to direct web requests to the TorrentFlux html directory. Edit the default site config (here I am using the vim editor. Substitute for the editor of your choice.):

$vim /etc/apache2/sites-enabled/000-default

There, you'll want find and change the document root to:

DocumentRoot /root/torrentflux/html/

And the path to the default directory to be that same patch:

<Directory /home/torrentflux/html/>

If you have trouble finding those lines, here's the configuration file in its entirety.

Save your changes, exit your editor, and then restart apache.

$apache2ctl restart

Now browse to your web address name again. If you see a login dialog, you've got TorrentFlux installed and working properly. Login with whichever username and password you like - this first login will become the admin login and password for your TorrentFlux install.

You'll then be presented with a list of settings to configure. For the most part, you can leave all of them alone, although feel free to tweak as you see fit. The only thing you must change is the path to where downloads are stored. Change it to the directory you created earlier:

/mnt/torrentflux-data/

Congrats! You're good to go. TorrentFlux will behave like many other Bittorrent clients you have used in the past. You can upload or point TorrentFlux directly to torrent files and it will queue them up to download. Then, click the "Run Torrent" (little green down-arrow button) to start downloading. When downloads are complete, TorrentFlux will continue seeding as long as you specify. To download completed files to your local computer, go to the "Directory" area and click the download button. TorrentFlux with tar-up the data and send it your way.

torrentflux

Enjoy Bittorrenting in the cloud. Feel free to leave any feedback or questions in the comments.

12 Comments

On 01.17.09 Santa Claus scribed these epic words:

rtorrent is wickedly fast and efficent. rtgui is a better front end (will even read RSS feeds!).

http://libtorrent.rakshasa.no/

http://code.google.com/p/rtgui/

On 01.17.09 Thomas scribed these epic words:

So, I’m curious how this is going to cost <$30/mon. Are you only running it as needed? Seems that will be heavily dependent on how quickly the torrents get downloaded and how long it takes to get your ratio to the level you want it (which depends on demand for the files you’re seeding, of course).

On 01.18.09 negatendo scribed these epic words:

Thomas: You’re probably right. That was a quick and not-so-thorough estimate. For some silly reason I didn’t account for the fact that the instance was probably going to have to be on all of the time. Once I gather some real data on how much it costs be sure and update this post.

On 01.18.09 Derek Reiff scribed these epic words:

This seems to be easier that I originally thought EC2 was. Can’t seem to get past the password requirement when first logging in through ssh, though. Will keep trying, though….

On 01.18.09 binnary scribed these epic words:

This can be done with Amazon S3 and reducing the final costs a lot than using EC2!

http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?S3Torrent.html

On 01.18.09 Lamnk scribed these epic words:

This is not new, i thought about it when EC2 came out but the idea got turned down when the bandwidth cost was announced. At 10c/GB you will end up paying at least $50 for 500GB transfer. You can easily find some cheapo P3 or Celeron dedicated boxes with 1000GB transfer for around $50. If you want to save more, a VPS will do the job. Just look on webhostingtalk.com for offers.

BTW, you can also seed torrent from S3 objects. But the cost is essentially the same as seeding from EC2 if your purpose is to increase your ratio.

On 01.19.09 Joel scribed these epic words:

I think amazon’s prices are high for keeping the server on all month. Check out slicehost — I have a slice which meets my needs for $20/month.

Watch out for bandwidth overages, though, with either vendor.

On 01.20.09 Jon scribed these epic words:

The tutorial is awesome. I ran through it and had Torrentflux installed in a short period of time. No problems at all. I’m just curious about the cost. This will be a trial run this month to see. This solves a big problem I was having at home.

On 01.21.09 HaloMan scribed these epic words:

I agree with Lamnk, for $50 you can find some cheapo dedicated box that will do the same thing, mind you, without the risk of Amazon terminating an account if they think it might be shady.

On 01.23.09 jth scribed these epic words:

Regarding Apache edits:

You might want to specify editing:

/etc/apache2/sites-available/default

the …sites-enabled/000-default is a symbolic link and editing that can cause problems if the editing ever converts it from a symlink to an actual file.

Its nitpicky I know, but something to consider

jth

On 02.04.09 usd6 scribed these epic words:

Great article, I learned from you was a big help. Thank you!

On 02.11.09 negatendo scribed these epic words:

binnary and Lamnk: Publishing content through Bittorent with Amazon S3 is not necessarily the same thing as what I’ve tried to do in my post.


Twitter Command Prefix
Google is Melting