Bittorrent Your High-Res Datasets

charlesleo · 09-25-2004, 10:33 PM

You could possibly incorporate something like Bittorrent to parse out localized versions of your Hi-res datasets past a certain km. Just have it included as part of the agreement or use it as an option if taxpayers get all nutty about this. I know NASA has issues with this however - I gather it opens up too many holes? Well, if this is the case then perhaps you can move your servers off-site. Perhaps Bill Gates can sponsor it as a good-will gesture... :lol:

rgovostes · 09-25-2004, 10:40 PM

This is a pretty good idea. Not only would this save a whole lot of bandwidth for NASA's servers, the downloads would be quite fast. However, I think this should be an option - many users would not want their bandwidth used without their permission.

However, BitTorrent is natively written in python - not to say there aren't clients in C or C++ that could be adapted for use. Most BitTorrent clients are released under the GPL.

Randy Kim · 09-26-2004, 12:46 AM

Another NASA fellow mentioned this. It makes sense, like in cases where one classroom is looking at one region of the world and all the machines relay data to each other. It's a thought worth pursuing.

There is an added challenge though. The data server also calculates each individual tile and mixes and matches up to 9 different spectrums of LandSat7. Soâ€¦ could this mean peer to peer distribution and distributed computing? Okay my head hurts now.

rgovostes · 09-26-2004, 04:10 AM

If interest in the project doesn't wane too much, the need for faster downloads could develop into a distributed server, like New York University's Coral (as an example, World Wind's homepage, Coralized).

In doing so, the satellite imagery would be shared by servers as well as users across the world, virtually eliminating latency and disk space - I imagine that 20 TB of satellite data would be no problem; according to the Coral stats page, their 85 clusters have transmitted 35 GB today alone. With the usefulness of satellite and USGS data, I'm sure NASA could find many more hosts...

This would have the added advantage that images could be adjusted for use with World Wind once, and every other client connected to the network would benefit from not having to process the image.

However, this would require more than a one-person development team, so perhaps it's not going to be happening any time soon?

RurouniJones · 09-26-2004, 01:43 PM

Agree, bittorrent is a great program, it is just a pity that if you ask anyone in academia or the mainstream press (or even an ordinary joe) you will get "Bittorrent? Isn't that a warez program?".

The result being that Bittorrent is never looked at as a serious file sitribution system that is tailor made for projects like this. Even if it is looked at by the techs the second someoen in "management" gets wind of it they shut it down faster than an arms dealer by the whitehouse... Guilty by association.

QBasicer · 09-26-2004, 01:46 PM

A lot of things come as a torrent these days, or at least Linux Distros...

I got this program through bittorrent... This is how it went:

Started it, and the site download was at ~30%.
When the torrent finished, the site download was at ~80%.

Quite the speed boost...Sorta. The site download was going at ~10KBps, the torrent at ~150KIps.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Help us get more High Resolution Imagery WorldWide	f0urtyfive	WorldWind General	55	01-21-2007 09:37 PM
who can tell me make this are high resolution	jetchina	Technical Support	2	01-30-2006 08:47 AM
High Resolution Imagery Not Displaying	yoway	Technical Support	0	10-13-2005 07:10 PM
Bittorrent Functionality for Data Download	tabdelgawad	Suggestion Box	16	11-07-2004 11:32 PM
Better control of high resolution downloads	Jugalator	Suggestion Box	0	09-26-2004 11:54 PM

09-25-2004, 10:33 PM	#1
charlesleo Junior Member Join Date: Sep 2004 Posts: 1	You could possibly incorporate something like Bittorrent to parse out localized versions of your Hi-res datasets past a certain km. Just have it included as part of the agreement or use it as an option if taxpayers get all nutty about this. I know NASA has issues with this however - I gather it opens up too many holes? Well, if this is the case then perhaps you can move your servers off-site. Perhaps Bill Gates can sponsor it as a good-will gesture... :lol:

09-25-2004, 10:40 PM	#2
rgovostes Junior Member Join Date: Sep 2004 Location: 42.485W -76.384N Posts: 28	This is a pretty good idea. Not only would this save a whole lot of bandwidth for NASA's servers, the downloads would be quite fast. However, I think this should be an option - many users would not want their bandwidth used without their permission. However, BitTorrent is natively written in python - not to say there aren't clients in C or C++ that could be adapted for use. Most BitTorrent clients are released under the GPL.

09-26-2004, 12:46 AM	#3
Randy Kim NASA Employee Join Date: Aug 2004 Location: Moffett Field, CA Posts: 116	Another NASA fellow mentioned this. It makes sense, like in cases where one classroom is looking at one region of the world and all the machines relay data to each other. It's a thought worth pursuing. There is an added challenge though. The data server also calculates each individual tile and mixes and matches up to 9 different spectrums of LandSat7. Soâ€¦ could this mean peer to peer distribution and distributed computing? Okay my head hurts now.

09-26-2004, 04:10 AM	#4
rgovostes Junior Member Join Date: Sep 2004 Location: 42.485W -76.384N Posts: 28	If interest in the project doesn't wane too much, the need for faster downloads could develop into a distributed server, like New York University's Coral (as an example, World Wind's homepage, Coralized). In doing so, the satellite imagery would be shared by servers as well as users across the world, virtually eliminating latency and disk space - I imagine that 20 TB of satellite data would be no problem; according to the Coral stats page, their 85 clusters have transmitted 35 GB today alone. With the usefulness of satellite and USGS data, I'm sure NASA could find many more hosts... This would have the added advantage that images could be adjusted for use with World Wind once, and every other client connected to the network would benefit from not having to process the image. However, this would require more than a one-person development team, so perhaps it's not going to be happening any time soon?

09-26-2004, 01:43 PM	#5
RurouniJones Junior Member Join Date: Sep 2004 Posts: 2	Agree, bittorrent is a great program, it is just a pity that if you ask anyone in academia or the mainstream press (or even an ordinary joe) you will get "Bittorrent? Isn't that a warez program?". The result being that Bittorrent is never looked at as a serious file sitribution system that is tailor made for projects like this. Even if it is looked at by the techs the second someoen in "management" gets wind of it they shut it down faster than an arms dealer by the whitehouse... Guilty by association.

09-26-2004, 01:46 PM	#6
QBasicer Junior Member Join Date: Sep 2004 Posts: 6	A lot of things come as a torrent these days, or at least Linux Distros... I got this program through bittorrent... This is how it went: Started it, and the site download was at ~30%. When the torrent finished, the site download was at ~80%. Quite the speed boost...Sorta. The site download was going at ~10KBps, the torrent at ~150KIps.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)