Torrents as a mechanism to reduce costs
Summary
- There are 8 posts — by 5 authors — in this topic.
- Latest post made by mf.kubke at 2010 Aug 02 18:38 NZST
Torrents are a very effective way to distribute large volumes of data in a cost-effective way. As a distributed system, it scales much better than providing a single file server. Torrents also provide a guarantee that the data hasn't been tampered with during transit because of its in-built encryption & checking. They could be very useful for quarterly DB dumps, etc. What do people think about promoting this as an option?
Tim.
Any recommended links to further relevant information about Torrents? On Mon, Aug 2, 2010 at 11:08 AM, Tim McNamara
<email obscured>>wrote: > Torrents are a very effective way to distribute large volumes of data in a > cost-effective way. As a distributed system, it scales much better than > providing a single file server. Torrents also provide a guarantee that the > data hasn't been tampered with during transit because of its in-built > encryption & checking. > > They could be very useful for quarterly DB dumps, etc. > > What do people think about promoting this as an option? > > Tim. > > ----------------------------------------- > Full text of this topic in The Open Government Ninjas: > http://groups.open.org.nz/r/topic/5cRWQbGEQjYssaIMXBKgZX > > To leave The Open Government Ninjas, email > <email obscured>?Subject=unsubscribe > > Start your own free groups and site with > OnlineGroups.Net http://onlinegroups.net > > Host your own online groups site with > GroupServer http://groupserver.org >
Not sure if this may be of use re:torrents http://www.plosone.org/article/info:doi/10.1371/journal.pone.0010071
-Kubke On Mon, Aug 2, 2010 at 11:13 AM, Jim McLeod <email obscured>> wrote: > Any recommended links to further relevant information about Torrents? > > On Mon, Aug 2, 2010 at 11:08 AM, Tim McNamara > <email obscured>>wrote: > > > Torrents are a very effective way to distribute large volumes of data in > a > > cost-effective way. As a distributed system, it scales much better than > > providing a single file server. Torrents also provide a guarantee that > the > > data hasn't been tampered with during transit because of its in-built > > encryption & checking. > > > > They could be very useful for quarterly DB dumps, etc. > > > > What do people think about promoting this as an option? > > > > Tim. > > > > ----------------------------------------- > > Full text of this topic in The Open Government Ninjas: > > http://groups.open.org.nz/r/topic/5cRWQbGEQjYssaIMXBKgZX > > > > To leave The Open Government Ninjas, email > > <email obscured>?Subject=unsubscribe > > > > Start your own free groups and site with > > OnlineGroups.Net http://onlinegroups.net > > > > Host your own online groups site with > > GroupServer http://groupserver.org > > > > ----------------------------------------- > Full text of this topic in The Open Government Ninjas: > http://groups.open.org.nz/r/topic/7cwjtWBKwFL0mv8jLd5XLJ > > To leave The Open Government Ninjas, email > <email obscured>?Subject=unsubscribe > > Start your own free groups and site with > OnlineGroups.Net http://onlinegroups.net > > Host your own online groups site with > GroupServer http://groupserver.org > -- M Fabiana Kubke Department of Anatomy | University of Auckland | New Zealand (+64) 9 373-7599 Ext 86002 | (+64)9 923 6002 (direct) | Mobile: (+64) 210 437 121 Skype: superfabs | http://twitter.com/Kubke | http://identi.ca/kubke | http://buildingblogsofscience.wordpress.com | http://sciblogs.co.nz/building-blogs-of-science | http://popscinz.wordpress.com
Hi Jim, Sorry for not providing a little bit more background. The article that Fabiana provided is a very good. Here's an introduction: ## Intro A torrent another name for peer-to-peer file sharing. It is a way to spread the load of offering files to download. It uses a system called the BitTorrent protocol [1]. The idea is that rather than downloading a single file, you download portions of the file from everybody that already has it. If 10 people have a resource, you end up downloading roughly 10% from each person. The portions are reassembled into the original file at the end. ## Positives As resources become more popular, the original provider ends up providing proportionally less bandwidth for the next download. When 100 people are connected, the department would only be providing roughly 1%. The distributed nature of the service has a few other advantages. It means that even if the original server goes down, the files are still accessible to everyone. ## Downsides They need special software if you're not running a Linux distribution, which generally provide BitTorrent clients out of the box. This may not be such a concern for open data people, because the entire audience is technically savvy. Torrents have a really bad name. They're strongly associated with illegal file sharing. I tend to think that this has happened because the system is so good at what it does... There will always be some people who choose not to offer their bandwidth for others. But this is no different than the status quo. Some terminology that may assist you when you're searching the Internet for more info: .torrent - a file type that has information about the file, such as where to look for trackers Tracker - a server that keeps track of people in the swarm Swarm - the number of people that are connected Seeder - someone providing the file to download Leecher - someone downloading the file
On 2 August 2010 11:13, Jim McLeod <email obscured>> wrote: > Any recommended links to further relevant information about Torrents? > > On Mon, Aug 2, 2010 at 11:08 AM, Tim McNamara > <email obscured>>wrote: > > > Torrents are a very effective way to distribute large volumes of data in > a > > cost-effective way. As a distributed system, it scales much better than > > providing a single file server. Torrents also provide a guarantee that > the > > data hasn't been tampered with during transit because of its in-built > > encryption & checking. > > > > They could be very useful for quarterly DB dumps, etc. > > > > What do people think about promoting this as an option? > > > > Tim. > [1] http://en.wikipedia.org/wiki/BitTorrent_(protocol)
On Mon, Aug 2, 2010 at 11:08 AM, Tim McNamara <email obscured>>wrote: > Torrents are a very effective way to distribute large volumes of data in a > cost-effective way. As a distributed system, it scales much better than > providing a single file server. Torrents also provide a guarantee that the > data hasn't been tampered with during transit because of its in-built > encryption & checking. > > They could be very useful for quarterly DB dumps, etc. > > What do people think about promoting this as an option? > Unfortunately, I'd say that unless a number of people are downloading a dataset at the same time or are happy to share data after they've finished (seeding), it wouldn't work out any more efficient - but is more complex for someone to set up. NZ Internet connections are pretty average and many have upload data caps, so the seeding rate is really bad compared with elsewhere (according to a major ISP I spoke with). I like the idea, I just don't think we have enough people or decent enough Internet yet :) Amazon S3 offers bittorrent support, so anyone publishing data using it can offer torrents easily enough. Rob :)
On Mon, Aug 2, 2010 at 11:08 AM, Tim McNamara <email obscured>> wrote: > Torrents are a very effective way to distribute large volumes of data in a > cost-effective way. As a distributed system, it scales much better than > providing a single file server. This conflates two things. Use of the bitorrent protocol for file transfer and the number of servers in use. It is entirely possible to run bittorrent with a single client and a single server, think of it as ftp with robust error checking. This scales well. It is entirely possible to script ftp to download from multiple / the best current servers, this has been done for many years by tools such as apt-spy. This scales well. Bittorrents' sweet-spot is when lots of people want the same file(s) at a similar time, when it can does a fabulous job (assuming a well connected network with distributed rather and global bottlenecks).
cheers stuart
On 2 August 2010 13:10, Robert Coup <email obscured>> wrote: > On Mon, Aug 2, 2010 at 11:08 AM, Tim McNamara > <email obscured>>wrote: > > > Torrents are a very effective way to distribute large volumes of data in > a > > cost-effective way. As a distributed system, it scales much better than > > providing a single file server. Torrents also provide a guarantee that > the > > data hasn't been tampered with during transit because of its in-built > > encryption & checking. > > > > They could be very useful for quarterly DB dumps, etc. > > > > What do people think about promoting this as an option? > > > > Unfortunately, I'd say that unless a number of people are downloading a > dataset at the same time or are happy to share data after they've finished > (seeding), it wouldn't work out any more efficient - but is more complex > for > someone to set up. > > NZ Internet connections are pretty average and many have upload data caps, > so the seeding rate is really bad compared with elsewhere (according to a > major ISP I spoke with). > My thoughts were that after a period of time a cadre of about a dozen institutional semi-permanent seeders, e.g. KAREN-affiliated public sector agencies, businesses & ISPs would develop. There is a tradition of ISPs providing Linux & BSD distros (that are several hundred mb each) for free: http://mirror.ihug.co.nz/ Is that list of seeders a realistic expectation? Tim.
One advantage I see of biotorrents is the way that it handles the searching of the data. Data can be searched by area as well as by licence, the latter being crucial for any reuse. Only played within the site once, but was enough to convince me it had the right elements for science data sharing (not surprising, since it was developed for that specific purpose). -kubke On Mon, Aug 2, 2010 at 1:32 PM, Tim McNamara
<email obscured>>wrote: > On 2 August 2010 13:10, Robert Coup <email obscured>> wrote: > > > On Mon, Aug 2, 2010 at 11:08 AM, Tim McNamara > > <email obscured>>wrote: > > > > > Torrents are a very effective way to distribute large volumes of data > in > > a > > > cost-effective way. As a distributed system, it scales much better than > > > providing a single file server. Torrents also provide a guarantee that > > the > > > data hasn't been tampered with during transit because of its in-built > > > encryption & checking. > > > > > > They could be very useful for quarterly DB dumps, etc. > > > > > > What do people think about promoting this as an option? > > > > > > > Unfortunately, I'd say that unless a number of people are downloading a > > dataset at the same time or are happy to share data after they've > finished > > (seeding), it wouldn't work out any more efficient - but is more complex > > for > > someone to set up. > > > > NZ Internet connections are pretty average and many have upload data > caps, > > so the seeding rate is really bad compared with elsewhere (according to a > > major ISP I spoke with). > > > > My thoughts were that after a period of time a cadre of about a dozen > institutional semi-permanent seeders, e.g. KAREN-affiliated public sector > agencies, businesses & ISPs would develop. There is a tradition of ISPs > providing Linux & BSD distros (that are several hundred mb each) for free: > http://mirror.ihug.co.nz/ > > Is that list of seeders a realistic expectation? > > Tim. > > ----------------------------------------- > Full text of this topic in The Open Government Ninjas: > http://groups.open.org.nz/r/topic/3d0oygKGB6ltbMubvgME1f > > To leave The Open Government Ninjas, email > <email obscured>?Subject=unsubscribe > > Start your own free groups and site with > OnlineGroups.Net http://onlinegroups.net > > Host your own online groups site with > GroupServer http://groupserver.org > -- M Fabiana Kubke Department of Anatomy | University of Auckland | New Zealand (+64) 9 373-7599 Ext 86002 | (+64)9 923 6002 (direct) | Mobile: (+64) 210 437 121 Skype: superfabs | http://twitter.com/Kubke | http://identi.ca/kubke | http://buildingblogsofscience.wordpress.com | http://sciblogs.co.nz/building-blogs-of-science | http://popscinz.wordpress.com
Loading…
Privacy | Acceptable Use | Terms of Service | About OnlineGroups.Net | Contact OnlineGroups.Net
Start an OnlineGroups.Net site for easier email collaboration in your organization.
Powered by GroupServer, the open source web-based mailing list manager.