Grid computing in NZ - could a multisector model work?
Summary
- There are 8 posts — by 6 authors — in this topic.
-
Posts with files From File Date Michael Birks Apr 01 23:16 NZDT - Latest post made by Michael Birks at Apr 01 23:16 NZDT
With the success of BeSTGRID[1], does anyone think that it would be possible to create some structure that would enable everyone conducting computationally expensive analyses to join in? Specifically, whether something like a spot market for computational resources could be created. I mean, spinning up a 12 machine Hadoop cluster on AWS running Nutch is fine. However, it strikes me as a missed opportunity to need to send data internationally when there are bound to be resources here. There are probably businesses that would be happy to allocate machines at 3am for batch processing by Auckland Uni, just as there are companies/individuals without machines who are likely to be happy to pay to get their data processed. Any thoughts welcome. Tim [1] bestgrid.org
It would be useful to get someone from the KAREN network to comment, the idea came up in the science/academic space a couple of years ago. Sent from my iPhone
On 31/03/2011, at 6:35 AM, Tim McNamara <email obscured>> wrote: > With the success of BeSTGRID[1], does anyone think that it would be possible > to create some structure that would enable everyone conducting > computationally expensive analyses to join in? > > Specifically, whether something like a spot market for computational > resources could be created. I mean, spinning up a 12 machine Hadoop cluster > on AWS running Nutch is fine. However, it strikes me as a missed opportunity > to need to send data internationally when there are bound to be resources > here. There are probably businesses that would be happy to allocate machines > at 3am for batch processing by Auckland Uni, just as there are > companies/individuals without machines who are likely to be happy to pay to > get their data processed. > > Any thoughts welcome. > > Tim > > [1] bestgrid.org > > ----------------------------------------- > Full text of this topic in The Open Government Ninjas: > http://groups.open.org.nz/r/topic/2mQwdTWml6fdLKuuiKyx0A > > To leave The Open Government Ninjas, email > <email obscured>?Subject=unsubscribe > > Start your own free groups and site with > OnlineGroups.Net http://onlinegroups.net > > Host your own online groups site with > GroupServer http://groupserver.org
A Geni company (Asterisk - Linux people) did some work in this area, especially for Weta Studios and looked at creating some sort of ongoing market for computational resources but I think that it all got shelved as being "too hard" with "not enough interest". Of course, that doesn't mean that a new initiative by someone with lots of energy wouldn't work. This may now be a better time to do this (weta stuff was at least 4-5 years ago). Regards Doug Hunt
On 31 March 2011 09:35, Tim McNamara <email obscured>> wrote: > With the success of BeSTGRID[1], does anyone think that it would be possible > to create some structure that would enable everyone conducting > computationally expensive analyses to join in? > > Specifically, whether something like a spot market for computational > resources could be created. I mean, spinning up a 12 machine Hadoop cluster > on AWS running Nutch is fine. However, it strikes me as a missed opportunity > to need to send data internationally when there are bound to be resources > here. There are probably businesses that would be happy to allocate machines > at 3am for batch processing by Auckland Uni, just as there are > companies/individuals without machines who are likely to be happy to pay to > get their data processed. > > Any thoughts welcome. > > Tim > > [1] bestgrid.org > > ----------------------------------------- > Full text of this topic in The Open Government Ninjas: > http://groups.open.org.nz/r/topic/2mQwdTWml6fdLKuuiKyx0A > > To leave The Open Government Ninjas, email > <email obscured>?Subject=unsubscribe > > Start your own free groups and site with > OnlineGroups.Net http://onlinegroups.net > > Host your own online groups site with > GroupServer http://groupserver.org >
> > With the success of BeSTGRID[1], does anyone think that it would be > possible > to create some structure that would enable everyone conducting > computationally expensive analyses to join in? > This is a really interesting question to ask. I'm currently working on the national Framework for eResearch Adoption[2]. This is looking at what we need in place, to help researchers to use technology, to do research at different scales (spatial, temporal, social, collaborative) than has been possible before. It includes high speed networks (KAREN), grid computing, high performance computing, data reuse & management infrastructure and services, analytical tools & models, collaboration tools, and tools to support discovery and access to scholarly works and other research outputs. The major recent development on the computational front is the National eScience Infrastructure (NeSI)[3]. This is essentially the group of research institutions that have supercomputing resource working together to share access to and investment in that resource, and make it available to others. I'm not sure whether there'll be the opportunity for people outside the CRIs and Unis to get access through that, but I can find out. A key point to note is that there are different types of supercomputer resource for different types of computationally intensive tasks. Broadly, there are two main types (with many sub distinctions). They are: 1. *Parallel computing* - Tasks that can be broken down into pieces, the computational work done, and then the pieces reassembled - examples include the way WETA uses their arrays of computers to render 3D animation for movies, aspects of computational chemistry, gene sequencing etc. Generally involves computers as clusters, either using commodity hardware or specialised hardware 2. *High performance computing* - tasks where a high degree of interaction between calculations within models are required - examples include weather forecasting, simulations of the human heart/cardiovascular system, etc - generally requires large supercomputers with a high degree of interconnect between memory/processing, e.g. NIWA's Cray, NIWA's new IBM p575 POWER6 supercomputer, Canterbury's BlueFern This is a generalisation of course, and there's lots of nuance around the way different problems can be addressed with different compute resources. Within NZ there are several examples of both broad types, and NeSI is working to bring those together into a sort of virtual national compute infrastructure. Whatever the shape of the problem you're addressing, if you're outside of a research org, any use of this infrastructure would likely be on a user pays basis (and to some extent this may be true if you're inside a research org). If you're doing highly parallel tasks it may or may not be cheaper to access this in the cloud than via local systems (although as you indicate, the cost of moving data from within NZ to outside NZ might change this equation). There are also varying levels of "driver's licence" required to be allowed to run code on a supercomputing resource. These vary from not very strict requirements at the parallel/cluster end, to fairly rigorous at the HPC end. Julian [1] bestgrid.org [2] http://www.eresearch.org.nz/fera [3] http://www.nesi.org.nz/
Thanks for your comprehensive note Julian. On 31 March 2011 15:32, Julian Carver <email obscured>> wrote: > > > > With the success of BeSTGRID, does anyone think that it would be > > possible > > to create some structure that would enable everyone conducting > > computationally expensive analyses to join in? > > > > This is a really interesting question to ask. [...] A key point to note is that there are different types of supercomputer
> resource for different types of computationally intensive tasks. Broadly, > there are two main types (with many sub distinctions). They are: > > 1. *Parallel computing* - Tasks that can be broken down into pieces, the > computational work done, and then the pieces reassembled - examples > include > the way WETA uses their arrays of computers to render 3D animation for > movies, aspects of computational chemistry, gene sequencing etc. > Generally > involves computers as clusters, either using commodity hardware or > specialised hardware > 2. *High performance computing* - tasks where a high degree of > interaction between calculations within models are required - examples > include weather forecasting, simulations of the human > heart/cardiovascular > system, etc - generally requires large supercomputers with a high degree > of > interconnect between memory/processing, e.g. NIWA's Cray, NIWA's new IBM > p575 POWER6 supercomputer, Canterbury's BlueFern > I was specifically thinking about embarrassingly parallel problems[1], which are what grid computing deals with best. I had envisaged a two-way model. The system would allow non-institutional users to add under utilised hardware to the pool, as well as being able to request resources from the pool. > Whatever the shape of the problem you're addressing, if you're outside of a > research org, any use of this infrastructure would likely be on a user pays > basis (and to some extent this may be true if you're inside a research > org). > If you're doing highly parallel tasks it may or may not be cheaper to > access > this in the cloud than via local systems (although as you indicate, the > cost > of moving data from within NZ to outside NZ might change this equation). > I think that it's necessary for everyone to pay. In fact, I was counting on it. Otherwise there would be no (financial) incentive for external parties to add their computing resources to the grid. Thinking more broadly though, I would be surprised if it were cheaper for data to be processed by infrastructure as a service providers. In the current environment, the Crown sould not be spending money on building a new science infrastructure if that is expected to be more expensive than alternatives that already exist. > There are also varying levels of "driver's licence" required to be allowed > to run code on a supercomputing resource. These vary from not very strict > requirements at the parallel/cluster end, to fairly rigorous at the HPC > end. > My outsider's 2c, rather than certification of skills, e.g. a "driver's licence", usage of the resource should be allocated through a pricing structure. That way, there will be an increased incentive for research to be carried out by skilled data analysts. High prices would also lead to an increase of supply, lowering the price and leading to more research. However, I'm not sure of the established norms in other areas of science, such as access to telescopes. Respecting cultural norms is quite important within science. If there is an established protocol about how these types of resources should be distributed, it makes sense to follow the norm. Julian > [1] http://en.wikipedia.org/wiki/Embarrassingly_parallel
[forwarding to the list]
---------- Forwarded message ---------- From: Tim McNamara <email obscured>> Date: 31 March 2011 14:19 Subject: Re: [ninjas] Grid computing in NZ - could a multisector model work? To: Doug Hunt <email obscured>> Hey Doug, Thanks for that. Very interesting. I think the environment has changed quite a bit in recent years: - emergence of cloud computing => more people happier with giving their data to others - lots more data => greater demand - budgets are tight => people willing to rent spare CPU cycles (greater supply) Some type of grid computing market would probably need to emerge from a research project. Either that or leverage one of the other market tools - the electricity sector has a very sophisticated electronic market & there's bound to be capability within NZX. However, I think academia has the connections, time and expertise to make it happen. There are already some mature tools for distributing work (e.g. BOINC). I think the hard part will be the platform that connects all of the devices. Tim On 31 March 2011 13:52, Doug Hunt <email obscured>> wrote: > A Geni company (Asterisk - Linux people) did some work in this area, > especially for Weta Studios and looked at creating some sort of > ongoing market for computational resources but I think that it all got > shelved as being "too hard" with "not enough interest". > > Of course, that doesn't mean that a new initiative by someone with > lots of energy wouldn't work. This may now be a better time to do > this (weta stuff was at least 4-5 years ago). > > Regards > > > Doug Hunt > > > > On 31 March 2011 09:35, Tim McNamara <email obscured>> wrote: > > With the success of BeSTGRID[1], does anyone think that it would be > possible > > to create some structure that would enable everyone conducting > > computationally expensive analyses to join in? > > > > Specifically, whether something like a spot market for computational > > resources could be created. I mean, spinning up a 12 machine Hadoop > cluster > > on AWS running Nutch is fine. However, it strikes me as a missed > opportunity > > to need to send data internationally when there are bound to be resources > > here. There are probably businesses that would be happy to allocate > machines > > at 3am for batch processing by Auckland Uni, just as there are > > companies/individuals without machines who are likely to be happy to pay > to > > get their data processed. > > > > Any thoughts welcome. > > > > Tim > > > > [1] bestgrid.org > > > > ----------------------------------------- > > Full text of this topic in The Open Government Ninjas: > > http://groups.open.org.nz/r/topic/2mQwdTWml6fdLKuuiKyx0A > > > > To leave The Open Government Ninjas, email > > <email obscured>?Subject=unsubscribe > > > > Start your own free groups and site with > > OnlineGroups.Net http://onlinegroups.net > > > > Host your own online groups site with > > GroupServer http://groupserver.org > > >
I think this is what greenbutton.co.nz does.
On 1/04/2011 8:51 p.m., laurence.millar wrote: > I think this is what greenbutton.co.nz does. > ----------------------------------------- > Full text of this topic in The Open Government Ninjas: ::snip:: GreenButton looks mostly focused on the image rendering side of things - they were involved in a fun 'Raygun' project with Weta last year - over more general purpose data analysis. Although, looking at their website, the plugin for Geneious looks like a step in the right direction. I thought of them as well, with Tim's original post, until I saw that they're leveraging Microsoft Azure, which I presume is, like Amazon's products, run out of places not-in-NZ. Tim, were you envisaging something like a BOINC tool where there was actually a cash payment per work unit completed? How much 'embarrassingly parallel' work is actually out there that isn't overly encumbered by the need for security, or some kind of time criticality that farming it out to potentially, Jo(e)'s Netbook is going to hold it up? And that isn't already being done by some other 'free' project?
The following file was added to this topic:
Loading…
Privacy Acceptable Use Terms of Service About OnlineGroups.Net Contact OnlineGroups.Net
Start an email groups site like this one with OnlineGroups.Net. Collaborate and share information online in your organization.