Chch earthquakes generated 660 GB of GIS data that everyone needed now - how NZ te

Ref http://www.nesi.org.nz/case-study-digital-support-in-national-crisis

I've just came across a really interesting article describing the role
of the eResearch community and the infrastructure that was required to
support the GIS response. I'm really glad to see agencies cooperating
like this. It looks like there were tons of players that really got
stuck in to make things happen:

"The community supported users from across New Zealand including the
Defence Force, National Crisis Management Centre, Police and Fire
Service, Land Information New Zealand, Landcare Research, GNS and
commercial geospatial data providers Koordinates (NZ) and Google (US,
Aus). Much of the data loading and redistribution to the Defence Force
took place in the Centre for eResearch labs at The University of
Auckland."

Technically, there were several interesting developments described. Of
special interest was something called BeSTGRID DataFabric. It seems to
be something like an FTP server?

Some things that I would like further info on:

Speaking  from a position of ignorance, I'm still fairly convinced
that the use of BitTorrent would have been a better solution. It
seemed like there was a problem with reliability. The size of the
files (biggest was 350GB) is a small movie.

One final point that struck me was that without public access to the
system, open licencing is stunted. Much of the data has been licenced
under CC-BY, which is great. The article doesn't describe where to get
it. However, if it's kept within KAREN/BeSTGRID/?/NeSI/, the potential
positive impact from liberal licences is diminished.


Please correct any mistakes in my reading of the original text or
inferences I've drawn!

On 2011-09-23, at 12:01 , Juha Saarinen wrote:
> Are you comparing it to commercial movies? Normal HD file sizes are
> somewhere in the 7-9GB region, Blu-Ray 24-50GB.
>
> Thing is, this is 2011 and 660GB is very little. They could've bought a
> bunch of 1TB hard drives, put the data onto those and couriered them off.

Indeed. CCC received the processed satellite imagery on the Sunday after the
earthquake on the 22nd via hard drive courtesy of the NZ Fire Service. There
had been talk of using Karen, but Karen didn't have the access control required
(my understanding), and the NZFS were very protective of the imagery at that
point.

It wasn't until LINZ stepped up later in March that the imagery was released
publicly.

Cheers Gav

Tim - happy to chat about the setup. The service is based on a distributed file
system called iRODS[1] and we were using the parallel streaming / pipelining
features within GridFTP[2]. (for those interested in the tech, this is the
technology behind the SaaS platform for transferring large scientific datasets,
GlobusOnline[3]. We're currently in planning for a transition and expansion of
this service, into NeSI[4], and with additional research institutions
connected, heading towards a national distributed file system for research.

There was an issue at the start as there were breakages in the KAREN network
due to the quake and a hub in Christchurch feeding some of our research
institutions with campuses elsewhere in NZ. As Gavin notes, TB drives were used
initially. Once we had this resolved, we were routinely distributing data
between multiple parties across the country, between Auckland, Napier,
Palmerston North, and Christchurch. This routine sharing and processing went on
for many weeks as different data sets were acquired and processed.

re comments on licensing, this was a difficult one to navigate. As a service
provider, we provided the security that was initially required, as defined by
the license for the original satellite image data negotiated by the involved
agencies. In the end licensing was resolved and the data has been available
since around March 2011[5].

cheers, Nick

[1] http://www.irods.org
[2] http://www.globus.org/toolkit/data/gridftp/
[3] https://www.globusonline.org/
[4] http://www.nesi.org.nz
[5] http://df.auckland.ac.nz/BeSTGRID/home/NZDF/