IMPORTING LARGE DATA ONTO OSM

Written by Jurg Wilbrink GIS.

On the 12th of June 2018 510 uploaded 13.367 roof materials and shapes to Open Street Map.

The import is a result of all the activities the data team of the Netherlands Red Cross (510) conducted in preparation and response to the hurricane ‘Irma’ that hit land on the island of St. Maarten on the 6th of September 2017. Often, the import of larger data sets onto OSM are considered difficult, time consuming, or in general confusing.

This blog post will describe the process of collection, cleaning, scripting, community buy-in, and finally, importing.

Irma Mapping Task

When the hurricane Irma got upscaled on the 4th of September to a CAT 4 storm and growing, the 510 team started its data collection and data response for St. Maarten. One of the first steps is to assess the state of the available map of the -to be – affected area. Accurate information on the situation pre-disaster will help the Red Cross’ response efficiency. After assessing the state of the map of the island, we found that a large share of the buildings were mapped incorrectly or were completely missing.

This is where the Missing Maps project kicked into action: a priority mapping task was created for the Dutch part of the island. All the mapping tasks are created through the Humanitarian OpenStreetMap Team’s (HOT) Tasking Manager; a tasking platform developed to create, organize, and track the progress of mapping actions on the largest open and web-based map database available: OpenStreetMap (OSM).

In principal, mapping is done through crowd mapping (events) where satellite imagery is traced to draw in buildings, roads, and waterways onto the map. The same was done for St. Maarten. After the process of acquiring the latest aerial photographs from St. Maarten from the French cadaster (IGN) (with thanks to the Dutch Kadaster) from February, 2017, and permission to map the French part of the Island (North side, Saint Martin) the call for action to our mapping community could be published:

Urgent mapping task #3491 was born. In the night of the 5th to the 6th of September the Dutch side (South side) of the island was mapped and validated for 95%. By 6 PM that evening the whole island (including North) was completely mapped by remote volunteers responding to the call.

 

 

Damage assessment

Part of the team deployed to St. Maarten to conduct the initial damage assessments on the ground as well as collect drone imagery of the island. The ground-based damage assessment was initially used to validate the initial damage assessment of the European Commission conducted by Copernicus. Mainly, the drone imagery were essential for the team in the Netherlands, as -due to cloud cover- no new aerial or satellite imagery could be acquired. Initially, third-party drone imagery was used, the imagery collected by RescUAV, that was shared openly on OpenAerialMap. After the first 510 drone imagery from a section of the island (Cay Bay) came in on the 13th of September, the comparison between pre- and post-disaster imagery could continue.

Adapting the online editor

The team adapted the online editor of OSM: the ID-editor, to be used for the damage classification of the buildings on the island. This adapted version of the ID-editor, was used to classify buildings into 4 different categories of visual damage ranging from no observable damage to complete visible damage. The damage classification tool follows the Harvard Humanitarian Initiative guidelines to conduct geospatially-based assessments of wind disaster damage to buildings.

Roof_Shapes and Roof_Materials

In parallel to the visual damage assessment of the buildings, a special mapping request was done to map all the roof types and the roof materials (in OSM language: roof_shape and roof_material). The collection of both datasets were done In collaboration with the Dutch cadaster, building on their expertise of interpreting high resolution aerial imagery. Through this work a total of 14.556 roof materials and shapes were collected initially in a separate database.

The data wasn’t directly collected on OSM to be sure not to ‘pollute’ the OSM database with potentially incorrectly collected data or data that is irrelevant to OSM. Subsequently, due to the urgency of the data and the lack of a standard operational procedure it was chosen to store the data separately.

OSM Import

After the data storm slowed down, and after the evaluation of the data, it was decided that the data set of roof shapes and materials were suitable and relevant to openly share on OSM. The import onto OSM has some regulations and policies to take into consideration. Most importantly, keeping in mind not to do any damage to existing data on OSM.

Figure 1 OSM Import steps

Prerequisites & Community buy-in

As we are working regularly with OSM data and are organizing quite a lot of mapathons yearly in The Netherlands, we checked the first step ‘prerequisites’ from the list. The buy-in for the OSM community in the geographic region was also pretty straight-forward. We are heavily involved in the OSM community of The Netherlands, and as St. Maarten is one of autonomous states of the Kingdom of The Netherlands, we considered it was safe to say we checked the second step after joining all the relevant mailing lists and emailed our plan to the imports mailing list.

License

The license compatibility was a matter of thinking of a license for the data, as 510 obtained the data through primary data collection. In the end, the data was released under the  ODbL license.

Import Plan

The import project and its permissions are registered in the import catalogue wiki and a separate wiki page is created to describe the import plan. In the import plan, the context of the data is described, the scheduled launch, and most importantly the way data is handled is described. The plan is designed to not do any damage to the existing OSM data. The data and all the scripts to clean up the data are added to the Netherlands Red Cross Github. Extracting the correct XML file using the Overpass Query Language gave us the opportunity to dive deep into the wonderful world of the Overpass API. Finally, the data owners are acknowledged on the contributors wiki.

Import Review

The import needs to reviewed by the imports authority. Therefore, the import team needs to register to the import mailing list and request permission to do the import. After some silence on the list, which we took as a good sign, a reminder was sent with a deadline attached to it. Some very useful feedback came back, and were added to the import plan and within a week, the import was officially approved.

Uploading

The uploading is the fun part. Be sure to make a dedicated OSM user account; in our case ‘StMaarten_Roofimports’. The cleaned database is combined with the latest OSM extract and the final upload is done through the Java OpenStreetMap (JOSM) software. A staggering 13.367 roof materials and shapes were uploaded to OSM, this equals 91.8% of all the buildings on Dutch St. Maarten.

Leave a Reply