The ISPRS semantic labelling benchmark
The ISPRS benchmark on object detection and 3D building reconstruction has demonstrated the usefulness of commonly available benchmark datasets to allow a comparison of state-of-the-art methods, and to reveal current deficits (see Rottensteiner et al., 2014).
The main motivation for the ISPRS Working group III/4 “3D Scene Analysis” to create and publish the follow-up benchmark on 2D semantic labelling was the observation that current very high resolution remote sensing data add significant new challenges to scene classification.
In contrast to images of low resolution we are confronted with a high intra-class variance, while at the same time the inter-class variance might be low. The relevant literature proposes several solutions to the problem of very high resolution remote sensing data classification. This concerns for instance a potential pre-segmentation, the feature extraction and selection, as well as classification paradigms and methods.
Many research communities work on the field of semantic analysis and classification of (urban) remote sensing images, such as photogrammetry, remote sensing, computer vision, or the geographical sciences. We believe that the 2D semantic labelling benchmark dataset offers the possibility to compare and discuss the respective methods in the different fields of research. In this regard the Geobia 2016 conference will offer an ideal platform to researchers from different communities to compare and discuss approaches and results.
We invite prospective authors to use the benchmark data for their experiments. In a common session during the conference the proposed approaches and results will be discussed. One aim is to prepare a journal paper based on the findings of the benchmark exercise and the conference session, and hence to make interlinks between the research fields more explicit. The most active contributors to the benchmark will be invited as co-authors.
Technical details:
On the benchmark website you can find a description of the two available datasets, a class definition and details on how to submit your results. There are two sets available: Vaihingen/Germany which shows a typical small city structure. The ground resolution of image data is 9cm. The second set was captured over the city of Potsdam, close to Berlin. This urban structure shows more complex object interactions. The ground sampling distance here is 5cm. Basically the entire area is split into several single tiles, and for each tile we have a high resolution orthoimage and a digital surface model available. These data are available after registration through the link on the website. For approximately half of the tiles we also provide reference label images, which can be used to train a classifier and/or to validate developed classification methods. For all tiles without ground truth labels we expect the participants to deliver the respective label images in a predefined format. In addition we expect the participants to describe their method, e.g. by adding a paper or a technical report, if applicable the ruleset (which will not be shared with others).
The results will then be evaluated and published on the benchmark website. On the front page you find a table which indicates most relevant accuracy measures for each class (one line per participant or method). In addition there is a link to further details which opens a new window with confusion matrices and visualisation of results and evaluation.
Reference
Rottensteiner, F., Sohn, G., Gerke, M., Wegner, J. D., Breitkopf, U., & Jung, J. (2014). Results of the ISPRS benchmark on urban object detection and 3D building reconstruction. ISPRS Journal of Photogrammetry and Remote Sensing, 93, 256-271. doi: 10.1016/j.isprsjprs.2013.10.004