DATA OPTIMIZATION FOR 3D MODELING AND ANALYSIS OF A FORTRESS ARCHITECTURE B.G. Marinoa, A. Masierob, ∗, F. Chiabrandoc, A.M. Linguad, F. Fissoreb, W. Błaszczak-Bake, A. Vettoreb a DiARC Department of Architecture, University of Studies Federico II, Naples Italy - bianca.marino@unina.it b Interdepartmental Research Center of Geomatics (CIRGEO), University of Padova, Viale dell’Università 16, Legnaro (PD) 35020, Italy - (andrea.masiero, francesca.fissore, antonio.vettore)@unipd.it c Department of Architecture and Design, Polytechnic of Turin, Viale Mattioli 39, Torino, 10125, Italy - filiberto.chiabrando@polito.it d Department of Environment, Land and Infrastructure Engineering, Polytechnic of Turin, C.so Duca degli Abruzzi 24, Torino, 10129, Italy - andrea.lingua@polito.it e Institute of Geodesy, University of Warmia and Mazury in Olsztyn, Oczapowskiego 2, 10-719 Olsztyn, Poland - wioleta.blaszczak@uwm.edu.pl KEY WORDS: Point Cloud Optimization, Data Reduction, Segmentation, Restoration, Cultural Heritage Buildings ABSTRACT: Thanks to the recent worldwide spread of drones and to the development of structure from motion photogrammetric software, UAV photogrammetry is becoming a convenient and reliable way for the 3D documentation of built heritage. Hence, nowadays, UAV photogrammetric surveying is a common and quite standard tool for producing 3D models of relatively large areas. However, when such areas are large, then a significant part of the generated point cloud is often of minor interest. Given the necessity of efficiently dealing with storing, processing and analyzing the produced point cloud, some optimization step should be considered in order to reduce the amount of redundancy, in particular in the parts of the model that are of minor interest. Despite this can be done by means of a manual selection of such parts, an automatic selection is clearly much more viable way to speed up the final model generation. Motivated by the recent development of many semantic classification techniques, the aim of this work is investigating the use of point cloud optimization based on semantic recognition of different components in the photogrammetric 3D model. The Girifalco Fortress (Cortona, Italy) is used as case study for such investigation. The rationale of the proposed methodology is clearly that of preserving high point density in the model in the areas that describe the fortress, whereas point cloud density is dramatically reduced in vegetated and soil areas. Thanks to the implemented automatic procedure, in the considered case study, the size of the point cloud has been reduced by a factor five, approximately. It is worth to notice that such result has been obtained preserving the original point density on the fortress surfaces, hence ensuring the same capabilities of geometric analysis of the original photogrammetric model. 1. INTRODUCTION Given the complexity and specificity of restoration issues in me- dieval fortress architectures, detailed, accurate and reliable anal- ysis of the current status of such cultural heritage architectures should be done in order to provide appropriate information to re- storers. In particular, the specific structure characteristics and the consequent applicable conservation and valorization strate- gies are fundamental factors to be taken into account in order to properly design the 3D survey and successfully extract and fully exploit the information from the obtained 3D model of the struc- ture. This work considers the survey and 3D modeling process of the Girifalco Fortress (Cortona, Tuscany, Italy). A fortress on the Cortona hilltop was originally built in the 5th century BC. How- ever, after such building has been damaged during battles, hence it has been rebuilt during the 17th century: the original walls were substituted in order to cope with the newly developed cannons. The current look of the fortress (visible in Fig. 1) is actually the same obtained after the realization of such new walls. The cul- tural heritage and historical importance of the fortress is also due to the famous families that owned it, and, in particular to the De Medici family, which is worldwide known for its important role during Italian Renaissance. Since the final interest in this work is that of providing suitable in- formation to the restorers, this work aims at investigating a proper ∗Corresponding author. Figure 1. Girifalco fortress, Cortona, Italy. integration between restoration goals and geomatics tools. The rationale is that of designing a survey and data processing driven by the needs of extracting the information suitable for properly supporting the conservation and restoration process. A preliminary step was the definition of the survey goals, in order to determine the building characteristics to be determined and de- tectable in the produced 3D model. Then, both UAV photogram- metry (Fig. 2 shows an orthophoto produced by properly pro- cessing UAV imagery) and Terrestrial Laser Scanning have been used in order to acquire the 3D information of interest for prop- erly describing the fortress. Nevertheless, this work focuses on the automatic processing of the 3D model obtained by means of UAV photogrammetry. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W11, 2019 GEORES 2019 – 2nd International Conference of Geomatics and Restoration, 8–10 May 2019, Milan, Italy This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLII-2-W11-809-2019 | © Authors 2019. CC BY 4.0 License. 809 Figure 2. Orthophoto of the Girifalco fortress. UAV photogrammetric survey of the Girifalco Fortress has been carried out on September 8, 2017, with the drone DJI Phantom 4. The UAV flew at an average altitude of 40 m over the ground (ground sample distance was 1.4 cm, approximately), acquiring 960 images of the fortress area, with (at least) 60% of overlap- ping between images (both along the same strip and between the closest images from different strips). Given the importance of properly describing the fortress walls, both nadir and oblique im- ages have been collected during the flight. Actually this ensured also a better camera network geometry (Fraser and Stamatopou- los, 2014, Chiabrando et al., 2017), which is of fundamental im- portance in order to ensure reliable reconstruction results, in par- ticular in the camera self-calibration case. Then Agisoft Photoscan 1.3.2 has been used in order to obtain a photogrammetric 3D model of the fortress. To this aim, the photogrammetric has been aided by the use of 27 targets, whose positions have been measured in a GNSS RTK survey, with av- erage accuracy (root mean square (RMS) error) of 1.6 cm. 10 targets have been used as control points, and 7 as check points, leading to an average planimetric and altimetric error of 2.0 cm and 2.4 cm, respectively. Agisoft Photoscan generated a photogrammetric point cloud of 40 Million points. The average point spacing on the central part of the model (the fortress area) is 3.0 cm. Actually, 3D modelling of large sites can lead to huge point clouds, which can lead to dramatically long computational times for pro- cessing and information extraction from such models. Conse- quently, several works recently considered the problem of reduc- ing the size of the generated point cloud while preserving its ge- ometric information in order to ease the processing phase and make it faster (Błaszczak-Bak, 2016). However, this work aims at exploiting a semantic interpretation of the points in the model in order to properly determine which areas should be sub-sampled more. To be more specific, since the fortress walls in the 3D model shall be further analyzed for restoration purposes, the goal is that of automatically extract such parts from the model and preserve their model resolution, whereas, the other parts of the generated point cloud (e.g. vegetation) will be significantly sub- sampled (actually they can also be discarded, if needed). Aiming at formulating a processing strategy that can be used for easily managing, interpreting, understanding and extracting in- formation from the generated model, in this work the following steps of 3D data processing has been considered: • automatic recognition of vegetated areas, by exploiting both spatial characteristics of tree point clouds and their intensity measurements (on either RGB imagery or LiDAR), • optimization of the point density in order to make the 3D model easier to be managed. This step can take advan- tage also from recently developed data optimization meth- ods (Błaszczak-Bak, 2016), if needed, • in particular, the produced dataset is optimized in order to ease the automatic extraction of information about the build- ing walls. This kind of procedure can be useful for in- stance for the automatic generation of semantic models (af- ter a proper segmentation and classification (Vosselman et al., 2004, Rabbani et al., 2006, Makuti et al., 2018)), e.g. BIMs and CityGML models. The the paper is organized as follows: first, Section 2. presents the proposed point cloud segmentation and classification proce- dure. Then, Section 3. deals with the data reduction and opti- mization problem. Finally, some conclusions will be drawn on the proposed procedure and on the possibility of automatically extracting information about damages on the building walls in Section 4.. 2. SEGMENTATION AND CLASSIFICATION This section aims at the automatic extraction of certain geometric information from the photogrammetric point cloud. In particu- lar, it is worth to notice that vegetated areas are of minor interest in the 3D model of the reconstructed fortress. Furthermore, the generated point cloud is very noisy on the vegetated areas. Since most of the fortress elements can be well approximated with sim- ple geometric shapes (e.g. planar surfaces), an automatic proce- dure based on the Random Sample Consensus (RANSAC) (Fis- chler and Bolles, 1981) is used in this work in order to extract the fortress parts from the point cloud. However, the presence of very noisy vegetated areas can negatively affect the outcomes of the RANSAC algorithm. Consequently, a vegetation classi- fication and segmentation procedure is applied before detecting the surfaces associated to the fortress. Despite the different im- plementation, the rationale of the proposed approach is relatively similar to the semantic segmentation principle (Long et al., 2015, Girshick et al., 2014, Noh et al., 2015). 2.1 Predictors Similarly to certain land cover classification methods (Huang et al., 2002, Foody, 2002, Gislason et al., 2006), intensities of the radiation coming from the area of interest and measured at several wavelengths can be used to estimate the vegetated areas. However, differently from multi-spectral classification techniques, which can exploit the Normalized Difference Vegetation Index (NDVI) in order to make such classification, in this case the use of a standard camera does not allow to compute such index. Nev- ertheless, such intensities can clearly be indicative of the vegeta- tion presence. In particular, the ratio between the green compo- nent and each of the other two is used in this work. Actually, the availability of 3D information about the point posi- tions and about the local point densities can also be exploited in order to determine if a point correspond to a vegetated area. To this aim, an approach based on considerations similar to those presented in (Habib and Lin, 2016) has been implemented. A lo- cal surface fitting is used to check whether points approximately lie on planar surfaces. For each considered point, all its neigh- borhood contained in the sphere centered in such point and with radius 0.5 m are used in order to determine the local variability of The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W11, 2019 GEORES 2019 – 2nd International Conference of Geomatics and Restoration, 8–10 May 2019, Milan, Italy This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLII-2-W11-809-2019 | © Authors 2019. CC BY 4.0 License. 810 the point cloud in such area. In particular, the covariance matrix of such point positions is computed: points on a fortress wall shall form a locally smooth surface, hence the third eigenvalue of the computed covariance matrix should be small. Differently, points on vegetated areas should be associated to comparable values of the covariance matrix eigenvalues. It is worth to notice that: • if the last eigenvalue of the covariance matrix is small with respect to the other ones, the neighborhood of the considered point can be quite well approximated by a planar surface • when the last eigenvalue is not small with respect to the oth- ers it is not possible to automatically conclude about such point being or not part of the vegetation. In accordance with the description provided above, the smallest eigenvalue of the local sample covariance matrix, along with the local point density and with the point intensities, is used as a pre- dictor of the point class, as described in the following subsection. 2.2 SVM classifier A number of methods have been proposed in the literature in or- der to properly classify data appertaining to certain categories. In particular machine learning and deep learning techniques have been widely used in order to improve the performance of classi- fiers. In particular, certain features are usually either automati- cally determined (as usual in deep learning methods) or manually selected (machine learning) is order to ease the solution of the classification problem (LeCun et al., 2015, Bishop, 2006, Facco et al., 2013). Among the possible methods, a Support Vector Machine (SVM) classifier has been used in this work (Suykens and Vandewalle, 1999). Two rectangular regions, containing approximately 1000 points each, have been manually selected on a fortress wall and on a vegetated area on the north-east area of the reconstructed model and their predictor values fed into the SVM in order to properly learn the classifier parameters. Then, the trained SVM is used to classify the vegetated areas on all the model. Figure 3, 4 and 5 show the results of the classification. In partic- ular, Fig. 3 shows the vegetated areas, whereas Fig. 4 compares the side view of the model before (a) and after (b) filtering out the vegetation points. It is worth to notice that, differently from the original point cloud, the filtered point cloud contains much less noisy points (Fig. 4(b)). Such result is also confirmed by the top view of the fortress model after filtering out the vegetation points (Fig. 5). 2.3 Detection of planar surfaces Once the vegetation has been filtered out, the remaining point cloud (Fig. 5) is used in order to detect the fortress surfaces. In particular, first, RANSAC is used to detect vertical surfaces. Then, horizontal regular surfaces are determined: surfaces at heights comparable with the altitudes of the walls are classified as ceil- ings, whereas the other points (and planar surfaces) are generi- cally classified as “ground”. Actually, it is worth to notice that “ground” points can be segmented further in order to determine the presence of other objects (e.g. cars). Despite this can be of great interest in several applications, for instance for digital ter- rain model generation and for more advanced information extrac- tion from the point cloud (Costantino and Angelini, 2011), this is out of the scope of this work. Figure 3. Extracted vegetation points. (a) (b) Figure 4. (a) West side view of the point cloud generated by Agisoft Photoscan. (b) Almost vegetation-free subsampled point cloud used for planar surface detection. Fig. 6 shows the resulting classification: “ground” points are shown (a), whereas the extracted fortress walls and ceiling are shown in (b) and (c). 3. SEMANTIC BASED MODEL OPTIMIZATION The previously presented segmentation procedure provides a se- mantic classification of the 3D point cloud that can be conve- niently exploited in order to change the size reduction of each point class depending on the specific user interests. In practice, since in this work the main interest is that of preserv- The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W11, 2019 GEORES 2019 – 2nd International Conference of Geomatics and Restoration, 8–10 May 2019, Milan, Italy This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLII-2-W11-809-2019 | © Authors 2019. CC BY 4.0 License. 811 Figure 5. Top view of the point cloud generated by Agisoft Pho- toscan after vegetation removal. ing as much geometric information about the fortress building as possible, all its points are kept. However, if a higher compres- sion is needed, model compression can be obtained by exploiting for instance the approach presented in (Błaszczak-Bak, 2016) for point cloud optimization while maintaining most of the geometric information of the original model. Differently, here vegetation and ground areas are highly com- pressed, by a factor (greater than) 100 and 20, approximately. Given the minor interest in these areas, random sub-sampling is applied to such points in order to obtain the desired compression at a low computational cost. Nevertheless, point cloud optimiza- tion can be applied to them as well, if needed. The resulting model is shown in Fig. 7, where is quite clear the different point density for the considered classes. The obtained model has a significantly smaller point cardinality with respect to the original one (it is reduced by more than a factor five). 4. DISCUSSION This paper presented a point cloud segmentation and compres- sion method, inspired by semantic segmentation, in order to re- duce the of point cloud size while preserving most of the geomet- ric information of interest. Indeed, since the currently common methods for 3D modeling of built heritage usually provide huge point clouds including also parts of minor interest for the user, it is important to reduce the size of such models in order to ease the computational burden for analyzing it and extracting any in- formation of interest, and also to reduce the storage requirements for such models. Actually, when dealing with applications of cultural heritage restora- tion, it is of fundamental importance to keep as much information as possible about the object of interest. This motivated the de- velopment of an automatic segmentation procedure in order to identify the most relevant parts of the model point cloud. Thanks to such implemented segmentation and classification pro- cedure, then it is possible to apply segmentation based point cloud optimization: the optimization function can be tailored to the user’s needs and specified differently for the identified point classes. In the considered case study this allowed the application of very different compression factors to the determined point classes. (a) (b) (c) Figure 6. Points labeled as ground (a) and as part of the fortress (side view (b), and top view (c). Despite the points associated to fortress surfaces (e.g. walls, ceil- ings) have not been sub-sampled, the overall point cloud com- pression in our case study was more than a factor five. Further- more, the application of recently developed point cloud optimiza- tion methods can reduce the cloud size further while minimizing the loss of geometric information (Błaszczak-Bak, 2016). Thanks to the preservation in the final model of the original point density, then, similarly to (Masiero et al., 2019), the obtained model is suitable for the application of surface and structural analysis (Korumaz et al., 2017, Tucci et al., 2016, Masiero et al., 2015). A secondary result of the applied method is the production of a model affected by less noise, and, in particular, once properly improved, the ground classification can also be considered for the generation of digital terrain models of the surveyed areas. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W11, 2019 GEORES 2019 – 2nd International Conference of Geomatics and Restoration, 8–10 May 2019, Milan, Italy This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLII-2-W11-809-2019 | © Authors 2019. CC BY 4.0 License. 812 Figure 7. Reduced size point cloud. REFERENCES Bishop, C. M., 2006. Pattern recognition and machine learning. springer. Błaszczak-Bak, W., 2016. New optimum dataset method in Li- DAR processing. Acta Geodyn. Geomater 13(4), pp. 381–8. Chiabrando, F., Lingua, A., Maschio, P. and Losè, L. T., 2017. The influence of flight planning and camera orientation in uavs photogrammetry. a test in the area of rocca san silvestro (li), tus- cany. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 42, pp. 163. Costantino, D. and Angelini, M., 2011. Features and ground auto- matic extraction from airborne lidar data. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Facco, P., Masiero, A. and Beghi, A., 2013. Advances on multi- variate image analysis for product quality monitoring. Journal of Process Control 23(1), pp. 89–98. Fischler, M. and Bolles, R., 1981. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24(6), pp. 381–395. Foody, G. M., 2002. Status of land cover classification accuracy assessment. Remote sensing of environment 80(1), pp. 185–201. Fraser, C. and Stamatopoulos, C., 2014. Automated target-free camera calibration. In: Proceedings of the ASPRS 2014 Annual Conference (Louisville, KY, USA), Vol. 2328. Girshick, R., Donahue, J., Darrell, T. and Malik, J., 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 580–587. Gislason, P. O., Benediktsson, J. A. and Sveinsson, J. R., 2006. Random forests for land cover classification. Pattern Recognition Letters 27(4), pp. 294–300. Habib, A. and Lin, Y.-J., 2016. Multi-class simultaneous adaptive segmentation and quality control of point cloud data. Remote Sensing 8(2), pp. 104. Huang, C., Davis, L. and Townshend, J., 2002. An assessment of support vector machines for land cover classification. Interna- tional Journal of remote sensing 23(4), pp. 725–749. Korumaz, M., Betti, M., Conti, A., Tucci, G., Bartoli, G., Bonora, V., Korumaz, A. G. and Fiorini, L., 2017. An integrated terres- trial laser scanner (TLS), deviation analysis (DA) and finite ele- ment (FE) approach for health assessment of historical structures. a minaret case study. Engineering Structures 153, pp. 224–238. LeCun, Y., Bengio, Y. and Hinton, G., 2015. Deep learning. na- ture 521(7553), pp. 436. Long, J., Shelhamer, E. and Darrell, T., 2015. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431– 3440. Makuti, S., Nex, F. and Yang, M., 2018. Multi-temporal classifi- cation and change detection using uav images. In: 2018 ISPRS TC II Mid-term Symposium Towards Photogrammetry 2020, 4–7 June 2018, Riva del Garda, Italy, International Society for Pho- togrammetry and Remote Sensing (ISPRS), pp. 651–658. Masiero, A., Chiabrando, F., Lingua, A., Marino, B., Fissore, F., Guarnieri, A. and Vettore, A., 2019. 3D modeling of Girifalco fortress. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLII-2/W9, pp. 479–478. Masiero, A., Guarnieri, A., Pirotti, F. and Vettore, A., 2015. Semi-automated detection of surface degradation on bridges based on a level set method. ISPRS - International Archives of Photogrammetry, Remote Sensing and Spatial Information Sci- ences 40(3), pp. 15–21. Noh, H., Hong, S. and Han, B., 2015. Learning deconvolution network for semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp. 1520–1528. Rabbani, T., Van Den Heuvel, F. and Vosselmann, G., 2006. Seg- mentation of point clouds using smoothness constraint. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 36(5), pp. 248–253. Suykens, J. A. and Vandewalle, J., 1999. Least squares sup- port vector machine classifiers. Neural processing letters 9(3), pp. 293–300. Tucci, G., Bonora, V., Fiorini, L. and Conti, A., 2016. The Flo- rence baptistery: 3-D survey as a knowledge tool for historical and structural investigations. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 41, pp. 977–984. Vosselman, G., Gorte, B. G., Sithole, G. and Rabbani, T., 2004. Recognising structure in laser scanner point clouds. ISPRS - In- ternational archives of photogrammetry, remote sensing and spa- tial information sciences 46(8), pp. 33–38. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W11, 2019 GEORES 2019 – 2nd International Conference of Geomatics and Restoration, 8–10 May 2019, Milan, Italy This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLII-2-W11-809-2019 | © Authors 2019. CC BY 4.0 License. 813 INTRODUCTION Segmentation and classification Predictors SVM classifier Detection of planar surfaces Semantic based model optimization Discussion