key: cord-0058808-hec8z3l9 authors: Iakushkin, Oleg; Budlov, Egor; Bainova, Ekaterina; Sedova, Olga title: Algorithm for Creating Massive Amounts of Unique Three-Dimensional Models and Materials from Rocks date: 2020-08-24 journal: Computational Science and Its Applications - ICCSA 2020 DOI: 10.1007/978-3-030-58817-5_8 sha: 7893b2eacabd217e5740331425371336afb9a575 doc_id: 58808 cord_uid: hec8z3l9 The article describes the algorithm for creating three-dimensional models of stones from their polygonal mesh and a prototype photo. This method extracts PBR textures from the source image of the object and makes them seamless. Then, using the Blender software, a UV scan of the model is built. At the next stage, the coordinates of the sweep seams are extracted, and the quality of the textures is improved. Then, the resulting textures are superimposed on the object following the UV scan. The result is a three-dimensional model of the object with the textures applied to it. Also, a program was implemented that allows you to perform the above actions with a click of a button, which makes the process of obtaining a finished model as simple as possible. Among other things, the model was successfully exported to other programs working with 3D graphics. Currently, one of the main tasks of 3D modelling is the creation of threedimensional models of archaeological artefacts, which should take into account the shape, material and size of the object [20] . Also no less important is the problem of generating texture of an object from a photograph of its prototype. Since some objects can deteriorate even from exposure to sunlight and under the influence of climatic conditions, the creation of electronic copies will facilitate the task of preserving unique artefacts [14, 16] . Creating three-dimensional digital copies of objects opens up several possibilities, such as: -Detailed familiarization with art objects and various coloured artefacts without physical risk [15] . -Usage of the objects in research and historical reconstruction [4, 13] . -Creating physical copies of objects [1] . As a result of researching cultural and historical monuments, a set of information that describes them is formed. However, the data obtained during this process is usually not enough for precise reconstruction, and the work of reconstructing the model takes a lot of time and effort. Although modern technologies provide ways to create textures from an image [11] automatically, it is often necessary to make additional changes manually, which requires special skills. This work aims to facilitate the task of constructing volumetric models of archaeological finds and providing artists with tools to change these automatically generated textures manually. 3D modelling has recently been one of the most intriguing and challenging fields in computer vision and graphics research. Some of the developments in this area are: -modelling of stones from sketches. -reconstruction of walls from generated stones [9] . -recreation of surfaces from video [18, 19] . -building of continuous textures using pictures [10] . Although modern technologies provide the ability to create textures from an image [5, 21] automatically, and research in this area continues [2, 3] , 3D models obtained using existing solutions often either require many input data, or do not look realistic, or require a significant amount of time to be spent on mastering the software and manual execution of all actions, which are serious drawbacks. Also, often there is a need to make further changes manually, which requires additional special skills. Our task is to create an algorithm that allows getting a three-dimensional model of the desired object, while using a small amount of input data ( Fig. 1) , and to implement it as an automated system. Such a model in appearance should be close to the original. Despite the complexity of the task, the system should not require special knowledge from the user in the subject area (in particular, 3D modelling or archaeology). It should also satisfy the following requirements: -High realism of the 3D model of the desired object obtained as a result of the program. -High degree of system automation. The process of building a model should not require any additional actions from the user other than launching the program. -High speed of execution. The time spent on the program should not be large, which will allow simulating large volumes of objects in a short time. -Availability to make manual changes to the resulting model by the user to further increase realism. The input data for the algorithm are an image of the object (photo, drawing, etc.) and its colourless model. Performing the following sequence of steps will allow us to get a realistic 3D model of the required object: I. Extraction of PBR maps from the original image. II. Improving the quality of the extracted maps. III. Generation of a colourless model. IV. UV-unwrapping of the model. V. Creation of a scalable PBR material. VI. Removal of seams. VII. Export of the results. Physically Correct Rendering (PBR) is a set of rendering rules that takes into account the behaviour of light in reality to create a realistic image. The extraction is performed with Python programming language using 'bitmap2material' algorithm. The results are illustrated on Fig. 2. The resolution of the extracted PBR maps is quite low. We can improve it by increasing the size of the texture itself. It can be done by repeatedly placing the same texture a few times in a row and column (Fig. 3) . For the 3D reconstruction process, we need information about the size and shape of the object. We can get them from a colourless 3D model. This task was already solved in another paper [8] , so we are going to use the result in our work (Fig. 4) . Now we need to project our 3D model to a 2D plane, which is called "UVunwrapping". 'UV' here means two-dimensional coordinate system. It is the process of 'unfolding' a mesh so that we can create a 2D texture which fits the 3D object. This way, we can preserve the most important characteristics of the original object, such as its shape, size and connectivity [12] . In our work we used popular 3D-editor "Blender" [6, 7, 17] . Its advantage is that although this software is free and open, it provides a vast range of possibilities for working with various 2D and 3D objects. 1. Group faces of the object into several groups based on their normal vectors. 2. For each group, calculate its average normal vector and derive the orthogonal projection from projecting 3D faces to a 2D plane. Proportional weighting coefficients (based on sizes) are also used, so vectors of larger faces take priority. 3. Place each group on a rectangular (UV) area as compactly as possible. The resulting UV scan was made using Blender smart UV project and is shown on Fig. 5 . If necessary, it can also be scaled and rotated for better quality. The next stage is to create a single scalable material from the generated PBR maps. This process can be divided into several steps: The resulting PBR material can be quickly applied to any number of models, which creates a possibility to model many objects of the same type in a short time. Although the model is now wrapped in texture, we can not call it completed yet. That is because we still have some visible seams on the model, which are not present on the prototype object. If we want the model look realistic, these seams should be removed. The following algorithm was used to camouflage the seams (each generated image should be of the same size as the original texture): 1. Create a black and white binary image with a black background and white lines along the edges of UV islands. 2. Calculate the average colour of texture. 3. Generate an image consisting of pixels having a random colour, where the colour of each pixel should differ from the average colour of the texture by several tints. 4. Create a binary image with random black and white pixels. 5. Use logical AND with images from steps 1 and 4. 6. Use the image from step 5 as a mask to combine original texture and image from step 3 (Fig. 6 ). The final step is to export the result to various other 3D editors (Fig. 7) . Although Blender is quite practical and convenient to work with, it should be possible to export the generated texture. With this, artists will be able to use the software most suitable for their manual adjustments, and programmers will be able to work in the environment most familiar to them. In this paper, we used "Unreal Engine 4" -cross-platform development environment. This algorithm was tested on various pairs of colourless models and photographs of stones (Fig. 8-9 ). Upon finishing all the steps, we get a completed 3D model. Although the resulting model already looks quite authentic, it is still ready for further manual improvements if necessary. Also, additional functional was created that allows to artificially break a generated model of a stone (Fig. 10 ). The execution time of the algorithm is presented in Table 1 . While the "extraction of PBR maps" part of the algorithm takes a lot more time than combining, we need to do it only once. After that, we can get a lot of differently shaped but similarly coloured objects fast. This will allow us to create a lot of 3D models of the same type in a short time. In this work, an algorithm was created that allows us to make a 3D model of the object from its real image and colourless model. It was implemented using Python programming language and 3D editor "Blender" 1 . The program itself can be run either as an extension for this software or independently in Docker container. This process was automatized and did not require any user actions until its completion. Further manual changes for better quality are also possible. The execution time is not too long, which allows us to create large amounts of unique three-dimensional models in a short time. The main tasks of the future work are: further improving the quality of the resulting model, addition of texture styling tools in automatic mode and complete integration into a web-service format. Exploring the capabilities of additive manufacturing technologies in (re)-framing tourist souvenirs 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction Decomposition of the modeling task of some objects of archeological research for processing in a distributed computer system Augmenting phenomenology: using augmented reality to aid archaeological phenomenology in the landscape A point set generation network for 3D object reconstruction from a single image The Essential Blender: Guide to 3D Creation with the Open Source Suite Blender Blender Foundations: The Essential Guide to Learning Blender 2.5 Reconstruction of stone walls in form of polygonal meshes from archaeological studies 3D reconstruction of landscape models and archaeological objects based on photo and video materials Texture generation for archaeological reconstructions Automated creation of unique editable textures for three-dimensional models of archaeological artefacts Developing a process for automating UV mapping and polygon reduction Interactive 3D visualization for theoretical virtual observatories 3D modelling and mapping for virtual exploration of underwater archaeology assets Controlled remote usage of private shared resources via Docker and NoVNC Cultural heritage 3D modelling and visualisation within an augmented reality environment, based on geographic information technologies and mobile platforms 3D Video and Its Applications Position tracking in 3D space based on a data of a single camera The architecture of the robot-finder based on SLAM and neural network 3D recording and modelling in archaeology and cultural heritage 3D virtual reconstruction and visualization of complex architectures -the 3D-arch project Acknowledgments. This research was partially supported by the Russian Foundation for Basic Research grant (project no. 17-29-04288).