Digitizing Reality: Automated 3D Point Cloud Data Processing Using AI - 25/02/2020
Intelligent Point Clouds
How can today's advanced technology solve the challenges that many organizations face after obtaining vast 3D point cloud datasets, such as the management, storage, registration, fusion and extraction of useful and actionable information?
Instruments for digitizing the 3D real environment are becoming smaller, more lightweight, cheaper and more robust. Accordingly, they are finding increasingly widespread usage, not only on surveying tripods for the highest accuracy, but also on mobile platforms such as autonomous vehicles, drones, helicopters, aircraft, robotic vacuum cleaners, trains, mobile phones, satellites and Martian rovers. Lidar uses laser scanning, while photogrammetry records images from one or more cameras which may be moving. Each laser scan records tens of millions of data point positions and colours in a point cloud, and hundreds of such point clouds may be combined. This article discusses the challenges that many companies and organizations face after obtaining vast 3D point cloud datasets, such as the management, storage, registration, fusion and extraction of useful and actionable information.
The first challenges that users face in performing 3D point cloud data processing include:
- Data storage: The amount of data recorded grows exponentially with time, creating large data repositories.
- Processing: The computing power required increases as new algorithms with useful functionality are released and with the volume of data.
- Sharing: Multiple stakeholders spread geographically around the world on mobile platforms all need to view the most up-to-date data at the same time.
Previously, a software application ran on a dedicated server in a data centre but, if the computer hardware broke down, the user either had to find a backup (which had to be standing by and ready) or suffer an interruption in service. Many companies guarantee a 24/7 level of service and so cannot tolerate this. However, cloud computing now gives users access, over a network, to applications running on a set of shared or pooled servers in a globally communicating network of data centres, giving speed and productivity improvements and resulting in increased competitiveness.
Big Data Analytics
Users face the difficult challenge of reducing the vast amounts of 3D point cloud data to generate useful and actionable information. Current methods for creating digital twin BIM models of buildings require users to inspect vast 3D point clouds to manually recognize and mark the outline positions of surfaces, straight edges, walls, floors, ceilings, pipes and other objects, which is time-consuming and susceptible to error. Some semi-automatic methods on laptops require users to recognize and mark some of these and the programme finds the rest. Again, such objects can be mislabelled. Fully automatic methods are becoming available on laptops but do not find all the useful information, so users must add to and correct what is found. The automatic method sometimes makes so many mistakes that it is quicker for the user to find and mark the structures manually.
What is considered 'useful information' may vary between applications. For example, in autonomous vehicles, useful information is an accurate 3D terrain model which can be used for safe navigation. In electricity pylon scanning, it is whether the pylon has its safety warning sign clearly in place and whether nearby vegetation is gradually encroaching on the power lines. In railway scanning, it is whether there has been any slippage or sag, plus an estimate of when gradually encroaching vegetation will become a hazard. Electricity supply companies and Network Rail, for example, are under UK government obligation to regularly inspect their assets and to perform preventative maintenance to ensure continuity of supply and travel.
Geometrical Object Recognition
Correvate has developed a suite of machine learning geometric image processing methods for fully automated basic object recognition – walls, floors (Figures 2 and 3), edges (Figure 4) and pipes (Figure 5).
See below for the rest of the article.
Artificial neural networks are extremely simplified models of living brains, and are trained to learn like people rather than being programmed by a master programmer. The learned knowledge or skills are stored in a distributed manner in the strengths or weights of the neuron interconnections. Some artificial neural networks learn on their own, while others require a teacher or instructor to tell them when they are right or wrong. Gradually, they get better and better at performing a task during the iterative learning cycles, which usually take a long time and require thousands of examples of the training data. Artificial neural networks are particularly good at recognition, classification and optimization tasks. However, their performance depends crucially on how they are trained and the types and the amount of training data. Many types of neural network have been developed and, most recently, Convolutional Neural Networks (CNN) used to perform deep learning have become very popular and are achieving very good results. In the case of object recognition, if the neural networks are only trained with examples of objects one wants to find, then all input data will be classified as one of those objects, even if it is not. So, the performance of the neural network is only as good as its training and the data that was used to train it. Neural networks are not as new as you might imagine given their current popularity in the media. Over 30 years ago, Selviah (1989) proved that the weighted interconnection layer of neural networks performs the same operation as a collection of correlators, operating in parallel, matching images from a database with input data and then the non-linear part of the neurons deciding which image matches the input most closely. The clever part is the way in which the training automatically works out which images to store in the database in the first place.
The conference room image (Figure 6) shows the impressive recognition results after training a new type of CNN with data from the Stanford Large-Scale 3D Indoor Spaces Dataset (S3DIS) using 70,000 3D objects of 13 types of structural objects (ceiling, floor, wall, beam, column, window, door) and movable objects (table, chair, sofa, bookcase, board) and clutter in 11 types of room. Each category of object is marked in a different colour, for example ‘chairs’ are marked in yellow, ‘boards’ are marked in orange, ‘beams’ are marked in red, ‘door’ is marked in green, ‘walls’ are marked in dark green, ‘floor’ is marked in blue, and so on. The accuracy of classification of objects is around 93.5%, which is comparable to human accuracy. The objects to be recognized can be chosen for each application simply by changing the training database.
Artificial Intelligence in the Cloud
As the AEC sector embraces digital technology, the amount of data produced grows exponentially, creating large data repositories. To generate useful and actionable information from this ‘big data’ requires leveraging smart analytical tools such as Artificial intelligence (AI). These are becoming more accessible, especially when hosted from the cloud. Both the cloud computing infrastructure and AI supply the tools to leverage and enable digital technology by providing convenient methods of working at scale, thus lowering the barriers to entry for users to these new ways of working. AI neural networks and deep learning require vast databases of thousands of examples for training, which can be conveniently stored in elastic expandable cloud storage on demand. AI software requires highly parallel processing on many parallel processors to carry out the training in a reasonable time, again easily available in cloud computing infrastructures.
The intelligent combination and use of available techniques such as laser scanning, automatic alignment, cloud computing and AI can not only speed up the analysis of vast datasets, but can also improve accuracy and release human activity to ensure that a product is correct and useful.
BENEFITS OF USING THE CLOUD
Auto-application updates: applications are updated automatically, so the user always has access to the most up-to-date optimized software and bug fixes.
Responsivity: dedicated development support teams continuously monitor user experience to optimize and, if necessary, rewrite code.
Scalability, flexibility and agility: Scalable elastic cloud environments on pools of servers, storage and networking resources scale up and down according to the number of users and the volume of their usage. They automatically scale up and down as users’ needs change.
Capital expenditure free: users have access to the highest power computers. Hardware is used efficiently as users do not need to purchase, manage and maintain large amounts of computer and storage hardware, resulting in lower hardware, power, cooling and IT management costs. Users only pay for what they use as the cloud resources automatically scale, so it is easier for small businesses to manage their business at any time of the day, from anywhere.
High speed: multiple computers run in parallel so many different parts of the same point cloud can be processed at the same time and many different users have no effect on speed or quality.
Security: the data is stored and communicated securely with a level of encryption chosen by the user. If security is a paramount concern, the software can run on a private cloud without internet connections in-house. Clouds can be configured to make use of certain data centres, such as within one country if international security is a concern.
Availability: if one server is busy or not available then another server takes its place to provide full availability.
Disaster recovery: data is stored in multiple locations at the same time so if storage hardware in one data centre breaks down, the calculation proceeds with little interruption as the data is backed up elsewhere. Data archiving facilities are automatically provided.
Latency: if latency is important, the cloud can be configured so that local clouds provide low latency to the user.
Increased collaboration: many users, located globally, and mobile users, can store, process, share and view datasets at the same time without any loss of speed or responsivity.
Reliability: the application software can make use of resources on cloud computing infrastructure provided by different vendors in different global regions.
Forward compatible: an open cloud architecture is forward compatible to match higher power computing resources as they are rolled out.
Sharing: all point cloud datasets are secure in one place and accessible at any time from any place by any employee, improving collaboration.
Last updated: 05/10/2020
The author thanks Charles Thomson+, Liang Cheng+, Jiayu Pan* and Hadi Baghsiahi+ (*UCL, +Correvate). For additional information, please contact Charles@vercator.co.uk.