Software for finding repeating items/patterns in point cloud

aurivus|REVIT, 3D Studio MAX, 3D Coat, Lightwave, ZBrush, VRMesh, TopoGun, Blender, Rhino, Trimble Sketchup Etc.
Scott
V.I.P Member
V.I.P Member
Posts: 1037
Joined: Tue Mar 29, 2011 7:39 pm
13
Full Name: Scott Page
Company Details: Scott Page Design- Architectural service
Company Position Title: Owner
Country: USA
Linkedin Profile: No
Location: Berkeley, CA USA
Has thanked: 205 times
Been thanked: 78 times
Contact:

Re: Software for finding repeating items/patterns in point c

Post by Scott »

neeravbm wrote:Scott, from what I understood, you want to measure dimensions within point cloud. For e.g. wall thickness, floor to ceiling height, etc. Am I correct? As I see it, the problem is that point cloud by itself doesn't know what a wall vs floor vs ceiling is. So there has to be a higher layer which segments and classifies point clouds and says that this group of points is a wall, this group of points is a floor, etc. Then you can measure distances between these higher order entities. As of now, I can't think of any way to directly measure distances between these higher order entities without actually creating these higher order entities. Please let me know if your reply was something totally different and I misunderstood it. Thanks!
Neerav,
Well stated -I appreciate your response. I guess we are stuck with vectorization and solid modeling for the moment (for dimensioning purposes). Point cloud visualization has improved immensely, so it would be nice to use the point cloud itself, as a stand alone deliverable. Visualization promotes and clarifies the project, dimensions get it built. Segmentation will likely be part of that process. Thanks!
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1236
Joined: Mon Jan 04, 2010 7:51 pm
14
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 62 times
Been thanked: 219 times
Contact:

Re: Software for finding repeating items/patterns in point c

Post by jedfrechette »

neeravbm wrote:If you had this software which could be instructed what to recognize, how often will you use it?
Our use case might be a little unique since we rely heavily on automatic model generation from point clouds rather than hand modeling, but classification regularly accounts for 50% or more of our post-processing time. With better tools I could easily see that time being cut in half.
neeravbm wrote:Can they provide separate photographs as output as well? If yes, are these 360 photographs or regular ones? With each photograph, do we also get the scanner position from where it was captured?
In some cases yes, but I think the challenge is that having separate photos (or even a colored point cloud) isn't guaranteed. Due to the extra time it takes on-site we rarely scan with color enabled, so most of our point clouds are x, y, z, intensity. Data collected with mobile platforms like GeoSLAM also commonly lack color information.

In addition to color, there are a bunch of other data channels that could potentially be used as classification features. For example, although we typically just work with the coordinates of each point, those coordinates actually represent a vector from the scanner to the measurement point and that vector should not pass through solid surfaces like walls. With static scans it is also pretty easy to estimate good surface normals for each point to provide even more information about its geometry.
neeravbm wrote:When you first start using the software, it will have no intelligence. You will have to start teaching it. So you'll select an object a few times in the point cloud and classify it as Object 1. This will be stored in your personal library now. You'll create many such objects from multiple point cloud files. Once the tool has been taught, any time you open a new point cloud file, it will find those objects for you automatically. Any machine learning system is going to have a few errors. So you'll need to validate the object that this tool finds. But in effect, you are creating a library of objects that you personally are interested in and that you personally come across in many of your different projects. Am I correct?
Exactly. I think the Training->Classification->Validation->Training feedback loop is critical and something that is currently being ignored at the user level. Yes the software should come with some pre-trained classifiers for common universal features, but I think the real value would start being realized once users can start training the system to detect features the developers never thought of. Going one step beyond that advanced users should be able to build their own classifiers. My ideal software would essentially be a classification platform that comes with batteries included. New users would be able to push a button and get a result using builtin pretrained classifiers. Intermediate users would be able to train the built in classifiers using their own data. Advanced users would be able to build their own classifiers from the ground up.

A couple examples of similar software for image classification are ERDAS Imagine for remote sensing data:

https://youtu.be/N6rjV2k-s18

and at a smaller scale ilastik:

http://ilastik.org/index.html
Jed
Scott
V.I.P Member
V.I.P Member
Posts: 1037
Joined: Tue Mar 29, 2011 7:39 pm
13
Full Name: Scott Page
Company Details: Scott Page Design- Architectural service
Company Position Title: Owner
Country: USA
Linkedin Profile: No
Location: Berkeley, CA USA
Has thanked: 205 times
Been thanked: 78 times
Contact:

Re: Software for finding repeating items/patterns in point c

Post by Scott »

Jed, Nice post and links -a good discussion overall... +1
neeravbm
V.I.P Member
V.I.P Member
Posts: 136
Joined: Thu Mar 16, 2017 3:29 pm
7
Full Name: Neerav Mehta
Company Details: Indoor Intelligence
Company Position Title: CTO
Country: USA
Linkedin Profile: No
Has thanked: 1 time
Been thanked: 6 times

Re: Software for finding repeating items/patterns in point c

Post by neeravbm »

Jed, I have a few quick questions for you. You mentioned that you rely on automatic model generation from point cloud and classification accounts for 50% of your post-processing time. What kind of models do you generate automatically from point cloud? Is it pipes using ClearEdge3D or is it more than pipes? Also what do you exactly expect the software to do in terms of classification? From what I understand, software will find locations of different objects in your point cloud. But how you want to classify that object, for e.g. call it a pump, or a valve, or a chair, is up to you, right?
Neerav Mehta
CTO, Indoor Intelligence
Creators of http://scantobim.xyz and http://rep3d.com
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1236
Joined: Mon Jan 04, 2010 7:51 pm
14
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 62 times
Been thanked: 219 times
Contact:

Re: Software for finding repeating items/patterns in point c

Post by jedfrechette »

neeravbm wrote:What kind of models do you generate automatically from point cloud? Is it pipes using ClearEdge3D or is it more than pipes?
When I said modeling I should have clarified that I was referring to lower level surface modeling, rather than object modeling. If we have different levels of data processing, e.g.:

Level 1: Measurements (e.g. point clouds)
Level 2: Surface (e.g. 3D polygonal mesh, terrain model)
Level 3: Features (e.g. valve, pump, chair, etc.)

Software like ClearEdge attempts to go from Level 1 to Level 3, which is a perfectly valid approach, but Level 3 features don't necessarily provide the same functionality as a Level 2 surface model. Most of our work requires Level 2 surfaces, which means that it is highly sensitive to the quality of the point cloud. If you have a person walking through your scan ClearEdge doesn't care because that person doesn't look like a pipe. If we build a surface model from that scan though, that person will be part of the model, which is usually not desireable.
neeravbm wrote:Also what do you exactly expect the software to do in terms of classification? From what I understand, software will find locations of different objects in your point cloud. But how you want to classify that object, for e.g. call it a pump, or a valve, or a chair, is up to you, right?
I would like software that I can use to efficiently assign points to arbitrary classes. I see creating the actual model, whether that is a Level 2 surface or a Level 3 feature as a separate process (although in many cases tightly coupled to the classification step). Exactly what objects are interesting can and does vary on a day-to-day and project-by-project basis. Also, as alluded to in the person example above, in many cases classification is as much about identifying points that represent things I want to exclude from the model as it is about identifying the points representing objects I want to model.

To give more concrete examples, on Monday, I might be scanning a city street with cars parked along the curb. I'm only interested in the street and the surrounding buildings so I need a classifier that can identify cars so that they can be excluded from my model. On Tuesday, I'm scanning a car to reconstruct it, so now I need a classifier tuned to identify the types of noise common in car scans, e.g. specular reflections off the headlights, mixed pixels and bogus distance measurements around the edges of the windshield, etc. On Wednesday I'm scanning a golf course and need to locate sprinkler heads marked with pink flags, so now I need a classifier that can identify pink flags. Those are all very specific scenarios and I could easily come up with many more. I think most people will have their own equally unique requirements, which is why I'm more interested in good tools rather than inflexible pre-built solutions that can't adapt to changing and diverse user needs.

To be generally useful I think any classification software would need to be a hybrid system, with both good manual tools that help humans work efficiently and perform the tasks machines can't do and automatic tools that can provide efficiencies wherever possible. The manual tools probably need to come first, after all if you want to train a classifier to identify sprinklers, first you're going to need to identify 1000 sprinklers by hand so you have labeled training data to build your classifier with.
Jed
neeravbm
V.I.P Member
V.I.P Member
Posts: 136
Joined: Thu Mar 16, 2017 3:29 pm
7
Full Name: Neerav Mehta
Company Details: Indoor Intelligence
Company Position Title: CTO
Country: USA
Linkedin Profile: No
Has thanked: 1 time
Been thanked: 6 times

Re: Software for finding repeating items/patterns in point c

Post by neeravbm »

Thanks, Jed, for the reply!

You mentioned that you want to efficiently assign points to arbitrary classes so that you can either convert points to a model or remove those points from your cloud. Do you spend more time finding the location of these classes in point cloud or actually assigning points to these classes? The reason I ask is finding the location of classes is much easier than assigning all the points to them. In other words, if the software gave you a 3D bounding box where most of the points belonged to your class, will it be useful to you?

It is interesting that you gave example of a car. Car is a category and is a higher-level construct than a specific model. To a machine, a Tesla S looks very different than a BMW. It's much easier to highlight a Tesla S in a point cloud and ask the software to find all other Tesla S's rather than highlight a Tesla S and ask the software to find all other cars. Latter requires lot more training than former. As first version of the software, is it still useful to you if the software finds exactly the same object that you highlighted rather than all the objects that fall under that category?
Neerav Mehta
CTO, Indoor Intelligence
Creators of http://scantobim.xyz and http://rep3d.com
Scott
V.I.P Member
V.I.P Member
Posts: 1037
Joined: Tue Mar 29, 2011 7:39 pm
13
Full Name: Scott Page
Company Details: Scott Page Design- Architectural service
Company Position Title: Owner
Country: USA
Linkedin Profile: No
Location: Berkeley, CA USA
Has thanked: 205 times
Been thanked: 78 times
Contact:

Re: Software for finding repeating items/patterns in point c

Post by Scott »

I don't want to hijack this thread -the following links do relate to the topic, though a separate thread for 'Image Processing' may also be needed if there is sufficient interest. I started looking for software used in medical imaging (MRI's CT scans), curious to see what was out there, and stumbled upon MATLAB and other resources. It was clear that the scanning world was, of course, a small branch of a much larger tree. 2D images and 3D constructs (point clouds & solid models) often overlap, so it's important to know the terminology in order to ask the right questions. To extract dimensions from point clouds and organize them in a sensible way, it's helpful to understand some of the processes needed to get you there: 'edge detection', 'segmentation', vectorization, and 2D image processing. The following videos (and chapter) do an excellent job explaining these terms:

IMAGE PROCESSING MADE EASY - MATLAB Video:
https://www.youtube.com/watch?v=1-jURfDzP1s
EDGE DETECTION WITH MATLAB -Video:
https://www.youtube.com/watch?v=Z4msmuJNpI4
CANNY EDGE DETECTOR - Computerphile Video:
https://www.youtube.com/watch?v=sRFM5IEqR2w
EDGE DETECTION AND SEGMENTATION- Book, Chapter 5:
https://pixhawk.ethz.ch/_media/eth_only ... tation.pdf

https://www.youtube.com/watch?v=oNsKjW3b77M ~4min.
Medical Image Analysis, DICOM Viewer - Point Cloud 3D Reconstruction (TUKE)
-Here Meshlab is used to take the CT images a step further, creating a point cloud and meshed model.
P.S.
I'd like to import my point clouds into such a grayscale viewer for the improved segmentation, edge detection, and animation tools. There is much overlap to be tapped out there.
neeravbm
V.I.P Member
V.I.P Member
Posts: 136
Joined: Thu Mar 16, 2017 3:29 pm
7
Full Name: Neerav Mehta
Company Details: Indoor Intelligence
Company Position Title: CTO
Country: USA
Linkedin Profile: No
Has thanked: 1 time
Been thanked: 6 times

Re: Software for finding repeating items/patterns in point c

Post by neeravbm »

Scott, thanks for the links! I'll have a look.

I didn't understand your last statement: "I'd like to import my point clouds into such a grayscale viewer for the improved segmentation, edge detection, and animation tools." Doesn't Revit provide the tools for segmentation and edge detection already? Also what is the use case you are thinking because of which you need these features?

You also mentioned about animation tools. Can you describe the use case for this as well?
Neerav Mehta
CTO, Indoor Intelligence
Creators of http://scantobim.xyz and http://rep3d.com
Scott
V.I.P Member
V.I.P Member
Posts: 1037
Joined: Tue Mar 29, 2011 7:39 pm
13
Full Name: Scott Page
Company Details: Scott Page Design- Architectural service
Company Position Title: Owner
Country: USA
Linkedin Profile: No
Location: Berkeley, CA USA
Has thanked: 205 times
Been thanked: 78 times
Contact:

Re: Software for finding repeating items/patterns in point c

Post by Scott »

neeravbm wrote:Scott, thanks for the links! I'll have a look.

I didn't understand your last statement: "I'd like to import my point clouds into such a grayscale viewer for the improved segmentation, edge detection, and animation tools." Doesn't Revit provide the tools for segmentation and edge detection already? Also what is the use case you are thinking because of which you need these features?

You also mentioned about animation tools. Can you describe the use case for this as well?
If all my clients only used Revit, life would be easier, though more limited.
For point cloud visualization alone, I think there is still much room for improvement. Segmentation is a way to isolate, group/ungroup objects in the point cloud. FARO Scene doesn't currently have these tools, so it's hard to say how useful they would be. Trimble RealWorks has a few segmentation tools, but doesn't allow the user to select the individual items or how to organize them. Medical imaging makes good use of segmentation (and animation) since grayscale images of soft tissue often benefits from post-colorization. Likewise, many point clouds lack color, so segmentation might help image enhancement.

I appreciate the way 2D images are organized in DICOM viewers: It takes a 3D object, creates a series of uniform slices, and delivers scaled orthophotos in a logical way. The user can see where the sections are cut (labeled horizontal/vertical section lines), and the order they were created. The slices can also be animated. This is all before any solid modeling is done. Having the ability to create/export a 2D layout from this process, would be very useful. The medical industry has already done this. Brains and buildings aren't all that different from a visualization point of view. Getting this organized information into standard A/E layout formats could be greatly improved. (PM me if you'd like)

P.S. Here is an interesting article from SPAR:
This Giant Library of 3D Images will Teach Machines How to Recognize Objects
http://www.spar3d.com/blogs/the-other-d ... e-objects/
Last edited by Scott on Sun May 07, 2017 11:57 pm, edited 1 time in total.
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1236
Joined: Mon Jan 04, 2010 7:51 pm
14
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 62 times
Been thanked: 219 times
Contact:

Re: Software for finding repeating items/patterns in point c

Post by jedfrechette »

neeravbm wrote:Do you spend more time finding the location of these classes in point cloud or actually assigning points to these classes? The reason I ask is finding the location of classes is much easier than assigning all the points to them. In other words, if the software gave you a 3D bounding box where most of the points belonged to your class, will it be useful to you.
Definitely assigning points to classes. I've been trying to think of a use case for just being able to locate class bounding boxes and so far haven't bee able to come up with an obvious one.

The ScanNet paper Scott linked to is interesting because I think it unintentionally highlights a couple of the core issues that exist right now. First, there was no existing software that the authors could use to manually label the points so they had to write their own. Even after that, users were taking 12 minutes to classify each scan. These are tiny scans that probably only took a couple minutes to collect so without even considering the additional machine time required, the scan time to classification time ratio is untenable for any type of production process. It would be ok if the automatic classifiers were good enough that only a small amount of manual labeling is required, but that's currently not the case.

Another recently released paper presents the Semantic3D benchmark: https://arxiv.org/abs/1704.03847

Even more so than the ScanNet paper, I think the Semantic3D paper captures the types of use cases that are most important to me. They succinctly summarize the goal of classification: "Given a set of points (here: dense scans from a static, terrestrial laser scanner), we want to infer one individual class label per point."

So far in this thread research on 3 very different approaches to related, but slightly different problems, has been referred to: Building Parser, ScanNet, and Semantic3D. Noteably, none of the algorithms tested in these papers perform very well, struggling to achieve even 80% accuracy. That's not even close to the level of accuracy you would need in a fully automatic production system. I don't say that as a criticism of the researchers, who are doing really important fundamental work; but simply to acknowledge that there is a long way to go before we have workable 1-click solutions.

In the mean-time, all of us practitioners have datasets, which are getting bigger and bigger, to deliver. To me a useful version 1.0 for a classification platform would have nothing more than extremely efficient manual segmentation tools. We're going to need to fix the classification for 20+ % of the data anyway, so we need to be able to do that efficiently. Any automatic tools, that even barely work would be a bonus at this point.

Since medical imaging got brought up, I would also point out that if you look at many of their classification tools the goal is not to take humans out of the loop it is to help humans work more efficiently. I think that's a good approach. After 4 million years of evolution we've gotten very good at identifying patterns.
Jed
Post Reply

Return to “Modeling Software”