I'm interested in learning how the scanning community is tackling the issue of verification. We need to certify the accuracy of the data we collect, and spend a lot of time with on-site calibration and accuracy verification...
Running Dim Control surveys is great for verifying the control that's used, but how do you verify the scan data? what certifications are you signing off on? I'm an old RLS who prior to Aerospace, spent 20+yrs in heavy civil / industrial settings. I had to certify the accuracy of the data we collected in-house, and stuff that was given to us by subcontractors.
With conventional survey equipment, calibration and certification processes are pretty much set in stone.
What's the trend regarding laser scanners and their data sets?
Calibration / Accuracy verification
- JosephChumbley
- I have made 10-20 posts
- Posts: 11
- Joined: Tue May 13, 2008 9:21 pm
- 15
- Full Name: Joseph Chumbley
- Company Details: Lockheed Martin Space Systems Co
- Company Position Title: Mech Eng Senior
- Country: USA
- Garth
- I have made 20-30 posts
- Posts: 21
- Joined: Fri Jan 15, 2010 2:58 am
- 14
- Full Name: Garth Archibald
- Company Details: GNS Science
- Company Position Title: Engineering Geological Surveyor
- Country: New Zealand
- Location: Wellington, New Zealand
- Contact:
Re: Calibration / Accuracy verification
Hi Joseph,
I would guess you are not the only Scanner struggling with this issue.
My approach is that it is not the scan data but the derived surfaces that must have errors assigned as the pointcloud data is not the deliverable but it is the volumes, sections, models etc that are important.
I look forward to others responses on this crucial issue.
Regards Garth
I would guess you are not the only Scanner struggling with this issue.
My approach is that it is not the scan data but the derived surfaces that must have errors assigned as the pointcloud data is not the deliverable but it is the volumes, sections, models etc that are important.
I look forward to others responses on this crucial issue.
Regards Garth
-
- I have made 10-20 posts
- Posts: 13
- Joined: Mon Oct 26, 2009 3:25 pm
- 14
- Full Name: Marc De Bruyne
- Company Details: Grontmij Industry
- Company Position Title: Technology Coordinator
- Country: Belgium
- Linkedin Profile: Yes
- Location: Belgium
- Contact:
Re: Calibration / Accuracy verification
Joseph,
Glad that somebody is starting up this discussion.
Somebody told me once: Redundancy is King...
And this statement never got out of my mind.
Quality is something that concerns me for 10 years now...talking about point cloud data.
We are working in the Process&Power business, and I always work with Service providers, even for the post processing.
We always do our engineering with point clouds; modeling is not an option for me.
The reasons why are all obvious for me, but not a part of this topic.
So...Quality control.
For me a laser scanner is an equipment just like a TPS: it delivers point measurements in XYZ.
A bit more points than a TPS, but I don’t know why I shouldn’t expect something less for a Laser scanning point.
I have to know the Quality, tolerance, environment situation (eg temperature, vibrations, etc)
Some guy's give me the answer:
- My scanner is been calibrated every year....
- My scanning software will generate you a deviation report.
But what does this really give me as a end customer?
To be honest, for me ...a "pretty picture" of an environment..
I can assure you, the early years we had some drama's with wrong data, fuzzy data, what ever.
N this cases the software reported “excellent data”
And I had no grip on it; once data was delivered: is this data really OK, Y/N?
In fact I was manually rechecking almost every step my service provider had done..digging in to the point cloud to be sure that there were no errors in.
So for me there's no compromising any more on Quality:
- Never scanning without TPS assistance
- For every area Plant benchmarks are set by the Surveyor that is responsible for the plant (typically the in house contractor)
- All targets are measured by TPS (minimum 4 in sight per scan)
- Scans are consolidated based on the TPS coordinates, and leveled out afterwards as an extra action by the scan software
-A xyz report from the targets is extracted out of the final consolidated point cloud.
-This xyz report is merged with the xyz report out of the TPS and there is a third comprising table generated.
So once my service provider delivers a hard disk to me, the only thing I check before I give the scan to my project team is this Q-report…and I can put my feet’s on the desk. (as a matter of speaking )
Because I know 2 things:
-The scanner is reliable
-The scan data is reliable, and will guaranty a perfect new integrated 3D design in the existing plant.
Glad that somebody is starting up this discussion.
Somebody told me once: Redundancy is King...
And this statement never got out of my mind.
Quality is something that concerns me for 10 years now...talking about point cloud data.
We are working in the Process&Power business, and I always work with Service providers, even for the post processing.
We always do our engineering with point clouds; modeling is not an option for me.
The reasons why are all obvious for me, but not a part of this topic.
So...Quality control.
For me a laser scanner is an equipment just like a TPS: it delivers point measurements in XYZ.
A bit more points than a TPS, but I don’t know why I shouldn’t expect something less for a Laser scanning point.
I have to know the Quality, tolerance, environment situation (eg temperature, vibrations, etc)
Some guy's give me the answer:
- My scanner is been calibrated every year....
- My scanning software will generate you a deviation report.
But what does this really give me as a end customer?
To be honest, for me ...a "pretty picture" of an environment..
I can assure you, the early years we had some drama's with wrong data, fuzzy data, what ever.
N this cases the software reported “excellent data”
And I had no grip on it; once data was delivered: is this data really OK, Y/N?
In fact I was manually rechecking almost every step my service provider had done..digging in to the point cloud to be sure that there were no errors in.
So for me there's no compromising any more on Quality:
- Never scanning without TPS assistance
- For every area Plant benchmarks are set by the Surveyor that is responsible for the plant (typically the in house contractor)
- All targets are measured by TPS (minimum 4 in sight per scan)
- Scans are consolidated based on the TPS coordinates, and leveled out afterwards as an extra action by the scan software
-A xyz report from the targets is extracted out of the final consolidated point cloud.
-This xyz report is merged with the xyz report out of the TPS and there is a third comprising table generated.
So once my service provider delivers a hard disk to me, the only thing I check before I give the scan to my project team is this Q-report…and I can put my feet’s on the desk. (as a matter of speaking )
Because I know 2 things:
-The scanner is reliable
-The scan data is reliable, and will guaranty a perfect new integrated 3D design in the existing plant.
- Matt Young
- Honorary Member
- Posts: 3929
- Joined: Fri Nov 30, 2007 3:03 pm
- 16
- Full Name: Matt Young
- Company Details: Baker Hicks
- Company Position Title: CAD-BIM Lead
- Country: UK
- Linkedin Profile: No
- Has thanked: 45 times
- Been thanked: 40 times
Re: Calibration / Accuracy verification
Joseph,
I could not explain it better.
Everything that we scan within industrial environments goes through the same process. We end up with a fully reportable control file that verifies the registration of the cloud data. When we deliver our point clouds we can say with confidence that the data is accurate and we can state how accurate.
Although it is not always required, we use the same methodology for projects other than the industrial ones.
Making sure that data is accurately measured and recorded is key to the success of laser scanning.
Matt
I could not explain it better.
Everything that we scan within industrial environments goes through the same process. We end up with a fully reportable control file that verifies the registration of the cloud data. When we deliver our point clouds we can say with confidence that the data is accurate and we can state how accurate.
Although it is not always required, we use the same methodology for projects other than the industrial ones.
Making sure that data is accurately measured and recorded is key to the success of laser scanning.
Matt
If you don't see that there is nothing, then you are kidding yourself.
-
- I have made 10-20 posts
- Posts: 13
- Joined: Mon Oct 26, 2009 3:25 pm
- 14
- Full Name: Marc De Bruyne
- Company Details: Grontmij Industry
- Company Position Title: Technology Coordinator
- Country: Belgium
- Linkedin Profile: Yes
- Location: Belgium
- Contact:
Re: Calibration / Accuracy verification
Matt,
Of course I work in industrial environments, where any inconsistency or clash can cost millions.
So, I don't want to claim that I'm totally against cloud to cloud registration, or Target to target registration.
If this is "fit for use" ...why not.
If I would be in Film industry, I don’t think my customer would be so concerned in the exact dimensions of the BAT-mobile? In that business there are other kinds of complex HQ requirements than we need.
My point is that I can't take the risk, too much involved in industrial environments.
So yes, in the early days of the HDS4500 we blasted 60/70 scans per day...now the maximum is 25.
Not always commercial defendable of course....but everybody knows, as long as we all are using the same equipment and software tools, higher quality always has to be paid.
Of course I work in industrial environments, where any inconsistency or clash can cost millions.
So, I don't want to claim that I'm totally against cloud to cloud registration, or Target to target registration.
If this is "fit for use" ...why not.
If I would be in Film industry, I don’t think my customer would be so concerned in the exact dimensions of the BAT-mobile? In that business there are other kinds of complex HQ requirements than we need.
My point is that I can't take the risk, too much involved in industrial environments.
So yes, in the early days of the HDS4500 we blasted 60/70 scans per day...now the maximum is 25.
Not always commercial defendable of course....but everybody knows, as long as we all are using the same equipment and software tools, higher quality always has to be paid.
- Matt Young
- Honorary Member
- Posts: 3929
- Joined: Fri Nov 30, 2007 3:03 pm
- 16
- Full Name: Matt Young
- Company Details: Baker Hicks
- Company Position Title: CAD-BIM Lead
- Country: UK
- Linkedin Profile: No
- Has thanked: 45 times
- Been thanked: 40 times
Re: Calibration / Accuracy verification
Its nice to see that more and more people in the industry are on the same page and that laser scanning has reached its maturity.
This will be good for us all.
This will be good for us all.
If you don't see that there is nothing, then you are kidding yourself.
- JosephChumbley
- I have made 10-20 posts
- Posts: 11
- Joined: Tue May 13, 2008 9:21 pm
- 15
- Full Name: Joseph Chumbley
- Company Details: Lockheed Martin Space Systems Co
- Company Position Title: Mech Eng Senior
- Country: USA
Re: Calibration / Accuracy verification
I'm glad to see others interested in this topic... Verification is a sticky issue...
I agree that running Dimensional Control Surveys to the targets allows a good 'apples to apples' comparison to the target locations (checking how close the scanner measurements compare with the Control Survey work)... But that's only a few points out of the 10's of millions of points gathered by the scanner at each set-up... How do you certify the accuracy for the remainder of this data?
In example, if it's a critical area, we will gather discrete measurements using conventional equipment (reflectorless, 1 second total station) for 60 - 100 or more points spread randomly throughout the scan area. Then by overlaying measurements on the scan data, we can come up with a calculated spread on the accuracy. This way we have a defensible statistical method for calculating an accuracy statement that does not rely on just the targets, but the actual elements being measured...
Let's face it, different surface types (material, paint, etc.) give different measurement results, distance from the scanner effects accuracy, and then you get to factor in proper warm-up time on the laser, atmospheric conditions, etc... All these need to be properly accounted for, and are major factors to the resultant Statement of Accuracy.
Any input on this type of validation effort would be welcome... We are shooting for final registered scan data that is in the "overall" millimeter accuracy ranges... relative accuracy (areas in close proximity) is easy to get, but when you start comparing a measurment taken 2 feet from the scanner vs one thats 150 feet away, things get tougher.
US Federal standards require validation on conventional survey equipment at the start and end of a project. And if the project lasts longer, accuracy validation is required during the length of the project (typically a minimum of weekly verifications). Is anyone going to this effort with their scanners?
I agree that running Dimensional Control Surveys to the targets allows a good 'apples to apples' comparison to the target locations (checking how close the scanner measurements compare with the Control Survey work)... But that's only a few points out of the 10's of millions of points gathered by the scanner at each set-up... How do you certify the accuracy for the remainder of this data?
In example, if it's a critical area, we will gather discrete measurements using conventional equipment (reflectorless, 1 second total station) for 60 - 100 or more points spread randomly throughout the scan area. Then by overlaying measurements on the scan data, we can come up with a calculated spread on the accuracy. This way we have a defensible statistical method for calculating an accuracy statement that does not rely on just the targets, but the actual elements being measured...
Let's face it, different surface types (material, paint, etc.) give different measurement results, distance from the scanner effects accuracy, and then you get to factor in proper warm-up time on the laser, atmospheric conditions, etc... All these need to be properly accounted for, and are major factors to the resultant Statement of Accuracy.
Any input on this type of validation effort would be welcome... We are shooting for final registered scan data that is in the "overall" millimeter accuracy ranges... relative accuracy (areas in close proximity) is easy to get, but when you start comparing a measurment taken 2 feet from the scanner vs one thats 150 feet away, things get tougher.
US Federal standards require validation on conventional survey equipment at the start and end of a project. And if the project lasts longer, accuracy validation is required during the length of the project (typically a minimum of weekly verifications). Is anyone going to this effort with their scanners?
- Nick M
- I have made 50-60 posts
- Posts: 56
- Joined: Wed Jul 22, 2009 2:34 am
- 14
- Full Name: Nick M
- Company Details: Consulting Company
- Company Position Title: Senior Surveyor
- Country: Australia
Re: Calibration / Accuracy verification
To expand on the validation of scanner accuray I agree with all the posts above inregards to utilising conventional survey methods to provide control and sample data to confirm accuracies. Before commencing critcial projects and after transportation we use a target range (6+ targets setup in our carpark with even distribution through the full field of view of the scanner). This allows us for a quick verification of scanner performance on a repeatabile site.
We have also run our scanner over a baseline range that is setup for conventional survey instruments which is setup and monitored by our lands department to ensure all survey equipment is accurate and certified for use and performing as specified. The scanner passed this test and it has proved useful when speaking to clients that this test has been done as the accuracies are proven in a familair and approved method.
On the flip side, whilst as a sureyor I wish to ensure desired accuracies are acheived at all times this can run into extremes. For example with GPS you don't calculate the algorithims manually to ensure the accuracies, nor do you critique the pole location and level for conventional single point locations (you have to have some faith in the machine and operators).
I believe as long as you have a sound understanding of the equipment and it's inherinet accuracies (and the project requirements) and have enough redundancy in your scanning and target aquisitions to prove accuracy all will be fine.
The hard part is knowing where to draw that fine line!
We have also run our scanner over a baseline range that is setup for conventional survey instruments which is setup and monitored by our lands department to ensure all survey equipment is accurate and certified for use and performing as specified. The scanner passed this test and it has proved useful when speaking to clients that this test has been done as the accuracies are proven in a familair and approved method.
On the flip side, whilst as a sureyor I wish to ensure desired accuracies are acheived at all times this can run into extremes. For example with GPS you don't calculate the algorithims manually to ensure the accuracies, nor do you critique the pole location and level for conventional single point locations (you have to have some faith in the machine and operators).
I believe as long as you have a sound understanding of the equipment and it's inherinet accuracies (and the project requirements) and have enough redundancy in your scanning and target aquisitions to prove accuracy all will be fine.
The hard part is knowing where to draw that fine line!