Modifying E57 scalars while keeping data structured

Please feel free to ask other members for help with certain projects
Post Reply
jukka.alander
I have made <0 posts
I have made <0 posts
Posts: 5
Joined: Mon Sep 23, 2019 9:06 am
Full Name: Jukka Alander
Company Details: Tietoa Finland Oy
Company Position Title: Project manager
Country: Finland
Linkedin Profile: Yes
Has thanked: 6 times

Modifying E57 scalars while keeping data structured

Post by jukka.alander »

Hello,

I'm looking for a software where it would be possible to modify (structured) E57 file contents like scalar field values without collapsing/modifying actual data structure of the file.

Currently we need to modify intensity values with a simple arithmetic operation. (Original problem is that intensity values from RTC360 are way too dark to be useable). This arithmetic operation is easily done with CloudCompare or FME, but with both software data structure gets modified in a way that ReCap wont read bubble views any longer - which we need.

Why this has to be done for E57's is that we get scans already registered and dont have access to Leica software like Cyclone or Register360.

Hiring a developer to create a custom tool with http://www.libe57.org/ is also possible, but we'd rather use existing software to do this.

Br,
Jukka
jedfrechette
V.I.P Member
V.I.P Member
Posts: 988
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 15 times
Been thanked: 61 times
Contact:

Re: Modifying E57 scalars while keeping data structured

Post by jedfrechette »

I'm not aware of any software that can do that out of the box, but it would be a very useful thing to have. The dark intensity values in RTC360 e57s are super annoying, but it would be useful for lot's of other tasks as well.

If you go the development route, what about reaching out to the main devs working on PDAL? I'm pretty sure they are happy to do sponsored development and you would be building on top of a library that is actively maintained, unlike libe57 which hasn't been touched in years.

I don't think the additions needed to be able to roundtrip structured e57 data through PDAL would be excessive. However, if a significant amount of work is needed that is something I would be interested in contributing financially to as well as there would be lots of ancillary benefits.
Jed
User avatar
Bohdan gutnyk
I have made 20-30 posts
I have made 20-30 posts
Posts: 25
Joined: Sat Feb 23, 2019 10:17 am
Full Name: Bohdan Gutnyk
Company Details: 3dm company in kiev
Company Position Title: 3dm
Country: Ukraine
Skype Name: bohdangutnyk
Linkedin Profile: No
Has thanked: 3 times
Been thanked: 19 times

Re: Modifying E57 scalars while keeping data structured

Post by Bohdan gutnyk »

I think you can create Recap project from structured data to get the bubble view inside the file. Then select all the points inside the recap and delete them, you will have panoramas but without the point cloud inside. Then modify e57 in cloudcompaire and import it into the project with bubble view in recap. You will get your point cloud with adjusted intensity, as well as a bubble view from a structured scan. It's a little longer, but it works.
jukka.alander
I have made <0 posts
I have made <0 posts
Posts: 5
Joined: Mon Sep 23, 2019 9:06 am
Full Name: Jukka Alander
Company Details: Tietoa Finland Oy
Company Position Title: Project manager
Country: Finland
Linkedin Profile: Yes
Has thanked: 6 times

Re: Modifying E57 scalars while keeping data structured

Post by jukka.alander »

Bohdan gutnyk wrote: Mon Nov 16, 2020 8:11 pm I think you can create Recap project from structured data to get the bubble view inside the file. Then select all the points inside the recap and delete them, you will have panoramas but without the point cloud inside. Then modify e57 in cloudcompaire and import it into the project with bubble view in recap. You will get your point cloud with adjusted intensity, as well as a bubble view from a structured scan. It's a little longer, but it works.
Thanks for the idea!

Actually I need to clarify; our problem affects the bubble views in Recap, because there are only two coloring options in bubble view; RGB and intensity, which are no use when scan is taken without colors and intensity is very dark. To my knowledge, ReCap builds bubble views from actual E57 points when importing to Recap (there are no embedded image files in our E57's). In ReCap's 3D View there are more coloring options, but clients want to use bubble views for measuring and viewing. This is why editing structured data is what I'm looking for.
User avatar
smacl
Global Moderator
Global Moderator
Posts: 493
Joined: Tue Jan 25, 2011 5:12 pm
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 141 times
Been thanked: 152 times
Contact:

Re: Modifying E57 scalars while keeping data structured

Post by smacl »

jedfrechette wrote: Mon Nov 16, 2020 6:13 pm I'm not aware of any software that can do that out of the box, but it would be a very useful thing to have. The dark intensity values in RTC360 e57s are super annoying, but it would be useful for lot's of other tasks as well.

If you go the development route, what about reaching out to the main devs working on PDAL? I'm pretty sure they are happy to do sponsored development and you would be building on top of a library that is actively maintained, unlike libe57 which hasn't been touched in years.

I don't think the additions needed to be able to roundtrip structured e57 data through PDAL would be excessive. However, if a significant amount of work is needed that is something I would be interested in contributing financially to as well as there would be lots of ancillary benefits.
Not sure what underlying E57 libraries are used by PDAL, but if you want to avoid libe57 Andy Moloney's implementation looks like the way to go. This library is kept up to date and I'm guessing this is what PDAL is based on if not libe57. For the type of in-place editing of intensities that is being looked for here, I'd say a direct C++ implementation could be a better option than PDAL.
jedfrechette
V.I.P Member
V.I.P Member
Posts: 988
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 15 times
Been thanked: 61 times
Contact:

Re: Modifying E57 scalars while keeping data structured

Post by jedfrechette »

PDAL uses their own e57 implementation that was contributed by Helix RE.

You could certainly write a standalone utility based on the asmaloney libe57 fork. I suggested PDAL because of the network effect. Extend the PDAL e57 IO code to support structured scans and you also get access to the rest of it's functionality for free.

If you went the PDAL route transforming the intensity values would probably be done with either a Python or Julia function (Matlab is also an option). Since the task is embarrassingly parallel it should be pretty straightforward to implement the transform in either language in a highly multithreaded way, or potentially even on the GPU, so performance should be pretty good.
Jed
jukka.alander
I have made <0 posts
I have made <0 posts
Posts: 5
Joined: Mon Sep 23, 2019 9:06 am
Full Name: Jukka Alander
Company Details: Tietoa Finland Oy
Company Position Title: Project manager
Country: Finland
Linkedin Profile: Yes
Has thanked: 6 times

Re: Modifying E57 scalars while keeping data structured

Post by jukka.alander »

Thank you for the input.

Just found out that Autodesk has published "instructions" to fix this issue by using PTX format and third party software like Faro Scene LT to edit intensities; https://knowledge.autodesk.com/support/ ... C-360.html

We can swap to PTX workflow, editing PTX should be much easier than E57. This is a next thing to try.
jedfrechette
V.I.P Member
V.I.P Member
Posts: 988
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 15 times
Been thanked: 61 times
Contact:

Re: Modifying E57 scalars while keeping data structured

Post by jedfrechette »

Well I guess that's one way to do it. I can't believe that Autodesk is seriously presenting this as a solution for the failure of their software to correctly read the and do a contrast stretch on the intensity values.
Jed
VXGrid
V.I.P Member
V.I.P Member
Posts: 179
Joined: Fri Feb 24, 2017 10:47 am
Full Name: Martin Graner
Company Details: PointCab GmbH
Company Position Title: Research and Development
Country: Germany
Linkedin Profile: No
Has thanked: 21 times
Been thanked: 27 times
Contact:

Re: Modifying E57 scalars while keeping data structured

Post by VXGrid »

jedfrechette wrote: Wed Nov 25, 2020 7:52 pm Well I guess that's one way to do it. I can't believe that Autodesk is seriously presenting this as a solution for the failure of their software to correctly read the and do a contrast stretch on the intensity values.
The question on the other hand is, why is Leica / the RTC providing such data?

There are two format types from which the software vendor could read the point cloud data
  1. A native format with the SDK from the hardware manufacturer.
  2. An exchange format like e57.
For a.) the software vendor knows from which hardware manufacturer the scan data came, and if he needs to apply calculations on the reflectiviy/colour values. (Should be in the documentation).
For b.) there is no such information (except perhaps some name fields from which the software vendor could read, but not everybody is using them the same way).

So in my opinion the histogramm adjustment should be done by the hardware manufacturer, if you are providing an exchange format.
jedfrechette
V.I.P Member
V.I.P Member
Posts: 988
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 15 times
Been thanked: 61 times
Contact:

Re: Modifying E57 scalars while keeping data structured

Post by jedfrechette »

If we're just talking about a contrast adjustment to improve visualization I don't see why the software needs to know what scanner captured the data. Photoshop doesn't need to know what camera was used when it applies a contrast stretch to a photo, and as indicated by Autodesk, Faro Scene doesn't need to know the scanner used to adjust the contrast of a scan.

As a user, I'd much rather have measured attributes like intensity stored as relatively "raw" values, regardless of format, and not have any post-process effects baked in so we can do quantitative analysis of them if we want. Again I'd refer to the analogy of digital photography. I want to shoot in raw, convert the data to an open lossless format (images=exr, scans=e57) and have the software I'm using to view the data handle the display transform and expose any user adjustments that are necessary.

I firmly agree though that vendors should definitely be writing more metadata to e57 files though to make the job of downstream users easier.
Jed
Post Reply

Return to “Request Help With Projects”