What Point Cloud format do you use and why?

Please post all your polls you want others opinion on.
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1079
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 34 times
Been thanked: 121 times
Contact:

Re: What Point Cloud format do you use and why?

Post by jedfrechette »

Shane, you may also want to check out TileDB:


youtu.be/lv-GExUVsHc
https://youtu.be/lv-GExUVsHc

It ticks most of your boxes for a high-performance open format, has been around for a few years, and is usable today for lidar and image data. I find their core argument pretty compelling:

https://tiledb.com/blog/debunking-purpo ... 2021-08-04

That blog post makes it in a slightly different context than this thread, but I agree with their core assertion that: We shouldn't be building specialized data stores because ultimately the data used in a lot of fields is pretty much the same. TileDB certainly isn't the first attempt at a universal data store (HDF comes to mind), but it's interesting that one of their initial use cases is point clouds.

I don't think the problems referred to in this thread exist because we lack good formats to store data in. The problems exist because our software isn't written to take advantage of them. Fundamentally I think that is entirely a user community problem not a technical one.

Las has become the defacto standard for aerial and mobile data because that's what the customers (mostly governments) required. Now that it supports arbitrary attributes I can easily see it becoming the standard for static scans as well if those same large customers start purchasing more of them and writing las requirements in to the specs. If that happens software vendors will add support.

The original development of e57 was largely driven by a couple people at one company, but when that company got bought out nobody else in the community kept moving the project forward and it stalled. Our user community just doesn't seem to value R&D enough, or have enough resources to support these types of projects so I would be surprised to see us come up with some new widely adopted uber-format that solves the issues raised here. If we do end up adopting something better I'm almost certain it will come from somewhere else.

I think that's OK. Ultimately our data isn't that complicated. It's some points with arbitrary attributes, images, and some metadata. There's lot's of good ways to store all of those things.
Jed
stavrosp
I have made <0 posts
I have made <0 posts
Posts: 1
Joined: Tue Aug 10, 2021 7:20 pm
Full Name: Stavros Papadopoulos
Company Details: TileDB
Company Position Title: CEO
Country: USA
Linkedin Profile: Yes

Re: What Point Cloud format do you use and why?

Post by stavrosp »

You may also want to check out this recent lightning talk at ESIP: https://twitter.com/tiledb/status/14251 ... 99488?s=20

Despite the fact that we built a powerful data format at TileDB, we still believe that we should depart from building a million different domain-specific formats. Instead, we should focus our energy on building engines on top of universal and intuitive data structures (like dense and sparse multi-dimensional arrays) and defining clean APIs that are easy for everyone to adopt. I am wondering what other folks in the community think about that.
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1079
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 34 times
Been thanked: 121 times
Contact:

Re: What Point Cloud format do you use and why?

Post by jedfrechette »

mschuetz wrote: Thu Aug 05, 2021 9:07 pmFor persistent storage with the ability to stream and real-time render, this early-stage project here might be interesting: https://github.com/copcio/copcio.github.io
As announced on the PDAL mailing list last Monday a Draft Spec for the COPC format has been released and the developers are actively seeking feedback. Any developers out there with intererest in this, now is your chance to give your input and contribute.
Jed
ananda
I have made 10-20 posts
I have made 10-20 posts
Posts: 18
Joined: Fri Apr 21, 2017 1:28 pm
Full Name: Ananda Fowler
Company Details: GEOHAWK GmbH
Company Position Title: Owner
Country: Austria
Linkedin Profile: Yes
Has thanked: 1 time
Been thanked: 7 times

Re: What Point Cloud format do you use and why?

Post by ananda »

I'll 3rd the COPC vote. It's the best of LAS/LAZ and EPT in one format.
e57 is the right direction for the data TLS generates, but as Jed mentioned, it wasn't ever well supported by the community.
I did my best by being on the committee for a while, but must admit that there were definitely points of inertia in the committee itself.
Also, at the outset, there was a lot of tug-of-war between vendors, which always has been a real problem for users...
It wasn't in the interest of certain enterprises to enable data to 'flee' their platform ;)
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1079
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 34 times
Been thanked: 121 times
Contact:

Re: What Point Cloud format do you use and why?

Post by jedfrechette »

I find it somewhat amusing that the rational for COPC compares it to a Tiff variant.

It seems like Las is becoming the point cloud equivalent of Tiff. It's probably not the best format for every use case, but it's kind of OK for most things and it is extensible enough that, one way or another, you can cram most of the things you might want to store in to it.
Jed
hobu
I have made <0 posts
I have made <0 posts
Posts: 1
Joined: Fri Sep 03, 2021 1:10 pm
Full Name: Howard Butler
Company Details: Hobu Inc
Company Position Title: President
Country: USA
Linkedin Profile: No
Been thanked: 1 time

Re: What Point Cloud format do you use and why?

Post by hobu »

Please participate in its creation with any comments on scope, use, or features at https://github.com/copcio/copcio.github.io.

The story of TIFF's extension with Cloud Optimized GeoTIFF is the playbook we are running with COPC. Take a widely supported archive format, add a minimal amount of organization to it in a way that doesn't disrupt the format's use as an archive container, and provide lots of open source tooling to boost uptake.

TIFFs stink for lots of reasons, but one thing they have going for them is just about everyone's software supports at least reading them. LAZ is much the same. Its downsides are well known, but much of the lidar industry has at least read support for it in their software. Proliferating a new format in the industry is extremely difficult, even if you are giving the software away as we did with LAZ.

We're not building the ultimate point cloud format. We're extending the well known LAZ with a little bit of metadata and utilizing a previously unused feature of the format (variable chunking) to provide an organized archive format that can be treated as unorganized LAZ by unsophisticated clients. This will mean it can be used by data providers and sophisticated clients immediately without requiring every downstream user to update their software.
User avatar
smacl
Global Moderator
Global Moderator
Posts: 738
Joined: Tue Jan 25, 2011 5:12 pm
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 288 times
Been thanked: 278 times
Contact:

Re: What Point Cloud format do you use and why?

Post by smacl »

hobu wrote: Fri Sep 03, 2021 1:37 pm Please participate in its creation with any comments on scope, use, or features at https://github.com/copcio/copcio.github.io.

The story of TIFF's extension with Cloud Optimized GeoTIFF is the playbook we are running with COPC. Take a widely supported archive format, add a minimal amount of organization to it in a way that doesn't disrupt the format's use as an archive container, and provide lots of open source tooling to boost uptake.

TIFFs stink for lots of reasons, but one thing they have going for them is just about everyone's software supports at least reading them. LAZ is much the same. Its downsides are well known, but much of the lidar industry has at least read support for it in their software. Proliferating a new format in the industry is extremely difficult, even if you are giving the software away as we did with LAZ.

We're not building the ultimate point cloud format. We're extending the well known LAZ with a little bit of metadata and utilizing a previously unused feature of the format (variable chunking) to provide an organized archive format that can be treated as unorganized LAZ by unsophisticated clients. This will mean it can be used by data providers and sophisticated clients immediately without requiring every downstream user to update their software.
Hi Howard, great to see you here on the Laser Scanning Forum and I look forward to seeing how COPC progresses.
RJGEOMATICS
Forum Supporter
Forum Supporter
Posts: 116
Joined: Tue Apr 23, 2019 7:20 pm
Full Name: Bart Man
Company Details: Laser Scanning
Company Position Title: Passionate about making amazing PCs
Country: Canada
Linkedin Profile: No
Has thanked: 35 times
Been thanked: 51 times

Re: What Point Cloud format do you use and why?

Post by RJGEOMATICS »

Must say. If anything, this string along with many other strings on the same topic, highlights the inconsistency of formats and expectations. Do I see suppliers and manufacturers sharing all their trade secrets and making everything open. "No"

Leica has created an interesting beast in the "LGS" format, but it is Pay to Play. The format is amazing in its compression ratios, and also that full HDR support exists within it, plus support for IFC and OBJ and many other interesting features.

Expecting all to play along with a free open source is a long way away, and would need be globally governed and enforced by law, not just any organization. We already have, I don't know how many different organizations trying to create standards from USIBD to IFMA to PlanBuild, to CanBIM, to OpenBim, onto Forensic associations and every flavor out there. Almost Daily, I see some organization trying to lead the charge to standardization, on everything from formats to methods, to specification sheets and how they report, and even some thinking that they create a one workflow is suitable for all hardware and software manufacturers...

The problem with strapping a generic open format to all, is that it also limits their freedom to create and add new.
Never mind the never ending story of evolution and that even with an open source format, this would require every software supplier to constantly create updates supporting every new addition of hardware advancement, that requires an extra line of code.

We don't see Apple, Google and Windows creating uniformity, we don't Autodesk sharing all its proprietary Object Based data principals, that will only work inside of Autodesk, and no other CAD or modeling software. Thus is realistic to think that all these, 3D Photo and Scanning platforms are going to provide a nice comprehensive open source solution? Realistically "No" We'll get partial data, but we'll never get complete data and full functionality when we move outside of the Supported and Authorized Ecosystem.

Hence why I have lowered my expectation, and resorted to ideally looking at basing PointCLouds on the *.pts agnostic approach, and separating the photogrammetry game from the PointCloud game. Also when we look beyond TLS (Tripod Laser Scanning) and move into, SLAM based, Ariborne, Mobile and other forms of PointClouds we loose even more support for Structured formats. Yes *.pts is very Primitive and simple. But it is supported by everyone and everything. And you can mix basically every form of PointCloud into a single cloud, even mixing TLS with Airborne with Photo Derived PointClouds. And also when it comes to modeling and meshing PointClouds, etc. Basically every platform out there can use a simple Basic PTS file.

Again, Leica has created a unique Pay to Play system with LGS. And we already see many software platforms paying the price to simply support the LGS files. Perhaps the Pay to Play is the real approach for those that want more complete functionality. And at that, yes, it is possible to create LGS files from Faro, and Z&F data etc. Something we commonly do for people, as many out there love the simplicity of the JetStream Viewer platform and that it is Standalone.

Call me a pessimist... But I have been in this game since 2001 and been in Geomatics since 1993... If anything the industry and standards have become more fractured over the past 30 years. Also manufacturers realize the benefits of selling code and software, over hardware. Selling software is a far greeter revenue stream for a company, and keeping it proprietary, insures return revenue, year after year. So to think that Autodesk, Bentley, Hexagon or Trimble or the others, are going to loosen the strings, is a dream. If anything, we'll see more and more Proprietary formats and specialized functions and sharing platforms be developed. It is Capitalism folks. That is the cold hard, brutal truth. And every supplier is in the race to lock everyone into their ecosystem.
VXGrid
V.I.P Member
V.I.P Member
Posts: 269
Joined: Fri Feb 24, 2017 10:47 am
Full Name: Martin Graner
Company Details: PointCab GmbH
Company Position Title: Research and Development
Country: Germany
Linkedin Profile: No
Has thanked: 77 times
Been thanked: 73 times
Contact:

Re: What Point Cloud format do you use and why?

Post by VXGrid »

RJGEOMATICS wrote: Fri Sep 03, 2021 7:38 pm Must say. If anything, this string along with many other strings on the same topic, highlights the inconsistency of formats and expectations. Do I see suppliers and manufacturers sharing all their trade secrets and making everything open. "No"

Leica has created an interesting beast in the "LGS" format, but it is Pay to Play. The format is amazing in its compression ratios, and also that full HDR support exists within it, plus support for IFC and OBJ and many other interesting features.

Expecting all to play along with a free open source is a long way away, and would need been globally governed and enforced by law, not just any organization. We already have I don't know how many different organization trying to create standards from USIBD to IFMA to PlanBuild, to CanBIM, to OpenBim, onto Forensic associations and every flavor out there.

The problem with strapping a generic open format to all, is that it also limits their freedom to create and add new.
Never mind the never ending story of evolution and that even with an open source format, this would require every software supplier to constantly create updates supporting every new addition of hardware advancement, that requires an extra line of code.

We don't see Apple, Google and Windows creating uniformity, we don't Autodesk sharing all its proprietary Object Based data principals, that will only work inside of Autodesk, and no other CAD or modeling software. Thus is realistic to think that all these 3D Photo and Scanning platforms are going to provide a nice comprehensive open source solution Realistically "No" We'll get partial data, but we'll never get complete data and full functionality when we move outside of the Supported and Authorized Ecosystem.

Hence why I have lowered my expectation, and resorted to ideally looking at basing PointCLouds on *.pts agnostic approach, and separating the photogrammetry game from the PointCloud game. Also when we look beyond TLS (Tripod Laser Scanning) and move into, SLAM based, Ariborne, Mobile and other forms of PointClouds we loose even more support of Structured formats. Yes *.pts is very primitivize and simple. But it is supported by everyone and everything. And you can mix basically every form of PointCloud into a single cloud, even mixing TLS with Airborne with Photo Derived PointClouds.

Again, Leica has created a unique Pay to Play system with LGS. And we already see many software platforms paying the price to simply support the LGS files. Perhaps the Pay to Play is the real approach for those that want more complete functionality. And at that, yes, it is possible to create LGS files from Faro, and Z&F data etc. Something we commonly do for people, as many out there love the simplicity of the JetStream Viewer platform and that it is Standalone.
In my opinion it is not about trade secrets, it is about empowering the users to choose whatever system/software they want to use in their processing workflow without punishing them if they decide for hardware A from manufacturer B.

When we leave things as they are, things will only get worse.
Sure data sharing with an open format won't be perfect, but at least if everybody (or at least most of the vendors) would write exchange files which holds most of their information, in a way that we don't need to dance the - I have so many extra rules to import this file, there is no general rule anymore - dance, then we are one step closer.

I see a lot of sharing in Apple, Google and Windows to be quite honest. Apple and Googles Android are just clones of Linux more or less and Windows is adapting a lot of features from Linux as well, since it is state of the art in any kind of server.
As well as Autodesk is opening their formats, dwg, dxf all read and writeable, their point cloud format can be access thru a SDK (yes you have to pay for that), but still you can use that.

If this exchange format is written in a way, that it supports the metadata from mobile mapping (or other moving scanner systems), software vendors can decide if they want to read that and use that or not (does not make sense to read additional data if you can't digest it).

As I said, with these file sizes generated Today, PTS won't be the answer. No compression, no data checksums, no additional information.
Yes bringing a cloud from software A to B is more or less always possible, but even the read/write speed is the worst possible.
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1079
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 34 times
Been thanked: 121 times
Contact:

Re: What Point Cloud format do you use and why?

Post by jedfrechette »

VXGrid wrote: Fri Sep 03, 2021 8:18 pm In my opinion it is not about trade secrets, it is about empowering the users to choose whatever system/software they want to use in their processing workflow without punishing them if they decide for hardware A from manufacturer B.
I 100% agree. For a concrete example of how open formats make our lives better, just consider digital imaging. Whether you are doing remote sensing with some obscure satellite based imaging sensor, shooting a Hollywood blockbuster, taking portraits at a wedding, or snapping a selfie at the pub, it is generally very easy to losslessly convert the images you're working with out of their original proprietary formats in to a widely used open format that you can work with in whatever applications you want.

Nobody passed any laws to make that happen. And, as I've said before, our scan data just isn't that complicated, it's not even very big, so I have a hard time buying that some super secret proprietary format that only the geniuses at $VENDOR could think up is necessary.

The thing I like about COPC is that it seems like a very pragmatic solution to some of the roadblocks we have now. Ultimately, I think all of this is more a question of momentum rather than any fundamental technical limitations. The world at large has been trending toward open data formats for a long time and I'm sure that's ultimately where we'll end up too.
Jed
Post Reply

Return to “Opinion Polls”