cyclone to Potree
-
- V.I.P Member
- Posts: 200
- Joined: Mon May 05, 2014 11:19 am
- 9
- Full Name: Malte Holm
- Company Details: TLS
- Company Position Title: Surveyor
- Country: Norway
- Linkedin Profile: Yes
- Has thanked: 2 times
- Been thanked: 1 time
cyclone to Potree
Hi All
I have done a lot of road scannings.
Lately my company has gotten Potree, where we want to upload a lot of this date, to share with our customers.
My regular procedure is to export an e57, use lastools to convert it into las and finally use the potree converter.
When i do this procedure, the computer runs i somekind of a loop in the tileing process, where it never ends. I can see in the commandopromt that the tileing process runs slower, as it gets further into the process and after around 24 hours, it almost doesnt calculate anymore.
I have around 100 gb of data, i try to convert.
Does anyone have experienced this kind of issue?
I have done a lot of road scannings.
Lately my company has gotten Potree, where we want to upload a lot of this date, to share with our customers.
My regular procedure is to export an e57, use lastools to convert it into las and finally use the potree converter.
When i do this procedure, the computer runs i somekind of a loop in the tileing process, where it never ends. I can see in the commandopromt that the tileing process runs slower, as it gets further into the process and after around 24 hours, it almost doesnt calculate anymore.
I have around 100 gb of data, i try to convert.
Does anyone have experienced this kind of issue?
- LeicaUK_RC
- V.I.P Member
- Posts: 777
- Joined: Sun Jul 17, 2011 9:30 am
- 12
- Full Name: Phil Marsh
- Company Details: Leica Geosystems part of Hexagon
- Company Position Title: Reality Capture Director UK
- Country: UK
- Skype Name: Phil.H.Marsh
- Linkedin Profile: Yes
- Location: Barnsley, UK
- Been thanked: 52 times
- Contact:
- smacl
- Global Moderator
- Posts: 1409
- Joined: Tue Jan 25, 2011 5:12 pm
- 13
- Full Name: Shane MacLaughlin
- Company Details: Atlas Computers Ltd
- Company Position Title: Managing Director
- Country: Ireland
- Linkedin Profile: Yes
- Location: Ireland
- Has thanked: 627 times
- Been thanked: 657 times
- Contact:
Re: cyclone to Potree
100gb in one LAS file or in a number of smaller LAS files? I'd expect the potree converted to take a few hours for this but 24 hours seems very excessive. Like any slow process, you need to figure out where the bottleneck is. The potree converter is basically building an octree comprising of lots of small files so really benefits from a very fast SSD. Also look at increasing the --flush-limit parameter to use more memory and reduce the number of disk writes. Potree is a great package but the documentation isn't so good and you really need to experiment a bit with it to get the best out of it.MalteHC wrote: ↑Tue May 26, 2020 7:05 am Hi All
I have done a lot of road scannings.
Lately my company has gotten Potree, where we want to upload a lot of this date, to share with our customers.
My regular procedure is to export an e57, use lastools to convert it into las and finally use the potree converter.
When i do this procedure, the computer runs i somekind of a loop in the tileing process, where it never ends. I can see in the commandopromt that the tileing process runs slower, as it gets further into the process and after around 24 hours, it almost doesnt calculate anymore.
I have around 100 gb of data, i try to convert.
Does anyone have experienced this kind of issue?
- geomontgomery
- V.I.P Member
- Posts: 186
- Joined: Thu Sep 13, 2018 3:23 pm
- 5
- Full Name: George Montgomery
- Company Details: ECE Design
- Company Position Title: Support Manager
- Country: USA
- Linkedin Profile: Yes
- Has thanked: 20 times
- Been thanked: 48 times
Re: cyclone to Potree
I used the same procedure you did to play around with potree. My cloud was 1/4 the size of yours though It did work in the end. Perhaps you're running out of temp space when the cloud is trying to convert? Have you tested with smaller clouds first just to see if you can get any product out?
-
- V.I.P Member
- Posts: 200
- Joined: Mon May 05, 2014 11:19 am
- 9
- Full Name: Malte Holm
- Company Details: TLS
- Company Position Title: Surveyor
- Country: Norway
- Linkedin Profile: Yes
- Has thanked: 2 times
- Been thanked: 1 time
Re: cyclone to Potree
It is in a single LAS.
It is quite a lot, especially doing to i have a very powerfull computer, dedicated to this kind of work.
My computer has 128 gb of ram and 2TB m.2 disk drive.
Where do i control the flush limit?
It is quite a lot, especially doing to i have a very powerfull computer, dedicated to this kind of work.
My computer has 128 gb of ram and 2TB m.2 disk drive.
Where do i control the flush limit?
smacl wrote: ↑Tue May 26, 2020 12:10 pm100gb in one LAS file or in a number of smaller LAS files? I'd expect the potree converted to take a few hours for this but 24 hours seems very excessive. Like any slow process, you need to figure out where the bottleneck is. The potree converter is basically building an octree comprising of lots of small files so really benefits from a very fast SSD. Also look at increasing the --flush-limit parameter to use more memory and reduce the number of disk writes. Potree is a great package but the documentation isn't so good and you really need to experiment a bit with it to get the best out of it.MalteHC wrote: ↑Tue May 26, 2020 7:05 am Hi All
I have done a lot of road scannings.
Lately my company has gotten Potree, where we want to upload a lot of this date, to share with our customers.
My regular procedure is to export an e57, use lastools to convert it into las and finally use the potree converter.
When i do this procedure, the computer runs i somekind of a loop in the tileing process, where it never ends. I can see in the commandopromt that the tileing process runs slower, as it gets further into the process and after around 24 hours, it almost doesnt calculate anymore.
I have around 100 gb of data, i try to convert.
Does anyone have experienced this kind of issue?
- smacl
- Global Moderator
- Posts: 1409
- Joined: Tue Jan 25, 2011 5:12 pm
- 13
- Full Name: Shane MacLaughlin
- Company Details: Atlas Computers Ltd
- Company Position Title: Managing Director
- Country: Ireland
- Linkedin Profile: Yes
- Location: Ireland
- Has thanked: 627 times
- Been thanked: 657 times
- Contact:
Re: cyclone to Potree
--flush-limit is on the potreeconverter command line so just add something like --flush-limit 10000000 to increase it to ten million points. You could also try playing around with the spacing to see if that improves things. Working with SCC, ff the data is from a very high density scanner such as an RTC360 I tend to reduce so that points within 3mm of one another are deleted which can cut the file size down by about 60%. Very easy to get masses of points you don't want or need very close to the scanner if you don't do something like this.
You do not have the required permissions to view the files attached to this post.
-
- I have made 40-50 posts
- Posts: 40
- Joined: Fri Nov 01, 2013 7:03 am
- 10
- Full Name: Markus Schuetz
- Company Details: potree.org
- Company Position Title: developer
- Country: Austria
- Linkedin Profile: No
- Been thanked: 5 times
Re: cyclone to Potree
PotreeConverter can be a bit slow and unstable for >10 billion points right now but here are some possible suggestions:
* You can consider using Entwine instead of PotreeConverter to build up the octree, since entwine is a bit better at creating an octree out of massive data sets. You can still use Potree to visualize the results:
** https://entwine.io/
** Some demos: https://potree.entwine.io/ (there is even one with 640 billion points)
* Check if the points inside the las file are in some way sorted by tile, or if they are spread all around. If they're spread randomly, then memory consumption will skyrocket and once you run out of memory, performance will drop to zero. It's advantageous to partition large point clouds into tiles of 1 to 50 million points, each, and then specify the folder containing the tiles as the input of the converter.
I was able to build an octree out of 18 billion points in around 16 hours, so 1 billion points / hour should be realistic. However, it also depends on whether the data has high depth complexity (buildings with interiors) or low depth complexity (aerial lidar scans). For the former, it's strongly recomended to partition it into smaller tiles since it improves memory consumption and reduces the amount of reads/writes that are necessary to process all the data.
In any case, I'm currently working on a complete rewrite of the converter which will be around 10x faster and produce 3 files instead of thousands to millions of files. It's still 1-3 months until it's ready, and it was mainly intented for the new WebGPU based rewrite of Potree but some sponsor is already helping out with some funding to bring it back to Potree 1.7 (next WebGL based release) as well. On that note, it'd be greatly appreciated if people/companies who've got good value out of Potree up until now could help out getting these future versions done by becoming sponsors at github! https://github.com/potree/potree
* You can consider using Entwine instead of PotreeConverter to build up the octree, since entwine is a bit better at creating an octree out of massive data sets. You can still use Potree to visualize the results:
** https://entwine.io/
** Some demos: https://potree.entwine.io/ (there is even one with 640 billion points)
* Check if the points inside the las file are in some way sorted by tile, or if they are spread all around. If they're spread randomly, then memory consumption will skyrocket and once you run out of memory, performance will drop to zero. It's advantageous to partition large point clouds into tiles of 1 to 50 million points, each, and then specify the folder containing the tiles as the input of the converter.
I was able to build an octree out of 18 billion points in around 16 hours, so 1 billion points / hour should be realistic. However, it also depends on whether the data has high depth complexity (buildings with interiors) or low depth complexity (aerial lidar scans). For the former, it's strongly recomended to partition it into smaller tiles since it improves memory consumption and reduces the amount of reads/writes that are necessary to process all the data.
In any case, I'm currently working on a complete rewrite of the converter which will be around 10x faster and produce 3 files instead of thousands to millions of files. It's still 1-3 months until it's ready, and it was mainly intented for the new WebGPU based rewrite of Potree but some sponsor is already helping out with some funding to bring it back to Potree 1.7 (next WebGL based release) as well. On that note, it'd be greatly appreciated if people/companies who've got good value out of Potree up until now could help out getting these future versions done by becoming sponsors at github! https://github.com/potree/potree
-
- I have made <0 posts
- Posts: 1
- Joined: Mon May 18, 2020 8:32 pm
- 3
- Full Name: Bogdan Negrea
- Company Details: LE34
- Company Position Title: LE34
- Country: Denmark
- Linkedin Profile: Yes
Re: cyclone to Potree
Hi everyone
I am a collegue of MalteHC and I have also tried to use potree converter with the above mentioned file and I can come with a bit of extra information.
First I dont think that the problem is potree as we have tiled single files of up to 225 gb without any problems.
The present .las file is created as an Cyclone export to .e57 which is then converted to .las.
I have tried to convert each individual .e57 file to .las(with sizes between 2 gb and 15 gb) and then use potree to tile them individually.
While some of the .las files ran fine, for some of them less than half the points were written to the tile output. If I merge these files and then run potree on the merged file it becomes extremly slow and looks like a never ending process.
I have also tried to import the e57 files to Recap and then export them back to .e57 and i could see that only 1/5 of the points were exported which is somehow corelated with the
previous experiment where potree wrote to the output a small amount of points.
I believe there is something wrong with the files coming out of Cyclone as we have no problems processing files exported from Trimble Real Works or Autodesk Recap but we only have problems with the files exported from Cyclone.
If anyone has any ideea of what is going wrong in the process would be much appreciated
I am a collegue of MalteHC and I have also tried to use potree converter with the above mentioned file and I can come with a bit of extra information.
First I dont think that the problem is potree as we have tiled single files of up to 225 gb without any problems.
The present .las file is created as an Cyclone export to .e57 which is then converted to .las.
I have tried to convert each individual .e57 file to .las(with sizes between 2 gb and 15 gb) and then use potree to tile them individually.
While some of the .las files ran fine, for some of them less than half the points were written to the tile output. If I merge these files and then run potree on the merged file it becomes extremly slow and looks like a never ending process.
I have also tried to import the e57 files to Recap and then export them back to .e57 and i could see that only 1/5 of the points were exported which is somehow corelated with the
previous experiment where potree wrote to the output a small amount of points.
I believe there is something wrong with the files coming out of Cyclone as we have no problems processing files exported from Trimble Real Works or Autodesk Recap but we only have problems with the files exported from Cyclone.
If anyone has any ideea of what is going wrong in the process would be much appreciated
- smacl
- Global Moderator
- Posts: 1409
- Joined: Tue Jan 25, 2011 5:12 pm
- 13
- Full Name: Shane MacLaughlin
- Company Details: Atlas Computers Ltd
- Company Position Title: Managing Director
- Country: Ireland
- Linkedin Profile: Yes
- Location: Ireland
- Has thanked: 627 times
- Been thanked: 657 times
- Contact:
Re: cyclone to Potree
Don't know what is going wrong, but if you want to send through one of the LAS files that is failing to me via wetransfer or similar I can run it through SCC and out to potree to see how it behaves. This would let you know if the problem with the LAS file produced by Cyclone or the potree converter application.