| IMPORTANT: This is the old version of the tools. The new version is available here: https://flightsim.to/file/26245/google-earth-decoder-optimization-tools-gedot-blender-addon as a Blender addon
TUTORIAL VIDEO: OjO made a great tutorial to create photogrammetry sceneries with the MSFS decoder, and my optimization tools. Feel free to have a look here: https://youtu.be/6fgTxEh_x84 and https://youtu.be/yRE37NAgwiA (for the merge_sceneries.py script)
IMPORTANT: for the merge_sceneries script to work properly, the source scenery and the destination scenery must share the same min LOD (17 here). The reason is that the min LOD determines the size of the tiles, and the size of the tiles from the source scenery must be the same as the one from the destination scenery.
Python blender scripts designed to be used with Google Earth Decoder Tool sceneries (https://github.com/Thalixte/Google-Earth-Decoder-optimisation-tools).
1) Presentation:
The archive contains four python scripts:
- scenery_optimisation.py: optimize Google Earth Decoder scenery (textures, Lods, CTD fix)
- fix_tiles_altitudes.py: fix wrong tiles altitudes that can occure when moving tiles on the x-y axis
- update_objects_LODs.py: update the LOD levels for the tiles of a scenery based on an array of minsizes
- merge_sceneries: merge all the files from a source Google Earth Decoder scenery to a destination Google Earth Decoder scenery
The scripts are intended to be used with a Google Earth Decoder min LOD of 17. I did not test them with another Google Earth Decoder minLod.
2) Prerequisites:
- Blender 2.83 (https://download.blender.org/release/Blender2.83/blender-2.83.9-windows64.msi). The blender version is important
- Node js (https://nodejs.org/dist/v14.15.1/node-v14.15.1-x64.msi)
- Blender2MSFS Toolkit: https://www.fsdeveloper.com/forum/resources/blender2msfs-toolkit.256/download
- Lily texture Packer IMPORTANT ! download version 1.1.x only: https://gumroad.com/l/DFExj
3) scenery_optimisation script:
3.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery).
It bakes the tiles texture files in order to reduce the number of the files in the final package, and reduce the scenery loading time.
It fixes an issue with the Google Earth Decoder tool, that breaks the LOD management system.
Look at this picture: https://flic.kr/p/2k3YCrd
This picture shows a wireframe representation of a big scenery created from the Google Earth Decoder tool. You can see that the scenery tiles are dark gray, which means there are a lot of vertices on each tile, because all the tiles are on the max level of detail (which is 19 here). The result is that this scene, from this point of view, displays more than 80M of vertices, which bottlenecks the GPU (and the CPU).
Normally, we should have this: https://flic.kr/p/2kayM52. In this case, we can see that the tiles have the correct LOD levels, and it leads to a reduced number of vertices (20M in this case, but it can be changed according to the LOD levels that are set in the optimisation script (see chap. 3.3 - Configuration)).
The problem here is that the Google Earth Decoder gives all the tiles the same origin, and changes the bounding box to go from this origin point to the last mesh vertice point.
Another way to see this problem, is to use the new MSFS SDK Debug LOD feature, that displays the current lod level of each tiles, and the size of the bounding sphere: https://flic.kr/p/2kautrR
The more the green or blue is the color, the more detailed is the tile. The more the yellow or red is the color, the less detailed is the tile.
Here, we can see that the more detailed tiles are the one that are far from the camera, which is the exact opposite result than the one we want to obtain to optimize our scenery framerate.
Now, if each tile has its own origin point, and has a bounding box corresponding to the real tile size, we obtain those results: https://flic.kr/p/2kayiyp and https://flic.kr/p/2kayiAo, which are far better from a LOD point of view, and better preserve the framerate.
This is the fix that the script applies to the tiles: it gives each tile its own origin point, and resizes the bounding boxes according to the real tile size.
The script also changes the LOD levels to better suit the LOD management system, and allows (if configured) to convert texture files into jpg format, in order to reduce the texture file size (but with a possible loss in the texture quality).
The script also applies ASOBO extension tags to the gltf files, in order to enable the road management and the collisions. It also fixes texture flickering issues.
The script automatically removes the orphaned scenery object xml files (scenery object xml files that do not have associated gltf and/or bin files)
3.2) Installation:
Just put the scenery_optimisation.py script, and the retrievepos.js script in a folder of your choice.
3.3) Configuration:
Change the following settings, according to your project:
- bake_textures_enabled: tells the script to optimize the textures by baking all the textures corresponding to the min Lod levels of the tiles (default is True). For instance, if you have a gltf file for a tile that is named 30604141705340627_LOD00.gltf, all the texture files corresponding to this tile and this LOD level (all the texture files that start with 30604141705340627_LOD00) will be baked into one single texture
- projects_folder: the parent folder that contains your sceneries
- project_name: the name of your project
- node_js_folder: the folder that contains the node js script that retrieves the Google Earth coords
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- target_lods: an array representing the minsize values per LOD, starting from a minLod of 17 (from the less detailed lod to the most detailed)
- project_file_name: the name of the xml file that embeds the project definition (by default, project_name.xml or author_name+project_name.xml)
- scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml)
- package_definitions_file_name: the name of the xml file that embeds the package definitions (by default, project_name.xml or author_name+project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
- output_texture_format: format of the final texture files (values are PNG_FORMAT, JPG_FORMAT, default is PNG_FORMAT)
- JPG_COMPRESSION_RATIO: if you choose the jpg format for the output texture files, indicates the compression ratio
3.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the scenery_optimisation python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
3.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- if no other backup exists, backup the scenery modelLib files into a backup folder, inside the project folder
- if necessary, install the node Js xhr2 module (npm install xhr2)
- install the PIP and Pillow libraries to allow texture format conversion
- if png texture files output format is selected, convert all remaining jpg texture files to PNG, then remove jpg files
- if jpg texture files output format is selected, convert all remaining png texture files to JPG, then remove png files, and compress all jpg texture files
- retrieve objects positions from Google earth, via the Google Earth API, and put the results in .pos files corresponding to the scenery tiles
- update the tiles position, using those .pos files
- place all tiles objects into corresponding sub folders, in order to group the objects corresponding to the same tile
- update the LODs of the scenery tiles, according to the target_lods defined in the configuration settings
- optimize the tiles, by baking the textures corresponding to LOD levels of the tiles, and by changing the bounding box of the tiles, in order to optimize the LODs
- applies ASOBO extension tags to the gltf files, in order to enable the road management and the collisions
- fix gltf doublesided attributes, in order to remove texture flickering issues
- removes the orphaned scenery object xml files (scenery object xml files that do not have associated gltf and/or bin files)
- automatically rebuild the scenery package, if the MSFS SDK is correctly installed, and MSFS 2020 is not running
4) fix_tiles_altitudes script :
4.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/).
When using the scenery_optimisation script, there are chances that the resulting tiles are slightly decaled in the x and/or y axis.
This decal can easily be fixed by going into the MSFS SDK, open the project, click on the "Save scenery..." button to order the scenery objects by type, then select all the tiles (not the lights, rectangles or polygons, just the tiles) and move them to the appropriate location. But this move can break some tiles altitude.
To fix it, save your scenery after moving the tiles to their appropriate location, then close your project and MSFS 2020, and run the fix_tile_altitudes script. Once the script has finished running and rebuidling your project, reopen the project in the MSFS SDK. The tiles position and altitude should now be correct.
4.2) Installation:
Just put the fix_tile_altitudes.py script, and the retrievepos.js script in a folder of your choice.
4.3) Configuration:
Change the following settings, according to your project:
- projects_folder: the parent folder that contains your sceneries
- project_name: the name of your project
- node_js_folder: the folder that contains the node js script that retrieves the Google Earth coords
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- project_file_name: the name of the xml file that embeds the project definition (by default, project_name.xml or author_name+project_name.xml)
- scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml)
- package_definitions_file_name: the name of the xml file that embeds the package definitions (by default, project_name.xml or author_name+project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
- fix_with_googleEarthDecoder_data: if set to True, tells the script to use the backup of the old objects.xml data (the one produced by the Google Earth Decoder tool) to fix tiles altitude. If set to False, tells the script to directly retrieve tiles altitude from the Google Earth API (default is True)
4.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the fix_tile_altitudes python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
The fix comes with two methods: the default one uses the backup of the old objects.xml data (the one produced by the Google Earth Decoder tool)the other method tries to retrieve the altitude directly based on the Google Earth data (using the Google Earth API)
4.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- if necessary, install the node Js xhr2 module (npm install xhr2)
- retrieve objects positions from Google earth, via the Google Earth API, and put the results in .pos files corresponding to the scenery tiles
- update the tiles altitude, using the backup objects.xml file, if fix_with_googleEarthDecoder_data is set to True, or using the .pos files generated by the Google Earth API, if fix_with_googleEarthDecoder_data is set to False
- automatically rebuild the scenery package, if the MSFS SDK is correctly installed, and MSFS 2020 is not running
5) update_objects_LODs script:
5.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/).
This script automates the process of changing the LOD levels for all the tiles of a scenery.
5.2) Installation:
Just put the update_object_LODs.py script in a folder of your choice.
5.3) Configuration:
Change the following settings, according to your project:
- projects_folder: the parent folder that contains your sceneries
- project_name: the name of your project
- target_lods: an array representing the minsize values per LOD, starting from a minLod of 17 (from the less detailed lod to the most detailed)
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- project_file_name: the name of the xml file that embeds the project definition (by default, project_name.xml or author_name+project_name.xml)
- scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml)
- package_definitions_file_name: the name of the xml file that embeds the package definitions (by default, project_name.xml or author_name+project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
5.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the update_object_LODs python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
5.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- update the LOD levels for each tile in the tile_object_name.xml files
- automatically rebuild the scenery package, if the MSFS SDK is correctly installed, and MSFS 2020 is not running
6) merge_sceneries script:
6.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/).
This script automates the process of merging a Google Earth Decoder source scenery into another Google Earth Decoder scenery.
6.2) Installation:
Just put the merge_sceneries.py script in a folder of your choice.
6.3) Configuration:
Change the following settings, according to your project:
- projects_folder: the parent folder that contains your sceneries
- src_project_name: the name of the scenery that you want to include in the final scenery
- dest_project_name: the name of the final project that should include both sceneries
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- src_project_file_name: the name of the xml file that embeds the source project definition (by default, src_project_name.xml or author_name+src_project_name.xml)
- dest_project_file_name: the name of the xml file that embeds the destination project definition (by default, dest_project_name.xml or author_name+dest_project_name.xml)
- src_scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml) for the scenery that you want to include in the final scenery
- dest_scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml) for the final project that should include both sceneries
- src_package_definitions_file_name: the name of the xml file that embeds the source package definitions (by default, src_project_name.xml or author_name+src_project_name.xml)
- dest_package_definitions_file_name: the name of the xml file that embeds the destination package definitions (by default, dest_project_name.xml or author_name+dest_project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
6.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the update_object_LODs python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
6.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- backup the modelLib files of the final project that should include both sceneries, into a backup subfolder, so called merge_sceneries
- copy all the xml files, gltf files, bin files and texture files from the source scenery to the destination scenery, overwritting the existing ones
- update the destination scenery scene xml file (objects.xml by default) to change the tiles guid corresponding to the source scenery tiles
- add the guid for the source tiles that does not exist in the destination scenery
- automatically rebuild the scenery package, if the MSFS SDK is correctly installed, and MSFS 2020 is not running
7) clean_package_files script:
7.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/).
This script automatically removes unused package files (.gltf, .bin and texture files), aka the files that are linked with tiles that have been removed from the scene. It appears that, if this cleaning is not done, the resulting package keep those files.
7.2) Installation:
Just put the clean_package_files.py script in a folder of your choice.
7.3) Configuration:
Change the following settings, according to your project:
- projects_folder: the parent folder that contains your sceneries
- project_name: the name of your project
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- project_file_name: the name of the xml file that embeds the project definition (by default, project_name.xml or author_name+project_name.xml)
- scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml)
- package_definitions_file_name: the name of the xml file that embeds the package definitions (by default, project_name.xml or author_name+project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
7.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the clean_package_files python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
7.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- browse the scenery object xml files in the object folder
- check the presence of the corresponding guid in the scene xml file
- if the guid is not found in the scene xml file, remove the files associated with this scenery object xml file (.gltf, .bin, and texture files), in order to reduce the resulting package size
- if files are associated to a LOD that is not used in the .xml files, those files are now removed
8) compress_built_package script:
8.1) Prerequisites:
- This script uses the Compressonator tool from GPUOpen. You can download it here: https://github.com/GPUOpen-Tools/compressonator/releases/download/V4.1.5083/CompressonatorCLI_x64_4.1.5083.exe
- it only works on already built package. So be sure to have built your package correctly first. The package should be in the folder <path_to_projects\project_name\Packages> folder
8.2) Presentation:
This script automatically compress DDS texture files from an already built package by applying the DXT1 compression (special thks to Archer374 for suggesting me this possible optimization). It can remove transparency component from the textures, but since Google Earth tiles do not use it, as far as i know, it is lossless.
8.3) Installation:
First, install the compressonator tool in the folder of your choice. Then, put the clean_package_files.py script in a folder of your choice.
8.4) Configuration:
Change the following settings, according to your project:
projects_folder: the parent folder that contains your sceneries
project_name: the name of your project
compressonatortool_folder: the folder that contains the compressonator exe (compressonatorcli.exe) that convert and compress the DDS texture files of the packages. This exe shoud be in the <path_to_compressonator\bin\cli> folder
author_name: the name of the author of the scenery
NB_PARALLEL_TASKS: number of parallel instance of compressonator running at the same time. If you experiment issues running the script, try lowering this setting
8.5) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the clean_package_files python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
8.6) Process:
- browse the built package for DDS files in the folder
- convert all DDS files to temporary BMP files
- convert and compress all BMP files to DXT1 format DDS files, in order to reduce the resulting package size
- remove temporary BMP files
If you want to donate, you will always be welcome to help me continue with more projects and update the existing ones https://paypal.me/Thalixte.
User Reviews
Review data
Share your thoughts
If you’ve used this add-on, share your thoughts with other pilots.
Write a reviewzaelu
Hello derekhe.
I was thinking at a way to make your program do manual Caches using MFS.
I wrote already this once and the site delete everything because I clicked out of the comment window so I will try again but I will be more short.
So the way I thought it could work is like this:
Your program has two features:
So how about you add a new option that Forces an LOD? For example LOW MEDIUM HIGH and ULTRA. ULTRA would be your "High LOD from now" and HIGH would be HIGH from MFS Manual Cache.
Also a way to rename the folders with cached files by the user and make them not to auto delete by your program
And here is how it will work:
A warning should be shown to user that the data can be very large. Like 20GB or more for a 3000 squared miles area.
===========
Another option would be that your program downloads normally the LOW LOD ass asked by MFS and saves the last session and after that it analyzes the area and presents the surface area and the estimated download size for user. And then user can activate a final option to download HIGH or ULTRA LOD for that particular Area. Even with MFS now shut down.
Would it be possible?
Also a question. Is your program now limited to 20GB max size of the cache? It seems its not passing over 20GB even if I set it to 30.
toshie27
Thalixte 11 month(s) ago
Hey, that's the limitations of the photogrammetry ! You can use 21 or 22 to get more detailed tiles, but it wil not work, as you will have too big iles, and the MSFS engine will not be able to render them (too much vertices).
Comonsys
Stopped on Line 74 "from bpy.app import binary_path_python" while running script. What can I do?
Thalixte 1 year(s) ago
Have you downloaded the last version ? Otherwise, try to run blender as an administrator
steveli
Right now, I'm frequently got an error message of "Google has blocked the request https://kh.google.com/rt/earth/BulkMetadata/pb=!1m2!1s!2u874". On the first time, I waited for 24 hours to get unblocked from google. But after a couple of hour usage, I got the same error message again - blocked by google. This time, I waited 48 hours to get unblocked. Now, I'm waiting on the 3rd time block from google. 48 hours passed already, I'm still blocked by google. Does anyone have any experience on how to get rid of this annoying block from google?
johnel34 1 year(s) ago
He says in the video that if you download an area of more than 1500 tiles, then Google might block you for a few hours, so you should only download a smaller area at a time. You can also try turning your router off and on again, and that should give you a new ip adress, so you can use Google again.
Thalixte 1 year(s) ago
I will update this week a version that get rids of the necessity to make googleearth requests to retrieve the tile coords. Stay tuned 😉
Mobias
Hi, I've been trying to use this today and when I run the script after a few minutes I get an error under 'BAKE TILE TEXTURE FILES' saying AttributeError 'Material' object has no attribute 'msfs_show_road_material'.
and then Python script failed, check message in the system console.
and then in line 1056 it highlights material.msfs_show_road_material = True
Is there something I'm doing wrong? Thanks for any help.
Just to add. I think this might be an issue with Lilly's texture packer rather than your optimisation tools.
Thalixte 1 year(s) ago
Hi. this is because tou have not installed and enabled the gltf2msfs blender addon. I will update a new version of the tools that get rid of this error, and which is more user friendly. Stay tuned 😉
Mobias 1 year(s) ago
Great thanks for your reply. I go your tool working ok anyway and made a scenery with it. Thanks for your hard work.
steveli
I am using this mod to making photogrammetry scenery and it works very well except one problem with "Lily texture packer":
Thalixte 1 year(s) ago
Hello, and thks for your feedback. As i am not the author of the Lily Texture packer, i cannot totally answer your question. But as far as i am concerned, i did not see any compression algorithm applied on by the addon. It only changes the UV map, and merge all the tile textures into one single file.
James1976
I used this tool with Blender and Earth 2 MSFS when I tried to get some Arizona towns to it's latest, but some towns like Buckeye, Surprise and Goodyear are still with Bing Maps. The State Farm Stadium is still University of Phoenix Stadium while Gila River Arena is still jobing.com Arena. Is there something I can do to fix these?
Thalixte 1 year(s) ago
Hello. Actually, there is no tool available to retrieve Bing maps fata (except in MS/Asobo's side). The author of the Google earth decoder tool told us he would bring the code in github, but he never did it. So there is no opppotunity for the moment to retrieve Bing data easily (and i have not the time to find a way to do it by myself, sorry). Maybe i will ask MS/Asobo if there is a possibility for them to extend their tools, but i don't think they will agree, because of copyright issues, and so son...
James1976 1 year(s) ago
Thanks, and I also tried to get the halfway made SoFi Stadium in Inglewood with parking spaces and the lake to get in there, but it only showed 1/4 made construction.
Kramnik
Hello, thx for your scripts, using them i was able to make a photogrammetry scenery of Taranto/Italy (you can find it here in flightsim.to) but i've a problem: texture files are THOUSANDS and the scenery cause a long time for starting FS, so i tried to use Lily texture packer, ..i correctly installed it in Blender addons, but using scenery_optimisation.py or compress_built_package.py, with bake_textures_enabled = True, ..i always get as result the same number of files created in Texture dir. So what i miss? i installed 1.1.3 version of LilyPacker (you wrote 1.1.x is good), must i try to use a different version? Thx in advance!
Thalixte 1 year(s) ago
Wat you have to check is that you have one texture per tile LOD. Depending on the scenery, yes, it can stil be thousands of textures. But compared to the number of textures withour packing them, yoy should have 10x less textures than without it
Kramnik 1 year(s) ago
After many tries, i found that all works correctly with scenery_optimisation and Lily packer if i start from the beginning, that is with a non-optimized yet package. The problem for my Taranto scenery is that i already made a final package without using Lily packer, and i made many adjustments from inside FS so i don't want to restart. Now i just released an upgrade of my Taranto scenery because with your scripts CLEAN and COMPRESS i was able to obtain a drastic reduction of files and dimensions. But it would be perfect if COMPRESS script could work with LilyPacker, inside the script i found a command (bake_textures_enabled = True) but it seems to do nothing. Can you help me? Thx again! (and sorry for my english..)
Thalixte 1 year(s) ago
Hi. Compress and bake are two different processes. The compress script only works on .DDS files. So it cannot be used for the tectures in the project (except you use directly DDS textures in your project).
The bake process reduces the number of texture files, by merging them for one gltf model.
Kramnik 1 year(s) ago
Hi, ..so last question is: is there a way to bake an already optimized package? And however, thanks for your scripts! i'll never be able to do a photogrammetry scenery without them!
Thalixte 1 year(s) ago
The bake process rely on gltf headers. If the "Scenery optimized Khronos glTF Blender I/O v1.2.75" is present in the gltf header, then, the bake process will be bypassed. What you can do is to change this line i the optimization script:
to
Kramnik 1 year(s) ago
Just tried but don't work, i tried with scripts scenery_optimisation.py and also bake_textures.py (the name of this script tells me that it should do exactly what i want), modified as suggested with 'bake_textures_required = bake_textures_enabled' ...nothing, bake don't start. Really thanks for your help attempts, i don't know if a way really exists
Thalixte 1 year(s) ago
Just PM me a link where i can download your project files, and i will see what i can do.
Tyrell63 Premium
Hi
I only have 2 small/big questions (for the beginning)
1.) where to put the Lily Texture Packer scripts? Same folder as the Google Earth Optimization scripts?
2.) After (of before?) I run the script.......can I remove unwanted buildings or parts of the import (e.g. to have only a special area with nice borders around (e.g. streets). Cause to area I want to import is never only rectangle....more "something else"...and there is a lot of "junk"around which is not needed/wanted. How can I remove this?
Many greetings from Singapore,
Lutz
Thalixte Premium 1 year(s) ago
1) Lily texture packer script is a blender addon. Download it, go to the addon manager in Blender, and load the archive. Then, once loaded, enable it in Blender.
2) For the area, i answered to you in PM.
Tyrell63 Premium 1 year(s) ago
Thank you. Its done
OjO
Hello again, I made a new tutorial, this time about the merge_sceneries script.
https://youtu.be/yRE37NAgwiA
Thalixte 2 year(s) ago
Great ! Will add the link. Thks a lot. Just a remark however: the fspackagetool_folder variable is important as it indicates the path to the build exe of the SDK (even if this build exe launch the Flightsimulator exe to really make the build).
OjO 2 year(s) ago
Thanks, but I am not sure I understand the remark: at 4:10 I say that the first half of that line (28) should be the folder where we installed the SDK (that's why I change it to F:\\MSFS SDK), while the second half should remain Tools\\bin (because these subfolders don't change, whatever the installation directory we chose for the SDK)
Thalixte 2 year(s) ago
Ah OK, i understand.
OjO 2 year(s) ago
I hope it's not too confusing, but the line is edited in the same way as in the optimization process, so I just assume that anybody who knows how to optimize a scenery has already an idea of what that line does 😀
Thalixte 2 year(s) ago
That's legit 😉
kvari
Is there a way to directly connect to Google live instead of this ?
Thalixte 2 year(s) ago
Hello! Can you explain more precisely what you mean ?
suomy
Do you know why this happens ? I've already used (scenery optimization/retrieverspos ) and also (scenery_optimisation/decoder) neither of them recovers the positions. Thanks
https://pasteboard.co/IoqZivhlRqcC.png
Thalixte 2 year(s) ago
Yes. It could be because you have been blacklisted by Google due to too many requests to the api. What are the longitude and latitude in the xml scene file ?
suomy 2 year(s) ago
Thanks for the answer.
oh. Ok, I thought while downloading the tiles it doesn't lock the positions.
In fact at the moment I'm blocked by google.
The positions are these.
lat="41.149258131456811" lon="-8.7071679949538634 City of Porto
theluckyboy 2 year(s) ago
Bonjour les amis,
I think I have the same problem with Google.
I have this kind of error message:
"ERROR ! Google has blocked the request https://kh.google.com/rt/earth/BulkMetadata/pb=!1m2!1s!2u874"
Thalixte 2 year(s) ago
No problem on my side today. try to change your public ip address to fix this issue.
suomy 2 year(s) ago
Yesterday I bought a vpn service (ExpressVPN), but it doesn't work, everything came in blocks as I showed in the photo.
OjO 2 year(s) ago
Hi, the ban from Google should end after no more than 24h, according to my experience. How big is the area that you are trying to import? Have you tried restarting the project from scratch, and I mean by using a new folder, then GED and then Thalixte's scripts?
theluckyboy 2 year(s) ago
All is right 24 hours after the Google ban.
I dont know what is the amount of data we can download with the Google Earth Decoder to avoid the ban during the optimization or the altitude fixing.
suomy 2 year(s) ago
I know all this, and I've never had any problems doing photogrammetry, I have several scenarios here in flightsim, since I updated (SDK 1.14.1) I'm having problems, I've tried everything with the help of Thalixte and still haven't had luck and i've also redid the projects and tried in new areas and i always have this problem of getting piled up and out of coordinates.
CyberDude
@Thalixte, thanks for creating and sharing this wonderful tool!
MistyEagleGaming
Hello, thanks for the amazing tool. However, is that anyway I can exclude the airport from the google photogrammetry ?
Thalixte 2 year(s) ago
Yes, once youd downloaded and optimize the tiles, go into the dev mode of MSFS, select the tiles corresponding to the airport, and remove them. Then, use terraforming polygons to blend the airport with the rest of the photogrammetry.
OjO
Hello, I made a video tutorial in English that shows how to use the scenery_optimisation script. I hope it's useful!
You find it here: https://youtu.be/6fgTxEh_x84
Thalixte 2 year(s) ago
Wonderful tutorial, very didactic and exhaustive. Just a remark: you can download more tiles from the msfs decoder tools. In my rig, i have 32Gb of RAM, and i can use a rectangle that covers approximatively 15000 data without any problems, with 17-19 Lods. More data can be problematic, but until 15000, it is good.
Also, Thanks for the "snap to ground" trick to correct tile elevation issues when moving them. Never thought about that. I made a specific script for that purpose (fix_tiles_altitude.py), but your tricks seems to make it easier to apply the correction 😉 .
Also, great colorimetry settings for the Benevento scenery. I guess i will try them for some future creations.
I added your video in the description of the tools.
OjO 2 year(s) ago
Thank you for the scripts. They are actually easy to use, once you know how to deal with scripts in Blender. And the results are great.
The color correction values are actually taken from the Brazilian video I used as a base for my tutorial (I think I said so in the video in Italian, but didn't do so in the English one - I'll put it in the description), so all the merits go to Rodukao.
The snap to ground technique is all mine, though 😋. I experimented a bit because I couldn't use your altitude script in the right way and I'm glad I could come up with a solution that can be applied while editing the scenery in MSFS. I've no idea why it works, but it does, so... no questions asked!
Regarding the number of tiles. I've kept the number extremely low because the problem in my case wasn't the RAM (I've got 32GB), but the part when the script tries to retrieve the coordinates from google. I experimented with 8000 tiles and every time I tried I had my IP blocked and/or all the tiles ended up stacked at the same coordinates (I've a fast internet connection, so it shouldn't be related to a poor connection). I've tried with the [0,0,0] pos file but couldn't solve the issue, so I decided to stick with smaller numbers and rely on the merge script. As I say in the video, people can try higher sizes, but I wanted to suggest those that gave me the smallest amount of issues (or none at all).
Speaking of the merge script... I was planning to do a short tutorial about it too, but I've got some strange issues with it, especially with altitudes (but there's also a problem with some smaller tiles disappearing too - it seems to me that it's a problem with the LOD, but I'm not sure). I want to test it a little bit more, but eventually I might have to ask you. Should I send you a PM, in case?
Thalixte 2 year(s) ago
Yes you can, for sure. As for th number of data, it is different from the number of tiles. The number of tiles correspond more to the number of metadata (the first retrieving step of the MSFS decoder). For the number of tiles, of course, it is better not to exceed 1500 tiles.
CyberDude 2 year(s) ago
@Signor Ojo, thank you very much for the EXCELLENT tutorial!
You have explained everything so well!I
OjO 2 year(s) ago
@CyberDude Thanks for the feedback!
theluckyboy
Bonjour Thalixte,
Merci pour ces scripts que tu as mis en place.
J'ai réussi à bien suivre tes instructions, mais j'ai un problème de generation de bâtiments sur la scène que je souhaite créer.
Comment empêcher l'autogeneration de bâtiments?
par exmple: https://ibb.co/pWBH8MB
Merci d'avance
Thalixte 2 year(s) ago
Bonjour. As-tu créé un polygone d'exclusion pour que le bâtiment auto généré n'apparaisse pas ?
theluckyboy 2 year(s) ago
Non, merci pour la réponse
theluckyboy 2 year(s) ago
Bonjour,
J'ai finalement réussi à créer quelques scènes de Blagnac et de ses environs (Purpan, En-Jacca) et aussi Labège. Maintenant, reste à améliorer la gestion des couleurs ...
Voici une petite vidéo du résultat: https://youtu.be/u5-TX0hcl9I
Excellent weekend.
Perschistence
What happens if you have two projects which overlap themselves in terms of the area they cover and merge them with the ,,merge_sceneries.py" script? Is it going to retain all the tiles, or will it delete them from one project?
Thalixte 2 year(s) ago
Here is an illustration of the quickly made Frankfurt scenery:
https://live.staticflickr.com/65535/51346108664_8a0d44ade5_3k.jpg
https://live.staticflickr.com/65535/51345362016_98d6b54d5f_3k.jpg
The link to get it: https://drive.google.com/file/d/1TQ8z8oGMw0Zl6BYzp7zxWHZQm65DZnKu/view?usp=sharing
The merge_sceneries keep the tiles from the first scenery indicated in the settings, and remove the corresponding tiles from the second one. To sum up, the second scenery indicated in the settings should always be the target scenery (the one that will be used to produce the package).
Some infos to compare (remember my scenery covers a bigger area):
number of files in the package:
size of the packages:
Perschistence 2 year(s) ago
The screenshots look good, I've got to say! Nice to know, because then it would not make any problems to import smaller parts of a big city and combine the projects later, in the end!
Could I for example import a city in one project with LOD 19 and in another project the center of the city with LOD 21, combine them in the end and get just the center of the city in full detail?
Thalixte 2 year(s) ago
Yes, of course, that's what i do regurarly. But i cannot make 17-21 tiles (too big when building and optimizing). I combine mostly 17-19 tiles with some 17-20 tiles, and it is already great. For instance, in this Frankfurt scenery, i retrieved some 17-20 tiles for the downtown, and the historical center. Later, i surely will merge them with the scenery i already brought to you (which is only 17-19 for the moment)...
Perschistence 2 year(s) ago
Uhh, nice, could you send me the final one too? Would be cool for screenshots^^
Thalixte 2 year(s) ago
There is no final one for the moment. And i don't know if i will release it soon, because there will be the next World Update on Germany, and maybe MS/Asobo will bring photogrammetry for this city, i don't know. But you can already enjoy the one i brought to you in the link.
Perschistence 2 year(s) ago
Thank you! 😊
Perschistence 2 year(s) ago
Do I get this right? When I have imported a town with LOD 19 and the center of the town in another project with LOD 21, I need to use the LOD19 as the dest_project_name and the LOD21 as src_project_name? Because I want to merge the LOD21 into the ,,destination" LOD19.
Should I first merge them and then run the optimization script on the final package, or should I first optimize both projects and then merge them together?
Thalixte 2 year(s) ago
First, optimize them, then merge them, and yes, you are right on the settings. Unsure also that both sceneries have the same min lod (17 for instance)
Perschistence 2 year(s) ago
Sorry, I don't understand your last sentence^^ Do all the projects need to have the same (lowest) LOD?
Thalixte 2 year(s) ago
Yes, they should
Perschistence 2 year(s) ago
I am encountering a problem at the moment. I try to merge a LOD 16 - 20 project with an LOD 16 - 21 Project but after about 6 minutes the FlightSimulator.exe console window is just closing itself without any error message, and it does not complete building the package. If I try to build it in game, the same happens after about 6 minutes. The game just closes down without any error message.
Thalixte 2 year(s) ago
Hi! This is because tou used 16 as min-lod. Using 16 will create bigger tiles thanu using 17, so it will leads to a too big cosmption of resources on your rig, and will crash. I strongly suggest to use 17 as min lods to produce tiles with correct dmensions..I sometimes use 16 as min lods in some sceneries (Thesaloniki for instance), and it worked, but it is very limit, especially if you use 21 as max lods. I personnaly never use 21 as max lods, as it will produce too big tiles, with too much meshes and vertices. I am also finishing the Frankfurt scenery (i hope you don't work on a similar one), as it is great to have this city in photogrammetry (i only hope Asobo will not anounce it to be in photogrammetry in the newt Wolrd Update).
Perschistence 2 year(s) ago
I now did merge some LOD 16-20 Projects with some 16-21 Projects and it did work for some reason. I guess my project yesterday was too big because it is trying to compile it in memory, as it says. Maybe they've set a limit. The package sources should probably stay well below 15 GB, but I am not 100% sure. I really did work on another Frankfurt project, but I liked yours even better than my actual one, just the LOD was too low. I merged LOD 16-21 for the center of the city with LOD 16-20 on the outside. I tried to combine it with LOD 16- 19 for the outside of the border of the city, but it is not compiling anymore. Maybe the project was too big again. Here you can see my try on a HQ version of Frankfurt but I would love to see yours too. Unfortunately, with my ,,brute force approach", I had to make mine a bit smaller. https://www.forum-3dcenter.org/vbulletin/showpost.php?p=12753846&postcount=2305
Thalixte 2 year(s) ago
Do you mind if i publish my scenery this week ? In my last version, the downtown is more detailed (min LOD of 20)
Perschistence 2 year(s) ago
Cool, why should I mind? I think more options are a good thing!
Thalixte 2 year(s) ago
Cool. So do you want me to add some POI in my scenery ? I don't know Frankfurt as well as you do, i guess. Maybe you can help me
Perschistence 2 year(s) ago
Hmm, I am not sure. You could use the highest buildings: https://www.wikiwand.com/en/List_of_tallest_buildings_in_Frankfurt
Thalixte 2 year(s) ago
Yes, that's what i did 😊
Perschistence 2 year(s) ago
Nice, I am looking forward trying your mod. 😊
Perschistence
I have an interesting question. There are some mods on flightsim.to, for example Frankfurt (Germany). They are about 1GB in size. When I try to do the same and import the whole town, my project gets about 10GB in size, or even 20GB. I can go down to LOD 16-19, then I get something that is just a few GB in size, but the quality of my town is way worse (and the project is still bigger). How can you achieve a size of just [email protected] like many city mods here and make it still look like LOD 21 or so? Maybe you can load it in blender first or something and massively simplify it?
I further investigated it.
My packages have a .bgl file what is multiple GB big in the scenery folder. The projects from flightsim.io also have a .bgl file but it is just a few kb in size. What is this?
Thalixte 2 year(s) ago
Hi! First of all, i rarely use 16 as a min LOD, because it produces bigger tiles, that are difficult to handle with Blender, and in the max level of details (21) are too big and cannot be built by the package builder.
Secondly, i mostly use 17-19 LODS, and in some specific places, i use 17-20 (to get a bette level of details near the ground). 17-21 is very very resource consuming. I suggest to use it on a landmark, not on a whole tile.
As of Frunkfurt scenery, according to the feedback, it does not seem to be optimized.
Just tested the Frankfurt scenery, and the area is not as big as i thought, so it explains the size of the package. Also, there are only 3 LODS per tiles, so i guess the author used 17-19 or 18-20 lod settings.
It is also not optimized considering the LOD spheres. In my rig, i have difficulties to maintain 30 fps with this scenery (i have an average of 27-28 fps, with the SU5 installed).
Perschistence 2 year(s) ago
Yes, the Frankfurt-Scenery performs terribly, it even tanks your framerate if you are 20 miles away.
If you take a look at the debug view you can see that it is nearly all the time using the highest LOD, you have to fly very, very far away to let it switch to the next one. The creator should have used your tools. 😉
I wish there would be a way to reconfigure the LOD for finished projects.
But if you try to import the whole town like in his 2 projects, even if you just use LOD 16-19 the project is going to be multiple times bigger. And his mod looks mostly like LOD 20. LOD 19 is way more ,,molten" and ,,blurred".
60% of the size is made up of the [Project_Name]-modelLib.BGL file, what is just about a few KB in a lot of downloadable mods like the frankfurt projects. Do you know what this is or have an idea if we could get rid of it?
It would save 60% space, what would be enormous.
Edit: Oh, now I see that he also has a pretty big BGL file. Maybe he just deleted way more stuff he does not need...
Thalixte 2 year(s) ago
I am making a quickly done scenery of Frankfurt. It is already optimized now, and covers a little bigger area than the addon you pointed me. When finished, i will give you a link so you can test and see the difference with an optimized scenery.
Perschistence 2 year(s) ago
Wow, that would be great, thank you very much!
Thalixte 2 year(s) ago
Ok, so here it is: https://drive.google.com/file/d/1TQ8z8oGMw0Zl6BYzp7zxWHZQm65DZnKu/view?usp=sharing
Perschistence 2 year(s) ago
Thank you! I am going to try it out as soon as I can. Wow, you brought it down to 626MB. I ask myself, is the 7z format so much more efficient than zip or rar?
Thalixte 2 year(s) ago
Yes, it can be
Perschistence
Cannot execute the compress_built_package.py because when I try it, this happens: https://i.ibb.co/Jq85XnM/Another-Instance.png
When I then lower NB_PARALLEL_TASKS to just 1 it also does not work and always shows "invalid commandline".
Thalixte 2 year(s) ago
You must install compressonator for this script to work. This script only compress DDS textures on an already built package. As mentioned in the section 8 of the description (or in the README.md file in the github repo), the script do the following:
Process:
Perschistence 2 year(s) ago
Do I have to install the compressonator in any specific way? I downloaded the .exe file from your link: https://github.com/GPUOpen-Tools/compressonator/releases/download/V4.1.5083/CompressonatorCLI_x64_4.1.5083.exe renamed it to "compressonatorcli.exe" placed it at "F:\\Google-Earth-Decoder-optimization-tools" and configured the "compress_built_package.py" as following:
After executing, I encountered the problem from my screenshot.
Perschistence 2 year(s) ago
Ahh, I now see, the Compressonator-File is a Setup^^
Perschistence
Sorry for writing so many comments. The JPEG compression seems to have no impact on the final package, because in the end it is converted to a DDS file what has exactly the same size. Or is it not meant to decrease the size of the final package?
Thalixte 2 year(s) ago
The JPG compression can have an impact if you change the compression setting. But it is not a losless process. That's why i prefer to keep PNG texture files
Perschistence 2 year(s) ago
I've set the "output_texture_format" to "JPG_FORMAT" and did not change the "JPG_COMPRESSION_RATIO" to something other than 0.75. Shouldn't this have changed the file sizes?
Thalixte 2 year(s) ago
I suggest you to rollback the texture_format to PNG_FORMAT.
Perschistence 2 year(s) ago
Hmm I am now running the optimization of another project and sometimes I see this in the log:
It is running for hours now, I ask myself if I can simply ignore it or if I have to restart the whole process again? Most of the time it seems to work. At least I get a lot of JPEG files. It wouldn't be too bad if I would just be missing some textures I guess...
I would simply stay with PNG compression but my project is 10GB in size and even the package compression in the end just reduces this by 10%. I guess, there is alot of potential to save storage and maybe even RAM with JPEG. Hmm, maybe I could do the compression manually via a batch script or something if it does not work.
Thalixte 2 year(s) ago
I think there is a miseunderstood: the optimization process is not just the fact of compress jpeg files.
First of all, 10 Gb is too heavy, even for my optimization scripts. I mostly use sceneries with less than 6 Gob of data. To do so, i split my sceneries in different packages.
The, the texture optimisation uses the Lily texture Packer in order to get only one texture per tile and LOD level. If you look at your package, you will see that, for one tile and one LOD level, you have a bunch of texture files.
The optimization process will reduce the number of files by associate only one texture file with one tile and LOD level. This feature help reducing the loading time of a scenery (until SU5, but i did not test it for the moment, currently downloading the 40 Gb update).
The optimization process will also redimension the tile bpunding box in order to be efficient with the LOD managament system of the simulation. This feature helps keeping stable framerate, even with an heavy photogrammetric scenery.
The optimization process also ensure that all tiles willl be correctly built, for instance, by converting all the textures in the same format (PNG or JPG, as you want, but i advice to use PNG, because in the end, if tou use the compress_built_package script, you will get optimized texture file size).
And, at last, the optimization process will update the gltf files to make them compatible with road traffic, and collision detection.
Perschistence 2 year(s) ago
I guess it is possible to save more space with JPEG than with the scenery_optimisation.py script if I accept a degradation in quality and therefore I shouldn't use the scenery optimisation script. But I don't know what the implications for the game are because JPEG first has to get decoded and is not meant for real-time game stuff. On the other hand, it could even perform better if ram or disc bandwidth is a problem and CPUs are pretty fast in decoding JPEGs.
Thalixte 2 year(s) ago
Guess what ? With the last update, memory footprint seems lighter. So, it is maybe not necessary to lose texture quality like this
Perschistence 2 year(s) ago
Seems amazing, +35% performance in 4k and +82% in 1080p. -66% ram usage. At least that is what golem.de measured: https://www.golem.de/news/flight-simulator-im-benchmark-test-sim-update-5-laesst-performance-abheben-2107-158478.html I am going to see if it also applies to most of the mods.