1) Presentation:
The archive contains four python scripts:
- scenery_optimisation.py: optimize Google Earth Decoder scenery (textures, Lods, CTD fix)
- fix_tiles_altitudes.py: fix wrong tiles altitudes that can occure when moving tiles on the x-y axis
- update_objects_LODs.py: update the LOD levels for the tiles of a scenery based on an array of minsizes
- merge_sceneries: merge all the files from a source Google Earth Decoder scenery to a destination Google Earth Decoder scenery
2) Prerequisites:
- Blender 2.83 (https://download.blender.org/release/Blender2.83/blender-2.83.9-windows64.msi). The blender version is important
- Node js (https://nodejs.org/dist/v14.15.1/node-v14.15.1-x64.msi)
- MSFS Toolkit: https://www.fsdeveloper.com/forum/resources/blender2msfs-toolkit.256/download
- Lily texture Packer: https://gumroad.com/l/DFExj
3) scenery_optimisation script:
3.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery).
It bakes the tiles texture files in order to reduce the number of the files in the final package, and reduce the scenery loading time.
It fixes an issue with the Google Earth Decoder tool, that breaks the LOD management system.
Look at this picture: https://flic.kr/p/2k3YCrd
This picture shows a wireframe representation of a big scenery created from the Google Earth Decoder tool. You can see that the scenery tiles are dark gray, which means there are a lot of vertices on each tile, because all the tiles are on the max level of detail (which is 19 here). The result is that this scene, from this point of view, displays more than 80M of vertices, which bottlenecks the GPU (and the CPU).
Normally, we should have this: https://flic.kr/p/2kayM52. In this case, we can see that the tiles have the correct LOD levels, and it leads to a reduced number of vertices (20M in this case, but it can be changed according to the LOD levels that are set in the optimisation script (see chap. 3.3 - Configuration)).
The problem here is that the Google Earth Decoder gives all the tiles the same origin, and changes the bounding box to go from this origin point to the last mesh vertice point.
Another way to see this problem, is to use the new MSFS SDK Debug LOD feature, that displays the current lod level of each tiles, and the size of the bounding sphere: https://flic.kr/p/2kautrR
The more the green or blue is the color, the more detailed is the tile. The more the yellow or red is the color, the less detailed is the tile.
Here, we can see that the more detailed tiles are the one that are far from the camera, which is the exact opposite result than the one we want to obtain to optimize our scenery framerate.
Now, if each tile has its own origin point, and has a bounding box corresponding to the real tile size, we obtain those results: https://flic.kr/p/2kayiyp and https://flic.kr/p/2kayiAo, which are far better from a LOD point of view, and better preserve the framerate.
This is the fix that the script applies to the tiles: it gives each tile its own origin point, and resizes the bounding boxes according to the real tile size.
The script also changes the LOD levels to better suit the LOD management system, and allows (if configured) to convert texture files into jpg format, in order to reduce the texture file size (but with a possible loss in the texture quality).
The script also applies ASOBO extension tags to the gltf files, in order to enable the road management and the collisions. It also fixes texture flickering issues.
The script automatically removes the orphaned scenery object xml files (scenery object xml files that do not have associated gltf and/or bin files)
3.2) Installation:
Just put the scenery_optimisation.py script, and the retrievepos.js script in a folder of your choice.
3.3) Configuration:
Change the following settings, according to your project:
- bake_textures_enabled: tells the script to optimize the textures by baking all the textures corresponding to the min Lod levels of the tiles (default is True). For instance, if you have a gltf file for a tile that is named 30604141705340627_LOD00.gltf, all the texture files corresponding to this tile and this LOD level (all the texture files that start with 30604141705340627_LOD00) will be baked into one single texture
- projects_folder: the parent folder that contains your sceneries
- project_name: the name of your project
- node_js_folder: the folder that contains the node js script that retrieves the Google Earth coords
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- target_lods: an array representing the minsize values per LOD, starting from a minLod of 17 (from the less detailed lod to the most detailed)
- project_file_name: the name of the xml file that embeds the project definition (by default, project_name.xml or author_name+project_name.xml)
- scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml)
- package_definitions_file_name: the name of the xml file that embeds the package definitions (by default, project_name.xml or author_name+project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
- output_texture_format: format of the final texture files (values are PNG_FORMAT, JPG_FORMAT, default is PNG_FORMAT)
- JPG_COMPRESSION_RATIO: if you choose the jpg format for the output texture files, indicates the compression ratio
3.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the scenery_optimisation python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
3.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- if no other backup exists, backup the scenery modelLib files into a backup folder, inside the project folder
- if necessary, install the node Js xhr2 module (npm install xhr2)
- install the PIP and Pillow libraries to allow texture format conversion
- if png texture files output format is selected, convert all remaining jpg texture files to PNG, then remove jpg files
- if jpg texture files output format is selected, convert all remaining png texture files to JPG, then remove png files, and compress all jpg texture files
- retrieve objects positions from Google earth, via the Google Earth API, and put the results in .pos files corresponding to the scenery tiles
- update the tiles position, using those .pos files
- place all tiles objects into corresponding sub folders, in order to group the objects corresponding to the same tile
- update the LODs of the scenery tiles, according to the target_lods defined in the configuration settings
- optimize the tiles, by baking the textures corresponding to LOD levels of the tiles, and by changing the bounding box of the tiles, in order to optimize the LODs
- applies ASOBO extension tags to the gltf files, in order to enable the road management and the collisions
- fix gltf doublesided attributes, in order to remove texture flickering issues
- removes the orphaned scenery object xml files (scenery object xml files that do not have associated gltf and/or bin files)
- automatically rebuild the scenery package, if the MSFS SDK is correctly installed, and MSFS 2020 is not running
4) fix_tiles_altitudes script :
4.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/).
When using the scenery_optimisation script, there are chances that the resulting tiles are slightly decaled in the x and/or y axis.
This decal can easily be fixed by going into the MSFS SDK, open the project, click on the "Save scenery..." button to order the scenery objects by type, then select all the tiles (not the lights, rectangles or polygons, just the tiles) and move them to the appropriate location. But this move can break some tiles altitude.
To fix it, save your scenery after moving the tiles to their appropriate location, then close your project and MSFS 2020, and run the fix_tile_altitudes script. Once the script has finished running and rebuidling your project, reopen the project in the MSFS SDK. The tiles position and altitude should now be correct.
4.2) Installation:
Just put the fix_tile_altitudes.py script, and the retrievepos.js script in a folder of your choice.
4.3) Configuration:
Change the following settings, according to your project:
- projects_folder: the parent folder that contains your sceneries
- project_name: the name of your project
- node_js_folder: the folder that contains the node js script that retrieves the Google Earth coords
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- project_file_name: the name of the xml file that embeds the project definition (by default, project_name.xml or author_name+project_name.xml)
- scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml)
- package_definitions_file_name: the name of the xml file that embeds the package definitions (by default, project_name.xml or author_name+project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
- fix_with_googleEarthDecoder_data: if set to True, tells the script to use the backup of the old objects.xml data (the one produced by the Google Earth Decoder tool) to fix tiles altitude. If set to False, tells the script to directly retrieve tiles altitude from the Google Earth API (default is True)
4.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the fix_tile_altitudes python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
The fix comes with two methods: the default one uses the backup of the old objects.xml data (the one produced by the Google Earth Decoder tool)the other method tries to retrieve the altitude directly based on the Google Earth data (using the Google Earth API)
4.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- if necessary, install the node Js xhr2 module (npm install xhr2)
- retrieve objects positions from Google earth, via the Google Earth API, and put the results in .pos files corresponding to the scenery tiles
- update the tiles altitude, using the backup objects.xml file, if fix_with_googleEarthDecoder_data is set to True, or using the .pos files generated by the Google Earth API, if fix_with_googleEarthDecoder_data is set to False
- automatically rebuild the scenery package, if the MSFS SDK is correctly installed, and MSFS 2020 is not running
5) update_objects_LODs script:
5.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/).
This script automates the process of changing the LOD levels for all the tiles of a scenery.
5.2) Installation:
Just put the update_object_LODs.py script in a folder of your choice.
5.3) Configuration:
Change the following settings, according to your project:
- projects_folder: the parent folder that contains your sceneries
- project_name: the name of your project
- target_lods: an array representing the minsize values per LOD, starting from a minLod of 17 (from the less detailed lod to the most detailed)
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- project_file_name: the name of the xml file that embeds the project definition (by default, project_name.xml or author_name+project_name.xml)
- scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml)
- package_definitions_file_name: the name of the xml file that embeds the package definitions (by default, project_name.xml or author_name+project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
5.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the update_object_LODs python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
5.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- update the LOD levels for each tile in the tile_object_name.xml files
- automatically rebuild the scenery package, if the MSFS SDK is correctly installed, and MSFS 2020 is not running
6) merge_sceneries script:
6.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/).
This script automates the process of merging a Google Earth Decoder source scenery into another Google Earth Decoder scenery.
6.2) Installation:
Just put the merge_sceneries.py script in a folder of your choice.
6.3) Configuration:
Change the following settings, according to your project:
- projects_folder: the parent folder that contains your sceneries
- src_project_name: the name of the scenery that you want to include in the final scenery
- dest_project_name: the name of the final project that should include both sceneries
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- src_project_file_name: the name of the xml file that embeds the source project definition (by default, src_project_name.xml or author_name+src_project_name.xml)
- dest_project_file_name: the name of the xml file that embeds the destination project definition (by default, dest_project_name.xml or author_name+dest_project_name.xml)
- src_scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml) for the scenery that you want to include in the final scenery
- dest_scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml) for the final project that should include both sceneries
- src_package_definitions_file_name: the name of the xml file that embeds the source package definitions (by default, src_project_name.xml or author_name+src_project_name.xml)
- dest_package_definitions_file_name: the name of the xml file that embeds the destination package definitions (by default, dest_project_name.xml or author_name+dest_project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
6.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the update_object_LODs python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
6.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- backup the modelLib files of the final project that should include both sceneries, into a backup subfolder, so called merge_sceneries
- copy all the xml files, gltf files, bin files and texture files from the source scenery to the destination scenery, overwritting the existing ones
- update the destination scenery scene xml file (objects.xml by default) to change the tiles guid corresponding to the source scenery tiles
- add the guid for the source tiles that does not exist in the destination scenery
- automatically rebuild the scenery package, if the MSFS SDK is correctly installed, and MSFS 2020 is not running
7) clean_package_files script:
7.1) Presentation:
This script applies the CTD issues fix (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/).
This script automatically removes unused package files (.gltf, .bin and texture files), aka the files that are linked with tiles that have been removed from the scene. It appears that, if this cleaning is not done, the resulting package keep those files.
7.2) Installation:
Just put the clean_package_files.py script in a folder of your choice.
7.3) Configuration:
Change the following settings, according to your project:
- projects_folder: the parent folder that contains your sceneries
- project_name: the name of your project
- fspackagetool_folder: the folder that contains the fspackagetool exe that builds the MSFS packages
- project_file_name: the name of the xml file that embeds the project definition (by default, project_name.xml or author_name+project_name.xml)
- scene_file_name: the name of the xml file that embeds the tile descriptions (by default, objects.xml)
- package_definitions_file_name: the name of the xml file that embeds the package definitions (by default, project_name.xml or author_name+project_name.xml)
- author_name: the name of the author of the scenery
- build_package_enabled: enable the package compilation when the script has finished (default is True)
7.4) Usage:
Open Blender in administrator mode.
Go in the Scripting view, then click on the Open icon, and choose the clean_package_files python script.
When the configuration is done, open the Blender system console Window (Window => Toggle System Console). Then, run the script.
7.5) Process:
- rename the modelLib folder, in order to fix CTD issues (see https://flightsim.to/blog/creators-guide-fix-ctd-issues-on-your-scenery/)
- browse the scenery object xml files in the object folder
- check the presence of the corresponding guid in the scene xml file
- if the guid is not found in the scene xml file, remove the files associated with this scenery object xml file (.gltf, .bin, and texture files), in order to reduce the resulting package size
242 Comments
huha001
Newby Question on Methodology:
first of all I would like to thank you very much for your scripts which made it very easy for me to generate my first scenery and made a 22 year old dream in the flightsimulator true to fly over my own house and recognize it :)
Looking on generating better quality sceneries I do have two questions on the general methodology after having applied your scripts:
1) I understand how to use a polygon in fs2020 and eliminate microsoft landscape. Is there a way to use a polygon to cut out google 3d objects from the newly generated landscape, as the google extract tool Earth2MsfsWPF allows only rectangles? I would like to cut out some of the google trees/landscapes and use the microsoft trees for that at the boundaries to get better transitions.
I have seen that you can import/export .gltf files in blender, but there may be a simpler way and I did not figure out how to assign this to the actual coordinates and may be put into a script.
2) Is there an automated way to generate the polygons. If i generate a polygon in fs2020 i do see 4 new files in <project>\PackageSources\scene
shapes.dbf, shapes.shx, shapes.shp and shapes.dbf but do not understaned the formats. Is there any documentation/link available on this?
1) Unfortunately, there is no possibility to select anything but rectangles with the Google Earth Decoder tool. My tip is to retrieve not too big tiles (with a min LOD of 17, you have tiles that cover a less large area than with a min 16 LOD). Than, i remove some tiles in the MSFS SDK in the border of the scene, and i cover the area with an exclusion and terraforming polygon. When you have made a polygon with the MSFS SDK, you can easily add new point, and make the shape that you want (and a shape that perfectly follow the borders of the scenery).
As i mentionned in a reply, if you want to edit the tiles in Blender, ensure to do it before the optimization process, as the Google Earth Decoder tool allow to loads the tiles in Blender in ar correct position. To sum up: with the Google Earth Decoder tool, tiles are correctly positionned in Blender, but their size is not correct for the simu. With my tool, the tiles are not correctly positionned in Blender, but their size is correct and optimized for the simu.
2) I do not myself take a look at the polygon format. Maybe it could be a great evolution to automatically generate a polygon that covers the area, but i prefer myself to create them in the MSFS SDK, and edit them as i want.
Bruchpilot80
is there anybody able to do a youtube-tutorial, how to use this tool? for people like me, english is not the native language, and without experiance with blender, it looks shocking, how many steps there are to do. i would really love to use your tool, because i made several scenerys, but i dont now how complicated it is to start with your tool. the videotutorial for the decoder was very usefull for me, when i had problems with next step. something like this would be great. thx for your work!
disastorm
Do you happen to know if there was any changes in the last world update that would affect the performance of this? I feel like that the stuff I had generated from this some time ago started performing worse after the update, although I suppose it could have been related to having a different flight plan or some caching thing or something.
cyrano
I'm struggling with understanding the LODs settings of the scripts. I'm trying to generate a less detailed mod. So using earth2MSFS I downloaded an area at 17-19 LODs instead of my usual 17-20.
Then I launched the scenery optimization script which runs perfectly. But when it comes to building the package, I take 0s and nothing is processed by msfs and there is no package.
I have tried the same on various areas including ones for which I have already a package at 17-20...
Any idea on why I can't build packages at 17-19 (I also tried 16-19 but it is the same). Is there a setting to change in the script when using 17-19?
* shows the scripts settings
* shows the msfs package builder message
* shows the folder directory with no package folder
* shows the first few files inside the modellib folder
I have also checked in that folder that there is no orphaned files
https://1drv.ms/u/s!An4eU-aBYbBGgYrPEeU-q1WbZciYkF4?e=u3t3lM
It looks like the builder does not find any "asset groups" whatever that means :(.
Once added back it works perfectly. Thanks for the support and pointing me in the right direction.
Voyager
Can't wait for them to finalize and release the DX12 version of MSFS...we'll ADORE the mesh shaders and sampler feedback !! :-D
It's because they will primarily bring the BEST features that DX12 has to offer, but that are not focused on visuals - those are: variable rate shading, mesh shaders, sampler feedback, etc.
I work with such technologies since 2016.
It's the reason 3080 has just 10GB of VRAM...but it is an insanely fast VRAM with the PAM4 analog signaling, the first of its kind...that's only been used only in laboratories and supercomputers up until now !
When they released their 2000 GPU series, the first "RTX" generation, nVidia actually made a small page where they presented all that cool stuff that Microsoft's DX12 Ultimate brings, but to a wider audience so that everyone can understand it.
And nVidia and AMD partnered with Microsoft around it, hence the name "Ultimate" - cuz it will finally unify all platforms.
For example, we have nVidia DLSS, and now AMD is going out with their Smart Super-Sampling - but both use Microsoft DirectML that is aone of the flagship features of DirectX12, both use machine learning and dedicated hardware cores for that (there are repos on GitHub for both Tensor DirectML and the AMD's solution, etc).
You notice how they are a bit silent whenever we ask them "will we get DLSS in DX12 update ??".
That's because we won't get DLSS...not only DLSS...but the full DirectML so that it will work on both nVidia and AMD hardware.
The same goes for these other features.
They designed that page something like 2 years ago.
Please check it out, it's 2 minutes, you'll like it:
https://www.nvidia.com/en-eu/geforce/technologies/directx-12-ultimate/
Those APIs (DX12 and vulkan) are great, performance wise. But they are also some kinds of beast for devs, especially the ones that were used to work on dx11. It is up to the devs now to manage the memory resources and the resource sharing between threads and shaders. This is the price to pay to have better performance. Also, consider that those APIs are especially designed to optimize the GPU architecture, especially the parallelization of tasks (because GPU are specifically designed to treat a bunch of parallelized tasks, in the shaders).
However, as for me, the way they plan to use DX12 is the right way: focusing on dx12 optimizations and resource management is far more important than bringing new effects introduced by the API.
Well you worked on the DirectX code for Reshade, you know it all.
I am fully counting on them to properly implement all of those things.
This simulator is actually one huge experiment by Microsoft, partnever with everyone, so that they bring the industry into the new realms.
By the way...have you seen...the Unreal Engine 5 demo...oh man...just wtf, like...we're looking at those cinematic grade textures and high-poly models, no optimization, just trillions of triangles there we go eeehaaa !
We're watching that and one colleague just jumped and started shouting "HOOOOW !!?? HOW do they do THAT !!??? WHYYYY GOD WHY !!??".
Hahaha.
Yeah, I suppose it's the result of all those things like DirectML, DirectStorage, etc. When a GPU has a crazy fast access to NVMe, CPU and RAM have the same with NVMe and GPU and vice versa...the internals of our machines will look different in a few years, very different, what do you think ? :-)
I mean, if we sum it all up - this simulator just YEARNS for all those features !
:-)
Lash123456
Hi... I get this problem... happens very rarely, but when Blender has built the package with scenery optimization, it sometimes fails to add the business, layout and manifest json files
Would you know why that might be?
Thanks
RiverSkyHigh
My position folder always remains empty now, despite receiving no error messages on the scripts. What might be causing this? I have the retrievepos.js properly directed.
Also, would it be possible to receive some clarification on the merge_sceneries script? What exactly is the meaning between "src_project" and "dest_project." Would it be possible to get an example on what the script structure would look like for this? Thanks!
The merging script helps at merging tiles from a source project to a destination project. For instance, i use it in this case: i retrieve a large area with 17-19 tiles. Then, i want to add more detailed tiles in some specific areas within the first area i retrieved. So i create a new project, retrieve and optimize the more detailed tiles. Then, when i have finished optimizing both the two projects, i use my merging script in order to replace the 17-19 tiles with the more detailed 17-20 tiles in the first project i made. In this case, the src_project is the second project i made (the one with the more detailed tiles), and the destination project is the one with the large area.
The merging script can also be used to make one big project by grouping the tiles from another sub projects.
RadoBG
Is there any tutorial that I can follow including this optimisation tools? I just keep getting error that says I havent installed xhr2 :(.
babulu73
Thank you so much. Your tool is just great (and i refer to it on every scenery i publish). You seem to have a great knowledge of blender and the SKD: Is it possible for you to create a script, to import a Earth Decoder optimized scenery into blender? Sometimes there are so many useless parts of a town, just creating thousands of unused polys. Or you have a litle forest in your scenery, that you would like to cut out, because it always looks rubbish if you blend it with the MSFS trees. Is there a way, to import Earth Decoder models into blender and crop them? Thank you very much!
Unfortunately, once the scenery has been optimized, it is not possible to import all the tiles at the same time, because they will share the same coords and bounding box. You can only import them individually in this case.
In Blender, File > Import > Gltf 2.0
comolas
Many thank you for this update
I've been trying to make a scene for 10 days, but I can't finish it because too many tiles are not in their place
I even tried to cut it in 10 zones to merge them afterwards but impossible anyway
Lash123456
I am getting this error again, but for the life of me dont know why.
https://ibb.co/5MY9Cdw
I had this before but I created a .pos file with [0,0,0] in it and this seemed to have fixed it.
But now this error has came up again despite me having this file... any ideas why its doing this now when all has been fine?
Thanks
https://ibb.co/mBcmgc1
This is one that worked in my current project
Doing part 2, it suddenly doesnt work.. I have not changed anything and always copied that .pos file into my new projects.. baffled why it no longer works when it has been fine previously.
I have been doing this, to put my created .pos file with [0,0,0] in it... all has been fine doing it this way.
If you are able to refer to my post on the 20th December
https://ibb.co/MGf6y4t
I think it downloaded with missing/corrupt date.. all ok now.
Thanks again for your responses, help and amazing scripts...
Pay day soon for me, so I will be sure to make a donation of good will for your excellent work!
MmAaMmOo
I was looking for that! thank you!