MapInfo Pro

Hi, 

I have a large seamless table of just over 10,000 tiles.  All the ghx and pprc files have been created for the tiffs.

When I open it in v15 I can zoom to the entire layer and, eventually, it will draw the entire map.

When I try and do the same in v21, no matter how long I wait, MapInfo crashes.  

I'm not really looking for solutions (as zoom layering works fine) I'm more interested in why? Is it a graphics thing or a memory allocation thing?  Am I missing a setting in v21?  why would 32 bit be better than 64 bit?

Thanks 

Nick

------------------------------
Nick Hall
Mapchester LTD
Manchester
------------------------------

The key difference is that when you open a raster in v21 it keeps the file open and uses a Windows file handle. Each process only gets about 2000 file handles so if you try to open more than 2000 rasters you run out of handles. When this happens, the engine will probably crash. Also, each raster tile is actually two files - the base level TIFF and the overview PPRC. So you can only open about 1000 tiles like this before the engine runs out of puff.

The old system in v15 (which is still present in v21) opens the raster when it needs some data and then immediately closes it again, so each time you require some raster data there is some overhead involved in opening the raster, verifying the format, loading metadata etc. It is not very efficient but it works for large seamless tables.

GDAL (the open-source raster library used in v21) has a hybrid system where it maintains a pool of file handles and allocates them to rasters on demand. So it can also open a large number of rasters simultaneously - with similar performance implications.

For the best performance, you can use the raster tools to merge all the tiles into a single large raster. The best format to use for this purpose is MRR.

Another alternative is to build a GDAL virtual raster (VRT) that merges all the TIFF tiles. This will be very much like a seamless table.

The only other thing you can do is prevent both the native TIFF driver and the GDAL driver from opening the TIFF files. The TIFF files will then fall through to the old (v15) system. The PPRC files will be ignored. You do this by turning off the TIFF drivers in the backstage settings for the raster engine. The problem with that solution is that it applies to any and all TIFF files you try to display.

------------------------------
Sam Roberts
Founder, Roberts Geospatial Engineering
Australia
------------------------------

Original Message:
Sent: 10-27-2022 10:51
From: Nick Hall
Subject: Seamless table v21 Vs v15

Hi,

I have a large seamless table of just over 10,000 tiles.  All the ghx and pprc files have been created for the tiffs.

When I open it in v15 I can zoom to the entire layer and, eventually, it will draw the entire map.

When I try and do the same in v21, no matter how long I wait, MapInfo crashes.

I'm not really looking for solutions (as zoom layering works fine) I'm more interested in why? Is it a graphics thing or a memory allocation thing?  Am I missing a setting in v21?  why would 32 bit be better than 64 bit?

Thanks

Nick

------------------------------
Nick Hall
Mapchester LTD
Manchester
------------------------------

Sam,

Thanks for that.  Interesting stuff.

I'll look at the MMR option as the same data may be required for Spectrum too so it may make that easier to manage rather than seamless tables.

Regards

Nick


------------------------------
Nick Hall
Mapchester LTD
Manchester
------------------------------

Original Message:
Sent: 10-28-2022 15:57
From: Sam Roberts
Subject: Seamless table v21 Vs v15

The key difference is that when you open a raster in v21 it keeps the file open and uses a Windows file handle. Each process only gets about 2000 file handles so if you try to open more than 2000 rasters you run out of handles. When this happens, the engine will probably crash. Also, each raster tile is actually two files - the base level TIFF and the overview PPRC. So you can only open about 1000 tiles like this before the engine runs out of puff.

The old system in v15 (which is still present in v21) opens the raster when it needs some data and then immediately closes it again, so each time you require some raster data there is some overhead involved in opening the raster, verifying the format, loading metadata etc. It is not very efficient but it works for large seamless tables.

GDAL (the open-source raster library used in v21) has a hybrid system where it maintains a pool of file handles and allocates them to rasters on demand. So it can also open a large number of rasters simultaneously - with similar performance implications.

For the best performance, you can use the raster tools to merge all the tiles into a single large raster. The best format to use for this purpose is MRR.

Another alternative is to build a GDAL virtual raster (VRT) that merges all the TIFF tiles. This will be very much like a seamless table.

The only other thing you can do is prevent both the native TIFF driver and the GDAL driver from opening the TIFF files. The TIFF files will then fall through to the old (v15) system. The PPRC files will be ignored. You do this by turning off the TIFF drivers in the backstage settings for the raster engine. The problem with that solution is that it applies to any and all TIFF files you try to display.

------------------------------
Sam Roberts
Founder, Roberts Geospatial Engineering
Australia

Original Message:
Sent: 10-27-2022 10:51
From: Nick Hall
Subject: Seamless table v21 Vs v15

Hi,

I have a large seamless table of just over 10,000 tiles.  All the ghx and pprc files have been created for the tiffs.

When I open it in v15 I can zoom to the entire layer and, eventually, it will draw the entire map.

When I try and do the same in v21, no matter how long I wait, MapInfo crashes.

I'm not really looking for solutions (as zoom layering works fine) I'm more interested in why? Is it a graphics thing or a memory allocation thing?  Am I missing a setting in v21?  why would 32 bit be better than 64 bit?

Thanks

Nick

------------------------------
Nick Hall
Mapchester LTD
Manchester
------------------------------

Sam,

I am interested in your explanation here as we use different approaches for seamless map sheets:

• In MapInfo v21 we use the Seamless Manager to create seamless maps.
• But these don't work in QGIS where we have had to create separate Virtual Raster files for the same seamless sheets.

You mention that an alternative in MI v21 is to build a GDAL virtual raster (.vrt). However, my experience is that I always get the error "Invalid OGR Dataset(s)" if I try to open a virtual raster (made in QGIS) using MI v21. Is there some way around this?

Any help would be appreciated!

Cheers

Matt

------------------------------
Matthew Lee
Deep End Services
South Yarra VIC
------------------------------

Original Message:
Sent: 10-28-2022 15:57
From: Sam Roberts
Subject: Seamless table v21 Vs v15

The key difference is that when you open a raster in v21 it keeps the file open and uses a Windows file handle. Each process only gets about 2000 file handles so if you try to open more than 2000 rasters you run out of handles. When this happens, the engine will probably crash. Also, each raster tile is actually two files - the base level TIFF and the overview PPRC. So you can only open about 1000 tiles like this before the engine runs out of puff.

The old system in v15 (which is still present in v21) opens the raster when it needs some data and then immediately closes it again, so each time you require some raster data there is some overhead involved in opening the raster, verifying the format, loading metadata etc. It is not very efficient but it works for large seamless tables.

GDAL (the open-source raster library used in v21) has a hybrid system where it maintains a pool of file handles and allocates them to rasters on demand. So it can also open a large number of rasters simultaneously - with similar performance implications.

For the best performance, you can use the raster tools to merge all the tiles into a single large raster. The best format to use for this purpose is MRR.

Another alternative is to build a GDAL virtual raster (VRT) that merges all the TIFF tiles. This will be very much like a seamless table.

The only other thing you can do is prevent both the native TIFF driver and the GDAL driver from opening the TIFF files. The TIFF files will then fall through to the old (v15) system. The PPRC files will be ignored. You do this by turning off the TIFF drivers in the backstage settings for the raster engine. The problem with that solution is that it applies to any and all TIFF files you try to display.

------------------------------
Sam Roberts
Founder, Roberts Geospatial Engineering
Australia

Original Message:
Sent: 10-27-2022 10:51
From: Nick Hall
Subject: Seamless table v21 Vs v15

Hi,

I have a large seamless table of just over 10,000 tiles.  All the ghx and pprc files have been created for the tiffs.

When I open it in v15 I can zoom to the entire layer and, eventually, it will draw the entire map.

When I try and do the same in v21, no matter how long I wait, MapInfo crashes.

I'm not really looking for solutions (as zoom layering works fine) I'm more interested in why? Is it a graphics thing or a memory allocation thing?  Am I missing a setting in v21?  why would 32 bit be better than 64 bit?

Thanks

Nick

------------------------------
Nick Hall
Mapchester LTD
Manchester
------------------------------

Hi Matt,

The only VRT files I have to test are some TIFF images that are geolocated and warped using a VRT. They work in MapInfo Pro v21.

You ought to check your MapInfo Pro Raster Preferences in the backstage area and make sure that the GDAL VRT format is enabled for processing and display.

Also, make sure you load the VRT via the Open Table option and in the file "Open" dialog make sure you have selected the "Raster Image" file type filter.

If the VRT does not load then you would have to contact support to see if they can help. It is possible, for example, that you are using a later version of GDAL that generates VRT files that are not compatible with the version of GDAL used in Pro.


------------------------------
Sam Roberts
Founder, Roberts Geospatial Engineering
Australia
------------------------------

Original Message:
Sent: 11-02-2022 01:55
From: Matthew Lee
Subject: Seamless table v21 Vs v15

Sam,

I am interested in your explanation here as we use different approaches for seamless map sheets:

• In MapInfo v21 we use the Seamless Manager to create seamless maps.
• But these don't work in QGIS where we have had to create separate Virtual Raster files for the same seamless sheets.

You mention that an alternative in MI v21 is to build a GDAL virtual raster (.vrt). However, my experience is that I always get the error "Invalid OGR Dataset(s)" if I try to open a virtual raster (made in QGIS) using MI v21. Is there some way around this?

Any help would be appreciated!

Cheers

Matt

------------------------------
Matthew Lee
Deep End Services
South Yarra VIC

Original Message:
Sent: 10-28-2022 15:57
From: Sam Roberts
Subject: Seamless table v21 Vs v15

The key difference is that when you open a raster in v21 it keeps the file open and uses a Windows file handle. Each process only gets about 2000 file handles so if you try to open more than 2000 rasters you run out of handles. When this happens, the engine will probably crash. Also, each raster tile is actually two files - the base level TIFF and the overview PPRC. So you can only open about 1000 tiles like this before the engine runs out of puff.

The old system in v15 (which is still present in v21) opens the raster when it needs some data and then immediately closes it again, so each time you require some raster data there is some overhead involved in opening the raster, verifying the format, loading metadata etc. It is not very efficient but it works for large seamless tables.

GDAL (the open-source raster library used in v21) has a hybrid system where it maintains a pool of file handles and allocates them to rasters on demand. So it can also open a large number of rasters simultaneously - with similar performance implications.

For the best performance, you can use the raster tools to merge all the tiles into a single large raster. The best format to use for this purpose is MRR.

Another alternative is to build a GDAL virtual raster (VRT) that merges all the TIFF tiles. This will be very much like a seamless table.

The only other thing you can do is prevent both the native TIFF driver and the GDAL driver from opening the TIFF files. The TIFF files will then fall through to the old (v15) system. The PPRC files will be ignored. You do this by turning off the TIFF drivers in the backstage settings for the raster engine. The problem with that solution is that it applies to any and all TIFF files you try to display.

------------------------------
Sam Roberts
Founder, Roberts Geospatial Engineering
Australia

Original Message:
Sent: 10-27-2022 10:51
From: Nick Hall
Subject: Seamless table v21 Vs v15

Hi,

I have a large seamless table of just over 10,000 tiles.  All the ghx and pprc files have been created for the tiffs.

When I open it in v15 I can zoom to the entire layer and, eventually, it will draw the entire map.

When I try and do the same in v21, no matter how long I wait, MapInfo crashes.

I'm not really looking for solutions (as zoom layering works fine) I'm more interested in why? Is it a graphics thing or a memory allocation thing?  Am I missing a setting in v21?  why would 32 bit be better than 64 bit?

Thanks

Nick

------------------------------
Nick Hall
Mapchester LTD
Manchester
------------------------------