MapInfo Pro

 View Only
  • 1.  Ordinance Survey Data - Geopackage Problems

    Posted 03-14-2024 06:33
    Edited by Ryan Cook 03-14-2024 07:03

    Hi,

    We've recently got access to the osdatahub premium packages where you can download data in gpkg format. Our trouble is, the data is just so large we can't seem to work with it efficiently. Even when we choose a small area of interest, rather than the whole of GB (which is really what we need), I am encountering practically unmanageable file sizes.  

    1. It is incredibly slow to run SQL/MapBasic queries/programs against the TAB linked to the gpkg (aka, one has simply 'opened' the gpkg in MapInfo). I have left a simple spatial query running for two days with no result. Navigating the map layer is also painfully slow. 
    2. 'Converting' the file to native tab (aka, one opens the gpkg and then delinks it by Saving Copy As) seems to speed up navigation and queries/programs ran against it, but I'm ending up with 20G DAT files for just a county of data. 

    Is anybody else having difficulty with OS data? 
    Is there something I should be doing when handling gpkg files? 
    Are there any tricks to getting tables into more manageable sizes?

    On that last point, even stripping the fields down takes an absolute age. In the end, it feels like something is off; we are heavy users of MapInfo and nothing on our drives comes anywhere near the 20GB file sizes or the slow running we are experiencing with the OS data. One of our client countries is Australia and we work effortlessly with the entire Australian road network which comes in at under 1GB. That a counties' worth of land sites in the UK is 20GB is peculiar. I appreciate file sizes are not necessarily related to geographic area sizes, and of course number of records and object complexity will come into it, but in the end what is the point of data tables that are too big to use?

    Any advice would be appreciated.  



    ------------------------------
    Ryan Cook
    ORH LTD
    ------------------------------



  • 2.  RE: Ordinance Survey Data - Geopackage Problems

    Employee
    Posted 29 days ago

    Hi Ryan

    Is that Geopackage file available for download or is it only for organizations with a "premium license"?



    ------------------------------
    Peter Horsbøll Møller
    Principal Presales Consultant | Distinguished Engineer
    Precisely | Trust in Data
    ------------------------------



  • 3.  RE: Ordinance Survey Data - Geopackage Problems

    Posted 23 days ago

    Hi Peter, it's via a premium licence aye. 



    ------------------------------
    Ryan Cook
    ORH LTD
    ------------------------------



  • 4.  RE: Ordinance Survey Data - Geopackage Problems

    Posted 24 days ago
    Edited by John Ievers 24 days ago

    Hi Ryan

    Ordnance Survey do pack a lot into the gpkg files. Which datasets are you using? The new NGD data has a lot of interesting data and certainly more attribute "heavy" than geometry "heavy".

    I don't use the MapInfo gpkg option to open these files any more - I use the OGR option, as this performs better and has more options to save as a MapInfo table

    If that helps....



    ------------------------------
    John Ievers
    CDR Group Limited
    Hope Valley, United Kingdom
    ------------------------------



  • 5.  RE: Ordinance Survey Data - Geopackage Problems

    Posted 23 days ago

    Hi John, hmm - not familiar with that method. Will have a rummage about! Thanks.



    ------------------------------
    Ryan Cook
    ORH LTD
    ------------------------------



  • 6.  RE: Ordinance Survey Data - Geopackage Problems

    Posted 22 days ago

    Hi Ryan,

    I had something similar with a Geopackage. I think the MI import sets all Character fields as 256 Chars. This can really add to the size of DAT file. I exported top mif - edited the field list and imported into MI and it returned to something more manageable and performed better. Maybe that is the reason it was so big for you?



    ------------------------------
    Martin Hodder
    Managing Consultant
    Higher Mapping Solutions (OEM)
    ------------------------------



  • 7.  RE: Ordinance Survey Data - Geopackage Problems

    Posted 21 days ago

    Hi Ryan,

    since you had another issue with operating on files located on a network drive: Is this Geopackage-File you're talking about located on a network drive?

    If so then you might know, that to evaluate any spatial query, MapInfo has to pull ALL the geometries of your file over the network to do the query evaluation locally. It's a desktop GIS-Software using in > 90% of its functionality only one cpu core, - even that should be considered when working with data of that size. Maybe a Database server that's made for working on this data amounts and a functionality to do those (spatial) queries on the server itself, for example PostgreSQL/PostGIS that can be accessed via MapInfo is the better storage for you than a Geopackage-File. BUT even on a database details in generating a query expression can make an extremly huge difference in processing time, - so even that might be a topic to look after when solving query-performance problems, like in your case.

    best regards

    SH




  • 8.  RE: Ordinance Survey Data - Geopackage Problems

    Posted 15 days ago

    Martin, 

    Yes, that sounds like something that would cause considerable slowdown. I will be sure to check the structure next time I pull something in from OS. 

    Ryan



    ------------------------------
    Ryan Cook
    ORH LTD
    ------------------------------



  • 9.  RE: Ordinance Survey Data - Geopackage Problems

    Posted 15 days ago

    Stefan,

    Thanks for the suggestion; I will have to explore alternative applications. 

    R



    ------------------------------
    Ryan Cook
    ORH LTD
    ------------------------------