MapInfo Pro

 View Only
Expand all | Collapse all

Elevation data on Mapping Data on data.gov.uk

  • 1.  Elevation data on Mapping Data on data.gov.uk

    Employee
    Posted 05-16-2019 12:48
    Edited by Chris Jenkins 05-16-2019 12:52
    I have been ploughing through the open data on the https://data.gov.uk/ website recently.   I started downloading open lidar data for Wales, 25cm through to 2m DSM and DTM's are avilable.  I have used MIP Advanced to stitch all of this together, it's compressed it into a 16Gb DTM.  As you can see in the map below, the data capture was for flooding purposes and focuses on the coast and river network, however as a result lots of urban areas have also been captured. 


    Out of interest I started inspecting my local area and found that I have been cycling through an Iron Age Hill Fort without ever noticing it.  In the map as you can see below is an extremely clear kidney bean shaped feature, using Slope and Curvature maps derived from the DTM the feature becomes even more evident.  I soon started noticing these all over the place.  It's very revealing of the turbulent History of the area and the need for fortification against invaders of the past.  


    It is very easy to see the ear/kidney bean shape fortification mound when viewing the slope map.

    This feature is called Mynedd Y Castell and dates back to c800 BC - 84AD according to www.ancientmonuments.uk

    Detailed information on the data capture can be found here.

    Contains Natural Resources Wales information © Natural Resources Wales and Database Right. All rights Reserved.

    ------------------------------
    Chris Jenkins
    Pitney Bowes Software Ltd
    HENLEY-ON-THAMES
    ------------------------------


  • 2.  RE: Elevation data on Mapping Data on data.gov.uk

    Posted 05-17-2019 05:17
    Lovely!
    How many files did you have to handle?
    Have you found a way of handling thousands of files for merging?

    ------------------------------
    John Ievers
    CDR Group
    Hope Valley, United Kingdom
    ------------------------------



  • 3.  RE: Elevation data on Mapping Data on data.gov.uk

    Employee
    Posted 05-17-2019 05:59
    Edited by Chris Jenkins 05-17-2019 06:05
    Hi John,  I agree it is lovely!  Very interesting to be able to pan and zoom into inspect any features at that level of detail anywhere in the country.  It was approximately 21500 ascii files contained in around 260 folders.  I did the tedious task of managing this manually whilst a certain Danish MapInfo Pro expert @Peter Møller was building a little application using the raster API (which ships with MIP Advanced) to merge these automatically.  It needs some tweaking but it helped do the final merge.  Would you see such a tool as being useful in MapInfo Pro Advanced?  I am of the opinion that a tool that can search for the .asc  files in multiple folders and then merge them would be a great addition.  

    ------------------------------
    Chris Jenkins
    Pitney Bowes Software Ltd
    HENLEY-ON-THAMES
    ------------------------------



  • 4.  RE: Elevation data on Mapping Data on data.gov.uk

    Posted 05-17-2019 07:39
    Most definitely would be useful.
    I am guessing that you separated out the 2m, 1m, 50cm and 25cm coverages.
    You can't populate the file list with 5-6,000 files to then merge in one operation as it is.
    Cheers

    ------------------------------
    John Ievers
    CDR Group
    Hope Valley, United Kingdom
    ------------------------------



  • 5.  RE: Elevation data on Mapping Data on data.gov.uk

    Employee
    Posted 05-17-2019 07:52
    Yes I separated them all. I would do this with the different resolutions even with the batch process.

    ------------------------------
    Chris Jenkins
    Pitney Bowes Software Ltd
    HENLEY-ON-THAMES
    ------------------------------



  • 6.  RE: Elevation data on Mapping Data on data.gov.uk

    Posted 05-19-2019 20:36
    In MapInfo Pro 17.03 we will have a new native raster capability called MapInfo Virtual Raster (MVR). The GDAL people have a similar concept called VRT but we have our own implementation that has numerous points of difference and advantages.

    In this example, I would recommend merging all the 0.25 / 0.5 / 1.0 / 2.0 LiDAR derived DEM rasters into separate MRR's (of common resolution) and then using an MVR to bring those rasters of different resolution together at run-time. (I would add the 5.0 metre raster available for the whole of the UK to complete the picture).

    Merging all data of different resolutions together into one raster means you either compromise the resolution, or you end up with a very large raster because you have had to interpolate to the highest resolution across the entire coverage. The MRR format has a "multi-resolution tile" concept to address this, but MVR provides a more elegant solution. By not merging into one mega-raster you minimise your total storage requirement and maximise your correctness by minimising the interpolation artefacts and processing pollution that compromise your data. It is win-win and the virtual raster engine ensures you use the most appropriate resolution data for the application (whether that be rendering or processing and analysis).

    Note that in 17.03 we are neither documenting or advertising this particular feature of MVR - it is a bit of an easter egg (and there are a few others as well). However, we could provide some guidance and insight into this technology via this forum.

    ------------------------------
    Sam Roberts
    Engineer, MapInfo Pro Advanced (Raster)
    Australia
    ------------------------------



  • 7.  RE: Elevation data on Mapping Data on data.gov.uk

    Posted 05-20-2019 06:11
    I like Easter Eggs Sam
    I need to see this in action.
    First question that springs to mind is - how will it handle different colour spreads, when each model has a different min and max value?
    Only a couple of weeks to release though.

    ------------------------------
    John Ievers
    CDR Group
    Hope Valley, United Kingdom
    ------------------------------



  • 8.  RE: Elevation data on Mapping Data on data.gov.uk

    Employee
    Posted 05-20-2019 02:53
    Thanks Chris, and yes I'll add the capability to create tab files for the ASCII files when I have a few minutes to spare :-)

    ------------------------------
    Peter Horsbøll Møller
    Pitney Bowes
    ------------------------------



  • 9.  RE: Elevation data on Mapping Data on data.gov.uk

    Posted 07-03-2019 07:38
    Hi @Peter Møller any update on this? It would be very useful and timely.

    @Sam Roberts do you have any pointers on the MVR virtual raster feature? Is this now default format if I merge multi-res rasters? I've just upgrade to 17.0.3 so keen to try this out.
    ​​

    ------------------------------
    Dave Townend
    Wireless Research
    BT Group plc
    London
    ------------------------------



  • 10.  RE: Elevation data on Mapping Data on data.gov.uk

    Employee
    Posted 07-03-2019 11:28
    Edited by Chris Jenkins 07-03-2019 11:29
    Hi @Dave Townend, Peter is on annual leave, I believe @Sam Roberts is in the office​​ but is on Australian time.  I imagine he will get back to your once he is in the office.​

    ------------------------------
    Chris Jenkins
    Pitney Bowes Software Ltd
    HENLEY-ON-THAMES
    ------------------------------



  • 11.  RE: Elevation data on Mapping Data on data.gov.uk

    Posted 07-03-2019 16:16
    In the 17.03 user interface we currently only generate virtual rasters (MVR) from the "Warp" tool and from the "Virtual Raster" tool. The latter is used to combine multiple source rasters into a multi-banded raster - it is not a merge. The main purpose is to provide some extra flexibility when rendering.

    In the future we will generate MVR files from other tools - Resample, Reproject, Clip, Merge,  Align, Combine, Calculator and others like Classify and Filter. The code already supports these operations in an MVR but we have provided no documentation or support, and it ought to be considered "in beta".

    So to do a "merge" operation (like I described in an earlier post) with an MVR today requires you to write the XML file by hand. I am currently writing a series of articles on writing XML files to access the new raster rendering engine and I will finish those soon. After that, I plan to write a series of articles on virtual rasters and I will post some example XML files. Here is an MVR that I sent to Andrei recently that merges three DTM rasters (Global, UK and LiDAR) and reprojects everything into the popular visualisation CRS (EPSG 3857) so it plays nice with Bing Maps. To display this MVR in MapInfo you just load it like you would any other raster.

    <?xml version="1.0" encoding="UTF-8"?>
    <VirtualRaster>
    	<Cache2File>False</Cache2File>
    	
    	<Raster>
    		<Name>DTM</Name> 
    		<OpenOnDemand>False</OpenOnDemand>
    		<OpenBaseSupport>False</OpenBaseSupport>
    		<File>Data\SRTMV2_30as_LandOnly.mrr</File>
    		<File>Data\UK_OS_Terrain_50m_Euro.mrr</File>
    		<File>Data\WalesLiDARDTM.mrr</File>
    	</Raster>
    	
    	<RasterOperationList Name = "MergeDTM">
    		<Merge>
    			<RasterInfo>
    				<Name>DTM</Name>
    				<UnderviewMapSize>32,32</UnderviewMapSize>
    				<UnderviewTileSize>256,256</UnderviewTileSize>
    				<BaseMapSize>32,32</BaseMapSize>
    				<BaseTileSize>1024,1024</BaseTileSize>
    				<OverviewMapSize>32,32</OverviewMapSize>
    				<OverviewTileSize>1024,1024</OverviewTileSize>
    				<CoordinateSystem>CoordSys Earth Projection 10,157,"m",0</CoordinateSystem>
    				<FieldInfo>
    					<Name>DTM</Name>
    					<Type>Continuous</Type>
    					<Compression>LZMA,9</Compression>
    					<OverviewCellCoverage>Coverage_Any</OverviewCellCoverage>
    					<ValidFlagPerBand>False</ValidFlagPerBand>
    					<Transform>
    						<Type>Simple</Type>
    						<Origin>Measure(0,1),Measure(0,1)</Origin>
    						<CellSize>Measure(30,1),Measure(30,1)</CellSize>
    					</Transform>
    					<BandInfo>
    						<Name>DTM</Name>
    						<DataType>REAL4</DataType>
    						<StoreDataType>REAL4</StoreDataType>
    						<RestrictDecimals>False</RestrictDecimals>
    						<Clip>False</Clip>
    						<Transform>False</Transform>
    						<PredEncode>False</PredEncode>
    						<PredEncodeType>None</PredEncodeType>
    						<NullValueType>Mask</NullValueType>
    						<DiscreteValue>False</DiscreteValue>
    					</BandInfo>
    				</FieldInfo>
    			</RasterInfo>
    			<InterpolationNearest>False</InterpolationNearest>
    			<UnderviewInterpolation>Cubic</UnderviewInterpolation>
    			<Layer>
    				<Raster Name = "DTM" Field = "0" Band = "0"/>
    			</Layer>
    		</Merge>
    	</RasterOperationList>
    	
    	<RasterInfo>
    		<FieldInfo>
    			<BandInfo>
    				<Name>DTM</Name>
    				<Operation Name = "MergeDTM" PrimaryRaster = "True"/>
    			</BandInfo>
    		</FieldInfo>
    	</RasterInfo>
    </VirtualRaster>

    This is what it looks like - here we see the 30 arc-second resolution global raster.

    MVR Merge Global
    Then if I zoom into the Wales coastline you can see the higher resolution 50 metre UK data. You can also see the 30 arc-second raster poking out underneath along the coastline as square edges.

    MVR Merge Local
    Then zoom in further to find the DTM generated from LiDAR data which is 2 metre resolution. You can see the boundary between the 50m raster and the 2m raster. This 2m raster is actually a DSM - it has all the trees and culture in it.

    MVR Merge Detail
    The MVR file and the images are attached.

    ------------------------------
    Sam Roberts
    Engineer, MapInfo Pro Advanced (Raster)
    Australia
    ------------------------------

    Attachment(s)



  • 12.  RE: Elevation data on Mapping Data on data.gov.uk

    Employee
    Posted 07-03-2019 15:51
    @Dave Townend, thanks for reaching out.

    The answer comes in three parts:

    1. Yes, the tool I created will now create TAB files for ASC files in a single and in multiple ​folders. Grab the latest version of the tool from Github: MapInfo Raster Tool 1.6.1.

    2. Currently there seems to be a problem if your folders contain too many ASC files. At least that's our finding, or more precisely the finding of @John Ievers. If you divide the ASC files into multiple sub folders it does seem to work.

    3. The issue in 2. is related to an issue in the ASC Reader. We are working on a fix for this. I'm not entirely up-to-date on the plans for this. Maybe @Sam Roberts or @Andrei Veselov can help you out here.

    And now back to my vacation :-)
    ​​​​​

    ------------------------------
    Peter Horsbøll Møller
    Pitney Bowes
    ------------------------------