More

Change NODATA pixel value in raster to arbitrary value

Change NODATA pixel value in raster to arbitrary value


I want to change the value in all pixels that are currently 0 to 800 in QGIS 2.6.1. 0 is listed as the no-data value in Properties > Transparency. I have unchecked the No Data Value box there. Then I have tried the following.

1) Set Settings > Option > Data Sources > Representation for NULL Values = 800. Save as… to make a new .tif raster layer, WITH and WITHOUT specifying a range for No Data Values in the Save As… dialog. I thought that maybe a range from -1 to 1 would handle real-representation zeroes better. Either way, the raster returned is just like the original, with no change to the no-data (value=0) pixels.

2) Processing Toolbox > Geoalgorithms > Raster > General Tools > Raster Calculator, with expression:ifelse(lt(a,1), (800), a). Still no dice, the no-data value is now-99999.

3) Raster menu > Raster Calculator, with expression:("my layer [email protected]" >= 0) * "my layer [email protected]". Interestingly, now the no-data value is-3.40282… E+38.

4) Processing Toolbox > Geoalgorithms > Raster > Edition > Reclassify Grid Values. This tool didn't work at all. Tried [0] Single, [1] Range, and [2] Simple Table options, with and without saving to a file. Log gave a bunch of errors about being unable to load modules. The path to the .dlls is correct.

5) Processing Toolbox > Geoalgorithms > Raster > Edition > Reclassify (Simple), with range option -1 to 1. Returned a raster with the no-data value = -99999.

For the record, all tests were done with the Null Representation = 800, but I also tried several of the methods with the default representation of NULL. Results were the same. I have also exported the raster as XYZ format and looked at it in CloudCompare; the null values are being expressed as the value reported in Properties > Transparency.


You can use the r.null.to function via GRASS to transform all NULL value cells into a user-defined value, which in your case is 800:

Note: You will have to create a GRASS Mapset before you can use r.null.to as it is not available in the Processing Toolbox. Once you created a mapset, use the GRASS option to add a raster layer, select Open GRASS Tools and search the Module list:


Raster Pixel Editor

Sometimes you want to change the values of the pixels in your rasters. It’s not something you’d necessarily want to do with imagery, but when working with categorical rasters or an elevation model, it can come in handy. Here’s an example that shows how to create a barrier wall for flood prevention on the Outer Banks in North Carolina. Essentially I’ll be creating a new raster which could then be applied to a flood model.

Start with your elevation data and a shapefile that delineates where the wall would go up. Keep in mind that it needs to be a polygon for this to work. When I did this, I created a polyline shapefile and then added a 1 m buffer to it.

From the Image Analysis Window, select the elevation layer, and add a function.

In the Editor that pops up, right click on the raster layer and insert a Constant Function. The constant should be the height of the wall that you want to build plus the underlying elevation. Because I’m looking at coastal data, 25 m is fine. If I was modeling the impact of constructing a dam in a high elevation area, I would want to add the elevation to the height of the dam.

After adding the Constant Function, keep the Editor open, and add a Clip Function. Navigate to the shapefile that you created for your area of interest. Select Outside for the Type, and check the box next to Use Input Features for Clipping Geometry. Click OK.

In the Editor window, add a Local Function above the Constant Function. Use the Plus icon to add a raster from disk and select the original elevation data. Under Operations, scroll down and select Maximum (Ignore NoData). Change the extent to Union. Click OK, and you’re all set.

You can create a shapefile for any area of interest and reuse this function chain to work as a pixel editor. You’re not limited to changing the values of the pixels, you can use any of the functions that you want like filtering, statistics, band math, etc.


2 Answers 2

I don't know exactly why RasterBand.GetMaximum() and RasterBand.GetMinimum() return None . Perhaps a raster attribute table needs to be computed before GTiff can access that value on the fly?

You can do similar operations on the numpy.ndarray variable barray .

The obvious problem here is if the nodata value is either the min or max, in which case that would be returned to your displeasure. Landsat has nodata=0.

You can solve that by making a masked array and using the masked array's min and max methods.

In order for gdal's .GetMinimum() & .GetMaximum() to return values, you first have to compute statistics on the band. Otherwise, it shows up as none. At least, that's my experience with Sentinel1 radar imagery and interferograms.
Some example code.


1 Answer 1

This is a bit different than extracting for a polygon, as you want to sample every pixel touched by the line, in the order they are touched (the polygon approaches don't care about pixel order).

It looks like it would be possible to adapt this approach to use rasterio instead. Given a line read from a shapefile using geopandas or fiona as a shapely object, you use the endpoints to derive a new equidistant projection that you use as dst_crs in a WarpedVRT and read pixel values from that. It looks like you would need to calculate the length of your line in terms of the number of pixels you want sampled, this is the width parameter of the WarpedVRT .

This approach may need to be adapted further if your line is not an approximately straight line between the endpoints.

If you want to just get the raw pixel values under the line, you should be able to use a mask in rasterio or rasterize directly, for each line. You may want to use the all_touched=True in the case of lines.


Advantages/Disadvantages of the Vector Model

In comparison with the raster data model, vector data models tend to be better representations of reality due to the accuracy and precision of points, lines, and polygons over the regularly spaced grid cells of the raster model. This results in vector data tending to be more aesthetically pleasing than raster data.

Vector data also provides an increased ability to alter the scale of observation and analysis. As each coordinate pair associated with a point, line, and polygon represents an infinitesimally exact location (albeit limited by the number of significant digits and/or data acquisition methodologies), zooming deep into a vector image does not change the view of a vector graphic in the way that it does a raster graphic (see fig. 1).

Vector data tend to be more compact in data structure, so file sizes are typically much smaller than their raster counterparts. Although the ability of modern computers has minimized the importance of maintaining small file sizes, vector data often require a fraction the computer storage space when compared to raster data.

The final advantage of vector data is that topology is inherent in the vector model. This topological information results in simplified spatial analysis (e.g., error detection, network analysis, proximity analysis, and spatial transformation) when using a vector model.

Alternatively, there are two primary disadvantages of the vector data model. First, the data structure tends to be much more complex than the simple raster data model. As the location of each vertex must be stored explicitly in the model, there are no shortcuts for storing data like there are for raster models (e.g., the run-length and quad-tree encoding methodologies).

Second, the implementation of spatial analysis can also be relatively complicated due to minor differences in accuracy and precision between the input datasets. Similarly, the algorithms for manipulating and analyzing vector data are complex and can lead to intensive processing requirements, particularly when dealing with large datasets.


ArcGIS Pro Nibble tool problem

I have a wawe intensity raster with gaps (NoData), and I want to Nibble the NoData pixels from neighbouring pixel values (wawe intensity values) in ArcGIS Pro, using a separate sea water mask covering the NoData areas (pixels in dark blue). The two raster are 32 bits floating point rasters and have identical raster statistics, extents and projection. In addition all the required env. variables seems to be correctly defined. The Nibble tools performs well without any error messages. BUT the output is a raster with values from -3.4 to +3.4. The same problem occurs when I use the Focal Statistics tool. same problems with integer rasters, Nibble output 0-255. The floating output raster has the same size 2.8 GB, so there are some data in there.. I have also built new pyramids and calculated raster statistics without any improvements..

Any ideas why this is so, and how to solve this problem?

Thanks for advising me to use GRIDs.

I solved the problem (went back to ArcGIS 10.17 as I am still most familiar with that SW).

  1. First I converted all NoData wawe-pixels to 0 to allow the Nibble tool too do the interpolation (within the coastal zone area).
  2. I then used this converted layer as the input raster to the Nibble-tool.
  3. The original wawe-raster (with its NoData pixels) aim as the input raster mask. The Nibble output gave me the intended result, as visualized in the enclosed images.

Frank


Raster Data Models (model properties of uniformly spaced grid cells)

Historical Motivation

General Concept of Suitability Analysis (i.e., appropriate land use based on characteristics of land)
  • very prolific (1700 projects over career) and influential
  • helped create U.S. National Park System
  • wrote first "National Plan" advocating conservation areas as well as development
  • resource-based planning (natural characteristics should influence city form) "multiple neighborhood-based centers determined by available resources"
  • importance of parks: "the cities that are best designed have about one-eigth of their area in parks and about one acre to 75 people." (Manning 1919)
  • Differed from "City Beautiful movement" which emphasized monumental civic centers and public buildings
How to implement these ideas in a predigital world? (not easy!)
      • mapped natural resources by survey (no airphotos!)
      • used light tables and hand tracing for spatial analysis

      • characteristics might be "good" in which case overlap areas are "suitable"
      • characteristics might be "bad" in which case overlap areas are "unsuitable"
      • characteristics might be "medium good" or "pretty bad" for intermediate cases

      Manning's Ideas & Methods Revisted and Popularized in 1960s by Ian McHarg (Penn) and Carl Steinitz (Harvard)

      Why not do overlays with Vector Model?
      • "Sliver problem" doesn't scale well
        • In an analysis with dozens of layers, can spend as much time on cleanup as analysis
        • Vector data model, particularly with polygons, is complex ("simple" overlay requires many computations)
        • Raster data model is a very efficient representation inside digital computers

        Raster Data Model

        • Orthophotos (2m x 2m 0.5m x 0.5m)
          • Cell value is pixel brightness in orthophoto
          • This is the default representation
          • Has odd naming restrictions (based on Fortran): 13 characters + underscore
          • Can also use "geoTIFF" simply by specifying extension (output: this_is_my_long_descriptive_filename.tif)
          • Can view rasters in base version, but not manipulate

          Illustrating basic raster operations in ArcGIS

          • Feature-to-raster tool (in ArcToolbox) to convert Cambridge block group polygons into a raster layer
            • 100 meter grid cells
            • use median housing value for value in each raster grid cell
            • Inverse-distance-weighted average of 12 closest sales
            • getting the grid in the right place (extents. )
            • getting the cell size right (coordinates and dimensions. )
            • masking off the parts not in Cambridge

            Tuesday, December 19, 2017

            Overview of GIS

            GIS - GIS is an integrated collection of computer software and data used to view and manage information about geographic places, analyse spatial relationships and model spatial processes

            Model - A model is a simplification to describe, predict or analyse reality. It is usually done to answer a question or solve a problem.
            Issues with models are that they represent a real world problem with several assumptions and simplifications involving compromise, sub-division, reclassification, generalization and imposition of temporal limits resulting in applying subjective constraints.
            GIS was developed in 1963 by Dr. Roger Tomlison who is regarded as the "Father of GIS".
            Location data is information that describes the location and properties (attributes) of features. It may be stored as raster or vector data.
            A map project references data files but does not contain them. Ex: ArcMap, QGIS
            Map document has an extension .mxd in ArcMap
            Vector data model defines objects with definite boundaries.
            Vector geometries are represented using (x,y,z) coordinate pairs: Point, (poly)line, Polygon
            Vector data uses geographic coordinates and attribute information to locate and determine features.
            Attribute types are: Nominal, Ordinal, Interval & Ratio.
            Nominal : Refers to QUALITY of a feature, NOT QUANTITY
            Ordinal : Refers to rank
            Interval: Refers to quality measurement that is linear (Ex: Temperature)
            Ratio : Quantity measurement that is linear, but has a fixed zero point.
            Raster data model: It is a mixture of cells (PIXEL-Picture Element) organized into rows and colmns where each cell contains a value representing information. Size increases exponentially with increasing cell size.
            Uses of raster:

            • Base map - As a background for vector layers
            • Surface map - Representing changing data in a landscape
            • Thematic map - Grouping values into classes or categories
            • Primary (mesuring data directly using instruments like GPS or techniques like surveying and remote sensing)
            • Secondary (Digitizing maps from physical maps, photographs, using photogrammetry)
            • -Resolution
            • -Scale
            • -Age
            • -Author
            • -Source
            • -Position & attribute
            • -Accuracy
            • -Completeness
            • -Metadata

            Vector file formats are those that can be stored as geodatabase classes in geodatabase and shapefile, CAD Ex: shp, shx, dbf, prj, xml.
            File geodatabase- Geodatabase is a collection of geographic datasets that is easily managed and scalable depending on the intended use. - IT IS EASIER TO STORE.
            It is a collection of files in a folder that can store, query and manage spatial and non-spatial data. It is composed of:

            • -Feature classes
            • -Feature dataset
            • -Raster dataset
            • -Non-spatial tables and
            • -Toolboxes

            Feature class - It is found in geodatabase files and is a collection of vectors with set attributes, but can also refer to annotations, multipoints or multipatches.
            Raster file formats- They are saved in geodatabase as mosaic model, and tiff, jpeg, GeoTIFF, jpeg2000, DEM formats. The format determines:

            1. -Conformal projection category preserves local angles. Ex: Mercator projection -A PROJECTION CANNOT BE BOTH EQUAL AREA AND CONFORMAL
            2. -Equal area projection category preserves area
            3. -Equidistant projection category preserves scale in agiven direction
            4. -Compromise projection category involves moderate distortion of SHAPE, AREA. DISTANCE, DIRECTION & SCALE.
            1. -60 zones that are divided by North or South
            2. -Each zone is a secant cylindrical mercator projection
            3. -Standard lines are approximately 180 km to eah side of the centrall meridian
            1. -store data in a central database where users can project them as needed.
            2. -make a quick map
            3. -when distortion of shape, area and distance are irrelevant
            4. -when spatial queies based on location and distance will not be performed.
            1. -Distance queries
            2. -To measure areas
            3. -GIS analysis
            4. -Editing GIS features
            5. -Correct visualisation

            Surface area refers to 3D measurement that measures along slopes and it is always bigger than 2D surface area.
            Categorical measurement refers to comparing categories of different factors using a common criteria.
            Ex:

            1. -Equal to (=)
            2. -Greater than (>)
            3. -Less than (<)
            4. -Greater than or equal to (>=)
            5. -Less than or equal to (<=)
            6. -Not equal to (<>)
            1. -LIKE
            2. -'-'
            3. -'%'
            4. Null values
            5. -IS
            6. -IS NOT operators are used to identify NULL values
            1. -Accessing spatial data to select records that meet a set of spatial criteria Ex: Test relationship of different datasets
            2. -Intersection
            3. -Adjacency
            4. -Containment
            5. -Distance
            6. Joining data
            7. -Combining data from multiple input tables into a single output table using a common key in the table
            • Attribute join involves appending the fields of one table to those of another through a field common to both tables
            • Benefit of attribute join is that all the data does not have to be stored in one table. NON-SPATIAL DATA can be mappable
            • -To find the nearest feature
            • -Contents of a polygon &
            • -Use as a measurement tool
            1. -Cheaper and less time-intensive to produce
            2. -Wide audience (accessible by anyone with internet access)
            3. -Easier to update
            4. -Interactive
            5. -Can be used to link to related information
            • -It is a thematic map in which vector areas are distinctly colored or shaded to represent classed values of a particular phenomenon.
            1. -Equal interval, Defined interval
            2. -Quantile
            3. -Natural breaks (in ArcMap)
            4. -Petty breaks (in QGIS)
            5. -Standard deviation (How much a feature's attribute value varies from the mean)
            6. -Subjective (Manual)
            7. -Unclassified (Unique values)

            Raster classification involves:

            • -Reclassifying cells to genaralize existing trends
            • -Creating themes in raster models. Ex: Chloropleth
            • -VECTOR -Feature classes
            • -RASTER -Raster datasets (A gridded spatial data model)
            • -NON-SPATIAL -Tables made-up of rows and columns
            • -Topology is defined as a set of geographic relationships of one or more feature classes with common geometries in a geodatabase.
            • -Topology describes how features are spatially related
            • -Shared features between feature classes can be managed using topology, nodes, edges and faces. Their relationship to one another and their features can be effectively discovered and assembled.
            • -Topology provides a mechanism to perform integrity checks on associated data thereby validating and maintaining better feature representations.
            1. -Poor precision
            2. -Time consuming
            3. -Manual rescaling
            4. -Hard to make changes to analysis
            5. Plenty of error propogation
            6. Problems with overlays:
            7. -Error propogation
            8. -Computationally intensive
            9. -Sliver creation
            10. -Scale
            1. -Tools
            2. -Input variables and
            3. -Connectors
          • Accuracy is defined as the closeness to true or known value
          • Precision is defined as closeness of two or more values to each other
            1. -ACTIVE Ex: LiDAR, RaDAR
            2. -PASSIVE Ex: Visible panchromatic, Visible multispectral, InfraRed, Thermal
            1. -BAND-1 useful for mapping coastal & aerosol studies
            2. -BLUE Bathymetric mapping to distinguish soil from vegetation
            3. -GREEN Emphasizes peak vegetation to assess plant vigour
            4. -RED Discrimitaes vegetatin slopes
            5. -NIR Emphasizes biomass content and shorelines
            6. -SWIR-1 Emphasizes moisture content of soil and vegetation penetrates thin clouds
            7. -SWIR-2 Improved SWIR-1
            8. -BAND-8 Panchromatic - 15 m resolution, sharper image definition
            9. -BAND-9 Improved detection of cirrus cloud contamination
            10. -BAND-10 TIRS-1 100m resolution, thermal imaging and estimated soil moisture
            11. -BAND-11 TIRS-2 Improved TIRS-1
            1. -Surface interpretation
            2. -Hydrological analysis
            3. -Statisticl analysis
            4. -Image classification
            1. -Using an established criteria
            2. -Reclassification into common values
            3. -Assigning weights to criteria
            4. -Overlay and
            5. -Evaluation
            1. -Spatial resolution
            2. -Spectral resolution
            3. -Radiometric resolution
            4. -Temporal resolution
            1. - GIS vendors developed server based softwares
            2. - Users could publish interactive maps on the web using GIS sofwares with interactivity and performance
            3. - In 1996 mapquest launched web service, users got directions but it was slow to load
            1. - In 2005 google maps developed tiles
            2. - Tiles load faster than one big map.
            3. -Maps are prepared at multiple scales. More data presented at each scale and web map loads only tiles needed by user.

            If x is a RasterStack/Brick, a RasterBrick will be returned if na.rm=TRUE . However, if na.rm=FALSE , a RasterLayer is returned, because the values would be the same for all layers.

            If x is a Raster* object: RasterLayer or RasterBrick. Cell values represent the size of the cell in km2, or the relative size if weights=TRUE

            If x is a SpatialPolygons* object: area if each spatial object in squared meters if the CRS is longitude/latitude, or in squared map units (typically meter)


            Where Geographical Information Systems are used

            GIS is used in nearly every field nowadays because its’ large content, high quality and highly-developed software. Here is a list of all the fields using GIS:

            • Education
            • Health
            • Insurance
            • Manufacturing
            • Petroleum
            • Public Safety
            • Real Estate
            • Retail
            • Sustainability
            • Telecommunications
            • Transportation
            • Electric and Gas Utilities
            • Natural Resources
            • Government
            • Water

            Geographical Information System is used everywhere from agriculture to public health care. Many various fields are already long been using GIS. The use for GIS is very wide which allows almost every field to benefit from it.

            GIS can be used for example in scientific investigations, resource management and development planning. For retail businesses GIS enables the chance to deliberately and exactly plan where to locate the next store. Whereas marketing companies are using GIS for making decisions of whom to market and where to market.

            Scientists are using GIS for example to compare population statistics to resources, like access capacity for healthcare. Biologists are tracking animals through GIS. City, state or federal officials are planning the responses in case of natural disasters by using GIS. In this case GIS helps to locate emergency shelters and to plan which roads should be safe to use in case of natural catastrophes.

            Different engineers are using GIS in multiple purposes such as support of design, implementation, and management of communication networks. Also for the infrastructure of internet connectivity, developing the road networks or transportation infrastructure.

            Real-world example:

            When hurricane Katrina happened there was alarming need for proper disaster management and tools for it. This disaster has seen by many as the first time when GIS was used as a tool for disaster management. GIS technology helped to share a great deal of data about the streets which were and were not accessible. GIS also gave the information where everywhere the flooding had reached. The effort of GIS data transmission was helping the whole situation and it gave relief efforts.

            There are no limits existing for the kind of information is possible to analyze using GIS.