Applier (hubdc.applier)

Basic tools for setting up a function to be applied over a raster processing chain. The Applier class is the main point of entry in this module.

See Applier Examples for more information.

class hubdc.applier.Applier(controls=None)

Bases: object

This class is the main point of entry in this module. For detailed usage examples see Applier Examples.

Attributes and properties are:

• inputRaster ApplierInputRasterGroup object containing all input rasters
• inputRasterArchive ApplierInputRasterArchiveGroup object containing all input raster archieves
• inputVector ApplierInputVectorGroup object containing all input vectors
• outputRaster ApplierOutputRasterGroup object containing all output rasters
• controls ApplierControls object containing all input rasters
• mainGrid Grid object

__init__(controls=None)

Parameters:controls – an ApplierControls object

apply(operatorType=None, operatorFunction=None, description=None, overwrite=True, *ufuncArgs, **ufuncKwargs)

Applies the operator blockwise over a raster processing chain and returns a list of results, one for each block.

The operator must be a subclass of ApplierOperator and needs to implement the ufunc() method to specify the image processing.

For example:

class MyOperator(ApplierOperator):
    def ufunc(self):
        # process the data

applier.apply(operator=MyOperator)

or:

def my_ufunc(operator):
    # process the data

applier.apply(operator=my_ufunc)

For detailed usage examples see Applier Examples.

Parameters:

• description – short description that is displayed on the progress bar
• ufuncArgs – additional arguments that will be passed to the operators ufunc() method.
• ufuncKwargs – additional keyword arguments that will be passed to the operators ufunc() method.

Returns:

list of results, one for each processed block

grid()

Returns the output Grid object.

class hubdc.applier.ApplierControls

Bases: object

Class for controlling various details of the applier processing.

__init__()

deriveExtent(grids, projection)

Derives the extent from the given Grid’s in the given Projection.

deriveGrid(inputRasterGroup)

Derives the grid from the given ApplierInputRasterGroup’s.

deriveProjection(grids)

Derives the projection from the given Grid’s.

deriveResolution(grids)

Derives the resolution from the given Grid.

setAutoExtent(autoExtent='intersection')

Define how the grid extent is derived from the input rasters. Possible options are listed in AutoExtent.

setAutoResolution(autoResolution='minimum')

Define how the grid resolution is derived from the input rasters. Possible options are listed in AutoResolution.

setBlockFullSize()

Set the block size to full extent.

setBlockSize(blockSize=Size(x=256, y=256))

Set the processing block x and y size. Pass an int defining x and y size to be the same, or a tuple (int, int) defining x and y size separately, or a Size object.

setExtent(extent=None)

Set the applier extent. Pass a (float, float, float, float) tuple defining the extent as (xMin, xMax, yMin, yMax), or an Extent object, or None (default) to derive the extent from the input rasters.

setGDALCacheMax(bytes=104857600)

For details see the GDAL_CACHEMAX Configuration Option.

setGDALDisableReadDirOnOpen(disable=True)

For details see the GDAL_DISABLE_READDIR_ON_OPEN Configuration Option.

setGDALMaxDatasetPoolSize(nfiles=100)

For details see the GDAL_MAX_DATASET_POOL_SIZE Configuration Option.

setGDALSwathSize(bytes=104857600)

For details see the GDAL_SWATH_SIZE Configuration Option.

setGrid(grid=None)

Set the applier grid defining the extent, resolution and projection. Pass a Grid object, or None (default) to derive the grid from the input rasters.

setNumThreads(nworker=None)

Set the number of pool worker for multiprocessing. Set to None to disable multiprocessing (recommended for debugging). Set to -1 to use all CPUs.

setNumWriter(nwriter=None)

Set the number of writer processes. Set to None to disable multiwriting (recommended for debugging).

setProgressBar(progressBar=None)

Set the progress display object. Default is an CUIProgress object. For suppressing outputs use an SilentProgress object

setProjection(projection=None)

Set the applier projection. Pass a Well-Known Text (WKT) projection string, or an Authority EPSG ID as int, or a Projection object, or None (default) to use the projection that is shared by all input rasters.

setResolution(resolution=None)

Set the applier resolution. Pass a float defining x and y resolution to be the same, or a (float, float) tuple defining x and y resolution separately, or a Resolution object, or None (default) to derive the resolution from the input rasters.

setWriteENVIHeader(createEnviHeader=True)

Set to True to create additional ENVI header files for all output rasters. The header files store all metadata items from the ENVI domain, so that the images can be correctly interpreted by the ENVI software. Currently only the native ENVI format and the GTiff format is supported.

class hubdc.applier.ApplierDefaults

Bases: object

Defauls values for various settings used inside an applier processing chain.

class GDALEnv

Bases: object

cacheMax = 104857600

disableReadDirOnOpen = True

maxDatasetPoolSize = 100

swathSize = 104857600

class GDALWarp

Bases: object

errorThreshold = 0.0

memoryLimit = 104857600

multithread = False

autoExtent = 'intersection'

autoResolution = 'minimum'

blockSize = Size(x=256, y=256)

nworker = None

nwriter = None

writeENVIHeader = True

class hubdc.applier.ApplierIO(filename)

Bases: object

Base class for io items. For internal use only.

__init__(filename)

filename()

Returns the filename.

operator()

Returns the operator object.

setOperator(operator)

Pass a handle on the operator object.

class hubdc.applier.ApplierIOGroup

Bases: object

__init__()

group(key)

operator()

setGroup(key, value)

setOperator(operator)

class hubdc.applier.ApplierInputRaster(filename)

Bases: hubdc.applier.ApplierIO

Class for handling input raster dataset.

__init__(filename)

Creates an instance from raster stored at filename.

array(overlap=0, resampleAlg=<Mock name='mock.gdal.GRA_NearestNeighbour' id='140313600122064'>, noDataValue=None, errorThreshold=0.0, warpMemoryLimit=104857600, multithread=False, grid=None)

Returns image data as 3-d numpy array of shape = (zsize, ysize, xsize), where zsize is the number of bands.

Parameters:

• overlap – the number of pixels to additionally read along each spatial dimension
• resampleAlg – GDAL resampling algorithm, e.g. gdal.GRA_NearestNeighbour
• noDataValue – explicitely set the noDataValue used for reading; this overwrites the noDataValue defined by the raster itself
• errorThreshold – error threshold for approximation transformer (in pixels)
• warpMemoryLimit – size of working buffer in bytes
• multithread – whether to multithread computation and I/O operations
• grid – explicitly set the Grid, for which image data is returned

bandArray(indicies, overlap=0, resampleAlg=<Mock name='mock.gdal.GRA_NearestNeighbour' id='140313600122064'>, noDataValue=None, errorThreshold=0.0, warpMemoryLimit=104857600, multithread=False)

Returns a band subset of the image data as 3-d numpy array of shape = (zsize, ysize, xsize), where zsize is the number of indicies.

Parameters:

• indicies – list of band indicies
• overlap – the number of pixels to additionally read along each spatial dimension
• resampleAlg – GDAL resampling algorithm, e.g. gdal.GRA_NearestNeighbour
• noDataValue – explicitely set the noDataValue used for reading; this overwrites the noDataValue defined by the raster itself
• errorThreshold – error threshold for approximation transformer (in pixels)
• warpMemoryLimit – size of working buffer in bytes
• multithread – whether to multithread computation and I/O operations

dataset()

Returns the Raster object.

fractionArray(categories, overlap=0, index=None)

Returns a stack of category fractions for the given categories as a 3-d numpy array of shape = (zsize, ysize, xsize), where zsize is the number of categories.

Parameters:

• categories – list of categories of interest
• overlap – the number of pixels to additionally read along each spatial dimension
• index – index to the band holding the categories

classmethod fromDataset(dataset)

Create an input raster from an Dataset.

metadataDict()

Returns the metadata dictionary.

metadataItem(key, domain)

Returns a metadata item.

noDataValue(default=None)

Return single image no data value. Only valid to use if all bands have the same no data value.

noDataValues(default=None)

Return band no data values.

sample(mask, resampleAlg=<Mock name='mock.gdal.GRA_NearestNeighbour' id='140313600122064'>, noDataValue=None, errorThreshold=0.0, warpMemoryLimit=104857600, multithread=False)

Returns all pixel profiles for which mask is True as a 2-d numpy array of shape = (zsize, samples). Note that pixel profiles are individually accessed, which is fast for sparse masks, but slow otherwise.

Parameters:

• overlap – the number of pixels to additionally read along each spatial dimension
• resampleAlg – GDAL resampling algorithm, e.g. gdal.GRA_NearestNeighbour
• noDataValue – explicitely set the noDataValue used for reading; this overwrites the noDataValue defined by the raster itself
• errorThreshold – error threshold for approximation transformer (in pixels)
• warpMemoryLimit – size of working buffer in bytes
• multithread – whether to multithread computation and I/O operations

class hubdc.applier.ApplierInputRasterGroup

Bases: hubdc.applier.ApplierIOGroup

Container for ApplierInputRaster and ApplierInputRasterGroup objects.

findRaster(ufunc=<function <lambda>>)

Returns the first ApplierInputRaster for that the user defined function ufunc(key, raster) matches. Returns None in case of no match.

findRasterKey(ufunc=<function <lambda>>)

Returns the first key for that the user defined function ufunc(key, raster) matches. Returns None in case of no match.

flatRasterKeys()

Returns an iterator over the keys of all contained ApplierInputRaster’s. Traverses the group structure recursively.

flatRasters()

Returns an iterator over all contained ApplierInputRaster’s. Traverses the group structure recursively.

classmethod fromFolder(folder, extensions, ufunc=None)

Returns an input raster group containing all input rasters that are located relativ to the given folder, matches one of the file extensions, and (optionally) matches the user defined filter ufunc(dirname, basename, extension). In the result, the file system folder structure is preserved.

Parameters:

• folder – root folder
• extensions – only rasters that matches one of the given extensions are included, e.g. [‘’, ‘.bsq’, ‘.tif’, ‘.vrt’].
• ufunc – function of form ufunc(dirname, basename, extension); only files that pass the filter function (i.e. return True) are included

static fromIndex(index)

Returns an input raster group containing all input rasters contained in the ApplierInputRasterIndex given by index.

groupKeys()

Returns an iterator over the keys of all directly contained ApplierInputRasterGroups’s. No recursion.

groups()

Returns an iterator over all directly contained ApplierInputRasterGroups’s. No recursion.

raster(key)

Returns the ApplierInputRaster named key.

rasterKeys()

Returns an iterator over the keys of all directly contained ApplierInputRaster’s. No recursion.

rasters()

Returns an iterator over all directly contained ApplierInputRaster’s. No recursion.

setRaster(key, value)

Add an ApplierInputRaster given by value and named key.

class hubdc.applier.ApplierInputRasterIndex

Bases: object

__init__()

static fromFolder(folder, extensions, ufunc=None)

insertFilename(key, filename, extent)

insertRaster(key, raster)

intersection(grid)

pickle(filename)

static unpickle(filename)

WGS84 = Projection(wkt=<Mockname='mock.osr.SpatialReference()'id='140313596641680'>)

class hubdc.applier.ApplierInputVector(filename, layerNameOrIndex=0)

Bases: hubdc.applier.ApplierIO

Class for handling the vector dataset given by it’s filename and layerNameOrIndex.

__init__(filename, layerNameOrIndex=0)

Parameters:

• filename – filename
• layerNameOrIndex (str or int) – layer name or index

array(initValue=0, burnValue=1, burnAttribute=None, allTouched=False, filterSQL=None, overlap=0, dtype=<Mock name='mock.float32' id='140313596607568'>)

Returns the vector rasterization of the current block in form of a 3-d numpy array of shape = (1, ysize, xsize).

Parameters:

• initValue – value to pre-initialize the output array
• burnValue – value to burn into the output array for all objects; exclusive with burnAttribute
• burnAttribute – identifies an attribute field on the features to be used for a burn-in value; exclusive with burnValue
• allTouched – whether to enable that all pixels touched by lines or polygons will be updated, not just those on the line render path, or whose center point is within the polygon
• filterSQL – set an SQL WHERE clause which will be used to filter vector features
• overlap – the number of pixels to additionally read along each spatial dimension

dataset()

Return the Vector object.

fractionArray(categories, categoryAttribute=None, oversampling=1, overlap=0)

Returns aggregated category fractions of the current block in form of a 3d numpy array of shape = (categories, ysize, xsize).

Parameters:

• categories – list of categories (numbers or names)
• categoryAttribute – attribute field on the features holding the categories
• oversampling – factor defining the relative degree of rasterization detail compared to the target resolution. If for example the target resolution is 30m and the oversampling factor is 10, then the categories are first rasterized at 3m, and finally aggregated to the target resolution.
• overlap – the number of pixels to additionally read along each spatial dimension

class hubdc.applier.ApplierInputVectorGroup

Bases: hubdc.applier.ApplierIOGroup

Container for ApplierInputVector and ApplierInputVectorGroup objects.

flatVectorKeys()

Returns an iterator over the keys of all contained ApplierInputVectors’s. Traverses the group structure recursively.

flatVectors()

Returns an iterator over all contained ApplierInputVector’s. Traverses the group structure recursively.

groupKeys()

Returns an iterator over the keys of all directly contained ApplierInputVectorGroups’s. No recursion.

groups()

Returns an iterator over all directly contained ApplierInputVectorGroups’s. No recursion.

setVector(key, value)

Add an ApplierInputVector given by value and named key.

vector(key)

Returns the ApplierInputVector named key.

class hubdc.applier.ApplierOperator(mainGrid, inputRaster, inputVector, outputRaster, controls, ufuncArgs, ufuncKwargs, operatorUFunc=None)

Bases: object

This is the baseclass for an user defined applier operator. For details on user defined operators see hubdc.applier.Applier.apply()

__init__(mainGrid, inputRaster, inputVector, outputRaster, controls, ufuncArgs, ufuncKwargs, operatorUFunc=None)

static aggregate(blockResults, grid, *args, **kwargs)

Overwrite this method to specify how to aggregate the list of block-wise return values. See Applier Examples for more information.

full(value, bands=1, dtype=None, overlap=0)

Returns a 3-d numpy array of shape = (zsize, ysize+2*overlap, xsize+2*overlap) filled with constant value.

grid()

Returns the Grid.

iblock()

isFirstBlock()

Returns wether or not the current block is the first one.

isLastBlock()

Returns wether or not the current block is the last one.

isLastXBlock()

Returns wether or not the current block is the last block in x direction.

isLastYBlock()

Returns wether or not the current block is the last block in y direction.

nblock()

nxblock()

nyblock()

subgrid()

Returns the current block Grid.

ufunc(*args, **kwargs)

Overwrite this method to specify the image processing. See Applier Examples for more information.

xblock()

xblockOffset()

xblockSize()

yblock()

yblockOffset()

yblockSize()

progressBar

Returns the ProgressBar.

class hubdc.applier.ApplierOptions

Bases: object

Enumeration types.

class AutoExtent

Bases: object

Options for automatic extent calculation.

intersection = 'intersection'

union = 'union'

class AutoResolution

Bases: object

Options for automatic resolution calculation.

average = 'average'

maximum = 'maximum'

minimum = 'minimum'

class hubdc.applier.ApplierOutputRaster(filename, driver=None, creationOptions=None)

Bases: hubdc.applier.ApplierIO

Class for creating and handling an output raster dataset.

__init__(filename, driver=None, creationOptions=None)

Parameters:

• filename – destination filename for output raster
• driver (hubdc.model.RasterDriver) –
• creationOptions (RasterCreationOptions) – e.g. for ENVI and GTiff files see http://www.gdal.org/frmt_various.html#ENVI and http://www.gdal.org/frmt_gtiff.html.

band(index)

Returns the ApplierOutputRasterBand for the given index.

bands()

Returns an iterator over all ApplierOutputRasterBand’s.

flatList()

Returns itself inside a list, i.e. [self].

setArray(array, overlap=0)

Write data to the output raster.

Parameters:

• array – 3-d numpy array of shape = (zsize, ysize, xsize) or 2-d numpy array of shape = (ysize, xsize)
• overlap – the amount of margin (number of pixels) to be removed from the image data block in each direction; this is useful when the overlap keyword was also used during data reading.

setMetadataDict(metadataDict)

Set metadata dictionary.

Parameters:metadataDict – dictionary of dictionaries for different metadata domains, e.g. {'ENVI': {'wavelength' : [482, 561, 655, 865, 1609, 2201], 'wavelength_units' : 'nanometers', 'band_names' : ['Blue', 'Green', 'Red', 'NIR', 'SWIR1', 'SWIR2']}}

setMetadataItem(key, value, domain)

Set image metadata item.

setNoDataValue(value)

Set no data value to all bands.

setZsize(zsize)

Specify the number of output bands. This is only required if the output is written band-wise.

zsize()

class hubdc.applier.ApplierOutputRasterBand(parent, index)

Bases: hubdc.applier.ApplierIO

Class for handling an output raster band dataset.

__init__(parent, index)

For internal use only.

setArray(array, overlap=0)

Write data to the output raster band.

Parameters:

• array – 3-d numpy array of shape = (1, ysize, xsize) or 2-d numpy array of shape = (ysize, xsize)
• overlap – the amount of margin (number of pixels) to be removed from the image data block in each direction; this is useful when the overlap keyword was also used during data reading.

setDescription(value)

Set band description.

setMetadataItem(key, value, domain='')

Set metadata item.

setNoDataValue(value)

Set no data value.

class hubdc.applier.ApplierOutputRasterGroup

Bases: hubdc.applier.ApplierIOGroup

Container for ApplierOutputRaster and ApplierOutputRasterGroup objects.

flatRasterKeys()

Returns an iterator over the keys of all contained ApplierOutputRaster’s. Traverses the group structure recursively.

flatRasters()

Returns an iterator over all contained ApplierOutputRaster’s. Traverses the group structure recursively.

raster(key)

Returns the ApplierOutputRaster named key.

setRaster(key, value)

Add an ApplierOutputRaster given by value and named key.