What and how can files be updated?

[wiesehahn]

Is there any documentation or examples specifically about updating files? I could not find it so far.
With gdal_formats() we can see which formats support it (rw+). But what can be updated and how do we do it.

E.g. I want to convert a GeoTIFF into COG, adding Overviews, setting Compression etc. Is this possible without temporarilly copying the file as we do in translate?

[ctoney]

For the case of converting GeoTIFF into COG it is required to re-write the file. This is also true in general for enabling compression on an uncompressed file (for formats that support compression), or for changing the type of compression used (e.g., with GeoTIFF format which supports several compression methods). The file must be re-written in order to do that. In the case of COG, there is also more involved, e.g., "Optimized layout of TIFF sections to minimize the number of GET requests needed by a reader doing random read access." This requires rewriting the file so we need translate() including the c("-of", "COG") command-line arguments passed in cl_arg.

Besides that, there are several things we can update in-place on files that support rw+. For existing documentation, a good starting point is the vignette "Raster API Tutorial" at:
https://usdaforestservice.github.io/gdalraster/articles/raster-api-tutorial.html
This is an R port of GDAL's Raster API tutorial which is written for C, C++ and Python.

The other relevant documentation and examples are those for class GDALRaster:
https://usdaforestservice.github.io/gdalraster/reference/GDALRaster-class.html

To open a raster dataset with update access:

ds <- new(GDALRaster, filename, read_only = FALSE)

where filename is the full path or relative to the current working directory. In some cases, filename may not refer to a physical file, but instead contain format-specific information on how to access a dataset (e.g., database connection string, URL, /vsiPREFIX/, etc.).

In the documentation for GDALRaster-class, there are several "set" methods that can be called on a dataset opened with update access. The methods ds$setGeoTransform(transform) and ds$setProjection(projection) would normally be used only with a newly created, uninitialized raster. These only edit the information on the raster, they do not perform any transformation or reprojection. An example is given for create():
https://usdaforestservice.github.io/gdalraster/reference/create.html

We can build or delete overviews with ds$buildOverviews(resampling, levels, bands). Note that for GTiff, overviews will be created internally if the dataset is open in update mode, while external overviews (.ovr) will be created if the dataset is open read-only.

We can set or delete a raster nodata value (per band). This does not change any pixel values, it only affects the value of the nodata marker that is defined on the dataset (for formats that support an intrinsic nodata value).

We can set the unit type, scale, offset and color interpretation values per band. We can get statistics which will generally then be "set back" on the raster if the format supports caching statistics. We can also set metadata items. See the GDAL Raster Data Model for a description of metadata:
https://gdal.org/user/raster_data_model.html#metadata

We can get/set color tables and raster attribute tables. These will then be available in other software such as QGIS. GDAL Raster Attribute Tables are well-supported in gdalraster and there is a vignette with details:
https://usdaforestservice.github.io/gdalraster/articles/raster-attribute-tables.html

We can write pixel data with ds$write(band, xoff, yoff, xsize, ysize, rasterData), where xoff/yoff are column/row offsets (number of pixels from the origin) where writing will start, xsize/ysize give the size of the region that will be written (number pixels in the x direction/y direction), and rasterData is vector of pixel values to write. The stand-alone function get_pixel_line(), and equivalent class method ds$get_pixel_line(xy) (new in gdalraster dev as the class method), can be used to translate geospatial xy coordinates to raster column/row (pixel/line).

Also note that when calling methods on a GDALRaster object, a value must be supplied for all parameters (for methods that have parameters). Methods that operate on bands take the band number as the first param. Some methods operate at the dataset level or band level, e.g., the get/set metadata methods. In that case, to specify dataset-level use band = 0. If a class method is called without the required params (i.e., an incorrect calling signature), an error will be returned like:

Error: could not find valid method

So just check the documentation and verify the required parameters are correct if that occurs.

After updating a raster dataset has been completed, it is important to call ds$close() to flush pending writes and perform proper cleanup.

Let me know if there are examples relating to update, or gaps in the documentation, that you think would be helpful to add.