How to Read Fill Value From Xarray Data Array
Reading and writing files¶
Xarray supports direct serialization and IO to several file formats, from simple Pickle files to the more flexible netCDF format (recommended).
netCDF¶
The recommended way to store xarray information structures is netCDF, which is a binary file format for cocky-described datasets that originated in the geosciences. Xarray is based on the netCDF data model, then netCDF files on disk direct represent to Dataset objects (more than accurately, a group in a netCDF file directly corresponds to a Dataset object. Come across Groups for more than.)
NetCDF is supported on almost all platforms, and parsers exist for the vast majority of scientific programming languages. Contempo versions of netCDF are based on the even more widely used HDF5 file-format.
Tip
If you lot aren't familiar with this data format, the netCDF FAQ is a skillful identify to outset.
Reading and writing netCDF files with xarray requires scipy or the netCDF4-Python library to be installed (the latter is required to read/write netCDF V4 files and use the pinch options described below).
We can save a Dataset to disk using the Dataset.to_netcdf() method:
In [1]: ds = xr . Dataset ( ...: { "foo" : (( "x" , "y" ), np . random . rand ( iv , 5 ))}, ...: coords = { ...: "x" : [ x , 20 , 30 , 40 ], ...: "y" : pd . date_range ( "2000-01-01" , periods = 5 ), ...: "z" : ( "x" , listing ( "abcd" )), ...: }, ...: ) ...: In [2]: ds . to_netcdf ( "saved_on_disk.nc" ) Past default, the file is saved as netCDF4 (bold netCDF4-Python is installed). You can control the format and engine used to write the file with the format and engine arguments.
Tip
Using the h5netcdf bundle by passing engine='h5netcdf' to open_dataset() can sometimes be quicker than the default engine='netcdf4' that uses the netCDF4 package.
We tin can load netCDF files to create a new Dataset using open_dataset() :
In [3]: ds_disk = xr . open_dataset ( "saved_on_disk.nc" ) In [four]: ds_disk Out[four]: <xarray.Dataset> Dimensions: (x: 4, y: 5) Coordinates: * x (x) int64 ten twenty xxx 40 * y (y) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-01-04 2000-01-05 z (x) object ... Data variables: foo (x, y) float64 ... Similarly, a DataArray can be saved to disk using the DataArray.to_netcdf() method, and loaded from deejay using the open_dataarray() function. Every bit netCDF files correspond to Dataset objects, these functions internally convert the DataArray to a Dataset before saving, so convert back when loading, ensuring that the DataArray that is loaded is always exactly the same equally the 1 that was saved.
A dataset tin can too be loaded or written to a specific group within a netCDF file. To load from a grouping, pass a group keyword argument to the open_dataset function. The group tin can be specified as a path-like string, e.g., to access subgroup 'bar' within group 'foo' laissez passer '/foo/bar' equally the grouping statement. When writing multiple groups in ane file, laissez passer manner='a' to to_netcdf to ensure that each telephone call does not delete the file.
Information is always loaded lazily from netCDF files. You can manipulate, slice and subset Dataset and DataArray objects, and no array values are loaded into memory until you try to perform some sort of actual computation. For an example of how these lazy arrays work, see the OPeNDAP section below.
There may exist small-scale differences in the Dataset object returned when reading a NetCDF file with different engines. For example, single-valued attributes are returned equally scalars by the default engine=netcdf4 , but every bit arrays of size (i,) when reading with engine=h5netcdf .
It is of import to note that when yous modify values of a Dataset, even one linked to files on disk, only the in-memory copy you are manipulating in xarray is modified: the original file on disk is never touched.
Tip
Xarray's lazy loading of remote or on-disk datasets is often but not always desirable. Before performing computationally intense operations, information technology is ofttimes a expert idea to load a Dataset (or DataArray) entirely into memory by invoking the Dataset.load() method.
Datasets have a Dataset.close() method to shut the associated netCDF file. However, it's oft cleaner to utilize a with statement:
# this automatically closes the dataset later utilize In [five]: with xr . open_dataset ( "saved_on_disk.nc" ) as ds : ...: print ( ds . keys ()) ...: KeysView(<xarray.Dataset> Dimensions: (x: 4, y: 5) Coordinates: * x (10) int64 x twenty 30 40 * y (y) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-01-04 2000-01-05 z (ten) object ... Information variables: foo (x, y) float64 ...) Although xarray provides reasonable support for incremental reads of files on deejay, it does not support incremental writes, which can be a useful strategy for dealing with datasets too big to fit into memory. Instead, xarray integrates with dask.array (meet Parallel computing with Dask), which provides a fully featured engine for streaming ciphering.
Information technology is possible to append or overwrite netCDF variables using the mode='a' argument. When using this choice, all variables in the dataset volition be written to the original netCDF file, regardless if they be in the original dataset.
Groups¶
NetCDF groups are not supported as part of the Dataset data model. Instead, groups can be loaded individually as Dataset objects. To do so, pass a group keyword argument to the open_dataset() role. The group can be specified as a path-like cord, e.g., to access subgroup 'bar' inside group 'foo' laissez passer '/foo/bar' as the group argument. In a like way, the group keyword argument tin be given to the Dataset.to_netcdf() method to write to a grouping in a netCDF file. When writing multiple groups in one file, pass mode='a' to Dataset.to_netcdf() to ensure that each call does not delete the file.
Reading encoded data¶
NetCDF files follow some conventions for encoding datetime arrays (every bit numbers with a "units" aspect) and for packing and unpacking data (as described by the "scale_factor" and "add_offset" attributes). If the argument decode_cf=Truthful (default) is given to open_dataset() , xarray volition effort to automatically decode the values in the netCDF objects co-ordinate to CF conventions. Sometimes this will fail, for example, if a variable has an invalid "units" or "calendar" attribute. For these cases, you can plow this decoding off manually.
You can view this encoding information (amidst others) in the DataArray.encoding and DataArray.encoding attributes:
In [half dozen]: ds_disk [ "y" ] . encoding Out[6]: {'zlib': False, 'shuffle': False, 'complevel': 0, 'fletcher32': Faux, 'face-to-face': Truthful, 'chunksizes': None, 'source': 'saved_on_disk.nc', 'original_shape': (5,), 'dtype': dtype('int64'), 'units': 'days since 2000-01-01 00:00:00', 'calendar': 'proleptic_gregorian'} In [seven]: ds_disk . encoding Out[seven]: {'unlimited_dims': prepare(), 'source': 'saved_on_disk.nc'} Annotation that all operations that manipulate variables other than indexing will remove encoding information.
Reading multi-file datasets¶
NetCDF files are oftentimes encountered in collections, e.k., with unlike files respective to different model runs or one file per timestamp. Xarray can straightforwardly combine such files into a single Dataset past making use of concat() , merge() , combine_nested() and combine_by_coords() . For details on the deviation between these functions encounter Combining information.
Xarray includes support for manipulating datasets that don't fit into memory with dask. If you lot have dask installed, y'all tin open multiple files simultaneously in parallel using open_mfdataset() :
xr . open_mfdataset ( 'my/files/*.nc' , parallel = True ) This function automatically concatenates and merges multiple files into a unmarried xarray dataset. Information technology is the recommended way to open multiple files with xarray. For more details on parallel reading, see Combining forth multiple dimensions, Reading and writing data and a weblog post past Stephan Hoyer. open_mfdataset() takes many kwargs that allow you to control its behaviour (for e.1000. parallel , combine , compat , join , concat_dim ). See its docstring for more than details.
Note
A mutual use-case involves a dataset distributed across a large number of files with each file containing a large number of variables. Ordinarily, a few of these variables need to exist concatenated along a dimension (say "fourth dimension" ), while the rest are equal beyond the datasets (ignoring floating point differences). The following control with suitable modifications (such as parallel=Truthful ) works well with such datasets:
xr . open_mfdataset ( 'my/files/*.nc' , concat_dim = "time" , combine = "nested" , data_vars = 'minimal' , coords = 'minimal' , compat = 'override' ) This command concatenates variables along the "time" dimension, but simply those that already contain the "time" dimension ( data_vars='minimal', coords='minimal' ). Variables that lack the "time" dimension are taken from the first dataset ( compat='override' ).
Sometimes multi-file datasets are non conveniently organized for easy use of open_mfdataset() . One can use the preprocess argument to provide a function that takes a dataset and returns a modified Dataset. open_mfdataset() will call preprocess on every dataset (corresponding to each file) prior to combining them.
If open_mfdataset() does not see your needs, other approaches are possible. The general pattern for parallel reading of multiple files using dask, modifying those datasets and and so combining into a unmarried Dataset is:
def change ( ds ): # change ds here return ds # this is basically what open_mfdataset does open_kwargs = dict ( decode_cf = True , decode_times = False ) open_tasks = [ dask . delayed ( xr . open_dataset )( f , ** open_kwargs ) for f in file_names ] tasks = [ dask . delayed ( modify )( chore ) for task in open_tasks ] datasets = dask . compute ( tasks ) # become a list of xarray.Datasets combined = xr . combine_nested ( datasets ) # or some combination of concat, merge As an instance, here's how we could guess MFDataset from the netCDF4 library:
from glob import glob import xarray as xr def read_netcdfs ( files , dim ): # glob expands paths with * to a list of files, similar the unix shell paths = sorted ( glob ( files )) datasets = [ xr . open_dataset ( p ) for p in paths ] combined = xr . concat ( datasets , dim ) return combined combined = read_netcdfs ( '/all/my/files/*.nc' , dim = 'fourth dimension' ) This function will work in many cases, but information technology's non very robust. First, it never closes files, which ways it volition fail if you demand to load more than a few 1000 files. 2nd, it assumes that you want all the information from each file and that it tin can all fit into memory. In many situations, y'all only need a small subset or an aggregated summary of the data from each file.
Here's a slightly more sophisticated example of how to remedy these deficiencies:
def read_netcdfs ( files , dim , transform_func = None ): def process_one_path ( path ): # use a context manager, to ensure the file gets airtight after employ with xr . open_dataset ( path ) as ds : # transform_func should do some sort of pick or # assemblage if transform_func is not None : ds = transform_func ( ds ) # load all data from the transformed dataset, to ensure nosotros can # use it after closing each original file ds . load () return ds paths = sorted ( glob ( files )) datasets = [ process_one_path ( p ) for p in paths ] combined = xr . concat ( datasets , dim ) return combined # hither we suppose nosotros only care about the combined hateful of each file; # you might as well employ indexing operations like .sel to subset datasets combined = read_netcdfs ( '/all/my/files/*.nc' , dim = 'time' , transform_func = lambda ds : ds . hateful ()) This pattern works well and is very robust. We've used similar code to procedure tens of thousands of files constituting 100s of GB of data.
Writing encoded data¶
Conversely, you can customize how xarray writes netCDF files on disk by providing explicit encodings for each dataset variable. The encoding argument takes a dictionary with variable names as keys and variable specific encodings as values. These encodings are saved as attributes on the netCDF variables on disk, which allows xarray to faithfully read encoded data back into retentivity.
Information technology is important to note that using encodings is entirely optional: if y'all do not supply any of these encoding options, xarray will write information to disk using a default encoding, or the options in the encoding aspect, if set. This works perfectly fine in most cases, simply encoding tin can be useful for additional control, especially for enabling compression.
In the file on disk, these encodings are saved as attributes on each variable, which permit xarray and other CF-compliant tools for working with netCDF files to correctly read the data.
Scaling and type conversions¶
These encoding options work on any version of the netCDF file format:
-
dtype: Any valid NumPy dtype or cord convertible to a dtype, east.thou.,'int16'or'float32'. This controls the blazon of the data written on disk. -
_FillValue: Values ofNaNin xarray variables are remapped to this value when saved on disk. This is of import when converting floating bespeak with missing values to integers on disk, consideringNaNis not a valid value for integer dtypes. Past default, variables with float types are attributed a_FillValueofNaNin the output file, unless explicitly disabled with an encoding{'_FillValue': None}. -
scale_factorandadd_offset: Used to convert from encoded data on deejay to to the decoded information in retention, according to the formuladecoded = scale_factor * encoded + add_offset.
These parameters tin be fruitfully combined to compress discretized data on deejay. For example, to save the variable foo with a precision of 0.1 in 16-chip integers while converting NaN to -9999 , we would apply encoding={'foo': {'dtype': 'int16', 'scale_factor': 0.one, '_FillValue': -9999}} . Compression and decompression with such discretization is extremely fast.
String encoding¶
Xarray can write unicode strings to netCDF files in two means:
-
Every bit variable length strings. This is only supported on netCDF4 (HDF5) files.
-
By encoding strings into bytes, and writing encoded bytes equally a character array. The default encoding is UTF-8.
By default, nosotros utilize variable length strings for compatible files and fall-dorsum to using encoded grapheme arrays. Character arrays can be selected even for netCDF4 files past setting the dtype field in encoding to S1 (respective to NumPy'south single-character bytes dtype).
If character arrays are used:
-
The string encoding that was used is stored on disk in the
_Encodingattribute, which matches an ad-hoc convention adopted by the netCDF4-Python library. At the time of this writing (October 2017), a standard convention for indicating string encoding for character arrays in netCDF files was nevertheless under discussion. Technically, yous can use any cord encoding recognized past Python if you feel the need to deviate from UTF-eight, by setting the_Encodingfield inencoding. But we don't recommend it. -
The character dimension name tin can be specified past the
char_dim_namefield of a variable'southencoding. If the proper name of the graphic symbol dimension is non specified, the default isf'cord{data.shape[-1]}'. When decoding character arrays from existing files, thechar_dim_nameis added to the variablesencodingto preserve if encoding happens, just the field tin can exist edited past the user.
Warning
Missing values in bytes or unicode string arrays (represented by NaN in xarray) are currently written to deejay every bit empty strings '' . This ways missing values will not be restored when data is loaded from disk. This behavior is likely to modify in the future (GH1647). Unfortunately, explicitly setting a _FillValue for string arrays to handle missing values doesn't work yet either, though we also hope to fix this in the hereafter.
Clamper based pinch¶
zlib , complevel , fletcher32 , continguous and chunksizes can be used for enabling netCDF4/HDF5's chunk based pinch, as described in the documentation for createVariable for netCDF4-Python. This only works for netCDF4 files and thus requires using format='netCDF4' and either engine='netcdf4' or engine='h5netcdf' .
Clamper based gzip pinch can yield impressive space savings, peculiarly for thin information, but it comes with significant performance overhead. HDF5 libraries can only read consummate chunks back into retentiveness, and maximum decompression speed is in the range of l-100 MB/southward. Worse, HDF5's pinch and decompression currently cannot exist parallelized with dask. For these reasons, we recommend trying discretization based pinch (described in a higher place) get-go.
Time units¶
The units and calendar attributes control how xarray serializes datetime64 and timedelta64 arrays to datasets on disk every bit numeric values. The units encoding should be a cord similar 'days since 1900-01-01' for datetime64 data or a string like 'days' for timedelta64 information. calendar should be 1 of the calendar types supported past netCDF4-python: 'standard', 'gregorian', 'proleptic_gregorian' 'noleap', '365_day', '360_day', 'julian', 'all_leap', '366_day'.
By default, xarray uses the 'proleptic_gregorian' calendar and units of the smallest time difference between values, with a reference time of the first fourth dimension value.
Coordinates¶
You tin can control the coordinates attribute written to disk by specifying DataArray.encoding["coordinates"] . If not specified, xarray automatically sets DataArray.encoding["coordinates"] to a space-delimited list of names of coordinate variables that share dimensions with the DataArray being written. This allows perfect roundtripping of xarray datasets but may not exist desirable. When an xarray Dataset contains not-dimensional coordinates that do non share dimensions with any of the variables, these coordinate variable names are saved nether a "global" "coordinates" attribute. This is not CF-compliant but again facilitates roundtripping of xarray datasets.
Invalid netCDF files¶
The library h5netcdf allows writing some dtypes (booleans, complex, …) that aren't immune in netCDF4 (see h5netcdf documentation). This feature is available through DataArray.to_netcdf() and Dataset.to_netcdf() when used with engine="h5netcdf" and currently raises a alert unless invalid_netcdf=Truthful is set:
# Writing circuitous valued data In [viii]: da = xr . DataArray ([ 1.0 + 1.0 j , 2.0 + 2.0 j , three.0 + iii.0 j ]) In [9]: da . to_netcdf ( "complex.nc" , engine = "h5netcdf" , invalid_netcdf = Truthful ) # Reading it dorsum In [ten]: xr . open_dataarray ( "complex.nc" , engine = "h5netcdf" ) Out[ten]: <xarray.DataArray (dim_0: 3)> array([ane.+1.j, 2.+2.j, three.+3.j]) Dimensions without coordinates: dim_0 Warning
Note that this produces a file that is probable to be not readable by other netCDF libraries!
Iris¶
The Iris tool allows easy reading of common meteorological and climate model formats (including GRIB and UK MetOffice PP files) into Cube objects which are in many ways very similar to DataArray objects, while enforcing a CF-compliant information model. If iris is installed, xarray tin can convert a DataArray into a Cube using DataArray.to_iris() :
In [xi]: da = xr . DataArray ( ....: np . random . rand ( 4 , 5 ), ....: dims = [ "10" , "y" ], ....: coords = dict ( x = [ 10 , 20 , 30 , 40 ], y = pd . date_range ( "2000-01-01" , periods = 5 )), ....: ) ....: In [12]: cube = da . to_iris () In [13]: cube Out[13]: <iris 'Cube' of unknown / (unknown) (x: 4; y: five)> Conversely, we can create a new DataArray object from a Cube using DataArray.from_iris() :
In [xiv]: da_cube = xr . DataArray . from_iris ( cube ) In [15]: da_cube Out[xv]: <xarray.DataArray (x: 4, y: 5)> array([[0.8529 , 0.235507, 0.146227, 0.589869, 0.574012], [0.06127 , 0.590426, 0.24535 , 0.340445, 0.984729], [0.91954 , 0.037772, 0.861549, 0.753569, 0.405179], [0.343526, 0.170917, 0.394659, 0.641666, 0.274592]]) Coordinates: * 10 (x) int64 10 20 30 forty * y (y) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-01-04 2000-01-05 OPeNDAP¶
Xarray includes support for OPeNDAP (via the netCDF4 library or Pydap), which lets usa access big datasets over HTTP.
For example, we can open a connection to GBs of weather data produced by the PRISM project, and hosted by IRI at Columbia:
In [16]: remote_data = xr . open_dataset ( ....: "http://iridl.ldeo.columbia.edu/SOURCES/.OSU/.PRISM/.monthly/dods" , ....: decode_times = False , ....: ) ....: In [17]: remote_data Out[17]: <xarray.Dataset> Dimensions: (T: 1422, Ten: 1405, Y: 621) Coordinates: * X (10) float32 -125.0 -124.958 -124.917 -124.875 -124.833 -124.792 -124.75 ... * T (T) float32 -779.5 -778.5 -777.5 -776.v -775.five -774.5 -773.v -772.5 -771.v ... * Y (Y) float32 49.9167 49.875 49.8333 49.7917 49.75 49.7083 49.6667 49.625 ... Data variables: ppt (T, Y, X) float64 ... tdmean (T, Y, 10) float64 ... tmax (T, Y, X) float64 ... tmin (T, Y, X) float64 ... Attributes: Conventions: IRIDL expires: 1375315200 Note
Like many existent-earth datasets, this dataset does not entirely follow CF conventions. Unexpected formats will unremarkably cause xarray'south automated decoding to fail. The way to work around this is to either set up decode_cf=False in open_dataset to turn off all use of CF conventions, or past merely disabling the troublesome parser. In this case, we set decode_times=False considering the time axis here provides the agenda attribute in a format that xarray does not look (the integer 360 instead of a string like '360_day' ).
Nosotros can select and slice this data any number of times, and zero is loaded over the network until nosotros wait at particular values:
In [18]: tmax = remote_data [ "tmax" ][: 500 , :: 3 , :: three ] In [19]: tmax Out[nineteen]: <xarray.DataArray 'tmax' (T: 500, Y: 207, X: 469)> [48541500 values with dtype=float64] Coordinates: * Y (Y) float32 49.9167 49.7917 49.6667 49.5417 49.4167 49.2917 ... * 10 (X) float32 -125.0 -124.875 -124.75 -124.625 -124.5 -124.375 ... * T (T) float32 -779.v -778.five -777.5 -776.5 -775.5 -774.5 -773.five ... Attributes: pointwidth: 120 standard_name: air_temperature units: Celsius_scale expires: 1443657600 # the data is downloaded automatically when nosotros brand the plot In [20]: tmax [ 0 ] . plot () 
Some servers require hallmark before we tin can access the information. For this purpose nosotros tin can explicitly create a backends.PydapDataStore and pass in a Requests session object. For case for HTTP Basic authentication:
import xarray as xr import requests session = requests . Session () session . auth = ( 'username' , 'countersign' ) shop = xr . backends . PydapDataStore . open ( 'http://case.com/data' , session = session ) ds = xr . open_dataset ( shop ) Pydap's cas module has functions that generate custom sessions for servers that use CAS unmarried sign-on. For example, to connect to servers that require NASA's URS hallmark:
import xarray as xr from pydata.cas.urs import setup_session ds_url = 'https://gpm1.gesdisc.eosdis.nasa.gov/opendap/hyrax/instance.nc' session = setup_session ( 'username' , 'password' , check_url = ds_url ) shop = xr . backends . PydapDataStore . open ( ds_url , session = session ) ds = xr . open_dataset ( shop ) Pickle¶
The simplest way to serialize an xarray object is to use Python's built-in pickle module:
In [21]: import pickle # use the highest protocol (-ane) because it is way faster than the default # text based pickle format In [22]: pkl = pickle . dumps ( ds , protocol =- 1 ) In [23]: pickle . loads ( pkl ) Out[23]: <xarray.Dataset> Dimensions: (x: 4, y: v) Coordinates: * 10 (x) int64 ten 20 30 40 * y (y) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-01-04 2000-01-05 z (x) object ... Information variables: foo (10, y) float64 ... Pickling is of import because it doesn't require any external libraries and lets you use xarray objects with Python modules similar multiprocessing or Dask. However, pickling is not recommended for long-term storage.
Restoring a pickle requires that the internal structure of the types for the pickled data remain unchanged. Because the internal design of xarray is still being refined, we make no guarantees (at this betoken) that objects pickled with this version of xarray volition work in future versions.
Note
When pickling an object opened from a NetCDF file, the pickle file will contain a reference to the file on deejay. If you desire to store the actual array values, load it into memory starting time with Dataset.load() or Dataset.compute() .
Lexicon¶
We tin can convert a Dataset (or a DataArray ) to a dict using Dataset.to_dict() :
In [24]: d = ds . to_dict () In [25]: d Out[25]: {'coords': {'x': {'dims': ('ten',), 'attrs': {}, 'data': [x, 20, 30, forty]}, 'y': {'dims': ('y',), 'attrs': {}, 'data': [datetime.datetime(2000, one, 1, 0, 0), datetime.datetime(2000, 1, 2, 0, 0), datetime.datetime(2000, 1, three, 0, 0), datetime.datetime(2000, one, 4, 0, 0), datetime.datetime(2000, 1, 5, 0, 0)]}, 'z': {'dims': ('x',), 'attrs': {}, 'data': ['a', 'b', 'c', 'd']}}, 'attrs': {}, 'dims': {'10': 4, 'y': 5}, 'data_vars': {'foo': {'dims': ('x', 'y'), 'attrs': {}, 'data': [[0.12696983303810094, 0.966717838482003, 0.26047600586578334, 0.8972365243645735, 0.37674971618967135], [0.33622174433445307, 0.45137647047539964, 0.8402550832613813, 0.12310214428849964, 0.5430262020470384], [0.37301222522143085, 0.4479968246859435, 0.12944067971751294, 0.8598787065799693, 0.8203883631195572], [0.35205353914802473, 0.2288873043216132, 0.7767837505077176, 0.5947835894851238, 0.1375535565632705]]}}} We tin can create a new xarray object from a dict using Dataset.from_dict() :
In [26]: ds_dict = xr . Dataset . from_dict ( d ) In [27]: ds_dict Out[27]: <xarray.Dataset> Dimensions: (10: 4, y: 5) Coordinates: * x (ten) int64 10 20 30 40 * y (y) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-01-04 2000-01-05 z (x) <U1 'a' 'b' 'c' 'd' Information variables: foo (x, y) float64 0.127 0.9667 0.2605 0.8972 ... 0.7768 0.5948 0.1376 Lexicon back up allows for flexible use of xarray objects. It doesn't require external libraries and dicts tin hands be pickled, or converted to json, or geojson. All the values are converted to lists, so dicts might exist quite large.
To export just the dataset schema without the data itself, utilize the data=Imitation option:
In [28]: ds . to_dict ( information = Fake ) Out[28]: {'coords': {'x': {'dims': ('10',), 'attrs': {}, 'dtype': 'int64', 'shape': (4,)}, 'y': {'dims': ('y',), 'attrs': {}, 'dtype': 'datetime64[ns]', 'shape': (5,)}, 'z': {'dims': ('x',), 'attrs': {}, 'dtype': 'object', 'shape': (4,)}}, 'attrs': {}, 'dims': {'x': 4, 'y': 5}, 'data_vars': {'foo': {'dims': ('x', 'y'), 'attrs': {}, 'dtype': 'float64', 'shape': (4, 5)}}} This can be useful for generating indices of dataset contents to betrayal to search indices or other automatic information discovery tools.
Rasterio¶
GeoTIFFs and other gridded raster datasets tin be opened using rasterio, if rasterio is installed. Here is an instance of how to apply open_rasterio() to read ane of rasterio's exam files:
In [29]: rio = xr . open_rasterio ( "RGB.byte.tif" ) In [30]: rio Out[thirty]: <xarray.DataArray (band: 3, y: 718, 10: 791)> [1703814 values with dtype=uint8] Coordinates: * band (band) int64 1 2 3 * y (y) float64 2.827e+06 2.826e+06 two.826e+06 2.826e+06 two.826e+06 ... * x (ten) float64 i.021e+05 1.024e+05 i.027e+05 ane.03e+05 1.033e+05 ... Attributes: res: (300.0379266750948, 300.041782729805) transform: (300.0379266750948, 0.0, 101985.0, 0.0, -300.041782729805, 28... is_tiled: 0 crs: +init=epsg:32618 The x and y coordinates are generated out of the file's metadata ( bounds , width , acme ), and they can exist understood as cartesian coordinates defined in the file's projection provided by the crs attribute. crs is a PROJ4 string which tin exist parsed past e.g. pyproj or rasterio. See Parsing rasterio geocoordinates for an example of how to convert these to longitudes and latitudes.
Additionally, y'all can utilize rioxarray for reading in GeoTiff, netCDF or other GDAL readable raster data using rasterio every bit well equally for exporting to a geoTIFF. rioxarray tin can too handle geospatial related tasks such as re-projecting and clipping.
In [31]: import rioxarray In [32]: rds = rioxarray . open_rasterio ( "RGB.byte.tif" ) In [33]: rds Out[33]: <xarray.DataArray (band: 3, y: 718, x: 791)> [1703814 values with dtype=uint8] Coordinates: * band (band) int64 1 2 3 * y (y) float64 two.827e+06 2.826e+06 ... 2.612e+06 2.612e+06 * 10 (ten) float64 1.021e+05 i.024e+05 ... three.389e+05 3.392e+05 spatial_ref int64 0 Attributes: STATISTICS_MAXIMUM: 255 STATISTICS_MEAN: 29.947726688477 STATISTICS_MINIMUM: 0 STATISTICS_STDDEV: 52.340921626611 transform: (300.0379266750948, 0.0, 101985.0, 0.0, -300.0417827... _FillValue: 0.0 scale_factor: 1.0 add_offset: 0.0 grid_mapping: spatial_ref In [34]: rds . rio . crs Out[34]: CRS.from_epsg(32618) In [35]: rds4326 = rds . rio . reproject ( "epsg:4326" ) In [36]: rds4326 . rio . crs Out[36]: CRS.from_epsg(4326) In [37]: rds4326 . rio . to_raster ( "RGB.byte.4326.tif" ) Zarr¶
Zarr is a Python parcel that provides an implementation of chunked, compressed, Northward-dimensional arrays. Zarr has the power to store arrays in a range of ways, including in retentiveness, in files, and in cloud-based object storage such as Amazon S3 and Google Cloud Storage. Xarray's Zarr backend allows xarray to leverage these capabilities, including the ability to shop and clarify datasets far too large fit onto disk (particularly in combination with dask).
Xarray can't open just any zarr dataset, because xarray requires special metadata (attributes) describing the dataset dimensions and coordinates. At this time, xarray tin can only open zarr datasets that have been written by xarray. For implementation details, see Zarr Encoding Specification.
To write a dataset with zarr, we use the Dataset.to_zarr() method.
To write to a local directory, we pass a path to a directory:
In [38]: ds = xr . Dataset ( ....: { "foo" : (( "x" , "y" ), np . random . rand ( 4 , v ))}, ....: coords = { ....: "ten" : [ 10 , xx , 30 , 40 ], ....: "y" : pd . date_range ( "2000-01-01" , periods = v ), ....: "z" : ( "x" , list ( "abcd" )), ....: }, ....: ) ....: In [39]: ds . to_zarr ( "path/to/directory.zarr" ) Out[39]: <xarray.backends.zarr.ZarrStore at 0x7f1bb8f8acf0> (The suffix .zarr is optional–only a reminder that a zarr store lives there.) If the directory does not exist, it will exist created. If a zarr shop is already present at that path, an error will be raised, preventing it from being overwritten. To override this behavior and overwrite an existing store, add mode='w' when invoking to_zarr() .
To shop variable length strings, convert them to object arrays first with dtype=object .
To read back a zarr dataset that has been created this way, nosotros use the open_zarr() method:
In [40]: ds_zarr = xr . open_zarr ( "path/to/directory.zarr" ) In [41]: ds_zarr Out[41]: <xarray.Dataset> Dimensions: (x: 4, y: v) Coordinates: * 10 (10) int64 ten 20 xxx forty * y (y) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-01-04 2000-01-05 z (x) <U1 dask.array<chunksize=(four,), meta=np.ndarray> Data variables: foo (x, y) float64 dask.array<chunksize=(4, 5), meta=np.ndarray> Cloud Storage Buckets¶
It is possible to read and write xarray datasets directly from / to deject storage buckets using zarr. This example uses the gcsfs parcel to provide an interface to Google Cloud Storage.
From v0.sixteen.2: general fsspec URLs are parsed and the store set upwards for yous automatically when reading, such that you tin open a dataset in a unmarried phone call. You should include any arguments to the storage backend every bit the cardinal storage_options , role of backend_kwargs .
ds_gcs = xr . open_dataset ( "gcs://<saucepan-proper noun>/path.zarr" , backend_kwargs = { "storage_options" : { "project" : "<project-name>" , "token" : None } }, engine = "zarr" , ) This besides works with open_mfdataset , assuasive you to laissez passer a list of paths or a URL to be interpreted equally a glob cord.
For older versions, and for writing, you must explicitly set up a MutableMapping instance and pass this, as follows:
import gcsfs fs = gcsfs . GCSFileSystem ( project = "<project-name>" , token = None ) gcsmap = gcsfs . mapping . GCSMap ( "<bucket-name>" , gcs = fs , check = True , create = False ) # write to the bucket ds . to_zarr ( store = gcsmap ) # read information technology back ds_gcs = xr . open_zarr ( gcsmap ) (or apply the utility role fsspec.get_mapper() ).
Zarr Compressors and Filters¶
There are many different options for compression and filtering possible with zarr. These are described in the zarr documentation. These options tin can be passed to the to_zarr method equally variable encoding. For instance:
In [42]: import zarr In [43]: compressor = zarr . Blosc ( cname = "zstd" , clevel = 3 , shuffle = 2 ) In [44]: ds . to_zarr ( "foo.zarr" , encoding = { "foo" : { "compressor" : compressor }}) Out[44]: <xarray.backends.zarr.ZarrStore at 0x7f1bba37f270> Note
Non all native zarr compression and filtering options take been tested with xarray.
Appending to existing Zarr stores¶
Xarray supports several ways of incrementally writing variables to a Zarr store. These options are useful for scenarios when it is infeasible or undesirable to write your entire dataset at once.
Tip
If you can load all of your data into a unmarried Dataset using dask, a single telephone call to to_zarr() will write all of your information in parallel.
Warning
Alignment of coordinates is currently non checked when modifying an existing Zarr store. It is up to the user to ensure that coordinates are consistent.
To add or overwrite entire variables, simply call to_zarr() with style='a' on a Dataset containing the new variables, passing in an existing Zarr store or path to a Zarr store.
To resize and then append values along an existing dimension in a store, set append_dim . This is a proficient option if data ever arives in a particular order, e.thou., for time-stepping a simulation:
In [45]: ds1 = xr . Dataset ( ....: { "foo" : (( "10" , "y" , "t" ), np . random . rand ( 4 , 5 , 2 ))}, ....: coords = { ....: "x" : [ 10 , 20 , 30 , 40 ], ....: "y" : [ 1 , 2 , 3 , 4 , v ], ....: "t" : pd . date_range ( "2001-01-01" , periods = 2 ), ....: }, ....: ) ....: In [46]: ds1 . to_zarr ( "path/to/directory.zarr" ) Out[46]: <xarray.backends.zarr.ZarrStore at 0x7f1bb8f03660> In [47]: ds2 = xr . Dataset ( ....: { "foo" : (( "10" , "y" , "t" ), np . random . rand ( 4 , five , 2 ))}, ....: coords = { ....: "x" : [ 10 , 20 , 30 , 40 ], ....: "y" : [ one , 2 , 3 , 4 , 5 ], ....: "t" : pd . date_range ( "2001-01-03" , periods = two ), ....: }, ....: ) ....: In [48]: ds2 . to_zarr ( "path/to/directory.zarr" , append_dim = "t" ) Out[48]: <xarray.backends.zarr.ZarrStore at 0x7f1bb8f039e0> Finally, you tin can use region to write to express regions of existing arrays in an existing Zarr shop. This is a good option for writing data in parallel from independent processes.
To scale this up to writing large datasets, the beginning step is creating an initial Zarr shop without writing all of its array data. This tin can be done by first creating a Dataset with dummy values stored in dask, and then calling to_zarr with compute=False to write only metadata (including attrs ) to Zarr:
In [49]: import dask.array # The values of this dask array are entirely irrelevant; only the dtype, # shape and chunks are used In [50]: dummies = dask . assortment . zeros ( thirty , chunks = ten ) In [51]: ds = xr . Dataset ({ "foo" : ( "10" , dummies )}) In [52]: path = "path/to/directory.zarr" # Now we write the metadata without computing whatsoever array values In [53]: ds . to_zarr ( path , compute = Simulated ) Out[53]: Delayed('_finalize_store-5faaeb29-b088-41a1-bfe0-c65dd1bbfc7f') Now, a Zarr store with the correct variable shapes and attributes exists that tin can exist filled out by subsequent calls to to_zarr . The region provides a mapping from dimension names to Python slice objects indicating where the data should be written (in index space, not coordinate space), e.m.,
# For convenience, we'll slice a unmarried dataset, merely in the existent use-case # we would create them separately possibly fifty-fifty from separate processes. In [54]: ds = xr . Dataset ({ "foo" : ( "x" , np . arange ( 30 ))}) In [55]: ds . isel ( x = slice ( 0 , 10 )) . to_zarr ( path , region = { "ten" : slice ( 0 , 10 )}) Out[55]: <xarray.backends.zarr.ZarrStore at 0x7f1bb8f079e0> In [56]: ds . isel ( ten = piece ( 10 , twenty )) . to_zarr ( path , region = { "x" : piece ( 10 , 20 )}) Out[56]: <xarray.backends.zarr.ZarrStore at 0x7f1bb8f07ac0> In [57]: ds . isel ( x = slice ( twenty , thirty )) . to_zarr ( path , region = { "x" : slice ( 20 , 30 )}) Out[57]: <xarray.backends.zarr.ZarrStore at 0x7f1bba8c84a0> Concurrent writes with region are safe equally long equally they change distinct chunks in the underlying Zarr arrays (or employ an appropriate lock ).
As a safety check to brand it harder to inadvertently override existing values, if you set region then all variables included in a Dataset must take dimensions included in region . Other variables (typically coordinates) need to be explicitly dropped and/or written in a split up calls to to_zarr with manner='a' .
GRIB format via cfgrib¶
Xarray supports reading GRIB files via ECMWF cfgrib python driver, if information technology is installed. To open a GRIB file supply engine='cfgrib' to open_dataset() :
In [58]: ds_grib = xr . open_dataset ( "example.grib" , engine = "cfgrib" ) We recommend installing cfgrib via conda:
conda install - c conda - forge cfgrib Formats supported by PyNIO¶
Xarray can also read GRIB, HDF4 and other file formats supported past PyNIO, if PyNIO is installed. To utilise PyNIO to read such files, supply engine='pynio' to open_dataset() .
We recommend installing PyNIO via conda:
conda install - c conda - forge pynio Warning
PyNIO is no longer actively maintained and conflicts with netcdf4 > 1.five.3. The PyNIO backend may exist moved outside of xarray in the future.
Formats supported by PseudoNetCDF¶
Xarray tin also read CAMx, BPCH, ARL PACKED BIT, and many other file formats supported by PseudoNetCDF, if PseudoNetCDF is installed. PseudoNetCDF can also provide Climate Forecasting Conventions to CMAQ files. In improver, PseudoNetCDF can automatically register custom readers that subclass PseudoNetCDF.PseudoNetCDFFile. PseudoNetCDF tin can identify readers either heuristically, or by a format specified via a key in backend_kwargs.
To employ PseudoNetCDF to read such files, supply engine='pseudonetcdf' to open_dataset() .
Add backend_kwargs={'format': '<format name>'} where <format proper noun> options are listed on the PseudoNetCDF page.
CSV and other formats supported by pandas¶
For more options (tabular formats and CSV files in detail), consider exporting your objects to pandas and using its wide range of IO tools. For CSV files, one might likewise consider xarray_extras.
Tertiary party libraries¶
More than formats are supported by extension libraries:
-
xarray-mongodb: Shop xarray objects on MongoDB
Source: https://xarray.pydata.org/en/stable/user-guide/io.html
0 Response to "How to Read Fill Value From Xarray Data Array"
Postar um comentário