swarmpal.io._paldata ==================== .. py:module:: swarmpal.io._paldata .. autoapi-nested-parse:: PalData tools for containing data Attributes ---------- .. autoapisummary:: swarmpal.io._paldata.logger Classes ------- .. autoapisummary:: swarmpal.io._paldata.PalDataItem swarmpal.io._paldata.PalMeta swarmpal.io._paldata.PalDataTreeAccessor swarmpal.io._paldata.PalProcess Functions --------- .. autoapisummary:: swarmpal.io._paldata.create_paldata Module Contents --------------- .. py:data:: logger .. py:class:: PalDataItem(fetcher: swarmpal.io._datafetchers.DataFetcherBase) An Item (i.e. dataset) that will be stored within a PalData object Intended use is through the `.from_...` creator tools :param fetcher: :type fetcher: DataFetcherBase .. rubric:: Examples >>> from swarmpal.io import PalDataItem >>> >>> # Create the item with a configuration >>> params = dict( >>> collection="SW_OPER_MAGA_LR_1B", >>> measurements=["F", "B_NEC"], >>> start_time="2016-01-01T00:00:00", >>> end_time="2016-01-02T00:00:00", >>> server_url="https://vires.services/ows", >>> ) >>> item = PalDataItem.from_vires(**params) >>> # "Initialise" - triggers the expensive part, fetching the data >>> item.initialise() >>> # Data is available as an xarray.Dataset >>> item.xarray >>> # or as a DataTree >>> item.datatree .. py:property:: xarray :type: xarray.Dataset | None xarray.Dataset containing the data, generated if not already present .. py:attribute:: _fetcher .. py:property:: dataset_name :type: str Name of the dataset, used as the datatree label .. py:property:: datatree :type: xarray.DataTree Create a new datatree containing only this dataset; labelled with the dataset name. .. py:property:: analysis_window :type: tuple[datetime.datetime] The start and end times of the analysis window (considering optional padding .. py:property:: magnetic_models :type: dict Dictionary of model names and details .. py:method:: _serialise_pal_metadata() .. py:method:: initialise() Trigger the fetching of the data and attach PAL metadata .. py:method:: _ensure_datetime(times: tuple[datetime.datetime | str]) -> tuple[datetime.datetime] :staticmethod: Converts times to datetimes if they are not already .. py:method:: _datetime_to_str(times: tuple[datetime.datetime]) -> tuple[str] :staticmethod: Convert datetime objects to iso strings .. py:method:: _get_start_end_times(params: dict) -> tuple[dict, tuple[datetime.datetime]] :staticmethod: Get start and end times from the job parameters Accounts for difference between VirES ("start_time", "end_time"), and HAPI ("start", "stop") .. py:method:: _update_start_end_times(params: dict, start: str, end: str) :staticmethod: Update the job parameters with new (start, end) times .. py:method:: _pad_times(params: dict) -> tuple[dict, tuple[datetime.datetime]] :staticmethod: Use job parameters to adjust time window Returns the modified params dictionary, as well as the analysis window .. py:method:: from_vires(**params) -> PalDataItem :staticmethod: Create PalDataItem from VirES source :param collection: :type collection: str :param measurements: :type measurements: list[str] :param start_time: :type start_time: str | datetime :param end_time: :type end_time: str | datetime :param models: :type models: list[str] :param auxiliaries: :type auxiliaries: list[str] :param sampling_step: :type sampling_step: str :param filters: :type filters: list[str] :param options: :type options: dict :param server_url: defaults to "https://vires.services/ows" :type server_url: str :param pad_times: This is handled specially by SwarmPAL and not passed to viresclient :type pad_times: tuple[timedelta] .. py:method:: from_hapi(**params) -> PalDataItem :staticmethod: Create PalDataItem from HAPI source :param server: :type server: str :param dataset: :type dataset: str :param parameters: :type parameters: str :param start: :type start: str :param stop: :type stop: str :param options: :type options: dict :param pad_times: This is handled specially by SwarmPAL and not passed to hapiclient :type pad_times: tuple[timedelta] .. py:method:: from_file(filename: os.PathLike | None = None, group: str | None = None, filetype: str = 'auto', **params) -> PalDataItem :staticmethod: Create a PalDataItem from a file :param filename: Path to the (netCDF / CDF) file to load :type filename: PathLike :param group: Group name within the netCDF file :type group: str :param filetype: Type of file to load. Can be "netcdf", "cdf" or "auto". If "auto", the file extension will be used to determine the type. :type filetype: str :param params: Additional parameters to pass to the fetcher :type params: dict :rtype: PalDataItem .. py:method:: from_manual(xarray_dataset: xarray.Dataset | None = None, **params) -> PalDataItem :staticmethod: Create a PalDataItem manually from an existing xarray Dataset :param xarray_dataset: An existing xarray.Dataset :type xarray_dataset: Dataset :rtype: PalDataItem .. py:class:: PalMeta .. py:method:: serialise(meta: dict) -> str :staticmethod: .. py:method:: deserialise(meta: str) -> dict :staticmethod: .. py:class:: PalDataTreeAccessor(datatree) Provide custom attributes and methods on XArray DataTrees for SwarmPAL functionality. See e.g. https://github.com/Unidata/MetPy/blob/main/src/metpy/xarray.py .. py:attribute:: _datatree .. py:method:: apply(palprocess: PalProcess) -> xarray.DataTree .. py:method:: quicklook(toolbox: str | None = None) .. py:property:: pal_meta :type: dict .. py:property:: magnetic_model_names :type: list[str] List of the model names used in the dataset .. py:property:: magnetic_model_name :type: str Model name if one and only one has been set .. py:method:: magnetic_residual(model: str = '') -> xarray.DataArray Magnetic data-model residual in NEC frame .. py:method:: to_cdf(filename: str, leaf: str, handler: str = 'pycdfpp') -> None Write one leaf of the datatree to a CDF file :param filename: Name of the file to create :type filename: str :param leaf: Location within the datatree :type leaf: str :param handler: CDF handler to use. Can be "pycdfpp" or "cdflib". Defaults to "pycdfpp". :type handler: str .. py:function:: create_paldata(*paldataitems: PalDataItem, **paldataitems_kw: PalDataItem) -> xarray.DataTree Generates a Datatree from a number of PalDataItems :returns: A Datatree containing Datasets defined from each PalDataItem :rtype: Datatree .. rubric:: Examples >>> from swarmpal.io import create_paldata, PalDataItem >>> >>> # Parameters to control a particular data request >>> data_params = dict( >>> collection="SW_OPER_MAGA_LR_1B", >>> measurements=["B_NEC"], >>> models=["IGRF"], >>> start_time="2016-01-01T00:00:00", >>> end_time="2016-01-01T03:00:00", >>> server_url="https://vires.services/ows", >>> options=dict(asynchronous=False, show_progress=False), >>> ) >>> # Create the datatree from a list of items >>> data = create_paldata( >>> PalDataItem.from_vires(**data_params) >>> ) >>> # Create the datatree from labelled items >>> data = create_paldata( >>> one=PalDataItem.from_vires(**data_params), >>> two=PalDataItem.from_vires(**data_params), >>> ) .. py:class:: PalProcess(config: dict | None = None, active_tree: str = '/', inplace: bool = True) Bases: :py:obj:`abc.ABC` Abstract class to define processes to act on datatrees .. py:attribute:: _active_tree :value: '/' .. py:property:: process_name :type: str :abstractmethod: .. py:method:: set_config(output_dataset: str = '', **kwargs) :abstractmethod: .. py:property:: output_dataset .. py:property:: active_tree :type: str Defines which branch of the datatree will be used .. py:property:: config :type: dict Dictionary that configures the process behaviour .. py:method:: __call__(datatree) -> xarray.DataTree Run the process, defined in _call, to update the datatree .. py:method:: _call(datatree) -> xarray.DataTree :abstractmethod: