Usage ===== As mentioned in the :doc:`quickstart ` page, EO Canvas allows you to use both high-level objects or to interact with the API in a more direct way. The two fundamental concepts of the Serverless Functions service are *processes* and *jobs*. A *process* describes either a SNAP or a Data Tailor computation. A *job*, on the other hand, is a submission of a process to the API, and it's used to keep track of its state, to retrieve the application logs and, finally, to download the results. .. note:: Users are not limited to use their local hard-drive to download the results, nor the HDA as source of data. Look at the :doc:`external storage ` page to learn more about it. To interact directly with the API, you can use the :class:`eocanvas.API` class: .. code-block:: python from eocanvas import API, Credentials # Without any argument, this will automatically load your credentials from the ~/.hdarc file api = API() # but it's possible to pass username and password on the fly as well api = API(Credentials("[username]", "[password]")) processes = api.get_processes() print(processes) >> [Process(api=, process_id='snap-function', ...] The available processes are predefined. New tools or custom functions might be added in future. Process inputs must be configured before submitting them. **SNAP** and **Data Tailor** has similar but different sets of inputs. SNAP ---- A **SNAP** process is configured through an XML object called a `Graph`. Such graph contains nodes for the inputs. Please refer to `official SNAP documentation `_ for further details. Since SNAP is being executed remotely, those inputs are defined as placeholders and then passed in through :class:`eocanvas.api.Input` objects. Extra configuration can be added using the :class:`eocanvas.api.ConfigOption` class. **EO Canvas** handles graphs through an embedded version of `Snapista `_. *Snapista* requires a local SNAP instance to work, to which it can submit the graph. The embedded version instead only allows to programmatically build or load the graph from an XML file. It also encodes it before it gets submitted to the API. Generally speaking, having pre-cooked graphs is a better option. The `demo` directory contains some examples. As already described in the :doc:`quickstart ` page, a graph instance can be created by loading a local XML file: .. code-block:: python from eocanvas.snap.graph import Graph graph = Graph.from_uri("olci_binding.xml") Please note how the input node has been defined: .. code-block:: xml Read false $img1 true Oa01_reflectance,Oa01_reflectance_err,Oa02_reflectance,Oa02_reflectance_err,Oa03_reflectance,Oa03_reflectance_err,Oa04_reflectance,Oa04_reflectance_err,Oa05_reflectance,Oa05_reflectance_err,Oa06_reflectance,Oa06_reflectance_err,Oa07_reflectance,Oa07_reflectance_err,Oa08_reflectance,Oa08_reflectance_err,Oa09_reflectance,Oa09_reflectance_err,Oa10_reflectance,Oa10_reflectance_err,Oa11_reflectance,Oa11_reflectance_err,Oa12_reflectance,Oa12_reflectance_err,Oa16_reflectance,Oa16_reflectance_err,Oa17_reflectance,Oa17_reflectance_err,Oa18_reflectance,Oa18_reflectance_err,Oa21_reflectance,Oa21_reflectance_err,CHL_NN,CHL_NN_err,CHL_OC4ME,CHL_OC4ME_err,altitude,latitude,longitude,detector_index,FWHM_band_1,FWHM_band_2,FWHM_band_3,FWHM_band_4,FWHM_band_5,FWHM_band_6,FWHM_band_7,FWHM_band_8,FWHM_band_9,FWHM_band_10,FWHM_band_11,FWHM_band_12,FWHM_band_13,FWHM_band_14,FWHM_band_15,FWHM_band_16,FWHM_band_17,FWHM_band_18,FWHM_band_19,FWHM_band_20,FWHM_band_21,frame_offset,lambda0_band_1,lambda0_band_2,lambda0_band_3,lambda0_band_4,lambda0_band_5,lambda0_band_6,lambda0_band_7,lambda0_band_8,lambda0_band_9,lambda0_band_10,lambda0_band_11,lambda0_band_12,lambda0_band_13,lambda0_band_14,lambda0_band_15,lambda0_band_16,lambda0_band_17,lambda0_band_18,lambda0_band_19,lambda0_band_20,lambda0_band_21,solar_flux_band_1,solar_flux_band_2,solar_flux_band_3,solar_flux_band_4,solar_flux_band_5,solar_flux_band_6,solar_flux_band_7,solar_flux_band_8,solar_flux_band_9,solar_flux_band_10,solar_flux_band_11,solar_flux_band_12,solar_flux_band_13,solar_flux_band_14,solar_flux_band_15,solar_flux_band_16,solar_flux_band_17,solar_flux_band_18,solar_flux_band_19,solar_flux_band_20,solar_flux_band_21,ADG443_NN,ADG443_NN_err,IWV,IWV_err,PAR,PAR_err,KD490_M07,KD490_M07_err,TSM_NN,TSM_NN_err,A865,A865_err,T865,T865_err,WQSF_lsb,WQSF_msb 0,0,4865,4091 WQSF_lsb_INVALID,WQSF_lsb_WATER,WQSF_lsb_LAND,WQSF_lsb_CLOUD,WQSF_lsb_TURBID_ATM,WQSF_lsb_CLOUD_AMBIGUOUS,WQSF_lsb_CLOUD_MARGIN,WQSF_lsb_SNOW_ICE,WQSF_lsb_INLAND_WATER,WQSF_lsb_COASTLINE,WQSF_lsb_TIDAL,WQSF_lsb_COSMETIC,WQSF_lsb_SUSPECT,WQSF_lsb_HISOLZEN,WQSF_lsb_SATURATED,WQSF_lsb_MEGLINT,WQSF_lsb_HIGHGLINT,WQSF_lsb_WHITECAPS,WQSF_lsb_ADJAC,WQSF_lsb_WV_FAIL,WQSF_lsb_PAR_FAIL,WQSF_lsb_AC_FAIL,WQSF_lsb_OC4ME_FAIL,WQSF_lsb_OCNN_FAIL,WQSF_lsb_KDM_FAIL,WQSF_lsb_BPAC_ON,WQSF_lsb_WHITE_SCATT,WQSF_lsb_LOWRW,WQSF_lsb_HIGHRW,WQSF_msb_ANNOT_ANGSTROM,WQSF_msb_ANNOT_AERO_B,WQSF_msb_ANNOT_ABSO_D,WQSF_msb_ANNOT_ACLIM,WQSF_msb_ANNOT_ABSOA,WQSF_msb_ANNOT_MIXR1,WQSF_msb_ANNOT_DROUT,WQSF_msb_ANNOT_TAU06,WQSF_msb_RWNEG_O1,WQSF_msb_RWNEG_O2,WQSF_msb_RWNEG_O3,WQSF_msb_RWNEG_O4,WQSF_msb_RWNEG_O5,WQSF_msb_RWNEG_O6,WQSF_msb_RWNEG_O7,WQSF_msb_RWNEG_O8,WQSF_msb_RWNEG_O9,WQSF_msb_RWNEG_O10,WQSF_msb_RWNEG_O11,WQSF_msb_RWNEG_O12,WQSF_msb_RWNEG_O16,WQSF_msb_RWNEG_O17,WQSF_msb_RWNEG_O18,WQSF_msb_RWNEG_O21,WQSF_REFLECTANCE_RECOM,WQSF_CHL_OC4ME_RECOM,WQSF_KD490_M07_RECOM,WQSF_PAR_RECOM,WQSF_W_AER_RECOM,WQSF_CHL_NN_RECOM,WQSF_TSM_NN_RECOM,WQSF_ADG443_NN_RECOM,WQSF_IWV_RECOM The *$img1* is an arbitrary placeholder and can be used to further configure the input: .. code-block:: python from eocanvas.api import Input, Config, ConfigOption inputs = Input(key="img1", url="http://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download/66b37374b6a632e1f39b3058") config = Config(key="img1", options=ConfigOption(uncompress=True, sub_path="xfdumanifest.xml")) Note that `img1` has been used as the key to configure the input. The dollar sign is only required in the graph. The `url` parameter is a valid WEkEO HDA download link and can be retrieved by using the HDA Python client: .. code-block:: python from hda import Client client = Client() query = { "dataset_id": "EO:EUM:DAT:SENTINEL-3:OL_2_WFR___", "dtstart": "2024-07-05T09:28:00.000Z", "dtend": "2024-07-05T09:30:00.000Z", "timeliness": "NT" } results = client.search(query) urls = results.get_download_urls() print(urls) >> ["http://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download/66b37374b6a632e1f39b3058", ...] inputs = Input(key="img1", url=urls[0]) The :class:`eocanvas.api.Config` object allows you to set two extra options: #. whether the product must be decompressed before it is passed to SNAP #. if uncompressed, what is the sub-path of the actual input file Once all the inputs are ready, you can instantiate a process class: .. code-block:: python from eocanvas.processes import SnapProcess process = SnapProcess(snap_graph=graph, eo_config=config, eo_input=inputs) Processes can be submitted to the Serverless Functions API that will return a `Job` object, used to check the status of the request. This can be done in two steps (useful if you want a reference to the job) or in one go: .. code-block:: python # Two steps job = process.submit() process.run(job) # Once you have the job, you can access the logs logs = job.logs # Otherwise you can directly call `run` and a job will be created under the hood process.run() The `run` method will block until the job is completed and the results downloaded locally. By default, results are downloaded in the current directory. A different one can be specified as well: .. code-block:: python process.run(dowload_dir="mydir") You can choose not to download the results instead. Just pass a flag to the process: .. code-block:: python process.run(dowload=False) In this case, the results can be downloaded at a later stage through the API: .. code-block:: python for result in job.results: result.download() Data Tailor ----------- A **Data Tailor** process is configured through an YAML object called a `Chain`. Please refer to `official Data Tailor documentation `_ for further details. Like SNAP, inputs are defined as placeholders and then passed in through :class:`eocanvas.api.Input` objects. **EO Canvas** handles Data Tailor chains through a lightweight class taken from the `official EUMDAC library `_ The :class:`eocanvas.datatailor.Chain` class can be instantiated from a file, or from a plain Python dictionary: .. code-block:: python from eocanvas.datatailor import Chain # Load it from a file chain = Chain.from_file("olci_resample.yaml") # Load it from a dictionary d = { 'product': 'OLL2WFR', 'filter': {'bands': ['chl_nn']}, 'projection': 'geographic', 'resample_resolution': [0.003, 0.003], 'format': 'netcdf4' } chain = Chain(**d) The inputs can be set similarly to SNAP .. code-block:: python inputs = Input(key="img1", url="http://gateway.prod.wekeo2.eu/hda-broker/api/v1/dataaccess/download/66c357dcb6a632e1f39b3131") Again, the url can be retrieved through the HDA Client as explained above. Once all inputs are set, you'd call a `DataTailorProcess` just as SNAP: .. code-block:: python from eocanvas.processes import DataTailorProcess process = DataTailorProcess(epct_chain=chain, epct_input=inputs) process.run() Data Tailor accepts a configuration paramater as well, similarly to SNAP. Please refer to the :class:`eocanvas.processes.DataTailorProcess` class. Shear Water ----------- Shearwater is a simple demo process based on a user application to show how other systems or user needs can be integrated into the API, such as AI models. The interface is quite simple, as it only accepts three parameters. Keep in mind that, at the moment, **Sindian** is the only valid value for the *area* parameter- .. code:: python from eocanvas.processes import ShearWaterProcess # No need to setup the input through other objects process = ShearWaterProcess(area="Sindian", start_day="2021-01-01", end_day="2021-01-02") process.run()