Usage in Python
===============

.. code-block:: python

  from hpc_campaign.manager import Manager
  from hpc_campaign.ls import ls
  from hpc_campaign.rm import rm

`Manager` is the class for creating, modifying, and viewing campaign archive files (`.aca`). It enables full lifecycle management for datasets, replicas, and archival storage locations.

A campaign archive name without the required `.aca` extension will be automatically corrected. Relative paths for archive names are resolved using the `campaignstorepath` defined in `~/.config/hpc-campaign/config.yaml` unless directly specified when creating a Manager object. 


**Global Usage and Options**

The manager command is invoked using the following general format:

.. code-block:: python

  manager = Manager("demoproject/test_campaign_001")
  manager.open(create=True)
  ...
  manager.close()


.. code-block:: python

  manager = Manager(archive=str(acafile), campaign_store=str(campaign_store))


The following options are available globally for the manager command to overwrite the default options:

* `campaign_store` specifies the path to the local campaign store used by the campaign manager instead of the default path set in `~/.config/hpc-campaign/config.yaml`
* `hostname` provides the host name, which must be unique for hosts within a campaign used by the campaign manager instead of the default hostname set in `~/.config/hpc-campaign/config.yaml`
* `keyfile` specifies the key file used to encrypt metadata in all subsequent data/image/text operation


**Manager sub-commands**

The [sub-command] argument can take one of the following values

* **add_archival_storage** Register an archival location (tape system, https, s3)
* **data** Add ADIOS2 or HDF5 files
* **image** Add images, embedded or remote optionally with an embedded thumbnail image
* **text** Add text files, embedded or just reference to remote file
* **add_time_series** Organizing a series of individual data files as a single entry with extra dimension for time
* **archived_replica** Create a replica of a dataset pointing to an archival storage location
* **close** Close the campaign archive after all operations.
* **delete_name** Delete a dataset by name (all replicas)
* **delete_replica** Delete a replica of a dataset
* **delete_time_series** Delete a time-series definition (but not the data objects themselves)
* **delete_uuid** Delete a dataset by UUID (all replicas)
* **info** List the content of a campaign archive
* **open** Open/create a campaign archive, optionally wipe all content to start afresh
* **upgrade** For upgrading an older ACA format to newer format


**1. add-archival-storage**

Records an archival storage location (e.g., tape system) to the list of known storage locations for the campaign. HTTPS and S3 server locations are considered to be archival locations for campaign management, since we cannot add datasets directly on these servers. 

The following `system` types are allowed and understood by ADIOS2 readers: 
* `https` for a file/tar accessible on a public HTTPS server
* `s3` for a file/tar accessible with the S3 protocol
* `fs` for a file/tar on a readable file system location
* `hpss` for the HPSS tape archive system - it cannot be read by ADIOS2
* `kronos` for the Kronos disk/tape archive system - typically readable by ADIOS2

Other `system` types that are allowed but for now not understood by ADIOS2 readers
* `http`
* `ftp`

.. code-block:: python

  manager.add_archival_storage(system = "fs", host="faketape", directory="/mnt/d/archive")

  host_id, dir_id, archive_id = manager.add_archival_storage(
    system = "https", 
    host="USERS.NCCS", 
    directory="~pnorbert/campaign-test",
    tarfilename="gray-scott-ensemble.tar"
    tarfileidx="gray-scott-ensemble.tar.idx"
    longhostname="users.nccs.gov",
    note="This is an example TAR file stored on an HTTPS server"
  )

Replicas of datasets then can be created (in the campaign file) by the `archived_replica` function. Note that hpc-campaign does not copy/move files on disk, someone else has to do that. These commands only record the action into the campaign archive file. 

If we put a TAR file there instead of individual files, we can just point to that, and use `archived_replica` one by one for each dataset. However, it is easier and provides more information, if we create an index file with the `hpc_campaign taridx` CLI command, then let the manager to create (i.e. record) a new replica for every dataset/replica already in the campaign archive that is also in the tar index. Moreover, if this index is available, ADIOS2 can read BP files, images and text directly from the tar file without the need of extracting them first (on file systems and https servers, that is).

**Return values**: `host_id`, `directory_id`, `archive_id`, use the directory_id and archive_id in follow-up `archived_replica` calls to identify this archival storage. Technically, `archive_id` is only needed if one host has multiple archive-storages under the same directory (e.g. individual files plus a TAR file in the same location for storing two replicas).

**2. data**

Adds one or more data files to the archive: currently valid HDF5 or ADIOS2 BP files are accepted.

.. note::

  A temporary file is created from HDF5 files during processing, so write access to the ``/tmp`` directory is required.


Example usage:

.. code-block:: python

  manager.data("data/heat.bp", name="heat")
  manager.data(["data/run_001.h5", "data/run_002.h5", "data/run_003.h5"])

Additional option (`name="<NAME>"`) can specify the representation name for one dataset in the campaign hierarchy. The same option can be applied to the text and image sub-commands.

**3. image**

Add an image file to the archive. By default, only a remote reference is stored for image files but it can be stored (`store=True``) or a thumbnail with a smaller resolution can be stored (`thumbnail=[64, 64]`)

Example usage:

.. code-block:: python

    manager.image("data/T0.png", name="T0")
    manager.image("data/T1.png", name="T1", store=True)
    manager.image("data/T2.png", name="T2", thumbnail=[64, 64])

Additional options for images include:
* `name="<NAME>"` representation name for the image in the campaign hierarchy

**Derived visualizations**

Use ``visualization`` when an image or image sequence is derived from variables
in a data set already in the campaign. The method adds the image data and also
records explicit metadata that links the visualization back to the source data
set and variable uses. This is useful in notebooks where a user reads a
variable, performs analysis, creates figures, and wants to save those figures
back into the campaign.

File-backed image inputs follow the same storage rule as ``image``: by default
the campaign records a reference to the image file, and ``store=True`` embeds
the image bytes in the archive. In-memory image inputs such as PNG/JPEG bytes,
PIL Images, and matplotlib Figures have no external file path, so they require
``store=True``.

Example usage:

.. code-block:: python

  manager.data("runs/run001/output.bp", name="run001/output.bp")

  # png_frames can be paths, PNG/JPEG bytes, PIL Images, or matplotlib Figures.
  # replace=True makes rerunning a notebook cell update the same visualization.
  manager.visualization(
      images=png_frames,
      source_dataset="run001/output.bp",
      name="density",
      kind="heatmap",
      color_by="rho",
      steps=[0, 1, 2],
      store=True,
      replace=True,
  )

  manager.visualization(
      images=overlay_frames,
      source_dataset="run001/output.bp",
      name="density_current_overlay",
      kind="heatmap-contour",
      color_by="rho",
      contour_by="current_z",
      replace=True,
  )

  manager.visualization(
      images=line_plot,
      source_dataset="run001/output.bp",
      name="mass_over_time",
      kind="line-plot",
      x_axis="time",
      y_axis="mass",
      replace=True,
  )

If ``name`` is a short token, the visualization sequence is stored under
``<source_dataset>/visualizations/<name>``. For advanced use, pass
``sequence_name`` to provide the exact sequence name. If image names are not
provided explicitly, they are generated as children of the sequence name, using
``steps`` when supplied.

**4. text**

Add one or more text files to the archive. If requested, text files are stored within the archive. In that case, zlib is used to compress the text file.

.. note::

  When storing text internally, be mindful of the resulting archive's size when adding large text files.

Example usage:

.. code-block:: python

  manager.text("input.json", name="input", store=True)
  file = Path("data/readme")
  manager.text(file, name="readme", store=True)


Additional options for text include:
* `name="<NAME>"` representation name for the image in the campaign hierarchy
* `store=True` stores the text file directly in the campaign archive instead of just a reference.

**5. add_time_series**

Organizes a sequence of datasets into a single named time-series. Subsequent calls with the same name will add datasets to the list, unless `replace=True` is used.

.. code-block:: python

  manager = Manager("test.aca")
  manager.open(create=True)
  manager.data("series/array00.bp", name="array00")
  manager.data("series/array01.bp", name="array01")
  manager.data("series/array02.bp", name="array02")
  manager.add_time_series("array", ["array00", "array01", "array02"])

  manager.data("series/array03.bp", name="array03")
  manager.add_time_series("array", ["array03"])
  manager.close()
  

.. code-block:: python

  hpc_campaign manager test.aca info
  ...
  Time-series and their datasets:
  array
    89635fe22f85314ebfc04c902bca42f3  ADIOS  Jun  9 08:49   array00
    cea0302ea4ce39ccabca6b40bbeb09d1  ADIOS  Jun  9 08:49   array01
    fad5daf925e13f938c2649d81a1821f2  ADIOS  Jun  9 08:49   array02
    180b6d3123a832d786e1d0ff99c7e303  ADIOS  Jun  9 08:49   array03

  Other Datasets:
  ...

  # ADIOS tools will present them as a single dataset with multiple steps
  bpls -l  test.aca array/*
    int64_t  array/Nx                          4*scalar = 10 / 10
    double   array/bpArray                     4*{10} = 0 / 9
    double   array/time                        4*scalar = 0 / 0

Additional options for text include:

* `replace=True` redefine the time-series starting with the current command


**6. archived_replica**

Indicates that a (replica of a) dataset has been manually copied or moved to an archival storage location. A new replica entry is created pointing to the archival host/directory. This sub-command only works if the metadata of the dataset is still included in the ACA file, so that it can be copied for the new replica. Therefore, always execute this sub-command before deleting the original replica from the ACA file. The two operations can be combined using the `move=True` option. This sub-command requires the use of `add_archival_storage` sub-command that adds the location (host/directory/tar file) to the campaign first. If many files are added to the archival location in a TAR file, it is better to use the `taridx` command to create an index of the tar file and then use that in the `add_archival_storage` operation to automatically create replicas of all datasets involved. However, this individual sub-command allows for placing the replica in a different relative path string than the original, while the tar indexing requires them to be placed exactly with the same relative paths. 

.. code-block:: python

    _host_id, dir_id, archive_id = manager.add_archival_storage("fs", "faketape", "/mnt/d/archive")
    manager.archived_replica("heat", dir_id, archiveid=archive_id, newpath="archivedheat.bp")


Additional options for text include:

* `archiveid:int` redefine the time-series starting with the current command
* `newpath` use this option if the replica's relative path under the archival directory is different from the relative path of the original replica under its own directory, 
* `replica:int` if a dataset has multiple replicas already, you must tell which one has been actually replicated for this record
* `move=True` indicate that the replica has been moved to new location, not copied. The original replica will be removed from the campaign archive. 


**7. close** 

Close the campaign archive after all operations. It is only for freeing up resources. Since all operations commit their changes, there is nothing left for close(). 


**8. delete_name**

Delete specific dataset (data, image, text) from a campaign archive file referring to their representation name. All replicas are deleted along with all embedded files. 
Example usage:

.. code-block:: python

  manager.delete_name("array00")

**9. delete_replica**

Delete specific replica of a dataset (data, image, text) from a campaign archive file. Embedded files are deleted only if it is the only replica referring to a file.
Example usage:

.. code-block:: python

  manager.delete_replica(12)

**10. delete_time_series**

Delete the time series definition (of a series of data object) from a campaign archive file. The data objects are not deleted, only this extra definition
Example usage:

.. code-block:: python

  manager.delete_time_series("array")


**11. delete_uuid**

Delete specific dataset (data, image, text) from a campaign archive file. All replicas are deleted along with all embedded files. 
Example usage:

.. code-block:: python

  manager.delete_uuid("180b6d3123a832d786e1d0ff99c7e303")

**12. info**

Prints the content and metadata of a campaign archive file.
Example usage:

.. code-block:: bash

  hpc_campaign manager demoproject/test_campaign_001 info [options]

The optional options allow listing replicas, entries that have been deleted and checksums. A complete list of options can be found in the help menu (`-h` option).


**13. open**

Open is the first function to be called after the manager object is created. It opens the campaign archive file (opens the database connection). By default, it will raise a FileNotFound error if the campaign archive does not exists. The `create=True` option will result in creating the campaign archive if it does not exists. 
The `truncate=True` option will wipe the content of an existing campaign archive to start afresh. 

.. code-block:: python

    manager = Manager(archive=str(api_archive), campaign_store=str(campaign_store))
    manager.open(create=True, truncate=True)

Additional options for opem include:
* `create=True` to create a new campaign archive if it does not exist already
* `truncate=True` if the campaign archive already exist, open and wipe its content


**14. upgrade**

An ADIOS2 release will only read the latest ACA version and throw errors if an older ACA files is opened. The `upgrade` sub-command will modify the old ACA to jump to the next version. It may be called multiple times to get to the current version. This is an in-place conversion. If an error occurs during conversion, all changes are cancelled, leaving the original file intact. 

.. code-block:: python

    manager = Manager(archive=str(api_archive), campaign_store=str(campaign_store))
    manager.open(create=True, truncate=True)
    new_version = manager.upgrade()
    print(new_version)


**Example creating an archive campaign file**

In this example we will create an archive campaign file with:
- the text json input file for a simulation
- the data generated by the simulation code
- analysis data generated by a code that reads the simulation data and produces histograms
- the images generated by a visualization code on the simulation data

Configuration:
- the `campaignpath` in `~/.config/hpc-campaign/config.yaml` is set to `/path/to/campaign-store`
- the path `/path/to/campaign-store/demoproject` is writable directory 
- the runs are made on a machine named OLCF in the Campaign hostname in `~/.config/hpc-campaign/config.yaml`
- all the files above are generated and stored in `${pwd}/runs`

.. code-block:: python

    manager = Manager(archive="demoproject/test_campaign_001")
    manager.open(create=True, truncate=True)
    manager.text("runs/input-configuration.json", store=True)
    manager.data(["runs/simulation-output.bp" ,"runs/simulation-chekpoint.bp"])
    manager.data("analysis/pdf.bp")
    manager.image("analysis/plot-2d.png", store=True)

    info_data = manager.info(True, False, False, False)
    output = format_info(info_data)
    print(output)
    manager.close()


.. code-block:: bash

  ADIOS Campaign Archive, version 0.7, created on Oct 18 14:29

  Hosts and directories:
    OLCF   longhostname = frontier05341.frontier.olcf.ornl.gov
      1. /path/to/simulation

  Other Datasets:
      3a4bf0b14cc33424a470862bd67ed007  TEXT   Oct 18 14:25   runs/input-configuration.json
      0fce4b1173f432f7ae5d2282df9077a6  ADIOS  Oct 18 14:25   runs/simulation-output.bp
      aa5d2282df9077a60fc643f5ab53b351  ADIOS  Oct 18 14:26   runs/simulation-chekpoint.bp
      b42d0da4a0793adca341ace1ff6e628d  ADIOS  Oct 18 14:28   analysis/pdf.bp
      85a0b724b22f37a4a79ad8a0cf1127d1  IMAGE  Oct 18 14:24   analysis/plot-2d.json


Comparing the campaign archive size to the data it points to can be done by the default method on each operating system.

.. code-block:: bash

  $ du -sh runs/*bp
  263M    simulation-chekpoint.bp
  3.8G    simulation-output.bp

  $ du -sh /path/to/adios-campaign-store/demoproject/test_campaign_001 info.aca
  127K     /path/to/adios-campaign-store/demoproject/test_campaign_001 info.aca