Installation

Prerequisits

This toolkit requires the following Python packages.

paramiko, a networking package to run a local service the I/O library can talk to to establish a remote connection
pyyaml, parsing yaml configuration files
PyNaCl for encoding metadata with keys
pillow for processing images
adios2 for adding ADIOS data
h5py for adding HDF5 data
redis for hpc_campaign cache to list local cache information (Redis is used by ADIOS2 for caching)

Besides these packages, Python needs support for Tcl/Tk (python3-tk), so that hpc_campaign connector can pop up a dialog for password entries.

How to install

The easiest way to install HPC Campaign Management is with pip3.

(.venv) $ pip3 install hpc-campaign
(.venv) $ hpc_campaign ls

HPC Campaign Management is just a bunch of Python scripts that can be installed in your Python environment from the repository,

(.venv) $ git clone https://github.com/ornladios/hpc-campaign.git
(.venv) $ cd hpc-campaign
(.venv) $ pip3 install -e .
(.venv) $ hpc_campaign ls

or can be accessed without installation by setting PYTHONPATH

$ git clone https://github.com/ornladios/hpc-campaign.git
$ cd hpc-campaign
$ export PYTHONPATH=${PWD}:$PYTHONPATH
$ python3 -m hpc_campaign ls

Versions

ACA 0.6 version is compatible with ADIOS 2.11.0

ACA 0.7 version is compatible with ADIOS 2.12.0

Setup

~/.config/hpc-campaign/config.yaml

There are three paths/names important in the campaign setup.

hostname is used in the campaign archive. Define a specific name the project agrees upon (e.g. OLCF, NERSC, ALCF) that identifies the generic location of the data and then use that name later to specify the modes of remote data access.
campaignstorepath is the directory where all the campaign archives are stored. This should be shared between project members in a center, and a private one on every member’s laptop. It is up to the project to determine what file sharing / synchronization mechanism to use to sync this directories.
cachepath is the directory where ADIOS can unpack metadata from the campaign archive so that ADIOS engines can read them as if they were entirely local datasets. The cache contains the metadata as well as data that have already been retrieved by previous read requests.

Use ~/.config/hpc-campaign/config.yaml to specify these options.

$ cat ~/.config/hpc-campaign/config.yaml

Campaign:
  hostname: OLCF
  campaignstorepath: /lustre/orion/csc143/proj-shared/campaign-store
  cachepath: /lustre/orion/csc143/proj-shared/campaign-cache
  verbose: 0

$ ls -R /lustre/orion/csc143/proj-shared/campaign-store
/lustre/orion/csc143/proj-shared/campaign-store/demoproject:
frontier_gray-scott_100.aca

$ hpc_campaign ls
demoproject/frontier_gray-scott_100.aca

~/.config/hpc-campaign/hosts.yaml

The campaign archive records locations of datasets as a host name plus directory and optionally a default protocol like https. The host name is an artifical identifier, and ADIOS needs to be told how to access data on that resource.

Use ~/.config/hpc-campaign/hosts.yaml to specify these access settings. ADIOS can use

SSH tunnels to launch a private server for the session. For this you need to have an account on that resource, and have ADIOS2 installed including the adios2_remote_server executable.
xRootD service, using various “transfer protocols” to reach the xRootD service (http, https or xroot protocols)
S3 protocol to reach Amazon AWS cloud as well as other S3-compatible storages. Set up an AWS profile (the S3 access key id and secret key in ~/.aws/credentials) and use that profile name here.

$ cat ~/.config/hpc-campaign/hosts.yaml

OLCF:
    dtn-ssh:
        protocol: ssh
        host: dtn.olcf.ornl.gov
        user: user007
        authentication: passcode
        serverpath: ~/dtn/sw/adios2/bin/adios2_remote_server -background -report_port_selection -v -v -l /ccs/home/user007/dtn/log.adios2_remote_server -t 16
        verbose: 1

NERSC:
    dtn-ssh:
        protocol: ssh
        host: dtn.nersc.gov
        user: user007
        authentication: publickey
        identity_file: ~/.ssh/nersc
        serverpath: ~/adios/master/dtn/bin/adios2_remote_server -background -report_port_selection -v -v -l /global/homes/u/user007/dtn/log.adios2_remote_server -t 16
        verbose: 1

ClusterViaJump:
    dtn-ssh:
        protocol: ssh
        host: login.example.org
        user: user007
        authentication: passcode
        jumphost: jump.example.org
        jumpuser: user007
        jumpidentity_file: ~/.ssh/jump_key
        serverpath: ~/dtn/sw/adios2/bin/adios2_remote_server -background -report_port_selection
        verbose: 1

UEDGE.Spin:
    spin:
        protocol: xrootd
        transfer_protocol: https  # http, https or xrootd (default is xrootd)
        host: adios2-xrootd-ingress.adios2-remote.development.svc.spin.uedge.org
        port: 443
        verbose: 0

Robinia:
    s3-transport:
        protocol: s3
        profile: robinia
        endpoint: https://projects.handsey.org.au
        AWS_EC2_METADATA: false
        recheck_metadata: false
        verbose: 0