CLI for interacting with the timer API
Project description
Globus Automate Timer CLI
This is a beta-version CLI for use with the (also in beta) Globus Timer API, to use primarily for scheduling recurring Globus transfers through Globus Automate.
As the CLI and service are still in beta, please feel free to email support at globus dot org with feedback or to resolve issues.
What is this Service/CLI for?
The Globus Timer service can be used to schedule recurring transfer tasks. For example, let's say we want to have a transfer automatically run every night to back up data. We submit a job to the Timer Service starting tonight, and with an interval of 1 day at which it will be re-run. In that request we provide the Timer service the same input we would give to the Globus Transfer Action Provider; that part of the request contains the information for what Globus endpoints we transfer to and from as well as other options relevant to the transfer.
Installation
This CLI requires Python version 3.5 or higher. See the Globus CLI docs for help on how to set up Python.
Once the appropriate version of Python is ready, install with pip install globus-timer-cli
. This will create a new command line
utility, globus-timer
which can be used for all interactions with
the service. Online documentation is always available using the
--help
option on any command. So, globus-timer --help
will provide
information about the options on the command while globus-timer job --help
will provide help text specific to working with jobs. When in
doubt, add --help
to a command for guidance on next steps.
Getting Started
To schedule transfers on your behalf, this CLI requires authentication through Globus Auth. Initially, the CLI will prompt you to authorize use via the Globus Auth system, with a link to open in a web browser. The page will request that you login to your Globus identity and consent to the service looking up your identity and interacting with the Globus Transfer service on your behalf, and then give you an authorization code to copy and paste back into the CLI. Some Globus endpoints require additional authentication for use; when this is necessary, the CLI will repeat the above process to get consent for using the Globus Transfer service and the specific endpoint(s) for that job.
Authentication information is thereafter cached in the file
~/.globus_timer_tokens.json
(so the authentication process is only needed on
the first use of the CLI, or of a given transfer endpoint when required); keep
this file secret.
After first use, you can determine what Globus Auth identity is being used for interacting with the service by running the command:
globus-timer session whoami
This will show the identity which will be used for running the Transfers. Be sure that this identity is authorized to work with the endpoints involved in the Transfer.
To remove your stored identity information so that you may re-authenticate, for example to invoke subsequent operations using a different Globus Auth identity, use the command:
globus-timer session logout
Note that should you wish to both logout, and revoke the Timer service's
permission to run further operations, you may use the command globus-timer session revoke
however this should be an extreme measure as it is much
preferred to properly delete jobs as described below.
Scheduling Periodic Globus Transfer operations
NOTE: To avoid confusion, please read the entirety of this section before using
the job transfer
sub-command.
The starting point for working with the Timer service is scheduling transfers to run periodically. An example of this command is below:
globus-timer job transfer \
--name example-job \
--label "Timer Transfer Job" \
--interval 28800 \
--start '2020-01-01T12:34:56' \
--source-endpoint ddb59aef-6d04-11e5-ba46-22000b92c6ec \
--dest-endpoint ddb59af0-6d04-11e5-ba46-22000b92c6ec \
--item "/share/godata/file1.txt" "~/new_file1.txt" false \
--item "/share/godata/file2.txt" "~/new_file2.txt" false
The parameters to the job transfer
command are as follows (also documented in
the CLI help: globus-timer job transfer --help
):
-
name
: A friendly name you can use to identify the job. However, note that all operations on the job (see below) are based on the job identifier (job_id
) which is returned in the output of this command. -
label
: A label string which will be used on the Globus Transfer tasks which will be created to perform the periodic transfer operations. You may use this to help you identify the source of the Globus Transfer task when viewing them in the Globus web application. If nolabel
string is provided, the Globus Transfer label will include thename
of this Timer job. -
interval
: The time, in seconds, between invocations of the Globus Transfer operation. The Transfer will be invoked after this number of fixed seconds. Note that this easily allows for intervals such as hourly (3600 seconds), daily (86400 seconds) or weekly (604800 seconds), but it doesn't allow for day of month or provide the ability to compensate for other time related changes like daylight savings time changes. This is presently a limitation of the Timer service. In some cases, the actual time the operation is started in Globus Transfer may be a few seconds longer than the specified interval, but any delay will not impact scheduling of the next operation (i.e. the job will stay on schedule and not "drift" due to delays in any particular execution). -
start
: A timestamp defining when the job should first be scheduled. Thus, the first invocation of the Globus Transfer operation will occur at the specified time and then again after eachinterval
number of seconds. The time may contain just a year, month and date, or may also contain the exact time of day as shown in the example. Unless a timezone is specified, the local timezone will be used. -
source-endpoint
anddest-endpoint
: Are the ids for Globus Transfer endpoints for the source and destination of the transfer operation respectively. These values can be retrieved from the Globus web application. -
item
: Specifies the exact locations on the source and destination endpoints to perform the transfer operation.item
should be followed by three values: the path on the source endpoint from which to start the transfer, the path on the destination endpoint where the data should be placed, and a true or false value indicating whether the transfer should be performed in a recursive manner. When the value is true, if the source path names a folder, the folder and all of its contained files and other folders will be transferred. If the value is false, the first (source) path should refer to a specific file.
Note that, as shown in the example, a single job creation command can
contain multiple specifications of the item
parameter. This means
that a single job can transfer from multiple paths between a single
source endpoint and a single destination endpoint.
As an alternative to specifying the item
parameter one or more
times, the item values may be stored in a file in which each line has
the same format as the item
parameter. The file may be specified
with the items-file
option which is used instead of the item
parameter. An example of the file which would perform the same
operations as the example above would look like:
# This is my items file
/share/godata/file1.txt, ~/new_file1.txt, false
/share/godata/file2.txt, ~/new_file2.txt, false
If this file was named transfer_items.csv
it would be specified with the parameter:
--items-file transfer_items.csv
Note that this is a CSV file. The individual parts of the item are separated
with commas, and lines starting with a #
are considered to be comments and are
not processed as items for the transfer job.
Additional Transfer Options
Some optional flags for setting certain Transfer options include:
--sync-level
, adjusting how Transfer should behave if files already exist at the destination--verify-checksum
, causing Transfer to retry if file checksums don't match--encrypt-data
, causing Transfer to encrypt data using TLS--preserve-timestamp
, causing Transfer to keep file timestamps the same at the destination
You may also specify a finite number of times or duration after which a job should stop:
--stop-after-runs
takes a number, telling Timer to run the job only that many times.--stop-after-date
takes a date argument, telling Timer to continue running the job only until the given date.
When either condition is reached, your job will still exist, and results will still be visible, but Timer will not run any more transfers. If you specify both, the job stops running as soon as one stop condition is met. If you update an existing job to modify either of these, Timer will resume running the job until it hits your new stop condition.
As always, all of the above options are also documented in globus-timer job transfer --help
.
Monitoring and Controlling Submitted jobs
After submitting the transfer job, the CLI should return some results containing
a job_id
, which identifies this job in the Timer service. To check the
status of your jobs, use:
globus-timer job status <job_id>
This command defaults to a summarized version of the job's information, which
does not include the full details for the corresponding task in Globus Transfer. To
check those, use -v/--verbose
:
globus-timer job status --verbose JOB_ID
Commands return date-times in ISO format, in UTC time, so the timezone most likely will not match the local time where you run the command. No need to worry: the actual start time is still equal to your submission's start time, etc.
The job description will also contain a value for status
indicating
the condition of the job. The status
may contain one of the
following values:
-
new
: The job has been received, but it has not yet been scheduled for its first execution (for example the start time has not yet been reached). -
loaded
: The job has reached its execution mode and is executing repeatedly on the requested interval. This is the "steady-state" for a job when it is executing as expected. -
updated
: If the job has recently been updated, it will be in this state until the job has transitioned to its new regular operation. This state should rarely be encountered as it is a transient state immediately after a job update. -
deleted
: Indicates that the job has been deleted. This would not normally be seen, but thejob status
command described above as well as other commands described below provide the option of retrieving jobs marked as deleted using a command line flag--show-deleted
.
A final important note: Last Result
in the non-verbose output extends only as
far as the Globus Automate system: SUCCESS
indicates you have successfully submitted
your job to the Timer service, which in turn successfully sent the task to the
Globus Transfer Action Provider. It's possible that the Globus Transfer service will
subsequently encounter some error running your transfer. Check the --verbose
output, which includes the actual response from Transfer, to be certain that
Transfer has run your job successfully.
To see all of your jobs which are outstanding, use the command:
globus-timer job list
This is particularly helpful if you do not have the job_id
for a
particular job available. It will show all of your jobs in the same
format as the job status
command described above. Similarly, the
--verbose/-v
and --show-deleted
options are supported.
Finally, a job may be removed from operation using the command:
globus-timer job delete <job_id>
The job will be removed from operation and no further transfers will
be performed. The same information provided via job status
will be
displayed, showing the final state of the job. As described above, the
job status
command may still retrieve the job information by
providing the --show-deleted
flag. This can be helpful to review the
final state of the job, such as the last transfer or error condition,
even if the output for the job delete
command is lost.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for globus_timer_cli-0.2.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e389a836ee2d4f6e354937d44816638b0baaf42c4d9db809d0a4bbaf5939003 |
|
MD5 | 211e3034fe21e3dd8f11d6e9850e6840 |
|
BLAKE2b-256 | d02e733f887cae7af8c5664bc6107c5464c249809c442cb96d6cbfcf23fc8fde |