Microsoft Azure Batch Extended Features
Project description
Microsoft Azure Batch Extensions
This project is a preview build of the Microsoft Azure command-line interface to demonstrate proposed features in Azure Batch. For further details on the Azure CLI, please check the official documentation.
The purpose of this project is to allow customers to try out proposed Batch features and provide feedback to help shape the direction of the Batch service. The features presented here may not be compatible with other Batch client SDKs and tools, nor will they necessarily be adopted into the core Batch service.
As these features are still in preview, they will be updated regularly, and refined based on customer feedback. Unfortunately this may result in occasional breaking changes, though every effort will be made to keep this to a minimum.
Features
Input data upload to Batch linked storage accounts
A new command to allow a user to upload a set of files directly into the storage account linked to their Azure Batch account.
Input data references using linked storage accounts
Input data stored in linked storage under a file group can be simply referenced by a task by using some new ResourceFile properties.
Automatic persistence of task output files to a file group
When declaring task output files, you can now persist outputs to a named file group, without the need to generate a container SAS URL.
Download job outputs from directly from storage
A new command to allow a user to download job output files from a file group in the storage account linked to their Azure Batch account.
Pool and job templates with parameterization
Templates allow pools and jobs to be defined in parameterized json files with a format inspired by ARM templates.
Task factories for automatic task generation on job submission
Task factories provide a way for a job and all its tasks to be created in one command instead of calling azure batch task create for each task. There are currently three kinds of task factory:
Task Collection - tasks are explicitly defined as a part of the job
Parametric Sweep - a set of tasks are created by substituting a range or sequence of values into a template
Per File - a template task is replicated for each available input file
Split job configuration and management with reusable application templates
Application templates provide a way to partition the details of a job into two parts.
All of the details about how the job should be processed are moved into the application template, creating a reusable definition that is independent of a particular account. Application templates are parameterized to allow the processing to be customized without requiring modification of the template itself.
Easy software installation via package managers
Integration with existing 3rd party package managers to streamline the installation of applications. Currently the following package managers are supported:
Chocolatey - for Windows
APT - as used by some Linux distros including Ubuntu, Debian, and Fedora.
Yum - a package manager used by some Linux distros including Red Hat Enterprise Linux, Fedora, CentOS.
Samples
Samples for all of the preview features can be found in samples.
SDK Installation
$ pip install azure-batch-extensionsCLI Extensions Installation
In order to make use of these features, you must have the Azure CLI installed. You can find futher instructions in the official documentation and in the Azure CLI GitHub repository.
This extension package can be installed to supplement the existing Azure CLI Batch commands. It can be installed using the CLI extension tools:
$ az extension add --source [URL to latest release package]Azure Batch account requirements
In order to make use of the new features previewed here, you will need an Azure Batch account with a linked storage account. For more information on this, see Create an Azure Batch account using the Azure Portal.
Contributing
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
Developer Installation
Preparing your machine
Install Python 3.5.x from http://python.org. Please note that the version of Python that comes preinstalled on OSX is 2.7.
Clone your repository and check out the master branch.
Create a new virtual environment “env” for Python 3.5 in the root of your clone. You can do this by running:
Windows
python -m venv <clone root>\envOSX/Ubuntu (bash)
python –m venv <clone root>/env
Activate the env virtual environment by running:
Windows
<clone root>\env\scripts\activate.batOSX/Ubuntu (bash)
. <clone root>/env/bin/activate
Install the dependencies and load the command module as a local package using pip.
python scripts/dev_setup.py
SDK Release History
1.0.0 (2017-10-03)
Support for Batch SDK version 4.0
Added support for extended common_resource_files in MultiInstanceSettings
Added support for native containers in tasks (including RepeatTask in a task factory)
0.2.0 (2017-07-29)
Support for Batch SDK version 3.1
Fix bug with pool OS version detection
Download from file group now supports prefix
Support detection of Linux command using /bin/sh
0.1.1 (2017-07-10)
Fix to support azure-cli-core v2.0.11
0.1.0 (2017-06-28)
Initial preview release.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
File details
Details for the file azure_batch_extensions-1.0.0-py2.py3-none-any.whl.
File metadata
- Download URL: azure_batch_extensions-1.0.0-py2.py3-none-any.whl
- Upload date:
- Size: 84.0 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 90c43aa138460c0fc2276187afa61a035a67bf8a93c3ef9e30fb7939540427f6 |
|
| MD5 | fed1a6b3c63751c9c9beb4307f8920b4 |
|
| BLAKE2b-256 | c97cdf43d231244aaffdebbf49a026c8ca831eb65c44bc157fb355acf03d90a2 |
