Batch Shipyard

Batch Shipyard is a tool to helpprovision, execute, and monitor container-based batch processing and HPCworkloads onAzure Batch. Batch Shipyardsupports both Docker andSingularity containers. No experience with theAzure Batch SDK is needed; runyour containers with easy-to-understand configuration files. All Azureregions are supported, including non-public Azure regions.

Additionally, Batch Shipyard provides the ability to provision and manageentire standalone remote file systems (storage clusters)in Azure, independent of any integrated Azure Batch functionality.

Major Features

Container Runtime and Image Management

• Support for multiple container runtimes includingDocker, Singularity, andKata Containers tuned for Azure Batchcompute nodes
• Automated deployment of container images required for tasks to compute nodes
• Support for container registries includingAzure Container Registryfor both Docker and Singularity images (ORAS), other Internet-accessiblepublic and private registries, and support forthe Sylabs Singularity Library andSingularity Hub
• Transparent support for GPU-accelerated container applications on bothDocker and Singularityon Azure N-Series VM instances
• Transparent assist for running Docker and Singularity containers utilizingInfiniband/RDMA on HPC Azure VM instances includingA-Series,H-Series,Hb/Hc-Series,and N-Series
• Integrated support forSingularity Encrypted Containers

Data Management and Shared File Systems

• Comprehensive data movementsupport: move data easily between locally accessible storage systems, remotefilesystems, Azure Blob or File Storage, and compute nodes
• Standalone Remote Filesystem Provisioningwith integration to auto-link these filesystems to compute nodes withsupport for NFS andGlusterFS distributed network file system
• Automatic shared data volume support for linking toRemote Filesystems,Azure Filevia SMB, Azure Blobvia blobfuse,GlusterFS provisioned directly on compute nodes,and custom Linux mount support (fstab)
• Support for automated on-demand, per-job distributed scratch spaceprovisioning via BeeGFS BeeOND

Monitoring

• Automated, integratedresource monitoringwith Prometheus and Grafanafor Batch pools and RemoteFS storage clusters
• Support for Batch Insights

Open Source Scheduler Integration

• Support for elastic cloud burstingon Slurm to Batch pools with automatedRemoteFS shared file system linking

Azure Ecosystem Integration

• Support forserverless executionbinding with Azure Functions
• Support for credential management throughAzure KeyVault

Azure Batch Integration and Enhancements

• Federationsupport: enables unified, constraint-based scheduling to collections ofheterogeneous pools, including across multiple Batch accounts and Azureregions
• Support for simple, scenario-based pool autoscaleand autopool to dynamically scale and control computing resources on-demand
• Support for Task Factorieswith the ability to generate tasks based on parametric (parameter) sweeps,randomized input, file enumeration, replication, and custom Python code-basedgenerators
• Support formulti-instance tasksto accommodate MPI and multi-node cluster applications packaged as Docker orSingularity containers on compute pools with automatic job completion andtask termination
• Seamless, direct high-level configuration support for popular MPI runtimesincluding OpenMPI, MPICH, MVAPICH, and Intel MPI with automatic configurationfor Infiniband, including SR-IOV RDMA VM sizes
• Seamless integration with Azure Batch job, task and file concepts along withfull pass-through of theAzure Batch APIto containers executed on compute nodes
• Support for Azure Batch task dependenciesallowing complex processing pipelines and DAGs
• Support for merge or final task specification that automatically dependson all other tasks within the job
• Support for job schedules and recurrences for automatic execution oftasks at set intervals
• Support for live job and job schedule migration between pools
• Support for Low Priority Compute Nodes
• Support for deploying Batch compute nodes into a specifiedVirtual Networkand pre-defined public IP addresses
• Automatic setup of SSH or RDP users to all nodes in the compute pool andoptional creation of SSH tunneling scripts to Docker Hosts on compute nodes
• Support for custom host imagesincluding Shared Image Gallery
• Support for Windows Containerson compliant Windows compute node pools with the ability to activateAzure Hybrid Use Benefitif applicable

Installation

Local Installation

Please see the installation guidefor more information regarding the various local installation options andrequirements.

Azure Cloud Shell

Batch Shipyard is integrated directly intoAzure Cloud Shelland you can execute any Batch Shipyard workload using your web browser orthe Microsoft Azure Androidand iOSapp.

Simply request a Cloud Shell session and type shipyard to invoke the CLI;no installation is required. Try Batch Shipyard nowin your browser.

Documentation and Recipes

Please refer to theBatch Shipyard Documentation on Read the Docs.

Visit theBatch Shipyard Recipessection for various sample container workloads using Azure Batch and BatchShipyard.

Batch Shipyard Compute Node Host OS Support

Batch Shipyard is currently compatible with popular Azure Batch supportedMarketplace Linux VMs,compliant Linux custom images,and native Azure BatchWindows Server with ContainersVMs. Please see theplatform image supportdocumentation for more information specific to Batch Shipyard support ofcompute node host operating systems.

Change Log

Please see theChange Logfor project history.

Please see this project's Code of Conduct andContributing guidelines.