sagemaker-python-sdk

授权协议 Apache-2.0 License
开发语言 Python
所属分类 神经网络/人工智能、 机器学习/深度学习
软件类型 开源软件
地区 不详
投 递 者 温凯
操作系统 跨平台
开源组织
适用人群 未知
 软件概览

SageMaker

SageMaker Python SDK

SageMaker Python SDK is an open source library for training and deploying machine learning models on Amazon SageMaker.

With the SDK, you can train and deploy models using popular deep learning frameworks Apache MXNet and TensorFlow.You can also train and deploy models with Amazon algorithms,which are scalable implementations of core machine learning algorithms that are optimized for SageMaker and GPU training.If you have your own algorithms built into SageMaker compatible Docker containers, you can train and host models using these as well.

For detailed documentation, including the API reference, see Read the Docs.

Table of Contents

  1. Installing SageMaker Python SDK
  2. Using the SageMaker Python SDK
  3. Using MXNet
  4. Using TensorFlow
  5. Using Chainer
  6. Using PyTorch
  7. Using Scikit-learn
  8. Using XGBoost
  9. SageMaker Reinforcement Learning Estimators
  10. SageMaker SparkML Serving
  11. Amazon SageMaker Built-in Algorithm Estimators
  12. Using SageMaker AlgorithmEstimators
  13. Consuming SageMaker Model Packages
  14. BYO Docker Containers with SageMaker Estimators
  15. SageMaker Automatic Model Tuning
  16. SageMaker Batch Transform
  17. Secure Training and Inference with VPC
  18. BYO Model
  19. Inference Pipelines
  20. Amazon SageMaker Operators in Apache Airflow
  21. SageMaker Autopilot
  22. Model Monitoring
  23. SageMaker Debugger
  24. SageMaker Processing

Installing the SageMaker Python SDK

The SageMaker Python SDK is built to PyPI and can be installed with pip as follows:

pip install sagemaker

You can install from source by cloning this repository and running a pip install command in the root directory of the repository:

git clone https://github.com/aws/sagemaker-python-sdk.git
cd sagemaker-python-sdk
pip install .

Supported Operating Systems

SageMaker Python SDK supports Unix/Linux and Mac.

Supported Python Versions

SageMaker Python SDK is tested on:

  • Python 3.6
  • Python 3.7
  • Python 3.8

AWS Permissions

As a managed service, Amazon SageMaker performs operations on your behalf on the AWS hardware that is managed by Amazon SageMaker.Amazon SageMaker can perform only operations that the user permits.You can read more about which permissions are necessary in the AWS Documentation.

The SageMaker Python SDK should not require any additional permissions aside from what is required for using SageMaker.However, if you are using an IAM role with a path in it, you should grant permission for iam:GetRole.

Licensing

SageMaker Python SDK is licensed under the Apache 2.0 License. It is copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. The license is available at:http://aws.amazon.com/apache2.0/

Running tests

SageMaker Python SDK has unit tests and integration tests.

You can install the libraries needed to run the tests by running pip install --upgrade .[test] or, for Zsh users: pip install --upgrade .\[test\]

Unit tests

We run unit tests with tox, which is a program that lets you run unit tests for multiple Python versions, and also make sure thecode fits our style guidelines. We run tox with all of our supported Python versions, so to run unit testswith the same configuration we do, you need to have interpreters for those Python versions installed.

To run the unit tests with tox, run:

tox tests/unit

Integrations tests

To run the integration tests, the following prerequisites must be met

  1. AWS account credentials are available in the environment for the boto3 client to use.
  2. The AWS account has an IAM role named SageMakerRole.It should have the AmazonSageMakerFullAccess policy attached as well as a policy with the necessary permissions to use Elastic Inference.

We recommend selectively running just those integration tests you'd like to run. You can filter by individual test function names with:

tox -- -k 'test_i_care_about'

You can also run all of the integration tests by running the following command, which runs them in sequence, which may take a while:

tox -- tests/integ

You can also run them in parallel:

tox -- -n auto tests/integ

Building Sphinx docs

Setup a Python environment, and install the dependencies listed in doc/requirements.txt:

# conda
conda create -n sagemaker python=3.7
conda activate sagemaker
conda install sphinx=3.1.1 sphinx_rtd_theme=0.5.0

# pip
pip install -r doc/requirements.txt

Clone/fork the repo, and install your local version:

pip install --upgrade .

Then cd into the sagemaker-python-sdk/doc directory and run:

make html

You can edit the templates for any of the pages in the docs by editing the .rst files in the doc directory and then running make html again.

Preview the site with a Python web server:

cd _build/html
python -m http.server 8000

View the website by visiting http://localhost:8000

SageMaker SparkML Serving

With SageMaker SparkML Serving, you can now perform predictions against a SparkML Model in SageMaker.In order to host a SparkML model in SageMaker, it should be serialized with MLeap library.

For more information on MLeap, see https://github.com/combust/mleap .

Supported major version of Spark: 2.4 (MLeap version - 0.9.6)

Here is an example on how to create an instance of SparkMLModel class and use deploy() method to create anendpoint which can be used to perform prediction against your trained SparkML Model.

sparkml_model = SparkMLModel(model_data='s3://path/to/model.tar.gz', env={'SAGEMAKER_SPARKML_SCHEMA': schema})
model_name = 'sparkml-model'
endpoint_name = 'sparkml-endpoint'
predictor = sparkml_model.deploy(initial_instance_count=1, instance_type='ml.c4.xlarge', endpoint_name=endpoint_name)

Once the model is deployed, we can invoke the endpoint with a CSV payload like this:

payload = 'field_1,field_2,field_3,field_4,field_5'
predictor.predict(payload)

For more information about the different content-type and Accept formats as well as the structure of theschema that SageMaker SparkML Serving recognizes, please see SageMaker SparkML Serving Container.

  • Sagemaker学习 1. 使用boto3对S3操作 学sagemaker前先学习一下boto3库。因为使用Sagemaker要配合S3服务,使用这里 先学习boto3对S3服务的控制。 导入 import boto3 指定使用AWS上哪种服务 #获取s3服务资源对象(相比client对象拥有更高级的抽象调用) s3 = boto3.resource('s3',region_name=regi

 相关资料
  • Amazon SageMaker Examples Example Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using Amazon SageMaker. �� Background Amazon SageMaker is a fully managed s

  • 我有一个python代码和一个模型,是预先训练和有一个model.pkl文件与我在同一个目录中的代码我,现在我必须运行或部署这到aws sagemaker但没有得到任何解决方案由于aws sagemaker只支持两个命令,分别用于训练和部署。 目前,我正在使用命令“python filename.py”运行该程序,并且它正在成功运行。我希望在aws sagemaker上运行同样的程序。 有什么解决

  • 我试图避免将现有的模型训练过程迁移到SageMaker,并避免创建一个定制Docker容器来承载我们训练过的模型。 我的希望是将我们现有的、经过培训的模型注入AWS通过sagemaker python sdk提供的预构建scikit学习容器中。我发现的所有示例都需要先培训模型,然后在SageMaker中创建模型/模型配置。然后使用方法进行部署。 是否可以为方法提供经过培训的模型,并将其托管在AWS

  • 我有一个500GB的csv文件和一个1.5 TB数据的mysql数据库,我想运行aws sagemaker分类和回归算法和随机森林。 aws sagemaker能支持吗?模型可以批量或分块读取和训练吗?它的任何例子

  • 我是AWS新手,尝试通过参考他们的演示来构建模型(从web控制台)。然而,当我尝试创建模型时,它给出了以下错误。 无法访问位于bucket\u namemodel\u name

  • 我是新来的。我在一个VPC中创建了一个笔记本,它有私有子网,kms默认加密密钥,根访问,没有直接的internet访问。我已附上政策,根据文件,该政策可完全访问IAM中的Sagemaker和S3。现在,当一位数据科学家试图在jupyter中运行他的代码时,出现了以下错误。我可以看到jar文件(/home/ec2 user/anaconda3/envs/python3/lib/python3.6/s