papermill is a tool for parameterizing, executing, and analyzingJupyter Notebooks.
Papermill lets you:
This opens up new opportunities for how notebooks can be used. Forexample:
Papermill takes an opinionated approach to notebook parameterization andexecution based on our experiences using notebooks at scale in datapipelines.
From the command line:
pip install papermill
For all optional io dependencies, you can specify individual bundleslike s3
, or azure
-- or use all
. To use Black to format parameters you can add as an extra requires ['black'].
pip install papermill[all]
This library currently supports Python 3.6+ versions. As minor Pythonversions are officially sunset by the Python org papermill will similarlydrop support in the future.
To parameterize your notebook designate a cell with the tag parameters
.
Papermill looks for the parameters
cell and treats this cell as defaults for the parameters passed in at execution time. Papermill will add a new cell tagged with injected-parameters
with input parameters in order to overwrite the values in parameters
. If no cell is tagged with parameters
the injected cell will be inserted at the top of the notebook.
Additionally, if you rerun notebooks through papermill and it will reuse the injected-parameters
cell from the prior run. In this case Papermill will replace the old injected-parameters
cell with the new run's inputs.
The two ways to execute the notebook with parameters are: (1) throughthe Python API and (2) through the command line interface.
import papermill as pm
pm.execute_notebook(
'path/to/input.ipynb',
'path/to/output.ipynb',
parameters = dict(alpha=0.6, ratio=0.1)
)
Here's an example of a local notebook being executed and output to anAmazon S3 account:
$ papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1
NOTE:If you use multiple AWS accounts, and you have properly configured your AWS credentials, then you can specify which account to use by setting the AWS_PROFILE
environment variable at the command-line. For example:
$ AWS_PROFILE=dev_account papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1
In the above example, two parameters are set: alpha
and l1_ratio
using -p
(--parameters
also works). Parameter values that look like booleans or numbers will be interpreted as such. Here are the different ways users may set parameters:
$ papermill local/input.ipynb s3://bkt/output.ipynb -r version 1.0
Using -r
or --parameters_raw
, users can set parameters one by one. However, unlike -p
, the parameter will remain a string, even if it may be interpreted as a number or boolean.
$ papermill local/input.ipynb s3://bkt/output.ipynb -f parameters.yaml
Using -f
or --parameters_file
, users can provide a YAML file from which parameter values should be read.
$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
alpha: 0.6
l1_ratio: 0.1"
Using -y
or --parameters_yaml
, users can directly provide a YAML string containing parameter values.
$ papermill local/input.ipynb s3://bkt/output.ipynb -b YWxwaGE6IDAuNgpsMV9yYXRpbzogMC4xCg==
Using -b
or --parameters_base64
, users can provide a YAML string, base64-encoded, containing parameter values.
When using YAML to pass arguments, through -y
, -b
or -f
, parameter values can be arrays or dictionaries:
$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
x:
- 0.0
- 1.0
- 2.0
- 3.0
linear_function:
slope: 3.0
intercept: 1.0"
Papermill supports the following name handlers for input and output paths during execution:
Local file system: local
HTTP, HTTPS protocol: http://, https://
Amazon Web Services: AWS S3 s3://
Azure: Azure DataLake Store, Azure Blob Store adl://, abs://
Google Cloud: Google Cloud Storage gs://
Read CONTRIBUTING.md for guidelines on how to setup a local development environment and make code changes back to Papermill.
For development guidelines look in the DEVELOPMENT_GUIDE.md file. This should inform you on how to make particular additions to the code base.
We host the Papermill documentationon ReadTheDocs.
pm.record("res_syn", res_syn) AttributeError: module 'papermill' has no attribute 'record' 难道我安装错了库???直接pip安装的,要不先提个issue1,issue2??哈哈 我怕提一个没人理我,我要让全世界的人都知道我面临的问题,这样才能有效地解决问题。 待续。。。 补充:20200528 其中micr
描述 (Description) 当路由匹配其相应的回调时使用它。 语法 (Syntax) router.execute(callback, args) 参数 (Parameters) callback - 当与路线匹配时执行。 args - 在execute方法中传递的参数。 例子 (Example) <!DOCTYPE html> <html> <head> <title>
全部显示 Execute 方法应用于 FileSearch 对象的情形。 开始对指定文件的搜索。返回一个 Long 类型,如果没有找到文件,则返回零 (0),如果找到一个或多个文件,则返回一个正数。 expression.Execute(SortBy, SortOrder, AlwaysAccurate) expression 必需。该表达式返回一个 FileSearch 对象。 SortByMs
execute命令用来把一个字符串当作Vimscript命令执行。在前面的章节我们曾经跟它打过交道, 不过随着对Vimscript中的字符串有更深入的了解,现在我们将再次认识它。 execute基本用法 执行下面的命令: :::vim :execute "echom 'Hello, world!'" Vim把echom 'Hello, world!'当作一个命令,而且尽职地在把它输出的同时将消息
3.4 execute 与 normal 为什么这两个命令值得单独拿出来讲,因为它们使得其他大部分 Vim 基本命令变得可编 程,用 VimL 编程。不仅是更高层次上的流程控制,更可以控制单个命令的执行,控制所 要执行的命令或参数。简单地说,就是可利用 VimL 语言的一切特性,拼接并生成将要执 行的 ex 命令,然后真正执行它。 :execute 将 VimL 的字符串(值)当作命令执行。 :n
我试图将随机生成的数据插入表中,下面是代码“ 表有3列-、、; 上面的查询在执行即时语句中如下所示: 这个块没有运行。当我试图运行单个execute immediate语句(例如 ORA给了我一个错误: ' execute immediate ' insert into EMP values(' | | prac _ seq . nextval | | ',' ' ' ||'name23 ' ',
问题内容: 我听说过的“Execute Around”是什么?为什么我要使用它,为什么我不想使用它? 问题答案: 基本上,是在这种模式下,你编写一种方法来执行始终需要执行的操作,例如资源分配和清理,并使调用者传递“我们想对资源进行的操作”。例如: 调用代码无需担心打开/清理的一面,它将由处理。 坦白说,这在Java中是很痛苦的,因为闭包是如此word,从Java 8 lambda表达式开始就可以像