当前位置：首页 > 软件库 > 神经网络/人工智能 > 机器学习/深度学习 >

dvc

授权协议 Apache-2.0 License

开发语言 Python

所属分类神经网络/人工智能、机器学习/深度学习

软件类型开源软件

地区不详

投递者柳英豪

操作系统跨平台

开源组织无

适用人群未知

软件概览

Website• Docs• Blog• Twitter• Chat (Community & Support)• Tutorial• Mailing List

Data Version Control or DVC is an open-source tool for data science and machinelearning projects. Key features:

Simple command line Git-like experience. Does not require installing and maintainingany databases. Does not depend on any proprietary online services.
Management and versioning of datasets and machine learningmodels. Data can be saved in S3, Google cloud, Azure, Alibabacloud, SSH server, HDFS, or even local HDD RAID.
Makes projects reproducible and shareable; helping to answer questions about howa model was built.
Helps manage experiments with Git tags/branches and metrics tracking.

DVC aims to replace spreadsheet and document sharing tools (such as Excel or Google Docs)frequently used as both knowledge repositories and team ledgers.DVC also replaces both ad-hoc scripts to track, move, and deploy different model versionsand ad-hoc data file suffixes and prefixes.

Contents

How DVC works

We encourage you to read our Get Started guide to better understand what DVCis and how it can fit your scenarios.

The easiest (but not perfect!) analogy to describe it: DVC is Git (or Git-LFS to be precise) & Makefilesmade right and tailored specifically for ML and Data Science scenarios.

Git/Git-LFS part - DVC helps store and share data artifacts and models, connecting them with a Git repository.
Makefiles part - DVC describes how one data or model artifact was built from other data and code.

DVC usually runs along with Git. Git is used as usual to store and version code (including DVC meta-files). DVC helpsto store data and model files seamlessly out of Git, while preserving almost the same user experience as if theywere stored in Git itself. To store and share the data cache, DVC supports multiple remotes - any cloud (S3, Azure,Google Cloud, etc) or any on-premise network storage (via SSH, for example).

The DVC pipelines (computational graph) feature connects code and data together. It is possible to explicitlyspecify all steps required to produce a model: input dependencies including data, commands to run,and output information to be saved. See the quick start section below orthe Get Started tutorial to learn more.

Quick start

Please read Get Started guide for a full version. Common workflow commands include:

Step	Command
Track data	`$ git add train.py` `$ dvc add images.zip`
Connect code and data by commands	`$ dvc run -n prepare -d images.zip -o images/ unzip -q images.zip` `$ dvc run -n train -d images/ -d train.py -o model.p python train.py`
Make changes and reproduce	`$ vi train.py` `$ dvc repro model.p.dvc`
Share code	`$ git add .` `$ git commit -m 'The baseline model'` `$ git push`
Share data and ML models	`$ dvc remote add myremote -d s3://mybucket/image_cnn` `$ dvc push`

Installation

There are four options to install DVC: pip, Homebrew, Conda (Anaconda) or an OS-specific package.Full instructions are available here.

Snap (Snapcraft/Linux)

snap install dvc --classic

This corresponds to the latest tagged release.Add --beta for the latest tagged release candidate,or --edge for the latest master version.

Choco (Chocolatey/Windows)

choco install dvc

Brew (Homebrew/Mac OS)

brew install dvc

Conda (Anaconda)

conda install -c conda-forge mamba # installs much faster than conda
mamba install -c conda-forge dvc

Depending on the remote storage type you plan to use to keep and share your data, you might need toinstall optional dependencies: dvc-s3, dvc-azure, dvc-gdrive, dvc-gs, dvc-oss, dvc-ssh.

pip (PyPI)

pip install dvc

Depending on the remote storage type you plan to use to keep and share your data, you might need to specifyone of the optional dependencies: s3, gs, azure, oss, ssh. Or all to include them all.The command should look like this: pip install dvc[s3] (in this case AWS S3 dependencies such as boto3will be installed automatically).

To install the development version, run:

pip install git+git://github.com/iterative/dvc

Package

Self-contained packages for Linux, Windows, and Mac are available. The latest version of the packagescan be found on the GitHub releases page.

Ubuntu / Debian (deb)

sudo wget https://dvc.org/deb/dvc.list -O /etc/apt/sources.list.d/dvc.list
sudo apt-get update
sudo apt-get install dvc

Fedora / CentOS (rpm)

sudo wget https://dvc.org/rpm/dvc.repo -O /etc/yum.repos.d/dvc.repo
sudo yum update
sudo yum install dvc

Comparison to related technologies

Data Engineering tools such as AirFlow <https://airflow.apache.org/>,Luigi <https://github.com/spotify/luigi>, and others - in DVC data,model and ML pipelines represent a single ML project focused on datascientists' experience. Data engineering tools orchestrate multiple dataprojects and focus on efficient execution. A DVC project can be used fromexisting data pipelines as a single execution step.
Git-annex - DVC uses the idea of storing the content of large files (which shouldnot be in a Git repository) in a local key-value store, and uses file hardlinks/symlinks instead ofcopying/duplicating files.
Git-LFS - DVC is compatible with manyremote storage services (S3, Google Cloud, Azure, SSH, etc). DVC alsouses reflinks or hardlinks to avoid copy operations on checkouts; thushandling large data files much more efficiently.
Makefile (and analogues including ad-hoc scripts) - DVC tracksdependencies (in a directed acyclic graph).
Workflow Management Systems - DVC is a workflowmanagement system designed specifically to manage machine learning experiments. DVC is built on top of Git.
DAGsHub - online service to host DVCprojects. It provides a useful UI around DVC repositories and integratesother tools.
DVC Studio - official onlineplatform for DVC projects. It can be used to manage data and models, runand track experiments, and visualize and share results. Also, itintegrates with CML (CI/CD for ML) <https://cml.dev/> for trainingmodels in the cloud or Kubernetes.

Contributing

Contributions are welcome! Please see our Contributing Guide for moredetails. Thanks to all our contributors!

Mailing List

Want to stay up to date? Want to help improve DVC by participating in our occasional polls? Subscribe to our mailing list. No spam, really low traffic.

Copyright

This project is distributed under the Apache license version 2.0 (see the LICENSE file in the project root).

By submitting a pull request to this project, you agree to license your contribution under the Apache license version2.0 to this project.

Citation

Iterative, DVC: Data Version Control - Git for Data & Models (2020)DOI:10.5281/zenodo.012345.

Barrak, A., Eghan, E.E. and Adams, B. On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects , in Proceedings of the 28th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2021. Hawaii, USA.

使用案例

dvc++里常用的一些快捷键

Dev-C++是一款挺不错的编译器。官方下载地址：点击打开链接现总结一些常用快捷键： 1. 几个快捷键：在工具->快捷键设置中可以设置自己习惯使用的快捷键，下面给出几种常用的快捷键，均为系统默认的。【Ctrl+N】新建源代码；【Ctrl+O】打开工程或文件；【Ctrl+S】保存；【F9】编译程序；【F10】运行；【F11】编译并运
DEV C++中Socket网络编程

Dev-C++是一个小巧开源的C/C++集成开发环境，对于大部分的Linux函数，都可以在引用头文件后直接调用，但开发socket网络程序却要多做一些工作。添加头文件不可以。添加 #pragma comment(lib,“ws2_32.lib”) 仍然报错学习自：http://blog.sina.com.cn/s/blog_62bed5da01013148.html 在程序中引用头文件 #
DVC数据集版本控制

1. 安装 pip install dvc 2. 初始化 mkdir dvc_example cd dvc_example git init dvc init 3. 添加数据 cp -r coco/test2017 dvc_example cp -r coco/val2017 dvc_example dvc add test2017 val2017 git add * git commit -
DVC 的使用

DVC 的基本使用 DVC (Data Version Control) 数据版本管理工具。DVC 类似于常用到的git代码管理工具。但 git 在工程中往往只是针对代码进行管理，换句话说就是对小文件进行管理，而 DVC 则解决了 git 的这个短板，它可以对工程中的大文件，数据进行管理。DVC 与 git 的结合使用，是目前工程中最具效率的搭档组合。DVC 官网 1. DVC 安装 pip in
Dvc (Data Control Version) 与 Git 合作关系

*为什么要使用Dvc ? * 1.做算法或者机器学习的时候，需要使用不同的数据集来验证算法或者模型的效果，也会需要针对同一份数据集，对比不同版本的模型效果。 2.不适用Git管理数据集的原因： Git 无法单独下载想要的数据集，git来管理大量数据集的话，可能git pull 或者 git push都会耗时很久，将算法模型代码和数据集耦合在一起。简单来说，Dvc负责管理真正的数据源，Git负责

dvc