Screenshot-to-code

授权协议 View license
开发语言 Python
所属分类 神经网络/人工智能、 机器学习/深度学习
软件类型 开源软件
地区 不详
投 递 者 湛玄裳
操作系统 跨平台
开源组织
适用人群 未知
 软件概览


A detailed tutorial covering the code in this repository: Turning design mockups into code with deep learning.

Plug: I write about learning Machine learning online and independent research �� Enjoy!

The neural network is built in three iterations. Starting with a Hello World version, followed by the main neural network layers, and ending by training it to generalize.

The models are based on Tony Beltramelli's pix2code, and inspired by Airbnb's sketching interfaces, and Harvard's im2markup.

Note: only the Bootstrap version can generalize on new design mock-ups. It uses 16 domain-specific tokens which are translated into HTML/CSS. It has a 97% accuracy. The best model uses a GRU instead of an LSTM. This version can be trained on a few GPUs. The raw HTML version has potential to generalize, but is still unproven and requires a significant amount of GPUs to train. The current model is also trained on a homogeneous and small dataset, thus it's hard to tell how well it behaves on more complex layouts.

A quick overview of the process:

1) Give a design image to the trained neural network

Insert image

2) The neural network converts the image into HTML markup

3) Rendered output

Screenshot

Installation

FloydHub

Click this button to open a Workspace on FloydHub where you will find the same environment and dataset used for the Bootstrap version. You can also find the trained models for testing.

Local

pip install keras tensorflow pillow h5py jupyter
git clone https://github.com/emilwallner/Screenshot-to-code.git
cd Screenshot-to-code/
jupyter notebook

Go do the desired notebook, files that end with '.ipynb'. To run the model, go to the menu then click on Cell > Run all

The final version, the Bootstrap version, is prepared with a small set to test run the model. If you want to try it with all the data, you need to download the data here: https://www.floydhub.com/emilwallner/datasets/imagetocode, and specify the correct dir_name.

Folder structure

|  |-Bootstrap                           #The Bootstrap version
  |  |  |-compiler                         #A compiler to turn the tokens to HTML/CSS (by pix2code)
  |  |  |-resources											
  |  |  |  |-eval_light                    #10 test images and markup
  |  |-Hello_world                         #The Hello World version
  |  |-HTML                                #The HTML version
  |  |  |-Resources_for_index_file         #CSS,images and scripts to test index.html file
  |  |  |-html                             #HTML files to train it on
  |  |  |-images                           #Screenshots for training
  |-readme_images                          #Images for the readme page

Hello World

HTML

Bootstrap

Model weights

  • Bootstrap (The pre-trained model uses GRUs instead of LSTMs)
  • HTML

Acknowledgments

  • Thanks to IBM for donating computing power through their PowerAI platform
  • The code is largely influenced by Tony Beltramelli's pix2code paper. Code Paper
  • The structure and some of the functions are from Jason Brownlee's excellent tutorial
 相关资料
  • 目前,自动化前端开发的最大阻碍是计算能力。但已有人使用目前的深度学习算法以及合成训练数据,来探索人工智能自动构建前端的方法。 Screenshot-to-code-in-Keras 是  Emil Wallner 实现的一个可根据设计草图生成基本 HTML 和 CSS 网站的神经网络。以下是该过程的简要概述: 1)给训练好的神经网络提供设计图像 2)神经网络将图片转化为 HTML 标记语言 3)渲

  • gnome-screenshot 是一款 GNOME桌面下的截图软件

  • Webpage Screenshot这个扩展不管是从功能上还是易用性上都非常不错,可以调整截屏的大小,而且是目前我所知道的唯一一个可以截取整个网页的截 屏扩展,当然也可以截取可见部分。同时在截屏之前可以对大小进行调整,支持多国语言界面,开发者Amina是中国人。另外,改扩展已经通过了Softpedia 网站的100% CLEAN认可,推荐有需要的同学安装此扩展。

  • Android Screenshot Library (ASL) 可以让你通过编程的方式抓取Android设备的屏幕,不需要root权限。ASL使用的是一个在后台运行的本地服务。这个服务通过Android Debug Bridge(ADB)在设备启动的时候启动。

  • iOS-Screenshot-Automator 用来为批量创建屏幕截图,根据你定义的每个语言和每种设备。 示例代码: #import "iOS-Screenshot-Automator/Helper.js"test("Login Test", loginTest);test("Route Test", routeDetailsTest);...function loginTest(target,

  • Android-screenshot-lib 是一个用于在 Android 应用中集成截屏功能的开发包。 该框架使用ddmlib捕获设备中的屏幕截图,并解决了有关使用ddmlib捕获屏幕截图的一些问题: ddmlib图像捕获速度很慢,每个图像大约600ms + 如果设备屏幕正在更新,则捕获的图像可能会显示部分更新的帧缓冲区 在设置的开发环境端(即执行Maven构建的机器)调用ddmlib-您的应用