awesome-nlp

授权协议 View license
开发语言 Python
所属分类 神经网络/人工智能、 机器学习/深度学习
软件类型 开源软件
地区 不详
投 递 者 柳俊健
操作系统 跨平台
开源组织
适用人群 未知
 软件概览

awesome-nlp

A curated list of resources dedicated to Natural Language Processing

Read this in English, Traditional Chinese

Please read the contribution guidelines before contributing. Please add your favourite NLP resource by raising a pull request

Contents

Research Summaries and Trends

Prominent NLP Research Labs

Back to Top

Tutorials

Back to Top

Reading Content

General Machine Learning

Introductions and Guides to NLP

Blogs and Newsletters

Videos and Online Courses

Back to Top

Books

Libraries

Back to Top

  • Node.js and Javascript - Node.js Libaries for NLP | Back to Top

    • Twitter-text - A JavaScript implementation of Twitter's text processing library
    • Knwl.js - A Natural Language Processor in JS
    • Retext - Extensible system for analyzing and manipulating natural language
    • NLP Compromise - Natural Language processing in the browser
    • Natural - general natural language facilities for node
    • Poplar - A web-based annotation tool for natural language processing (NLP)
    • NLP.js - An NLP library for building bots
    • node-question-answering - Fast and production-ready question answering w/ DistilBERT in Node.js
  • Python - Python NLP Libraries | Back to Top

    • TextAttack - Adversarial attacks, adversarial training, and data augmentation in NLP
    • TextBlob - Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of Natural Language Toolkit (NLTK) and Pattern, and plays nicely with both ��
    • spaCy - Industrial strength NLP with Python and Cython ��
      • textacy - Higher level NLP built on spaCy
    • gensim - Python library to conduct unsupervised semantic modelling from plain text ��
    • scattertext - Python library to produce d3 visualizations of how language differs between corpora
    • GluonNLP - A deep learning toolkit for NLP, built on MXNet/Gluon, for research prototyping and industrial deployment of state-of-the-art models on a wide range of NLP tasks.
    • AllenNLP - An NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.
    • PyTorch-NLP - NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer representations, common NLP metrics such as BLEU
    • Rosetta - Text processing tools and wrappers (e.g. Vowpal Wabbit)
    • PyNLPl - Python Natural Language Processing Library. General purpose NLP library for Python. Also contains some specific modules for parsing common NLP formats, most notably for FoLiA, but also ARPA language models, Moses phrasetables, GIZA++ alignments.
    • PySS3 - Python package that implements a novel white-box machine learning model for text classification, called SS3. Since SS3 has the ability to visually explain its rationale, this package also comes with easy-to-use interactive visualizations tools (online demos).
    • jPTDP - A toolkit for joint part-of-speech (POS) tagging and dependency parsing. jPTDP provides pre-trained models for 40+ languages.
    • BigARTM - a fast library for topic modelling
    • Snips NLU - A production ready library for intent parsing
    • Chazutsu - A library for downloading&parsing standard NLP research datasets
    • Word Forms - Word forms can accurately generate all possible forms of an English word
    • Multilingual Latent Dirichlet Allocation (LDA) - A multilingual and extensible document clustering pipeline
    • NLP Architect - A library for exploring the state-of-the-art deep learning topologies and techniques for NLP and NLU
    • Flair - A very simple framework for state-of-the-art multilingual NLP built on PyTorch. Includes BERT, ELMo and Flair embeddings.
    • Kashgari - Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding.
    • FARM - Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
    • Haystack - End-to-end Python framework for building natural language search interfaces to data. Leverages Transformers and the State-of-the-Art of NLP. Supports DPR, Elasticsearch, HuggingFace’s Modelhub, and much more!
    • Rita DSL - a DSL, loosely based on RUTA on Apache UIMA. Allows to define language patterns (rule-based NLP) which are then translated into spaCy, or if you prefer less features and lightweight - regex patterns.
    • Transformers - Natural Language Processing for TensorFlow 2.0 and PyTorch.
    • Tokenizers - Tokenizers optimized for Research and Production.
    • fairSeq Facebook AI Research implementations of SOTA seq2seq models in Pytorch.
    • corex_topic - Hierarchical Topic Modeling with Minimal Domain Knowledge
    • Sockeye - Neural Machine Translation (NMT) toolkit that powers Amazon Translate.
    • DL Translate - A deep learning-based translation library for 50 languages, built on transformers and Facebook's mBART Large.
  • C++ - C++ Libraries | Back to Top

    • InsNet - A neural network library for building instance-dependent NLP models with padding-free dynamic batching.
    • MIT Information Extraction Toolkit - C, C++, and Python tools for named entity recognition and relation extraction
    • CRF++ - Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing tasks.
    • CRFsuite - CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data.
    • BLLIP Parser - BLLIP Natural Language Parser (also known as the Charniak-Johnson parser)
    • colibri-core - C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams and skipgrams in a quick and memory-efficient way.
    • ucto - Unicode-aware regular-expression based tokenizer for various languages. Tool and C++ library. Supports FoLiA format.
    • libfolia - C++ library for the FoLiA format
    • frog - Memory-based NLP suite developed for Dutch: PoS tagger, lemmatiser, dependency parser, NER, shallow parser, morphological analyzer.
    • MeTA - MeTA : ModErn Text Analysis is a C++ Data Sciences Toolkit that facilitates mining big text data.
    • Mecab (Japanese)
    • Moses
    • StarSpace - a library from Facebook for creating embeddings of word-level, paragraph-level, document-level and for text classification
  • Java - Java NLP Libraries | Back to Top

    • Stanford NLP
    • OpenNLP
    • NLP4J
    • Word2vec in Java
    • ReVerb Web-Scale Open Information Extraction
    • OpenRegex An efficient and flexible token-based regular expression language and engine.
    • CogcompNLP - Core libraries developed in the U of Illinois' Cognitive Computation Group.
    • MALLET - MAchine Learning for LanguagE Toolkit - package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text.
    • RDRPOSTagger - A robust POS tagging toolkit available (in both Java & Python) together with pre-trained models for 40+ languages.
  • Kotlin - Kotlin NLP Libraries | Back to Top

    • Lingua A language detection library for Kotlin and Java, suitable for long and short text alike
    • Kotidgy — an index-based text data generator written in Kotlin
  • Scala - Scala NLP Libraries | Back to Top

    • Saul - Library for developing NLP systems, including built in modules like SRL, POS, etc.
    • ATR4S - Toolkit with state-of-the-art automatic term recognition methods.
    • tm - Implementation of topic modeling based on regularized multilingual PLSA.
    • word2vec-scala - Scala interface to word2vec model; includes operations on vectors like word-distance and word-analogy.
    • Epic - Epic is a high performance statistical parser written in Scala, along with a framework for building complex structured prediction models.
    • Spark NLP - Spark NLP is a natural language processing library built on top of Apache Spark ML that provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment.
  • R - R NLP Libraries | Back to Top

    • text2vec - Fast vectorization, topic modeling, distances and GloVe word embeddings in R.
    • wordVectors - An R package for creating and exploring word2vec and other word embedding models
    • RMallet - R package to interface with the Java machine learning tool MALLET
    • dfr-browser - Creates d3 visualizations for browsing topic models of text in a web browser.
    • dfrtopics - R package for exploring topic models of text.
    • sentiment_classifier - Sentiment Classification using Word Sense Disambiguation and WordNet Reader
    • jProcessing - Japanese Natural Langauge Processing Libraries, with Japanese sentiment classification
  • Clojure | Back to Top

    • Clojure-openNLP - Natural Language Processing in Clojure (opennlp)
    • Infections-clj - Rails-like inflection library for Clojure and ClojureScript
    • postagga - A library to parse natural language in Clojure and ClojureScript
  • Ruby | Back to Top

  • Rust

    • whatlang — Natural language recognition library based on trigrams
    • snips-nlu-rs - A production ready library for intent parsing
    • rust-bert - Ready-to-use NLP pipelines and Transformer-based models
  • NLP++ - NLP++ Langauge | Back to Top

Services

NLP as API with higher level functionality such as NER, Topic tagging and so on | Back to Top

  • Wit-ai - Natural Language Interface for apps and devices
  • IBM Watson's Natural Language Understanding - API and Github demo
  • Amazon Comprehend - NLP and ML suite covers most common tasks like NER, tagging, and sentiment analysis
  • Google Cloud Natural Language API - Syntax Analysis, NER, Sentiment Analysis, and Content tagging in atleast 9 languages include English and Chinese (Simplified and Traditional).
  • ParallelDots - High level Text Analysis API Service ranging from Sentiment Analysis to Intent Analysis
  • Microsoft Cognitive Service
  • TextRazor
  • Rosette
  • Textalytic - Natural Language Processing in the Browser with sentiment analysis, named entity extraction, POS tagging, word frequencies, topic modeling, word clouds, and more
  • NLP Cloud - SpaCy NLP models (custom and pre-trained ones) served through a RESTful API for named entity recognition (NER), POS tagging, and more.
  • Cloudmersive - Unified and free NLP APIs that perform actions such as speech tagging, text rephrasing, language translation/detection, and sentence parsing

Annotation Tools

  • GATE - General Architecture and Text Engineering is 15+ years old, free and open source
  • Anafora is free and open source, web-based raw text annotation tool
  • brat - brat rapid annotation tool is an online environment for collaborative text annotation
  • doccano - doccano is free, open-source, and provides annotation features for text classification, sequence labeling and sequence to sequence
  • INCEpTION - A semantic annotation platform offering intelligent assistance and knowledge management
  • tagtog, team-first web tool to find, create, maintain, and share datasets - costs $
  • prodigy is an annotation tool powered by active learning, costs $
  • LightTag - Hosted and managed text annotation tool for teams, costs $
  • rstWeb - open source local or online tool for discourse tree annotations
  • GitDox - open source server annotation tool with GitHub version control and validation for XML data and collaborative spreadsheet grids
  • Label Studio - Hosted and managed text annotation tool for teams, freemium based, costs $
  • Datasaur support various NLP tasks for individual or teams, freemium based
  • Konfuzio - team-first hosted and on-prem text, image and PDF annotation tool powered by active learning, freemium based, costs $
  • UBIAI - Easy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as OCR annotation for invoice labeling, costs $

Techniques

Text Embeddings

Word Embeddings

Sentence and Language Model Based Word Embeddings

Back to Top

Question Answering and Knowledge Extraction

Back to Top

Datasets

Back to Top

  • nlp-datasets great collection of nlp datasets
  • gensim-data - Data repository for pretrained NLP models and NLP corpora.

Multilingual NLP Frameworks

Back to Top

  • UDPipe is a trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files. Primarily written in C++, offers a fast and reliable solution for multilingual NLP processing.
  • NLP-Cube : Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New platform, written in Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API).
  • UralicNLP is an NLP library mostly for many endangered Uralic languages such as Sami languages, Mordvin languages, Mari languages, Komi languages and so on. Also some non-endangered languages are supported such as Finnish together with non-Uralic languages such as Swedish and Arabic. UralicNLP can do morphological analysis, generation, lemmatization and disambiguation.

NLP in Korean

Back to Top

Libraries

  • KoNLPy - Python package for Korean natural language processing.
  • Mecab (Korean) - C++ library for Korean NLP
  • KoalaNLP - Scala library for Korean Natural Language Processing.
  • KoNLP - R package for Korean Natural language processing

Blogs and Tutorials

Datasets

  • KAIST Corpus - A corpus from the Korea Advanced Institute of Science and Technology in Korean.
  • Naver Sentiment Movie Corpus in Korean
  • Chosun Ilbo archive - dataset in Korean from one of the major newspapers in South Korea, the Chosun Ilbo.
  • Chat data - Chatbot data in Korean
  • Petitions - Collect expired petition data from the Blue House National Petition Site.
  • Korean Parallel corpora - Neural Machine Translation(NMT) Dataset for Korean to French & Korean to English
  • KorQuAD - Korean SQuAD dataset with Wiki HTML source. Mentions both v1.0 and v2.1 at the time of adding to Awesome NLP

NLP in Arabic

Back to Top

Libraries

  • goarabic - Go package for Arabic text processing
  • jsastem - Javascript for Arabic stemming
  • PyArabic - Python libraries for Arabic
  • RFTokenizer - trainable Python segmenter for Arabic, Hebrew and Coptic

Datasets

  • Multidomain Datasets - Largest Available Multi-Domain Resources for Arabic Sentiment Analysis
  • LABR - LArge Arabic Book Reviews dataset
  • Arabic Stopwords - A list of Arabic stopwords from various resources

NLP in Chinese

Back to Top

Libraries

  • jieba - Python package for Words Segmentation Utilities in Chinese
  • SnowNLP - Python package for Chinese NLP
  • FudanNLP - Java library for Chinese text processing

Anthology

  • funNLP - Collection of NLP tools and resources mainly for Chinese

NLP in German

  • German-NLP - Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

NLP in Polish

  • Polish-NLP - A curated list of resources dedicated to Natural Language Processing (NLP) in polish. Models, tools, datasets.

NLP in Spanish

Back to Top

Libraries

  • spanlp - Python library to detect, censor and clean profanity, vulgarities, hateful words, racism, xenophobia and bullying in texts written in Spanish. It contains data of 21 Spanish-speaking countries.

Data

Word and Sentence Embeddings

NLP in Indic languages

Back to Top

Data, Corpora and Treebanks

Corpora/Datasets that need a login/access can be gained via email

Language Models and Word Embeddings

Libraries and Tooling

NLP in Thai

Back to Top

Libraries

  • PyThaiNLP - Thai NLP in Python Package
  • JTCC - A character cluster library in Java
  • CutKum - Word segmentation with deep learning in TensorFlow
  • Thai Language Toolkit - Based on a paper by Wirote Aroonmanakun in 2002 with included dataset
  • SynThai - Word segmentation and POS tagging using deep learning in Python

Data

  • Inter-BEST - A text corpus with 5 million words with word segmentation
  • Prime Minister 29 - Dataset containing speeches of the current Prime Minister of Thailand

NLP in Danish

NLP in Vietnamese

Libraries

  • underthesea - Vietnamese NLP Toolkit
  • vn.vitk - A Vietnamese Text Processing Toolkit
  • VnCoreNLP - A Vietnamese natural language processing toolkit
  • PhoBERT - Pre-trained language models for Vietnamese
  • pyvi - Python Vietnamese Core NLP Toolkit

Data

  • Vietnamese treebank - 10,000 sentences for the constituency parsing task
  • BKTreeBank - a Vietnamese Dependency Treebank
  • UD_Vietnamese - Vietnamese Universal Dependency Treebank
  • VIVOS - a free Vietnamese speech corpus consisting of 15 hours of recording speech by AILab
  • VNTQcorpus(big).txt - 1.75 million sentences in news
  • ViText2SQL - A dataset for Vietnamese Text-to-SQL semantic parsing (EMNLP-2020 Findings)
  • EVB Corpus - 20,000,000 words (20 million) from 15 bilingual books, 100 parallel English-Vietnamese / Vietnamese-English texts, 250 parallel law and ordinance texts, 5,000 news articles, and 2,000 film subtitles.

NLP for Dutch

Back to Top

  • python-frog - Python binding to Frog, an NLP suite for Dutch. (pos tagging, lemmatisation, dependency parsing, NER)
  • SimpleNLG_NL - Dutch surface realiser used for Natural Language Generation in Dutch, based on the SimpleNLG implementation for English and French.

NLP in Indonesian

Datasets

Libraries & Embedding

NLP in Urdu

Datasets

Libraries

Other Languages

  • Russian: pymorphy2 - a good pos-tagger for Russian
  • Asian Languages: Thai, Lao, Chinese, Japanese, and Korean ICU Tokenizer implementation in ElasticSearch
  • Ancient Languages: CLTK: The Classical Language Toolkit is a Python library and collection of texts for doing NLP in ancient languages
  • Hebrew: NLPH_Resources - A collection of papers, corpora and linguistic resources for NLP in Hebrew

Back to Top

Credits for initial curators and sources

License

License - CC0

  • 早有耳闻在Github上搜“awesome”真的会给人awesome,平时也偶尔看到一些很不错的带有awesome标题的项目,但也没去整理,今天就先整理几个关于人工智能等的一些项目。 机器学习 awesome-machine-learning-on-source-code:关于机器学习和源代码的研究论文、数据集和软件项目的列表。 Machine-Learning-Tutorials: 包含机器学习

 相关资料
  • Awesome Awesome Node.js A curated list of awesome lists that are about or related to Node.js. Inspired by the awesome list thing, going deeper down the rabbit hole. �� Meta stuff about this awesome li

  • awesome-chinese-nlp A curated list of resources for NLP (Natural Language Processing) for Chinese 中文自然语言处理相关资料 图片来自复旦大学邱锡鹏教授 Contents 列表 1. Chinese NLP Toolkits 中文NLP工具 Toolkits 综合NLP工具包 Popular NLP T

  • A curated list of awesome things related to Vite.js This awesome list is for Vite 2.x and onward. Vite 1.x's list is archived. Resources Official Resources 文档 GitHub Repo Release Notes Vue 3 Docs Awes

  • Awesome Python 是一个资源整理集合,由 vinta 发起和维护。内容包括:Web框架、网络爬虫、网络内容提取、模板引擎、数据库、数据可视化、图片处理、文本处理、自然语言处理、机器学习、日志、代码分析等。 这个系列没有推荐 Python 书籍、经典博文、交互教程,所以另外推荐:《25本免费的Python电子书》、《学习Python编程的11个(教程)资源》、《PythonMonk:Py

  • Font Awesome 是一个图标工具包。其已经被重新设计并从头构建。除此之外,还增加了一些功能,比如 icon font ligature、SVG 框架、流行的前端库(如 React)的官方 NPM 包,以及对新 CDN 的访问。Font Awesome 已扩展至 7,865 个图标。

  • awesome-android 收录了来自 github 或其他网站的关于 Android 的大部分库。