2018年最強自然語言模型 Google BERT 資源彙總

望江小汽車發表於2019-03-03

本文介紹了一種新的語言表徵模型 BERT——來自 Transformer 的雙向編碼器表徵。與最近的語言表徵模型不同,BERT 旨在基於所有層的左、右語境來預訓練深度雙向表徵。BERT 是首個在大批句子層面和 token 層面任務中取得當前最優效能的基於微調的表徵模型,其效能超越許多使用任務特定架構的系統,重新整理了 11 項 NLP 任務的當前最優效能記錄。

BERT 相關資源

加更BERT在中文和小資料集上的應用方法以及實驗效果

標題 說明 附加
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 原始論文 20181011
Reddit 討論 作者討論
BERT-pytorch Google AI 2018 BERT pytorch implementation
論文解讀:BERT模型及fine-tuning 習翔宇 論文解讀
最強NLP預訓練模型!谷歌BERT橫掃11項NLP任務記錄 論文淺析
【NLP】Google BERT詳解 李入魔 解讀
如何評價 BERT 模型? 解讀論文思想點
NLP突破性成果 BERT 模型詳細解讀 章魚小丸子 解讀
谷歌最強 NLP 模型 BERT 解讀 AI科技評論
預訓練BERT,官方程式碼釋出前他們是這樣用TensorFlow解決的 論文復現說明 20181030
谷歌終於開源BERT程式碼:3 億引數量,機器之心全面解讀 20181101
為什麼說 Bert 大力出奇跡? 20181121
BERT fine-tune 實踐終極教程 BERT在中文資料集上的fine tune全攻略 20181123
BERT在極小資料下帶來顯著提升的開源實現 張俊 20181127

BERT 論文內容精要

模型結構

其中的主要模組 Transformer 來自 Attention Is All You Need

2018年最強自然語言模型 Google BERT 資源彙總

模型輸入

2018年最強自然語言模型 Google BERT 資源彙總

預訓練方法

遮蔽語言模型(完形填空)和預測下一句任務。

實驗

2018年最強自然語言模型 Google BERT 資源彙總
2018年最強自然語言模型 Google BERT 資源彙總
2018年最強自然語言模型 Google BERT 資源彙總

模型分析

Effect of Pre-training Tasks

2018年最強自然語言模型 Google BERT 資源彙總

Effect of Model Size

2018年最強自然語言模型 Google BERT 資源彙總

Effect of Number of Training Steps

2018年最強自然語言模型 Google BERT 資源彙總

Feature-based Approach with BERT

2018年最強自然語言模型 Google BERT 資源彙總

結論

Recent empirical improvements due to transfer learning with language models have demonstrated that rich, unsupervised pre-training is an integral part of many language understanding systems. Inparticular, these results enable even low-resource tasks to benefit from very deep unidirectional architectures.Our major contribution is further generalizing these findings to deep bidirectional architectures, allowing the same pre-trained model to successfully tackle a broad set of NLP tasks. While the empirical results are strong, in some cases surpassing human performance, important future work is to investigate the linguistic phenomena that may or may not be captured by BERT.


BERT 相關資源

標題 說明 附加
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 原始論文 20181011
Reddit 討論 作者討論
BERT-pytorch Google AI 2018 BERT pytorch implementation
論文解讀:BERT模型及fine-tuning 習翔宇 論文解讀
最強NLP預訓練模型!谷歌BERT橫掃11項NLP任務記錄 論文淺析
【NLP】Google BERT詳解 李入魔 解讀
如何評價 BERT 模型? 解讀論文思想點
NLP突破性成果 BERT 模型詳細解讀 章魚小丸子 解讀
谷歌最強 NLP 模型 BERT 解讀 AI科技評論
預訓練BERT,官方程式碼釋出前他們是這樣用TensorFlow解決的 論文復現說明 20181030
谷歌終於開源BERT程式碼:3 億引數量,機器之心全面解讀 20181101

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (Submitted on 11 Oct 2018)

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE benchmark to 80.4% (7.6% absolute improvement), MultiNLI accuracy to 86.7 (5.6% absolute improvement) and the SQuAD v1.1 question answering Test F1 to 93.2 (1.5% absolute improvement), outperforming human performance by 2.0%. Comments: 13 pages

摘要:本文介紹了一種新的語言表徵模型 BERT,意為來自 Transformer 的雙向編碼器表徵(Bidirectional Encoder Representations from Transformers)。與最近的語言表徵模型(Peters et al., 2018; Radford et al., 2018)不同,BERT 旨在基於所有層的左、右語境來預訓練深度雙向表徵。因此,預訓練的 BERT 表徵可以僅用一個額外的輸出層進行微調,進而為很多工(如問答和語言推斷任務)建立當前最優模型,無需對任務特定架構做出大量修改。

BERT 的概念很簡單,但實驗效果很強大。它重新整理了 11 個 NLP 任務的當前最優結果,包括將 GLUE 基準提升至 80.4%(7.6% 的絕對改進)、將 MultiNLI 的準確率提高到 86.7%(5.6% 的絕對改進),以及將 SQuAD v1.1 的問答測試 F1 得分提高至 93.2 分(提高 1.5 分)——比人類表現還高出 2 分。

Subjects: Computation and Language (cs.CL) Cite as: arXiv:1810.04805 [cs.CL] (or arXiv:1810.04805v1 [cs.CL] for this version) Bibliographic data Select data provider: Semantic Scholar [Disable Bibex(What is Bibex?)] No data available yet Submission history From: Jacob Devlin [view email] [v1] Thu, 11 Oct 2018 00:50:01 GMT (227kb,D)

Reddit 討論

2018年最強自然語言模型 Google BERT 資源彙總
2018年最強自然語言模型 Google BERT 資源彙總
2018年最強自然語言模型 Google BERT 資源彙總

官方復現 google-research bert

最近谷歌釋出了基於雙向 Transformer 的大規模預訓練語言模型,該預訓練模型能高效抽取文字資訊並應用於各種 NLP 任務,該研究憑藉預訓練模型重新整理了 11 項 NLP 任務的當前最優效能記錄。如果這種預訓練方式能經得起實踐的檢驗,那麼各種 NLP 任務只需要少量資料進行微調就能實現非常好的效果,BERT 也將成為一種名副其實的骨幹網路。

Introduction

BERT, or Bidirectional Encoder Representations from Transformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks.

Our academic paper which describes BERT in detail and provides full results on a number of tasks can be found here: arxiv.org/abs/1810.04….

To give a few numbers, here are the results on the SQuAD v1.1 question answering task:

SQuAD v1.1 Leaderboard (Oct 8th 2018) Test EM Test F1
1st Place Ensemble - BERT 87.4 93.2
2nd Place Ensemble - nlnet 86.0 91.7
1st Place Single Model - BERT 85.1 91.8
2nd Place Single Model - nlnet 83.5 90.1

And several natural language inference tasks:

System MultiNLI Question NLI SWAG
BERT 86.7 91.1 86.3
OpenAI GPT (Prev. SOTA) 82.2 88.1 75.0

Plus many other tasks.

Moreover, these results were all obtained with almost no task-specific neural network architecture design.

If you already know what BERT is and you just want to get started, you can download the pre-trained models and run a state-of-the-art fine-tuning in only a few minutes.

復現 bert_language_understanding

Pre-training of Deep Bidirectional Transformers for Language Understanding

復現 BERT-keras

Keras implementation of BERT(Bidirectional Encoder Representations from Transformers)

復現 pytorch-pretrained-BERT

PyTorch version of Google AI's BERT model with script to load Google's pre-trained models.

BERT的資料集 GLUE

GLUE 來自論文 GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

摘要

For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems.

相關文章