Implementing bert4keras Myself

Description

Currently, BERT has been basically implemented, and official weights can be successfully loaded. It has been verified that the model output is consistent with keras-bert, so you can use it with confidence.

The original intention of this project is for the convenience of modification and customization, so it may be updated frequently.

Therefore, stars are welcome, but forking is not recommended, as the version you fork might quickly become outdated.

Usage

Quick installation:

pip install git+https://www.github.com/bojone/bert4keras.git

Reference code:

#! -*- coding: utf-8 -*-
# Test code availability

from bert4keras.models import build_transformer_model
from bert4keras.tokenizers import Tokenizer
import numpy as np

config_path = '../../kg/bert/chinese_L-12_H-768_A-12/bert_config.json'
checkpoint_path = '../../kg/bert/chinese_L-12_H-768_A-12/bert_model.ckpt'
dict_path = '../../kg/bert/chinese_L-12_H-768_A-12/vocab.txt'

tokenizer = Tokenizer(dict_path) # Build tokenizer
model = build_transformer_model(config_path, checkpoint_path) # Build model, load weights

# Encoding test
token_ids, segment_ids = tokenizer.encode(u'Language Model')
print(model.predict([np.array([token_ids]), np.array([segment_ids])]))

The examples previously given in "When BERT Meets Keras: This Might Be the Easiest Way to Open BERT" based on keras-bert are still applicable to this project; you only need to change the loading method of the base_model to the one used in this project.

Currently, it is only guaranteed to support Python 2.7. The experimental environment is TensorFlow 1.8+ and Keras 2.2.4+. (Some friends have tested it and found that Python 3 also works directly without errors. Python 3 users can try it out. However, I haven’t tested it myself, so I don’t guarantee it.)

Of course, if any contributors find bugs, please feel free to point them out, provide corrections, or even submit Pull Requests!

Background

Previously, I had been using the keras-bert implemented by the expert CyberZHG. If the goal is purely to call and fine-tune BERT within Keras, keras-bert is already quite satisfactory.

However, if you want to modify the internal structure of BERT on the basis of loading official pre-trained weights, then keras-bert is relatively difficult to meet our needs. This is because, for the sake of code reusability, keras-bert encapsulates almost every small module into a separate library. For example, keras-bert depends on keras-transformer, which depends on keras-multi-head, which in turn depends on keras-self-attention. With such layers of dependencies, modifying it becomes quite a headache.

Therefore, I decided to rewrite a Keras version of BERT, striving to implement it completely within a few files to reduce these dependencies, while retaining the feature of being able to load official pre-trained weights.

Acknowledgements

Thanks to CyberZHG for implementing keras-bert. This implementation has referred to the source code of keras-bert in many places. I sincerely thank him for his selfless contribution.

When reposting, please include the address of this article: https://kexue.fm/archives/6915

For more detailed reposting matters, please refer to: "Scientific Space FAQ"