From e41212c7151d09db704682a762257aa6e024b64a Mon Sep 17 00:00:00 2001
From: Manuel Romero <mrm8488@gmail.com>
Date: Tue, 17 Mar 2020 17:29:11 +0100
Subject: [PATCH] Create model card for CodeBERTaPy (#3309)

---
 model_cards/mrm8488/CodeBERTaPy/README.md | 123 ++++++++++++++++++++++
 1 file changed, 123 insertions(+)
 create mode 100644 model_cards/mrm8488/CodeBERTaPy/README.md

diff --git a/model_cards/mrm8488/CodeBERTaPy/README.md b/model_cards/mrm8488/CodeBERTaPy/README.md
new file mode 100644
index 00000000..95f471a5
--- /dev/null
+++ b/model_cards/mrm8488/CodeBERTaPy/README.md
@@ -0,0 +1,123 @@
+---
+language: code
+thumbnail:
+---
+
+# CodeBERTaPy
+
+CodeBERTaPy is a RoBERTa-like model trained on the [CodeSearchNet](https://github.blog/2019-09-26-introducing-the-codesearchnet-challenge/) dataset from GitHub for `python` by [Manuel Romero](https://twitter.com/mrm8488)
+
+The **tokenizer** is a Byte-level BPE tokenizer trained on the corpus using Hugging Face `tokenizers`.
+
+Because it is trained on a corpus of code (vs. natural language), it encodes the corpus efficiently (the sequences are between 33% to 50% shorter, compared to the same corpus tokenized by gpt2/roberta).
+
+The (small) **model** is a 6-layer, 84M parameters, RoBERTa-like Transformer model – that’s the same number of layers & heads as DistilBERT – initialized from the default initialization settings and trained from scratch on the full `python` corpus for 4 epochs.
+
+## Quick start: masked language modeling prediction
+
+```python
+PYTHON_CODE = """
+fruits = ['apples', 'bananas', 'oranges']
+for idx, <mask> in enumerate(fruits):
+  print("index is %d and value is %s" % (idx, val))
+""".lstrip()
+```
+
+### Does the model know how to complete simple Python code?
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline(
+    "fill-mask",
+    model="mrm8488/CodeBERTaPy",
+    tokenizer="mrm8488/CodeBERTaPy"
+)
+
+fill_mask(PYTHON_CODE)
+
+## Top 5 predictions:
+
+'val' # prob  0.980728805065155
+'value'
+'idx'
+',val'
+'_'
+```
+
+### Yes! That was easy 🎉 Let's try with another Flask like example
+
+```python
+PYTHON_CODE2 = """
+@app.route('/<name>')
+def hello_name(name):
+    return "Hello {}!".format(<mask>)
+
+if __name__ == '__main__':
+    app.run()
+""".lstrip()
+
+
+fill_mask(PYTHON_CODE2)
+
+## Top 5 predictions:
+
+'name' # prob  0.9961813688278198
+' name'
+'url'
+'description'
+'self'
+```
+
+### Yeah! It works 🎉 Let's try with another Tensorflow/Keras like example
+
+```python
+PYTHON_CODE3="""
+model = keras.Sequential([
+    keras.layers.Flatten(input_shape=(28, 28)),
+    keras.layers.<mask>(128, activation='relu'),
+    keras.layers.Dense(10, activation='softmax')
+])
+""".lstrip()
+
+
+fill_mask(PYTHON_CODE3)
+
+## Top 5 predictions:
+
+'Dense' # prob   0.4482928514480591
+'relu'
+'Flatten'
+'Activation'
+'Conv'
+```
+
+> Great! 🎉
+
+## This work is heavely inspired on [CodeBERTa](https://github.com/huggingface/transformers/blob/master/model_cards/huggingface/CodeBERTa-small-v1/README.md) by huggingface team
+
+<br>
+
+## CodeSearchNet citation
+
+<details>
+
+```bibtex
+@article{husain_codesearchnet_2019,
+	title = {{CodeSearchNet} {Challenge}: {Evaluating} the {State} of {Semantic} {Code} {Search}},
+	shorttitle = {{CodeSearchNet} {Challenge}},
+	url = {http://arxiv.org/abs/1909.09436},
+	urldate = {2020-03-12},
+	journal = {arXiv:1909.09436 [cs, stat]},
+	author = {Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},
+	month = sep,
+	year = {2019},
+	note = {arXiv: 1909.09436},
+}
+```
+
+</details>
+
+> Created by [Manuel Romero/@mrm8488](https://twitter.com/mrm8488)
+
+> Made with <span style="color: #e25555;">&hearts;</span> in Spain