2.8 KiB
language, thumbnail
| language | thumbnail |
|---|---|
| english |
Longformer-base-4096 fine-tuned on SQuAD v2
Longformer-base-4096 model fine-tuned on SQuAD v2 for Q&A downstream task.
Longformer-base-4096
Longformer is a transformer model for long documents.
longformer-base-4096 is a BERT-like model started from the RoBERTa checkpoint and pretrained for MLM on long documents. It supports sequences of length up to 4,096.
Longformer uses a combination of a sliding window (local) attention and global attention. Global attention is user-configured based on the task to allow the model to learn task-specific representations.
Details of the downstream task (Q&A) - Dataset 📚 🧐 ❓
SQuAD v2 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering.
| Dataset | Split | # samples |
|---|---|---|
| SQuAD2.0 | train | 130k |
| SQuAD2.0 | eval | 12.3k |
Model fine-tuning 🏋️
The training script is a slightly modified version of this one
Model in Action 🚀
import torch
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
tokenizer = AutoTokenizer.from_pretrained("mrm8488/longformer-base-4096-finetuned-squadv2")
model = AutoModelForQuestionAnswering.from_pretrained("mrm8488/longformer-base-4096-finetuned-squadv2")
text = "Huggingface has democratized NLP. Huge thanks to Huggingface for this."
question = "What has Huggingface done ?"
encoding = tokenizer.encode_plus(question, text, return_tensors="pt")
input_ids = encoding["input_ids"]
# default is local attention everywhere
# the forward method will automatically set global attention on question tokens
attention_mask = encoding["attention_mask"]
start_scores, end_scores = model(input_ids, attention_mask=attention_mask)
all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())
answer_tokens = all_tokens[torch.argmax(start_scores) :torch.argmax(end_scores)+1]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))
# output => democratized NLP
If given the same context we ask something that is not there, the output for no answer will be <s>
Created by Manuel Romero/@mrm8488 | LinkedIn
Made with ♥ in Spain