c-my/transformers
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6,204 Commits 171 Branches 49 Tags
master
Commit Graph
11 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Quentin Lhoest a7d73cfdd4 fix rag index names in eval_rag.py example (#8730) 2020-11-24 17:04:47 +01:00
Quentin Lhoest 8062fa63c5 Fix rag finetuning + add finetuning test (#8585) 
* replace init_ddp_connection for index init

* style

* add finetune test

* add test data

* move generate tensors to device

* add test on EM metric

* style

* allow multi process test

* keep gloo process group for retrieval

* add multi-gpu test

* use custom accelerator

* clean test finetune

* minor

* style

* style

* typo

* use python call instead of imported main fumction

* return_dict fix in modeling_rag

* use float32 in retrieval

* store as float32 as well in the custom knowledge dataset example

* style

* rename to finetune_rag

* style

* update readme

* rename utils and callbacks to utils_rag and callbacks_rag

* fix test

* patrick's comments

* generate dummy data in the finetue test script

* remove dummy data files

* style
 2020-11-20 19:05:03 +01:00
Quentin Lhoest 62cd9ce9f8 fix missing return dict (#8653) 2020-11-19 15:17:18 +01:00
Quentin Lhoest 033f29c625 Allow Custom Dataset in RAG Retriever (#7763) 
* add CustomHFIndex

* typo in config

* update tests

* add custom dataset example

* clean script

* update test data

* minor in test

* docs

* docs

* style

* fix imports

* allow to pass the indexed dataset directly

* update tests

* use multiset DPR

* address thom and patrick's comments

* style

* update dpr tokenizer

* add output_dir flag in use_own_knowledge_dataset.py

* allow custom datasets in examples/rag/finetune.py

* add test for custom dataset in distributed rag retriever
 2020-10-19 19:42:45 +02:00
Quentin Lhoest 466115b279 Fix missing reference titles in retrieval evaluation of RAG (#7817) 2020-10-16 10:15:49 +02:00
Quentin Lhoest 0c64b18840 Fix bert position ids in DPR convert script (#7776) 
* fix bert position ids in DPR convert script

* style
 2020-10-14 05:30:02 -04:00
Quentin LhoestandPatrick von Platen cf1c88e092 [RAG] Fix retrieval offset in RAG's HfIndex and better integration tests (#7372) 
* Fix retrieval offset in RAG's HfIndex

* update slow tests

* style

* fix new test

* style

* add better tests

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
 2020-09-25 16:12:46 +02:00
Quentin LhoestandSylvain Gugger 0f16dd0ac2 Add DPR to models summary (#6690) 
* add dpr to models summary

* minor

* minor

* Update docs/source/model_summary.rst

qa -> question answering

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update docs/source/model_summary.rst

qa -> question ansering (cont'd)

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2020-08-25 09:57:28 +02:00
Quentin Lhoest 9a8c168f56 Sort unique_no_split_tokens to make it deterministic (#6461) 
* change unique_no_split_tokens's type to set

* use sorted list instead of set

* style
 2020-08-14 10:36:58 +02:00
Quentin Lhoest 4fedc1256c Fix tests imports dpr (#5576) 
* fix test imports

* fix max_length

* style

* fix tests
 2020-07-07 16:35:12 +02:00
Quentin LhoestandLysandre fbd8792195 Add DPR model (#5279) 
* beginning of dpr modeling

* wip

* implement forward

* remove biencoder + better init weights

* export dpr model to embed model for nlp lib

* add new api

* remove old code

* make style

* fix dumb typo

* don't load bert weights

* docs

* docs

* style

* move the `k` parameter

* fix init_weights

* add pretrained configs

* minor

* update config names

* style

* better config

* style

* clean code based on PR comments

* change Dpr to DPR

* fix config

* switch encoder config to a dict

* style

* inheritance -> composition

* add messages in assert startements

* add dpr reader tokenizer

* one tokenizer per model

* fix base_model_prefix

* fix imports

* typo

* add convert script

* docs

* change tokenizers conf names

* style

* change tokenizers conf names

* minor

* minor

* fix wrong names

* minor

* remove unused convert functions

* rename convert script

* use return_tensors in tokenizers

* remove n_questions dim

* move generate logic to tokenizer

* style

* add docs

* docs

* quality

* docs

* add tests

* style

* add tokenization tests

* DPR full tests

* Stay true to the attention mask building

* update docs

* missing param in bert input docs

* docs

* style

Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
 2020-07-07 08:56:12 -04:00