* make transformers-cli cross-platform
Using "scripts" is a useful option in setup.py particularly when you want to get access to non-python scripts. However, in this case we want to have an entry point into some of our own Python scripts. To do this in a concise, cross-platfom way, we can use entry_points.console_scripts. This change is necessary to provide the CLI on different platforms, which "scripts" does not ensure. Usage remains the same, but the "transformers-cli" script has to be moved (be part of the library) and renamed (underscore + extension)
* make style & quality
* simplify cache vars and allow for TRANSFORMERS_CACHE env
As it currently stands, "TRANSFORMERS_CACHE" is not an accepted variable. It seems that the these variables were not updated when moving from version pytorch_transformers to transformers. In addition, the fallback procedure could be improved. and simplified. Pathlib seems redundant here.
* Update file_utils.py
The CONTRIBUTING file pins to a specific version of isort, so we might as well install that in `dev` . This makes it easier for contributors so they don't have to manually install the specific commit.
* Add disable_outgoing to pretrained items
Setting disable_outgoing=True disables outgonig traffic:
- etags are not looked up
- models are not downloaded
* parameter name change
* Remove forgotten print
* add "info" command to CLI
As a convenience, add the info directive to CLI. Running `python transformers-cli info` will return a string containing the transformers version, platform, python version, PT/TF version and GPU support
* Swap f-strings for .format
Still supporting 3.5 so can't use f-strings (sad face)
* Add reference in issue to CLI
* Add the expected fields to issue template
This way, people can still add the information manually if they want. (Though I fear they'll just ignore it.)
* Remove heading from output
* black-ify
* order of imports
Should ensure isort test passes
* use is_X_available over import..pass
* style
* fix copy-paste bug
* Rename command info -> env
Also adds the command to CONTRIBUTING.md in "Did you find a bug" section
In batch_encode_plus we have to ensure that the tokenizer has a pad_token_id so that, when padding, no None values are added as padding. That would happen with gpt2, openai, transfoxl.
closes https://github.com/huggingface/transformers/issues/2640
- mostly stylistic streamlining
- removed 'additional context' sections. They seem to be rarely used and might cause confusion. If more details are needed, users can add them to the 'details' section
Bram Vanroy