transformers/model_cards/huseinzol05/albert-tiny-bahasa-cased/README.md

language

language
malay

Bahasa Albert Model

Pretrained Albert tiny language model for Malay and Indonesian, 85% faster execution and 50% smaller than Albert base.

Pretraining Corpus

albert-tiny-bahasa-cased model was pretrained on ~1.8 Billion words. We trained on both standard and social media language structures, and below is list of data we trained on,

1. dumping wikipedia.
2. local instagram.
3. local twitter.
4. local news.
5. local parliament text.
6. local singlish/manglish text.
7. IIUM Confession.
8. Wattpad.
9. Academia PDF.

Preprocessing steps can reproduce from here, Malaya/pretrained-model/preprocess.

Pretraining details

• This model was trained using Google Albert's github repository on v3-8 TPU.
• All steps can reproduce from here, Malaya/pretrained-model/albert.

Load Pretrained Model

You can use this model by installing torch or tensorflow and Huggingface library transformers. And you can use it directly by initializing it like this:

from transformers import AlbertTokenizer, AlbertModel

model = BertModel.from_pretrained('huseinzol05/albert-tiny-bahasa-cased')
tokenizer = AlbertTokenizer.from_pretrained(
    'huseinzol05/albert-tiny-bahasa-cased',
    do_lower_case = False,
)

Example using AutoModelWithLMHead

from transformers import AlbertTokenizer, AutoModelWithLMHead, pipeline

model = AutoModelWithLMHead.from_pretrained('huseinzol05/albert-tiny-bahasa-cased')
tokenizer = AlbertTokenizer.from_pretrained(
    'huseinzol05/albert-tiny-bahasa-cased',
    do_lower_case = False,
)
fill_mask = pipeline('fill-mask', model = model, tokenizer = tokenizer)
print(fill_mask('makan ayam dengan [MASK]'))

Output is,

[{'sequence': '[CLS] makan ayam dengan ayam[SEP]',
  'score': 0.05121927708387375,
  'token': 629},
 {'sequence': '[CLS] makan ayam dengan sayur[SEP]',
  'score': 0.04497420787811279,
  'token': 1639},
 {'sequence': '[CLS] makan ayam dengan nasi[SEP]',
  'score': 0.039827536791563034,
  'token': 453},
 {'sequence': '[CLS] makan ayam dengan rendang[SEP]',
  'score': 0.032997727394104004,
  'token': 2451},
 {'sequence': '[CLS] makan ayam dengan makan[SEP]',
  'score': 0.031354598701000214,
  'token': 129}]

Results

For further details on the model performance, simply checkout accuracy page from Malaya, https://malaya.readthedocs.io/en/latest/Accuracy.html, we compared with traditional models.

Acknowledgement

Thanks to Im Big, LigBlou, Mesolitica and KeyReply for sponsoring AWS, Google and GPU clouds to train Albert for Bahasa.