GPT2-Medium Training from Scratch on Google Colab for Any Language - Tips & Tricks by Twenkid

My tips for training on Colab on Tesla T4 16GB, based on the obstacles that I had to overcome. I trained on a custom-built Bulgarian dataset. https://youtu.be/F-Xt-cK4L-g The code is based on: Arshabhi Kayal's tutorial: https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171 However his example was for local training on a modest Nvidia Geforce RTX 2060 (6 GB) for GPT2-Small (3 times smaller) and a fixed dataset. The code in the experiments in this video was extended and debugged for application in the Colaboratory which has its hurdles and for gradual extension of the dataset after each training epoch, without retraining the tokenizer (see Dataset Skip None in the video). ...

Some impotant points and discoveries: * Google Colab hurdles (the dataset should be sampled in parts, can't run too long epochs at once)

* The inputs/labels output of tokenization after changing the dataset should be filtered (Dataset Skip None)

* Etc.

...

Съвети за машинно обучение на GPT2-Medium модел на български или друг език от нулата през Google Colaboratory, от Тодор Арнаудов - Тош/Twenkid. Следва продължение. Errata: ~ 2:05 Tesla K80, not P100.

#gpt2 #transformers #tutorial