Update README.md

README.md

@@ -1,11 +1,30 @@
 ---
-license: apache-2.0
+language:
+- th
+- en
+tags:
+- openthaigpt
 ---
 <br>
-（Quickstart）
 
+## How to use
 ```python
-from transformers import AutoModel, AutoConfig
-config = AutoConfig.from_pretrained("Konthee/CLIPTextCamembertModelWithProjection-contrastive", trust_remote_code=True)
-model = AutoModel.from_pretrained("Konthee/CLIPTextCamembertModelWithProjection-contrastive", trust_remote_code=True)
-```
+from transformers import AutoModel,AutoProcessor
+model = AutoModel.from_pretrained("openthaigpt/CLIPTextCamembertModelWithProjection-contrastive", trust_remote_code=True)
+processor = AutoProcessor.from_pretrained("openthaigpt/CLIPTextCamembertModelWithProjection-contrastive", trust_remote_
+```
+
+### Preprocessing
+
+Texts are preprocessed with the following rules:
+
+- Replace HTML forms of characters with the actual characters such asnbsp;with a space and \\\\\\\\\\\\\\\\<br /> with a line break [[Howard and Ruder, 2018]](https://arxiv.org/abs/1801.06146).
+- Remove empty brackets ((), {}, and []) than sometimes come up as a result of text extraction such as from Wikipedia.
+- Replace line breaks with spaces.
+- Replace more than one spaces with a single space
+- Remove more than 3 repetitive characters such as ดีมากกก to ดีมาก [Howard and Ruder, 2018]](https://arxiv.org/abs/1801.06146).
+- Word-level tokenization using [[Phatthiyaphaibun et al., 2020]](https://zenodo.org/record/4319685#.YA4xEGQzaDU) ’s `newmm` dictionary-based maximal matching tokenizer.
+- Replace repetitive words; this is done post-tokenization unlike [[Howard and Ruder, 2018]](https://arxiv.org/abs/1801.06146). since there is no delimitation by space in Thai as in English.
+- Replace spaces with <\\\\\\\\\\\\\\\\_>. The SentencePiece tokenizer combines the spaces with other tokens. Since spaces serve as punctuation in Thai such as sentence boundaries similar to periods in English, combining it with other tokens will omit an important feature for tasks such as word tokenization and sentence breaking. Therefore, we opt to explicitly mark spaces with <\\\\\\\\\\\\\\\\_>.
+
+<br>