Commit
·
6eab37f
1
Parent(s):
f0c0638
Update README.md
Browse files
README.md
CHANGED
@@ -6,7 +6,12 @@ model-index:
|
|
6 |
- name: t5-small-generation-code-documentation
|
7 |
results: []
|
8 |
widget:
|
9 |
-
- text:
|
|
|
|
|
|
|
|
|
|
|
10 |
---
|
11 |
|
12 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
@@ -58,4 +63,4 @@ The following hyperparameters were used during training:
|
|
58 |
- Transformers 4.30.0
|
59 |
- Pytorch 2.0.0
|
60 |
- Datasets 2.1.0
|
61 |
-
- Tokenizers 0.13.3
|
|
|
6 |
- name: t5-small-generation-code-documentation
|
7 |
results: []
|
8 |
widget:
|
9 |
+
- text: >-
|
10 |
+
def get_training_corpus(threshold=256): dataset_corpus = dataset['train']
|
11 |
+
for start_idx in range(0, len(dataset_corpus), 1000): samples =
|
12 |
+
dataset_corpus[start_idx : start_idx + 1000] samples = [sample for sample in
|
13 |
+
samples['func_code_tokens'] if len(sample) < threshold] yield samples
|
14 |
+
pipeline_tag: text2text-generation
|
15 |
---
|
16 |
|
17 |
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
|
|
63 |
- Transformers 4.30.0
|
64 |
- Pytorch 2.0.0
|
65 |
- Datasets 2.1.0
|
66 |
+
- Tokenizers 0.13.3
|