Upload folder using huggingface_hub

README.md

@@ -0,0 +1,202 @@
+---
+base_model: Salesforce/codegen2-16B_P
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Salesforce/codegen2-16B_P",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "wte",
+    "lm_head"
+  ],
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "qkv_proj",
+    "out_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dad68fd47d0ca79638a2185e50edcf12c6f04458c1dd7d82cd89b8ba210d950c
+size 1298520480

added_tokens.json

@@ -0,0 +1,945 @@
+{
+  "\t\t": 50294,
+  "\t\t\t": 50293,
+  "\t\t\t\t": 50292,
+  "\t\t\t\t\t": 50291,
+  "\t\t\t\t\t\t": 50290,
+  "\t\t\t\t\t\t\t": 50289,
+  "\t\t\t\t\t\t\t\t": 50288,
+  "\t\t\t\t\t\t\t\t\t": 50287,
+  "  ": 50286,
+  "   ": 50285,
+  "    ": 50284,
+  "     ": 50283,
+  "      ": 50282,
+  "       ": 50281,
+  "        ": 50280,
+  "         ": 50279,
+  "          ": 50278,
+  "           ": 50277,
+  "            ": 50276,
+  "             ": 50275,
+  "              ": 50274,
+  "               ": 50273,
+  "                ": 50272,
+  "                 ": 50271,
+  "                  ": 50270,
+  "                   ": 50269,
+  "                    ": 50268,
+  "                     ": 50267,
+  "                      ": 50266,
+  "                       ": 50265,
+  "                        ": 50264,
+  "                         ": 50263,
+  "                          ": 50262,
+  "                           ": 50261,
+  "                            ": 50260,
+  "                             ": 50259,
+  "                              ": 50258,
+  "                               ": 50257,
+  "<dummy_0>": 50295,
+  "<dummy_1>": 50296,
+  "<dummy_2>": 50297,
+  "<dummy_3>": 50298,
+  "<eom>": 50300,
+  "<mask_100>": 51100,
+  "<mask_101>": 51099,
+  "<mask_102>": 51098,
+  "<mask_103>": 51097,
+  "<mask_104>": 51096,
+  "<mask_105>": 51095,
+  "<mask_106>": 51094,
+  "<mask_107>": 51093,
+  "<mask_108>": 51092,
+  "<mask_109>": 51091,
+  "<mask_10>": 51190,
+  "<mask_110>": 51090,
+  "<mask_111>": 51089,
+  "<mask_112>": 51088,
+  "<mask_113>": 51087,
+  "<mask_114>": 51086,
+  "<mask_115>": 51085,
+  "<mask_116>": 51084,
+  "<mask_117>": 51083,
+  "<mask_118>": 51082,
+  "<mask_119>": 51081,
+  "<mask_11>": 51189,
+  "<mask_120>": 51080,
+  "<mask_121>": 51079,
+  "<mask_122>": 51078,
+  "<mask_123>": 51077,
+  "<mask_124>": 51076,
+  "<mask_125>": 51075,
+  "<mask_126>": 51074,
+  "<mask_127>": 51073,
+  "<mask_128>": 51072,
+  "<mask_129>": 51071,
+  "<mask_12>": 51188,
+  "<mask_130>": 51070,
+  "<mask_131>": 51069,
+  "<mask_132>": 51068,
+  "<mask_133>": 51067,
+  "<mask_134>": 51066,
+  "<mask_135>": 51065,
+  "<mask_136>": 51064,
+  "<mask_137>": 51063,
+  "<mask_138>": 51062,
+  "<mask_139>": 51061,
+  "<mask_13>": 51187,
+  "<mask_140>": 51060,
+  "<mask_141>": 51059,
+  "<mask_142>": 51058,
+  "<mask_143>": 51057,
+  "<mask_144>": 51056,
+  "<mask_145>": 51055,
+  "<mask_146>": 51054,
+  "<mask_147>": 51053,
+  "<mask_148>": 51052,
+  "<mask_149>": 51051,
+  "<mask_14>": 51186,
+  "<mask_150>": 51050,
+  "<mask_151>": 51049,
+  "<mask_152>": 51048,
+  "<mask_153>": 51047,
+  "<mask_154>": 51046,
+  "<mask_155>": 51045,
+  "<mask_156>": 51044,
+  "<mask_157>": 51043,
+  "<mask_158>": 51042,
+  "<mask_159>": 51041,
+  "<mask_15>": 51185,
+  "<mask_160>": 51040,
+  "<mask_161>": 51039,
+  "<mask_162>": 51038,
+  "<mask_163>": 51037,
+  "<mask_164>": 51036,
+  "<mask_165>": 51035,
+  "<mask_166>": 51034,
+  "<mask_167>": 51033,
+  "<mask_168>": 51032,
+  "<mask_169>": 51031,
+  "<mask_16>": 51184,
+  "<mask_170>": 51030,
+  "<mask_171>": 51029,
+  "<mask_172>": 51028,
+  "<mask_173>": 51027,
+  "<mask_174>": 51026,
+  "<mask_175>": 51025,
+  "<mask_176>": 51024,
+  "<mask_177>": 51023,
+  "<mask_178>": 51022,
+  "<mask_179>": 51021,
+  "<mask_17>": 51183,
+  "<mask_180>": 51020,
+  "<mask_181>": 51019,
+  "<mask_182>": 51018,
+  "<mask_183>": 51017,
+  "<mask_184>": 51016,
+  "<mask_185>": 51015,
+  "<mask_186>": 51014,
+  "<mask_187>": 51013,
+  "<mask_188>": 51012,
+  "<mask_189>": 51011,
+  "<mask_18>": 51182,
+  "<mask_190>": 51010,
+  "<mask_191>": 51009,
+  "<mask_192>": 51008,
+  "<mask_193>": 51007,
+  "<mask_194>": 51006,
+  "<mask_195>": 51005,
+  "<mask_196>": 51004,
+  "<mask_197>": 51003,
+  "<mask_198>": 51002,
+  "<mask_199>": 51001,
+  "<mask_19>": 51181,
+  "<mask_1>": 51199,
+  "<mask_200>": 51000,
+  "<mask_201>": 50999,
+  "<mask_202>": 50998,
+  "<mask_203>": 50997,
+  "<mask_204>": 50996,
+  "<mask_205>": 50995,
+  "<mask_206>": 50994,
+  "<mask_207>": 50993,
+  "<mask_208>": 50992,
+  "<mask_209>": 50991,
+  "<mask_20>": 51180,
+  "<mask_210>": 50990,
+  "<mask_211>": 50989,
+  "<mask_212>": 50988,
+  "<mask_213>": 50987,
+  "<mask_214>": 50986,
+  "<mask_215>": 50985,
+  "<mask_216>": 50984,
+  "<mask_217>": 50983,
+  "<mask_218>": 50982,
+  "<mask_219>": 50981,
+  "<mask_21>": 51179,
+  "<mask_220>": 50980,
+  "<mask_221>": 50979,
+  "<mask_222>": 50978,
+  "<mask_223>": 50977,
+  "<mask_224>": 50976,
+  "<mask_225>": 50975,
+  "<mask_226>": 50974,
+  "<mask_227>": 50973,
+  "<mask_228>": 50972,
+  "<mask_229>": 50971,
+  "<mask_22>": 51178,
+  "<mask_230>": 50970,
+  "<mask_231>": 50969,
+  "<mask_232>": 50968,
+  "<mask_233>": 50967,
+  "<mask_234>": 50966,
+  "<mask_235>": 50965,
+  "<mask_236>": 50964,
+  "<mask_237>": 50963,
+  "<mask_238>": 50962,
+  "<mask_239>": 50961,
+  "<mask_23>": 51177,
+  "<mask_240>": 50960,
+  "<mask_241>": 50959,
+  "<mask_242>": 50958,
+  "<mask_243>": 50957,
+  "<mask_244>": 50956,
+  "<mask_245>": 50955,
+  "<mask_246>": 50954,
+  "<mask_247>": 50953,
+  "<mask_248>": 50952,
+  "<mask_249>": 50951,
+  "<mask_24>": 51176,
+  "<mask_250>": 50950,
+  "<mask_251>": 50949,
+  "<mask_252>": 50948,
+  "<mask_253>": 50947,
+  "<mask_254>": 50946,
+  "<mask_255>": 50945,
+  "<mask_256>": 50944,
+  "<mask_257>": 50943,
+  "<mask_258>": 50942,
+  "<mask_259>": 50941,
+  "<mask_25>": 51175,
+  "<mask_260>": 50940,
+  "<mask_261>": 50939,
+  "<mask_262>": 50938,
+  "<mask_263>": 50937,
+  "<mask_264>": 50936,
+  "<mask_265>": 50935,
+  "<mask_266>": 50934,
+  "<mask_267>": 50933,
+  "<mask_268>": 50932,
+  "<mask_269>": 50931,
+  "<mask_26>": 51174,
+  "<mask_270>": 50930,
+  "<mask_271>": 50929,
+  "<mask_272>": 50928,
+  "<mask_273>": 50927,
+  "<mask_274>": 50926,
+  "<mask_275>": 50925,
+  "<mask_276>": 50924,
+  "<mask_277>": 50923,
+  "<mask_278>": 50922,
+  "<mask_279>": 50921,
+  "<mask_27>": 51173,
+  "<mask_280>": 50920,
+  "<mask_281>": 50919,
+  "<mask_282>": 50918,
+  "<mask_283>": 50917,
+  "<mask_284>": 50916,
+  "<mask_285>": 50915,
+  "<mask_286>": 50914,
+  "<mask_287>": 50913,
+  "<mask_288>": 50912,
+  "<mask_289>": 50911,
+  "<mask_28>": 51172,
+  "<mask_290>": 50910,
+  "<mask_291>": 50909,
+  "<mask_292>": 50908,
+  "<mask_293>": 50907,
+  "<mask_294>": 50906,
+  "<mask_295>": 50905,
+  "<mask_296>": 50904,
+  "<mask_297>": 50903,
+  "<mask_298>": 50902,
+  "<mask_299>": 50901,
+  "<mask_29>": 51171,
+  "<mask_2>": 51198,
+  "<mask_300>": 50900,
+  "<mask_301>": 50899,
+  "<mask_302>": 50898,
+  "<mask_303>": 50897,
+  "<mask_304>": 50896,
+  "<mask_305>": 50895,
+  "<mask_306>": 50894,
+  "<mask_307>": 50893,
+  "<mask_308>": 50892,
+  "<mask_309>": 50891,
+  "<mask_30>": 51170,
+  "<mask_310>": 50890,
+  "<mask_311>": 50889,
+  "<mask_312>": 50888,
+  "<mask_313>": 50887,
+  "<mask_314>": 50886,
+  "<mask_315>": 50885,
+  "<mask_316>": 50884,
+  "<mask_317>": 50883,
+  "<mask_318>": 50882,
+  "<mask_319>": 50881,
+  "<mask_31>": 51169,
+  "<mask_320>": 50880,
+  "<mask_321>": 50879,
+  "<mask_322>": 50878,
+  "<mask_323>": 50877,
+  "<mask_324>": 50876,
+  "<mask_325>": 50875,
+  "<mask_326>": 50874,
+  "<mask_327>": 50873,
+  "<mask_328>": 50872,
+  "<mask_329>": 50871,
+  "<mask_32>": 51168,
+  "<mask_330>": 50870,
+  "<mask_331>": 50869,
+  "<mask_332>": 50868,
+  "<mask_333>": 50867,
+  "<mask_334>": 50866,
+  "<mask_335>": 50865,
+  "<mask_336>": 50864,
+  "<mask_337>": 50863,
+  "<mask_338>": 50862,
+  "<mask_339>": 50861,
+  "<mask_33>": 51167,
+  "<mask_340>": 50860,
+  "<mask_341>": 50859,
+  "<mask_342>": 50858,
+  "<mask_343>": 50857,
+  "<mask_344>": 50856,
+  "<mask_345>": 50855,
+  "<mask_346>": 50854,
+  "<mask_347>": 50853,
+  "<mask_348>": 50852,
+  "<mask_349>": 50851,
+  "<mask_34>": 51166,
+  "<mask_350>": 50850,
+  "<mask_351>": 50849,
+  "<mask_352>": 50848,
+  "<mask_353>": 50847,
+  "<mask_354>": 50846,
+  "<mask_355>": 50845,
+  "<mask_356>": 50844,
+  "<mask_357>": 50843,
+  "<mask_358>": 50842,
+  "<mask_359>": 50841,
+  "<mask_35>": 51165,
+  "<mask_360>": 50840,
+  "<mask_361>": 50839,
+  "<mask_362>": 50838,
+  "<mask_363>": 50837,
+  "<mask_364>": 50836,
+  "<mask_365>": 50835,
+  "<mask_366>": 50834,
+  "<mask_367>": 50833,
+  "<mask_368>": 50832,
+  "<mask_369>": 50831,
+  "<mask_36>": 51164,
+  "<mask_370>": 50830,
+  "<mask_371>": 50829,
+  "<mask_372>": 50828,
+  "<mask_373>": 50827,
+  "<mask_374>": 50826,
+  "<mask_375>": 50825,
+  "<mask_376>": 50824,
+  "<mask_377>": 50823,
+  "<mask_378>": 50822,
+  "<mask_379>": 50821,
+  "<mask_37>": 51163,
+  "<mask_380>": 50820,
+  "<mask_381>": 50819,
+  "<mask_382>": 50818,
+  "<mask_383>": 50817,
+  "<mask_384>": 50816,
+  "<mask_385>": 50815,
+  "<mask_386>": 50814,
+  "<mask_387>": 50813,
+  "<mask_388>": 50812,
+  "<mask_389>": 50811,
+  "<mask_38>": 51162,
+  "<mask_390>": 50810,
+  "<mask_391>": 50809,
+  "<mask_392>": 50808,
+  "<mask_393>": 50807,
+  "<mask_394>": 50806,
+  "<mask_395>": 50805,
+  "<mask_396>": 50804,
+  "<mask_397>": 50803,
+  "<mask_398>": 50802,
+  "<mask_399>": 50801,
+  "<mask_39>": 51161,
+  "<mask_3>": 51197,
+  "<mask_400>": 50800,
+  "<mask_401>": 50799,
+  "<mask_402>": 50798,
+  "<mask_403>": 50797,
+  "<mask_404>": 50796,
+  "<mask_405>": 50795,
+  "<mask_406>": 50794,
+  "<mask_407>": 50793,
+  "<mask_408>": 50792,
+  "<mask_409>": 50791,
+  "<mask_40>": 51160,
+  "<mask_410>": 50790,
+  "<mask_411>": 50789,
+  "<mask_412>": 50788,
+  "<mask_413>": 50787,
+  "<mask_414>": 50786,
+  "<mask_415>": 50785,
+  "<mask_416>": 50784,
+  "<mask_417>": 50783,
+  "<mask_418>": 50782,
+  "<mask_419>": 50781,
+  "<mask_41>": 51159,
+  "<mask_420>": 50780,
+  "<mask_421>": 50779,
+  "<mask_422>": 50778,
+  "<mask_423>": 50777,
+  "<mask_424>": 50776,
+  "<mask_425>": 50775,
+  "<mask_426>": 50774,
+  "<mask_427>": 50773,
+  "<mask_428>": 50772,
+  "<mask_429>": 50771,
+  "<mask_42>": 51158,
+  "<mask_430>": 50770,
+  "<mask_431>": 50769,
+  "<mask_432>": 50768,
+  "<mask_433>": 50767,
+  "<mask_434>": 50766,
+  "<mask_435>": 50765,
+  "<mask_436>": 50764,
+  "<mask_437>": 50763,
+  "<mask_438>": 50762,
+  "<mask_439>": 50761,
+  "<mask_43>": 51157,
+  "<mask_440>": 50760,
+  "<mask_441>": 50759,
+  "<mask_442>": 50758,
+  "<mask_443>": 50757,
+  "<mask_444>": 50756,
+  "<mask_445>": 50755,
+  "<mask_446>": 50754,
+  "<mask_447>": 50753,
+  "<mask_448>": 50752,
+  "<mask_449>": 50751,
+  "<mask_44>": 51156,
+  "<mask_450>": 50750,
+  "<mask_451>": 50749,
+  "<mask_452>": 50748,
+  "<mask_453>": 50747,
+  "<mask_454>": 50746,
+  "<mask_455>": 50745,
+  "<mask_456>": 50744,
+  "<mask_457>": 50743,
+  "<mask_458>": 50742,
+  "<mask_459>": 50741,
+  "<mask_45>": 51155,
+  "<mask_460>": 50740,
+  "<mask_461>": 50739,
+  "<mask_462>": 50738,
+  "<mask_463>": 50737,
+  "<mask_464>": 50736,
+  "<mask_465>": 50735,
+  "<mask_466>": 50734,
+  "<mask_467>": 50733,
+  "<mask_468>": 50732,
+  "<mask_469>": 50731,
+  "<mask_46>": 51154,
+  "<mask_470>": 50730,
+  "<mask_471>": 50729,
+  "<mask_472>": 50728,
+  "<mask_473>": 50727,
+  "<mask_474>": 50726,
+  "<mask_475>": 50725,
+  "<mask_476>": 50724,
+  "<mask_477>": 50723,
+  "<mask_478>": 50722,
+  "<mask_479>": 50721,
+  "<mask_47>": 51153,
+  "<mask_480>": 50720,
+  "<mask_481>": 50719,
+  "<mask_482>": 50718,
+  "<mask_483>": 50717,
+  "<mask_484>": 50716,
+  "<mask_485>": 50715,
+  "<mask_486>": 50714,
+  "<mask_487>": 50713,
+  "<mask_488>": 50712,
+  "<mask_489>": 50711,
+  "<mask_48>": 51152,
+  "<mask_490>": 50710,
+  "<mask_491>": 50709,
+  "<mask_492>": 50708,
+  "<mask_493>": 50707,
+  "<mask_494>": 50706,
+  "<mask_495>": 50705,
+  "<mask_496>": 50704,
+  "<mask_497>": 50703,
+  "<mask_498>": 50702,
+  "<mask_499>": 50701,
+  "<mask_49>": 51151,
+  "<mask_4>": 51196,
+  "<mask_500>": 50700,
+  "<mask_501>": 50699,
+  "<mask_502>": 50698,
+  "<mask_503>": 50697,
+  "<mask_504>": 50696,
+  "<mask_505>": 50695,
+  "<mask_506>": 50694,
+  "<mask_507>": 50693,
+  "<mask_508>": 50692,
+  "<mask_509>": 50691,
+  "<mask_50>": 51150,
+  "<mask_510>": 50690,
+  "<mask_511>": 50689,
+  "<mask_512>": 50688,
+  "<mask_513>": 50687,
+  "<mask_514>": 50686,
+  "<mask_515>": 50685,
+  "<mask_516>": 50684,
+  "<mask_517>": 50683,
+  "<mask_518>": 50682,
+  "<mask_519>": 50681,
+  "<mask_51>": 51149,
+  "<mask_520>": 50680,
+  "<mask_521>": 50679,
+  "<mask_522>": 50678,
+  "<mask_523>": 50677,
+  "<mask_524>": 50676,
+  "<mask_525>": 50675,
+  "<mask_526>": 50674,
+  "<mask_527>": 50673,
+  "<mask_528>": 50672,
+  "<mask_529>": 50671,
+  "<mask_52>": 51148,
+  "<mask_530>": 50670,
+  "<mask_531>": 50669,
+  "<mask_532>": 50668,
+  "<mask_533>": 50667,
+  "<mask_534>": 50666,
+  "<mask_535>": 50665,
+  "<mask_536>": 50664,
+  "<mask_537>": 50663,
+  "<mask_538>": 50662,
+  "<mask_539>": 50661,
+  "<mask_53>": 51147,
+  "<mask_540>": 50660,
+  "<mask_541>": 50659,
+  "<mask_542>": 50658,
+  "<mask_543>": 50657,
+  "<mask_544>": 50656,
+  "<mask_545>": 50655,
+  "<mask_546>": 50654,
+  "<mask_547>": 50653,
+  "<mask_548>": 50652,
+  "<mask_549>": 50651,
+  "<mask_54>": 51146,
+  "<mask_550>": 50650,
+  "<mask_551>": 50649,
+  "<mask_552>": 50648,
+  "<mask_553>": 50647,
+  "<mask_554>": 50646,
+  "<mask_555>": 50645,
+  "<mask_556>": 50644,
+  "<mask_557>": 50643,
+  "<mask_558>": 50642,
+  "<mask_559>": 50641,
+  "<mask_55>": 51145,
+  "<mask_560>": 50640,
+  "<mask_561>": 50639,
+  "<mask_562>": 50638,
+  "<mask_563>": 50637,
+  "<mask_564>": 50636,
+  "<mask_565>": 50635,
+  "<mask_566>": 50634,
+  "<mask_567>": 50633,
+  "<mask_568>": 50632,
+  "<mask_569>": 50631,
+  "<mask_56>": 51144,
+  "<mask_570>": 50630,
+  "<mask_571>": 50629,
+  "<mask_572>": 50628,
+  "<mask_573>": 50627,
+  "<mask_574>": 50626,
+  "<mask_575>": 50625,
+  "<mask_576>": 50624,
+  "<mask_577>": 50623,
+  "<mask_578>": 50622,
+  "<mask_579>": 50621,
+  "<mask_57>": 51143,
+  "<mask_580>": 50620,
+  "<mask_581>": 50619,
+  "<mask_582>": 50618,
+  "<mask_583>": 50617,
+  "<mask_584>": 50616,
+  "<mask_585>": 50615,
+  "<mask_586>": 50614,
+  "<mask_587>": 50613,
+  "<mask_588>": 50612,
+  "<mask_589>": 50611,
+  "<mask_58>": 51142,
+  "<mask_590>": 50610,
+  "<mask_591>": 50609,
+  "<mask_592>": 50608,
+  "<mask_593>": 50607,
+  "<mask_594>": 50606,
+  "<mask_595>": 50605,
+  "<mask_596>": 50604,
+  "<mask_597>": 50603,
+  "<mask_598>": 50602,
+  "<mask_599>": 50601,
+  "<mask_59>": 51141,
+  "<mask_5>": 51195,
+  "<mask_600>": 50600,
+  "<mask_601>": 50599,
+  "<mask_602>": 50598,
+  "<mask_603>": 50597,
+  "<mask_604>": 50596,
+  "<mask_605>": 50595,
+  "<mask_606>": 50594,
+  "<mask_607>": 50593,
+  "<mask_608>": 50592,
+  "<mask_609>": 50591,
+  "<mask_60>": 51140,
+  "<mask_610>": 50590,
+  "<mask_611>": 50589,
+  "<mask_612>": 50588,
+  "<mask_613>": 50587,
+  "<mask_614>": 50586,
+  "<mask_615>": 50585,
+  "<mask_616>": 50584,
+  "<mask_617>": 50583,
+  "<mask_618>": 50582,
+  "<mask_619>": 50581,
+  "<mask_61>": 51139,
+  "<mask_620>": 50580,
+  "<mask_621>": 50579,
+  "<mask_622>": 50578,
+  "<mask_623>": 50577,
+  "<mask_624>": 50576,
+  "<mask_625>": 50575,
+  "<mask_626>": 50574,
+  "<mask_627>": 50573,
+  "<mask_628>": 50572,
+  "<mask_629>": 50571,
+  "<mask_62>": 51138,
+  "<mask_630>": 50570,
+  "<mask_631>": 50569,
+  "<mask_632>": 50568,
+  "<mask_633>": 50567,
+  "<mask_634>": 50566,
+  "<mask_635>": 50565,
+  "<mask_636>": 50564,
+  "<mask_637>": 50563,
+  "<mask_638>": 50562,
+  "<mask_639>": 50561,
+  "<mask_63>": 51137,
+  "<mask_640>": 50560,
+  "<mask_641>": 50559,
+  "<mask_642>": 50558,
+  "<mask_643>": 50557,
+  "<mask_644>": 50556,
+  "<mask_645>": 50555,
+  "<mask_646>": 50554,
+  "<mask_647>": 50553,
+  "<mask_648>": 50552,
+  "<mask_649>": 50551,
+  "<mask_64>": 51136,
+  "<mask_650>": 50550,
+  "<mask_651>": 50549,
+  "<mask_652>": 50548,
+  "<mask_653>": 50547,
+  "<mask_654>": 50546,
+  "<mask_655>": 50545,
+  "<mask_656>": 50544,
+  "<mask_657>": 50543,
+  "<mask_658>": 50542,
+  "<mask_659>": 50541,
+  "<mask_65>": 51135,
+  "<mask_660>": 50540,
+  "<mask_661>": 50539,
+  "<mask_662>": 50538,
+  "<mask_663>": 50537,
+  "<mask_664>": 50536,
+  "<mask_665>": 50535,
+  "<mask_666>": 50534,
+  "<mask_667>": 50533,
+  "<mask_668>": 50532,
+  "<mask_669>": 50531,
+  "<mask_66>": 51134,
+  "<mask_670>": 50530,
+  "<mask_671>": 50529,
+  "<mask_672>": 50528,
+  "<mask_673>": 50527,
+  "<mask_674>": 50526,
+  "<mask_675>": 50525,
+  "<mask_676>": 50524,
+  "<mask_677>": 50523,
+  "<mask_678>": 50522,
+  "<mask_679>": 50521,
+  "<mask_67>": 51133,
+  "<mask_680>": 50520,
+  "<mask_681>": 50519,
+  "<mask_682>": 50518,
+  "<mask_683>": 50517,
+  "<mask_684>": 50516,
+  "<mask_685>": 50515,
+  "<mask_686>": 50514,
+  "<mask_687>": 50513,
+  "<mask_688>": 50512,
+  "<mask_689>": 50511,
+  "<mask_68>": 51132,
+  "<mask_690>": 50510,
+  "<mask_691>": 50509,
+  "<mask_692>": 50508,
+  "<mask_693>": 50507,
+  "<mask_694>": 50506,
+  "<mask_695>": 50505,
+  "<mask_696>": 50504,
+  "<mask_697>": 50503,
+  "<mask_698>": 50502,
+  "<mask_699>": 50501,
+  "<mask_69>": 51131,
+  "<mask_6>": 51194,
+  "<mask_700>": 50500,
+  "<mask_701>": 50499,
+  "<mask_702>": 50498,
+  "<mask_703>": 50497,
+  "<mask_704>": 50496,
+  "<mask_705>": 50495,
+  "<mask_706>": 50494,
+  "<mask_707>": 50493,
+  "<mask_708>": 50492,
+  "<mask_709>": 50491,
+  "<mask_70>": 51130,
+  "<mask_710>": 50490,
+  "<mask_711>": 50489,
+  "<mask_712>": 50488,
+  "<mask_713>": 50487,
+  "<mask_714>": 50486,
+  "<mask_715>": 50485,
+  "<mask_716>": 50484,
+  "<mask_717>": 50483,
+  "<mask_718>": 50482,
+  "<mask_719>": 50481,
+  "<mask_71>": 51129,
+  "<mask_720>": 50480,
+  "<mask_721>": 50479,
+  "<mask_722>": 50478,
+  "<mask_723>": 50477,
+  "<mask_724>": 50476,
+  "<mask_725>": 50475,
+  "<mask_726>": 50474,
+  "<mask_727>": 50473,
+  "<mask_728>": 50472,
+  "<mask_729>": 50471,
+  "<mask_72>": 51128,
+  "<mask_730>": 50470,
+  "<mask_731>": 50469,
+  "<mask_732>": 50468,
+  "<mask_733>": 50467,
+  "<mask_734>": 50466,
+  "<mask_735>": 50465,
+  "<mask_736>": 50464,
+  "<mask_737>": 50463,
+  "<mask_738>": 50462,
+  "<mask_739>": 50461,
+  "<mask_73>": 51127,
+  "<mask_740>": 50460,
+  "<mask_741>": 50459,
+  "<mask_742>": 50458,
+  "<mask_743>": 50457,
+  "<mask_744>": 50456,
+  "<mask_745>": 50455,
+  "<mask_746>": 50454,
+  "<mask_747>": 50453,
+  "<mask_748>": 50452,
+  "<mask_749>": 50451,
+  "<mask_74>": 51126,
+  "<mask_750>": 50450,
+  "<mask_751>": 50449,
+  "<mask_752>": 50448,
+  "<mask_753>": 50447,
+  "<mask_754>": 50446,
+  "<mask_755>": 50445,
+  "<mask_756>": 50444,
+  "<mask_757>": 50443,
+  "<mask_758>": 50442,
+  "<mask_759>": 50441,
+  "<mask_75>": 51125,
+  "<mask_760>": 50440,
+  "<mask_761>": 50439,
+  "<mask_762>": 50438,
+  "<mask_763>": 50437,
+  "<mask_764>": 50436,
+  "<mask_765>": 50435,
+  "<mask_766>": 50434,
+  "<mask_767>": 50433,
+  "<mask_768>": 50432,
+  "<mask_769>": 50431,
+  "<mask_76>": 51124,
+  "<mask_770>": 50430,
+  "<mask_771>": 50429,
+  "<mask_772>": 50428,
+  "<mask_773>": 50427,
+  "<mask_774>": 50426,
+  "<mask_775>": 50425,
+  "<mask_776>": 50424,
+  "<mask_777>": 50423,
+  "<mask_778>": 50422,
+  "<mask_779>": 50421,
+  "<mask_77>": 51123,
+  "<mask_780>": 50420,
+  "<mask_781>": 50419,
+  "<mask_782>": 50418,
+  "<mask_783>": 50417,
+  "<mask_784>": 50416,
+  "<mask_785>": 50415,
+  "<mask_786>": 50414,
+  "<mask_787>": 50413,
+  "<mask_788>": 50412,
+  "<mask_789>": 50411,
+  "<mask_78>": 51122,
+  "<mask_790>": 50410,
+  "<mask_791>": 50409,
+  "<mask_792>": 50408,
+  "<mask_793>": 50407,
+  "<mask_794>": 50406,
+  "<mask_795>": 50405,
+  "<mask_796>": 50404,
+  "<mask_797>": 50403,
+  "<mask_798>": 50402,
+  "<mask_799>": 50401,
+  "<mask_79>": 51121,
+  "<mask_7>": 51193,
+  "<mask_800>": 50400,
+  "<mask_801>": 50399,
+  "<mask_802>": 50398,
+  "<mask_803>": 50397,
+  "<mask_804>": 50396,
+  "<mask_805>": 50395,
+  "<mask_806>": 50394,
+  "<mask_807>": 50393,
+  "<mask_808>": 50392,
+  "<mask_809>": 50391,
+  "<mask_80>": 51120,
+  "<mask_810>": 50390,
+  "<mask_811>": 50389,
+  "<mask_812>": 50388,
+  "<mask_813>": 50387,
+  "<mask_814>": 50386,
+  "<mask_815>": 50385,
+  "<mask_816>": 50384,
+  "<mask_817>": 50383,
+  "<mask_818>": 50382,
+  "<mask_819>": 50381,
+  "<mask_81>": 51119,
+  "<mask_820>": 50380,
+  "<mask_821>": 50379,
+  "<mask_822>": 50378,
+  "<mask_823>": 50377,
+  "<mask_824>": 50376,
+  "<mask_825>": 50375,
+  "<mask_826>": 50374,
+  "<mask_827>": 50373,
+  "<mask_828>": 50372,
+  "<mask_829>": 50371,
+  "<mask_82>": 51118,
+  "<mask_830>": 50370,
+  "<mask_831>": 50369,
+  "<mask_832>": 50368,
+  "<mask_833>": 50367,
+  "<mask_834>": 50366,
+  "<mask_835>": 50365,
+  "<mask_836>": 50364,
+  "<mask_837>": 50363,
+  "<mask_838>": 50362,
+  "<mask_839>": 50361,
+  "<mask_83>": 51117,
+  "<mask_840>": 50360,
+  "<mask_841>": 50359,
+  "<mask_842>": 50358,
+  "<mask_843>": 50357,
+  "<mask_844>": 50356,
+  "<mask_845>": 50355,
+  "<mask_846>": 50354,
+  "<mask_847>": 50353,
+  "<mask_848>": 50352,
+  "<mask_849>": 50351,
+  "<mask_84>": 51116,
+  "<mask_850>": 50350,
+  "<mask_851>": 50349,
+  "<mask_852>": 50348,
+  "<mask_853>": 50347,
+  "<mask_854>": 50346,
+  "<mask_855>": 50345,
+  "<mask_856>": 50344,
+  "<mask_857>": 50343,
+  "<mask_858>": 50342,
+  "<mask_859>": 50341,
+  "<mask_85>": 51115,
+  "<mask_860>": 50340,
+  "<mask_861>": 50339,
+  "<mask_862>": 50338,
+  "<mask_863>": 50337,
+  "<mask_864>": 50336,
+  "<mask_865>": 50335,
+  "<mask_866>": 50334,
+  "<mask_867>": 50333,
+  "<mask_868>": 50332,
+  "<mask_869>": 50331,
+  "<mask_86>": 51114,
+  "<mask_870>": 50330,
+  "<mask_871>": 50329,
+  "<mask_872>": 50328,
+  "<mask_873>": 50327,
+  "<mask_874>": 50326,
+  "<mask_875>": 50325,
+  "<mask_876>": 50324,
+  "<mask_877>": 50323,
+  "<mask_878>": 50322,
+  "<mask_879>": 50321,
+  "<mask_87>": 51113,
+  "<mask_880>": 50320,
+  "<mask_881>": 50319,
+  "<mask_882>": 50318,
+  "<mask_883>": 50317,
+  "<mask_884>": 50316,
+  "<mask_885>": 50315,
+  "<mask_886>": 50314,
+  "<mask_887>": 50313,
+  "<mask_888>": 50312,
+  "<mask_889>": 50311,
+  "<mask_88>": 51112,
+  "<mask_890>": 50310,
+  "<mask_891>": 50309,
+  "<mask_892>": 50308,
+  "<mask_893>": 50307,
+  "<mask_894>": 50306,
+  "<mask_895>": 50305,
+  "<mask_896>": 50304,
+  "<mask_897>": 50303,
+  "<mask_898>": 50302,
+  "<mask_899>": 50301,
+  "<mask_89>": 51111,
+  "<mask_8>": 51192,
+  "<mask_90>": 51110,
+  "<mask_91>": 51109,
+  "<mask_92>": 51108,
+  "<mask_93>": 51107,
+  "<mask_94>": 51106,
+  "<mask_95>": 51105,
+  "<mask_96>": 51104,
+  "<mask_97>": 51103,
+  "<mask_98>": 51102,
+  "<mask_99>": 51101,
+  "<mask_9>": 51191,
+  "<sep>": 50299
+}

checkpoint-384/README.md

@@ -0,0 +1,202 @@
+---
+base_model: Salesforce/codegen2-16B_P
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.14.0

checkpoint-384/adapter_config.json

@@ -0,0 +1,35 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "Salesforce/codegen2-16B_P",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": [
+    "wte",
+    "lm_head"
+  ],
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "qkv_proj",
+    "out_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-384/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dad68fd47d0ca79638a2185e50edcf12c6f04458c1dd7d82cd89b8ba210d950c
+size 1298520480

checkpoint-384/global_step384/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74222527a8d877aad3828526b44fcd608a869281719e287460ece659cf7e6341
+size 3835348208

checkpoint-384/global_step384/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ece7bcfaf4f3ca150548a156f7231b1b36cd78a5505a41560070d06fed760e10
+size 3835355312

checkpoint-384/global_step384/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bb290b78c0cd5662548db0bc02fd952d40ab7510f8ca6469dcaa1cbb3b4dbd4
+size 3835355312

checkpoint-384/global_step384/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b823f70f5fbf5779276919bb20fd9129c28027d89bf82341c154368a2aff88aa
+size 3835348144

checkpoint-384/global_step384/mp_rank_00_model_states.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e15cb581a7335807bfb546dc9e2ed89361ad018c4e7be245d7c6562c8bc53be8
+size 2699642858

checkpoint-384/latest

@@ -0,0 +1 @@
+global_step384

checkpoint-384/rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa85d56c5321c002e1647994d628dab5e3cd4796535b50075ffd820498b048ef
+size 15024

checkpoint-384/rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c48defcc06f8ea9896fee0c140c14a21cc584eb37340579e4c5bc5664d987e95
+size 15024

checkpoint-384/rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:378cbab57014e78b0d18d5892f0da74720b514dc32520dc03258162eda1f4f2f
+size 15024

checkpoint-384/rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb0271db62a3a09bbba0639e805ac86f2f917ee9fc8dc044148b606f8c44ebec
+size 15024

checkpoint-384/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89a5746b43df091022c92ec7086b154664f55227131ae07280a849ce36e06dd6
+size 1064

checkpoint-384/trainer_state.json

@@ -0,0 +1,306 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9986996098829649,
+  "eval_steps": 500,
+  "global_step": 384,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.002600780234070221,
+      "grad_norm": 0.02962934412062168,
+      "learning_rate": 1.282051282051282e-06,
+      "loss": 0.619,
+      "step": 1
+    },
+    {
+      "epoch": 0.02600780234070221,
+      "grad_norm": 0.06379027664661407,
+      "learning_rate": 1.282051282051282e-05,
+      "loss": 0.6962,
+      "step": 10
+    },
+    {
+      "epoch": 0.05201560468140442,
+      "grad_norm": 0.0363883376121521,
+      "learning_rate": 2.564102564102564e-05,
+      "loss": 0.7759,
+      "step": 20
+    },
+    {
+      "epoch": 0.07802340702210664,
+      "grad_norm": 0.03419478237628937,
+      "learning_rate": 3.846153846153846e-05,
+      "loss": 0.8087,
+      "step": 30
+    },
+    {
+      "epoch": 0.10403120936280884,
+      "grad_norm": 0.04424262419342995,
+      "learning_rate": 4.985507246376812e-05,
+      "loss": 0.7775,
+      "step": 40
+    },
+    {
+      "epoch": 0.13003901170351106,
+      "grad_norm": 0.22272075712680817,
+      "learning_rate": 4.840579710144928e-05,
+      "loss": 0.7476,
+      "step": 50
+    },
+    {
+      "epoch": 0.15604681404421328,
+      "grad_norm": 0.049193304032087326,
+      "learning_rate": 4.695652173913044e-05,
+      "loss": 0.6617,
+      "step": 60
+    },
+    {
+      "epoch": 0.18205461638491546,
+      "grad_norm": 0.04189423844218254,
+      "learning_rate": 4.5507246376811595e-05,
+      "loss": 0.7254,
+      "step": 70
+    },
+    {
+      "epoch": 0.20806241872561768,
+      "grad_norm": 0.033223457634449005,
+      "learning_rate": 4.405797101449275e-05,
+      "loss": 0.7454,
+      "step": 80
+    },
+    {
+      "epoch": 0.2340702210663199,
+      "grad_norm": 0.023022688925266266,
+      "learning_rate": 4.2608695652173916e-05,
+      "loss": 0.7263,
+      "step": 90
+    },
+    {
+      "epoch": 0.26007802340702213,
+      "grad_norm": 0.1517011970281601,
+      "learning_rate": 4.115942028985507e-05,
+      "loss": 0.7241,
+      "step": 100
+    },
+    {
+      "epoch": 0.28608582574772434,
+      "grad_norm": 0.041623640805482864,
+      "learning_rate": 3.971014492753624e-05,
+      "loss": 0.647,
+      "step": 110
+    },
+    {
+      "epoch": 0.31209362808842656,
+      "grad_norm": 0.03412195295095444,
+      "learning_rate": 3.8260869565217395e-05,
+      "loss": 0.6991,
+      "step": 120
+    },
+    {
+      "epoch": 0.3381014304291287,
+      "grad_norm": 0.02426602691411972,
+      "learning_rate": 3.681159420289855e-05,
+      "loss": 0.7115,
+      "step": 130
+    },
+    {
+      "epoch": 0.3641092327698309,
+      "grad_norm": 0.023634808138012886,
+      "learning_rate": 3.536231884057971e-05,
+      "loss": 0.6992,
+      "step": 140
+    },
+    {
+      "epoch": 0.39011703511053314,
+      "grad_norm": 0.1857312172651291,
+      "learning_rate": 3.3913043478260867e-05,
+      "loss": 0.7133,
+      "step": 150
+    },
+    {
+      "epoch": 0.41612483745123535,
+      "grad_norm": 0.057914506644010544,
+      "learning_rate": 3.246376811594203e-05,
+      "loss": 0.637,
+      "step": 160
+    },
+    {
+      "epoch": 0.44213263979193757,
+      "grad_norm": 0.0314478725194931,
+      "learning_rate": 3.1014492753623195e-05,
+      "loss": 0.69,
+      "step": 170
+    },
+    {
+      "epoch": 0.4681404421326398,
+      "grad_norm": 0.02375701256096363,
+      "learning_rate": 2.9565217391304352e-05,
+      "loss": 0.7052,
+      "step": 180
+    },
+    {
+      "epoch": 0.494148244473342,
+      "grad_norm": 0.017046812921762466,
+      "learning_rate": 2.811594202898551e-05,
+      "loss": 0.6963,
+      "step": 190
+    },
+    {
+      "epoch": 0.5201560468140443,
+      "grad_norm": 0.14757999777793884,
+      "learning_rate": 2.6666666666666667e-05,
+      "loss": 0.699,
+      "step": 200
+    },
+    {
+      "epoch": 0.5461638491547465,
+      "grad_norm": 0.03953570872545242,
+      "learning_rate": 2.5217391304347827e-05,
+      "loss": 0.6362,
+      "step": 210
+    },
+    {
+      "epoch": 0.5721716514954487,
+      "grad_norm": 0.031761154532432556,
+      "learning_rate": 2.3768115942028988e-05,
+      "loss": 0.6929,
+      "step": 220
+    },
+    {
+      "epoch": 0.5981794538361509,
+      "grad_norm": 0.019830092787742615,
+      "learning_rate": 2.2318840579710145e-05,
+      "loss": 0.6936,
+      "step": 230
+    },
+    {
+      "epoch": 0.6241872561768531,
+      "grad_norm": 0.017688650637865067,
+      "learning_rate": 2.0869565217391303e-05,
+      "loss": 0.692,
+      "step": 240
+    },
+    {
+      "epoch": 0.6501950585175552,
+      "grad_norm": 0.18702688813209534,
+      "learning_rate": 1.9420289855072467e-05,
+      "loss": 0.7103,
+      "step": 250
+    },
+    {
+      "epoch": 0.6762028608582574,
+      "grad_norm": 0.03623680770397186,
+      "learning_rate": 1.7971014492753624e-05,
+      "loss": 0.6185,
+      "step": 260
+    },
+    {
+      "epoch": 0.7022106631989596,
+      "grad_norm": 0.026319777593016624,
+      "learning_rate": 1.652173913043478e-05,
+      "loss": 0.7065,
+      "step": 270
+    },
+    {
+      "epoch": 0.7282184655396619,
+      "grad_norm": 0.018396981060504913,
+      "learning_rate": 1.5072463768115944e-05,
+      "loss": 0.6869,
+      "step": 280
+    },
+    {
+      "epoch": 0.7542262678803641,
+      "grad_norm": 0.016413649544119835,
+      "learning_rate": 1.3623188405797103e-05,
+      "loss": 0.6865,
+      "step": 290
+    },
+    {
+      "epoch": 0.7802340702210663,
+      "grad_norm": 0.1341114193201065,
+      "learning_rate": 1.2173913043478261e-05,
+      "loss": 0.7022,
+      "step": 300
+    },
+    {
+      "epoch": 0.8062418725617685,
+      "grad_norm": 0.03741007670760155,
+      "learning_rate": 1.072463768115942e-05,
+      "loss": 0.6272,
+      "step": 310
+    },
+    {
+      "epoch": 0.8322496749024707,
+      "grad_norm": 0.024399157613515854,
+      "learning_rate": 9.27536231884058e-06,
+      "loss": 0.6793,
+      "step": 320
+    },
+    {
+      "epoch": 0.8582574772431729,
+      "grad_norm": 0.016972342506051064,
+      "learning_rate": 7.82608695652174e-06,
+      "loss": 0.7078,
+      "step": 330
+    },
+    {
+      "epoch": 0.8842652795838751,
+      "grad_norm": 0.014587855897843838,
+      "learning_rate": 6.376811594202898e-06,
+      "loss": 0.7041,
+      "step": 340
+    },
+    {
+      "epoch": 0.9102730819245773,
+      "grad_norm": 0.13855686783790588,
+      "learning_rate": 4.927536231884058e-06,
+      "loss": 0.6831,
+      "step": 350
+    },
+    {
+      "epoch": 0.9362808842652796,
+      "grad_norm": 0.03484239801764488,
+      "learning_rate": 3.4782608695652175e-06,
+      "loss": 0.6321,
+      "step": 360
+    },
+    {
+      "epoch": 0.9622886866059818,
+      "grad_norm": 0.022825093939900398,
+      "learning_rate": 2.028985507246377e-06,
+      "loss": 0.6889,
+      "step": 370
+    },
+    {
+      "epoch": 0.988296488946684,
+      "grad_norm": 0.019488025456666946,
+      "learning_rate": 5.797101449275362e-07,
+      "loss": 0.6797,
+      "step": 380
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 384,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 8.415557240450187e+18,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-384/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:946ab9e47c5e0b67ce7697f3723adb4ce2e58ae28b9f35acb22e930e952f463a
+size 6904

checkpoint-384/zero_to_fp32.py

@@ -0,0 +1,760 @@
+#!/usr/bin/env python
+
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: Apache-2.0
+
+# DeepSpeed Team
+
+# This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
+# copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
+# the future. Once extracted, the weights don't require DeepSpeed and can be used in any
+# application.
+#
+# example:
+#   python zero_to_fp32.py . output_dir/
+#   or
+#   python zero_to_fp32.py . output_dir/ --safe_serialization
+
+import argparse
+import torch
+import glob
+import math
+import os
+import re
+import gc
+import json
+import numpy as np
+from tqdm import tqdm
+from collections import OrderedDict
+from dataclasses import dataclass
+
+# while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
+# DeepSpeed data structures it has to be available in the current python environment.
+from deepspeed.utils import logger
+from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
+                                            FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
+                                            FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
+
+
+@dataclass
+class zero_model_state:
+    buffers: dict()
+    param_shapes: dict()
+    shared_params: list
+    ds_version: int
+    frozen_param_shapes: dict()
+    frozen_param_fragments: dict()
+
+
+debug = 0
+
+# load to cpu
+device = torch.device('cpu')
+
+
+def atoi(text):
+    return int(text) if text.isdigit() else text
+
+
+def natural_keys(text):
+    '''
+    alist.sort(key=natural_keys) sorts in human order
+    http://nedbatchelder.com/blog/200712/human_sorting.html
+    (See Toothy's implementation in the comments)
+    '''
+    return [atoi(c) for c in re.split(r'(\d+)', text)]
+
+
+def get_model_state_file(checkpoint_dir, zero_stage):
+    if not os.path.isdir(checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
+
+    # there should be only one file
+    if zero_stage <= 2:
+        file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
+    elif zero_stage == 3:
+        file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
+
+    if not os.path.exists(file):
+        raise FileNotFoundError(f"can't find model states file at '{file}'")
+
+    return file
+
+
+def get_checkpoint_files(checkpoint_dir, glob_pattern):
+    # XXX: need to test that this simple glob rule works for multi-node setup too
+    ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
+
+    if len(ckpt_files) == 0:
+        raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
+
+    return ckpt_files
+
+
+def get_optim_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
+
+
+def get_model_state_files(checkpoint_dir):
+    return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
+
+
+def parse_model_states(files):
+    zero_model_states = []
+    for file in files:
+        state_dict = torch.load(file, map_location=device, weights_only=False)
+
+        if BUFFER_NAMES not in state_dict:
+            raise ValueError(f"{file} is not a model state checkpoint")
+        buffer_names = state_dict[BUFFER_NAMES]
+        if debug:
+            print("Found buffers:", buffer_names)
+
+        # recover just the buffers while restoring them to fp32 if they were saved in fp16
+        buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
+        param_shapes = state_dict[PARAM_SHAPES]
+
+        # collect parameters that are included in param_shapes
+        param_names = []
+        for s in param_shapes:
+            for name in s.keys():
+                param_names.append(name)
+
+        # update with frozen parameters
+        frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
+        if frozen_param_shapes is not None:
+            if debug:
+                print(f"Found frozen_param_shapes: {frozen_param_shapes}")
+            param_names += list(frozen_param_shapes.keys())
+
+        # handle shared params
+        shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
+
+        ds_version = state_dict.get(DS_VERSION, None)
+
+        frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
+
+        z_model_state = zero_model_state(buffers=buffers,
+                                         param_shapes=param_shapes,
+                                         shared_params=shared_params,
+                                         ds_version=ds_version,
+                                         frozen_param_shapes=frozen_param_shapes,
+                                         frozen_param_fragments=frozen_param_fragments)
+        zero_model_states.append(z_model_state)
+
+    return zero_model_states
+
+
+def parse_optim_states(files, ds_checkpoint_dir):
+    total_files = len(files)
+    state_dicts = []
+    for f in tqdm(files, desc='Loading checkpoint shards'):
+        state_dict = torch.load(f, map_location=device, mmap=True, weights_only=False)
+        # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
+        # and also handle the case where it was already removed by another helper script
+        state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
+        state_dicts.append(state_dict)
+
+    if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
+        raise ValueError(f"{files[0]} is not a zero checkpoint")
+    zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
+    world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
+
+    # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
+    # parameters can be different from data parallelism for non-expert parameters. So we can just
+    # use the max of the partition_count to get the dp world_size.
+
+    if type(world_size) is list:
+        world_size = max(world_size)
+
+    if world_size != total_files:
+        raise ValueError(
+            f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
+            "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
+        )
+
+    # the groups are named differently in each stage
+    if zero_stage <= 2:
+        fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
+    elif zero_stage == 3:
+        fp32_groups_key = FP32_FLAT_GROUPS
+    else:
+        raise ValueError(f"unknown zero stage {zero_stage}")
+
+    fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
+    return zero_stage, world_size, fp32_flat_groups
+
+
+def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
+    """
+    Returns fp32 state_dict reconstructed from ds checkpoint
+
+    Args:
+        - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
+
+    """
+    print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
+
+    optim_files = get_optim_files(ds_checkpoint_dir)
+    zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
+    print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
+
+    model_files = get_model_state_files(ds_checkpoint_dir)
+
+    zero_model_states = parse_model_states(model_files)
+    print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
+
+    if zero_stage <= 2:
+        return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+    elif zero_stage == 3:
+        return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                                          exclude_frozen_parameters)
+
+
+def _zero2_merge_frozen_params(state_dict, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+    frozen_param_fragments = zero_model_states[0].frozen_param_fragments
+
+    if debug:
+        num_elem = sum(s.numel() for s in frozen_param_shapes.values())
+        print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        state_dict[name] = frozen_param_fragments[name]
+
+        if debug:
+            print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _has_callable(obj, fn):
+    attr = getattr(obj, fn, None)
+    return callable(attr)
+
+
+def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+
+    # Reconstruction protocol:
+    #
+    # XXX: document this
+
+    if debug:
+        for i in range(world_size):
+            for j in range(len(fp32_flat_groups[0])):
+                print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
+
+    # XXX: memory usage doubles here (zero2)
+    num_param_groups = len(fp32_flat_groups[0])
+    merged_single_partition_of_fp32_groups = []
+    for i in range(num_param_groups):
+        merged_partitions = [sd[i] for sd in fp32_flat_groups]
+        full_single_fp32_vector = torch.cat(merged_partitions, 0)
+        merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
+    avail_numel = sum(
+        [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
+
+    if debug:
+        wanted_params = sum([len(shapes) for shapes in param_shapes])
+        wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
+        # not asserting if there is a mismatch due to possible padding
+        print(f"Have {avail_numel} numels to process.")
+        print(f"Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    total_numel = 0
+    total_params = 0
+    for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
+        offset = 0
+        avail_numel = full_single_fp32_vector.numel()
+        for name, shape in shapes.items():
+
+            unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
+            total_numel += unpartitioned_numel
+            total_params += 1
+
+            if debug:
+                print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
+            state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
+            offset += unpartitioned_numel
+
+        # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
+        # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
+        # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
+        # live optimizer object, so we are checking that the numbers are within the right range
+        align_to = 2 * world_size
+
+        def zero2_align(x):
+            return align_to * math.ceil(x / align_to)
+
+        if debug:
+            print(f"original offset={offset}, avail_numel={avail_numel}")
+
+        offset = zero2_align(offset)
+        avail_numel = zero2_align(avail_numel)
+
+        if debug:
+            print(f"aligned  offset={offset}, avail_numel={avail_numel}")
+
+        # Sanity check
+        if offset != avail_numel:
+            raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero2_merge_frozen_params(state_dict, zero_model_states)
+
+    _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def zero3_partitioned_param_info(unpartitioned_numel, world_size):
+    remainder = unpartitioned_numel % world_size
+    padding_numel = (world_size - remainder) if remainder else 0
+    partitioned_numel = math.ceil(unpartitioned_numel / world_size)
+    return partitioned_numel, padding_numel
+
+
+def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
+    if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
+        return
+
+    if debug:
+        for i in range(world_size):
+            num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
+            print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
+
+        frozen_param_shapes = zero_model_states[0].frozen_param_shapes
+        wanted_params = len(frozen_param_shapes)
+        wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
+        avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
+        print(f'Frozen params: Have {avail_numel} numels to process.')
+        print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
+
+    total_params = 0
+    total_numel = 0
+    for name, shape in zero_model_states[0].frozen_param_shapes.items():
+        total_params += 1
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+
+        param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
+        state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
+
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+    print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
+
+
+class GatheredTensor:
+    """
+    A pseudo tensor that collects partitioned weights.
+    It is more memory efficient when there are multiple groups.
+    """
+
+    def __init__(self, flat_groups, flat_groups_offset, offset, partitioned_numel, shape):
+        self.flat_groups = flat_groups
+        self.flat_groups_offset = flat_groups_offset
+        self.offset = offset
+        self.partitioned_numel = partitioned_numel
+        self.shape = shape
+        self.dtype = self.flat_groups[0][0].dtype
+
+    def contiguous(self):
+        """
+        Merge partitioned weights from flat_groups into a single tensor.
+        """
+        end_idx = self.offset + self.partitioned_numel
+        world_size = len(self.flat_groups)
+        pad_flat_param_chunks = []
+
+        for rank_i in range(world_size):
+            # for each rank, we need to collect weights from related group/groups
+            flat_groups_at_rank_i = self.flat_groups[rank_i]
+            start_group_id = None
+            end_group_id = None
+            for group_id in range(len(self.flat_groups_offset)):
+                if self.flat_groups_offset[group_id] <= self.offset < self.flat_groups_offset[group_id + 1]:
+                    start_group_id = group_id
+                if self.flat_groups_offset[group_id] < end_idx <= self.flat_groups_offset[group_id + 1]:
+                    end_group_id = group_id
+                    break
+            # collect weights from related group/groups
+            for group_id in range(start_group_id, end_group_id + 1):
+                flat_tensor = flat_groups_at_rank_i[group_id]
+                start_offset = self.offset - self.flat_groups_offset[group_id]
+                end_offset = min(end_idx, self.flat_groups_offset[group_id + 1]) - self.flat_groups_offset[group_id]
+                pad_flat_param_chunks.append(flat_tensor[start_offset:end_offset])
+
+        # collect weights from all ranks
+        pad_flat_param = torch.cat(pad_flat_param_chunks, dim=0)
+        param = pad_flat_param[:self.shape.numel()].view(self.shape).contiguous()
+        return param
+
+
+def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
+    param_shapes = zero_model_states[0].param_shapes
+    avail_numel = sum([flat_group.numel() for flat_group in fp32_flat_groups[0]]) * world_size
+
+    # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
+    # param, re-consolidating each param, while dealing with padding if any
+
+    # merge list of dicts, preserving order
+    param_shapes = {k: v for d in param_shapes for k, v in d.items()}
+
+    if debug:
+        for i in range(world_size):
+            print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
+
+        wanted_params = len(param_shapes)
+        wanted_numel = sum(shape.numel() for shape in param_shapes.values())
+        # not asserting if there is a mismatch due to possible padding
+        avail_numel = fp32_flat_groups[0].numel() * world_size
+        print(f"Trainable params: Have {avail_numel} numels to process.")
+        print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
+
+    # params
+    # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
+    # out-of-core computing solution
+    offset = 0
+    total_numel = 0
+    total_params = 0
+    flat_groups_offset = [0] + list(np.cumsum([flat_tensor.numel() for flat_tensor in fp32_flat_groups[0]]))
+    for name, shape in tqdm(param_shapes.items(), desc='Gathering sharded weights'):
+        unpartitioned_numel = shape.numel()
+        total_numel += unpartitioned_numel
+        total_params += 1
+        partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
+
+        if debug:
+            print(
+                f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
+            )
+
+        # memory efficient tensor
+        tensor = GatheredTensor(fp32_flat_groups, flat_groups_offset, offset, partitioned_numel, shape)
+        state_dict[name] = tensor
+        offset += partitioned_numel
+
+    offset *= world_size
+
+    # Sanity check
+    if offset != avail_numel:
+        raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
+
+    print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
+
+
+def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
+                                               exclude_frozen_parameters):
+    state_dict = OrderedDict()
+
+    # buffers
+    buffers = zero_model_states[0].buffers
+    state_dict.update(buffers)
+    if debug:
+        print(f"added {len(buffers)} buffers")
+
+    if not exclude_frozen_parameters:
+        _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
+
+    _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
+
+    # recover shared parameters
+    for pair in zero_model_states[0].shared_params:
+        if pair[1] in state_dict:
+            state_dict[pair[0]] = state_dict[pair[1]]
+
+    return state_dict
+
+
+def to_torch_tensor(state_dict, return_empty_tensor=False):
+    """
+    Convert state_dict of GatheredTensor to torch tensor
+    """
+    torch_state_dict = {}
+    converted_tensors = {}
+    for name, tensor in state_dict.items():
+        tensor_id = id(tensor)
+        if tensor_id in converted_tensors:  # shared tensors
+            shared_tensor = torch_state_dict[converted_tensors[tensor_id]]
+            torch_state_dict[name] = shared_tensor
+        else:
+            converted_tensors[tensor_id] = name
+            if return_empty_tensor:
+                torch_state_dict[name] = torch.empty(tensor.shape, dtype=tensor.dtype)
+            else:
+                torch_state_dict[name] = tensor.contiguous()
+    return torch_state_dict
+
+
+def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir,
+                                             tag=None,
+                                             exclude_frozen_parameters=False,
+                                             lazy_mode=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
+    ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
+    via a model hub.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+        - ``lazy_mode``: get state_dict in lazy mode. It returns a dict of pesduo tensor instead of torch tensor, which is more memory efficient.
+          Convert the pesduo tensor to torch tensor by ``.contiguous()``
+
+    Returns:
+        - pytorch ``state_dict``
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
+        # do the training and checkpoint saving
+        state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
+        model = model.cpu() # move to cpu
+        model.load_state_dict(state_dict)
+        # submit to model hub or save the model to share with others
+
+    In this example the ``model`` will no longer be usable in the deepspeed context of the same
+    application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
+
+    Note: the above usage may not work if your application doesn't have sufficient free CPU memory.
+    You may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
+    the checkpoint. Or you can load state_dict in lazy mode ::
+
+        from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
+        state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, lazy_mode=True) # not on cpu
+        for name, lazy_tensor in state_dict.item():
+            tensor = lazy_tensor.contiguous()  # to cpu
+            print(name, tensor)
+            # del tensor to release memory if it no longer in use
+    """
+    if tag is None:
+        latest_path = os.path.join(checkpoint_dir, 'latest')
+        if os.path.isfile(latest_path):
+            with open(latest_path, 'r') as fd:
+                tag = fd.read().strip()
+        else:
+            raise ValueError(f"Unable to find 'latest' file at {latest_path}")
+
+    ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
+
+    if not os.path.isdir(ds_checkpoint_dir):
+        raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
+
+    state_dict = _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
+    if lazy_mode:
+        return state_dict
+    else:
+        return to_torch_tensor(state_dict)
+
+
+def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
+                                               output_dir,
+                                               max_shard_size="5GB",
+                                               safe_serialization=False,
+                                               tag=None,
+                                               exclude_frozen_parameters=False):
+    """
+    Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
+    loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
+
+    Args:
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``output_dir``: directory to the pytorch fp32 state_dict output files
+        - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
+        - ``safe_serialization``:  whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+        - ``exclude_frozen_parameters``: exclude frozen parameters
+    """
+
+    # Dependency pre-check
+    if safe_serialization:
+        try:
+            from safetensors.torch import save_file
+        except ImportError:
+            print('If you want to use `safe_serialization`, please `pip install safetensors`')
+            raise
+    if max_shard_size is not None:
+        try:
+            from huggingface_hub import split_torch_state_dict_into_shards
+        except ImportError:
+            print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
+            raise
+
+    # Convert zero checkpoint to state_dict
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir,
+                                                          tag,
+                                                          exclude_frozen_parameters,
+                                                          lazy_mode=True)
+
+    # Shard the model if it is too big.
+    weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
+    if max_shard_size is not None:
+        filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
+        # an memory-efficient approach for sharding
+        empty_state_dict = to_torch_tensor(state_dict, return_empty_tensor=True)
+        state_dict_split = split_torch_state_dict_into_shards(empty_state_dict,
+                                                              filename_pattern=filename_pattern,
+                                                              max_shard_size=max_shard_size)
+    else:
+        from collections import namedtuple
+        StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
+        state_dict_split = StateDictSplit(is_sharded=False,
+                                          filename_to_tensors={weights_name: list(state_dict.keys())})
+
+    # Save the model by shard
+    os.makedirs(output_dir, exist_ok=True)
+    filename_to_tensors = state_dict_split.filename_to_tensors.items()
+    for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
+        shard_state_dict = {tensor_name: state_dict[tensor_name] for tensor_name in tensors}
+        shard_state_dict = to_torch_tensor(shard_state_dict)
+        output_path = os.path.join(output_dir, shard_file)
+        if safe_serialization:
+            save_file(shard_state_dict, output_path, metadata={"format": "pt"})
+        else:
+            torch.save(shard_state_dict, output_path)
+        # release the memory of current shard
+        for tensor_name in list(shard_state_dict.keys()):
+            del state_dict[tensor_name]
+            del shard_state_dict[tensor_name]
+        del shard_state_dict
+        gc.collect()
+
+    # Save index if sharded
+    if state_dict_split.is_sharded:
+        index = {
+            "metadata": state_dict_split.metadata,
+            "weight_map": state_dict_split.tensor_to_filename,
+        }
+        save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
+        save_index_file = os.path.join(output_dir, save_index_file)
+        with open(save_index_file, "w", encoding="utf-8") as f:
+            content = json.dumps(index, indent=2, sort_keys=True) + "\n"
+            f.write(content)
+
+
+def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
+    """
+    1. Put the provided model to cpu
+    2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
+    3. Load it into the provided model
+
+    Args:
+        - ``model``: the model object to update
+        - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
+        - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
+
+    Returns:
+        - ``model`: modified model
+
+    Make sure you have plenty of CPU memory available before you call this function. If you don't
+    have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
+    conveniently placed for you in the checkpoint folder.
+
+    A typical usage might be ::
+
+        from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
+        model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
+        # submit to model hub or save the model to share with others
+
+    Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
+    of the same application. i.e. you will need to re-initialize the deepspeed engine, since
+    ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
+
+    """
+    logger.info(f"Extracting fp32 weights")
+    state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
+
+    logger.info(f"Overwriting model with fp32 weights")
+    model = model.cpu()
+    model.load_state_dict(state_dict, strict=False)
+
+    return model
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("checkpoint_dir",
+                        type=str,
+                        help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
+    parser.add_argument("output_dir",
+                        type=str,
+                        help="directory to the pytorch fp32 state_dict output files"
+                        "(e.g. path/checkpoint-12-output/)")
+    parser.add_argument(
+        "--max_shard_size",
+        type=str,
+        default="5GB",
+        help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
+        "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
+        "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
+        "without CPU OOM issues.")
+    parser.add_argument(
+        "--safe_serialization",
+        default=False,
+        action='store_true',
+        help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
+    parser.add_argument("-t",
+                        "--tag",
+                        type=str,
+                        default=None,
+                        help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
+    parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
+    parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
+    args = parser.parse_args()
+
+    debug = args.debug
+
+    convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
+                                               args.output_dir,
+                                               max_shard_size=args.max_shard_size,
+                                               safe_serialization=args.safe_serialization,
+                                               tag=args.tag,
+                                               exclude_frozen_parameters=args.exclude_frozen_parameters)

logs/events.out.tfevents.1738893626.apolo.2381507.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88633ed8a22a5f6c0c0a70e3b0d09eddfb5a64f018c65a729e77c75d6e84fc5d
+size 6829

logs/events.out.tfevents.1738894521.apolo.2389024.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:039d9de97b83309f2ab5b48653fd2d7e0e7a0dce791111b881f8f206fe9b81d4
+size 6415

logs/events.out.tfevents.1738894780.apolo.2393100.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d0d28805640478d1ad9727f750eb7a9dbfe5270e50dccc85a97f3d23c3492ba
+size 6621

logs/events.out.tfevents.1738895567.apolo.2399779.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7b03d4a1b9a5da4e9715345ef670697f4cbae426e0cc33524eafb725250c4e1
+size 6835

logs/events.out.tfevents.1738895868.apolo.2404670.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d47513bb9522eaf28ec3c8e32c8d9660eb16c2a78a3857e7f1df65b8d464b353
+size 6834

logs/events.out.tfevents.1738896201.apolo.2410067.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61476e8e694469e468dfad1328a2f69d2aa90a57199e595f2fe1e1bfa7a27007
+size 14958

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}