ritulk/MPNET_finetuned_on_stsb_multi_mt_dataset

SentenceTransformer based on ritulk/MPNET-fine-tuned-political-clustering

This is a sentence-transformers model finetuned from ritulk/MPNET-fine-tuned-political-clustering. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: ritulk/MPNET-fine-tuned-political-clustering
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ritulk/MPNET_finetuned_on_stsb_multi_mt_dataset")
# Run inference
sentences = [
    'Zwei Weißkopfseeadler auf einem Ast.',
    'Zwei Adler sitzen auf einem Ast.',
    'Ein Mann, der in einem Raum auf dem Boden sitzt, klimpert auf einer Gitarre.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.6568
spearman_cosine	0.6576

Semantic Similarity

• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.676
spearman_cosine	0.675

Semantic Similarity

• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.7641
spearman_cosine	0.7619

Training Details

Training Dataset

PhilipMay/stsb_multi_mt

• Size: 5,749 training samples
• Columns: text, text_pair, and score
• Approximate statistics based on the first 1000 samples:

  	text	text_pair	score
  type	string	string	float
  details	min: 7 tokens mean: 14.58 tokens max: 49 tokens	min: 7 tokens mean: 14.6 tokens max: 47 tokens	min: 0.0 mean: 0.45 max: 1.0

• Samples:

  text	text_pair	score
  Ein Flugzeug hebt gerade ab.	Ein Flugzeug hebt gerade ab.	1.0
  Ein Mann spielt eine große Flöte.	Ein Mann spielt eine Flöte.	0.7599999904632568
  Ein Mann streicht geriebenen Käse auf eine Pizza.	Ein Mann streicht geriebenen Käse auf eine ungekochte Pizza.	0.7599999904632568

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 1,500 evaluation samples
• Columns: text, text_pair, and score
• Approximate statistics based on the first 1000 samples:

  	text	text_pair	score
  type	string	string	float
  details	min: 6 tokens mean: 25.19 tokens max: 65 tokens	min: 7 tokens mean: 25.21 tokens max: 70 tokens	min: 0.0 mean: 0.42 max: 1.0

• Samples:

  text	text_pair	score
  Ein Mann mit einem Schutzhelm tanzt.	Ein Mann mit einem Schutzhelm tanzt.	1.0
  Ein kleines Kind reitet auf einem Pferd.	Ein Kind reitet auf einem Pferd.	0.95
  Ein Mann verfüttert eine Maus an eine Schlange.	Der Mann füttert die Schlange mit einer Maus.	1.0

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 5
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	spearman_cosine
0.2778	100	0.6009	0.9181	-
0.5556	200	0.4724	0.8744	-
0.8333	300	0.449	0.8405	-
-1	-1	-	-	0.6576
0.2778	100	0.0781	0.9378	-
0.5556	200	0.0772	0.9290	-
0.8333	300	0.2281	0.8876	-
1.1111	400	0.3267	0.9336	-
1.3889	500	0.2936	0.8612	-
1.6667	600	0.2283	0.8569	-
1.9444	700	0.2448	0.8589	-
2.2222	800	0.1877	0.8418	-
2.5	900	0.1693	0.8351	-
2.7778	1000	0.1635	0.8588	-
3.0556	1100	0.1642	0.8260	-
3.3333	1200	0.1027	0.8380	-
3.6111	1300	0.0983	0.8407	-
3.8889	1400	0.0978	0.8317	-
4.1667	1500	0.1187	0.8376	-
4.4444	1600	0.0977	0.8465	-
4.7222	1700	0.0686	0.8492	-
5.0	1800	0.0587	0.8485	-
-1	-1	-	-	0.6750
0.2778	100	0.0656	0.0464	-
0.5556	200	0.0564	0.0454	-
0.8333	300	0.0498	0.0496	-
1.1111	400	0.042	0.0408	-
1.3889	500	0.0384	0.0416	-
1.6667	600	0.0319	0.0427	-
1.9444	700	0.0332	0.0427	-
2.2222	800	0.0249	0.0416	-
2.5	900	0.0232	0.0408	-
2.7778	1000	0.0219	0.0415	-
3.0556	1100	0.0215	0.0409	-
3.3333	1200	0.0158	0.0402	-
3.6111	1300	0.0171	0.0387	-
3.8889	1400	0.0152	0.0393	-
4.1667	1500	0.0126	0.0389	-
4.4444	1600	0.0124	0.0389	-
4.7222	1700	0.0118	0.0393	-
5.0	1800	0.0127	0.0391	-
-1	-1	-	-	0.7619

Framework Versions

• Python: 3.11.12
• Sentence Transformers: 4.0.2
• Transformers: 4.50.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}