LorMolf/mnrl-toolbench-roberta-base

SentenceTransformer based on FacebookAI/roberta-base

This is a sentence-transformers model finetuned from FacebookAI/roberta-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: FacebookAI/roberta-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("LorMolf/mnrl-toolbench-roberta-base")
# Run inference
sentences = [
    "I'm planning a family vacation and I need to find a child-friendly hotel with recreational activities for kids. Can you suggest some options in our destination city? Also, provide me with information about nearby attractions that would be suitable for children.",
    'def webcams_travel_webcams_list_limit_limit_offset:\n\t"""\n\tDescription:\n\tThis is a modifier. Returns the list of webcams sliced by {limit}. The optional offset is given by {offset}. Required: {limit}. The maximum value for {limit} is 50. {offset} defaults to 0. If limit is not given, then a default of limit=10 is applied.\n\n\tArguments:\n\t---------\n\t- limit : NUMBER (required)\n\t  Description: Maximum number of webcams in the result list.\n\t"""',
    'def check_username_askfm:\n\t"""\n\tDescription:\n\tCheck username on Ask.fm\n\n\tArguments:\n\t---------\n\t- username : string (required)\n\t  Default: username\n\t"""',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Device Aware Information Retrieval

• Dataset: dev
• Evaluated with src.port.retrieval_evaluator.DeviceAwareInformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.03
cosine_accuracy@3	0.15
cosine_accuracy@5	0.22
cosine_accuracy@10	0.41
cosine_precision@1	0.03
cosine_precision@3	0.0567
cosine_precision@5	0.05
cosine_precision@10	0.052
cosine_recall@1	0.0117
cosine_recall@3	0.06
cosine_recall@5	0.085
cosine_recall@10	0.171
cosine_ndcg@1	0.03
cosine_ndcg@3	0.0535
cosine_ndcg@5	0.0675
cosine_ndcg@10	0.1056
cosine_mrr@10	0.1191
cosine_map@100	0.1261

Training Details

Training Dataset

Unnamed Dataset

• Size: 30,000 training samples
• Columns: sentence_0, sentence_1, and sentence_2
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	min: 25 tokens mean: 58.56 tokens max: 110 tokens	min: 26 tokens mean: 88.11 tokens max: 512 tokens	min: 26 tokens mean: 98.59 tokens max: 512 tokens

• Samples:

  sentence_0	sentence_1	sentence_2
  My family and I are going on a road trip across Spain. We want to navigate using a map with Spanish labels. Can you provide us with a raster map tile of Spain with Spanish labels? Also, we need to find the correct address for a restaurant in Barcelona. The restaurant's address is Carrer de Mallorca 123, Barcelona.	def maptiles_getmaptilewithspanishlabels: """ Description: Raster Map Tiles with Spanish Labels. Please see tutorial on how to use the Spanish world map. Arguments: --------- - z : NUMBER (required) Description: zoom (from 0 up to zoom 19) Default: 3 - x : NUMBER (required) Description: X-number of tile (see documentation) Default: 4 - y : NUMBER (required) Description: Y-number of tile (see documentation) Default: 2 """	def morning_star_stock_v2_get_short_interest: """ Description: Mapped to Short Interest section in Quote tab Arguments: --------- - performanceId : STRING (required) Description: Value of performanceId field from .../auto-complete or /get-summary or .../get-movers endpoints Default: 0P0000OQN8 """
  I want to explore the music of [artist]. Can you search for their tracks on Soundcloud and show me the search results? Also, find the album with the id '67890' on Deezer and provide its details. Additionally, give me the information about the song with the URL 'https://soundcloud.com/song/12345'.	def soundcloud__song_info: """ Description: Get basic information of a song. Arguments: --------- - track_url : STRING (required) Default: https://soundcloud.com/user-977421934/the-phoenix """	def sms77io_get_webhooks: """ Description: Retrieves all existing webhooks. Arguments: --------- - p : STRING (required) Description: API key from Sms77.io. """
  I'm planning a road trip with my friends and we need to rent a van. Can you suggest some reliable car rental companies in the area? Additionally, provide us with a list of scenic routes and landmarks along the way.	def working_days__1_3_add_working_days: """ Description: Add (or remove) any number of working days to a date. Arguments: --------- - country_code : STRING (required) Description: The ISO country code (2 letters). See available countries & configurations Default: US - start_date : STRING (required) Description: The start date (YYYY-MM-DD) Default: 2013-12-31 - increment : NUMBER (required) Description: The number of working days you want to add to your start date (positive or negative integer but not zero) Default: 10 """	def coupons_all: """ Description: get all coupons Arguments: --------- """

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• num_train_epochs: 1
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss	dev_cosine_ndcg@10
-1	-1	-	0.1364
0.0333	500	0.5494	-
0.0667	1000	0.5672	-
0.1	1500	0.9232	-
0.1333	2000	1.3901	-
0.1667	2500	1.3868	-
0.2	3000	1.3869	0.1663
0.2333	3500	1.3866	-
0.2667	4000	1.3864	-
0.3	4500	1.3864	-
0.3333	5000	1.3865	-
0.3667	5500	1.3865	-
0.4	6000	1.3865	0.1316
0.4333	6500	1.3866	-
0.4667	7000	1.3865	-
0.5	7500	1.3863	-
0.5333	8000	1.3864	-
0.5667	8500	1.4115	-
0.6	9000	1.3871	0.1867
0.6333	9500	1.3864	-
0.6667	10000	1.3868	-
0.7	10500	1.3866	-
0.7333	11000	1.3863	-
0.7667	11500	1.3866	-
0.8	12000	1.3863	0.1902
0.8333	12500	1.3864	-
0.8667	13000	1.3864	-
0.9	13500	1.3864	-
0.9333	14000	1.3866	-
0.9667	14500	1.3864	-
1.0	15000	1.3864	0.1056

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 4.0.2
• Transformers: 4.51.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}