Shuu12121/CodeSearch-ModernBERT-Crow-Preview

CodeSearch-ModernBERT-Crow🐦‍⬛

This is a sentence-transformers model finetuned from Shuu12121/CodeModernBERT-Crow. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Shuu12121/CodeModernBERT-Crow
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Complete the connection preface. At this point, we should have received the client connection preface string.\nNow we need to make sure that the client sent a settings frame along with the preface, update our settings,\nand send an empty settings frame in response to the client preface.\n\n@throws Http2Exception',
    'protected void completeConnectionPreface() throws Http2Exception {\n        if (TraceComponent.isAnyTracingEnabled() && tc.isDebugEnabled()) {\n            Tr.debug(tc, "completeConnectionPreface entry: about to send preface SETTINGS frame");\n        }\n        FrameSettings settings;\n        // send out a settings frame with any HTTP2 settings that the user may have changed\n        if (Constants.SPEC_INITIAL_WINDOW_SIZE != this.streamReadWindowSize) {\n            settings = new FrameSettings(0, -1, -1, this.muxLink.config.getH2MaxConcurrentStreams(), (int) this.streamReadWindowSize, this.muxLink.config.getH2MaxFrameSize(), -1, false);\n        } else {\n            settings = new FrameSettings(0, -1, -1, this.muxLink.config.getH2MaxConcurrentStreams(), -1, this.muxLink.config.getH2MaxFrameSize(), -1, false);\n        }\n\n        this.frameType = FrameTypes.SETTINGS;\n        this.processNextFrame(settings, Direction.WRITING_OUT);\n\n        if (Constants.SPEC_INITIAL_WINDOW_SIZE != muxLink.maxReadWindowSize) {\n            // the user has changed the max connection read window, so we\'ll update that now\n            FrameWindowUpdate wup = new FrameWindowUpdate(0, (int) muxLink.maxReadWindowSize, false);\n            this.processNextFrame(wup, Direction.WRITING_OUT);\n        }\n    }',
    'def assert_text_present_in_table(table_id, text, options = {})\r\n      options[:just_plain_text] ||= false\r\n      perform_assertion { assert(the_table_source(table_id, options).include?(text), "the text #{text} not found in table #{table_id}") }\r\n    end',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: code-docstring-retrieval
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8349
cosine_accuracy@3	0.91
cosine_accuracy@5	0.9266
cosine_accuracy@10	0.9455
cosine_precision@1	0.8349
cosine_precision@3	0.3033
cosine_precision@5	0.1853
cosine_precision@10	0.0945
cosine_recall@1	0.8349
cosine_recall@3	0.91
cosine_recall@5	0.9266
cosine_recall@10	0.9455
cosine_ndcg@10	0.8929
cosine_mrr@10	0.8758
cosine_map@100	0.8772

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,394,248 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 3 tokens mean: 53.75 tokens max: 914 tokens	min: 18 tokens mean: 187.98 tokens max: 3510 tokens	min: 1.0 mean: 1.0 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  // NewMockUnit creates a new mock instance	func NewMockUnit(ctrl *gomock.Controller) *MockUnit { mock := &MockUnit{ctrl: ctrl} mock.recorder = &MockUnitMockRecorder{mock} return mock }	1.0
  // SetNextPageToken sets the NextPageToken field's value.	func (s *ListBudgetsForResourceOutput) SetNextPageToken(v string) *ListBudgetsForResourceOutput { s.NextPageToken = &v return s }	1.0
  // addHandler adds a handler for a puType/event.	func (r *registerer) addHandler(puType common.PUType, event common.Event, handler common.EventHandler) { r.handlers[puType][event] = handler }	1.0

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}