Exl Quants Please

#15

by rjmehta - opened Jul 1, 2024

Jul 1, 2024

•

@turboderp @LoneStriker Exl quants please if possible but not sure if exllama supports gemma.

Jul 3, 2024

It doesn't, yet. 9B may be supported soon, though really it needs support in flash-attn to work correctly. 27B will not work at all without it.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment