sanitation@lemmy.today to Technology@lemmy.worldEnglish · 3 days agoIf open weight models are the future, U.S. AI companies are going to have a hard timewww.fastcompany.comexternal-linkmessage-square88fedilinkarrow-up1283arrow-down13
arrow-up1280arrow-down1external-linkIf open weight models are the future, U.S. AI companies are going to have a hard timewww.fastcompany.comsanitation@lemmy.today to Technology@lemmy.worldEnglish · 3 days agomessage-square88fedilink
minus-squareAsafum@lemmy.worldlinkfedilinkEnglisharrow-up1·2 days agoThanks for the tip! I do have a 5080 so I’ll have to look up exllamav3
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up2·edit-22 days agoYou want this one: https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more: https://huggingface.co/turboderp/Qwen3.6-27B-exl3 For faster inference at the cost of a little more VRAM usage: https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3 And you run those in: https://github.com/theroyallab/tabbyAPI And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead.
minus-squareAsafum@lemmy.worldlinkfedilinkEnglisharrow-up2·2 days agoThank you for all the links! Much appreciated :)
Thanks for the tip! I do have a 5080 so I’ll have to look up exllamav3
You want this one:
https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main
Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more:
https://huggingface.co/turboderp/Qwen3.6-27B-exl3
For faster inference at the cost of a little more VRAM usage:
https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3
And you run those in:
https://github.com/theroyallab/tabbyAPI
And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead.
Thank you for all the links! Much appreciated :)