{"id":692,"date":"2024-03-06T16:25:43","date_gmt":"2024-03-06T07:25:43","guid":{"rendered":"https:\/\/elosove.com\/?p=692"},"modified":"2024-03-06T16:27:39","modified_gmt":"2024-03-06T07:27:39","slug":"rinna-nekomata-14b%e3%83%a2%e3%83%87%e3%83%ab%e3%82%92windows%e3%83%ad%e3%83%bc%e3%82%ab%e3%83%ab%e7%92%b0%e5%a2%83%e3%81%a7%e5%8b%95%e3%81%8b%e3%81%99","status":"publish","type":"post","link":"https:\/\/elosove.com\/?p=692","title":{"rendered":"rinna nekomata-14b\u30e2\u30c7\u30eb\u3092Windows\u30ed\u30fc\u30ab\u30eb\u74b0\u5883\u3067\u52d5\u304b\u3059"},"content":{"rendered":"\n

rinna nekomata\u30e2\u30c7\u30eb\u3092\u30ed\u30fc\u30ab\u30eb\u74b0\u5883\u3067\u52d5\u304b\u3057\u3066\u307f\u3088\u3046\u3068\u3057\u305f\u306e\u3067\u3059\u304c\u3001\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u30ac\u30a4\u30c9\u304c\u3042\u307e\u308a\u898b\u5f53\u305f\u3089\u306a\u304b\u3063\u305f\u306e\u3067\u66f8\u3044\u3066\u307f\u307e\u3057\u305f\u3002<\/p>\n\n\n\n

\u74b0\u5883<\/strong><\/h2>\n\n\n\n
\n

windows11<\/p>\n\n\n\n

RAM 64GB<\/p>\n\n\n\n

VRAM 16GB (RXT4090 laptop)<\/p>\n<\/blockquote>\n\n\n\n

\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb<\/h2>\n\n\n\n

\u516c\u5f0f\u30da\u30fc\u30b8<\/a>\u306erinna\/nekomata-14b-instruction\u30e2\u30c7\u30eb\u306e\u30da\u30fc\u30b8\u3092\u53c2\u8003\u306b\u52d5\u304b\u3057\u3066\u307f\u307e\u3059\u3002<\/p>\n\n\n\n

<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
conda<\/span> <\/span>create<\/span> <\/span>-n<\/span> <\/span>rinna<\/span> <\/span>python=<\/span>3.9<\/span><\/span>\nactivate<\/span> <\/span>rinna<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
How to use the model\u306e\u30b5\u30f3\u30d7\u30eb\u30b3\u30fc\u30c9\u304b\u3089torch,transformers\u3092\u7528\u610f\u3057\u307e\u3059\u3002(CUDA\u306f11.8\u3092\u9078\u629e)<\/p>\n\n\n\n
\u3053\u306e\u307e\u307e\u30b5\u30f3\u30d7\u30eb\u30b3\u30fc\u30c9\u3092\u52d5\u304b\u3059\u3068\u30a8\u30e9\u30fc\u304c\u51fa\u308b\u306e\u3067\u3001\u30a8\u30e9\u30fc\u306e\u6307\u793a\u306b\u5f93\u3063\u3066\u4ed6\u306e\u30e9\u30a4\u30d6\u30e9\u30ea\u3082\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u307e\u3059\u3002<\/p>\n\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
pip<\/span> <\/span>install<\/span> <\/span>torch<\/span> <\/span>torchvision<\/span> <\/span>torchaudio<\/span> <\/span>--index-url<\/span> <\/span>https:\/\/download.pytorch.org\/whl\/cu118<\/span><\/span>\npip<\/span> <\/span>install<\/span> <\/span>transformers<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
\u4ee5\u4e0a\u3067\u52d5\u304d\u307e\u3059\u304c\u3001\u3053\u306e\u307e\u307e\u3067\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u30b5\u30a4\u30ba\u306e\u5909\u66f4\u304c\u51fa\u6765\u306a\u3044\u306e\u3067bitsandbytes\u3092\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u307e\u3059\u3002<\/p>\n\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
pip<\/span> <\/span>install<\/span> <\/span>tiktoken\u3000transformers_stream_generator\u3000einops<\/span> <\/span>accelerate<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
windows\u306b\u3088\u304f\u3042\u308b\u3053\u3068\u3067\u3059\u304c\u3001bitsandbytes\u306fwindows\u30b5\u30dd\u30fc\u30c8\u304c\u66d6\u6627\u306a\u306e\u3067\u79c1\u306e\u5834\u5408\u306f\u30a8\u30e9\u30fc\u304c\u51fa\u307e\u3057\u305f\u3002<\/p>\n\n\n\n
\u89e3\u6c7a\u7b56\u3068\u3057\u3066\u306fbitsandbytes not working with windows 11<\/a>\u3067\u7d39\u4ecb\u3055\u308c\u3066\u3044\u308bbitsandbytes-windows-webui<\/a>\u304b\u3089\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u307e\u3059\u3002<\/p>\n\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
pip<\/span> <\/span>install<\/span> <\/span>https:\/\/github.com\/jllllll\/bitsandbytes-windows-webui\/releases\/download\/wheels\/bitsandbytes-0.41.1-py3-none-win_amd64.whl<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
\u4ee5\u4e0a\u3092\u307e\u3068\u3081\u308b\u3068\u4e0b\u8a18\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n\n
Package<\/th>Version<\/th><\/tr><\/thead>
accelerate<\/td>0.27.2<\/td><\/tr>
bitsandbytes<\/td>0.41.1<\/td><\/tr>
einops<\/td>0.7.0<\/td><\/tr>
tiktoken <\/td>0.6.0<\/td><\/tr>
transformer<\/td>4.38.2<\/td><\/tr>
transformers-stream-generator<\/td>0.0.4<\/td><\/tr>
torch
torchaudio
torchvision<\/td>		2.2.1+cu118
2.2.1+cu118
0.17.1+cu118<\/td><\/tr><\/tbody><\/table>
Python 3.9.18<\/figcaption><\/figure>\n\n\n\n
\u3000<\/p>\n\n\n\n
\u30e2\u30c7\u30eb\u306e\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9<\/h2>\n\n\n\n
How to use the model\u306e\u30b5\u30f3\u30d7\u30eb\u30b3\u30fc\u30c94\u884c\u76ee\u3067\u5b9f\u884c\u3055\u308c\u307e\u3059\u3002(AutoTokenizer.from_pretrained)<\/p>\n\n\n\n
DL\u30d5\u30a9\u30eb\u30c0\u306f”C:\\Users\\\u81ea\u5206.cache\\huggingface\\hub\\models–rinna–nekomata-14b-instruction\\snapshots\\\u30cf\u30c3\u30b7\u30e5\\pytroch_model.bin<\/em><\/strong>“\u306728GB\u306e\u30e2\u30c7\u30eb\u304c\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u3055\u308c\u307e\u3059\u3002<\/p>\n\n\n\n
\u30c0\u30a6\u30f3\u30ed\u30fc\u30c9\u95a2\u9023\u306e\u30a8\u30e9\u30fc\u304c\u51fa\u308b\u5834\u5408\u306f\u30b3\u30de\u30f3\u30c9\u30d7\u30ed\u30f3\u30d7\u30c8\u3092\u7ba1\u7406\u8005\u3068\u3057\u3066\u5b9f\u884c\u3057\u3066\u307f\u3066\u304f\u3060\u3055\u3044\u3002<\/p>\n\n\n\n
\u3000<\/p>\n\n\n\n
\u30b5\u30f3\u30d7\u30eb\u30b3\u30fc\u30c9\u5b9f\u884c\u7d50\u679c<\/h2>\n\n\n\n
\u305d\u306e\u307e\u307e\u5b9f\u884c\u3059\u308b\u3068\u4e0b\u8a18\u306e\u30b3\u30fc\u30c9\u3067\u30e2\u30c7\u30eb\u306e\u5c55\u958b\u65b9\u6cd5\u304c\u9078\u3070\u308c\u308b\u306e\u3067\u3001\u79c1\u306e\u74b0\u5883\u3067\u306fRAM13GB\u7a0b\u5ea6\u3001VRAM13GB~15GB\u6d88\u8cbb\u3055\u308c\u63a8\u8ad6\u6642\u9593\u306f70~90sec\u7a0b\u5ea6\u304b\u304b\u308a\u307e\u3057\u305f\u3002<\/p>\n\n\n\n
\u305d\u3053\u3067\u3001bitsandbytes\u3092\u9811\u5f35\u3063\u3066\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u3057\u305f\u6069\u6075\u3092\u53d7\u3051\u308b\u305f\u3081load_in_4bit\u3001load_in_8bit\u30aa\u30d7\u30b7\u30e7\u30f3\u3092\u3064\u3051\u3066\u901f\u5ea6\u8a08\u6e2c\u3057\u3066\u307f\u307e\u3057\u305f\u3002<\/p>\n\n\n\n
<\/td>VRAM [GB]<\/td>RAM [GB]<\/td>\u901f\u5ea6 [sec]<\/td><\/tr>
load_in_4bit<\/td>13.6<\/td>20~23(\u5c55\u958b\u6642\u306e\u307f)<\/td>12.9<\/td><\/tr>
load_in_8bit<\/td>11<\/td>20~23 (\u5c55\u958b\u6642\u306e\u307f)<\/td>11.5<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\u3061\u306a\u307f\u306b\u30aa\u30d7\u30b7\u30e7\u30f3\u306e\u3064\u3051\u65b9\u306e\u30b5\u30f3\u30d7\u30eb\u306f\u4e0b\u8a18\u306e\u901a\u308a\u3067\u3059\u3002<\/p>\n\n\n\n
<\/circle><\/circle><\/circle><\/g><\/svg><\/span><\/path><\/path><\/svg><\/span>
model<\/span> <\/span>=<\/span> <\/span>AutoModelForCausalLM.from_pretrained<\/span>(<\/span><\/span>\n                        <\/span>"rinna\/nekomata-14b-instruction"<\/span>,<\/span><\/span>\n                        <\/span>device_map<\/span>=<\/span>"cuda",<\/span><\/span>\n                        <\/span>trust_remote_code<\/span>=<\/span>True,<\/span><\/span>\n                        <\/span>load_in_4bit<\/span>=<\/span>True,<\/span><\/span>\n                        <\/span>bf16<\/span>=<\/span>True<\/span> )<\/span><\/span><\/code><\/pre><\/div>\n","protected":false},"excerpt":{"rendered":"
rinna nekomata\u30e2\u30c7\u30eb\u3092\u30ed\u30fc\u30ab\u30eb\u74b0\u5883\u3067\u52d5\u304b\u3057\u3066\u307f\u3088\u3046\u3068\u3057\u305f\u306e\u3067\u3059\u304c\u3001\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u30ac\u30a4\u30c9\u304c\u3042\u307e\u308a\u898b\u5f53\u305f\u3089\u306a\u304b\u3063\u305f\u306e\u3067\u66f8\u3044\u3066\u307f\u307e\u3057\u305f\u3002 \u74b0\u5883 windows11 RAM 64GB VRAM 16GB (RXT40 […]<\/p>\n","protected":false},"author":1,"featured_media":657,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[26,23],"class_list":["post-692","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech","tag-llm","tag-rinna"],"_links":{"self":[{"href":"https:\/\/elosove.com\/index.php?rest_route=\/wp\/v2\/posts\/692","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/elosove.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/elosove.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/elosove.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/elosove.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=692"}],"version-history":[{"count":7,"href":"https:\/\/elosove.com\/index.php?rest_route=\/wp\/v2\/posts\/692\/revisions"}],"predecessor-version":[{"id":699,"href":"https:\/\/elosove.com\/index.php?rest_route=\/wp\/v2\/posts\/692\/revisions\/699"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/elosove.com\/index.php?rest_route=\/wp\/v2\/media\/657"}],"wp:attachment":[{"href":"https:\/\/elosove.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=692"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/elosove.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=692"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/elosove.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=692"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
bitsandbytes-windows<\/a>\u3068\u3044\u3046\u30d7\u30ed\u30b8\u30a7\u30af\u30c8\u304c\u3042\u308a\u307e\u3059\u304c\u3001\u3053\u308c\u306fbitsandbytes\u306ev0.37.5\u3067\u66f4\u65b0\u304c\u6b62\u307e\u3063\u3066\u3044\u308b\u306e\u3067\u30a4\u30f3\u30b9\u30c8\u30fc\u30eb\u306b\u6210\u529f\u3057\u3066\u3082v0.39\u4ee5\u4e0a\u3092\u4f7f\u3046\u3088\u3046\u306b\u30a8\u30e9\u30fc\u304c\u51fa\u307e\u3059\u3002<\/p>\n\n\n\n