Watch the video:
0:00 - Explanation
0:45 - Why this is unique
1:20 - Starting Vicuna & Oobabooga install
2:00 - Installing CPU-only Vicuna & Oobabooga (NEW)
3:00 - Vicuna CPU Desktop Shortcut (NEW)
3:45 - Using Vicuna & Oobabooga in CPU mode (NEW)
4:40 - GPU vs CPU-only install
5:38 - Installing Vicuna & Oobabooga for GPU & CPU
6:00 - Downloading GPU and/or CPU Vicuna model
7:07 - GPU vs CPU Vicuna speedWhat is Vicuna
Vicuna is a model from the Team with members from UC Berkley, CMU, Stanford, and UC San Diego. Vicuna-13B is an open-source chatbot trained by fine-tuning LLaMA. Their website boasts: “Preliminary evaluation using GPT-4 as a judge shows Vicuna-13B achieves more than 90%* quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90%* of cases.” - vicuna.lmsys.org.
It’s incredible that they managed to train such a powerful AI with just $300.
What this is
The previous article and video shows you the installation script for installing the GPU model.
The CPU-only model for those without GPUs, GPUs powerful enough, or just not Nvidia GPUs - This is for you.
The CPU setting didn’t work on install, but I’ve updated the All-in-one script for installation, with a more streamlined and faster setup where you can download the Vicuna CPU model, and use it immediately.
This is the updated article.
All-in-one script
Recently I’ve been creating simple PowerShell scripts that should install everything and get it up and working. While these usually work well, there have been some issues with Mamba, the Python version they’re using and installing packages. See this issue.
It’s worth a shot. The script I created runs the automated installers, and later gets to downloading the model automagically. See the video for more.
Open a PowerShell window where you intend to ‘install’ this project. Installing really is just having the files there, and deleting them will ‘uninstall’ the project.
So, head to Desktop and create a new folder called AI for example. Click an open spot in the path at the top of the Windows Explorer, select all text and type powershell and hit enter.
Run: iex (irm vicuna.tc.ht)
Follow the prompts on screen.
Should you want to install JUST the Ooba webui, you can use the command.
Manual install
If you’re interested in the GPU model, see the GPU Vicuna Installation Manual install section.
oobabooga/text-generation-webui
Download and extract the oobabooga-windows.zip zip from oobabooga/text-generation-webui. Open the oobabooga-windows folder and you’ll see a few batch scripts.
Running install.bat should download and set up the folder structure for the project.
Vicuna model (CPU)
The Vicuna model on HuggingFace doesn’t work, so we’ll download a different version. Download the Vicuna CPU model from HuggingFace.
Instead of opening download-model.bat, we’ll download and place the files ourself as using the bat file will download revision 1 and 2 of the model – wasting a full 8.14GB.
Head to the link above, and click the small down arrow next to the download size for ggml-vicuna-13b-4bit-rev1.bin.
So, open oobabooga-windows, then in models create eachadea_ggml-vicuna-13b-4bit. Drop the bin file in there.
The model should now download and set up.
Editing the start.bat
Open start-webui.bat with a text editor like Notepad.
Edit the line call python server.py --auto-devices --cai-chat to say call python server.py --auto-devices --chat --model eachadea_ggml-vicuna-13b-4bit. Save and close.
Then run start-webui.bat.
Note that the above is DIFFERENT from the GPU-only version.
Using the web-ui
To use the interface, wait for the project to completely open and you should see a URL in the console. Head to that URL in your browser and you’ll see the interface!
While the PowerShell script does all of this for you, it may not always work. Flaky install scripts and connections to servers can always cause it to fall apart mid-install, and continuing with a manual installation may be nessecary.
