Watch the video:
0:00 - Explanation
0:22 - What is Vicuña?
1:36 - What is Oobabooga?
2:18 - One-click install script
4:59 - Using the WebUI
5:45 - Basic chat & VRAM usage
6:24 - Guide responses in certain ways (POWERFUL)
7:14 - Preset AI characters/Personas
7:49 - Creating a character/AI persona
8:11 - Limitations
9:02 - Parameters
9:48 - Fine-Tuning
10:05 - Speech-To-Text and Text-To-Speech (TTS/STT)What is Vicuna
Vicuna is a model from the Team with members from UC Berkley, CMU, Stanford, and UC San Diego. Vicuna-13B is an open-source chatbot trained by fine-tuning LLaMA. Their website boasts: “Preliminary evaluation using GPT-4 as a judge shows Vicuna-13B achieves more than 90%* quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90%* of cases.” - vicuna.lmsys.org.
It’s incredible that they managed to train such a powerful AI with just $300.
This guide is for the GPU version of Vicuna and Oobabooga. Should you want the CPU version (compatible with CPUs and AMD) see the updated article.
All-in-one script
Recently I’ve been creating simple PowerShell scripts that should install everything and get it up and working. While these usually work well, there have been some issues with Mamba, the Python version they’re using and installing packages. See this issue.
It’s worth a shot. The script I created runs the automated installers, and later gets to downloading the model automagically. See the video for more.
Open a PowerShell window where you intend to ‘install’ this project. Installing really is just having the files there, and deleting them will ‘uninstall’ the project.
So, head to Desktop and create a new folder called AI for example. Click an open spot in the path at the top of the Windows Explorer, select all text and type powershell and hit enter.
Run: iex (irm vicuna.tc.ht)
Follow the prompts on screen.
Should you want to install JUST the Ooba webui, you can use the command.
Manual install
oobabooga/text-generation-webui
Download and extract the oobabooga-windows.zip zip from oobabooga/text-generation-webui. Open the oobabooga-windows folder and you’ll see a few batch scripts.
Running install.bat should download and set up the folder structure for the project.
Vicuna model
The Vicuna model on HuggingFace doesn’t work, so we’ll download a different version. Download the Vicuna model from HuggingFace.
Open download-model.bat, enter L and hit enter (it should correspond with L) None of the above).
Enter the following text and hit enter: anon8231489123/vicuna-13b-GPTQ-4bit-128g.
The model should now download and set up.
Alternatively, head to the Files and vesrions tab, and click the small down arrow next to the size for each file.
Move them all to oobabooga-windows\text-generation-webui\models\anon8231489123_vicuna-13b-GPTQ-4bit-128g (the last part of the path will be a folder you create).
Editing the start.bat
Open start-webui.bat with a text editor like Notepad.
Edit the line call python server.py --auto-devices --cai-chat to say call python server.py --auto-devices --chat --wbits 4 --groupsize 128 --model anon8231489123_vicuna-13b-GPTQ-4bit-128g. Save and close.
Then run start-webui.bat.
Using the web-ui
To use the interface, wait for the project to completely open and you should see a URL in the console. Head to that URL in your browser and you’ll see the interface!
While the PowerShell script does all of this for you, it may not always work. Flaky install scripts and connections to servers can always cause it to fall apart mid-install, and continuing with a manual installation may be nessecary.
