Stanford Alpaca is an instruction-following large language model developed by researchers at Stanford University. Released in March 2023, it was created by fine-tuning LLaMA 7B on 52,000 instruction-following demonstrations generated using OpenAI's text-davinci-003 model. The project attracted attention for showing that instruction-following models could be produced from existing foundation models at relatively low cost.
Development Alpaca was developed as an inexpensive research model for studying instruction-following systems. Using a modified version of the Self-Instruct method, the researchers expanded 175 human-written examples into a dataset of 52,000 demonstrations using text-davinci-003. Data generation cost less than US$500, while fine-tuning LLaMA 7B took about three hours on eight 80 GB Nvidia A100 GPUs and was estimated to cost less than US$100.
In a preliminary evaluation, Alpaca performed similarly to text-davinci-003, receiving 90 preferences compared with 89 for text-davinci-003. The researchers cautioned that the evaluation was limited and that Alpaca inherited weaknesses from both LLaMA and its training data.
Release and limitations Stanford released the dataset and training code for research use through the project's GitHub repository. Commercial use was restricted because the model was derived from LLaMA and its instruction data was generated through OpenAI's API.
Stanford also launched a public demonstration but removed it shortly afterward, citing hosting costs and inadequate content filtering. The model could produce hallucinations, misinformation, toxic language, and stereotypical responses, and was not intended for general deployment.
See also List of large language models List of software developed at universities Vicuna LLM
External links
References