Deploy and configure local language models in your Prisme.ai environment with the prismeai-llm microservice
prismeai-llm
microservice enables you to integrate open-source language models into your Prisme.ai environment. This service supports both LocalAI and Ollama as runtime engines, giving you flexibility in how you deploy and manage your models.
prismeai-llm
microservice allows you to:
quay.io/go-skynet/local-ai
. You can also configure it to use Ollama as an alternative runtime.
prismeai-llm
microservice, ensure you have:
./models
directory:
prismeai-apps/values.yaml
configuration:
Create Model Directory
/models
directory in your specified storage volume.Download Model Files
Add Configuration Files
Special Configuration for Embedding Models
Restart the Service
Test Text Generation
phi-2
with your installed model name (e.g., mistral-7b-instruct
, orca
, or airoboros
).You should receive a streamed response containing the generated text.Test Embedding Generation
bert
with your installed embedding model name (e.g., mpnet
).You should receive a response containing a vector of floating-point numbers representing the text embedding.Slow Response Times
DEBUG: true
in your environmentModel Loading Errors
Out of Memory Errors
Access Project Settings
Configure Model Settings
Save and Test
Adding New Models
/models
directory:
GPU Acceleration
Custom Model Templates
custom-format.tmpl
).System
, .Messages
, etc.