Skip to main content
A fine-tuned model answers only while it is loaded on a GPU. A loaded model is warm. When other models need the room, it can be unloaded, and it goes cold. The next call to a cold model waits while it loads again. That wait is a cold start, and it can take several seconds (cold starts). Base models have no cold starts. Keep warm keeps a fine-tuned model loaded, so its calls never wait for a cold start. It helps most for a model called now and then. It doesn’t guarantee a latency. This request keeps the model a family serves warm:
Send {"enabled": false} to switch keep warm off. The answer is the family or the model.
  • Switching a model off stops its own keep warm.
  • A model kept warm through its family and on its own is one warm model. It stays warm until neither asks for it.
Families and models show these fields:

​
Checks

To be sure you change what you last read, add a check. If it fails, nothing changes.
  • To a family, send served: the adapter of the model it serves, or null for none. If the family has moved since, you get 409 served_changed.
  • To a model, send adapter: its ID as you saw it. If the number names another model, you get 409 model_changed.

​
Limits

An account keeps at most its tier’s kept_warm models warm at once. Tiers gives the number for each tier, and effective_limits on /v1/account gives yours. Each model counts once, whether its family, its own switch or both keep it warm. Each size’s room for kept-warm models is shared by every account. Both limits are checked before the model is loaded. Lowering a limit switches no model off. A kept-warm family whose move would go over either limit waits, still kept warm, until there is room (what a move needs).

​
Errors

​
Cost

Keep warm is billed for every hour it is on, whether or not the model is called. Each hour is at the base model’s price, billed by the started minute. Pricing lists the prices. A model is billed once, however many switches ask for it. Keep warm and storage explains how each hour is held and charged.