Skip to content

Deploy models with Cog#

Cog containers are Docker containers that serve an HTTP server for running your model. You can deploy them anywhere that Docker containers run.

The server inside Cog containers is coglet, a Rust-based inference server that handles HTTP requests, worker process management, and run execution.

This guide assumes you have a model packaged with Cog. If you don't, follow our getting started guide, or start from one of the examples in the Cog repository.

Build a Docker image#

Build your model into a Docker image:

cog build -t my-model

The image contains your model code, dependencies, the Cog runtime, and everything in between. It serves an HTTP server on port 5000 when run.

Run the model#

You have several options for running a built image.

Docker#

Run the image directly with Docker. This is the approach you'd use for production deployment.

# If your model uses a CPU:
docker run -d -p 5001:5000 my-model

# If your model uses a GPU:
docker run -d -p 5001:5000 --gpus all my-model

The server listens on port 5000 inside the container (mapped to 5001 above).

cog serve#

For local development, cog serve builds the image and starts the server with your project directory mounted in:

cog serve

By default the server runs on port 8393 and the container port is published on 127.0.0.1 (localhost), so it is only reachable from your local machine. The server process inside the container binds to 0.0.0.0; use --host to control which host interface the Docker port mapping is published on.

Use -p to choose a different port:

cog serve -p 5000

Make a prediction#

Once the server is running, make predictions by sending a POST request to the /predictions endpoint. Inputs go inside an "input" object in the JSON body.

[!NOTE] The examples below use localhost:5001, matching the Docker command above (-p 5001:5000). If you used cog serve, use localhost:8393 by default, or the port you passed with -p.

curl http://localhost:5001/predictions -X POST \
    -H "Content-Type: application/json" \
    -d '{"input": {"prompt": "a photo of a cat", "steps": 50}}'

{
  "status": "succeeded",
  "output": "data:image/png;base64,...",
  "metrics": {
    "predict_time": 4.52
  }
}

[!IMPORTANT] Inputs must be wrapped in an "input" object. {"input": {"scale": 2.0}} is correct; {"scale": 2.0} is not.

To discover what inputs your model accepts, view the OpenAPI schema:

curl http://localhost:5001/openapi.json

Passing file inputs#

File inputs (cog.Path or cog.File types) are passed as strings inside the "input" object. There are two ways to do this:

1. HTTP/HTTPS URLs

Pass a URL to a publicly accessible file. The server downloads it inside the container:

curl http://localhost:5001/predictions -X POST \
    -H "Content-Type: application/json" \
    -d '{"input": {"image": "https://example.com/photo.jpg"}}'

2. Data URLs (base64)

To pass a local file, encode it as a data URL:

# Construct a data URL from a local file
DATA_URL=$(python3 -c "
import base64, mimetypes
with open('input.jpg', 'rb') as f:
    data = base64.b64encode(f.read()).decode()
mime = mimetypes.guess_type('input.jpg')[0] or 'application/octet-stream'
print(f'data:{mime};base64,{data}')
")

curl http://localhost:5001/predictions -X POST \
    -H "Content-Type: application/json" \
    -d "{\"input\": {\"image\": \"$DATA_URL\"}}"

[!NOTE] The HTTP API only accepts JSON (application/json). Multipart form uploads are not supported. When you use cog run -i [email protected], the CLI handles the base64 encoding for you automatically.

Getting output files#

When a model returns a file output (cog.Path or cog.File), the response contains a base64-encoded data URL by default:

{
  "status": "succeeded",
  "output": "data:image/png;base64,iVBORw0KGgo..."
}

To have the server upload output files to external storage instead, start the server with the --upload-url flag. The server then uploads each file output to that URL prefix and returns the resulting URL in the response.

With cog serve:

cog serve --upload-url https://example.com/upload/

When running the image directly with Docker, override the command to start the server with --upload-url:

docker run -d -p 5001:5000 my-model \
    python -m cog.server.http --upload-url https://example.com/upload/

With an upload URL configured, file outputs are uploaded and the response contains the uploaded URL instead of a data URL:

{
  "status": "succeeded",
  "output": "https://example.com/upload/image.png"
}

Run a one-off prediction#

The Docker and cog serve commands above leave an HTTP server running. If you instead want to run a single prediction and exit — without starting a server — use cog run:

cog run my-model -i [email protected]

This starts the container, runs one prediction, prints the result, and exits. File inputs are passed with the @ prefix (e.g. -i [email protected]), and the CLI handles base64 encoding for you.

Health checks#

The server exposes a GET /health-check endpoint that returns the current status of the model container. Use this for readiness probes in orchestration systems like Kubernetes.

curl http://localhost:5001/health-check

The response includes a status field with values like STARTING, READY, BUSY, SETUP_FAILED, or DEFUNCT. See the HTTP API reference for full details.

Stop the server#

If you started the container with docker run -d, stop it with:

docker kill <container-id>

If you used cog serve, press Ctrl+C in the terminal. (cog run exits on its own once the prediction finishes, so there's nothing to stop.)

Concurrency#

By default, the server processes one run at a time. To enable concurrent runs, make your run() method async and decorate it with @cog.concurrent(max=N):

import cog

class Runner(cog.BaseRunner):
    @cog.concurrent(max=4)
    async def run(self) -> str:
        return "hello world"

The deprecated concurrency.max field in cog.yaml is still supported and takes precedence over the decorator by baking COG_MAX_CONCURRENCY into the image.

Environment variables#

You can configure runtime behavior with environment variables:

  • COG_SETUP_TIMEOUT: Maximum time in seconds for the setup() method (default: no timeout).
  • COG_MAX_CONCURRENCY: Number of concurrent prediction slots (default: 1). Overrides both @cog.concurrent and deprecated cog.yaml concurrency.

See the environment variables reference for the full list.

Next steps#