Runbooks on Spinnaker

API Rate Limiting

Mon, 01 Jan 0001 00:00:00 +0000

If your Spinnaker deployment has API clients interacting with it, enabling and knowing how to operate the API rate limiter can help keep Spinnaker reliable through spikes of heavy traffic or rogue clients.

Intro

The API Rate Limiter currently supports limiting individual authenticated and anonymous principals, bucketing requests into windows that are refreshed every interval as well as global & per-principal learning mode. Both the capacity, as well as the window size, are configurable globally and per-principal.

Caching: Account Name Based Sharding

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Account name based sharding is applicable to the caching function of a clouddriver. In case of HA mode, it is applicable to clouddriver-caching pods only. This feature is added at caching scheduler(both redis & sql backed) level which means it’s applicable to any cloud provider.

This is a configurable feature. When configured, all the caching agents of an account are run by a single caching pod. The logic of identifying the account whose caching agents are to be run in a particular pod is dependent on the name of the account(hashcode) and the number of available caching pods. When the pods are scaled up or down, the agents will resettle to the same or a different pod based on the sharding logic.

Orca: Quality of Service

Mon, 01 Jan 0001 00:00:00 +0000

EXPERIMENTAL: This feature is still in an early adoption / experimental phase. While you can use it today (Orca v6.71.0), Netflix is currently running this in learning mode / judiciously enabling in response to on-call events.

Spinnaker ships with an optional Quality of Service (QoS) module that can be used to manage the amount of active executions running at any given time. By default, this QoS module is disabled, but can be enabled and tuned with a handful of knobs and different strategies. Before we dive into the configuration settings, we’ll go over how QoS works.

Orca: Zombie Executions

Mon, 01 Jan 0001 00:00:00 +0000

Aliases: orphaned execution

A zombie Execution is one that has a status in the database of RUNNING but there are no messages in Orca’s work queue or unacked set—the pipeline or task is not doing anything.

Diagnosis

Logs will be emitted regularly for Executions that are currently running in Orca via the QueueProcessor class, which will look similar to the following example. If no logs have been emitted for over 10 minutes for a RUNNING Execution, it is very likely a zombie.

Sharding Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Intro

This document shows you how to shard traffic to different Spinnaker services based upon configured criteria. The general pattern is to define a selector class in your configuration. Endpoints will then be selected based upon the criteria specified in the selectors.

As an example, Netflix creates read-only shards for Clouddriver to better manage requests.

Selectors exist at these levels:

Application
Execution type (i.e, Pipeline vs Orchestration)
Origin
Authenticated User
Deployment account
Cloud Provider

You will need to modify your Spinnaker deployment pipelines to ensure the infrastructure for each shard is correctly created, otherwise traffic could be sent to non-existent shards.