Feature flags

Overview

A feature flag decides, for each request, which variation of a feature your app serves: on or off, or one of several values such as a model name, a number or a JSON object. Flags belong to one environment of a project, so production and staging have their own.

Open Developer tools → Telemetry → Feature flags (/feature-flags). The page lists the current environment’s flags with whether each is on, its type, its targeting and the app it is for. Search by key, name or description, and filter by state (on, off, archived) or app.

The SDKs fetch the environment’s flags with its API key and evaluate them inside your app: no network call per evaluation, and your users’ attributes never leave the process. Each evaluation is stamped on the active span, so you can compare cost, latency and evaluations per variation in Telemetry.

How a flag decides

Each flag has:

Part	What it does
Key	What your code asks for, such as new-checkout. Lowercase letters, digits, dots, dashes and underscores; it cannot be changed.
Type	Boolean, string, number or JSON. A boolean flag has exactly two variations, true and false.
Variations	Two to twenty values the flag can serve.
Served while off	The variation everyone gets while the flag is off or archived.
Targeting rules	Checked in order. The first rule whose conditions all hold decides.
Everyone else	What is served while the flag is on and no rule matches.
App	Optional. The service.name of the one app the flag is for; leave it empty for every app in the environment.

A rule or Everyone else serves either one variation or a percentage rollout across variations. Rollout percentages must add up to exactly 100%, in steps of 0.001%.

Conditions

A condition compares one attribute of the evaluation context with one or more values:

Operator	Matches when the attribute
equals / does not equal	is (or is not) the value
is one of / is not one of	is (or is not) any of the values
contains / starts with	contains, or starts with, any of the values
is less than, is at most, is greater than, is at least	compares as a number
is version, is a version before/after (or equal)	compares as a semantic version, such as 2.1.0 or 2.0.0-beta.1

A missing attribute never matches, including for “does not equal” and “is not one of”. When the attribute is a list (for example groups), a positive operator matches when any item does, and a negated one when no item does.

Segments

A segment is a named, reusable set of conditions of one environment, such as beta-users (groups is one of beta) or eu-enterprise (country is one of the EU countries and plan equals enterprise). A context is in a segment when every condition holds. Manage them under Feature flags → Segments.

In a rule, choose the operator is in segment or is not in segment and list segment keys: is in segment holds when the context is in any of them, is not in segment when it is in none. Segments cannot name other segments. A flag that names a segment the environment does not have is refused, and a segment some flag names cannot be deleted. Segment changes go through change control like flag changes, and the SDKs receive segments with the flags.

Percentage rollouts

Each user lands in one of 100,000 buckets, worked out from the flag key, the flag’s own random salt and the bucketing attribute (the context’s key unless you choose another, such as tenant). The same user always lands in the same bucket, in every SDK and on every server, so:

• raising a rollout from 20% to 50% keeps everyone who already had the feature,
• two flags with the same percentage do not roll out to the same people,
• a context without the bucketing attribute is bucket 0 (the first variation with any share).

Create and change flags

1. Click New flag, enter a key and name, and choose the type.
2. Set the variations and the variation served while the flag is off.
3. Add targeting rules, and choose what everyone else gets.
4. Click Create flag. A new flag is off.
5. Turn it on with the switch in the list. Every switch asks you to confirm.

The History button shows every saved version, who made it and who approved it, and the comment. Revert to this version restores that version’s variations, rules and everyone-else serving as a new version; whether the flag is on does not change.

Archive turns a flag off and stops sending it to SDKs, so code asking for it gets its own default. Its history is kept and Restore brings it back, still off.

SDKs pick up any change within about 30 seconds.

Change control

Flag changes follow the organization’s policy change control:

• with approval required, every create, change, switch, archive and revert waits for a second person with Approve policy changes, and is applied only if the flag has not changed in the meantime;
• a change freeze stops flag changes, except for the people it exempts;
• when comments are required, each change asks for one.

Pending flag changes appear on Policy changes for people who may view feature flags.

AI flags

Three presets in New flag → Start from give AI features a starting point:

Preset	Flag	Read it with
Model switch	A string flag whose variations are provider/model ids, such as openai/gpt-4o and openai/gpt-4o-mini.	flags.model(key, context, default)
Prompt version	A JSON flag whose variations name Prompt Hub prompt versions: { "name": "support-reply", "version": "1.2.0" } (the text support-reply@1.2.0 works too).	flags.prompt(key, context, default), which fetches the served version from Prompt Hub
Kill switch	A boolean flag that serves true while the feature may run. Switch the flag off and it serves false, the safe default, everywhere.	flags.killSwitch(key, context) (kill_switch in Python): true when the switch is engaged

The served variation is stamped on the span as for any flag, so the model or prompt version shows up beside the call’s cost, latency and evaluations. Until rules arrive, or for an unknown flag, a kill switch is not engaged: the feature keeps running rather than stopping because Shield360 could not be reached.

Compare variations

The chart icon on a flag’s row opens its variations side by side over the last 24 hours, 7 days or 30 days:

Column	What it is
Traces	Traces with a span carrying shield360.flag.<key>, by the variation it carries.
Calls	Model calls (spans with gen_ai.system) in those traces; in a trace with no model call, the stamped spans themselves.
Cost per call	Mean gen_ai.usage.cost per call.
Latency p50 / p95	Call durations.
Error rate	Calls whose status is an error.
Evaluations passed / mean eval score	Evaluations of those calls, including people’s thumbs up and down (manual_feedback).

A trace belongs to the variation one of its spans carries; if one trace was served two variations, it counts under one of them. Every value in the query (the flag’s attribute, the window) is a bound parameter.

Scheduled rollouts

Under Feature flags → Schedules, schedule a flag’s rollout as steps at given times, for example 10% now, 50% at noon and 100% tomorrow. Each step can serve a rollout or a variation by default, serve one variation to everyone (clearing the rules), or switch the flag on or off.

A job applies each step within about a minute of its time, through change control as the person who made the schedule:

• a change freeze that stops them holds the step (and the ones after it) until it lifts;
• when approval is required, the step becomes a pending change for a second person, and an alert says so;
• otherwise it is saved as a new version with the schedule named in its comment, and an alert says so.

Steps run in order. A step that fails (the flag was archived, or a variation it names was removed) ends the schedule. Cancel stops the steps not yet run; steps already applied stay.

Automatic flips

Under Feature flags → Automations, change a flag automatically when:

• a budget reaches a threshold: once per budget period, the first time the budget’s measured spend reaches the percentage (budgets are measured hourly). For example, at 90% of the Support budget serve the cheaper model to everyone.
• a detection rule fires: when the rule raises an alert, at most once per cooldown. For example, when the jailbreak detection fires, switch the agent’s kill switch off.

The budget or rule must belong to the same project and environment as the flag. An automation acts through change control as the person who made it, exactly like a schedule step: held by a freeze (it fires again once the freeze lifts), a pending change when approval is required, or applied as a new version. Each outcome raises an alert (feature_flag.changed_automatically, or feature_flag.automatic_change_held when it waits, is held or fails) and is written to the audit log.

Experiments

Under Feature flags → Experiments, compare a flag’s variations on one primary metric, each against a control variation, from the spans that carry the flag, from when the experiment starts until you stop it (at most 90 days):

Metric	Kind	Test
Conversion: evaluation passed (one evaluation, or any; manual_feedback is people’s thumbs up)	Rate	Two-proportion z-test
Conversion: a span attribute equals a value (per trace: converted when any of its spans carries it)	Rate	Two-proportion z-test
Error rate	Rate	Two-proportion z-test
Cost per call	Mean	Welch’s t-test
Latency (mean)	Mean	Welch’s t-test
Numeric span attribute (mean)	Mean	Welch’s t-test

For each variation the result shows the sample size, the estimate with its 95% interval (Wilson for a rate, Student’s t for a mean), and against the control the difference with its interval, the p-value and whether it is significantly better or worse. Everything is computed from ClickHouse aggregates (counts, sums, means and variances).

What the numbers assume:

• Independent observations. Each span (call or evaluation) is one observation; spans of one user or one trace are not independent, which makes intervals somewhat too narrow.
• Large samples. The tests rely on the normal approximation. No result is shown until every variation has the experiment’s minimum sample (100 by default, at least 30) and, for a rate, at least 5 conversions and 5 non-conversions. Until then the result says insufficient data.
• A fixed horizon. The p-value is for one look at a sample size you decided in advance. Checking again and again and stopping the first time something looks significant inflates false positives: set the minimum sample before you start and judge the result once it is reached.
• Two-sided, 5%. With more than two variations the 5% is divided by the number of comparisons with the control (Bonferroni), and the interval of the difference uses the same adjusted level.
• Means of skewed data. Latency and cost are skewed; Welch’s test compares means, which large samples make reliable, not medians.

Who can do what

Permission	Allows	Held by
flags.read (View feature flags)	See flags and their history	Owners, administrators, analysts, the Security role, members and viewers
flags.manage (Manage feature flags)	Create, change, switch, archive and revert flags; manage segments, schedules, automations and experiments	Owners and administrators

Both are checked on the server for every request.

Read flags in your app

The SDKs use the same Shield360 URL and API key as tracing: shield360Url / shield360ApiKey (TypeScript), shield360_url / shield360_api_key (Python), Shield360URL / Shield360APIKey in Go’s Config, or Shield360Url / Shield360ApiKey in .NET’s options, or the SHIELD360_URL and SHIELD360_API_KEY environment variables. Use the API key of the environment whose flags you want.

import Shield360 from 'shield360';

Shield360.init({ applicationName: 'checkout-api' });
await Shield360.flags.ready(); // optional: wait for the first fetch

const context = { key: user.id, email: user.email, tenant: user.tenantId, plan: user.plan };

if (Shield360.flags.isEnabled('new-checkout', context)) {
  // the new checkout
}
const model = Shield360.flags.getFlag('model-choice', context, 'gpt-4o-mini');

import shield360

shield360.init(application_name="checkout-api")
shield360.flags.wait_until_ready()  # optional: wait for the first fetch

context = {"key": user.id, "email": user.email, "tenant": user.tenant_id, "plan": user.plan}

if shield360.flags.is_enabled("new-checkout", context):
    ...  # the new checkout
model = shield360.flags.get_flag("model-choice", context, "gpt-4o-mini")

shield360.Init(shield360.Config{ApplicationName: "checkout-api"})
flags := shield360.Flags()
flags.WaitUntilReady(5 * time.Second) // optional

evalCtx := map[string]any{"key": user.ID, "plan": user.Plan}
if flags.IsEnabled(ctx, "new-checkout", evalCtx) {
  // the new checkout
}
model := flags.GetString(ctx, "model-choice", evalCtx, "gpt-4o-mini")

var flags = Shield360Sdk.Flags;
await flags.WaitUntilReadyAsync(); // optional

var context = new Dictionary<string, object?> { ["key"] = user.Id, ["plan"] = user.Plan };
if (flags.IsEnabled("new-checkout", context))
{
    // the new checkout
}
var model = flags.GetString("model-choice", context, "gpt-4o-mini");

The AI helpers, in each SDK:

const model = Shield360.flags.model('support-model', context, 'openai/gpt-4o-mini');
const { prompt, version } = await Shield360.flags.prompt('support-prompt', context, { name: 'support-reply' }, { variables: { tone: 'friendly' } });
if (Shield360.flags.killSwitch('ai-summaries', context)) return cachedSummary;

model = shield360.flags.model("support-model", context, "openai/gpt-4o-mini")
result = shield360.flags.prompt("support-prompt", context, {"name": "support-reply"}, variables={"tone": "friendly"})
if shield360.flags.kill_switch("ai-summaries", context):
    return cached_summary

model := flags.Model(ctx, "support-model", evalCtx, "openai/gpt-4o-mini")
p := flags.Prompt(ctx, "support-prompt", evalCtx, flags.PromptRef{Name: "support-reply"}, flags.PromptOptions{})
if flags.KillSwitch(ctx, "ai-summaries", evalCtx) {
  return cachedSummary
}

var model = flags.Model("support-model", context, "openai/gpt-4o-mini");
var prompt = await flags.PromptAsync("support-prompt", context, new PromptRef("support-reply"));
if (flags.KillSwitch("ai-summaries", context)) return cachedSummary;

prompt returns the prompt’s name, version, the served variation and what Prompt Hub answered (prompt, or error when it could not be fetched); it never throws.

The SDKs never throw or raise into your app. You get the default you pass (or false from isEnabled / is_enabled):

• until the first fetch has completed,
• for a flag that does not exist, is archived, or is for another app,
• when the flag’s type does not match your default.

While Shield360 cannot be reached, the last rules fetched keep being served.

getFlagDetail / get_flag_detail return the value with the served variation’s key and the reason (RULE_MATCH, FALLTHROUGH, OFF, FLAG_NOT_FOUND, and so on).

Setting	TypeScript	Python	Default
Poll interval	Shield360.flags.configure({ pollIntervalMs })	shield360.flags.configure(poll_interval=…) (seconds)	30 seconds, with ±10% jitter; at least 5 seconds
Request timeout	timeoutMs	timeout	10 seconds
Stamp spans	recordOnSpan	record_on_span	on

Variations on spans

Each evaluation that serves a variation sets shield360.flag.<key> to the variation’s key on the active span (the active Activity in .NET, the span in the context.Context in Go), for example shield360.flag.model-choice = b. Evaluate the flag inside the span you want to compare (the request or the LLM call); Compare variations and Experiments read these attributes.

Fetching rules yourself

GET /api/flags/rules with Authorization: Bearer <environment API key> returns the environment’s flags that are not archived and its segments, as the SDKs evaluate them, and a version. Send the ETag back as If-None-Match and an unchanged environment answers 304 Not Modified. A signed-in session is refused: the answer is always the key’s own environment.