Skip to content
FeatlyFeatly

Segments and experiments

Segments

Section titled “Segments”

A segment is a named, reusable audience: a set of conditions, defined once, referenced by many flags and configs.

Instead of repeating user.plan == "pro" AND user.country IN [...] across a dozen flags, define a pro-users segment and target it with a single InSegment condition. Edit the segment once and every flag that references it follows.

Segment "beta-testers":
  user.email EndsWith "@ourcompany.com"
  OR user.betaOptIn Equals true


Flag rule:
  IF context InSegment "beta-testers"  ->  variant "on"

Segments are first-class entities with their own screen in the dashboard and their own API. A rule references a segment by key, and segment membership is evaluated as part of the same first-match-wins rules walk.

Experiments

Section titled “Experiments”

An experiment is an A/B test built on the flag engine. It combines:

  • Deterministic bucketing — the same targetingKey is assigned to the same variant for the life of the experiment (the MurmurHash3 split), so a user’s experience is consistent.
  • Exposure events — when a user is bucketed into a variant, Featly can emit an exposure event, and your code can emit custom events (featly.Events.TrackAsync("checkout.completed")). Together these let you measure the effect of a variant on a metric.
if (await featly.Flags.IsEnabledAsync("new-checkout", ctx))
{
    await featly.Events.TrackAsync("checkout.started");
}

Bucketing being deterministic and shared between the SDK and the server means exposure attribution is consistent no matter where evaluation happens.

Statistical significance

Section titled “Statistical significance”

The analytics endpoint (and the Experiments dashboard) reports, per metric, a two-proportion z-test comparing each variant’s conversion rate against a baseline — the flag’s default variant, or the first variant observed if the default was never exposed. A variant crossing the conventional p < 0.05 threshold is flagged isSignificant, and the highest-converting variant among those significantly better than the baseline is named the metric’s winner (a significant drop never wins).

This is a fixed-horizon test: it assumes you decide on a sample size (or a running duration) in advance and look once. Checking a running experiment repeatedly and stopping as soon as it turns significant inflates the false-positive rate — a well-known pitfall with fixed-horizon tests. Sequential analysis, which is safe to peek at anytime, is a possible future addition.

Next

Section titled “Next”