Does Nesting Bias a Permutation Test?

Shows what treating nested sequences as independent would cost. The comparison is run twice on the same data: once with the ordinary permutation test, which shuffles single sequences, and once with actor, which shuffles whole actors (the persons whose sessions they are, or the teams of students). The result places the two side by side, with the ICC and design effect that explain any difference between them. See the sections Nested data and actor and ICC and design effect of permutation for what these quantities mean.

Usage

permutation_diagnostics(
  x,
  y = NULL,
  actor,
  iter = 1000L,
  alpha = 0.05,
  level = c("overall", "edges"),
  seed = NULL
)

Arguments

x: A netobject_group (every pair of groups is diagnosed) or a netobject (then y is required). Transition methods only ("relative", "frequency", "co_occurrence").
y: A netobject to compare with x, or NULL.
actor: Character. Column identifying the actor each sequence belongs to, as in permutation.
iter: Integer. Permutation iterations for each of the two tests. Default 1000.
alpha: Numeric. Significance level. Default 0.05.
level: Character. "overall" (default): one row per group pair. "edges": one row per edge per pair.
seed: Integer or NULL. RNG seed; both tests use the same seed.

Value

A data.frame.

With level = "overall", one row per compared pair:

pair: "<x> vs <y>".
n_sequences, n_actors: Sequences and distinct actors in the pair.
design: "between" (every actor in one group), "within" (every actor in both groups) or "mixed".
icc, icc_ci_lower, icc_ci_upper: How alike the sequences of one actor are, with a 95% interval; as printed by permutation. NA interval with fewer than 3 actors.
deff_edges: Median over edges of the design effect, the ratio of the actor-level to the sequence-level null variance of the edge difference (Kish, 1965).
deff_global: The same ratio for the global M statistic.
p_global_sequence, p_global_actor: Permutation p-values of M when sequences or whole actors are reassigned.
sig_edges_sequence, sig_edges_actor: Edges with p < alpha under each test.
edges_changed: Edges significant under one test but not the other.
min_p_actor: Smallest p-value an exact actor-level test can produce, max(1 / arrangements, 1 / (iter + 1)).

With level = "edges", one row per edge present in either network: pair, from, to, diff, icc (per-edge ANOVA ICC, not bias-corrected; NA where the share does not vary), null_sd_sequence, null_sd_actor, deff (NaN where the edge difference never varies under either null), p_sequence, p_actor, changed.

The ICC and design effects are those of the actor-level run (see the clustering element of permutation); the p-values and significance counts compare it with a separate ordinary run. Errors with class nestimate_actor_unsupported for association networks and nestimate_actor_missing when actor is not a column of the networks' metadata or sequence data.

How to read the result

deff_edges, deff_global: The design effect (Kish, 1965): actor-level over sequence-level null variance. 1 means the two shuffles give the same chance variation; above 1 the actor-level one varies more, below 1 less.
edges_changed: Edges significant under one test but not the other. Edges with p-values close to alpha can flip from Monte Carlo error alone; increase iter before reading much into one or two.
min_p_actor above alpha: Too few actors: the actor-level test cannot reject anything.

References

Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall. (jackknife, ch. 11)

Kish, L. (1965). Survey Sampling. Wiley. (design effect)

Anderson, M. J., & ter Braak, C. J. F. (2003). Permutation tests for multi-factorial analysis of variance. Journal of Statistical Computation and Simulation, 73(2), 85-113.

Examples