permutation() tests whether two networks differ more than chance would
produce. The sequences (or, for association networks, the rows) behind
both networks are pooled, the group labels are shuffled iter times, both
networks are re-estimated on every shuffle, and each observed difference
is compared with the shuffled ones. It tests every edge, the network as a
whole and, with measures, the nodes' centralities.
It compares two netobjects from build_network() estimated with the
same method on the same nodes, for every built-in method and for
registered estimators whose result keeps the data (cleaned_data, see
register_estimator()). A netobject_group (or an mcml, converted with
as_tna()) is tested pair by pair: every pair of its groups when y is
NULL, or the groups of the same name in a second netobject_group. Two
wtna_mixed results of wtna() are tested on their transition and
co-occurrence networks, and two net_edge_betweenness() results on their
edge betweenness.
A small p-value says that the observed difference is larger than the
shuffles produce; it does not say how large or how probable the
difference is, which bayes_compare() estimates. With many edges, some
fall below alpha by chance: set adjust to correct for that. The
shuffle treats every sequence as exchangeable; when sequences are nested
in persons or teams, set actor, and see what the nesting changes with
permutation_diagnostics().
Usage
permutation(
x,
y = NULL,
iter = 1000L,
alpha = 0.05,
paired = FALSE,
adjust = "none",
measures = NULL,
nlambda = NULL,
seed = NULL,
actor = NULL
)
# S3 method for class 'net_permutation'
print(x, ...)
# S3 method for class 'net_permutation'
summary(object, ...)
# S3 method for class 'net_permutation_group'
print(x, ...)
# S3 method for class 'net_permutation_group'
summary(object, ...)
# S3 method for class 'wtna_perm_mixed'
print(x, ...)
# S3 method for class 'wtna_perm_mixed'
summary(object, ...)Arguments
- x
The first network: a
netobjectfrombuild_network()(a barecograph_networkis converted to one), anetobject_groupormcml, awtna_mixed, or anet_edge_betweenness()result. For theprint()methods: anet_permutation,net_permutation_grouporwtna_perm_mixed.- y
The second network, of the same kind as
x, estimated with the same method on the same nodes (their order may differ); other methods or nodes raise an error. DefaultNULL: whenxis anetobject_groupor anmcml, every pair of its groups is tested. When both arenetobject_groups, the groups present in both under the same name are tested, and having none in common raises an error.- iter
The number of shuffles, at least 2. Default
1000. The smallest attainable p-value is1 / (iter + 1).- alpha
The significance level, strictly between 0 and 1, that marks an edge or centrality as significant (
sig,diff_sig) and that thenestimate_few_actorswarning is checked against. It does not change the p-values. Default0.05.- paired
Whether the two networks hold paired observations (the i-th sequence or row of
xwith the i-th ofy, such as the same people measured twice). Each shuffle then swaps the two members of a pair at random instead of shuffling all labels, soxandyneed the same number of sequences (afterconcat) or rows; otherwise an error is raised. It cannot be combined withactor, of which it is the special case of one actor per pair. DefaultFALSE.- adjust
The multiplicity correction for the p-values, a method of
stats::p.adjust():"none"(default),"holm","hochberg","hommel","bonferroni","BH","BY"or"fdr". The correction runs over the tested edges: the cells non-zero in either network (a self-loop included), each pair counted once in an undirected network, whose matrix stays symmetric. Cells zero in both networks keep their raw p-value, 1. The centrality p-values are corrected separately, over all states and measures. Any other value raises an error of classnestimate_bad_inputbefore anything runs.- measures
Centrality measures to test as well as the edges: any of
"OutStrength","InStrength","ClosenessIn","ClosenessOut","Closeness","Betweenness","BetweennessRSP","Diffusion"and"Clustering"("InCloseness"and"OutCloseness"are accepted for"ClosenessIn"and"ClosenessOut"), or"all"for the nine. DefaultNULL(edges only). Every state and measure gets the observed difference, an effect size and a p-value from the same shuffled networks as the edge test. The centralities are computed as bynet_centrality()with its defaults: self-loops left out, heavier edges as shorter steps, andDiffusionrange-normalised to 0–1 in each network. An unknown name raises an error, and so does anymeasureswithnet_edge_betweenness()input.- nlambda
Ignored, kept so existing calls run. Default
NULL. The shuffled glasso networks use thenlambdathe networks were built with (see Details); a value other thanNULLgives a warning of classnestimate_ignored_argument.- seed
An integer that makes the shuffles reproducible. Default
NULL. The session's own random stream is left as it was; with grouped input each pair, or each shared group, is tested from the same seed.- actor
The column identifying the actor each sequence belongs to: the person whose sessions they are, or the team of a student. It is looked up in the network's metadata (such as
"student_id"when sessions are nested in students, or"Group"for students nested in teams) and then in its wide sequence data. Whole actors are then shuffled, and the ICC and design effect are reported; see the sections Nested data and actor and ICC and design effect. DefaultNULL: sequences are shuffled one by one. A value that is not one column name, oractorwithpaired = TRUE, raises an error of classnestimate_bad_actor; a column that is absent or has missing values,nestimate_actor_missing; a column whose values do not give one id per sequence (or, withconcat, that joins rows of different actors),nestimate_actor_misaligned; a method other than the transition methods,nestimate_actor_unsupported.- ...
- object
For the
summary()methods: anet_permutation,net_permutation_grouporwtna_perm_mixed.
Value
A net_permutation object. print() shows the settings, the
number of edges tested and significant, the global test and, with
actor, the ICC and design effects; summary() returns the edge table,
one row per edge present in either network; cograph::splot() draws
the difference network. Grouped input gives a net_permutation_group, a
named list of net_permutation results (one per group pair, named
"<group i> vs <group j>", when y is NULL; one per shared group
name otherwise), whose print() prints each and whose summary()
stacks the edge tables with a leading group column naming the pair or
group. Two wtna_mixed inputs give a wtna_perm_mixed, the
net_permutation results of the transition and cooccurrence
networks, whose print() prints both and whose summary() returns a
list of their two edge tables. The print() methods return their input
invisibly. The net_permutation's fields:
- x, y
The two networks compared.
- diff
The observed difference matrix,
x - y.- diff_sig
The observed difference where
p < alpha, else 0.- p_values
The p-value matrix (adjusted when
adjustis not"none").- effect_size
The effect-size matrix: the observed difference divided by the SD of the shuffled differences.
- summary
The edge table
summary()returns: one row per edge present in either network (an undirected network has one row per unordered pair), with columnsfrom,to,weight_x,weight_y,diff,effect_size,p_valueandsig.- global
The global test, one row per statistic:
statistic("M", the sum of absolute differences, and"S", the largest absolute difference),observed, andp_valuefrom the same shuffles as the edge test. Absent for edge-betweenness input.- method
The network estimation method (
"edge_betweenness"for edge-betweenness input).- source_method
For edge-betweenness input: the source networks' method.
- edge_betweenness
For edge-betweenness input: the
invertsetting and the source method used.- iter, alpha, paired, adjust
The settings used.
- actor
The
actorcolumn name, orNULL.- n_actors
The number of distinct actors, or
NULL.- null_sd
The SD of each edge's shuffled differences (the effect-size denominator), as a matrix.
- null_sd_m
The SD of the global statistic M over the shuffles. Absent for edge-betweenness input.
- clustering
Only with
actor: a one-row data frame withn_sequences,n_actors,design("between","within"or"mixed"),iccwithicc_ci_lowerandicc_ci_upper(how alike the sequences of one actor are),deff_edges(the median design effect over edges),deff_global(the design effect for M) andmin_p, the smallest attainable p-value.- clustering_edges
Only with
actor: one row per edge of the edge table, withfrom,to,icc,null_sd_actor,null_sd_sequenceanddeff.- centralities
Only with
measures: a list ofstats(one row per state and measure, withstate,centrality,diff_true,effect_sizeandp_value),diffs_true(the observed differences, one row per state and one column per measure) anddiffs_sig(the same wherep < alpha, else 0).
Details
For transition methods, per-sequence count matrices are computed once and
each shuffle only re-sums them, which keeps a large iter fast; the
networks' scaling and threshold are applied to every shuffled network
as to the observed one. For the other methods the estimator is re-run on
every shuffle (cor and pcor through a lighter direct path).
For glasso, every shuffled network is estimated as build_network()
estimated the observed ones, with x's params (each sample's own
lambda path, nlambda, lambda.min.ratio, gamma,
penalize.diagonal, refit), so the observed difference and its null
come from one estimator (van Borkulo et al. 2022). Two glasso networks
built with different settings raise an error of class
nestimate_bad_input.
A network that rests on one sequence, or one-hot data with no actor or
session, has nothing to shuffle and raises an error of class
nestimate_single_sequence.
Wide rows combined with params = list(concat = k) are shuffled as the
merged sequences the estimator used, keeping their internal joins. A
merged sequence must lie within one actor when actor is set (class
nestimate_actor_misaligned otherwise), and paired tests need the same
number of merged sequences. Both networks must use the same weighted
setting (class nestimate_bad_input otherwise).
A refit that fails or produces non-finite weights makes the permutation
statistic undefined for that shuffle. The test then raises an error of
class nestimate_permutation_failed that reports the number of failed
shuffles, and returns no p-values: failed shuffles are neither counted as
non-exceedances nor dropped from the null.
For net_edge_betweenness() input, the source networks are shuffled and
edge betweenness is recomputed on each shuffle. Both objects must come
from the same source method and use the same invert setting.
The centrality block follows the layout of
tna::permutation_test(measures = ): a centrality whose shuffled
differences do not vary gets an effect size of NaN or Inf, as there,
rather than the 0 the edge table uses.
What is tested
Two kinds of question are answered from the same shuffles.
- Edge tests
One test per edge: is the difference in this edge's weight,
x - y, larger in absolute value than the shuffles produce?summary()reports each with an effect size (the observed difference divided by the SD of the shuffled differences, 0 when that SD is 0) and a p-value,(number of shuffles at least as extreme + 1) / (iter + 1).- Global test
One test for the whole network: do the two networks differ at all? Two statistics: M, the sum of the absolute differences over all cells of the weight matrix (the total amount of difference; an undirected edge counts twice), and S, the largest absolute difference. Being a single test, it needs no multiplicity correction.
print()shows it.
The letters follow this function's historical convention. In nct(), M
is instead the largest absolute edge difference and S the difference in
global strength; the sum of absolute edge differences here measures total
edge difference, not a difference in global strength. The smallest
attainable p-value is 1 / (iter + 1); with the default iter = 1000 it
is 0.000999, meaning that no shuffle came close.
Nested data and actor
Shuffling treats every sequence as an exchangeable unit. When several
sequences come from the same actor (sessions of one person, students of
one team), actor names the column identifying that actor, and the
shuffle respects it (Good, 2005; Anderson & ter Braak, 2003): an actor
whose sequences are all in one group moves to the other group as a whole,
and an actor with sequences in both groups has their labels shuffled
among their own sequences only. Mixed designs combine the two. The
observed differences do not change; only the p-values and effect sizes
do. With few actors there are few distinct ways to shuffle them, and a
warning of class nestimate_few_actors is raised when no p-value could
fall below alpha.
actor is available for transition networks built from sequences
("relative", "frequency", "co_occurrence"). On one-hot data the
actor column is held per row rather than per unit, so the call raises an
error of class nestimate_actor_misaligned. The other methods, association networks ("cor",
"pcor", "glasso", ...) among them, do not keep row identifiers after
estimation and raise an error of class nestimate_actor_unsupported.
ICC and design effect
With actor, print() also reports:
- ICC
The intraclass correlation, the proportion of the total variance that lies between actors (Shrout & Fleiss, 1979). An ICC close to 0 indicates little evidence of a nesting effect. It is the one-way ANOVA ICC of each sequence's transition shares within each group, averaged over edges weighted by edge frequency and jackknife bias-corrected, with a 95% interval from the leave-one-actor-out jackknife (Efron & Tibshirani, 1993).
- Design effect
The ratio of the variance of an estimate under the clustered design to its variance had the units been sampled independently (Kish, 1965). Here it is the variance of the shuffled differences when whole actors are moved, divided by the variance when single sequences are moved, both drawn in the same run, reported as the median over edges and for the global statistic M. For equal numbers of sequences per actor m, Kish gives the approximation
1 + (m - 1) * ICC.
Reading the printed output
Permutation Test: Transition Network (relative probabilities) [directed]
Iterations: 1000 | Alpha: 0.05 | Actor: Group (200 actors)
Nodes: 9 | Edges tested: 78 | Significant: 42
Global test (networks differ overall?): M = 2.612 (p = 0.000999) | ...
Nesting in Group: ICC = -0.002 [95% CI -0.006, 0.002] | between design
Design effect (1 = nesting does not matter): edges 1.03 | global 1.20
Line 2: the settings, and the actor column with its number of actors.
Line 3: the edges present in either network and how many differ at
alpha. Line 4: the global test. Lines 5 and 6, only with actor: the
ICC with its interval, whether actors sit in one group (between), in
both (within) or either (mixed), and the design effects. With actor,
when no p-value can fall below alpha (too few actors, or too small an
iter), a last line says so.
References
Anderson, M. J., & ter Braak, C. J. F. (2003). Permutation tests for multi-factorial analysis of variance. Journal of Statistical Computation and Simulation, 73(2), 85-113.
Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall.
Good, P. (2005). Permutation, Parametric, and Bootstrap Tests of Hypotheses (3rd ed.). Springer.
Kish, L. (1965). Survey Sampling. Wiley.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428.
van Borkulo, C. D., van Bork, R., Boschloo, L., Kossakowski, J. J., Tio, P., Schoevers, R. A., Borsboom, D., & Waldorp, L. J. (2023). Comparing network structures on three aspects: A permutation test. Psychological Methods, 28(6), 1273-1285.
See also
permutation_diagnostics() to compare the actor-level and
ordinary tests side by side; bayes_compare() for the Bayesian
complement, which instead of "is this difference more extreme than
chance?" answers "how probable is a difference, and how large?";
build_network(), bootstrap_network()
Examples
s1 <- data.frame(V1 = c("A","B","C"), V2 = c("B","C","A"))
s2 <- data.frame(V1 = c("A","C","B"), V2 = c("C","B","A"))
n1 <- build_network(s1, method = "relative")
n2 <- build_network(s2, method = "relative")
perm <- permutation(n1, n2, iter = 10)
# \donttest{
set.seed(1)
d1 <- data.frame(V1 = sample(LETTERS[1:4], 20, TRUE),
V2 = sample(LETTERS[1:4], 20, TRUE),
V3 = sample(LETTERS[1:4], 20, TRUE))
d2 <- data.frame(V1 = sample(LETTERS[1:4], 20, TRUE),
V2 = sample(LETTERS[1:4], 20, TRUE),
V3 = sample(LETTERS[1:4], 20, TRUE))
net1 <- build_network(d1, method = "relative")
net2 <- build_network(d2, method = "relative")
perm <- permutation(net1, net2, iter = 100, seed = 42)
print(perm)
#> Permutation Test: Transition Network (relative probabilities) [directed]
#> Iterations: 100 | Alpha: 0.05
#> Nodes: 4 | Edges tested: 16 | Significant: 3
#> Global test (networks differ overall?): M = 3.858 (p = 0.0297) | S = 0.567 (p = 0.109)
summary(perm)
#> from to weight_x weight_y diff effect_size p_value sig
#> 1 A A 0.33333333 0.2222222 0.11111111 0.6504300 0.49504950 FALSE
#> 2 A B 0.06666667 0.3333333 -0.26666667 -1.9871473 0.04950495 TRUE
#> 3 A C 0.20000000 0.2222222 -0.02222222 -0.1329339 0.93069307 FALSE
#> 4 A D 0.40000000 0.2222222 0.17777778 1.0449465 0.25742574 FALSE
#> 5 B A 0.30769231 0.3333333 -0.02564103 -0.1106148 0.98019802 FALSE
#> 6 B B 0.38461538 0.0000000 0.38461538 1.9215279 0.03960396 TRUE
#> 7 B C 0.23076923 0.2500000 -0.01923077 -0.1103038 0.99009901 FALSE
#> 8 B D 0.07692308 0.4166667 -0.33974359 -1.9910325 0.07920792 FALSE
#> 9 C A 0.33333333 0.1111111 0.22222222 1.0147102 0.38613861 FALSE
#> 10 C B 0.66666667 0.3333333 0.33333333 1.3787271 0.24752475 FALSE
#> 11 C C 0.00000000 0.2222222 -0.22222222 -1.3964532 0.26732673 FALSE
#> 12 C D 0.00000000 0.3333333 -0.33333333 -1.8265261 0.12871287 FALSE
#> 13 D A 0.00000000 0.5000000 -0.50000000 -1.8205956 0.10891089 FALSE
#> 14 D B 0.00000000 0.2000000 -0.20000000 -0.9828935 0.45544554 FALSE
#> 15 D C 0.33333333 0.2000000 0.13333333 0.5436513 0.56435644 FALSE
#> 16 D D 0.66666667 0.1000000 0.56666667 2.5117967 0.01980198 TRUE
# Students are nested in teams, and Achiever is a team-level label:
# permute whole teams, not single students
net <- build_network(group_regulation_long, method = "relative",
actor = "Actor", action = "Action", time = "Time",
group = "Achiever")
permutation(net, iter = 100, actor = "Group", seed = 1)
#> Grouped Permutation Test
#> Groups: High vs Low
#>
#> -- High vs Low --
#> Permutation Test: Transition Network (relative probabilities) [directed]
#> Iterations: 100 | Alpha: 0.05 | Actor: Group (200 actors)
#> Nodes: 9 | Edges tested: 78 | Significant: 43
#> Global test (networks differ overall?): M = 2.612 (p = 0.0099) | S = 0.210 (p = 0.0099)
#> Nesting in Group: ICC = -0.002 [95% CI -0.006, 0.002] | between design
#> Design effect (1 = nesting does not matter): edges 1.03 | global 1.26
# }