Research workspace
Experiment pending· no human-vs-AI result exists yet
Research Mode

Where inside Qwen does information useful for distinguishing human-written from AI-generated text emerge?

Qwen3-8B-Base is frozen. For each document we read its hidden states at every depth and ask whether a simple linear classifier can tell human-written from fully AI-generated text using only that representation. The model is never fine-tuned.

Secondary questions

  1. At what layer does separation first become linearly accessible?
  2. Is the strongest layer intermediate or final?
  3. Does RMSNorm change separability?
  4. Does activation magnitude matter?
  5. Does attention increase separation?
  6. Does the MLP increase separation?
  7. Does the signal generalize across AI generators?
  8. Does the signal survive domain shift?

How results are worded on this site

  • A layer detects AI. → A linear classifier can access information correlated with the human-vs-AI label from this representation.
  • RMSNorm causes AI detection. → Separability changed across these representations, suggesting normalization may alter the accessibility of label-related information.
  • Qwen knows this is AI text. → (never claimed)
  • A detector proves authorship. → A probe score is evidence about a text's statistical properties, not proof of who wrote it.
  1. 1Scientific question
    Where inside a frozen Qwen3-8B-Base does information useful for distinguishing human-written from fully AI-generated text become linearly accessible, and which computations (attention, MLP, normalization) change that accessibility?
  2. 2Hypothesis
    Linear accessibility of the label is higher at some intermediate depths than at the embeddings, and changes across block sub-steps. This is a hypothesis to test, not an expected result; no direction is assumed for normalization.
  3. 3Dataset and access
    Pangram EditLens (pangram/editlens_iclr), human-written vs fully AI-generated only; AI-edited text excluded. Access is gated (manual approval, non-commercial terms) and has not been granted. LatentLens will not use unofficial copies to get around the gate.
  4. 4Model and representation
    Qwen/Qwen3-8B-Base @ 49e3418f, frozen. Each document: ⟨start⟩ + up to 512 tokens; pooled residual stream at all 37 depths (mean and last-token; one chosen on validation); seven sites inside five blocks.
  5. 5Splits and leakage controls
    Published train / val / test splits plus held-out generator (test_llama) and domain (test_enron) splits. The run stops if any id, identical text or EditLens source_id group appears in two splits.
  6. 6Probe and baselines
    Standardised L2 logistic regression per depth (C by grouped cross-validation inside train). Baselines: shuffled labels, length, token count, perplexity under the model, stylometric features; length-matched test subset; per-generator results.
  7. 7Metrics and uncertainty
    ROC-AUC (primary), PR-AUC, balanced accuracy, F1. Depth and pooling chosen on validation only; the test split used once, at the selected depth and at the final depth, with bootstrap 95% intervals. Exploratory comparisons (RMSNorm, attention vs MLP) use validation with paired bootstrap differences.
  8. 8Limitations
    One model, one dataset. A successful probe shows predictive, label-correlated information under this protocol; it does not show that the model uses that information, identify a mechanism, or provide reliable proof of authorship for any individual text.
Data and method

Dataset, extraction and probing protocol

Dataset: Pangram EditLens

pangram/editlens_iclr on Hugging Face: human-written, AI-edited and fully AI-generated texts with generator and source metadata. The first task uses only human-written vs fully AI-generated text. Published splits: train, val, test, plus test_llama (held-out generator family) and test_enron (domain shift: e-mails).

Status: access is gated (manual approval, non-commercial terms). Access has not been granted to this project yet, so no human-vs-AI result exists.

Hidden-state extraction

  • document = <|endoftext|> + up to 512 tokens; the start token is excluded from pooling
  • all 37 depths: mean pooling and last-token pooling (one is chosen on validation)
  • seven sites inside 5 blocks for the RMSNorm and attention-vs-MLP studies
  • per-document mean token RMS and mean next-token loss (perplexity baseline)
  • frozen model, inference mode, explicit hooks, nothing stored per token

Probing protocol (fixed in advance)

  • StandardScaler + L2 logistic regression; C by grouped CV inside train
  • layer and pooling selected on validation only
  • test evaluated once: at the selected depth and at the final depth
  • leakage audit (ids, identical texts, EditLens source_id groups) must pass first
  • controls: shuffled labels, length, token count, perplexity, stylometry, length-matched test subset, per-generator and held-out splits
Pipeline status. Every script runs end to end and is covered by automated structural tests on the small Qwen3-0.6B-Base checkpoint with a synthetic fixture (never shown as a result). The layer-by-layer runner used on the laptop matched the ordinary forward pass to within 1e-4 on Qwen3-0.6B-Base in float32. On Qwen3-8B in bfloat16 it was compared with an independent offloaded transformers run: 6/6 and 12/14 top-1 tokens agreed and log-probabilities differed by up to 0.46 nats. That is agreement within bfloat16 noise on a small check, not proof of numerical equivalence; the runner remains an implementation under validation.

Results

Each result below is either a recorded measurement or an explicit pending state with the commands that produce it. No placeholder charts are drawn.

8Research

Human-vs-AI separability across Qwen

Validation ROC-AUC of a linear probe at every depth. Chance is 0.5. The best depth is chosen on validation; the test set is used once.
Experiment pendingLayerwise probe on Qwen3-8B-Base

Needs the EditLens dataset (pangram/editlens_iclr is gated: request access on Hugging Face, then `hf auth login`) and a hidden-state extraction run with Qwen3-8B-Base. No values are shown until the experiment has run.

Commands that produce this result
python scripts/latentlens/extract_hidden_states.py --model Qwen/Qwen3-8B-Base --dataset hf:pangram/editlens_iclr --split train --limit 4000 --loader full --device cuda --dtype bfloat16 --batch-size 8
python scripts/latentlens/extract_hidden_states.py --model Qwen/Qwen3-8B-Base --dataset hf:pangram/editlens_iclr --split val --limit 1000 --loader full --device cuda --dtype bfloat16 --batch-size 8
python scripts/latentlens/extract_hidden_states.py --model Qwen/Qwen3-8B-Base --dataset hf:pangram/editlens_iclr --split test --limit 2000 --loader full --device cuda --dtype bfloat16 --batch-size 8
python scripts/latentlens/extract_hidden_states.py --model Qwen/Qwen3-8B-Base --dataset hf:pangram/editlens_iclr --split test_llama --limit 1000 --loader full --device cuda --dtype bfloat16 --batch-size 8
python scripts/latentlens/extract_hidden_states.py --model Qwen/Qwen3-8B-Base --dataset hf:pangram/editlens_iclr --split test_enron --limit 1000 --loader full --device cuda --dtype bfloat16 --batch-size 8
python scripts/latentlens/train_layer_probes.py --representations data/latentlens/reps/qwen3-8b-editlens_iclr --extra-test test_llama --extra-test test_enron
9Research

Representation journey (PCA)

PCA is an exploratory visualization. Quantitative evidence comes from held-out classifier metrics. Fitted on training documents, shown on validation documents.
Experiment pendingPCA of pooled representations

Runs on the same extraction as the layer probes; fitted on train, shows validation documents. No values are shown until the experiment has run.

Commands that produce this result
python scripts/latentlens/build_pca_artifacts.py --representations data/latentlens/reps/qwen3-8b-editlens_iclr
10Research

RMSNorm and activation magnitude

Does normalization change how linearly accessible the label is, and does activation magnitude carry label-related information? Block RMSNorm only; Q/K RMSNorm is a separate per-head operation.
Experiment pendingRMSNorm and magnitude probes

Runs on the same extraction as the layer probes (component captures at the default layers). No values are shown until the experiment has run.

Commands that produce this result
python scripts/latentlens/probe_rmsnorm.py --representations data/latentlens/reps/qwen3-8b-editlens_iclr
11Research

Attention vs MLP

At which transformation inside a block does linear separability change? Compared between residual-stream states that differ by exactly one update.
Experiment pendingComponent probes inside five blocks

Runs on the same extraction as the layer probes (component captures at the default layers). No values are shown until the experiment has run.

Commands that produce this result
python scripts/latentlens/probe_components.py --representations data/latentlens/reps/qwen3-8b-editlens_iclr