Jacob Garcia · Hugging Face Model Foundry
Fast Weight Time Machine Lab
Interactive fast-weight binding explorer. This showcase backs up the trained artifacts, measured evaluation, and complete runnable source.
Verified project card
# Fast-Weight Time Machine Fast-Weight Time Machine tests temporary variable binding with an explicit, sequence-local weight matrix. A controller receives key/value writes, produces a write strength, and accumulates outer products in a fast memory. A later query key reads the memory in one matrix-vector operation. The learned parameters stay fixed between examples; only the fast matrix changes inside each sequence. The historical anchor is Schmidhuber's 1992 [Learning to Control Fast-Weight Memories](https://doi.org/10.1162/NECO.1992.4.1.131), which described feedforward controllers producing context-dependent weight changes and included adaptive temporary-variable binding. This project is a modern outer-product interpretation tested against a similarly sized GRU. It does not claim to reproduce the original implementation or experiments exactly. ## Verified results The models trained on 16,000 sequences containing four writes and 12 distractors. Each held-out condition contained 4,000 new sequences. | Condition | Fast weights, 2,508 params | GRU, 3,050 params | | --- | ---: | ---: | | 4 bindings, 12 distractors | 100.00% | 37.40% | | 8 bindings, 12 distractors | 100.00% | 27.03% | | 4 bindings, 64 distractors | 100.00% | 31.15% | | 8 bindings, 64 distractors | 99.98% | 23.83% | This task is intentionally aligned with the fast-weight architecture: an explicit outer-product matrix can bind a key and value, while the GRU must compress all bindings into one recurrent vector. The comparison demonstrates the inductive bias; it is not a general claim that fast weights outperform GRUs on arbitrary sequences. ## Reproduce ```powershell uv run python projects/fast-weight-time-machine/train.py ```
Evaluation snapshot
{
"benchmark": "Temporary variable binding",
"training_examples": 16000,
"results": {
"fast_weight": {
"parameters": 2508,
"training_epochs": 20,
"four_pairs_12_distractors": {
"accuracy": 1.0,
"cross_entropy": 0.07871694164350629
},
"eight_pairs_12_distractors": {
"accuracy": 1.0,
"cross_entropy": 0.09045482566580176
},
"four_pairs_64_distractors": {
"accuracy": 1.0,
"cross_entropy": 0.07799763092771173
},
"eight_pairs_64_distractors": {
"accuracy": 0.99975,
"cross_entropy": 0.09526938293129206
}
},
"gru": {
"parameters": 3050,
"training_epochs": 50,
"four_pairs_12_distractors": {
"accuracy": 0.374,
"cross_entropy": 1.337967962026596
},
"eight_pairs_12_distractors": {
"accuracy": 0.27025,
"cross_entropy": 2.0145242735743523
},
"four_pairs_64_distractors": {
"accuracy": 0.3115,
"cross_entropy": 2.1618038713932037
},
"eight_pairs_64_distractors": {
"accuracy": 0.23825,
"cross_entropy": 2.5799436271190643
}
}
},
"training_history": {
"fast_weight": [
{
"variant": "fast_weight",
"epoch": 1,
"training_loss": 1.0295030027627945,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 2,
"training_loss": 0.030936013227180828,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 3,
"training_loss": 0.010835887420745123,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 4,
"training_loss": 0.006198568949623714,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 5,
"training_loss": 0.004087505533197333,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 6,
"training_loss": 0.002923791433521916,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 7,
"training_loss": 0.002205265745047539,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 8,
"training_loss": 0.0017280159813780633,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 9,
"training_loss": 0.0013937751747786053,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 10,
"training_loss": 0.001148996146632329,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 11,
"training_loss": 0.0009644523434828789,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 12,
"training_loss": 0.0008224385064317002,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 13,
"training_loss": 0.0007094257014123575,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 14,
"training_loss": 0.0006176355989679457,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 15,
"training_loss": 0.0005433608903475698,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 16,
"training_loss": 0.00048117371394284185,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 17,
"training_loss": 0.000429233452046497,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 18,
"training_loss": 0.0003849036833603999,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 19,
"training_loss": 0.00034724543285038736,
"validation_accuracy": 1.0
},
{
"variant": "fast_weight",
"epoch": 20,
"training_loss": 0.00031471504258834536,
"validation_accuracy": 1.0
}
],
"gru": [
{
"variant": "gru",
"epoch": 1,
"training_loss": 2.30176459796845,
"validation_accuracy": 0.148
},
{
"variant": "gru",
"epoch": 2,
"training_loss": 2.2673269188593306,
"validation_accuracy": 0.181
},
{
"variant": "gru",
"epoch": 3,
"training_loss": 2.20308005999005,
"validation_accuracy": 0.214
},
{
"variant": "gru",
"epoch": 4,
"training_loss": 2.073779126954457,
"validation_accuracy": 0.275
},
{
"variant": "gru",
"epoch": 5,
"training_loss": 1.9181506936512296,
"validation_accuracy": 0.324
},
{
"variant": "gru",
"epoch": 6,
"training_loss": 1.7754394440423875,
"validation_accuracy": 0.3175
},
{
"variant": "gru",
"epoch": 7,
"training_loss": 1.6792653269237943,
"validation_accuracy": 0.3405
},
{
"variant": "gru",
"epoch": 8,
"training_loss": 1.6136303837337191,
"validation_accuracy": 0.3415
},
{
"variant": "gru",
"epoch": 9,
"training_loss": 1.5580230913464985,
"validation_accuracy": 0.355
},
{
"variant": "gru",
"epoch": 10,
"training_loss": 1.5131285304114932,
"validation_accuracy": 0.349
},
{
"variant": "gru",
"epoch": 11,
"training_loss": 1.4721362969231984,
"validation_accuracy": 0.351
},
{
"variant": "gru",
"epoch": 12,
"training_loss": 1.4381263615593078,
"validation_accuracy": 0.371
},
{
"variant": "gru",
"epoch": 13,
"training_loss": 1.4131997729104662,
"validation_accuracy": 0.369
},
{
"variant": "gru",
"epoch": 14,
"training_loss": 1.3921367497671218,
"validation_accuracy": 0.366
},
{
"variant": "gru",
"epoch": 15,
"training_loss": 1.3752350996411036,
"validation_accuracy": 0.36
},
{
"variant": "gru",
"epoch": 16,
"training_loss": 1.3600610343236772,
"validation_accuracy": 0.375
},
{
"variant": "gru",
"epoch": 17,
"training_loss": 1.350090401513236,
"validation_accuracy": 0.3675
},
{
"variant": "gru",
"epoch": 18,
"training_loss": 1.3397450371394082,
"validation_accuracy": 0.375
},
{
"variant": "gru",
"epoch": 19,
"training_loss": 1.3330339874540056,
"validation_accuracy": 0.3735
},
{
"variant": "gru",
"epoch": 20,
"training_loss": 1.3186499731881278,
"validation_accuracy": 0.3745
},
{
"variant": "gru",
"epoch": 21,
"training_loss": 1.3137422989285181,
"validation_accuracy": 0.376
},
{
"variant": "gru",
"epoch": 22,
"training_loss": 1.3108702462816995,
"validation_accuracy": 0.3715
},
{
"variant": "gru",
"epoch": 23,
"training_loss": 1.3006561313356673,
"validation_accuracy": 0.3715
},
{
"variant": "gru",
"epoch": 24,
"training_loss": 1.291312327460637,
"validation_accuracy": 0.369
},
{
"variant": "gru",
"epoch": 25,
"training_loss": 1.2901321441408187,
"validation_accuracy": 0.374
},
{
"variant": "gru",
"epoch": 26,
"training_loss": 1.283520429853409,
"validation_accuracy": 0.374
},
{
"variant": "gru",
"epoch": 27,
"training_loss": 1.2761712244578771,
"validation_accuracy": 0.3745
},
{
"variant": "gru",
"epoch": 28,
"training_loss": 1.2719707943144298,
"validation_accuracy": 0.3785
},
{
"variant": "gru",
"epoch": 29,
"training_loss": 1.26511964343843,
"validation_accuracy": 0.375
},
{
"variant": "gru",
"epoch": 30,
"training_loss": 1.2616949081420898,
"validation_accuracy": 0.378
},
{
"variant": "gru",
"epoch": 31,
"training_loss": 1.2597541241418748,
"validation_accuracy": 0.3705
},
{
"variant": "gru",
"epoch": 32,
"training_loss": 1.2553172735940843,
"validation_accuracy": 0.3755
},
{
"variant": "gru",
"epoch": 33,
"training_loss": 1.250368530788119,
"validation_accuracy": 0.368
},
{
"variant": "gru",
"epoch": 34,
"training_loss": 1.2492833989007133,
"validation_accuracy": 0.3785
},
{
"variant": "gru",
"epoch": 35,
"training_loss": 1.2454108302555387,
"validation_accuracy": 0.379
},
{
"variant": "gru",
"epoch": 36,
"training_loss": 1.2419150027017745,
"validation_accuracy": 0.3835
},
{
"variant": "gru",
"epoch": 37,
"training_loss": 1.2380726621264504,
"validation_accuracy": 0.383
},
{
"variant": "gru",
"epoch": 38,
"training_loss": 1.2354776140243289,
"validation_accuracy": 0.3705
},
{
"variant": "gru",
"epoch": 39,
"training_loss": 1.2319395958431183,
"validation_accuracy": 0.3765
},
{
"variant": "gru",
"epoch": 40,
"training_loss": 1.2327304643297952,
"validation_accuracy": 0.3855
},
{
"variant": "gru",
"epoch": 41,
"training_loss": 1.2301562305480715,
"validation_accuracy": 0.378
},
{
"variant": "gru",
"epoch": 42,
"training_loss": 1.2272724480856032,
"validation_accuracy": 0.376
},
{
"variant": "gru",
"epoch": 43,
"training_loss": 1.223879613573589,
"validation_accuracy": 0.3785
},
{
"variant": "gru",
"epoch": 44,
"training_loss": 1.220575767850119,
"validation_accuracy": 0.379
},
{
"variant": "gru",
"epoch": 45,
"training_loss": 1.2178495970983354,
"validation_accuracy": 0.3805
},
{
"variant": "gru",
"epoch": 46,
"training_loss": 1.21574023980943,
"validation_accuracy": 0.371
},
{
"variant": "gru",
"epoch": 47,
"training_loss": 1.2135314487275624,
"validation_accuracy": 0.376
},
{
"variant": "gru",
"epoch": 48,
"training_loss": 1.2135897912676372,
"validation_accuracy": 0.3765
},
{
"variant": "gru",
"epoch": 49,
"training_loss": 1.210650818688529,
"validation_accuracy": 0.377
},
{
"variant": "gru",
"epoch": 50,
"training_loss": 1.209133299570235,
"validation_accuracy": 0.3735
}
]
}
}
Backed-up artifact tree
README.md__pycache__/app.cpython-311.pyc__pycache__/data.cpython-311.pyc__pycache__/model.cpython-311.pycapp.pyartifacts/fast-weight-time-machine/evaluation.jsonartifacts/fast-weight-time-machine/fast_weight.safetensorsartifacts/fast-weight-time-machine/gru.safetensorsdata.pydata/binding_test.npzmodel.pyrequirements.txttrain.py