Systems

Worse is better - Wikipedia

Richard Gabriel's uncomfortable argument that a simpler, uglier design that ships will beat the 'right thing' that never quite does.

From Wikipedia, the free encyclopedia

Unix philosophy - Wikipedia

Small programs that each do one thing and compose through text streams — the design philosophy that keeps outliving its obituaries.

Language Models are Few-Shot Learners

GPT-3's few-shot results made scale itself the headline: capability arriving from parameter count, not architecture novelty.

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Computer latency: 1977-2017

Dan Luu measures keystroke-to-screen latency across decades and finds a 1977 Apple II beating most modern machines, worse-is-better made empirical.

Conway's law - Wikipedia

Organizations ship their own communication structure as software architecture, so if you ignore the org chart the org chart wins anyway.

From Wikipedia, the free encyclopedia

Cloudflare Workers AI

Cloudflare's pitch for running model inference at the edge, next to your users instead of in one distant datacenter, and why latency is the product.

Run machine learning models, powered by serverless GPUs, on Cloudflare's global network.