A write up HERE on Hiroshi Yoshida (1876 - 1950) and above his wood block print Mt. Otensho from 'Twelve Subjects of the Japan Alps' (1926).
Image from HERE.
Adam Phillips is a Welsh psychoanalyst and essayist. I have never read anything by him, and until today had never heard of him. Yet this extended interview with him in The Paris Review makes me want to read his books.
It ends as follows:
Often one hears or reads accounts in which people will say, Well,
he may have treated his children, wives, friends terribly, but look at
the novels, the poems, the paintings. I think it’s a terrible equation.
Obviously one can’t choose to be, as it were, a good parent or a good
artist, but if the art legitimates cruelty, I think the art is not worth
having. People should be doing everything they can to be as kind as
possible and to enjoy each other’s company. Any art, any anything, that
helps us do that is worth having. But if it doesn’t, it isn’t.
An incredible calendar of minitaure scenes - made by Tatsuya Tanaka (since 2011). One for every day of the year. Each day there is a scene of daily life with miniature people interacting with objects that stand in for full-scale objects. HERE
Image Copyright Tanaka Tatsuya
Data Colada is an excellent blog published by three behaviourial scientists: Uri Simonsohn, Leif Nelson and Joe Simmons.
HERE is a wonderful solution to a common problem. The problem is for users of the open source statistical software called R. Millions of people rely on this software for statistical analysis - and although it was developed by academic statisticians, it is now very widely used.
Here is Data Colada's explanation of the problem:
R itself has some reproducibility problems..., but the big problem is its packages: the addon scripts that users install to enable R to do things like run meta-analyses, scrape the web, cluster standard errors, format numbers, etc. The problem is that packages are constantly being updated, and sometimes those updates are not backwards compatible. This means that the R code that you write and run today may no longer work in the (near or far) future because one of the packages your code relies on has been updated. But worse, R packages depend on other packages. Your code could break after a package you don't know you are using updates a function you have never even used.
An elegant solution to a real problem.
An excellent piece by Kiran Sidhu in The Guardian:
My younger self would never have understood Wilf. We would never have found each other. I would have asked: what can you learn by standing still? The philosopher Immanuel Kant lived his entire life in Königsberg and barely travelled outside the city. He would wake at 5am every day, he’d have lunch at the same restaurant at the same time every day, he’d go for a walk in the same park on the same route, every day. Perhaps a myopic existence for many, but a life no less full. I was beginning to appreciate lives lived like this, as if under a microscope. I began to understand the beauty of a microscopic life. The thing with life is that it’s not stagnant, we give meaning to whatever we choose to assign meaning to. It felt freeing to know this.
From a re-issue of a self-published book of photos taken by Paul Graham in the 1980s (HERE):
Along with many, if not most, of my friends, I was unemployed in the early 1980s. There were no jobs to speak of, and none to be found. Claiming unemployment benefit required that you attend the local Department of Employment office in person to sign paperwork in front of them confirming you were unemployed. As the number of people without work increased dramatically, the queues to ‘sign on’ lengthened, then lengthened again. Waiting times moved into hours, mornings, afternoons, then whole days. Scheduled interview appointments meant waiting beside a partitioned cubicle to be summoned, or for a staff member to randomly appear. It took a long while for me to realise precisely what it was I was witnessing here: these offices were where political ideology and citizens’ lives collided.
In early 1980’s Britain, workers became poker chips in a strategic game, as Margaret Thatcher enacted an agenda to reshape British society, abandoning the ‘postwar consensus’ and asserting the political class as the sole arbiter of power. Her ruling Conservative government forced through policies privatising state industries, engaging in direct confrontation with the unions, and redefining the UK’s social and economic priorities.
Thatcher triumphed on all fronts — major industries like steel, coal, and the railways were sold off to corporate interests at a discount, and union power was vanquished as an effective counterweight. The main weapon was the threat of mass unemployment. Never overtly stated, it was the bludgeon that hung over every negotiation, every worker, every regular life. The consequence of this radical agenda was literally millions of unemployed, sluiced into an unprepared Department of Employment, and Department of Social Services, who never got the memo, and weren’t supposed to, anyway.
Text & Image Copyright P. Graham
A brilliant write-up in The New Yorker on the work of Elisabeth Bik, who finds and exposes image manipulation in bio-medical journal articles. She has a remarkable talent for recognising matching portions of images, even when one of the images has been re-scaled or otherwise edited. HERE
Chris Killip is a British photographer. He was born in Douglas, Isle of Man in 1946, and after training as a hotel manager decided in 1964 to become a photographer. In the 1970s he took a series of photos in the north-east of England. HERE is an exhibition of these photos that was held at the the J. Paul Getty Museum in LA in 2017.
The photo above was taken in Skinningrove, a small fishing village. Killip says that “... it was very difficult to gain access to photograph there. Simon’s father had drowned in an incident at sea. They had this ritual where they came out and took Simon out to sea so that he wouldn’t become fearful of it. It’s very formal. He’s dressed very formally. I was on the boat and nobody spoke.”
Image copyright C. Killip.
An excellent, but now quite old, analysis in The Atlantic about how the New York Public Library has embraced digital innovation. Well written and engaging throughout.
I'm going to give you the conclusion to this article here to solve the tl;dr problem. There are two things the library has done to create such cool projects. First, I'm convinced the NYPL is succeeding online because of desire. The library's employees care about the digital aspects of their institution, and the institution supports their innovation. I mean this in the most fundamental way possible and as a damning critique of media companies. Second, the library sees its users as collaborators in improving the collections the library already has. While serving them online costs the library some money, they are creating value, too, by opening up conduits into the library for superusers.
Way back in 2011 Robin Sloan had a brilliant insight into what was good about good blogging (HERE). His idea is that the secret to blogging is to work in public, but reveal nothing.
But that can't be all of it. In my opinion another great thing to do is to find a way to share with readers what you have learnt through both public and private work. So it is also great to work in private, and reveal everything.
Recently I have published a new book with my long term collaborator Joss Langford (HERE). It explains pretty much everything that Joss and I have learnt over the years about how to build strategic research relationships with universities. The book has been written for practitioners, and is designed to help innovators develop more effective approaches to benefitting from early stage university research.
In my experience, I have found that the knowledge of how to successfully partner with universities was held by individuals as part of their personal skill set. They knew what to do, but it was hard to figure out how they did what they did. This knowledge was quite volatile as most companies didn’t have the equivalent of a Tech Transfer Office.
This means that the individuals who were given the task of extracting value from university projects often needed to learn through their own experience and mistakes. Even if you have been successful at running small PhD projects with universities it doesn’t mean that you will be successful at larger strategic collaborations. There are published case studies in the academic literature that people can read, but many of these are complicated by explanations of a theoretical framework. There are also edited volumes of case studies on successful academic-company partnerships, these explain the What, When and Why of the partnerships, but they do not generally address the How.
What Joss and I thought was missing was a cheap, plain English guide for practitioners. A PlayBook is by construction not academic, so our definition of success is that the books become well thumbed working tools - they are kept on the desks of practitioners, and end up full of highlighted passages and annotations.
A delightful piece HERE in The Guardian about the role the UK has played in recent decades revolutionising the quality of the World's football pitches. The top groundsmen are British, the best place to study 'turf science' is at Myerscough College in Preston, and the reserach knowledge base is centred at STRI (the Sports Turf Research Institute) in Bingley West Yorkshire, close to Bradford.
An excellent piece HERE in Granta by Jonathan Lee: How to Start a Novel.
The first sentence of a novel is an entryway. An open door. But how many
doors have you seen in your life that you’ve actually wanted to walk
through? It takes energy and trust to cross a threshold. You may have to
take your shoes off. Stop fidgeting with your phone. Prepare. Be alert.
Some clumsy fool with a baseball bat could be waiting on the other
side, ready to clobber you to death with his unsubtle story.
From the latest newsletter by Craigmod. Huh. (HERE)
Consider the washbasin, underloved or disregarded, but here, huh, lots of detail, understated love: delicate tile, two styles — square and tall — with subtle color offsets, well-worn texture of wooden wall beams (high-quality?), pleasing natural color of wall itself (earthen, no wallpaper), wooden window frame with wooden slats mirroring vertical tiles (old screen, endearing), non-fussy containers for soap and hand sanitizer, paper towels in acrylic box prepped and ready to be pulled on (so often not the case), and then the metal washbowl propped against the corner, rubber circular base, the circle of the base connecting to the circle of the soap dispenser to the circle of the fūrin bell hanging off to the left pulling the whole scene together, ending, finally, with the eyes finding the spigot, bottom right, gleaming and polished and clean and clearly adored in the simple way a spigot can be adored, late evening light, the whole scene seemingly acrostic, spelling out: come traveler, brush your teeth in this fine space.
Image & text Copyright Craig Mod 2021
HERE is a very well written and readable shorter piece by Rob Goodman and Jimmy Soni that explains the early work of Claude Shannon and his 'invention' of information theory.
A wonderful piece HERE by David Cornwell, on the deeply affectionate collaboration between his parents Jane and David - who wrote as John le Carré.
Very few, very wise people saw through them both, of whom the most recent and the most absolute is Richard Ovenden, who examined the papers my father loaned to the Bodleian library in Oxford and observed a “deep process of collaboration”. His analysis is a perfect match for my recollection: “A rhythm of working together that was incredibly efficient … a kind of cadence from manuscript, to typescript, to annotated and amended typescripts … with scissors and staplers being brought to bear … getting closer and closer to the final published version.”
Berelson & Steiner (1964) Human behavior: An inventory of scientific findings, is a classic of social science research. It offered a threefold grand summary of all of the social science research they had analysed: (1) Some do, some don't. (2) The differences aren't very great. (3) It's more complicated than that.
HERE is a summary article about a wonderful piece of social science research which stretches the third of the classes of study in this 3 fold summary. In short it shows that nonviolent protests are twice as likely to succeed as armed conflicts – and those which have engaged more than 3.5% of the population have never failed to bring about change.
A thought provoking and non-hysterical article by Nicholson Baker in the New York Magazine about how the Covid-19 virus came to be. The introductory paragraph summarises his story:
What happened was fairly simple, I’ve come to believe. It was an accident. A virus spent some time in a laboratory, and eventually it got out. SARS-CoV-2, the virus that causes COVID-19, began its existence inside a bat, then it learned how to infect people in a claustrophobic mine shaft, and then it was made more infectious in one or more laboratories, perhaps as part of a scientist’s well-intentioned but risky effort to create a broad-spectrum vaccine. SARS-2 was not designed as a biological weapon. But it was, I think, designed. Many thoughtful people dismiss this notion, and they may be right. They sincerely believe that the coronavirus arose naturally, “zoonotically,” from animals, without having been previously studied, or hybridized, or sluiced through cell cultures, or otherwise worked on by trained professionals. They hold that a bat, carrying a coronavirus, infected some other creature, perhaps a pangolin, and that the pangolin may have already been sick with a different coronavirus disease, and out of the conjunction and commingling of those two diseases within the pangolin, a new disease, highly infectious to humans, evolved. Or they hypothesize that two coronaviruses recombined in a bat, and this new virus spread to other bats, and then the bats infected a person directly — in a rural setting, perhaps — and that this person caused a simmering undetected outbreak of respiratory disease, which over a period of months or years evolved to become virulent and highly transmissible but was not noticed until it appeared in Wuhan.
Here is a wonderful piece on a dictionary of coastal irish words and expressions which have been collected by Manchán Magan.
A coastal Irish speaker, walking the beach at night, might have equally expected to hear stranach (the murmuring of water rushing from shore), or the whisper of caibleadh (distant spirit voices drifting in over the waves). They knew the ceist an taibhse
(the question for the ghost)—a riddle used to determine if someone they
met along the way was human or supernatural. Many words describe ways
of predicting the weather, or fishing fortunes, by paying attention to
birds or wind direction; to the sea’s sounds; or to the colors in a
The poet and advocate for poetry, Edward Hirsch, published a best-selling book in 1999 called How to Read a Poem and Fall in Love with Poetry. That book had an addendum of poetic terms, which he later expanded to be a book of its own - A Poet's Glossary, published in 2014. Here is Hirsch's entry for Filídh.
The filídh were a professional caste of poets in early Ireland who were often credited with the supernatural power of prophecy. The words fili and filídh are etymologically connected to “seer.” These poets, who were the successors of the druids and could practice divination, were magicians and lawgivers. They were the highest-ranking members of a group called the áes dána (literally, “the people of skill, craft”). In English, the word bard usually denotes a Celtic poet, but the filídh were in fact more aristocratic and enjoyed greater privileges than the bards. Their poetry is nonetheless called bardic, since they were entrusted with an oral tradition, the full knowledge of the tribe, which predated Christianity. Their education was daunting and they spent years at a dedicated school where poetry was studied as a craft. There were seven orders of filídh; the highest grade, the ollamh, studied for twelve years. The filídh practiced an elaborate form of syllabic poetry and mastered complex metrical forms, which employed both internal and end-rhymes, consonance, alliteration, and other devices of sound. They learned by heart at least 300 poetic meters, 250 primary stories, and 100 secondary stories. They recited traditional tales and topographical lore. They also served as crucial advisors and historical chroniclers, who remembered the genealogies of their patrons. They were so bound by tradition that there is little change in their work for the four centuries from 1250 to 1650. The poet Giolla Bríghde Mac Con Midhe explained in the thirteenth century:
If poetry were to be suppressed, my people,
if we were without history, without ancient lays,
forever, but the father of each man,
everyone will pass unheralded.
Ted Hughes said that the fili “was the curator and re-animator of the inner life which held the people together and made them what they were.”
In an unpublished manuscript essay of 1940, called The Philosophy of Breaking Through, the American ecologist Ed Ricketts (1897 - 1948) tried to address the experience that humans occasionally have of moments of transcendental insight, many of which lead to the creation of new things. Ricketts gave many different examples, but he considered the pattern he had identified as a human universal, which was often only missed due to obstacles we have in our own perceptions: obstacles created by the way we create routines to deal with life. The explicit articulation of this theme by the Californian poet Robinson Jeffers (1887 –1962), was of particular significance to Ricketts:
No doubt some few wise people know this merely through living. But many of us can achieve at least a clearer and more easily conveyed conscious expression of it through the spiritual motifs underlying literature, especially in poetry. . .
But in some of Jeffers’ poems, the thing is stated clearly, with full conscious recognition, and with that exact economy of words which we associate with scientific statements: ‘Humanity is the mold to break away from, the crust to break through, the coal to break into fire, . . . The atom to be split.’
It was this phrase ‘…the crust to break through’, taken from Jeffers’ allegorical narrative in free verse, Roan Stallion, that Ricketts’ used to describe an experience that he found otherwise difficult to put into words.
Ricketts’ essay describes a common pattern for these moments of breaking through. Often, there is a tension between what is, and what might be so, or between two opposing views: ‘The struggle is between opposing forces, each honest in its own right and without evasion but limited in scope and vision’. It was, Ricketts argued, through the resolution of these tensions, that a statistically rare break through, or epiphany happens. And at that point in time there is clarity of insight that may lead to artistic creation, a discovery, or an invention.
Little wonder, then, that at the very moment that an individual tries out something really new, they go beyond what their social group already knows. They break through. And when an individual reaches beyond what is already known, they do not ignore or negate what they have inherited and learned, they build on it.
In response to questions in 2016 about the awarding of a Nobel prize to Bob Dylan, the American music critic Greil Marcus has the following to say about Dylan's song “Like a Rolling Stone”:
I can’t listen to that song without feeling as if I’m hearing it for the first time. Every note in that song, every word, every inflection is a breakthrough. There is an energy that has come to bear on all the people in that room, all the people playing that song in that moment, that is taking them past themselves, taking them somewhere they’ve never been, somewhere they’ve never played, they’ve never sung with that kind of synchronicity. Every person playing off every other, and every person stepping into a realm where he’s never been before in terms of passion, expressiveness, intensity. Taking a form and pushing it to its absolute limit and pushing yourself past your limits. That’s what you hear over those six minutes.
Let’s put it this way: a song like that, a work of art like that, comes to no artist more than once. But it doesn’t necessarily come in anyone’s lifetime. We are lucky we are alive when that song can be played.
A great interview (HERE) with the Nobel prize winner Frank Wilczek - on his work in theoretical physics over the past 50 years.
Have you had to put these projects on hold because of the pandemic?
Oh, no. No! They’ve been fostered because of it. I’m not schlepping around and going to conferences and traveling. I mean, I’m eating better. I’ve lost 15 pounds, taken up juggling, doing exercise. It’s given me time to do creative thinking. In a way I’ve been going back to school, as if I were a graduate student. I want to learn more about machine learning. I’m going to be 70 in May, but I feel younger now than I’ve felt for many years.
At that time Macondo was a village of twenty adobe houses, built on the bank of the river of clear water that ran along a bed of polished stones, which were white and enormous, like prehistoric eggs. The world was so recent that many things lacked names, and in order to indicate them it was necessary to point.
One Hundred Years of Solitude.
HERE is a wonderful interview with Gabriel García Márquez in The Paris Review from 1981.
I had forgotten just how good this was. HERE is an article from 2007 by Laura Barton, about her rock'n'roll pilgrimage to Boston's Route 128 - the site and subject of Jonathan Richman's legendary song Roadrunner.
Roadrunner is one of the most magical songs in existence. It is a song
about what it means to be young, and behind the wheel of an automobile,
with the radio on and the night and the highway stretched out before
you. It is a paean to the modern world, to the urban landscape, to the
Plymouth Roadrunner car, to roadside restaurants, neon lights, suburbia,
the highway, the darkness, pine trees and supermarkets. As Greil Marcus
put it in his book Lipstick Traces: "Roadrunner was the most obvious
song in the world, and the strangest."
A superb piece in the New York Times (HERE) on how Neil Sheehan got hold of US Government papers on the Vietnam war, and published them as a front page story in 1971.
He also revealed that he had defied the explicit instructions of his confidential source, whom others later identified as Daniel Ellsberg, a former Defense Department analyst who had been a contributor to the secret history while working for the Rand Corporation. In 1969, Mr. Ellsberg had illicitly copied the entire report, hoping that making it public would hasten an end to a war he had come passionately to oppose.
Contrary to what is generally believed, Mr. Ellsberg never “gave” the papers to The Times, Mr. Sheehan emphatically said. Mr. Ellsberg told Mr. Sheehan that he could read them but not make copies. So Mr. Sheehan smuggled the papers out of the apartment in Cambridge, Mass., where Mr. Ellsberg had stashed them; then he copied them illicitly, just as Mr. Ellsberg had done, and took them to The Times.
Over the next two months, he strung Mr. Ellsberg along. He told him that his editors were deliberating about how best to present the material, and he professed to have been sidetracked by other assignments. In fact, he was holed up in a hotel room in midtown Manhattan with the documents and a rapidly expanding team of Times editors and reporters working feverishly toward publication.
This reads like a spy story.
A good essay HERE by John Naughton in The Observer on the rise of unregulated machine learning algorithms. It ends with a thought experiment:
Imagine what it would be like if we gave the pharmaceutical industry the leeway that we currently grant to tech companies. Any smart biochemist working for, say, AstraZeneca, could come up with a strikingly interesting new molecule for, say, curing Alzheimer’s. She would then run it past her boss, present the dramatic results of preliminary experiments to a lab seminar after which the company would put it on the market. You only have to think of the Thalidomide scandal to realise why we don’t allow that kind of thing. Yet it is exactly what the tech companies are able to do with algorithms that turn out to have serious downsides for society.
Here is a very cool project called Dynamicland, that is trying to change the way that humans collaborate with computers, physical artefacts and each other. Their mission is to, "...incubate a humane dynamic medium whose full power is accessible to all people".
I just got a copy of A Grammar of Typography: Classical Design in the Digital Age by Mark Argetsinger. It is a big book: 8 1/2 x 12 inches, 530 pages. Lavishly illustrated and beautifully designed and made.
What a way to start another year of lockdowns.
This Editorial in The Guardian (HERE) is about the scientific generosity of Prof. Zhang Yongzhen, who works for the Shanghai Public Health Clinical Centre. Prof Zhang was the first person to obtain a whole genome sequence of Sars-CoV-2 on 5th January 2020. He immediately began sharing his work and within days the Jenner Institute, Moderna, and BioNTech had all designed vaccines.
I believe we have a moral obligation to share good ideas. It is how we have learnt ourselves - all human culture is built on sharing. We just need to pass things on.
Annie Dillard in The Writing Life has this to say about open and enthusiastic sharing.
"... the impulse to keep to yourself what you have learned is not only shameful, it is destructive. Anything you do not give freely and abundantly becomes lost to you. You open your safe and find ashes".
From Uniform books, (HERE), is a scholarly exigesis of the TV programme Detectorists. It includes a Foreword by the programme's writer and lead actor, Mackenzie Crook:
I was detecting on my own… and dug down four inches to find an exquisite bronze hawking whistle. I took a few minutes to unclog the mud with a piece of straw, then held it to my lips and blew. The note that issued from the whistle was a ghost, a sound unheard for centuries, and the last person to hear that sound, that exact sound, was the person who dropped it just yards from where I was standing. And it wasn’t a faint, feeble ghost either: it was an urgent, piercing shrill that echoed across the field and back through time.
HERE is the web site of a new documentary, The Bit Player, on Claude Shannon. The film was commissioned by the IEEE Information Theory Society - and celebrates Shannon's ground breaking 1948 paper in the Bell System Technical Journal, and the impact that this work on communication theory has had on modern life.
One of my first posts on Data Deluge in 2009 was on 'evidence' as a general concept. I noted that lawyers are pretty hung up about evidence, and that the way that evidence in a scientific, engineering and statistical sense, interacts with the law was the subject of a very large and authorative US Federal government Handbook on how to use scientific evidence in law courts. This manual is now in its 3rd edition (HERE).
The manual has extensive chapters on science, statistics and engineering, and how
evidence from these different fields of endeavour should be considered
by lawyers and judges.
The chapter on How science works is by David Goodstein - it is excellent. It includes a well informed and irreverent description of what actually happens in science rather than what theoretically happens.
Optics: a soap bubble exhibiting interference colours.
Coloured mezzotint [?] by M. Rapine, c. 1883, after B. Desgoffe. (HERE).
Image above derived from an original image by S. Taneda of a progressive wave of concentric circles created by surface flow around a vibrating cylinder immersed halfway under a water surface. Original by S. Taneda, published in Fluid Dynamics Research, Vol.1, No.1 (1986) 1.
A wonderful story HERE about a record breaking image that was accidentally created by someone doing a Fine Art degree at the University of Hertfordshire. Regina Valkenborgh was interested in capturing images without modern technology, so she created a pinhole camera from a beer can lined with photographic paper. She left the camera in one of the University's telescopes in 2012. Eight years later it was removed and the image was developed. The image is a superposition of 2,953 arcs, each of which is a record of the Sun's path in the sky. The University claims it is "...the longest exposure image ever taken".
For printing and typography nerds, a piece on how the German designer Erik Spiekermann has been able to create a hybrid digital - letterpress printing process (HERE).
One of the two or three people I follow on Twitter is the American author Craig Mod.
Craig is based in Japan, and over the past 5 years he has developed an interesting set of email newsletters, and a distinctive approach to long-distance walks across Japan.
His most recent long walk has just finished (HERE): it was nearly 700 kilometers over five weeks. Most of that distance was along the Tokaido, the historic highway that connects Tokyo and Kyoto. During the walk he sent a daily email and photo (which were great), and invited readers to respond to his walk. HERE is his immediate post-walk summary, but he will no doubt write and publish a book on his experiences.
Bill Ryder-Jones is a musician who lives and works in my home town of West-Kirby. As a teenager he played in The Coral, leaving it to go solo in 2008. His latest album Yawny Yawn was released in 2019, it's a solo piano re-working of an earlier album called Yawn. He has a small studio in the town, also called Yawn, which is located in a quiet back street close to the sea front (he has his coffee breaks at the Aubergine - the cafe my son works at). His lyrics have some of the tenderness and strangeness of other bands from this part of the world - such as Echo and the Bunnymen. Examples include: 'There’s a fortune to be had from telling people you’re sad' and the following couplet: 'I remember what we did and when / and the smell of your breath / and even all the names of your dickhead friends'.
From a piece in The Quietus on him (HERE) the following:
Ryder-Jones holds great affection for West Kirby, a picturesque town near the coast, seven miles from Birkenhead. “If you turn left out of my mother’s house you can see the river Dee and North Wales,” he describes. “I love being at the water, I walk the long way to my studio every day so I can be by it. I walk from my house to the studio through a lovely park, then on to the beach and back around, it’s one of my favourite places in the world. It’s funny how I tried to escape it for all of my 20s before I gave in and said ‘this is where I’m meant to be’. I still live in my West Kirby bubble, and I live there mainly because no one cares what I do. If I go to Liverpool, people say ‘oh I like your music,’ whereas in West Kirby they just remember me from school. Or they wanna ask how The Coral are doing,” he laughs.
The Significance Filter, the Winner's Curse and the Need to Shrink.
E.W. van Zwet & E.A. Cator
September 22, 2020
The ‘significance filter’ refers to focusing exclusively on statistically significant results. Since frequentist properties such as unbiasedness and coverage are valid only before the data have been observed, there are no guarantees if we condition on significance. In fact, the significance filter leads to overestimation of the magnitude of the parameter, which has been called the ‘winner's curse’. It can also lead to undercoverage of the confidence interval. Moreover, these problems become more severe if the power is low. While these issues clearly deserve our attention, they have been studied only informally and mathematical results are lacking. Here we study them from the frequentist and the Bayesian perspective. We prove that the relative bias of the magnitude is a decreasing function of the power and that the usual confidence interval undercovers when the power is less than 50%. We conclude that failure to apply the appropriate amount of shrinkage can lead to misleading inferences.