Cremieux on FDA Reform, Fake Data, and the Columbia Admissions Hack

Episode Summary

Drs. Anish Koka and Anthony DiGiorgio sit down with independent science writer Cremieux, the pseudonymous Substack author known for auditing primary data and calling out flawed research. The conversation ranges from FDA drug approval reform (Cremieux proposes a two-pathway system separating safety approval from insurer-facing efficacy approval), the divide between well-evidenced GLP-1s and unregulated peptides like BPC-157, and the HRT black-box warning controversy, to the Columbia and NYU admissions data leaks that made national news and implicated NYC mayor Zohran Mamdani. The back half digs into research methodology: why observational studies rarely replicate RCT findings, how Cremieux debunked a widely-cited Jonathan Haidt social media/cognition study, why the "administrator growth" chart doctors love to cite is likely exaggerated, and how publication bias correction methods work.

Chapter Markers

00:00 Introduction and guest background

02:00 Why the pseudonym "Cremieux"

03:11 How he got into primary data analysis and writing

04:46 FDA drug approval reform — a two-pathway proposal

05:57 The peptide market: GLP-1s vs. unproven peptides like BPC-157

09:39 Should drug authority come from the FDA or "bubble up" from trusted voices?

11:25 Regulatory capture and ideas for FDA reform

16:10 HRT, the black box warning, and communicating risk to patients

21:40 Decentralizing medical decision-making

22:45 DEI in medicine and the Columbia/NYU admissions data leak

34:28 Proving discrimination: causal inference standards for university admissions

36:42 Busting myths from observational studies with multiverse analysis

40:29 Debunking the Jonathan Haidt social media/cognition study

46:46 Does the US healthcare system explain lower life expectancy?

49:56 Is the administrator-to-physician growth chart accurate?

54:22 How publication bias correction methods work

Co-Host Handles

@anish_koka and @drdigiorgio

Show Handle

@drsloungepod

Subscribe Links
Spotify: https://open.spotify.com/show/44vw8eirsKKnjgNIrdDvrR
Apple Podcasts: https://podcasts.apple.com/us/podcast/the-doctors-lounge/id1832097658
YouTube: https://www.youtube.com/@TheDoctorsLoungePod

Mr. Jordan Lasker X Handle / Substack

X handle: https://x.com/cremieuxrecueil

Substack: https://substack.com/@cremieux

🔗 Connect with the Hosts:

• Dutch Rojas on X

• Dr. Anthony DiGiorgio on X

• Dr. Anish Koka on X

• Dr. Dan Choi on X

• Dr. Sanat Dixit on X

Transcript

Anish Koka MD: we are live at the doctor's lounge. Normally we record on Thursdays, but today we are recording on Tuesday. So I hope you guys are tuning in because we've got a great guest. I guess today is one of the most widely read independent science writers operating now. some people think he also was one of most controversial. He publishes under the name Crémieux He has 36,000 substack subscribers and he's read and amplified with some of most influential figures in technology and finance. What I like that he does really which is kind of where I found him, I think a few years ago, is that he really goes back to the primary data and he actually checks citations. he's doing the that journalists I think should be doing. And a lot of us citizen journalists just find them so lacking. they're just kind of stenographers of whatever some credentialed person is saying and it's incredibly frustrating to see that so, he routinely, does data analysis and you can disagree with what he says, but he reruns the analysis. He does the hard work and arrives at whatever conclusions he's driving at. he recently a detailed rating Marty Makary self-described top 10 reforms and you're one the, Trump administration. He did not give them a good grade. we're going to talk to him about that. And we're also going to cover a number of different topics when it comes to GLP-1s, when it comes to what he thinks about getting GLP-1s to be more price-efficient, drug approval, pharmaceutical advertising. So it's going to be a wide-ranging topic. I'm really looking forward to it. But Mr. Cremu, welcome to the Doctors' Lounge. Thanks for coming on.

Cremieux: Thanks for having me on.

Anish Koka MD: So start with, Crémieux is not your real name, of course, but it is named after And in preparing this, I, of course read a little bit about who is because I wanted to be in on the joke that or in on it. But I figured I'd have you tell you tell us the story about how you, why you chose that.

Cremieux: Yeah. So When I describe my politics, I kinda tell people I'm a forty an 1848 liberal, and is among those who would be cast in that sort of way. It's a liberal, classical liberal, basically, what you would call it today. Crémieux was known for being kind of the Abraham Lincoln of France. He ended slavery in the French colonies, he allowed the persecuted Jews in North Africa to come to France to escape all the persecution there. and he did just a bunch of really wonderful things. He was pro-free trade, he was pro-freedom many, many ways. And I really admire the guys and so I was like, great, we'll just use this as my profile picture and all that.

Anish Koka MD: How you ⁓ into ⁓ what you're doing in terms of ⁓ primary analysis and writing about it and where did this all come from?

Cremieux: It came from arguing with people long ago when I was a kid and it just kind of kept going. but starting a Twitter came from friends coaxing me into just doing it because I was doing it behind the scenes. I was people and people and looking reading stuff online and like contacting authors and going out and going, Hey look, you have you had this error here or there ⁓ or You did this thing wrong or you could do this and you could learn a little more about that. Or like this paper contradicts this thing. Sorry, just like swallow own spit. down like the wrong pipe or whatever. And my friends were like, You should put all this stuff online. You should stop doing this kind of like in Discord chats or like on Telegram. You should actually go and like put your thoughts online and then people will like it. And then I was like, one day I go, okay, sure, I'll do that. in my first month I got something like sixty five hundred followers. Next month it like thirteen thousand and then it's just been growing from there.

Anish Koka MD: Wow, wow, it's like, could certainly write a book on how go viral because there's so many of your threads do go viral because your fingers does seem to be on the pulse of things. why don't start with the rewind a little bit to your thoughts on, you a lot of thoughts on the FDA, drug approvals, where are you a? kind of a meta sense of, the thousand foot view of you how do you think drug approvals should should go in this country?

Cremieux: I think they should be considerably liberalized, but they should also I think there should be a two pathways basically. I think the FDA should approve drugs for use without like any sort of push for insurance coverage. they should have a safety approval basically, so if the drug is safe, ⁓ people can go and get it if they want to pay for it. Totally up to them. It might be a drug that actually works, but if it's something that they want, then they can pursue it if they really think they need it. and then they should have one actually pushes toward insurers towards coverage ⁓ where they assess the And those are it's like s basically saying Have a pathway for people to get drugs after phase one. Have a pathway for people to get drugs after phase two if they want to pay for them out of their own pocket. And then also have a pathway where you do the phase three. Your company invests a lot of money in getting the big sample that's population representative and that can confirm or disconfirm if there is an like efficacy to the drug and then have the FDA approve like ⁓ substantially and then push insurers to actually cover the drug for the affected populations. I basically want choice, more freedom, and still the FDA's role. I do think there is a role for a regulator still.

Anthony DiGiorgio: So that essentially what the peptide market is doing now? I mean they they haven't cleared phase one, but yeah, yeah, it is. ⁓ but they haven't cleared phase one, but essentially, right, people can buy them if they want.

Anish Koka MD: So.

Cremieux: No, I think the peptide market's kind of a mess. Yeah. Yeah. Yeah, they can go out of pocket. They can do whatever they wish to do, which is I think totally fine. I encourage people to have that freedom, but I don't really encourage people to use those drugs per se. I do encourage people sometimes to use GLP ones because they're, well understood. the only one that like people use en masse that is not approved yet is Redatru Tide. And that seems be totally fine. It'll probably be approved relatively soon, beginning of next year probably. But the other stuff, like BPC 157 or CJC twelve ninety five with DAC, like all of that stuff is just really crazy and I don't really encourage use, but I do believe that people, if they want to access it, should be able to access it.

Anish Koka MD: So how do you arrive at so you're crystals? Sure. Why not? How you arrive at?

Anthony DiGiorgio: Healing crystals?

Cremieux: If if they want it, it's crazy.

Anish Koka MD: you're some distinctions here, right? You're saying you are, how are arriving at, what's your for that, hey look, GLP-1s, know, whatever, RETA, the triple peptide, Tirzepatide are you arriving the fact that those work, but all other peptides are not so great?

Cremieux: Yeah, so I think it just is the conciliance of evidence. For the GLP ones, Tirzepatide and Semaglutide there's just excellent evidence that they work, and the FDA has approved them on that basis. They've gone through the phase ones, twos, and threes. We can see them work. we have a million and one differ well, probably actually more than millions of anecdotes, honestly, if we're getting out of numbers. it's something like a fifth of Americans have tried a GLP one at this point, and it's And they definitely seem to work. Ever almost everybody who uses them sees considerable weight loss efficacy. and the like diabetics who use them see considerable improvements to their HBA1C. But we don't see really the same thing for many other drugs among this like peptide class. Like I've not there are plenty of actually approved wonderful peptide drugs. But when I talk about peptides, I'm usually meaning unapproved ones that are kind of on scant ⁓ clinical evidence. there are some among them that actually do have considerable clinical evidence, but they're used in ways that have never actually been tested very widely. So like thymosin alpha one has been used alongside a variety of therapies for people who have viral issues, and that has approvals in some countries. There's lots of clinical evidence that it works People use it as kind of like a general immune defen booster thing though, and that's bizarre and probably doesn't work. just because there's no real mechanistic way it should work, and there's really no clinical evidence to support it working like that. Plus, we don't have any sort of obvious anecdotes that would support it. you can see that melanotan, despite a lack of like much clinical evidence, melanotan too as an example, does make people much, much, much darker. And so we definitely know that it works for tanning. You can just see it. It's it goes without saying. But something Like BPC 157, where it's supposed to heal you or reduce chronic pain or do a million different things, nothing so obvious. And really no good mechanistic arguments that can outweigh the risks on those sorts of things. So while I do support people having access to it, I would probably I don't recommend it. just because there are the considerable mechanistic reasons for like indications of risk and then nothing really on the efficacy side.

Anish Koka MD: so that's why you still, but why does it have to be a governmental organization? Why does it, the FDA that have to have to show efficacy? Why, why can't we just go to trusted people? what does CREMU think about BPC and GLP ones and, kind of allow that to bubble up? have gone from, 6,000 to whatever 360,000 people, 36,000 sub-stack folks. So why, you don't think, just like cars don't have a, you're approved for use. You don't see a world where, folks bubble up were authoritative, not because they the credentials to be authoritative, but they were actually. And I'm saying that because, there's so gaming that can happen when it comes to the FDA, right? When you the FDA being the the maker of gold in terms of valuations and whatnot. Of course, there's a massive amount of lobbying political influence and you get stuff through the door that may not work well, which has been, Vinay Prasad's one of Vinay Prasad's strong arguments. it's a two part question. One, why do you think that we need an FDA? Why can't it be something else? Cause they're, inherently corruptible. And two, tell me your sense of what Vinay's argument has been, he's been the most vociferous probably a high level in terms of the FDA isn't working very well. Not that it's not working at all. I don't like the fact that people make to be his argument. That's not his argument. ⁓ I'm sure his argument is that look, we need to improve the hit rate a significant amount because right now we're just, we're not doing it well enough.

Cremieux: Yeah. So I think there could be, for example, competing regulatory agencies. it's just that we don't really have anything like that established and it's would be hard to do that. Transitioning to that would be very difficult. I would love to see something like that. I would love to see insurers able to do their own vetting of drugs and whatnot. It's just that we're not really set up for that and it's easier to change things by altering the FDA or working within existing structures rather than trying to go for something wholly new. there are plenty of great models out there for how this could work. Like Robin Hanson has talked a lot about for example merging health and life insurance and also just changing how those things interact with the medical industry, how they interact with approvals. I believe it could be worthwhile, for example, for an insurer to invest in an experimental treatment. Or allowing people to do like certain right to try things. there's all sorts of options out there for how, we could decentralize the FDA or go to multiple competing regulators. I just don't see them as very realistic. So I think we do have to continue having the FDA effectively. Now as for how the FDA really runs, I think it's just it's a mess. it does a lot of really good stuff. Like it's quite efficient when it comes to PDU FA. it's increasingly efficient when it comes to generic stuff, but it's still pretty like bad. There's a still a lot of regulatory capture. They're still pretty bad on devices. They should probably they could probably down regulate a lot of class threes to class twos and a lot of class twos to class ones. there's a of regulatory chaos right now. And it's unclear if an approved or like signed off on trial design is going to actually result in results that the FDA will actually accept and then approve the drug on the basis of. and it's a total mess. And I feel like we could do that a lot better just by, instituting those sort of basic competency checks we've had in prior administrations. Sometimes, I mean not all the time, there's always been issues. the FDA is often enough too liberal with approvals. That's part of what Vinay has pointed out with the medical reversal stuff. Like, have you guys read Vinay's paper on the oncology drugs that have been approved on the basis of endpoints besides overall survival, like minimal residual disease and whatnot?

Anish Koka MD: Yep.

Cremieux: And like just down the line, none of that seems to actually translate very little of it. Some of it does translate. But most of it doesn't translate into overall survival benefits. And it just becomes mess. Like they never should have been approved. They were those outcomes were not very good. We definitely need stronger vetting and like a move towards ⁓ greater statistical rigor and larger trials. but the FDA also makes a lot of those things difficult. One of my favorite reforms that I often mention is that in China, in order to get their trials going quicker, they made it so the IRB burdens were reduced by doing a singular approval if you have multiple sites in your trial. So in the US, you might have to get IRB approval at every site in your trial. So if you do one hospital, you gotta get approval here, another hospital, you gotta get approval here, another hospital, and so on and so forth. But in China, if one site approves, like the IRB, the ethics board, approves the trial, then everybody inherits the permission to Go and run the trial to have it go at all the sites, which is a brilliant reform. The FDA should probably adopt something like this. and they've been adopting some pretty neat stuff, especially under Vinay and Makary back when they were in, now that they're out. I don't really know what their the direction is a little less certain. but they had some cool ideas, and I would like to see them used. I'd like to see them both increase the rigor while increasing the pace of approvals. it's really hard to say though exactly. what all the issues are, where everything should go. I don't know.

Anthony DiGiorgio: There there is the option for centralized IRB in the US. a few studies I'm involved in ⁓ we have as well. But I think the bigger issue is the amount of paperwork that goes into an FDA trial or even just to getting a centralized IRB. I I've seen multiple years for a retrospective review through an IRB, which is ridiculous. and the amount of paperwork for an FDA trial, I mean, you have to go in and

Anish Koka MD: Yeah.

Anthony DiGiorgio: sign so many pieces of documentation, watch so many modules, navigate so many unfriendly websites. it's really painful. So I think the trial design, the IRB issue is one, but not the major issue, I think, holding up. I think that the China, China's really done a good job, not just the IRB issue, but in really reducing that friction of phase one data. And I know they're exporting a lot of that phase one data to us.

Cremieux: ⁓ yeah, of issues.

Anthony DiGiorgio: and pharma companies are increasingly investing in in phase one out of China.

Anish Koka MD: think the stance that you have on, on HRT the one that kind of identifies your general, here. And, a interesting one, in the sense that you weren't necessarily, well, tell me about you thought. So just give background, hormone replacement was something that medicine did because there was real world. data that suggested, my goodness, all these women menopause, they seem to be having started to have having more heart attacks. So maybe supplementing with hormone replacement, HRT, hormone therapy will be protective. And so then folks went and did actual study, the famous WHI study. in that study was ⁓ negative randomized control trial. this tension between observational data and then randomized control trial data. at point, became, OK, all HRT is bad. And nobody really prescribed for decades. And now over last 10 years, the usual thing with RCTs, anyone that's in the histomology space knows there's no RCTs perfect. No RCT ever answers every question. There's always going to be some subgroup that you're like, well, maybe it works in that subgroup. Right. so now, we're, and this FDA for FDA, I don't know who exactly is super pro ⁓ HRT, but it seems like they've really, they've panels. to try to get folks understanding that HRT is good. So that's the context, that's the background. What were your thoughts on this all played out?

Cremieux: Yeah. So I generally support availability of HRT. It's quite good. Like it a lot of women when they take it, they feel so much better. Like, and that's good. Having patients feel better is like pretty good outcome. the issue that I took with what the FDA did on HRT, they increased the availability, kind of. They didn't really change the availability per se. They did something to patient perceptions, which was removing the black box warning. They remove the warning saying that it causes cancer. But it does cause cancer. and there's like fairly evidence that it causes cancer, so it's like they probably shouldn't have removed the warning. But the idea was to get more women to use it so they would feel better. And they do feel better on it, so you have some benefit. Like you should probably take it anyway. It'll make you feel better, even if there is a slight increase in the risk of cancer. But what don't like is that They remove the warning entirely when it was an accurate warning. I don't feel like the FDA should ever do that. I feel like they should keep accurate warnings, and if their goal is to increase usage of a treatment, then just tell people it's still worth it. It's worthwhile. Like you will feel better on this thing, quite likely. Talk to your doctor, it might work for you. They might have experience of patients who felt a ton better. Yeah.

Anish Koka MD: So would say, look, this the risk of cancer by approximately X based on the randomized control trials we have. But you will feel You decide what you want to do. ⁓ But again, when I there and have these conversations, a lot of it is, how do you contextualize that for patients? Because ⁓ the numbers, ⁓ Cremio, pretty small, It's like.

Cremieux: Exactly.

Anish Koka MD: WHO study, it's like eight additional cases where 10,000 women per year. So and course, that's like a general number. Right. So that's a general number that you then have to try to apply to the 50 year old, 60 year old. the 60 or the 50 year old that had a history of uterine mom with uterine cancer. mean, it becomes quite challenging. So, I think you may have the.

Cremieux: Yeah. Becomes a mess.

Anish Koka MD: There's a certain classifications that are okay with some level of uncertainty, right? It's like yes testosterone may increase my risk of dying from a heart attack or an arrhythmia, but I want to do it and I understand that right? A lot of folks are gonna have trouble with that.

Cremieux: ⁓ yeah. Yeah. it it's troubling for doctors. You it does suck having to explain to a patient, like, well, you have a slight increase in this risk here, you might get breast cancer, but I still am gonna recommend the for you or like, yeah, testosterone will help with your depression, or you'll be more wakeful, or you'll have more energy. but also your lipids will be a little harder to control and you'll have issues not you know. It does get bothersome like that.

Anthony DiGiorgio: But that's that's what being a doctor is. I mean, that that's why I think AI is still not replacing us because those nuanced conversations ⁓ like ninety-nine percent of the challenge in our job, right? I can I do the same thing with every time I consent a patient for surgery. Yeah, you might get better, you might get worse, it might be a complication, it might not be a complication, you might not get better even though the surgery was perfect. Right. So you have to have these nuanced convers and patients appreciate that and I think they get it. So is there a world where

Anish Koka MD: You

Anthony DiGiorgio: I mean, to take a step back and like look at almost a Hayekian view, with decentralized decision making, is there a world where we decentralize those decisions, let it happen on the individual patient doctor level, without the heavy hand of government ⁓ with its on the scale?

Cremieux: Yeah. I mean to an extent that's what we have. I mean you are having those patients all the time, or those conversations all the time. we can have more.

Anish Koka MD: Well, yeah. Yeah. The, the, issue is, that the government seems to be, bipolar, right? And it depends on what administration you have. So you have, X number of years and it's not just the government. It's just academic, academia or whatnot, right? It goes with what's vogue. So patients come to you not with this like blank slate. Patients are coming to you for HRT or patients are going for testosterone. And it's because they've heard about it, through culturally. And in this case, of course, now you have this alignment of certain academics with the government that's kind of pushing. HRT, maybe because, certain members of ⁓ HHS very pro, ⁓ hormone replacement. ⁓ but that an interesting dynamic. Yes. Right. Exactly. The I clearly, you know, the mayor of New York needs to get on some TRT. We watched him, you know, struggle ⁓ to bench whatever he benched. ⁓ we got to we got get him on some TRT. ⁓

Cremieux: Yeah. HR T and T R T. Ha ha ha ha ha ha. ⁓ it was bad. He couldn't do one plate.

Anish Koka MD: right. ⁓ moving on to that is germane there was just a hearing that in Congress today on DEI in medicine, specifically in medical education and physician training. and speaking about Momdani. Um, you, you were, you were basically in the New York times because of some data that you got ahold of from Columbia. Columbia is an elite school in New York and you were, you acquired, um, admissions data, with detailed analysis of who was getting admitted, what their scores were. can you talk about that whole period of time? It was pretty fascinating. You became like a national news story. and have influenced story obviously very influential in political sphere when it came the elections.

Cremieux: Yeah. Yeah. So this all started a little while before with the hack of NYU. So NYU, you know, close by. it's like a It feels like its campus is like a community college in New York. It's not not a good campus. And it's priced extremely high for what it is. It's a very weird school. but NYU got hacked by the same hacker who gave me the Columbia dilator. And looked through the data, promoted a lot of their findings, went on the Charlie Kirk show and talked about it with Charlie. That was first time talking to Charlie. and People loved it. And the hacker, I started talking to them. and then they were like, hey, I have some new data, and I don't want to deal with all the press this time, so why don't you just take the data, give it to whoever you like, give it to people in government, give it to people in press, release your own findings. Tell the world happened at or what's going on at Columbia. And so I did. And the hackers' motivation was to affirmative action, because it is illegal. The ⁓ twenty twenty three SFFA Students Fair Admissions versus Harvard decision declared that affirmative action is a form of like, illegal racial discrimination done by universities. And if they want to keep their public funding or any access to publicly backed loans, they should have to abide by that law. They should have to, treat students fairly regardless of their race. And I found that they weren't. both NYU and Columbia are continuing to practice affirmative action long after the decision. And very likely that the Columbia. has continued actually practicing affirmative action. and so this was just pretty neat. And then one day the hacker goes, hey, Memdani's in here. And I go. How interesting. So we look, and Mamdani, as it turned out, lied about his race in order to apply to Colombia. his father was an African American or African studies professor, or maybe it's African American studies. I have to look at the d the which one it was. But his father from Uganda is an African studies professor at Colombia. ⁓ And so already had a considerable advantage in getting in. But he took it further and he said he was black. He didn't say was African American. and at the time he also wasn't American, he didn't have a citizenship yet. he certainly isn't black, and he knew the difference, and he had the option to fill out any sort of complex categorization of himself that he wanted, but he didn't fill out anything. He left the like fill in the blank empty. unlike some other students who put in weird stuff like Swiss German. Like just people put in whatever they want, and Mamdani elected not to. He elected to instead misrepresent that he was black and try to get in in this way by taking advantage of the affirmative action they have going on there. believe that whatever happened at the interview stage might have ⁓ dissuaded the department from actually letting him in because it's kind of obvious he's not black, so very funny thing happened there. But the basic gist of it is Affirmative action, everybody, including Mamdani, knew it was going on and tried to take advantage of it, and it's still going on, very likely to today. The demographics can't be what they are, Because it's just not really possible given all the test scores and the GPAs and everything like that. we'll have data on this relatively soon because there's a thing called iPads, the integrated post-secondary educational data system, that the US government, after I wrote this piece, decided to reform. So I basically gave the Department of Education a outline for how to reform this big thing that the NCES, the basically a statistics bureau within the Department of Education, maintains that gathers all the university secondary education sort of data. data sorry, university education data like every year in. I gave them way to reform it and then they put out a memo last year saying they were gonna reform it. And so this admission cycle we be getting expansive data that'll allow us to see if every university or if any unit particular university is still practicing affirmative action violation of the law. And that's basically where the story is right now.

Anthony DiGiorgio: So in the nineteen nineties, California actually passed a law outlawing race-based admissions. and every school in California had to abide by this, right? So that's actually been the law in California for way longer than the Students for Fair Admission. there were some leaders from California institutions in Congress today testifying that they have not been discriminating based on race since that was put effect in the nineties.

Anish Koka MD: ⁓ I think.

Anthony DiGiorgio: we're if Chris Congressman Cromot was there, what would have asked them? and what sort of proof do you think you could have brought to the table to say that, maybe there are some institutions in California which still do discriminate?

Cremieux: Yeah. So I would have asked them for their applicable data. I would have just said, give us the raw data, let us see it. I'd like to see where the qualifications that you gather from students stand relative to the classes you admit. Kinda the main way we'll figure out how people are doing affirmative action in the future is we will look at all the qualifications that they gather and all the data that's available to the different admissions departments, and we'll how that should stack up under the sorts of like ⁓ criteria systems that like they use. If they say that use X, Y, and Z different indicators in order to judge students, then they should be able to supply the data to show us that that's how they did the selection. it should be very little effort, actually. But Mysteriously, whenever are asked to do this, they don't tend to be very forthcoming with the data. Harvard in SF V Harvard case did eventually get asked to bring forward the data and they showed their admissions models and like what they used to determine how students got in. it was very clear, like, wow, unless you are putting race in as one of the criteria for admissions, then there's no way, based the data that your admissions department has gathered, for them have selected the student bodies they did. It basically becomes You have to affirm that you're using a given admissions model. And that's what I would ask them. I would ask them, what are your models? What are the criteria? Are you just being wishy washy in a way that leads to some like massive discrepancies between what should be happening under fair admissions. Give us a model that works. And they don't, if they can't affirmatively justify it, then something is up. And I feel like that's what we should transition towards is they should have to affirmatively justify everything. They should be able to ahead of admissions cycle show us what they're going to do and then admit accordingly. I feel like that's really the only fair way to do it.

Anthony DiGiorgio: Anisha, you're trying to talk, you're muted.

Cremieux: ⁓ yeah.

Anish Koka MD: Is it libertarian to allow universities do what want to do and select the body composition in way ⁓ they want? Of ⁓ that butts up against rights law. But in perfect world, I gather would be okay with discriminating or ⁓ not as they want. Just don't do with federal dollars. that?

Cremieux: Exactly. Yeah. If they're gonna use federal funds for it, if they're gonna use federal lands, if they're gonna request federal grants for their professors and whatnot, and those that's going to be split off to fund their departments, then you have to admit fairly. but if they're not going to take any money out of the public purse, I see too much of an issue with them going and doing whatever they like. But That's probably not ideal. if they were to cut themselves off from the federal government, they'd for one, they'd have a hard time running themselves. they'd advantage considerably the universities that haven't cut themselves off, and they'd probably be out competed in the marketplace of research and ⁓ prestige and everything else. So it's there's really way for them to actually go about and do that. If they tried, they probably just wouldn't work out too well for them. But I'd love for them to give it a go. If they really want to continue discriminating, they can just say it and then go do it. ⁓ just not with my money.

Anish Koka MD: Now we know, we know that it's happening though. ⁓ course we we know it's been happening for a while. this is now your number six of, the GOP ⁓ being in power yet not a single has faced any significant sanction. Why do you think that is?

Cremieux: Yeah. ⁓ if I had to chalk it up to anything right now, be that the administration is kind of chaotic and they don't have enough people actually in. if they had enough people to staff everything sufficiently, they'd have a lot more people employed in the federal government right now, in appointee roles and whatnot. But they just kinda don't. There's not enough. They have hiring issues, they have issues with competence and they have competing goals as well. So like some people want to basically take knowledge we have and then go, hey, we know you're discriminating, we're gonna come after you, and then try and negotiate some sort of different settlement. Like the administration has reached settlements with ⁓ different universities that we know there are issues with, and this precludes them from going back and like, exacting something out like know, getting another pound of flesh from Columbia or whatever, because they've made a deal and the deal is this or that and it was negotiated by these people. But these other people in the Avenue might want to do something else and it's just not happening because not all coordinating. if they really wanted to be like to go hard on this, they would push the iPads reform they've already committed to and who knows they'll actually end up doing it. We'll have to see when the data is supposed to come around, I think around August. and then they would use that immediately. They would take it to the DOJ and they'd go, Hey, we still have evidence that these people, these universities, are discriminating. And then they would prosecute immediately. But shall if they even have the people to do that. If is capable of like throwing together the analyses at us you know, stand up in court and be able to show Center everything else and you'll be able actually make a case. I just don't know if they can. It's kinda difficult. it is a lot of work and I don't know if they have enough workers really.

Anish Koka MD: interesting.

Anthony DiGiorgio: I think one of the one of the things you're you're best at articulating is the shortcomings causal inference ⁓ in research, right? you really showing the the burden of proof and how observational studies tend to. But here what you're talking about is essentially causal inference, on admissions data. so I think the the point that you're trying to make and is that when it does come out it has to be quite strong. And it often is that these things go on. But speaking to your bigger issue with these kind of observational studies, how do you get to ⁓ a level of proof that you think hold in a court of

Cremieux: What I think we should do there is force universities to act in the affirmative. I think we should force them be the ones like proving that they didn't discriminate, basically. ⁓ need to move to that sort of standard if we're gonna continue to allow them, ⁓ public funding. don't really see any other way in which to make it possible to properly prosecute the when they are acting up. So what we should do basically ⁓ say you have to submit data that you obtained on your patient on your students, and then you have to show that you a given admissions model that you've committed to ahead of time and that the results of that model. model are what actually result in in your student body. if they can't do that, then they be just become ineligible. this would be a big change. but I it's realistically the only way to actually implement any sort of change with meaningful pressure on the universities. ⁓ They are just crafty. They will just continue to lie, they will just continue to mislead, and there's really no standard that really works, unfortunately. ⁓ I think it's what have to transition to.

Anthony DiGiorgio: Do you have a speaking of you know causal inference and your critique of observational studies, do you have any favorites to stand? I have some favorites of yours that you've pointed out that have gone sort of counterintuitive against the grain, but what are some of your favorite, ⁓ myths that you've myths, excuse me, that you've based on observational poor observational studies?

Cremieux: man, so I don't know if I have any particular favorites, but I do have a paper that I have been preparing for a little while. I have a multiverse analysis running right now in the background here. actually for that paper. And what I is I look through every little estimator I can, like target trial emulation studies. I use their estimators. I use all sorts of different things, ⁓ and I look to see Can I recover from observational data the estimates from these ⁓ large well-known RCTs? And do all sorts of matching and stuff and I find no. It's not possible go from observational data to the RCT results. and that's just like a very broad finding. It's not on any particular one. ⁓ I actually just had the AI just pushed the PDF up on the side of the screen here. Great. Now it's actually done. Breaking data. Yeah, basically it just like doesn't work. so I've done a very comprehensive evaluation of it. ⁓ I posted some of the graphs from it on Twitter. it's kind of like a little spoiler of, this is what's coming. But I just think in general, observational methods are so, so incredibly poor. especially in the area of nutrition, where my people

Anthony DiGiorgio: Breaking data on the doctor's lounge.

Cremieux: I feel like every observational finding in nutrition is just the sum of all the biases out there. And Ioannidas once said something kinda like that. And it's it's just kinda stuck with me. And ever since I heard that I've been like, wow, there's really nothing else that explains this nutrition stuff. It's not about like does eating more wheat actually like make you a Superman? Nothing like that. It's about just who's eating it. Yeah, I feel like that's how everything has shown up there. It all just collapses. It's gotten it's so cokey. And it's amazing that we even have like just yesterday there was a what was it, a new New York Times headline about eating spicy foods and living longer? And it's just like why? Why even look at this? It's not it definitely isn't real. It's de definitely just like people who like spicy food.

Anish Koka MD: Ha ha ha.

Anthony DiGiorgio: Drives clicks.

Cremieux: Live longer. It could it could be like, it just happened to be that these like Chinese people are eating Sichuan or something like that. It's like they live longer. Go okay, great. What this tell us? Should we all eat Sichuan? And the answer is probably not. It's not gonna do anything for you. But that's just how it is. ⁓ Just jumbled mess.

Anish Koka MD: You But my, my, my, my, one of my favorites was, this, this was ties into the affirmative action because what's problem really affirmative action? yes, it's unfair. I mean, but one of the really things we, don't think is talked about is that, okay, we've affirmative action that's been going on for now since what? 1969 since Martin Luther King was assassinated. probably around there, there's been some sort of firm action that's been happening. So now you have like this massive class of folks that have been pushed ahead, because of their ability, but pushed ahead because of their, checking a box some category, right? And these, and a lot of these folks are doing, ⁓ you know, are academics, researchers, right? Because, you know, you can... You can push people ahead, but if they have to go like run a Chinese restaurant, as an example, like they're going to fail. mean, they're not going to be able to do it because it's like, well, I've showed up and I, I am, I have checked off this box, this box and this box there. Therefore I will be able to run this business and know how to do it. It's like, no, you are going to fail, guess they may not fail. They may not fail in the warm welcoming. Sorry, Anthony cocoon world of, academia where, where, they can punt, they keep pumping out.

Cremieux: Yeah.

Anish Koka MD: interesting papers on spicy food living longer and stuff. so ⁓ segue, I'm not talking about Dr. DiGiorgio Dr. DiGiorgio is an exception. But ⁓ but had great post. ⁓ You had a great post on an article Jonathan Haidt published. The tweet is powerful, new longitudinal study finds that adolescents who increase their social media

Cremieux: Yeah. Yeah.

Anthony DiGiorgio: I like my warm cocoon here in academia.

Anish Koka MD: over a two year period showed lower cognitive performance compared to those who did not increase. And so, it just, you read that and I'm sure a bunch of us have the same reaction. It's like, you know, I roll, it's like, oh, okay, just screen time is killing everyone. It's like the same thing that happened with like video games and like Al and Tipper Gore, right? Back in, the eighties, it's like, oh, everyone is getting more violent because of these violent video games. it just feels right. And then there's, then there's of course some, it happens, yeah.

Cremieux: ⁓ Yeah. ⁓ god.

Anthony DiGiorgio: I thought it was the rap music.

Anish Koka MD: And of course there's happy group of academics that are like, let us make this empiric, right? tell us a little bit about what did. of us, look at that, we roll our eyes, we're like, okay, we mark Jonathan Haidt as, all right, this guy ⁓ may not be most robust ⁓ of people when says things, and we move on. You did something more, so tell us what you did with that tweet.

Cremieux: Ha ha. Yeah. So when I saw that, I was like I bullshit. ⁓ I just ⁓ believe it. don't believe like in a lot of things affecting cognitive ability or like a bunch of other things. Like I think in general should assume everything affects everything else is gonna have a smaller effect when you really get down to it and like look at it very rigorously. Like I found over my like long

Anish Koka MD: Ha ha.

Cremieux: of like looking into different data sets, every time that I try and look at something, I find like a null. I just don't get anything interesting. and I use all like the latest methods and everything. I use like, crazy new event study estimators and I use all sorts of like interesting ⁓ stuff. And I like all sorts of triangulation exercises. And I just find nothing all the time. And so I become of sour the idea of people finding all these crazy results. Whenever I see them I'm just like you probably didn't get that Through any sort of rigorous methods. And I look, and as it turns out, they are usually not very rigorous. They are usually something very basic or wrong or the estimand, ⁓ the theoretical thing that they think they're estimating, isn't right. It's not what corresponds to the verbal, like the sentence they've laid out. They go, ⁓ we found X causes Y. And it's like. You didn't estimate if X causes Y, actually. You estimated something that is like X is related to Y, maybe. it could be like a proxy like X is related to Z and that might transfer to Y, and we don't really know. And I just got kind of bothered. I looked this paper and I go, ⁓ look, a friend of mine, he has access to the data set. Well I'm gonna send my friend some code. And I'm gonna him. Re-estimate these because the authors just correlated X and Y, and X and Y were related. And just it was curious to me. I noticed only looked at four out of the nine tests available in this data set. And I was like, why they exclude the other ones? I honestly couldn't So I said, of all, look all the data, rerun their methods, but look at everything. Two, Estimate it properly. Don't look at just does X cause Y. Look at does X at time one affect Y at time two? for some reason the authors didn't do that. They had the data to do it and they just didn't do it. And I'm like Why didn't you do it? is it because you wouldn't have found what you wanted to find, which was that social media is like killing the kids' brains? And it turns that's that was probably the case. When you actually look at if X at times one has any effects at on Y at time or whatever, nothing. There's nothing there. ⁓ I at all of this, it turns out that on the test they didn't look at, ⁓ the results were much weaker. In some of the coefficients were not even the right directions. and if you didn't, ⁓ cut down the sample in very weird ways, you also get a stronger result. Some of their results evaporated just by using the full sample, which they never really justified using a smaller portion of it in the first place. It's just very bizarre. and then I looked, well, what if I compare siblings who use different amounts of social media? is the one who uses more social media going to be the one that's, dumber? And it's like, no, actually it turns out that there's really no effect. so look over time, as you should do, as the author should have done, nothing. Look, siblings, nothing. use the whole set, much less ⁓ of result than they got. And I was just thinking to myself, like, wow, they really are either just like lazy or dishonest, or there's something wrong with them. And the article still stands. The article is out there. People are talking about it. Jonathan Haidt is talking about it still. people like Jonathan Hyde, align with Jonathan Hayde. They are like looking at this article as a piece of evidence in favor of their theory that social media is pernicious and harmful and it hurts the kids. And isn't. Like it just doesn't show up in the data. It's not actually supported at all when you actually look into it. In fact, just as supported as like saying social media affects like your African admixture. And I looked at this and I was like, Yeah, to the extent that, it says it makes you like less intelligent, it also says it like makes you like blacker or something like that. It makes you like have a different family background. It makes you have changes your genes. It changes your brain volume. And it's just like all of these are things that cannot be affected. and you can use past measures and it affects those past measures. And it's like, well obviously that's wrong. that's a negative control. It's temporally prior to the thing you're claiming is affecting stuff. But they check any of this. They just went lazy. X causes Y based on the correlation. ⁓ And Jonathan was happy to repeat it because he hates Social media.

Anthony DiGiorgio: All right, two of my favorites, if we could

Cremieux: Ha ha ha.

Anthony DiGiorgio: go hide ANISH

Anish Koka MD: No, no, go ahead, go ahead.

Anthony DiGiorgio: so I was gonna say if somebody comes and tells you that Americans live because of our healthcare system, that's the reason we have a lower life expectancy. is that true? Is that a randomized trial or is that a wet street causes rain? And then the second one, that I actually I and many of my colleagues guilty of of posting I think has to do with diagnostic measurement ⁓ is does the growth in really outpace growth of physicians as much as that famous chart shows?

Cremieux: Yeah. So on the first one, American life expectancy. It doesn't actually lag behind because of our healthcare system. Where the healthcare system can actually affect people's lifespans, we tend to do better. Like with cancer survival rates, we have a lot better results than pretty much all of Europe. We do better than the rest of the developed world when it comes to the that doctors in the US can actually touch. We spend money, we have newer drugs, we give out drugs more readily. Like we do stuff that allows our patients to live longer. Where we lag is diseases of affluence. We're fatter. We are more likely to have diabetes. We're more likely to manage it poorly because the patients like don't adhere to their medications well. we're more likely to get into car crashes. We're more likely to shoot each other. It's all stuff that's like Americans violent and reckless, and it's like, well yeah, violent, reckless, they eat a lot. Cool. If you get rid of that, then America can actually, live longer than a lot of its European peers. It can live longer even than some East Asian countries. which is pretty wild. ⁓ but if you

Anthony DiGiorgio: And statistically if you're if you're counting things like violence and car accidents that affect younger folks, that's gonna drag down life expectancy greater than, the difference in cancer survival.

Cremieux: Yeah. Because that mostly affects older people. Like it's massively, massively age related. And mean, caused by a part of the aging process. Just a But America doesn't really lag when you get down to like the things that we can actually throw money at, because we do throw money at everything, and we are pretty good at we have like more competent doctors when it comes to screenings, we have more competent doctors when it comes to giving out the latest treatments, and just seems to work. But If you overlook those things and you just look at America spends more, it looks like America's pretty bad. But just cause we are less healthy and more violent. well. We can fix things. ⁓ We have self cars coming. We have self driving cars in Get right now. You can go ⁓ drive a Tesla and they self drive pretty damn well. Take ⁓ on the highway and they're like, You sit back. It's great. and if Americans had that and they were using it appropriately, could theoretically cut down our delta with ⁓ Western Europe and that would eliminate about ten percent of it. Like that's it's huge. It's just it's incredible how far behind we are in terms of ⁓ how you how unsafe are behind the wheel. It's it's a bunch of ridiculous things like that. But we're not really lagging when it comes to life expectancy if you account for all the facts that stand before us like that. and Romy the other

Anthony DiGiorgio: The famous chart showing the growth in administrators over the growth in physicians. We all feel that as doctors. We look back five years ago and there was one person, now there's three people in that administrative role. So why is that chart wrong? And is there a better way to look at it?

Cremieux: ⁓ yes, that's right. Yeah. Yeah. I mean so there has been considerable administrative growth in healthcare. But that chart is made by ⁓ Walworth and Himmelstein and they For one, they don't reveal their methods anymore. They used to say like say like, ⁓ these were our sources, here's this or that. And now they kinda don't. They just stopped. And it's very bizarre. and when people question the like, two thousand to X growth in admin, they don't even justify themselves anymore. They just say, ⁓ it's, it's legit, trust us. And it's like, well, I I don't trust you. When I look at the actual underlying data, I see like, you know, a fifty percent increase, not a two thousand times increase. ⁓ It's just orders of magnitude difference. There's very considerable administrative growth in healthcare, and like we everybody feels it. ⁓ you do to interface with like a million different people and you have like a huge coding department and everything feels absurd all the But nevertheless, it's like that increase corresponds to twice as many people, not two thousand times more people. ⁓ If it really were to be as as the charts say, then It wouldn't even be every other person in healthcare is an admin, it would be almost everybody in healthcare is an admin. It just doesn't make any sense. It doesn't actually work out. If we use the CMS's data, if we use very broad or narrow definitions, we still see this increase over time. That's considerable, just not that that huge. That chart though just goes wild. The one ⁓ don't know if you guys have like if your audience is familiar with the chart we're talking about, but it's effectively a fraudulent chart. It's just like it's completely made up these days. there used to be a version that was okay and now it's become just ridiculous. the don't even justify it anymore.

Anish Koka MD: You Yeah, I think it's hard to get the... Anthony has said this many times and we all feel it as clinicians that are working in hospitals. mean, it's insanity what it takes in some hospitals to discharge a patient now compared to like what I used to do. I'm old enough to... When I was in training and a patient was ready to be discharged, I would go to the patient's chart and write discharge now. Like that's what I use or DC home. Like that was it. didn't do it. Didn't do anything else. You know, there's no medic and medicine reconciliation and nothing anyway. So, and, of course, every time you add some layer of whatever, now you have like somebody who's checking on med rec if it's okay, what I mean? It's like this massive mushrooming things. don't know. I don't know what exactly the, the number Like, is it, really 2000 X or is it a hundred X or something? It is as you. said also in terms of how you feel it, it is remarkably significant. And there's a lot of demand for administrators all the time. ⁓ frustrating. And of course, this doesn't even take into account the clinicians that are like 20 % or something FTE. And then the rest of the time they're running around, wrapping on the for not, doing your

Cremieux: Mm-hmm.

Anish Koka MD: Dictation or something in time anyway, so I don't know. So I might my spirit is with that chart I don't know. what the exact number is of it You do something again, this is related where you do a job of taking Meta analyses and of driving a truck through it ⁓ in the sense of destroying it you demonstrate that there's publication bias. Can you talk a little bit about

Cremieux: Whoops, it sounds like you're you're you I I couldn't hear you for a minute. I think your mic turned off.

Anish Koka MD: sorry. yeah.

Cremieux: Sorry, I couldn't hear you for a minute.

Anish Koka MD: Yep, that's okay. Can you hear me now?

Cremieux: I hear you now, I hear you now. Sorry.

Anish Koka MD: Okay. There's a, you do publication bias, kind of correction, a corrective to that. And you've done it with the FDA when it comes to food dyes. Can you talk a little bit about how you do that?

Cremieux: I think I only heard a few words in that, but I heard publication bias and food dies.

Anish Koka MD: Yes, so you do you do a correction, right?

Cremieux: Yeah. ⁓ Yeah. So there are ton of different publication bias corrections out there. Basically what these are are their ways of attempting to account for the fact that the studies that enter the meta-analysis are just the studies in the literature and the studies you can get your hand on hands on from like that you know are out there but haven't been published 'cause filed or by some of their authors. you attempt to overcome the fact that there are unpublished studies by Checking out the signatures selected publication. And one of those is that the p-values, the standard errors relative to the estimates and all that, they are configured in a way that leads to significant results a lot of the time. So a study might have a small amount of power, it might not have a very large sample, it might not have achieved So it'll have to achieve, if it's to get a p-value under 0.05 to reach that significance threshold, might have to get huge effect size. And as a result, the effect size will be greatly exaggerated in order to achieve statistical significance. And if they didn't do it any other way, they wouldn't get that, and it might be harder for them to actually get the paper published. So We notice this so we can correlate the standard errors and the sizes of the effects, and then as a result, up with an that you can shift the intercept on, and it'll you reduce it generally, or maybe it'll make it larger if there's negative publication bias, ⁓ and it'll move towards a more realistic estimate. And we have assessments this because we have some literatures that are unbiased. So like for example, there are two educational funders out there who it so if you get funding from them, like a grant from them to run a study, you have to publish the study. and they have to have it analyzed independently. and there are several other publication bias correction methods beyond this. Like you can fit what's called a selection model you have some number of parameters that are like if a study has a p-value in the range of like 0.05 to 0.01, it's just significant, like barely significant, and yet they clearly do something to manipulate it to get it that way, then you downweight it accordingly. And using these different methods, you generally do that effect sizes and meta-analyses exaggerated considerably upwards, and if you correct, they go down. And then when you review By comparing them to pre-registered replication studies that are large and that they say what they're gonna do ahead of time, you generally find that those render ⁓ much, much, smaller They tend to actually be smaller than the publication bias correction methods will say they should be. So the publication bias correction method might say, well the effect should be half as large. And then you look at the replication ⁓ and it's actually, ⁓ ten percent as as the actual study. As an example, I saw a very ridiculous ⁓ study yesterday where this person, Alex ⁓ Lugavere, was ⁓ positing the study from twenty fourteen and he goes, Well, they told people that they didn't sleep very well, ⁓ as a result, they scored fifteen IQ points lower on a standardized test. And it's like, Hold just telling people they didn't sleep very well reduced their IQ by a full standard deviation? That just seem realistic. And as it out, that didn't replicate. And if you had looked at the entire literature on using this sort of sham, these lying to people and like seeing what happened to their cognitive results, ⁓ you would Found the effect size is actually a lot smaller when you apply these methods, you would predict from the eventual failed replications. And that's genuine that's just what you get all the time. You look at meta-analyses and you often end up looking at like just an accumulation of garbage. sometimes you have fraudulent studies in there. So like there was a very famous meta-analysis on IQ and motivation, the proceedings of the National Academy of Science, and they finally retracted it, I think, last late last year. after something like fifteen years of being published, it was wild. because the studies supporting the very large effect size they found turned out to have made up one of the very Rare. Should be done more often. Yeah.

Anthony DiGiorgio: So there's okay.

Anish Koka MD: How do you so you've done a job kind of showing how meta-analyses themselves are kind of these selected cherry picking and some some outright fraud of course. ⁓ how you prevent how you ha we have that automatic reaction to to certain things? Like when Jonathan Haidt says that, like the automatic reaction is okay, that's that's BS. ⁓ Now, how but of course that that means that we are in our own little

Cremieux: ⁓ yeah. Yeah.

Anish Koka MD: bias echo right? I mean so how does one like have you made any mistakes in retrospect on your on the side of ⁓ where you think for sure something's right? ⁓ everything progresses. the question you always have ask yourself okay, I believe this. What new data point would shift me from what I believe? And if will you from what you believe, then you're no different than any every other, kind of hyperpartisan, who's pseudoscientific.

Cremieux: Yeah. So I actually have several posts where I'm just like, I think I used to believe this thing and now I'm wrong I think it's wrong now. Or I like I've changed my view on this, or I've qualified this. For example, some years ago, before I had a Twitter, I used to really believe in a massive effect size for the effect of exercise depression. Meaning, like, if you go and exercise, you go on a like a few mile run, you'll greatly improve your happiness and you'll be, feeling a lot better. And we should thus get psychiatrists to go and recommend like going to the gym, basically. And then Some RCTs came out, and it turns out when you tell people to go to the gym, and they for one, they have low adherence, they often don't do it. And if you do get them to do it, you give them like incentives to go and do it, it has really small effects. and then so I then I started thinking, well, I guess it must have no effect. It must just be wrong. It must just be that the people who can go exercise are gonna be less depressed if they something like that, some sort of selection effect. but then I updated again because it turned out there was some Mendelian randomization evidence that showed, yeah, okay, actually it does seem Like exercise leads to less depression. It's just a very small effect. And I had to qualify for belief even further. So I went from huge effect ⁓ to effect to modest effect. And I that's where the RCTs have sort of ended up. And if you don't update like that, then you'll it's like you ever heard of the term simulated annealing? It's like that. You have a system where it updates between points rapidly at first. Like you go big change, smaller change, smaller change, smaller change, smaller change, until you have a more qualified view that is actually more representative of what the reality is. because you've tuned yourself like a model. You've ⁓ taken in a lot of different data points and you've slowly come around to the sort golden ⁓ moderate of everything. hopefully with some elimination of some of the potentially fraudulent studies along the way.

Anish Koka MD: Do you do you worry ⁓ sorry, go ahead, Anthony.

Anthony DiGiorgio: So we had Jay Bhattacharya on I was gonna say, we had Jay Bhattacharya on here, and I he's trying to tackle that with the NIH, and I give him a lot of credit for this in looking at the reproducibility crisis and fit mandating that lot data sets now be public when you publish a study. I get you know, some of the studies I'm involved in, the data sets are out there, they have their own DOI. So after we publish the study, you can click on the link to the data set, download the data we used, and run your own analysis. but I worry about the effect that you may get when people are just choosing willy-nilly to run whatever analysis they want. And I think of that, I don't know if you're familiar with the Silverson study, right? Where they gave the same data set on soccer fouls to a bunch of different data analyst teams. And basically it they came up with a whole gamut of results, right? Some this exact same data set. Some teams said there was bias in soccer fouls, some teams said there was a reverse bias, some teams said there was no bias, but the exact same data set. So how are we going to avoid that?

Cremieux: Mm. Yeah.

Anthony DiGiorgio: Again, I think more data is probably better, but how are we going to avoid the fact that now you've just got hundreds of different people running analyses on those public data sets and people are going to come up with whatever analytic results that they want just by tweaking their models?

Cremieux: Yeah. do just have to kind of establish standards for rigor. one of the things that's really missing from all of these studies so far, where they've given out many data or the same data set to many different teams, is that none of them used simulated data with a known ground truth and then attempted to figure out which aspects the teams actually predict the result correct. for some reason they've all just looked at the fact that there's variation and then gone, Ooh, look, there's a lot of variation in what results you can get. Now I'm thinking to myself, every time I read one these papers, Well, why you actually just give out a data set where you know the facts, you know what's true? it's very easy to do these days. And then see, is there still And if there is, let's try and figure what's motivating this variation. I don't know why nobody's done that. I feel like that's one of the crucial things we have to see done before we can advance on this question. I think we're actually kinda stuck until we do that. we have to know Are people right or wrong? a lot of the time when we have a kind of a we have a method called multiverse analysis. And I was just telling you guys I was running a multiverse analysis, and people are unfamiliar, a multiverse analysis, also called a specification curve analysis, ⁓ is you fit every model. You just do everything. So you have a bunch covariates, you have a bunch of things you can control for, ⁓ you have a of subgroups, you have a bunch of years of data, you have different things that like different independent and dependent variables that could be, considered proxies for one another. you run literally every analysis and you arrange all the results from the smallest effect to the largest effect. And this effectively is you acting as all of those different teams. And then you output like the median result. Or you cut down to the results that are realistic and possible. Like you move the temporally impossible results. Or you check if they align with the possible results or something like that. You like look for bias in that way. And you can use this method to get a more realistic image of like what the literature actually could say. but the problem is, I compared this to RCT results, and even doing that, even fitting every possible model you can with your dataset, doesn't get you the RCT result reliably. It just work. and I feel like this is gonna be one of the most fun findings in the paper. I have some nice little illustrations of this. I'll send you guys some ⁓ pictures from it later. But Just gist is, you can do anything you want with the data and still it's wrong. it doesn't even matter. all these teams can disagree. You can be your own thousands of teams and it still just is pointless because you didn't have the ability to even infer the thing from the study.

Anish Koka MD: All right, sir. Well

Anthony DiGiorgio: Yeah, I mean that that's an issue with so much research is it's so hard to so hard to get to what ground truth is. So many outcomes, you know, s i in medicine outcomes are subjective often. Like if you look at any study looking at back pain, right, it's a totally subjective outcome. it it's really hard to I think to even find what the ground truth is in a lot of these studies.

Cremieux: Yeah, yeah. It becomes a complete mess.

Anish Koka MD: Excellent. Well, sir, we've been talking for a while. We've been talking for about an hour. Thank you for giving us all your time. This has been great. there's a a number of number of posts. And we've got as we talked before we went live, there's so many different posts that we could talk about. ⁓ Haha. You cover a very expansive breadth of stuff and it's fairly impressive the breadth of stuff that you do cover. So it's always super interesting. You always learn something, even if you you know, even if you may disagree. So sir, thank you so much for coming on. Thank you for chatting. All right. your handle yeah absolutely your handle is at Cremux ri I yeah go You you you you you tell us your handle.

Cremieux: Thank you for having me on.

Anthony DiGiorgio: never gonna be able to pronounce it'll be in the show notes, but please go ahead. your substack and your Twitter handle.

Cremieux: I got you. And I the name has probably held back my growth a little bit. So it's kinda it's even more remarkable.

Anthony DiGiorgio: But i it it's remarkable stuff you're putting forward and I highly recommend anyone subscribe to your Substack. Obviously it gives you a follow on X and tune in 'cause I one of the best I enjoy reading all your stuff. so thank you for doing what you do.

Anish Koka MD: Excellent.

Cremieux: Thank you guys for having me on. It's been a lot of fun.

Anish Koka MD: Thanks so much.