Chat about Novel Coronavirus - 2019-nCoV - COVID-19

17% of Londoners likely to have had it. Rest of the UK is 5%.

https://www.theguardian.com/world/video/2020/may/21/london-17-of-population-may-have-had-coronavirus-says-matt-hancock-video

You can cuddle with your grandkids w/o worries etc.

As long as you are sure that you have not carried it on your person after touching something previously touched by an infected person

Interesting - on the claimed accuracy of the antibody tests that have been approved by PHE

https://www.sciencemediacentre.org/expert-reaction-to-phe-laboratory-evaluations-of-roche-and-abbott-antibody-tests/

You can cuddle with your grandkids w/o worries etc.

As long as you are sure that you have not carried it on your person after touching something previously touched by an infected person

Also, as long as they haven't touched everything and are now carrying it.

And it's contrary to government guidelines, but, yeah.

Greenbank is right, I think.
I'll have to read it all but the false positive, false negative is the whole problem. If the test is accurate and not specific etc etc.

As long as you are sure that you have not carried it on your person after touching something previously touched by an infected person.

One can argue that people may relax and not take as much care around them if they think they’re ok.

Just not even shocked any more....

Tens of thousands of Covid-19 tests have been double-counted in the Government’s official tally, public health officials have admitted.

Diagnostic tests which involve taking saliva and nasal samples from the same patient are being counted as two tests, not one.

The Department of Health and Social Care and Public Health England each confirmed the double-counting.

This inflates the daily reported diagnostic test numbers by over 20 per cent, with that proportion being much higher earlier on in the crisis before home test kits were added to the daily totals.

Almost 350,000 more tests have been reported in Government data than people tested since the start of the pandemic.

The discrepancy is in large part explained by the practice of counting salvia and nasal samples for the same individual twice.

Happy to have you chime in.

For the record, and I've said this to him, I've never doubted him being right about something. I just didn't understand what he was trying to express here:

So if 5% of the population have had it then using 98.5% and 99.5%
figures we get:-
-ve result would be 99.973% accurate
+ve result would be 91.284% accurate

If 1% of the population have had it then:-
-ve would be 99.995% accurate
+ve would be 66.779% accurate

If 10% of the population have had it then:-
-ve would be 99.944% accurate
+ve would be 95.673% accurate

I've come to the conclusion that this is a claim about the test's result accuracy in the general public when deployed, and as a count (albeit expressed as a percentage). I noted this a few times yesterday, but it was never acknowledged that this was in fact the source of the misunderstanding (or I missed it as I slowly had more beers/was making dinner). I.e.:

"Just to reiterate, I do understand that the overall number of accurate results will depend on the prevalence of the disease in the population. But the accuracy of the test, in my understanding, should be independent of this."

This may all be down to clumsy language on my part, though. What I was trying to express was the difference between the accuracy of the test as a pharma company expresses it on the tin (presumably something akin to an F1 score). This is in contrast to the accuracy (which I think greenbank was expressing), which is that achieved in the population as is influenced due to other factors (prevalence being one of them). The latter does not impact the former.

Assuming I understand everyone now.

I think it's Bayes fault.

http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_Probability/BS704_Probability6.html

https://www.sciencedirect.com/science/article/pii/S2405844020304163

This may all be down to clumsy language on my part, though. What I was trying to express was the difference between the accuracy of the test as a pharma company expresses it on the tin (presumably something akin to an F1 score). This is in contrast to the accuracy (which I think greenbank was expressing), which is that achieved in the population as is influenced due to other factors (prevalence being one of them). The latter does not impact the former.

No, I don't think you're understanding what I've said. It doesn't have to involve the general population at all, the same problems occur when you use the same test group you used to measure the sensitivity/specificity.

Imagine you have a test group of 10,000 people. You know everyone's exact status through other testing. 9,500 (95%) are negative. 500 (5%) are positive.

Imagine the test has a specificity and a sensitivity of 95% (i.e. you get 5% of false positives and 5% of false negatives).

Now apply your test to all 10,000 people in your test group.

Consider just the positive results, where could these have come from?

Firstly there are 95% of the 500 people who are truly positive. So that's 475 people. (The other 5% get a false negative result.)

The other positive results will be the false positives from the people who are truly negative. How many of them will there be?

That'll be 5% of the 9,500 people who are truly negative. That's another 475 people.

How accurate is a positive result if it's only correct for 475 out of 950 people who test positive? 50%.

So even if you apply your own test to the same test group you used to measure the sensitivity and specificity of the test you find that a positive result is not as accurate as you expected.

It has nothing to do with the general public, the above numbers come from the same people who were used to calibrate the test, although the general population prevalence can skew the numbers even more if the prevalence in the population is different to that of the test group that was used to help measure sensitivity and specificity.

Can you do this in a 2 x 2 table? that might help?

I'm going for a run, that might help.

HA!
I don't think anyone is being obtuse or deliberately LFGSS.

I know that. I'm trying to strip it back to basics as much as possible to make it easier for the "Aha!" moment to appear.

(I read too many Martin Gardner books as a child.)

We now have two 10k groups. The test group from which the sensitivity and specificity are learned ("the same test group you used to measure the sensitivity/specificity"). And a second 10k (which may or may not be the same as the first, but in your example they are). This second group is having the test applied to them. The important thing is that these are two different things which I think we both agree on.

The first group is used to judge and measure the test. Sensitivity and specificity are learned from this group. Whatever you do with the test after this, sensitivity and specificity do not change (unless they were wrong to start).

The second group is that which the test being used on. In this case, we can say that the second 10,000 people in your example are being used as an analogue for the general population, no? The results will depend on the makeup of this 10,000 people. They will not mimic reality (that is, the tests are not perfect).

So even if you apply your own test to the same test group you used to measure the sensitivity and specificity of the test you find that a positive result is not as accurate as you expected.

The tests are not 100% accurate, this was never a point of confusion.

So again, the sensitivity and specificity which are learned from the first group is what I thought at the very beginning you were claiming shifts, depending on prevalence in the population (this is what I quoted in my reply to Chalfie, and if you remember, the first thing I said to you after you replied was: "Ah, sorry. So the raw number of false positives/false negatives will shift depending on how many true negatives/true positives there are. Okay - I mistook your "-ve" and "+ve" to be analogues for specificity/sensitivity.")

The results in the second group (be it a sample or the general population) are dependent on the makeup of that population and the sensitivity/specificity of the test. I agree with this and always have.

If I'm still misunderstanding you we can either take this off public chat or you can rest assured that you tried and I'm a moron.

To be honest, I'm a bit embarrassed here as everyone seems to get it but me. I understand how this stuff works (when it comes to the type of the stats I do). I've published papers that rely on robust precision and recall testing. I've spent lots of time worrying about false/true positives/negatives in my datasets and analyses. I just don't quite get what it is you're trying to convince me of if it's not what I've already laid out (or if it's just that the results will not be a perfect match for reality, then we never disagreed).

I don't really get it yet, I'm trying to do something else. I've read enough to be dangerous but need to have the aha moment so i can explain it back. make sense?

I'm not sure talking about groups/populations is helpful. You just test an individual and there are 2 possibilities: a positive result or a negative. Of the positives approx. 50% are false results, which is a failure of the test. Of the negatives, 99% are accurate and 1% are false negatives. It is much more likely that the testee is negative so the false positives are fairly common and false negatives are an (unlikely) test failure on top of an (unlikely) result, meaning a very small proportion. Hence, negative results are more likely to be true than positive results.

You are correct that the sensitivity and specificity never change. Just the likelihood of the result being true or false, which is what @Greenbank meant by "66% accurate" or whatever (i.e. 66% of these are true, the rest are test failures)

I don't think you actually disagree with each other

We now have two 10k groups. The test group from which the sensitivity and specificity are learned ("the same test group you used to measure the sensitivity/specificity"). And a second 10k (which may or may not be the same as the first, but in your example they are). This second group is having the test applied to them. The important thing is that these are two different things which I think we both agree on.

Yep.

The first group is used to judge and measure the test. Sensitivity and specificity are learned from this group. Whatever you do with the test after this, sensitivity and specificity do not change (unless they were wrong to start).

Yep.

The second group is that which the test being used on. In this case, we can say that the second 10,000 people in your example are being used as an analogue for the general population, no? The results will depend on the makeup of this 10,000 people. They will not mimic reality (that is, the tests are not perfect).

I used the same group of people again to highlight the fact that even if you use the same group of people you can get weird looking results for the accuracy of a specific outcome (not the test in general).

In my example above the overall accuracy was 95%. I don't doubt that. I've been talking about the accuracy of a positive result of the test. With the numbers above if you get a positive result then it's only a 50:50 chance of being accurate, despite the test having an overall accuracy of 95%.

So even if you apply your own test to the same test group you used to measure the sensitivity and specificity of the test you find that a positive result is not as accurate as you expected.

The tests are not 100% accurate, this was never a point of confusion.

Yes, but they're not 95% accurate for people who get a positive result.

So again, the sensitivity and specificity which are learned from the first group is what I thought at the very beginning you were claiming shifts,

No, I was calculating the accuracy of the test per outcome and showing how it differs massively for a negative and a positive result.

depending on prevalence in the population (this is what I quoted in my reply to Chalfie, and if you remember, the first thing I said to you after you replied was: "Ah, sorry. So the raw number of false positives/false negatives will shift depending on how many true negatives/true positives there are. Okay - I mistook your "-ve" and "+ve" to be analogues for specificity/sensitivity.")

The point is that an individual test may be 95% accurate but only if you know your true status (in which case the test is pointless).

When you get your test result the only thing you know is your result, so you have to look at the estimated/calculated accuracy for the individual result, and that is where they can be skewed well away from the expected 95%.

(I know you're not a moron. I'm probably using the wrong words/terms all over the place, apologies if I am.)

I'm not sure talking about groups/populations is helpful.

You're probably right, but because I misunderstood the point from the start (I think!), it was necessary to distinguish between the calculated sensitivity/specificity and the actual test results.

You just test an individual and there are 2 possibilities: a positive result or a negative. Of the positives approx. 50% are false results, which is a failure of the test. Of the negatives, 99% are accurate and 1% are false negatives. It is much more likely that the testee is negative so the false positives are fairly common and false negatives are an (unlikely) test failure on top of an (unlikely) result, meaning a very small proportion. Hence, negative results are more likely to be true than positive results.

I agree with this (and Greenbank's many examples of it using actual numbers). There is zero that is contentious about this.

You are correct that the sensitivity and specificity never change. Just the likelihood of the result being true or false.

This is what I assumed, but because of the initial confusion, I thought there may be something happening in medicine which I didn't understand or know about. Hence my original confusion/question.

I don't think you actually disagree with each other

Ha, all I can say is that I don't disagree with your take on it.

tl;dr
the test result you get might not be right.

When you get your test result the only thing you know is your result, so you have to look at the estimated/calculated accuracy for the individual result, and that is where they can be skewed well away from the expected 95%.

Absolutely. I never disagreed with this and I think we've just been talking at each other because we've picked up on particular things which stood out as odd to us (or maybe I just did this). It's hard to talk these things through on the internet when you're generally doing something else. Much better suited to a pub.

It's hard to talk these things through on the internet when you're generally doing something else. Much better suited to a pub.

Indeed.

But, for fear of overdoing it, here's where prevalence comes in...

If I know I'm negative then taking the test I've described above I know that the result will be correct 95% of the time.

If I know I'm positive then taking the test I've described above I know that the result will be correct 95% of the time.

But people don't know whether they are negative or positive, hence the need to take the test. This is where prevalence comes in. Let's say it's 10% amongst the population (to make my numbers easy again). [EDIT] Changed from 5% to make it obvious this is nothing to do with the test sensitivity/specificity.

Also I'm only going to do the calculations for a positive test result.

So there's a 90% chance I'm truly negative. If I'm truly negative then the chance of a false positive result is 5%. This route gives me a 4.5% chance of getting a positive result.

Conversely there's a 10% chance I'm truly positive. If I'm truly positive then the chance of a true positive result is 95%. This route gives a 9.5% chance of getting a positive result.

So a positive result in a test (given these assumptions) gives me 67.9% chance (9.5% / (9.5%+4.5%)) of getting an accurate result given the balance between truly positive and false positive. (That's not accuracy for the test overall. If you do the same calculations for a negative outcome it's 99.7% or similar.)

Your status when you take the test is unknown, so you have to use the general prevalence to answer the question of "How accurate is a specific outcome?"

If you change the prevalence amongst the population you're altering the odds for your own infection status, which affects the derived accuracies for the two test outcomes. It's not so much that the numbers/status of other people that are infected affects your test, but it's the prevalence that needs to be taken into account for your possible initial infection status.

Did I miss something, or wasn't there a claim that the antibody tests only gave false negatives but not false positives?