Chat about Novel Coronavirus - 2019-nCoV - COVID-19

I take great glee in putting massive data processing costs into every project, costs go something like
We speak to your suppliers and source data directly: £x
We speak to your systems team and work through them: £1.2x
We work with your marketing team to get the data: £5x

Meh. I have to look at a lot of logs from our product. The main component generates anything up to 50GB a day.

All has to be treated under GDPR too.

$ df -g /ZZZZZZ
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/ZZZZZZfs 1119252.27 221534.26   81% 600988211    51% /ZZZZZZ

It's amazing how much IT stuff that may be presented as a very swishy iPhone app (e.g. phone banking apps) are actually backended by layers and layers of increasingly old and crufty goblinry

True, but the weak link in the chain here was a new component backended by fuckwits.

At last, some good news

https://www.standard.co.uk/news/world/cannabis-chemical-prevent-coronavirus-a4563806.html

authors name checks out

1 Attachment

Great spot!

I mean. Who could predict that underfunding public health, failing to deliver on it problems that were supposed to bring lab data in from multiple disparate suppliers (who don't update without payment, so it's all bodged to work), failing to deliver on software that was supposed to be the case management software, and asking the staff you do have to effectively work with notepad and excel could lead to this?

Have any of you guys actually looked into what they are using Excel for?

Had a chat with somebody who does Covid data for the Cabinet Office earlier

The testing labs have relatively disparate systems but they all have the option to provide extracts in csv or XLS or XLSX.

Those are submitted and Excel is used to aggregate the data from the (ten or so I think) spreadsheets into a single spreadsheet. From there, the process changes depending on who is using it. The CO use R to transform the data from that point. No idea what PHE do although it sounds like Access is involved which really is stupid.

Anyway, I'm sure there are better tools to aggregate a small number of relatively small spreadsheets but isnt the issue here more about process and quality control rather than the tool in use?

Note I'm not even trying to defend the possibility that Access is involved. That really is inexcusable.

I mean. I only work for PHE.

I'm aware of that. I wasnt arguing against anything you said.

Is my understanding of how Excel is used in this context correct?

I'll wait for someone who works in hospital data to come along in a.minute.

Sorry. But pissy.

I'll wait for someone who works in hospital data to come along in a.minute

I was an NHS information analyst for four years. It was my first experience of NHS Informatics.

It was one solid shitshow of using inappropriate tools for the job. We ran about a third of a billion quid of commissioning data covering a huge region of patients through spreadsheets and excel. Often being encouraged to write vbs to run queries outside of Excel to essentially splice multiple maxed out spreadsheets together . It was a horror show.

Didnt mean to sound pissy but I actually wasnt replying to you. Was more voicing my opinion that this is more a process failure than Excel being totally inappropriate for the task they are supposedly using it for. Wasnt commenting on systems or funding.

It's just a mess. This is my understanding of the situation from my experience. I've never worked an outbreak, I missed SARS and ebola.
Testing from hospitals comes into SGSS a SQL environment. Data comes out of that to be imported manually into the case management system (which has been earmarked for update and replace for at least 5 years). This allows health protection teams to do the job of identifying cases,clusters, situations, outbreaks. And to manage them with local authorities. So that's one data flow. From SGSS to the team doing the dirty work.(which is where I used to work)
Getting data out of that system with the linked information you need is very difficult.

The data on the webpages you see are all in bi, I don't understand why they're not doing shiny apps, there are a number of very good r data people in PHE London. And a lot of the stuff for public health data (fingertips for example) is generated in r. And makes use of a big SQL environment.

I don't understand as each of the epidemiology reports we receive from the epi team is clearly generated in r, but output as a PDF. We haven't been provided anything like a shared resource of tables for just our data that we're allowed to see.

I don't understand how or why access is being used, as I thought it wasn't used anymore.

I think the amount and type of information coming in has got much bigger than the processing capability of the team setting up the data gathering.

(Edit: I meant I sounded pissy. And I am. I'm sad. The people I worked with for the last 5 years are working really hard. And their organisation is being hammered)

. And their organisation is being hammered

I really sympathise with this. My first experience of managing government reform was helping shift PH out of the PCTs and have been close to all the changes since. Hasnt the PHE budget been cut in half since?

Anyway, I'm sure there are better tools to aggregate a small number of relatively small spreadsheets but isnt the issue here more about process and quality control rather than the tool in use?

Seeing as you moved the chat over here, I'll reply here.

Part of the points that I made on the other thread were that using Excel as your tool for this job makes the process and quality control harder if not impossible.

Lack of granular change control tools.
Lack of good software engineering and automated testing tools.
Lack of data type and structure checking.
Etc.

I think that most, if not all of the implications that I listed still hold up to scrutiny.

I joined in 2015, which wasn't long after it was formed. Within the first year, there were mergers of two HPTs into one. There were offers of redundancy. Since 2010, yes I know formed in 2012, there have been a succession of cuts - i'm not sure how much. But it's hard to tell when you're in London. When you talk to trainees who have been to other places PHE centres outside are pretty bare.
We were considered overresourced with access to an epidemiology team (for the outbreak investigation) AND 2.5 information officer/analysts to put data in and do stuff with it for other reasons.

What a crying shame.

Don't vote Tory.

Good idea. I'll get right on that.

Although. That nice Sunak man is going to balance the books. So.....

The people in PHE are great.
Mostly, you can do outbreak control with simple bits of information:
name, age, gender, ethnicity, geographic location, date symptoms started, date they ended, test result, date of test result. It's not BIG DATA.

So I think it's easy to fall into the problem of "excel will work". Because it has and will. I'm pretty sure Ebola was notebooks and pencils and excel.
People work with what they're familiar with and what is available and what they can set up quickly.

Which leads to multiple "databases" for lots of different projects rather than "A BIG THING TO DO IT ALL".

You've got a couple of years before you can act on it.

We might all be dead by then.

Daily figures just released for Manchester University are crazy (200+ a day now).