Why I Reject Your Data Papers

I believe that research papers are innocent (“accept”) until proven guilty (“reject”). Here are some crimes that move your data mining paper into the guilty camp quite quickly, and make me feel sad:

  1. No baseline. You present results of your wonderful new machine learning algorithm compared only to variations of itself (e.g., results with different parameters). No baseline or alternative approach results presented. Quantitative results on data only make relative (not absolute) sense: If your algorithmic magic is 99% accurate, is a simple linear regression baseline 98% or 58% accurate? There’s a huge difference here.
  2. Excessive/unmotivated data pruning. You prune your dataset to fit your particular use case, and do not give any concrete motivation for doing so. The most common of these is the “we removed all of the users with fewer than X entries,” with no explanation as to why (overall), why X rather than some other value, what effect this had, or what would happen to users who do not have X amount of data.
  3. Your metrics aren’t measuring your claims. Your metrics don’t measure your claims. You’re claiming an increase in user satisfaction (or long-term engagement, or recommendation quality), but are only measuring prediction accuracy.
  4. You’ve evaluated your algorithm… manually. You evaluate your algorithm by manually inspecting test instances. “Hey look! It works for these two cases, so surely this is cool!” I admit that I have sometimes done this while debugging algorithms, but (as far as I recall) I have never reported results in this way. If you’re going to do this, you need to (a) not be the one evaluating (i.e., run a user study), and (b) do this on way more than the handful of instances that you can do.
  5. You do more than one thing on this list.

And some minor issues to keep in mind:

  1. I haven’t read every paper out there, and assuming that I’m going to read all of the papers you cite (“we do what [76] does.”) is a stretch.
  2. Related work should discuss why work is related. It’s called “Related” work, and not “Here’s a list of what others have done”
  3. As a reader, I will skip your “the rest of this paper is organized as follows” paragraph. I don’t care much about knowing how your paper is structured; I would much rather you spend the time/space telling me about what you have found and its implications.
  4. While I know that some people feel very strongly about this, I don’t care if your related work is Section 2, or Section [Conclusion-1] as long as it is readable where it is. Are you discussing related work results compared to yours? That should come after you have presented your results. Are you discussing related work as a motivation for yours? Perhaps that should appear after the Introduction.

#wwdh What Works in Digital Health?

Random notes from the “What Works in Digital Health” workshop:

  • Personal Tracking as Health Informatics (paper)
  • There’s no such thing as gaining a pound (paper)
  • Trajectories of depression (paper will soon be here)
  • A wicked problem is a problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize.
  • Root cause analysis is a method of problem solving used for identifying the root causes of faults or problems.
  • Action research is either research initiated to solve an immediate problem or a reflective process of progressive problem solving led by individuals working with others in teams or as part of a “community of practice” to improve the way they address issues and solve problems.
  • My slides on Data Science in Digital Health.
  • Beyond Personal Informatics (workshop)
  • Man vs. Smart House: A Cautionary Tale (video)
  • Formative vs. summative assessment (definitions)
  • The tea cup stretch.
  • HANDI “Supporting the Architects of the Digital Health Revolution”
  • The UK dallas project is thinking beyond traditional health and social care to consider how new ideas and technology can be used to improve the way people live.

Notes from @RoyalSocMed #telemed Event

Brief notes and links from Mainstreaming medical apps; reducing NHS costs; improving patient outcomes event at the Royal Society of Medicine.

  • 90% of apps are not used after 6 months. 80% of apps do not generate revenue to support a business. 500 million people will use a health app by 2017, a soon to be $26 billion industry. >80% of surgical trainees use a mobile phone at work. Why do people not use health apps? 37% the sheer number of them is confusing. What would convince you to use a health app? 50+% if the app is free.
  • Depression will be the largest cuse of disability by 2020 (WHO). £105 bn annual cost in UK. 75% of those with a diagnosable mental illness receive no treatment. People with poor mental health die 15-20 years younger.
  • EC green paper on mHealth (April 2014).
  • A study of the content of 47 smoking cessation apps shows an inverse relation between app popularity and behavioural evidence; here is a similar paper for chronic pain management apps; for melanoma diagnostic apps; for opioid conversion apps; a study evaluating the content of cancer apps.
  • The HSCIC is the national provider of information, data and IT systems for commissioners, analysts and clinicians in health and social care.
  • Our Mobile Health reviews and tests medical apps.
  • Orange is into healthcare? @orangehcare. mHealth grand tour trial on real-time feedback and monitoring of glucose levels in diabetics.
  • Phones grow all sorts of bacteria.
  • Seven deadly sins of app development: (1) inequality in app availability vs incidence of disease, (2) lack of expertise; no medical specialty involvement in app development, (3) plagerism- copying information from the web into an app, (4) commercial bias/interests- apps with no consumer reviews, promoting private practice, (5) inaccuracy and errors e.g. in melanoma detection and calculator apps, (6) lack of security & confidentiality, (7) lack of testing, evidence to support effect (e.g., this review, and this paper).
  • Behaviour change COM-B framework (Capability, Opportunity, Motivation); 93 behaviour change techniques have been grouped in this paper.
  • MyHealthApps: patients helping each other find apps that are useful to them.
  • BCN Health App were demo’ing their TCApp for eating disorder management.
  • The Alliance of European Life Sciences Law Firms. What is a medical device? Includes software intended by the manufacturer for diagnosis, prevention, monitoring, treatment or alleviation of disease. Yet “stand alone software for general purposes when used in a healthcare setting is not a medical device.” Yet…
  • “Medical device stand-alone software including apps” UK guidance online. “Mobile Medical Applications” US FDA guidance online.
  • Patient view: compliance = automation + simplicity
  • Big White Wall 27k members, 27% of UK adult population covered in contracts
  • Prof Meddis (Essex): anyone that can command an iPhone can produce their own hearing aide. Here is BioAid.

Extracting Large Datasets from the Google App Engine Datastore

The Google App Engine (GAE) is a very interesting service for quickly deploying web apps or building systems that use Google’s infrastructure. I’ve recently been working on a project that uses the GAE to receive large amounts of data from a mobile app that is part of a feasibility trial with 20 or so participants. Therefore, the two key things that I wanted to accomplish are (a) getting a back-end that receives/processes/stores data up and running as quickly as possible and (b) having a way to extract/download the data for offline analysis. The GAE was perfect for part (a), and the system was set up: the data that rolled in was easily stored in the NDB data store.

Now the headache: getting data out of the system. What ways could this be done?

  • Don’t even try: using a front-end request handler. These are limited to 60-seconds, hardly enough time to read through/format/process a large datastore.
  • In the old days, it was possible to back up data from the developer console. This limits data extraction to admins only, which is not ideal. This post describes how this was done, and worked for me… until the console was upgraded and the ‘Backup’ functionality was removed (does anyone know where it is?).
  • The bulkloader. This post describes how to use it, and I got this working. Simple and command-line based. Hours and hours later, the bulkloader was still trudging along and would then fail, which seemed to happen if my machine momentarily lost it’s WiFi connection. I didn’t see any way to modify how the bulkloader throttles data download; so I’d only really suggest using this method if your data is very small.
  • Using the remote API shell, and paging through the data store entities. I got this working too, and this seemed to be faster than the bulkloader.. but would hit some over quota limits after some time (even though my project has billing enabled). Again, probably ideal for small data, and admin-access only.

What I did in the end was inspired from the post that wanted me to start the backup from the admin dashboard:

  1. I found this post that explains how to transfer data from the data store to BigQuery. What it is actually doing is transferring data from the data store to the cloud storage, and the uploading it from cloud storage into BigQuery.
  2. I wrote a map-reduce job that does the same as the first part of that post. The problem I encountered was that the example and  all the documentation points to using the “FileOutputWriter” in “mapreduce.output_writers” which does not exist anymore (therefore producing ImportErrors). By digging around the code, I found that I could replace it with “GoogleCloudStorageConsistentOutputWriter”
  3. I ran the map-reduce job by hitting the URL with a get request. My code then redirects to the mapreduce status page. The whole thing took approximately 10 hours to complete.
  4. I downloaded the data using gsutil (the -m flag enables multithreaded downloads):
    gsutil -m cp -R gs://my-bucket/my-dir .

2014 in Review

My last year in review ended with “I think that 2014 is going to be a year of prioritising and getting things finished.” I think I (mostly!) managed to do just that, with most of my work this year focusing on one project, Emotion Sense. My #YearInReview:

  • Most of the year was dedicated to analysing our growing Emotion Sense data and our results are now taking shape. Gillian presented our first poster. There is so much more yet to come from this; I hope that my 2015 year in review will have this as its first point.
  • We received a grant from the EPSRC’s Impact Acceleration Follow-On Fund, to help turn Emotion Sense into a commercial product for healthcare measurement, monitoring, and intervention. Emotion Sense has also been accepted as an i-Teams project which will kick off in early 2015.
  • We received funding from the Medical Research Council to test and further refine Q Sense: a smoking cessation intervention that (building from Emotion Sense) uses smartphone sensing to guide quitters on a personalised, data driven journey away from tobacco. Felix presented a poster about the project as well.
  • A paper about group co-location in social networks – lead by Chloë – was published in PLoS ONE (arxiv link).
  • I submitted/published a book chapter on “The Anatomy of Mobile Location-based Recommender Systems” (preprint) for the upcoming second version of the Recommender Systems Handbook.
  • I (finally?) published some old work from my time at UCL on crowdsourcing public transit status updates via a smartphone app (pdf).
  • Advait published the results of his masters project analysing shared bicycle systems around the world in an upcoming special edition of Transportation (Special Issue on Emerging, Passively Generated Datasets for Travel Behavior and Policy Analysis).
  • I gave talks in the SACHI Group (University of St. Andrews), Institute for Public Health and Dept. of Psychiatry (University of Cambridge), the E-Health Unit (University College London), and the Dept of Primary Care and Dept. of Education (University of Oxford), at Health 2.0 London, and at Data Science London’s Healthcare hackathon. Every talk was about smartphones (e.g., these slides).
  • I did less consulting work than last year, but what I am doing is more on track with my other work (… smartphones!).

I guess that the trend should continue: I hope that 2015 is going to have Emotion Sense written all over it.

Random Links from #health2eu

I’m at Health 2.0 Europe: London November 10-11. Brief, brief notes:

The “From Health to Wellness” non-auditorium session (where I presented about Emotion Sense).