Personal Insights Data

Lately, I’ve been on a bit of a kick to combine my closeted data nerd tendencies and my recent interest in self-improvement. Thus, this year is about deciding what data I need to start tracking and setting up a system to store and analyze it. This has been partly inspired by Scott Adams’ book How to Fail at Almost Everything and Still Win Big, and partly inspired by using the tools I have (geekery) to solve the existing problems I have (dissatisfied with how life has gone to this point).

As part of this, I’m trying to make the data collection process as streamlined as possible. This means getting a place to store data on a daily basis (MongoDB), and methods for gathering a lot of the data I send (lots of API action, using IFTTT where possible, and assorted fun), visual displays are going to be a part of it (ChartJS), etc. The hard part is going to be the analysis portion, which I may have to lean on Clint for some help with, but I’m looking forward to this.

Activity Data to Be Gathered

As you can imagine, I’m going to be working to gather a LOT of information. On the hard data side of the house, here’s just a few of the things I’d like to track:

  • Sleep duration, step count — FitBit via IFTTT, using my Charge HR.
  • Bicycling data — via the Strava API. I paid for a full account last week, so this should have some more interesting stuff in it.
  • Weightlifting data — I should do some research. I’m sure there’s a tracking tool out there with an API, but it might be easier and more cost-effective to just roll my own.
  • My medications, including time of day, dosage size, etc. Probably going to have to log this manually.
  • Habit frequency (and in some cases, duration) — things like meditation (frequency/duration), teeth brushing (frequency), etc.
  • Computer time — number of tweets on a given day, facebook interactions, # of emails sent, etc., can be tackled via various tools, including things like Time Sink. It’d be great if there were a usage tracker plugin for Chrome.
  • Time spent writing (word count per day) — need to interface to WordPress and see if Scrivener supports Applescript.
  • Reading volume — pages per day, book progress, etc. Probably going to have to make Goodreads a part of this.
  • Environmental variables — locations throughout the day, weather conditions, amount of daylight exposed to, amount of social contact, and so on. Basically, I want to look at the things you don’t normally notice, or necessarily even consider a part of your day-to-day satsifcation with life/happiness.

Honestly, that’s just the tip of the iceberg. I could see making a point to track everything that happens, right down to paying bills (does it impact my mood? probably.), or even (pardon the crudity here) when I take a dump.

Life Indicators

These would be the things I’d like to track in order to ascertain how life is going. There’s two types of these indicators — hard and subjective. Hard metrics will be things that can be measured directly — what’s my resting heart rate (FitBit)? What’s my VO2 Max (sports lab), body weight (scale), cholesterol (lab tests), and whatever else you can think of. On the flip side of the coin, there’s the subjective “how I feel about these things” — which will probably be on a scale of 1-5 for simplicity. How rested do I feel? How is my stress level? How do I feel about my parenting for the day? What’s my mood, where 1=everyone die and 5=I love everyone?

It’s the subjective stuff that’s going to be the most important metric, because it reflects how I feel about things. And it can even be influenced by the hard indicators mentioned above (body weight can impact self-image, for example).

Daily Gathering and Intermittent Gathering

The other hard part of this is going to be getting data into the system on a regular basis. Sure, some of it can be scripted1, and some of it will be automated, but ultimately, I need to put a system in place that simplifies the process of gathering that data.

To that end, I’m thinking of hashing together a quick iOS app that will ask on a preset time interval (hourly or daily) to rank the subjectives from 1-5 and have an interface to allow me to manually enter data for things like lab data (medical or sports) so that I can keep data flowing at a regular pace.


The ultimate goal here is achieving personal insights with data instead of a therapist2 — to isolate the things that are causing me to be dissatisfied with life and to reduce, minimize, or yes, even outsource them, and then to find the things that are making life better and to craft the time for making them a larger part of my life.

What do you think? What should I track? What sorts of features should I roll into this thing?

Image Credits: Csaba Nagy/.

2 thoughts on “Personal Insights Data”

  1. This is an interesting area. Stephen Wolfram has been tracking data for some time ( and has inspired someone to write a keylogger: (Andrej Karpathy wrote one that records less but displays more

    Russel Poldrack got himself about 100 MRIs of his brain over a period of weeks and, to look for correspondences of blood flow in his brain to various aspects of his life, recorded a lot of personal data: (a nice side-effect of his project is the prepackaged analytics virtual machine they made, which you could reuse for some of your data:

    Anand Sharma got interested in visualizing all this data and made a product that pulls in data from various sources like Fitbit, Strava, location tracker app Moves, and computer use tracker RescueTime to make some nice Hollywood-inspired visualizations: (his app is called

    And on behavior change, this wrist shocking thing looks promising:

    Good luck with your project.

    • Fuck, this is great stuff, Nate. Thank you. Pavlok looks particularly interesting, but I’m hoping that the cold, hard data of “hey, this particular thing is fucking me up” will be enough of a shock that I can a.) start changing that habit without resorting to jolting myself, and b.) see the data change on-the-fly as I make alterations to my behavior and hopefully form some sort of positive-feedback loop.

Comments are closed.