Actions on Google for my Tesla

I’ve been meaning to add the possibility to control the AC in my car from Google Home for a while but never got around to it. A few nights ago a finally set to action.

Google Home is basically Google’s take on the smart home. It is an ecosystem of things. The google home smart speaker is integrated with the google home app which is integrated with the google voice assistant, etc. etc.. In the Google Home app, you can setup devices you have around the house that are compatible, such as lights, power outlets, thermostats, etc. Once they are setup in the app, you can control them using google assistant or google home smart speakers. So, if I want to turn to be able to turn on the AC in my car by saying “Ok Google, turn on car AC”, I must make sure I can have a device in google home with the name “car AC” and when I issue the command “turn on”, it will send a signal to my car to turn the AC on. To do that I started looking at Actions on Google.

(Do I really need to build something for this? No, there are a couple of actions on google integrations already that you can simply add. However, when you do, you basically give away all control of your car to a third party. With this control and physical access to the car, you can essentially walk up to the car, unlock it, and drive away. Now, I’m not too paranoid about this, even though I probably should be. But, in this case, instead of giving someone the chance to drive away with my car, I much rather learn how to build this myself. After all, it’s really quite easy.)

Actions on Google allow you to “integrate with google assistant”. This currently means that you can extend your mobile apps to work with the voice assistant, to create conversational apps (basically apps that you use through voice/text… yes, chat bots), or connect to smart devices (i.e. google home). For my purposes, I do not want to create a conversational app. If I did, I would have to say things like “Ok google, talk to my car. -Ok, talking to your car. -Hi I’m your car what can i do for you? -Turn on AC. -Ok, AC turning on”. Zzz. I much rather just open the app on my phone and press the climate button. So, instead I went for smart device integration.

Actions on Google for smart home devices is really quite simple. You need an Oauth server that can issue a token, and an https endpoint that respond to commands issued by the google assistant. We are simply building a service that acts as a server for a particular device type. There are only four commands possible: Sync, Query, Execute, and Disconnect. WIth an access token received from your Oauth server, the Sync command should respond with a list of devices that this user possesses. Query should respond with the state of the queried devices. Execute should abide to whatever the particular command is (e.g. TurnOn should turn on the device), and Disconnect should do what ever needs to be done whenever the user unlinks their google home with your service.

So I first implemented the Oauth server. There are essentially two routes required: auth and token. The auth route should let a user authorize themselves and tell Google that it is ok for your service to share access to their devices. The server responds to google through a redirect route including an authorization code. The token route should take an authorization code and return an access token (as well as a refresh token that can be used to refresh the access token). For more information go here.

In my case I let a user authenticate with their tesla account, and use their access token as the access token (together with some other stuff). This means I never have to bother storing any information on my server, but let Google store that in the token instead.

Next, the action route. Google issues POST requests to the route with access token in an Authorization header. The post data is JSON and includes a requestId and a list of intents. The intent specifies if it is Sync, Query, Execute or Disconnect.

Sync does not send any additional data, but simply expects to have the devices associated with the user returned. For this, I lookup the devices associated with the tesla account (as given by the access token) and return the device data mapped to data expected by Google, including an id that can be used to identify the device when issuing commands to the Tesla API in the future. For each device Google also expects the type of device, and its traits. The device type is not really important as far as i can tell although it will dictate what it looks like in the Google Home app. But the traits specify what commands can be sent by google assistant. In my case I want to be able to the the heat on or off, so I picked the device type action.devices.types.HEATER (CAR doesn’t exist as a type), and the trait is actions.devices.traits.OnOff.

For Query, Google sends a list of devices it wants the status of. Here I simply iterate through the list, and fetch the status of the climate of each device. (The Tesla API has four distinct states fetched through four different routes: climate, drive, vehicle, and charge.) With a device trait of OnOff, Google expects a state named on to be true or false. I set it to true of the climate is on.

Finally, (I don’t bother with Disconnect as there is nothing to do), I implement Execute. Execute is a bit of a handful as it will send a list of commands as a set of devices and executions. Here I cheat majorly and assume there will be only one command, with one device, and one execution. The execution of action.devices.commands.OnOff carries a parameter of on that is set to on when the command is to turn the device on. Depending on this value I either turn the climate on or off through the Tesla API. Finally, Google expects you to respond with the final status of the commands for each device.

Finally, when setting up the Action on the Actions on Google Console, you simply add the auth and token routes for your Oauth server, and the action route for your Action. When complete, hitting test will set the action up for you to use through your own google assistant only.

All in all, there are a few concepts required to learn around Actions on Google, but once you understand those, enabling you to control something through google assistant is really as simple as writing a small rest API. Next I might add more capabilities, such as adding both the car heater, and the car charger, so that I can start or stop charging at home. I might also add the capability to set the temperature I want in my car, although I very rarely change this so it’s very far down my todo list.

And here’s the code.

Posted in Uncategorized

Hass.io addon for backup to S3

I’ve been playing around with Home Assistant lately. I was surprised to not find a way to make external backups automatically. Most what I read recommended making snapshots and copy the snapshot over a samba mount manually. I wanted a way to directly make a backup and upload it to AWS S3. So late last night i threw an addon together. It can be found on github here.

Posted in Uncategorized

In Sweden, Digital strategist at Iteam

I have not updated this in a loooooong time. Figured I should at least add some minor updates before I write any other posts.

In January 28, 2017, me and my family left Glasgow and Scotland to move to Sweden. We had no plans on what to do so I did some freelance stuff. (30 years of programming has some benefits these days.) Not long after I decided to join Iteam as a digital strategist. It meant I left academia for industry. I no longer do research, but instead help companies and organisations think in digital terms – what change does digital mean to existing business and practices. My long stretch in looking into the future by applying research through design, now benefits my ability to help seeing opportunities and what is possible for existing businesses and organisations as well as new enterprise. My long experience in building stuff indeed comes in handy at times, where I mentor more junior colleagues and help strengthen our technology skills.

I now live in Veddige, close to Varberg, south of Gothenburg, on the Swedish west coast. Feel free to give me a shout if you want to meet up!

Posted in Uncategorized

Workshop at NordiCHI on Mobile Wellbeing

We will organise a workshop at NordiCHI later this year in Gothenburg, Sweden. The workshop is called Mobile Wellbeing and is to be focused around mental wellbeing and use and design of mobile technology. The workshop is organised by me, John Rooksby (UK), Alexandra Weilenmann (SE), Thomas Hillman (SE), Pål Dobrin (SE), and Juan Ye (UK). If you are interested in the workshop theme, consider writing a position paper and join us in Gothenburg on October 23. The deadline for position papers is August 25.

You should visit the workshop website for more information, but at the workshop we aim to discuss the following three questions:

  • In what ways do mobile technologies affect mental wellbeing?
  • How can mobile technologies be designed to support and improve mental wellbeing and to mitigate negative effects?
  • What strategies and practices can be developed for using mobile technology in ways that do not harm and instead support improvement of our wellbeing?

See you in Göteborg!

 

Posted in Uncategorized

Forget-me-not: Honorable Mention at CHI 2016

presenting_forgetmenotLast week I went to San Jose for a week to attend CHI. Among other things I got to present my note Forget-me-not: History-less Mobile Messaging. This note received an Honorable Mention which is given to the top 5% papers and notes at the conference.

The paper is based on work done by a group of level 3 students. The group project was to design, build, and study a mobile text messaging app without history. This is what they did and turned into the app forget-me-not. The student project lasted a year, and throughout the project the students received first best presentation at the intermediate project presentations at half-time, and finally they received best L3 project in Computing Science in the school. I could not be prouder of these students. Topping that up with also being able to turn the work into a paper receiving an honorable mention surely is the cherry on the cake!

The final paper discuss mobile text messaging and the role of messaging history. To do that we study what happens with mobile text messaging when there is no history. By interviewing 10 participants after using the app for 2 weeks, we gain insights into how they perceive the app, and how they perceive messaging through it. We found that messaging requires effort, but allows users to be relaxed about what they write. History turns out to be useful in the ability to “scroll up” to see the past messages which allows for distractions. Not having history instead makes it more engaging. It is also discussed how not having history allows for sending messaging you don’t want to have on record, such as bank details, or planning a secret birthday party. Read the whole paper, where we discuss the design of the app, and discuss the method of deliberately removing history to study history.

 

Posted in Uncategorized

Pass The Ball at CHI 2015 in Seoul

This morning I presented a paper at CHI entitled Pass The Ball: Enforced Turn-Taking in Activity Tracking. See the abstract below

We have developed a mobile application called Pass The Ball that enables users to track, reflect on, and discuss physical activity with others. We followed an iterative design process, trialling a first version of the app with 20 people and a second version with 31. The trials were conducted in the wild, on users’ own devices. The second version of the app enforced a turn-taking system that meant only one member of a group of users could track their activity at any one time. This constrained tracking at the individual level, but more successfully led users to communicate and interact with each other. We discuss the second trial with reference to two concepts: socialrelatedness and individual-competence. We discuss six key lessons from the trial, and identify two high-level design implications: attend to “practices” of tracking; and look within and beyond “collaboration” and “competition” in the design of activity trackers.  

Posted in Uncategorized

Ffmpeg and Chromecast

I’ve been struggling recently with transcoding media files and streaming to Chromecast. Starting with the excellent project castnow over on github I wanted to be able to 1) stream media files directly from rar archives, and 2) create a web interface to start media files. It is not meant to be overly ambitious but just something useful to use at home.

Among several problems I’ve encountered so far, one has been especially annoying and turned out to have a very simple fix. The transcoding is done using ffmpeg. What I’ve been doing is to let ffmpeg reencode any media file I give it, into an mp4 with h264 and aac. This works most of the time, however for some mkv-s there’s been no image when transcoded. Casting the mkv directly to the chromecast gives you moving pictures, but it has no sound (since Chromecast does not support 5.1 audio as of yet as far as I understand).

The first attempt at a solution was to then not reencode the video but to simply copy the original. That is simple using ffmpeg flag: -vcodec copy. Unfortunately this still doesn’t work. However encoding the video to a file and then casting the file to the chromecast works. Thus there was something going on when the output from ffmpeg is streamed directly. I’ve still not worked out what is going on, but I’ve found a solution. Instead of creating a mp4 container, encoding everything into a mkv (or matroska) suddenly makes everything work just fine. The final line is


cat some-media-file | ffmpeg -i - -f matroska -vcodec h264 -acodec aac -ac2 pipe:1 | stream-to-chromecast

So far this seems to work all the time, however it is somewhat unnecessary to encode h264 if the video is already h264. In my project I therefor check codecs and set the flags for ffmpeg accordingly.

The project is written in Node.JS and is available on the following github repository.

Posted in Uncategorized

Why and How to Quickly Build Apps – Make No Decisions

Today I was invited to give a talk at the Mobile Apps Group Meetup in Glasgow. I decided to talk briefly about my own app development in my research, why it involves quickly building apps, and how I tend to do that.

I first gave the premise of my work: To understand an app (or the ideas manifested in the app), it needs to be built, so that it can be studied in use. In my view, we can only know what an artefact is once it is in use. We cannot know what it is prior to that.

I then explained how I suggest people to do that. None of the points are anything new, but it is hopefully something that people will start doing once they hear it often enough so I figured it is worth talking about. Concisely I’m trying to convey that you should make decisions when they are easy to make and refrain from it when it is time consuming.

Thus the process is:

  • Sketch a lot of ideas. Sketch on paper or in any other material that is easy to produce and easy to discard.
  • Make a mockup using Sketch, Photoshop, or anything else that allow you to create what you want different screens of your app should look like. This is where decisions are made. From the sketches made previously, pick one, make it into images that will say exactly what the app will look like on screen. The purpose of this mockup is to describe what is to implemented in the next phase.
  • Now you build. But make no decisions what so ever. Just transfer the decisions manifested in the mockups, down to the font sizes, margins, and colours. As soon as you start playing around with margins, font sizes, and colours, you start loosing a lot of time. The reason is because it is not as easy and efficient as it would have been if you would have done this already when creating the mockups, so you should not do it now! If it makes it easier, pretend that the mockups come from a paying customer who pays you to implement an app that looks exactly like he has decided and you have no room for creative suggestions.

In my experience, following this simple rule of making all decisions while creating the mockups, and making no decisions when implementing, makes implementing it a breeze. I think one of the reasons for this is because when you try to make decisions in the implementation phase you not only need to think about how you would like it to look, but you also need to think about how to make it so. Having the decision made means you only need to think about how to make it.

Posted in Uncategorized

Lived Informatics at CHI, Toronto

Today I presented our paper ‘Personal Tracking as Lived Informatics’ at CHI. By interviewing 22 people about how they were using personal tracking devices, such as Nike Fuelband, Jawbone Up, FitBit, and mobile apps (e.g. RunKeeper and MyFitnessPal).

Among the findings, we uncover how the people we talk to use multiple trackers, and track multiple things. They switch between trackers, as well as what they track, over time. While they say that they do not share tracked data to social networks, they do track together with people in their lives. Furthermore, while they track for a long time, they rarely look at their historical data.

We discuss our findings and present an alternative view to Personal Informatics which we term Lived Informatics. Lived Informatics emphasises the emotionality of tracking and that tracking is something done with an outlook for the future, rather than part of a rational scientific process for optimising self as seen in Quantified Self and in Personal Informatics.

We are very happy the paper got an Honorable Mention.

The paper:

Rooksby, J., Rost, M., Morrison, A. and Chalmers, M. (2014) Personal Tracking as Lived Informatics. To appear in Proceedings of CHI’14, Toronto, Canada.

Posted in Uncategorized

Guest lecture in Gothenburg

On friday I gave a guest lecture in a course at LinCS at the University of Gothenburg. The course was ‘Understanding and designing for social media practices’ and is aimed at exploring methods for studying online social media.

The lecture was based on two previous studies of foursquare. The first one was a primarily interview based study where we looked at performative aspects of using foursquare. The second one deals with a study of log data from foursquare. The talk discussed what it means for people to share their location through the means of checking in (or what it means to check in), and discussed what this means for the genera of data, and showed examples of communicative features in the log data from foursquare.

The slides can be found on slideshare.

Posted in Uncategorized