Controlling Hue bulbs with Insteon Dimmers using Node Red and Home Assistant

The house I recently moved into contains 40+ 6" recessed lights all containing incandescent or halogen bulbs ranging from 120 to 150 watts each. Basically they had flood lights installed in the house. These lights were all extremely bright and completely inefficient. I am in the process of replacing all of them with these Hue Downlights, which are only 9 watts each, and plenty bright at 650 lumens. On top of that I can now control the color or temperature of every light in the house.

In my previous house I controlled Hue lighting from Indigo and used physical Insteon dimmers. I want to achieve the same thing in the new house, but using Home Assistant. I feel like I caused myself a bit of headache by wanting to use the Insteon dimmers, because they have lights on them that show the brightness level., and I am not able to handle this light not properly reflecting the brightness of the Hue bulbs being controlled. I also wanted to be able to make them be able to dim or brighten the hue bulbs by holding down the dimmer paddle like intended with standard bulbs.

Whats been done so far?

I've successfully got the Insteon dimmers setup and working with Home Assistant using Insteon-MQTT running in a Docker container. I'm able to control the dimmers through the Home Assistant UI and see their state reflected by interacting with them manually. You can read about how I set this part up here:

Time to actually control something!

My first attempt at making all of this work was by using normal HA automations written in YAML. In the beginning it seemed like this was going to work without any issue. Turning the lights on and off was extremely straight forward. 15 minutes later if thats all I wanted I would have been done.

Why it got complicated

The Insteon Dimmers when brightening or dimming only send payloads of in UP, DOWN, and STOP. They do not know their actual brightness level, and only report that level after they send STOP. If I actually wanted to change the brightness I needed to kick off a script that would increase or decrease the brightness at an interval until the STOP payload was received. This seemed easy enough, and I did get that working as well. There is even an example of it in the Home Assistant docs. This is now the point where I'm my own worst enemy and everything fell apart. After the STOP payload came through I'd stop the script, and then the dimmer would report its brightness and since I was changing the light brightness independently of the dimmer brightness things would be out of sync and the light would either brighten or dim to match the dimmer itself. I got it pretty close by adjusting the timing and amount the brightness changed each loop, but it was never quite right. Here's an example of just the yaml file for brightness changing from the dimmer.

- id: dimmer_brightness
  alias: "Dimmer Brightness Change"
  trigger:
    platform: state
    entity_id:
      - light.dimmer
  condition:
    condition: template
    value_template: >-
      {% set manual_state = state_attr(trigger.entity_id, "manual_state") %}
      {% set is_not_manual = (manual_state == "STOP" or manual_state == "None") %}
      {% set dimmer_on = (trigger.from_state.state == 'on' and trigger.to_state.state == 'on') %}
      {% set from_off_to_on = trigger.from_state.state == 'off' and trigger.to_state.state == 'on' %}
      {% set brightness_change = trigger.from_state.attributes.brightness != trigger.to_state.attributes.brightness %}
      {% set needs_update = state_attr(state_attr("sensor.insteon_map", state_attr(trigger.entity_id, 'address')), 'brightness') != trigger.to_state.attributes.brightness %}
      {{ dimmer_on and brightness_change and is_not_manual and needs_update  }}
  action:
    - service: light.turn_on
      data_template:
        entity_id: >-
          {% set address = state_attr(trigger.entity_id, 'address') %}
          {{ state_attr("sensor.insteon_map", address) }}
        brightness: >-
           {{ trigger.to_state.attributes.brightness }}

Does that explain why I felt it got too complicated? I needed to check all sorts of different conditions and try to present that in a sane way through yaml. I started to go slightly crazy. In the end I had 9 yaml files with the different parts of the automations to keep everything in sync.

What were my options?

I'm sure there are plenty of options, but the two I looked into were Node Red and AppDaemon. My initial instinct was to try and use AppDaemon since I'm relatively comfy writing Python, and it's a familiar path to have to take from my experiences using Indigo. I ended up deciding to give Node Red a try first though, since the barrier to entry as well as the visual representation was appealing and seemed like it might be easier to manage later once I've forgotten how everything works at a later date.

So, I Installed and Setup Node Red

Adding the Dimmers to HA

The dimmers are added pretty much as you'd expect based on how I walked through setting up insteon-mqtt with home assistant. There are a few differences though, which are the addition of json_attributes_topic which is going to be how we are able to track the manual interactions, as well as the addition of a unique_id for no reason other than I didn't like the UI complaining to me about it not having one.

- name: dimmer_hallway
  platform: mqtt
  schema: json
  state_topic: "insteon/41.b3.5e/state"
  command_topic: "insteon/41.b3.5e/level"
  json_attributes_topic: "insteon/41.b3.5e/manual_state"
  brightness: true
  unique_id: 41.b3.5e

There will be an additional attribute that will show up called manual_state which will say STOP once you've released the dimmer paddle.

Creating a Generic HA Config and Node Red Flow

My goal was to make it so that for 99% of the use cases I would be able to simply add a new hue bulb and "pair" it to a dimmer switch and have it work with little effort. To do that I ended up with the concept of sync_entity_id, sync_address, address, sync_entites, sync_dimmers, and sync_bulbs.

There very well may be a much more straight forward way of doing it but I found myself in the situation where I needed a place to store some additional meta data that was manually configured for my lights and dimmers. To do this I heavily utilized the flexibility of customize.yaml.

#----------------------------------------------------------
#         Hallway Dimmers
#----------------------------------------------------------
light.dimmer_hallway:
  address: 41.b3.5e
  is_sync_dimmer: true
  sync_entity_id: light.hallway
  friendly_name: Hallway Dimmer

light.dimmer_hallway_master:
  address: 41.b8.f6
  is_sync_dimmer: true
  sync_entity_id: light.hallway
  friendly_name: Hallway Dimmer Master

#----------------------------------------------------------
#          Hallway Bulbs
#----------------------------------------------------------
light.hallway:
  sync_address: 41.b3.5e
  is_sync_bulb: true
  sync_entities:
    - light.dimmer_hallway
    - light.dimmer_hallway_master

Hallway Example of customize.yaml

What are these used for?

These probably wont make a ton of sense at the moment but will be more clear once we get into the Node Red flow.

Dimmers

address - the actual insteon device address
is_sync_dimmer - so we know if the dimmer is supposed to talk to a bulb
sync_entity_id: id of the bulb to reference

Bulbs

sync_address - the address of the insteon device to control
is_sync_bulb - so we know if the bulb is supposed to talk to a dimmer
sync_entities - list of dimmers to update when the light changes

Creating the Flows

Now that the Home Assistant configuration is handled its time to create the flow. The main thing to remember here is that the state of the bulb is always the most important truth. By taking this approach we wont end up in any weird update loops and also still have the ability to use any third party apps to control the bulbs and the dimmers will still respond appropriately. You can download the Entire Node Red Flow otherwise I've broken it up into chunks that are linked to as we go.

Bulb Flow

Here's what the flow for the lights changing looks like, and you can download it here: light-flow.json

The first thing that happens is we check if the bulb is a sync_bulb, and if it is we move on to a function node which gets the list of sync_entities so that we can update their state.

let attrs = msg.data.new_state.attributes;
let payload = msg.payload;

if(attrs.hasOwnProperty("sync_entities")) {
    msg.payload = attrs.sync_entities;
    msg.original_payload = payload;
    return msg;
}

Sync Entities Node

Next those entities are piped through the split node, so that we can go get each dimmer individually using its it. The split node sends an individual msg for each entity_id in the sync_entities list. It sets msg.payload to each of those entity_ids, which go through a ha-get-entities node that uses the entity_id as its property its searching on.

Finally there is a check using a switch node that looks at the original payload to see if the bulb was turning on or off. From there if its off we turn the dimmer off, and if its on there is one last check to make sure the bulb wasn't changing state due to a manual interaction with the dimmer.

let attrs = msg.payload.attributes;
let hasManual = attrs.hasOwnProperty("manual_state");

if(!hasManual || hasManual && attrs.manual_state == "STOP") {
    return msg;
}

Stop Node

Dimmer Flow

The dimmer flow is similar to the bulb flow but even more straight forward. You can download it here: dimmer-flow.json

It starts out exactly the same where the first node is looking for events coming from any entity starting with light.dimmer. The next node is a simple switch checking if it is a sync dimmer. The Stop node is the same as the one from the bulb flow, which is checking to make sure the dimmer isn't being manually interacted with.

Finally we check if its an on or an off command with the on/off switch node. If its off then we simply turn the bulb off, but if its on we make one check to make sure the bulb is actually off before turning it on. This was done to handle the edge case of using a bulb standalone as well as in a hue light group. Turning the bulb on individually in hand turns on the group as well. When the group turned on it triggered an update of the dimmer from the bulb, and if you don't check against the light group being on already it would turn on the whole group. (thats confusing, and I'm not sure a better way to explain...)

Manual Dimming

This is where things started to get a bit more tricky. Now I wanted to actually start handing the manual interaction with the dimmers, meaning if I hold down the dimmer paddle I now want the bulbs to start dimming and then stop when I let off the paddle. That sounds pretty straight forward, but it ends up becoming a bit of a headless interaction because we will be getting zero feedback about state until the paddle is released, so the speed and way it behaves is all based on my best guess and referencing what I used to have when I did a similar thing using Indigo. I used the timing from this Indigo Hue Plugin. You can download the flow here: manual-dim-flow.json

For the manual interactions we will be responding directly to the MQTT messages that are coming over the manual_state channel, and the first node will simply be looking for the STOP payload and if it's received it will kill the 400ms timer.

If the payload is not STOP then it's a matter of looking up the bulb that listens to this dimmer, by using the dimmer address and looking up the bulb that has that address set as its sync_address. Once the light is found I decided to create a JSON object of "instructions" that will be iterated over by the timer. The below javascript is what is found in the Dim/Brighten function node.

let light = msg.payload;
let step = 30, max = 20, transition = .4;
let curBrightness = 0, nextBrightness = 0;

msg.instructions = { "increase": true, "transition": transition, "step": step, "count": 0, "max": max, "brightness": 0};

//brightness attribute may not exist if brightening from off state
if(light.attributes.hasOwnProperty('brightness')) {
    curBrightness = light.attributes.brightness;
}

if(msg.manual_state == "DOWN") {
    nextBrightness = curBrightness - step;
    nextBrightness = nextBrightness < 0 ? 0 : nextBrightness;
    msg.instructions.increase = false;
} else {
    nextBrightness = curBrightness + step;
    nextBrightness = nextBrightness > 255 ? 255 : nextBrightness;
}

msg.instructions.brightness = nextBrightness
msg.light = light;

return msg;

Dim/Brighten Node

It creates an instructions object containing properties saying if its increasing or decreasing brightness depending on if manual_state is set to UP or DOWN, how fast the transition is, the step of how much to change the brightness, and the brightness to be set. Next immediately start updating the brightness of the bulb so there isn't a feeling of delay.

From here the stoptimer node will get kicked off and it will fire every 400ms. The Update Instructions function node will then use and update that instructions object to increase or decrease brightness appropriately and also killing the timer if the button is held down longer than it takes to completely turn on or off the bulb.

let inst = msg.instructions;

inst.count++;

if(inst.count >= inst.max || inst.brightness > 255 || inst.brightness < 0) {
    msg.payload = "stop"
    return [null, msg]
}


if(inst.increase) {
    inst.brightness += inst.step;
} else {
    inst.brightness -= inst.step;
}

return [msg, null];

Update Instructions Node

Manual Fast On

There is one final interaction that I wanted to handle which is the simple fact that if the light was on and the dimmer was on, and I tapped the dimmer again to increase its brightness all the way to 100% nothing would happen.

This would come through as a payload mode of normal and a state of ON. You could also add things to this such as support for "fast" which would be a double tap of the switch for an instant on or off message. I decided to handle fast in the same way I handled normal. You can download the flow here: manual-on-off-flow.json

First uses a get entities node to find the dimmer entity being pressed using its address. Once we've got the dimmer, it's time to figure out if the dimmer was manually pressed and was already on. This is done in the Manual Tap node.

let payload = msg.payload;

msg.dimmer = msg.dimmers[0];
delete msg.dimmers;

let device_on = payload.reason == "device" && payload.state == "ON";
let device_off = payload.reason == "device" && payload.state == "OFF";
let fast = payload.mode == "fast";

let fast_on = msg.dimmer.state == "off" && device_on && fast;
let fast_off = msg.dimmer.state == "on" && device_off && fast;

let fast_on_from_on = msg.dimmer.state == "on" && device_on && fast;
let fast_off_from_off = msg.dimmer.state == "off" && device_off && fast

let on_from_on = msg.dimmer.state == "on" && device_on

if(fast_on || on_from_on) {
    msg.brightness = payload.brightness;
    return msg;
}

Manual Tap Node

There's more script there than necessary to solve what I actually needed, but I figured I'd try and capture a few of the various states incase I ever wanted to respond to them differently in the future.

Closing Thoughts

Overall I'm pretty happy with my setup. It's certainly not perfect but 99% of the time it works exactly as you'd expect. One major consideration to be made is you really need to commit to the idea that if the mqtt broker, insteon-mqtt, or home assistant isnt reachable the switches will do nothing. I've spent some time figuring out the best way to mitigate this, and my current solution is nothing runs on wifi, and everything is connected to a UPS. Obviously if the power goes out so do the lights, but the UPS is in place to prevent something from not rebooting properly during a short power outage as well as piece of mind that something wont get corrupt by shutting down improperly.

Another is how I'm handling manual dimmer changes. In the current setup the MQTT node is listening to insteon/+/manual_state which is any dimmer in my house. If someone were to manually dim one at the same time as someone else, whoever stops first will stop both lights. After writing this I'm not 100% sure, but there may even be an issue using two switches at the exact same time. I've never tested but assume bad things happen. At the end of the day its only myself and my wife living in this house and the likelihood of this happening is slim.