Filtering DHT temperature sensor

I have some temperature sensors hooked up to Pi’s in the chicken coop and the basement, but was getting some nasty readings. The sensor in the coop would give erratic values of 5% humidity a few times an hour and the sensor in the basement was continuously dropping to zero.

Before filtering, results from my coop

I’m assuming you have the sensors hooked up already, if not the posts I got my inspiration from are linked at the bottom and there is a how-to.

The Idea of filtering the results is explained in the links as well, but in short: By leaving out all results outside of the standard deviation and calculating the means you get more consistent results.

In my case I collect 60 values, calculate means and standard deviation and exclude all erratic values. I use the Adafruit_DHT library to read the sensor, math and numpy to do the calculations and urllib.request to send the values to Domoticz. You can install all of those with pip3 (sudo pip3 install).

Here is the code:

import math
import numpy
import threading
from time import sleep
from datetime import datetime
import sys
import Adafruit_DHT
import urllib.request

# parameters
DHT_type    = 22
OneWire_pin = 4
sensor_idx  = 93
url_json    = "http://192.168.1.104:8080/json.htm?type=command&param=udevice&idx="

filtered_temperature = [] # here we keep the temperature values after removing outliers
filtered_humidity = [] # here we keep the filtered humidity values after removing the outliers

lock = threading.Lock() # we are using locks so we don't have conflicts while accessing the shared variables
event = threading.Event() # we are using an event so we can close the thread as soon as KeyboardInterrupt is raised

# function which eliminates the noise
# by using a statistical model
# we determine the standard normal deviation and we exclude anything that goes beyond a threshold
# think of a probability distribution plot - we remove the extremes
# the greater the std_factor, the more "forgiving" is the algorithm with the extreme values
def eliminateNoise(values, std_factor = 2):
    mean = numpy.mean(values)
    standard_deviation = numpy.std(values)

    if standard_deviation == 0:
        return values

    final_values = [element for element in values if element > mean - std_factor * standard_deviation]
    final_values = [element for element in final_values if element < mean + std_factor * standard_deviation]

    return final_values

# function for processing the data
# filtering, periods of time, yada yada
def readingValues():
    seconds_window = 60 # after this many second we make a record
    values = []
    
    while not event.is_set():
        counter = 0
        while counter < seconds_window and not event.is_set():
            temp = None
            humidity = None
            try:
                humidity, temp = Adafruit_DHT.read_retry(DHT_type, OneWire_pin)

            except IOError:
                print("we've got IO error")

            if math.isnan(temp) == False and math.isnan(humidity) == False:
                values.append({"temp" : temp, "hum" : humidity})
                counter += 1

            sleep(1)

        lock.acquire()
        filtered_temperature.append(numpy.mean(eliminateNoise([x["temp"] for x in values])))
        filtered_humidity.append(numpy.mean(eliminateNoise([x["hum"] for x in values])))
        lock.release()

        values = []

def Main():
    # here we start the thread
    # we use a thread in order to gather/process the data separately from the printing proceess
    data_collector = threading.Thread(target = readingValues)
    data_collector.start()

    while not event.is_set():
        if len(filtered_temperature) > 0: # or we could have used filtered_humidity instead
            lock.acquire()

            # here you can do whatever you want with the variables: print them, file them out, anything
            temperature = filtered_temperature.pop()
            humidity = filtered_humidity.pop()
#            print('{},{:.01f},{:.01f}' .format(datetime.now().strftime("%Y-%m-%d %H:%M:%S"), temperature, humidity))

            cmd = url_json  + str(sensor_idx) + "&nvalue=0&svalue=" + str(temperature) + ";" + str(humidity) + ";0"
#            print (cmd)
            urllib.request.urlopen(cmd)

            lock.release()
        # wait a second before the next check
        sleep(1)

    # wait until the thread is finished
    data_collector.join()
    
if __name__ == "__main__":
    try:
        Main()

    except KeyboardInterrupt:
        event.set()

You can also download this from my github.
On your pi do ‘wget’ and paste the URL using CTRL=SHIFT+V

I run the script on startup and get results every minute and a bit.

After almost 24 hours, the graph in Domoticz now looks like this:

Graph after filtering.

I’m happy with the result and I hope you will enjoy a cleaner graph as well. Like I said, I only connected the dots between a tutorial to read the sensor and send data to domoticz and a how to to filter DHT results

Let me know how it worked for you!

OTGW on remote pi with ser2net

Now I am building my own domotica system with Domoticz, I bought and built a open therm gateway (OTGW) to intercept and modify the communications between my boiler and the Honeywell Round thermostat I have in the livingroom.

The situation is as follows, I have a rPi in the basement, connected to an RFLink and the smart meter. (The water meter is on the to-add list.) In the attic I have a rPi running some other software and a Synology NAS. The latter fails in running Domoticz and device sharing seems tedious with two different devices so I went with the pi.

Connected everything, setup device sharing per the manual and everything went fine. Or so I thought. Every time I changed the setpoint on the master Domoticz, a new device called ‘setpoint’ would appear on the slave and the setpoint would not be changed on the thermostat/boiler.
Googling this problem got me on two tracks: Write a script on the slave that fixes this or use the OTGW remotely via SER2NET. I chose the latter because I dind’t like the extra software on the pi in the attic anyway and this seemed less of a hassle.

Installing ser2net

SSH into your device and enter the following:

sudo apt update
sudo apt upgrade
sudo apt install ser2net -y

Find out where your OTGW lives and enter:

dmesg | grep tty

To be honest, I couldn’t make out which was the OTGW but since the other USB device is a hdd, I made an educated guess and it worked. I went with /dev/ttyUSB0
Next we will edit the ser2net config by opening the file.

sudo nano /etc/ser2net.config

In this file at the bottom, enter:
4000:raw:0:/dev/ttyUSB0:9600,NONE,1STOPBIT,8DATABITS
which breaks down into: ‘PORT:STATE:TIMEOUT:DEVICE:OPTIONS’ where 0 disables timeout and our options are the baudrate, parity, number of stop bits and number of data bits

Next, add a hardware device to your domoticz choosing ‘OpenTherm Gateway with LAN interface’ for type and connecting to the IP of your device and the port (4000 in our case*) you specified.
Allow new devices for 5 minutes and you’ll see the devices come to your Domoticz server!

* Don’t forget to open this port in your firewall if you have any. In case of UFW

sudo ufw allow from 192.168.1.0/24 to any port 4000 comment 'Allow OTGW ser2net on local lan'

Have fun!

Setting up internal DNS(+cache) on Synology NAS

First thing I did when I received my second hand NAS I bought a completely useless domain name to access it via the internet. Although we have a dynamic IP address I don’t bother setting up DDNS but just change the dns settings from my provider when my IP changes.

Why internal DNS?

That said, I wanted applications that work via the internet outside my home to work the same way in home. It had to, considering the WAF factor. Until now I had to use the internal IP on my network and the domain name from outside our network.

After some googling it ended up being rather simple, but there are a lot of versions out there and I thought I’d just add another one to complete the chaos we all love so much.

In time this setup will be replaced by a raspberry Pi running PiHole, but that will be in a later post.

How did I do it?

I installed the DNS server package. No need to open or forward ports since everything is inside your local network.

After installing I created a new master zone, with my domain name as domain name and host id. It automatically cointained the NS and second A record. I only created the first A record forwarding the domain name to my Synology.

The next step probably isn’t necessary, because most devices and routers can handle two or more dns servers, but I turned on the resolution service and chose the cloudflare dns service to use if my Synology doesn’t know the record a client is looking for.

Now I just had to change the dns settings in my router to look for the record on my Synology first and renew the lease on all devices so apps like DS file could also use the domain name to login when inside my home network.

A little bonus

The following adds a little bonus to the fact that you use your own dns server. The synology can cache your queries so next time you visit a website you visit before your Synology will know immediately where to send you and doesn’t have to look it up first.

The following settings work for me, but everyone uses their system differently so maybe you could do with a smaller cache or need a bigger one or you need the resources for something totally different.

There you go! You can now set up everything pointing to the same domain name and made your home network faster in the process. If you have any questions or remarks, din’t hesitate to leave them here and I will get in touch asap.

The Genesis blog

A.k.a. my first blog post on this website. Don’t expect a whitepaper or something. I just needed some space in my head and that’s best done by writing stuff and since writing on a piece of paper results in tons of head scratching wondering what the hell I wrote down. I decided it was best done in digital form and then, why not on the internet.

I love browsing other peoples how-to’s and always find my own situation just a tad bit different but often lack the balls to comment or just don’t want to sign up to yet another service to do so.

Since I don’t like (having) a lot (of different) internet services I try to do most things myself. Same goes for this website, it is hosted on my Synology NAS in the attic. I have just ordered my first Raspberry Pi and have some neat plans and love reading books. In English and in Dutch so some of my posts will be in Dutch as well.

I’m an autodidact and I’ll love to learn to code and write my own software so that’s what I’ll try to do the coming year(s). There’s not a lot of spare time in my schedule, but you’re welcome to join me in my journey or just tag along some trips.

Happy browsing and please don’t hesitate to ask any questions you might already have or to send me your remarks. I’ll try to make it as easy as possible.