Wednesday, 16 April 2014

Weather data website

My first blog post was about reading data from weather sensors. I used the rtl_433 package. I did not have a conclusion. I got it to work. I did not capture any rtl_433 compatible sensors near me. I had an idea to run a webserver that’s displays a temperature captured with a SDR.

I reinstalled my homemade antenna. It’s a dipole antenna for CB frequencies. It’s and pretty wide range antenna so it captures all kinds of frequencies pretty well. Had some fun with it. I was scanning through the 430~ region and noticed that this antenna picks those frequencies pretty well. I remembered all the rtl_433 fun and tried it again.

Installed the rtl_433. It did not take long when it started decoding supported weather probes. At first I picked up only one but we’ll get back on that later.

temp

As you can see it also sends other fun data.

When I launched my command “rtl_433 –p 35 2>rtl433test.txt”

The “–p 35” flag deals with the frequency correction because rtl-sdrs are not perfect. I used  kalibrate-rtl to calibrate. It found out that my rtl-sdr error is 35 ppm.

It outputted everything in the text file.

PYTHON

Now I needed something to extract the necessary info from the text file. I used python to do the job. Little Google search lead to results and also used what I have studied lately.  It is HERE I know this thing is not very efficient. Probably there is a better way to solve my problem, but that’s the thing I made and it works. It imports the 433test.txt, searches for the terms „humidity” and “temp”. Then it prints the whole line where the term is present. Instead of printing the line it saves it into an html file. Had problems with it saving into txt and then presenting them in the website. So I used html because it is just almost plain text.  It gets the info from 433test.txt, outputs the necessary values to “temp.html” and “niiskus.html” and then waits for 60 seconds.

HTML page:

I used the power of Google and some free tools to craft that webpage.  The most important things are of course the temperature and humidity. The other stuff I added because of fun. Also I presented the path it takes to present that information. The page automatically refreshes after 60 seconds so you don’t have to refresh it manually.

Server:

Domain part was simple, registered a free domain called kukktemp.tk – go figure. Used its free DNS feature and pointed the DNS to my home IP.

I needed something to run this webserver on. I picked up my Raspberry Pi and installed apache2 on it. I did not need any MySQL or other stuff. I only needed the webserver. I configured the apache server and other parts.

Port forwarded the Pi and VOILA the Pi is available to the internet. I also forwarded the SSH.

I ran the commands and it started working surprising or that’s what I thought.

PROBLEMS:

I started noticing that sometimes it reports the temperature as 0 degrees. I found out that some kind of transmitter, that is picked up by the rtl_433 but it is glitches so it only shows 0 degrees. Enough to mess with my data. I had to thoughts how to solve this problem. ONE. Edit the python script or TWO edit the rtl_433 files.  With python script it would glitch when the sensor reports 0 degrees legimately so I moved to editing the rtl_433 files.

I noticed that, that phantom sensors is using a different protocol than the one that sends good data. So again I downloaded the rtl_433 from github and started poking at it.

When I removed the whole protocol it stated screaming because it does not find the references it needs. So then I looked for the part where it actually prints the data to the screen. I found the protocol I needed and deleted the lines where I normally prints the temperature. Also then I noticed I was able to do  even more that deleting  lines.  I saw that I was able to edit everything. Next time when I make a new revision, I probably would make new keywords and stuff so the python would have an easier time picking it up. We’ll see. It has running strong for the past days. I have no intensions to pull the plug. Let’s see where this thing leads us.

kukktemposx

KUKKTEMP.TK

 

EDIT 20.04.2014

Added a line in the rtl_433 files so it would also print the time and date for further logging and data collection.

kell

 

EDIT > 26.04.2014

I have noticed over a week that the website does not work after a while. The problem: that file what python need to process gets really long. I mean line 100k lines, then it does not work anymore. So I modified the code so at the beginning it would make a backup and then with that backup when it’s done reading data it cleans it. So every time it has only couple of lines to process. Let’s see how long that works! You can see the change HERE

EDIT > 3.05.2014

Made an Android application that basically serves the same purpose as the website itself. Only now it is much simpler to check the data out.

Couple of days ago I added the log to the website. It scrolls automatically to the end. It sounds simple but it pretty difficult to get working.

Here’s a screenshot:

kukktempapk

Kukktemp.apk

 

The website has ran almost two weeks I think. A lot of data has been gathered.

EDIT: 10.06.2014

It has ended. Everything is HERE.