Nowadays there is no possibility not to read about the new next thing: Internet of Thins (IOT). As a Java developer who mainly develops web applications and the backends IOT is not the daily business (at least in my case) but if you search hard enough you’ll find a usecase.
Usecase
In my case the usecase was my new office. In winter it was very cold, in summer it was really (really!) hot so I had the idea it would be a good idea to have a temperature logger to let my bosses see how hot it was in the office. Additionally I wanted to log the outer temperature to correlate it with the temperature inside the office.
Of course I did not want to buy a completed solution because:
- I wanted to try out IOT on my own
- I wanted my data not to be pushed somewhere into the cloud
- I’m stingy
So I decided to build up my own solution.
Technics
As this should be a learning project I wanted to use
- spring-boot
- a MQTT message broker
- ELK stack
- my old Raspberry Pi model B
Docker
and it was clear, the technical solution would not be trivial but complex and excessive. In the end the technical draft looked like this:
Docker was removed from my requirements list – took too much time to learn.
Implementation
Upgrade of my Virtual Private Server (VPS)
The first thing (yes, I know, should have been the last step…) was to upgrade my VPS. All the things I wanted to install needed more HDD space and RAM I had, so I upgraded to a VPS providing
- 1 virtual CPU core
- 2 GB of RAM
- 40 GB of HDD
This small machine seemed to be enough to run all pieces of my solution plus some additional things. My suggestion were correct, the final version runs smoothly with this hardware.
Raspberry Pi as temperature sensor
Using the Raspberry Pi model B (512MB RAM, 1x 700 MHz CPU) was easier I thought. I just had to buy a DS18S20 temperature sensor, connect it to the Pi and do some bash magic.
Thanks to a very good documenation (in german which can be found here) my Pi was able to do temperature measurements in the first try.
Grabbing the temperature information using Java was not very hard, I used the Apache Commons Exec project to fire some bash commands and parse the results.
MQTT Message Broker
Choosing a MQTT Message Broker was not that easy, there are many implementations with pro and cons. I decided to use RabbitMQ with the MQTT plugin because it also provides AMQP which I may use in later projects.
Java and MQTT
When I started the Java implementation I just knew I wanted to use spring-boot, but I had no idea how to send and receive MQTT messages. There is a client called paho provided by Eclipse, but I wanted to stay in the Spring universe so I decided to give the MQTT support provided by spring-integration a try.
After some testing (unfortunately the spring-integration documentation not always is that clear) I was able to send and receive my temperature information via MQTT.
Using the ELK stack
Installing the ELK stack is not that hard, there is plenty of documentation in the web. But writing the Logstash configuration matching my temperature logfiles was hard (at least for me). I ended up using much trial and error in this script:
input {
file{
path => "temperature.recipient\logs\received-temperatures.log"
}
}
filter{
grok {
match => { "message" => "%{DATESTAMP:logtimestamp} \[MQTT Call: %{DATA:mqttCaller}\] %{LOGLEVEL} %{JAVACLASS} - {measureTimestamp=%{DATESTAMP:measureTimestamp:timestamp}, room=%{DATA:room}, degreesCelsius=%{NUMBER:degreesCelsius:float}, degreesFahrenheit=%{NUMBER:degreesFahrenheit:float}}"}
}
date{
match => ["measureTimestamp", "dd.MM.yyyy HH:mm:ss"]
target => "measureTimestamp"
}
mutate {
convert => ["degreesCelsius", "float"]
convert => ["degreesFahrenheit", "float"]
}
}
output {
elasticsearch{
protocol => "http"
}
}
Pitfalls
Of course I had some problems during implementation and installation of my temperature logging project.
Using HTTPS on the server
In my local development environment there was no need to think about HTTPS, everything worked fine using HTTP.
Configuring RabbitMQ to use HTTPS was a hard time for me (remember, I’m a Java developer, no Admin!). I did everything I found in the documentation (see here and here), but it did not work.
The simple solution was to enable both listener ports 1883 AND 8883 and block port 1883 by the firewall.
While running your Java application you normally do not see details when a SSL connection is established. If it fails, you have no idea why. In this case the JVM argument
-Djavax.net.debug=SSL
helps a lot.
Kibana RAM consumption
My Kibana 4.3.0 consumed lots of RAM which my VPS did not have which resulted in very high system load due to swapping.
The reason for that is a memory leak in the Kibana server component. The complete error description and a workaround can be found here.
Results
In the end I was successful and I have a well working temperature logger and a big analysis toolset provided by kibana.
Unfortunately I moved to a new well temperatured office in the meanwhile so there is no more need for a temperature logger there. But, as you can see, the temperature logger also works at home 😉
The Java sourcecode can be found in my BitBucket Git repository. Feel free to have a look at the sourcecode, create issues at BitBucket or post some comments.
Flexguse's Blog 