Data logging with open source hardware and software in the energy sector

Solar panels, Arduino, PHP, MySQL, and Flotr

Explore how to monitor simple climatic conditions with the Arduino in the context of solar power generation. Specifically, get an introduction to basic temperature sensing with a TMP36 chip and light with a light-emitting diode (LED), and store the information for later retrieval. Then smooth out selected results and display in a chart, using PHP as a data processor, MySQL® as storage, and Flotr as a charting library.

Share:

Colin Beckingham (colbec@start.ca), Researcher, Freelance

Colin Beckingham is a freelance researcher, writer, and programmer who lives in eastern Ontario, Canada. Holding degrees from Queen's University, Kingston, and the University of Windsor, he has worked in a rich variety of fields including banking, horticulture, horse racing, teaching, civil service, retail, and travel and tourism. The author of database applications and numerous newspaper, magazine, and online articles, his research interests include open source programming, VoIP, and voice-control applications on Linux. You can reach Colin at colbec@start.ca.



19 July 2011

Also available in Chinese Russian Japanese Portuguese

Overview

As power generation moves towards cleaner and smarter sources, small-scale photovoltaic panel arrays are springing up on roofs and in backyards. Some of these installations are sophisticated with a high level of monitoring and self-adjustment; others have no self-awareness. In the latter case, it is still helpful for the owner to have a good and inexpensive source of basic operating data.

You want to know if your panels are working optimally given the conditions. Both light and temperature, with the addition of wind speed and other factors, can influence panel output. More light generates more power, but as more power is generated the cells heat up, which reduces their efficiency. Airflow helps to dissipate heat. Ideally, you want bright light and cool conditions, and a good breeze or other artificial cooling.

The Arduino acts as an intermediary between the sensors and your data storage. It is a straightforward matter to make a permanent record and display the data as required. This article examines data logging of temperature using the TMP36 temperature sensor and a simple light-emitting diode (LED) for light.


Arduino

The Arduino is an inexpensive, adaptable, and programmable open source microprocessor that can read data input in the form of voltage at its analog pins. See Resources for basic information about the unit and a good developerWorks introduction to this unit in a game context. With sensors connected to specific input pins, the unit programmatically reads the data at those pins. What it does with that information depends on how you configure the hardware.

The simplest way of getting the data from the Arduino is with the unit connected directly to a host computer Universal Serial Bus (USB) interface, reading the data as if it were a serial connection. However, the Arduino can also act independently of a computer given a power supply and an alternate communications channel. You can add extension boards ("shields") to the Arduino to store data directly to a microSD memory card, transmit data over a cat5e network cable through mini web server, or even transmit data wireless to a compatible receiver.

Figure 1 shows an Arduino microprocessor with an Ethernet shield mounted and a cat5 Ethernet and power supply cables attached, which gives you an idea of the size of the unit.

Figure 1. Arduino with Ethernet shield
Photo of the Arduino with an ethernet assembly attached.

Some sensors have their own data smoothing internal circuitry. The simpler ones do not. Therefore, the question arises whether the Arduino should do any necessary processing before storage or transmission, or should data simply be reported raw as observed for later processing on another machine. Either is perfectly acceptable. This project simply reports the raw observations for later processing. This gives the option to change the algorithm for smoothing at any time if necessary. When a smoothing process is applied, it is baked in and probably not reversible. In a real-time data situation where the smoothing requirements are well understood, it makes more sense to have the Arduino do the smoothing.


Sensors

You can get a wide variety of sensors to work with the Arduino. Here, the TMP36 and an LED provide a simple, inexpensive, and easily repeatable alternative.

The TMP36 is a specially constructed transistor. Give it a voltage and it returns another voltage that varies consistently according to the ambient temperature on a different combination of connectors. Record the output voltage and perform some simple arithmetic to find the temperature in degrees Celsius or Fahrenheit according to preference. See Resources for help on both the chip and how to connect it to the Arduino.

It seems simple enough to log temperature data in a solar generation context, but there are a couple hurdles to surmount. The first issue, related to ambient temperature, is determining where to locate the sensor. To avoid false readings you want to place it out of direct sunlight and in a weatherproof location that provides enough ventilation. Secondly, a solar panel consists of a number of non-identical cells, each of which runs at its own temperature in the same ambient conditions. In ideal circumstances, you measure all the cells and take the average. An alternate is to try a number of them to identify a representative average cell and monitor it alone.

An LED can both light up when given current at a suitable voltage and generate a voltage when exposed to light (see Mike Cook's article in Resources). The greater the intensity of the light, the more voltage the Arduino sees at the connected pin. Unlike the temperature sensor, which is designed to be mappable into a specific temperature scale, you have to experiment to see how the output from your LED translates to light intensity on a known scale.

Practical setup

Imagine that your solar array location has power, no Ethernet local area network (LAN) jack to plug into, but is within range of a wireless access point. Then the local power can fire up both the access point and the Arduino, with the Arduino plus Ethernet shield plugged into the repeater. You then have analog pins 3 to 6 available for sensor input; assuming pins 1 and 2 are reserved for use by the Ethernet shield (see Resources for more about pin usage). Four pins means you can have four sensors, say two temperature sensors, one for ambient and the other for panel temperature, a light sensor, and a wind speed sensor. This is a rich set of data for a passive panel setup.

You can achieve the actual wiring between the Arduino and the sensors over short distances with regular cat3 telephone cable, which is an unshielded bundle of two pairs of 24 American wire gauge (AWG) wire. This also gives you the option of using readily available telephone adapters and cabling.

Working with the Arduino and Ethernet shield combination is not as easy as an Arduino that is USB/serial connected alone. Here are some details to consider:

  • Normally you don't poll voltages at pins that have no sensors connected to them.
  • Through a barrel connector, the Arduino can accept power at a range of different voltages (see Resources for details). Analog pin voltage readings seem to depend critically on the voltage of the power supply to the Arduino. Make sure that you do your calibration of sensors with the same power supply unit that you intend to use in the field. Advanced users make use of the Analog Reference (AREF) pin to provide a reference voltage.
  • Attach your temperature sensor to the back of a solar cell. Duct tape seems to work quite well to hold the sensor against the cell even in warm conditions over the short term, but make sure that any exposed wires or connectors are both protected and insulated by good electrical tape. It is not unusual for a cell to rise to a normal operating cell temperature (NOCT) of 50 degrees C (124 F) in bright light when the ambient temperature is 20 degrees C (69 F) and there is a steady breeze blowing.
  • The Arduino with Ethernet shield is quite power hungry. If you use a set of fully charged AA size batteries to provide voltage and current they last only a few hours. Advanced users look at the sleep modes available to control power usage.

Figure 2 shows TMP36 sensors attached to a solar rack and panel. On the right, under black electrical tape, is a sensor for ambient temperature taken from the metal pole. To the left is a second sensor attached to the back of a solar cell with duct tape. The duct tape retains adhesion a lot better than the consumer-grade electrical tape, at least in the short term.

Figure 2. Sensors on panel
photo of sensors attached to a solar panel

Figure 3 shows a green LED inside an inverted test tube for protection against rain.

Figure 3. Light sensor
photo of the light sensor taped to a wooden post

The Arduino program (or sketch, as Arduino calls it) that allows access to the data collected is a simple web server that you access with a browser or other script (see Listing 1). The basic code is provided by Arduino as part of the examples collection and requires some editing to adapt to your own requirements (see Resources).

Listing 1. Arduino web server
// based on example web server sketch from http://arduino.cc/
#include <SPI.h>
#include <Ethernet.h>
//
byte mac[] = { 0x!!, 0x!!, 0x!!, 0x!!, 0x!!, 0x!! };
byte ip[] = { 192,168,0,xxx };
Server server(80);
//
void setup()
{
  // start the Ethernet connection and the server:
  delay(10000);
  Ethernet.begin(mac, ip);
  server.begin();
}
//
void loop()
{
  // listen for incoming clients
  Client client = server.available();
  if (client) {
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();
	  //
          // output the value of each analog input pin
          for (int analogChannel = 3; analogChannel < 7; analogChannel++) {
            //client.print("analog input ");
            client.print(analogChannel);
            client.print(" : ");
            client.print(analogRead(analogChannel));
            client.println("<br />");
          }
          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        } 
        else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
    // close the connection:
    client.stop();
  }
}

Listing 1 begins by specifying the header libraries needed, the Media Access Control (MAC) address of the Ethernet shield, and the Internet Protocol (IP) address by which you intend to know the Arduino. It also sets up a server object and a variable used to store a value that can be later discarded. The setup() subroutine runs just once on startup and initializes the server. The loop() subroutine then loops forever, waiting for a Hypertext Transfer Protocol (HTTP) request and then answering with the data needed. In response to a request, it sends a standard HTTP response header, collects readings from the relevant pins, and sends the data package in text format. In this case, the data provided is the voltage experienced at the instant of the request on analog pins 3, 4, 5, and 6. No data is sent for pins 1 and 2 because they are used by the interface between the Arduino board and the Ethernet shield. Change the range of channels to read only the pins you are using.

The response will be something like the output shown in Listing 2.

Listing 2. Arduino response
3 : 292
4 : 288
5 : 286
6 : 280

This gives the number of the pin and the value from a scale of 0 to 1023, the two numbers separated by a colon. The output can be more specific than just specifying the analog pin number, but the generic number allows flexibility because it does not tie down any one pin to any one job.


Data storage

The next thing to consider is how the data is stored. In this case, you are using MySQL as the back end, as shown in Listing 3.

Listing 3. Back-end table
CREATE TABLE IF NOT EXISTS `readings` (
  `readid` int(11) NOT NULL AUTO_INCREMENT,
  `tstamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `pin` int(11) NOT NULL,
  `value` float NOT NULL,
  PRIMARY KEY (`readid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;

This code creates a table with an ID field, the timestamp defaulting to the system time of the entry, the pin the reading relates to, and a field for the actual reading, which in this case is stored as a float.


Data collection and smoothing

With the back end defined, you can now use PHP to query the Arduino, analyze the response, and store the observations in the back end, as shown in Listing 4.

Listing 4. PHP data logging script
<?php
// reader for panel monitors
$fp = fopen("http://192.168.0.xxx","r");
$mysqli = new mysqli("$server",$username,$password,$mysolar);
while ($line = fgets($fp,32)) {
  $line = str_replace("
","",trim($line)); $key = substr($line,0,1); switch ($key) { case 3: // things to do break; case 4: // things to do break; case 5: // this is light from LED recorded as is $val = (int) substr($line,4); $sql = "insert into readings values(NULL,NULL,$key,$val)"; $result = $mysqli->query($sql); //echo "val = $val\n"; break; case 6: // temperature from a TMP36 attached to back of solar cell $val = (int) substr($line,4); $tmp = degc($val); // convert to Celsius/Centigrade $tmp -= 5; // calibration $sql = "insert into readings values(NULL,NULL,$key,$tmp)"; $result = $mysqli->query($sql); break; default: //echo "Found strange key $key!\n"; break; } } function degc($v) { $t = $v * (5/1024); $t -= 0.5; $t *= 100; return round($t,1); } ?>

This script begins by opening the connection to the Arduino in a file pointer for reading. Then it opens the connection to the MySQL back end for data output. It then waits for the response from the Arduino and goes into a loop to read lines reported. Each of the four anticipated lines contains a Hypertext Markup Language (HTML) break tag, which makes it easier to read the output in a browser while testing. The script removes this break. The resulting lines begin with the number of the pin followed by the voltage found. Focusing on the pin number, it then goes into a switch structure that takes a different action depending on which sensor is connected to the pin.

Calibration takes place in this step. That is, if you find that a sensor consistently overestimates by a certain value, you can subtract that value before the final value is reported. There is a specific function for converting the measured millivolts from the TMP36 to degrees Celsius, changing from a range of 0-1023 to 1 to 5, finding the zero point and scaling the result by 100. Lastly, it stores the cleaned up and rounded value in the database. How do you know if your sensor needs correction? One way is to measure a cell that behaves similarly with a point and shoot emissivity temperature reader.

The insert SQL statement starts with two NULL values, allowing the back end to substitute an auto-increment ID number for the first and the current timestamp for the second. It then plugs in the number of the pin and the reading for that pin. You can use this script from your crontab list for regular processing or run as required from the command line.


Chart display

With the back end filling up with data, the next task is to make the data easily readable. There are a number of charting libraries available, each with its own strengths and weaknesses. Flotr (see Resources) is one example that uses JavaScript to display a chart in a browser window. The code shown in Listing 5 reads from the data stored in the readings table and presents the temperature and light data in separate charts.

Listing 5. PHP chart generator using Flotr
<?php
// read solar data and display in flotr chart
$doctype = "<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN'
    'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'>
<html xmlns='http://www.w3.org/1999/xhtml' xml:lang='en'>
<head>
<title>LOG reader</title>
<meta http-equiv='Content-Type' content='text/html;charset=UTF-8' />
<script language=\"javascript\" type=\"text/javascript\"
		src=\"../flotr/lib/prototype.js\"></script>
<script language=\"javascript\" type=\"text/javascript\" 
		src=\"../flotr/lib/base64.js\"></script>
<script language=\"javascript\" type=\"text/javascript\"
		src=\"../flotr/lib/canvas2image.js\"></script>
<script language=\"javascript\" type=\"text/javascript\"
		src=\"../flotr/lib/canvastext.js\"></script>
<script language=\"javascript\" type=\"text/javascript\" 
		src=\"../flotr/flotr.js\"></script>
</head><body>";
$mysqli = new mysqli($server,$username,$password,$dbname);
// get temp and light data and chart it
$sensors = array('Temp'=>6,"Light"=>5);
foreach ($sensors as $label=>$sensor) {
  $sql = "select tstamp,value from readings 
	    where pin=$sensor order by tstamp asc";
  $result = $mysqli->query($sql) or die($mysqli->error);
  $mydata1 = "[ ";
  while ($row = $result->fetch_array()) {
        $cfdoy = $row[0];
        $cfdoy = (strtotime($cfdoy)-(5*60*60))*1000;
        $cfamt = $row[1];
        $mydata1 .= "[ $cfdoy , $cfamt ] ,";
  }
  $mydata1 = substr($mydata1,0,-2)." ]";
  $leg = ($sensor == 18) ? "dC" : "mV";
  $title = ($sensor == 18) ? "Cell Temperature" : "Light";
  $conth .= "<div>$label</div>\n
      <div id=\"container$sensor\" style='width:600px; height:250px;'>
      <script type='text/javascript'>
          var f = Flotr.draw(
              $('container$sensor'), [
              { // => first series
                  data: ".$mydata1.",
                  label: '$leg',   
                  htmlText: false,
                  lines: {show: true}
              }],
              {  
                xaxis: {
                  title: 'Time',
                  mode:'time', 
                  noTicks: 10,
                  labelsAngle:45 
                },
                yaxis: {
                  title: '$leg',
                  noTicks: 8
                },
                title: '$title',
                selection: {
                  mode: 'x', 
                  color: '#B6D9FF',
                  fps: 20
                },
                mouse: {
                  track: true,
                  relative: true,
                  margin: 5,
                  trackFormatter: function(obj) { 
                      var dd = parseInt(obj.x);   
                      var d = new Date(dd);
                      return (d.getHours()+4) + ':' 
			  + d.getMinutes() + ' | ' + obj.y; 
                  },
                  position: 'sw',
                  lineColor: '#FF3F19',
                  trackDecimals: 1,
                  sensibility: 2,  
                  fillOpacity: 0.4 
                }
              }  
      );
        </script></div>";
}
echo "$doctype"."<html><body>".$conth."</body></html>";
?>

The code in Listing 5 begins by defining a string that is the start of the Extensible Hypertext Markup Language (XHTML) output, including the references to the JavaScript Flotr libraries. Then it opens a connection to the MySQL server where the data is stored and sets up an array containing the numbers of the pins used, steps through the array elements fetching data for the relevant sensor and substituting it into the JavaScript script.

The script is set up to treat X axis data as time values on the 24-hour clock for both charts and contains mouse instructions that allow you to hover over the produced line and display the related coordinates.

During the display phase, you might think about smoothing. This can be the use of a moving average or other weight-based scheme that moderates extreme values.

Using real data generated from this setup sampled every five minutes, the charting script produces the output shown in Figure 4 with no smoothing. The conditions were sunny in May in Ontario, with occasional clouds. You can get a copy of the data from the Download section.

Figure 4. Example output
example output

Note the close correlation between the light levels and cell temperature, which is to be expected.

Conclusion

The open source Arduino is useful as a simple and adaptable processor for sensor data in a solar generation context. A wide variety of sensors are available, and as long as a sensor produces a suitable output voltage, or alternatively a digital output, that can be cleanly registered by the Arduino, it can collect and report that data. The TMP36 and LED as sensors are about as basic as you can get—serious users will upgrade from these alternatives.

Good numbers for the behavior of panels in varying conditions can be helpful to managers of photovoltaic units because, given the data record, they can form a baseline of observations. Managers can then make changes according to reasoning and advice and then assess whether later panel manipulation has had a positive effect or not.


Download

DescriptionNameSize
Data samplereadings.sql.zip1KB

Resources

Learn

Get products and technologies

  • Arduino: Visit the main site.
  • MySQL community edition: Access a freely downloadable version fo this open source database.
  • PHP: Check out the latest release of this general-purpose scripting language.
  • IBM trial software: Innovate your next open source development project using trial software, available for download or on DVD.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Open source on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Open source, Industries
ArticleID=733302
ArticleTitle=Data logging with open source hardware and software in the energy sector
publish-date=07192011