The Coding Slim Jim: July 2011

Thursday, July 14, 2011

Data decimation

The some times u just cant send out 100s of MBs of data, hell sometimes you cant process it. Basically you need to decimate the results into a usable shape. The average joe understands averages and maximums. So break it down based on what you need to see for the current level of zoom.

function decimate($points, $bounds)
{
  $lngcnt = 20;
  $latcnt = 20;
  
  $cdl = array();
  foreach($points as &$p)
  {
    $lngper = ($p['long'] - $bounds[1])/($bounds[3] - $bounds[1]);
    $latper = ($p['lat']  - $bounds[0])/($bounds[2] - $bounds[0]);
    $latidx = intVal($latcnt*$latper);
    $lngidx = intVal($lngcnt*$lngper); 
    $val = $p['cpm'];

    #create arrays if needed
    if(!isset($cdl[$lngidx]))
    {
      $cdl[$lngidx] = array();
    }

    if(!isset($cdl[$lngidx][$latidx]))
    {
      $cdl[$lngidx][$latidx] = array();
      $cp = &$cdl[$lngidx][$latidx];
      $cp['sum'] = $val;
      $cp['cnt'] = 1;
      $cp['max'] = $val;
      $cp['max_long'] = $p['long'];
      $cp['max_lat']  = $p['lat'];
      # $cp['min'] = $val;
      # $cp['min_long'] = $p['long'];
      # $cp['min_lat']  = $p['lat'];
    }
    else
    {
      $cp = &$cdl[$lngidx][$latidx];
      $cp['sum'] += $val;
      $cp['cnt']++;
      if ($val > $cp['max'])
      {
        $cp['max']      = $val;
        $cp['max_long'] = $p['long'];
        $cp['max_lat']  = $p['lat'];
      }
      # if ($val < $cp['min'])
      # {
      #   $cp['min']      = $val;
      #   $cp['min_long'] = $p['long'];
      #   $cp['min_lat']  = $p['lat'];
      # }
    }
  }
  
  $res = array();
  foreach($cdl as $lng => &$lngcdl)
  {
    foreach($lngcdl as $lat => &$cp)
    {
      if($cp['cnt'] > 0)
      {
        $pnt = array();
        $pnt['cpm']      = $cp['sum']/$cp['cnt'];
 $pnt['long1']    = ($bounds[3] - $bounds[1])*$lng/$lngcnt + $bounds[1];
 $pnt['lat1']     = ($bounds[2] - $bounds[0])*$lat/$latcnt + $bounds[0];
 $pnt['long2']    = ($bounds[3] - $bounds[1])*($lng+1)/$lngcnt + $bounds[1];
 $pnt['lat2']     = ($bounds[2] - $bounds[0])*($lat+1)/$latcnt + $bounds[0];
        $pnt['max']      = $cp['max']     ;
        $pnt['max_long'] = $cp['max_long'];
        $pnt['max_lat']  = $cp['max_lat'] ; 
        # $pnt['min']      = $cp['min']     ;
        # $pnt['min_long'] = $cp['min_long'];
        # $pnt['min_lat']  = $cp['min_lat'] ;
 array_push($res, $pnt);
      }
    }
  }
  return $res;
}

php sqlite3 queries.

Getting sqlite into php is easy as Here is an example query. Of course im forced to use sqlite in the first place because im running an iphone as a web server.

try 
{
  $db = new PDO("sqlite:data.sqlite");
}
catch(PDOException $e)
{
  echo $e->getMessage();
  echo "<br><br>Database -- NOT -- loaded successfully .. ";
  die( "<br><br>Query Closed !!! $error");
}

#away with ye hackers..
$bbox = sqlite_escape_string($_GET['b']);
$bounds = split(" ", $bbox);
if ( count($bounds) != 4) { die("[]"); } 

$sql  = $db->prepare("SELECT * FROM readings WHERE lat >= ? AND long >= ? AND lat <= ? AND long <= ? LIMIT 1000");
if (! $sql->execute($bounds)) { die("[]"); } 
$query = $sql->fetchAll(PDO::FETCH_ASSOC); 

$decimated = decimate($query, $bounds);
jsonOutput($decimated);

A php sqlite to json converter page

Next I build basically a quick converter page to convert an sqlite query result into json

function jsonOutput($query)
{
  #json convert
  $ofirst = true;
  print "[\n";
  foreach ($query as $row)
  {
    if($ofirst) $ofirst = false;
    else       print ",\n";
    
    $first = true;
    print "{";
    foreach ($row as $key => $value)
    {
      if($first) $first = false;
      else       print ",";
      if(is_numeric($value))
        print $key . ': ' . $value;
      else
        print $key . ': "' . $value . '"';
    }
    print "}";
  }
  print "\n]\n";
}

And ajax test page

While building the radiation map prototype i put together a ajax data grabber here it is

<html>
<head>
<script type="text/javascript">
function loadData(str)
{
var xmlhttp;    
if (window.XMLHttpRequest)
  {// code for IE7+, Firefox, Chrome, Opera, Safari
  xmlhttp=new XMLHttpRequest();
  }
else
  {// code for IE6, IE5
  xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
  }
xmlhttp.onreadystatechange=function()
  {
  if (xmlhttp.readyState==4 && xmlhttp.status==200)
    {
    document.getElementById("txtHint").innerHTML=xmlhttp.responseText;
    }
  }
xmlhttp.open("GET","/safecast/data.php?lat=1&long=2",true);
xmlhttp.send();
}
</script>
</head>
<body>

<div id="txtHint">
<button onclick="loadData('')">Get Data</button>
</div>

</body>
</html>

Tuesday, July 12, 2011

perl to translate chars

$a =~ tr/[A-Z]/[a-z]/;

Try it with

perl -e '$a = "ABCD"; $a =~ tr/[A-Z]/[a-z]/; print $n;'$a;

Monday, July 11, 2011

google maps radiation maping for safecast

Just spent the day hacking together a google maps heat map mash up for safecast.org here is the prototype. Still needs alot of work... but it should be better than what they have at the moment..

Radiation Map Prototype type for Safecast.org

Will blog more about how it all works later.. But interesting thing to note is that its running of my iPhone web-server... very geeky...

Update: Ok.. its fallen over 4 times already... its an iphone calm down people...