The Coding Slim Jim

Thursday, July 14, 2011

Data decimation

The some times u just cant send out 100s of MBs of data, hell sometimes you cant process it. Basically you need to decimate the results into a usable shape. The average joe understands averages and maximums. So break it down based on what you need to see for the current level of zoom.

function decimate($points, $bounds)
{
  $lngcnt = 20;
  $latcnt = 20;
  
  $cdl = array();
  foreach($points as &$p)
  {
    $lngper = ($p['long'] - $bounds[1])/($bounds[3] - $bounds[1]);
    $latper = ($p['lat']  - $bounds[0])/($bounds[2] - $bounds[0]);
    $latidx = intVal($latcnt*$latper);
    $lngidx = intVal($lngcnt*$lngper); 
    $val = $p['cpm'];

    #create arrays if needed
    if(!isset($cdl[$lngidx]))
    {
      $cdl[$lngidx] = array();
    }

    if(!isset($cdl[$lngidx][$latidx]))
    {
      $cdl[$lngidx][$latidx] = array();
      $cp = &$cdl[$lngidx][$latidx];
      $cp['sum'] = $val;
      $cp['cnt'] = 1;
      $cp['max'] = $val;
      $cp['max_long'] = $p['long'];
      $cp['max_lat']  = $p['lat'];
      # $cp['min'] = $val;
      # $cp['min_long'] = $p['long'];
      # $cp['min_lat']  = $p['lat'];
    }
    else
    {
      $cp = &$cdl[$lngidx][$latidx];
      $cp['sum'] += $val;
      $cp['cnt']++;
      if ($val > $cp['max'])
      {
        $cp['max']      = $val;
        $cp['max_long'] = $p['long'];
        $cp['max_lat']  = $p['lat'];
      }
      # if ($val < $cp['min'])
      # {
      #   $cp['min']      = $val;
      #   $cp['min_long'] = $p['long'];
      #   $cp['min_lat']  = $p['lat'];
      # }
    }
  }
  
  $res = array();
  foreach($cdl as $lng => &$lngcdl)
  {
    foreach($lngcdl as $lat => &$cp)
    {
      if($cp['cnt'] > 0)
      {
        $pnt = array();
        $pnt['cpm']      = $cp['sum']/$cp['cnt'];
 $pnt['long1']    = ($bounds[3] - $bounds[1])*$lng/$lngcnt + $bounds[1];
 $pnt['lat1']     = ($bounds[2] - $bounds[0])*$lat/$latcnt + $bounds[0];
 $pnt['long2']    = ($bounds[3] - $bounds[1])*($lng+1)/$lngcnt + $bounds[1];
 $pnt['lat2']     = ($bounds[2] - $bounds[0])*($lat+1)/$latcnt + $bounds[0];
        $pnt['max']      = $cp['max']     ;
        $pnt['max_long'] = $cp['max_long'];
        $pnt['max_lat']  = $cp['max_lat'] ; 
        # $pnt['min']      = $cp['min']     ;
        # $pnt['min_long'] = $cp['min_long'];
        # $pnt['min_lat']  = $cp['min_lat'] ;
 array_push($res, $pnt);
      }
    }
  }
  return $res;
}

php sqlite3 queries.

Getting sqlite into php is easy as Here is an example query. Of course im forced to use sqlite in the first place because im running an iphone as a web server.

try 
{
  $db = new PDO("sqlite:data.sqlite");
}
catch(PDOException $e)
{
  echo $e->getMessage();
  echo "<br><br>Database -- NOT -- loaded successfully .. ";
  die( "<br><br>Query Closed !!! $error");
}

#away with ye hackers..
$bbox = sqlite_escape_string($_GET['b']);
$bounds = split(" ", $bbox);
if ( count($bounds) != 4) { die("[]"); } 

$sql  = $db->prepare("SELECT * FROM readings WHERE lat >= ? AND long >= ? AND lat <= ? AND long <= ? LIMIT 1000");
if (! $sql->execute($bounds)) { die("[]"); } 
$query = $sql->fetchAll(PDO::FETCH_ASSOC); 

$decimated = decimate($query, $bounds);
jsonOutput($decimated);

A php sqlite to json converter page

Next I build basically a quick converter page to convert an sqlite query result into json

function jsonOutput($query)
{
  #json convert
  $ofirst = true;
  print "[\n";
  foreach ($query as $row)
  {
    if($ofirst) $ofirst = false;
    else       print ",\n";
    
    $first = true;
    print "{";
    foreach ($row as $key => $value)
    {
      if($first) $first = false;
      else       print ",";
      if(is_numeric($value))
        print $key . ': ' . $value;
      else
        print $key . ': "' . $value . '"';
    }
    print "}";
  }
  print "\n]\n";
}

And ajax test page

While building the radiation map prototype i put together a ajax data grabber here it is

<html>
<head>
<script type="text/javascript">
function loadData(str)
{
var xmlhttp;    
if (window.XMLHttpRequest)
  {// code for IE7+, Firefox, Chrome, Opera, Safari
  xmlhttp=new XMLHttpRequest();
  }
else
  {// code for IE6, IE5
  xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
  }
xmlhttp.onreadystatechange=function()
  {
  if (xmlhttp.readyState==4 && xmlhttp.status==200)
    {
    document.getElementById("txtHint").innerHTML=xmlhttp.responseText;
    }
  }
xmlhttp.open("GET","/safecast/data.php?lat=1&long=2",true);
xmlhttp.send();
}
</script>
</head>
<body>

<div id="txtHint">
<button onclick="loadData('')">Get Data</button>
</div>

</body>
</html>

Tuesday, July 12, 2011

perl to translate chars

$a =~ tr/[A-Z]/[a-z]/;

Try it with

perl -e '$a = "ABCD"; $a =~ tr/[A-Z]/[a-z]/; print $n;'$a;

Monday, July 11, 2011

google maps radiation maping for safecast

Just spent the day hacking together a google maps heat map mash up for safecast.org here is the prototype. Still needs alot of work... but it should be better than what they have at the moment..

Radiation Map Prototype type for Safecast.org

Will blog more about how it all works later.. But interesting thing to note is that its running of my iPhone web-server... very geeky...

Update: Ok.. its fallen over 4 times already... its an iphone calm down people...

Friday, June 24, 2011

Golden rules of Software engineering

One mustn't imply the existing code is a reeking pile of spaghetti...
Managers understand that no engineering feat is impossible, why cant Software engineers..
All problems no matter how complex can be solved with meetings and PowerPoint slides...
Bug trackers are for time tracking and customer communication not fixing problems in source code....
When defining how an interface looks the words "Best and World Class" implies exactly what colors, layout and content is needed...
Designers and Customers are gods. If they tell you to they want the confirm button to play Mozart at the top volume then by god it needs to play Mozart irrespective of what the user feedback says...

Sadly I have encountered each and every one of these items in my own personal experience.