Difference between revisions of "Main Page/Research/MSB/Data processing"

From phurvitz
< Main Page‎ | Research‎ | MSB
Jump to: navigation, search
 
(→‎Data processing scripts)
 
(10 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
__FORCETOC__
 +
==Jonathan's processing stuff==
 
Job submit: http://ubi.cs.washington.edu/infrastructure/submitJob_phurvitz.html
 
Job submit: http://ubi.cs.washington.edu/infrastructure/submitJob_phurvitz.html
  
Line 4: Line 6:
  
 
Output data: http://www.cs.washington.edu/research/projects/ubicomp3/phurvitz/gis.washington.edu/phurvitz/msb/data/
 
Output data: http://www.cs.washington.edu/research/projects/ubicomp3/phurvitz/gis.washington.edu/phurvitz/msb/data/
 +
 +
==Pushing files==
 +
use rsync:
 +
e.g., to copy from the SD card to the HD
 +
<pre>
 +
rsync -rivh --ignore-existing --size-only /cygdrive/g/msp-data/ /cygdrive/c/users/phurvitz/htdocs/phurvitz/msb/data/pmh04_20071012/
 +
</pre>
 +
 +
e.g., to copy from the HD to the web server
 +
<pre>
 +
rsync -rivh --ignore-existing --size-only /cygdrive/c/users/phurvitz/htdocs/phurvitz/msb/data/pmh04_20071012/ /cygdrive/p/public_html/msb/data/pmh04_20071012/
 +
</pre>
 +
 +
A script to automate this pushing process (assuming the SD card is mounted on drive ''G'':
 +
 +
<pre>
 +
#! /usr/bin/perl
 +
use strict;
 +
 +
# get the args
 +
if ($#ARGV == -1) {
 +
    print "Usage: $0 <subject_id>\n";
 +
    exit;
 +
}
 +
 +
# get the dirs
 +
my $sddata = "g:/msp-data/";
 +
my $hddata = "c:/users/phurvitz/htdocs/phurvitz/msb/data/$ARGV[0]/";
 +
my $msbdata = "p:/public_html/msb/data/$ARGV[0]/";
 +
 +
# do the dirs exist?
 +
if (! -d $sddata) {
 +
  print "$sddata does not exist\n";
 +
  exit;
 +
}
 +
 +
if (! -d $hddata) {
 +
  print "$hddata does not exist\n";
 +
  exit;
 +
 +
}
 +
if (! -d $msbdata) {
 +
  print "$msbdata does not exist\n";
 +
  exit;
 +
}
 +
 +
# munge to cygdir pathnames
 +
$sddata =~ s/://g;
 +
$sddata =~ s/^/\/cygdrive\// ;
 +
$hddata =~ s/://g;
 +
$hddata =~ s/^/\/cygdrive\// ;
 +
$msbdata =~ s/://g;
 +
$msbdata =~ s/^/\/cygdrive\// ;
 +
 +
# copy from the SD card to the HD
 +
system ("rsync -rivh --ignore-existing --size-only $sddata $hddata");
 +
 +
# copy from the HD to the webdir
 +
system ("rsync -rivh --ignore-existing --size-only $hddata $msbdata");
 +
</pre>
 +
 +
==Data processing scripts==
 +
To download MSB files: [[../Scripts/msb.get.data.pl|msb.get.data.pl]]
 +
 +
To read MSB files: [[../Scripts/read.msb.files.R|read.msb.files.R]]
 +
 +
To remove duplicates in the ''class.csv'' file: [[../Scripts/msb_remdupes.pl|msb_remdupes.pl]]
 +
 +
To conflate timestamps between the the phone log file and the MyExperience file: [[../Scripts/conflate_timestamp.R|conflate_timestamp.R]]
 +
 +
A master script to download and process data
 +
 +
==How the hell does it work?==
 +
[[/how it works|how it works]]

Latest revision as of 17:18, 27 January 2009

Jonathan's processing stuff

Job submit: http://ubi.cs.washington.edu/infrastructure/submitJob_phurvitz.html

Job status: http://ubi.cs.washington.edu/assist/index.php/Job_Status

Output data: http://www.cs.washington.edu/research/projects/ubicomp3/phurvitz/gis.washington.edu/phurvitz/msb/data/

Pushing files

use rsync: e.g., to copy from the SD card to the HD 

rsync -rivh --ignore-existing --size-only /cygdrive/g/msp-data/ /cygdrive/c/users/phurvitz/htdocs/phurvitz/msb/data/pmh04_20071012/

e.g., to copy from the HD to the web server 

rsync -rivh --ignore-existing --size-only /cygdrive/c/users/phurvitz/htdocs/phurvitz/msb/data/pmh04_20071012/ /cygdrive/p/public_html/msb/data/pmh04_20071012/

A script to automate this pushing process (assuming the SD card is mounted on drive G: 

#! /usr/bin/perl
use strict;

# get the args
if ($#ARGV == -1) {
    print "Usage: $0 <subject_id>\n";
    exit;
}

# get the dirs
my $sddata = "g:/msp-data/";
my $hddata = "c:/users/phurvitz/htdocs/phurvitz/msb/data/$ARGV[0]/";
my $msbdata = "p:/public_html/msb/data/$ARGV[0]/";

# do the dirs exist?
if (! -d $sddata) {
  print "$sddata does not exist\n";
  exit;
}

if (! -d $hddata) {
  print "$hddata does not exist\n";
  exit;

}
if (! -d $msbdata) {
  print "$msbdata does not exist\n";
  exit;
}

# munge to cygdir pathnames
$sddata =~ s/://g;
$sddata =~ s/^/\/cygdrive\// ;
$hddata =~ s/://g;
$hddata =~ s/^/\/cygdrive\// ;
$msbdata =~ s/://g;
$msbdata =~ s/^/\/cygdrive\// ;

# copy from the SD card to the HD
system ("rsync -rivh --ignore-existing --size-only $sddata $hddata");

# copy from the HD to the webdir
system ("rsync -rivh --ignore-existing --size-only $hddata $msbdata");

Data processing scripts

To download MSB files: msb.get.data.pl

To read MSB files: read.msb.files.R

To remove duplicates in the class.csv file: msb_remdupes.pl

To conflate timestamps between the the phone log file and the MyExperience file: conflate_timestamp.R

A master script to download and process data

How the hell does it work?

how it works

Retrieved from "http://faculty.washington.edu/phurvitz/mediawiki/index.php?title=Main_Page/Research/MSB/Data_processing&oldid=1780"