Stochastic Nonsense » mysql

Querying Databases in R

earl — Fri, 14 Aug 2009 16:00:36 +0000

One of the first things you’ll want to do in R is set it up to talk to databases. The easiest way to do this is using ODBC, via package RODBC.

To get the package, run

> install.packages(RODBC)

Once you have RODBC installed, you call it in R as follows. But it’s very simple: a bit of setup, then sqlQuery will run your sql and return the results in a data frame.

library(RODBC)

db <- odbcConnect( dsn='your dsn name' )
sql <- 'select page_id, count(*) as cnt
           from document_ads
           group by page_id
           having count(*) > 1'

results <- sqlQuery(db, sql, errors=T, rows_at_time=1024)
str(results)
'data.frame':	282432 obs. of  2 variables:
 $ page_id: int  17646774 17115332 17606022 15899428 17099174 17283774 8604200 16315025 17259751 17283270 ...
 $ cnt            : int  489 1119 132 113 148 200 112 121 1135 633 ...

On Windows, you setup the DSNs in the ODBC Data Sources inside the control panel; on MacOS, mysql includes a program called ODBC Administrator; on linux, you'll have to install unixODBC .

Also, it's often convenient to write code that caches your query results, particularly if the query takes a while. I've found that the easiest thing to do is write the results into a data file and check for the file existence like such:

filename <- 'query cache.RData'
if (!file.exists(filename)){
   # don't have a cached copy so run the query
   library(RODBC)
   [snip]
   query1 <- sqlQuery(db, sql, errors=T, rows_at_time=1024)

   # save the query results for the future
   save(list=c('query1', 'sql'), file=filename)
   rm(list=c('query1', 'sql') )
}
load(file=filename)



MySQL, Batch Imports, and Rails
earl — Thu, 13 Aug 2009 13:00:48 +0000
I really love Rails, but it’s not the most performant code in the world.  Though it doesn’t often arise in CRUD programming, if you do any sort of stats, ML, or data analytics, you’ll frequently find yourself wanting to import lots of data into your db.  You could create an ActiveRecord object for each row, but this is glacial, requiring one round trip to the db server per row, and is likely to abuse the kindness of your dba.  Instead, there is a wonderful gem called ar-extensions that allows you to access mysql’s native bulk import facilities.  To use it you just call Model.import with arrays of data and their corresponding fields.  For example, say I have a table like this:

mysql> describe adsense_analytics_days;
+------------------+----------+------+-----+---------+----------------+
| Field            | Type     | Null | Key | Default | Extra          |
+------------------+----------+------+-----+---------+----------------+
| id               | int(11)  | NO   | PRI | NULL    | auto_increment |
| page_id          | int(11)  | NO   | MUL | NULL    |                |
| impressions      | int(11)  | YES  |     | NULL    |                |
| clicked          | int(11)  | YES  |     | NULL    |                |
| ecpm             | float    | YES  |     | NULL    |                |
| ctr              | float    | YES  |     | NULL    |                |
| cpc              | float    | YES  |     | NULL    |                |
| revenue          | float    | YES  |     | NULL    |                |
| start_date       | date     | YES  | MUL | NULL    |                |
| end_date         | date     | YES  |     | NULL    |                |
| created_at       | datetime | YES  |     | NULL    |                |
+------------------+----------+------+-----+---------+----------------+
11 rows in set (0.09 sec)

mysql> 



This has a corresponding model AdsenseAnalyticsDay.  Batch importing with rails is then trivial:

require 'ar-extensions'
require 'ar-extensions/import/mysql'

# instead of
if false
  rows.each do |row|
    AdsenseAnalyticsDay.create( ) # etc
  end
end

# you can accomplish a bulk import from, eg, a csv as such:
f = File.new('bulk_import.csv', 'r')
data = []
while line = f.gets
  puts "#{line}" if rand(1000) >= 999
  # pid, impr, clicked, ecpm, ctr, cpc, revenue, start_date, end_date, created_at
  d = line.split(',')
  (0..2).each{ |i| d[i] = d[i].to_i  }
  (3..6).each{ |i| d[i] = d[i].to_f }
  data << d[0..8]
end
f.close

fields = [:page_id, :impressions, :clicked, :ecpm, :ctr, :cpc, :revenue, :start_date, :end_date]
AdsenseAnalyticsDay.import(fields, data, {:validate => false })


where the csv looks like:


MacBook-2:work earl$ head bulk_import.csv
0,344,5,0.755814,0.0145349,0.052,0.26,2009-08-06,2009-08-06,2009-08-10 19:49:12
1,8,1,0,0.125,0,0,2009-08-06,2009-08-06,2009-08-10 19:49:12
2,32,9,76.875,0.28125,0.273333,2.46,2009-08-06,2009-08-06,2009-08-10 19:49:12
4,16,1,1.875,0.0625,0.03,0.03,2009-08-06,2009-08-06,2009-08-10 19:49:12
6,17,2,8.82353,0.117647,0.075,0.15,2009-08-06,2009-08-06,2009-08-10 19:49:12
12,15,1,80,0.0666667,1.2,1.2,2009-08-06,2009-08-06,2009-08-10 19:49:12
34,5,0,0,0,0,0,2009-08-06,2009-08-06,2009-08-10 19:49:12
36,2,0,0,0,0,0,2009-08-06,2009-08-06,2009-08-10 19:49:12
39,46,2,11.7391,0.0434783,0.27,0.54,2009-08-06,2009-08-06,2009-08-10 19:49:12
41,3,0,0,0,0,0,2009-08-06,2009-08-06,2009-08-10 19:49:12
MacBook-2:work earl$ 





Writing MySQL Query Results to Disk
earl — Wed, 15 Jul 2009 01:57:07 +0000
Notes to myself: how to easily write query results to disk using mysql.

mysql -h main-backup.local -u earl -e "select count(*) from adsense_analytics_days;" -p collegelist_development > csvname.csv;



where h specifies the name of the mysql server, u the username, e the query, p the database.
This will output a tsv file; to turn it into csv try using sed to transform tabs into commas or with tr .