Mark's Logs

Reading, thinking, and seeing.

N9, MeeGo, Et Al

| Comments

Tools

  1. Nokia Suite:
    • limited support, for backup and sync
    • Contacts: cannot sync group
    • only on Windows
  2. NokiaLink:
  3. Qt SDK
    • for OS X, Windows and Linux; online or offline
    • Developer account required
  4. Flasher
  5. Repository

Dev Mode

  • Settings -> Security -> Developer mode

Remove Native Apps?

  • Native apps for Chinese Mainland, such as qunar, kaixin, cannot be purged before applying some tricks.
  • Package mp-harmattan-003-pr has dependencies on those apps, which if been removed, the phone would be frozen as a rock.
  • This site provides a work round, a patched package.

Selected Apps

  1. Foursquare: download and dpkg -i
  2. Skype: apt-get install account-plugin-skype l10n-app-skype
  3. Gtalk: apt-get install account-plugin-google-talk l10n-app-google
  4. Twitter: apt-get install twitter l10n-app-twitter webupload-service-twitter
  5. Facebook: apt-get install facebook l10n-app-facebook webupload-service-facebook

Go Lang Notes

| Comments

TL;DR

0. Resources

I. Interaction with other language

C

  • cgo. Include C-libs or even valid C-code by placing these statements as comments immediately above the import “C” line:
1
2
3
4

//#include <stdio.h>
//#include <stdlib.h> 
import "C"
import "unsafe"
  • C.uint, C.long, C.random():
1
2
3

var i int
C.uint(i)
int(C.random())
  • Strings convertion: C.CString(s), C.GoString(cs):
1
2

cstring := C.CString(gostring)
defer C.free(unsafe.Pointer(cstring))
  • Memory allocations made by C code are not known to Go’s memory manager:
1

defer C.free(unsafe.Pointer(Cvariable))
  • Pseudo #cgo directives:
hdfs.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

// #cgo linux CFLAGS: -I/opt/jdk/include -I/opt/jdk/include/linux
// #cgo linux LDFLAGS: -Llib -lhdfs -L/opt/jdk/jre/lib/amd64/server -ljvm
// #cgo darwin LDFLAGS: -L/usr/lib/java -lhdfs -framework JavaVM
// #include "hdfs.h"
/*
int getlen(char*** ptr) {
    int i = 0;
    while (ptr[i] != NULL) ++i;
    return i;
}
int getl(char*** ptr) {
    int i = 0;
    while (ptr[0][i] != NULL) ++i;
    return i;
}
char* getstring(char*** ptr, int i, int j) {
    return ptr[i][j];
}
*/
import "C"
  • Pointer arithmetic:
1
2

var p *C.hdfsFileInfo
p = (*C.hdfsFileInfo)(unsafe.Pointer(uintptr(unsafe.Pointer(info)) + uintptr(i)*unsafe.Sizeof(C.hdfsFileInfo{})))

C++

  • SWIG (Simplified Wrapper and Interface Generator) supports for calling C++/C code from Go on Linux.

    • Write the SWIG interface file for the lib to be wrapped
    • SWIG generates the C stub functions
    • called using the cgo machinery

  • SWIG does NOT understand all of C++.

II. Basic constructs and elementary data types

Filenames, Keywords, Identifiers

  • Filenames may not contain spaces or any other special chars.
  • Identifiers: valid if begin with a letter (even Unicode) and followed by 0 or more letters or Unicode digits.
  • _ is blank identifier.
  • anonymous
  • 25 keywords or reserved words: break, case, chan, const, continue, default, defer, else, fallthrough, for, func, go, goto, if, import, interface, map, package, range, return, select, struct, switch, type, var
  • 36 predeclared identifiers: append, bool, byte, cap, close, complex, complex64, complex128, copy, false, float32, float64, imag, int, int8, int16, int32, int64, iota, len, make, new, nil, panic, print, println, real, recover, string, true, uint, uint8, uint16, uint32, uint64, uintptr
  • Used punctuation chars: .,,,;,:,...
  • Used delimiters: (), [], {}
  • Automatic semicolon insertion. However, multiple statements must be seperated by ; on one line.

Basic structure and components

Packages, import and visibility

  • Every go-file belongs to one (and only one) package. Many different .go files can belong to one package.
  • Package must be indicated on the first line. A standalone exec belongs to package main; each go app contains one package main; a package name is written in lowercase letters.
  • Package is compiled as a unit; each dir contains one package by convetion.
  • Every piece of code is compiled only once.
  • Apart from _, Ids of code-objs have to be unique in a package.

  • Visibility Rule:

    • Id start with an uppercase, then the ‘obj’ with this identifier is visible in code outside the package: exported, like public in OO.
    • Id start with a lowercase are not visible outside the package: like private.

  • alias package: import fm "fmt"

Install_name on OS X

| Comments

On OS X, the loader dyld does have a search path, defined in the DYLD_FRAMEWORK_PATH and DYLD_LIBRARY_PATH variables. However, these are empty on OS X by default, so they rarely matter.

Sometimes, we want to install a third-party library to a location, which is not system-defined, not /usr/local/lib nor /usr/lib, for some personal reasons or when you did not have root privilege. Suppose it was libdummy. If libdummy’s install_name was just libdummy.1.dylib or so, and you were building a program which links against libdummy. After the compilation, you checked the shared libraries your program used:

otool -L program

then you could see libdummy.1.dylib in the output, just libdummy.1.dylib in that line. Ja, the linker stored install_name there, not the location of the library. Then you let the program run, but the dyld said it could not find the proper libdummy.1.dylib. That’s the story a lot of people would experience on OS X.

On Linux, we could modify /etc/ld.so.conf in order to include some other directories when the loader searches for a library. However, on OS X things are different. Also, we would not like to set DYLD_ variables every time launching the program, nor add those variables in .zshrc, .bashrc, …

De facto, as we know install_name matters, we could simply employ it.

On Darwin platform, gcc has some platform dependent options, such as -dynamic, -arch, -bundle, and -install_name plays an important role here.

gcc -o libdummy.dylib -install_name ${PREFIX}/lib/libdummy.dylib ...

would set install_name for libdummy.dylib to a well-defined path. Next time when linking your program against libdummy, the linker would store that path. Use otool -D to print the install_nam for specified library.

Besides absolute paths, we could use other techniques as well: @executable_path, @loader_path, @rpath. This article describes these very well.

For already built libraries and programs, there is no need to rebuild them. On OS X, there is a very useful tool: install_name_tool.

  • change install_name for a library: install_name_tool -id "new_install_name" libdummy.dylib
  • change linked install_name in a program: install_name_tool -change "old_install_name" "new_install_name" program

One more thing. If you use cmake to generate Makefile for you project, you could solve the install_name issue like this:

SET(CMAKE_INSTALL_NAME_DIR @executable_path)

Replace @executable_path with your own choice.

Remote Exposure

| Comments

1. Choosing the Right Equipment

Cameras

  • Film SLR is still a popular camera choice for a remote expedition.

    • old manual SLRs, as well as most rangefinders, possess a mechanical shutter system that can operate without any power source;
    • downsides: no instantly review; more experience needed; physical medium of film;
    • fSLR can be a great backup.

  • DSLRs

    • pros: larger sensors, modular and interchangeable lenses, live view, longer battery life, faster, auto-bracketing;
    • cons: heavier and bulkier;
    • HDR capability.

  • Compact Cameras

    • disadvantages: heavy noise, low battery life, slow autofocus, shutter lag, non-interchangeable lenses; even no viewfinder; lack of ‘RAW’;
    • considerable only when:
      1. the climb is very difficult: light weight is more important than bringing back good images;
      2. snapshots to complement your personal memories;
      3. a high-end compact could serve as a useful backup.

  • EVILs (Electonic Viewfinder Interchangeable Lens Cameras)

    • designed for street photography; close to APS-C DSLRs (with consumer lenses);
    • cons: absence of an optical viewfinder;
    • trade-off between weight and some special features and more choices in lenses (against DSLR).

  • The Big Stuff

    • a bad idea.

  • Summary

    • DSLR or EVIL; compact as a backup.
    • it depends.

Lenses

  • balance versatility and features with weight and portability;
  • no need to focus on large apertures;
  • sometimes it is inconvenient to change lenses;
  • a wide lens is generally the most useful; default choice;
  • oftentimes, telephoto lenses are a necessity; 200mm should be enough; secondary choice;
  • ultra wide lenses are tricky to use;
  • prime lenses: for experienced;
  • gather some statistics in photo archives.

Carrying Systems

  • have your camera accessible at all times;
  • backpack with survival items;
  • without fuss;
  • size should be adapted;
  • all the equipment you might use should be accessible;
  • reasonably weather resistant;
  • simple design: internal separators and a small pocket for memory cards and batteries;
  • lightweight;
  • camera backpacks are worst, chest bag and shoulder bag are not that good;
  • a belt system is the best solution: a padded belt and a few detachable pouches.

Battery and Memory Strategies

  • at least enough cards to get you through one day of shooting;
  • cold is the deadliest threat to batteries;
  • careful and constant monitoring power sources;

Odds and Ends

  • lens caps, UV filters;
  • lens hoods;
  • microfiber cloth;
  • need permission from every recognizable person in images;

Dead Weight

  • tripod: isn’t very useful in the mountains -> find a platform stable enough;
  • artifical lighting: flash is more trouble than help;
  • filters: UV or clear filters only;
  • laptop: no;

Bring with

  • For Hiking and Less Demanding Climbs, e.g.:

    • Nikon D90 w/ 2+ batteries, 12GB SD cards;
    • Nikkor 16-35mm f/4 VR, 70-300mm f/4.5-5.6 VR (equipped w/ UV);
    • Think Tank Skin belt system;

  • For Difficult Climbs, e.g.:

    • EP-1 Pen m43, w/ 12GB SD cards;
    • Zuiko 14-42mm f/3.5-5.6;

25 Recipes for Getting Started With R

| Comments

1. Downloading and Installing R

Noting special

2. Getting Help on a Function

help(), args(), example(), ?func

3. Viewing the Supplied Documentation

help.start()

4. Searching the Web for Help

RSiteSearch("hey phrase")

5. Reading Tabular Datafiles

read.table("target.txt", stringsAsFactor=FALSE)

will prevent read.table() from interpreting character string as factor.

Other options: na.strings=".", header=T, etc.

6. Reading from CSV Files

read.csv("filename", header=T, as.is=T)

as.is=T indicates that R should not interpret nonnumeric data as a factor.

7. Creating a Vector

c(...)

8. Computing Basic Statistics

  • na.rm=T ignore NA values in data, otherwise result would be invalidated;
  • lapply() is magic.

9. Initializing a Data Frame from Column Data

dfrm <- data.frame(v1, v2, v3, v4)
dfrm <- as.data.frame(list.of.vectors)

10. Selecting Data Frame Column by Position

  • dfrm[[n]], dfrm[, n]: returns nth column;
  • dfrm[n]: returns a data frame of nth column;
  • dfrm[c(n1, n2, ..., nk)], dfrm[, c(n1, n2, ..., nk)]: returns a data frame of k columns.

11. Selecting Data Frame column by Name

Similar to the previous section.

  • dfrm[["name"]], dfrm$name: returns one column, called name;
  • dfrm["name"], dfrm[c("name1", "name2", ..., "name3")], dfrm[, "name"], dfrm[, c(...)].

12. Forming a Confidence Interval for a Mean

  • t.test(x): apply to sample x, to determine a confidence interval;
  • conf.level argument: see intervals at other levels.

13. Forming a Confidence Interval for a Proportion

  • prop.test(n, x): sample size n, x successes;
  • use conf.level argument for other confidence levels.

14. Comparing the Means of Two Samples

By default, t.test() assumes data is not paired. Test with two sample x, y:

t.test(x, y, paired=T)

15. Testing a Correlation for Significance

  • cor.test(x, y, method="Spearman") for nonnormally data; default is Pearson method.
  • the function returns several values, including p-value from the test of significance. p < 0.05 indicates that the correlation is likely significant; otherwise, not.

16. Creating a Scatter Plot

  • plot(x, y): two parallel vectors x and y;
  • plot(dfrm): for data frame.

17. Creating a Bar Chart

  • barplot(vector);
  • ref tapply();
  • barchart() from lattice package.

18. Creating a Box Plot

  • boxplot(vector)

19. Creating a Histogram

  • hist(vector, number.of.bins): 7 bins by default;
  • histogram() from lattice package.

20. Performing Simple Linear Regression

  • lm(y ~ x): y ~ x is a model formula

21. Performing Multiple Linear Regression

  • lm(y ~ u + v + w)

22. Getting Regression Statistics

  • m <- lm(y ~ u + v + w)

      anova(m)                        ANOVA table
  coef(m), coefficients(m)        Model coefficients
  confint(m)                      Confidence intervals for the regression coefficients
  deviance(m)                     Residual sum of squares
  effects(m)                      Vector of orthogonal effects
  fitted(m)                       Vector of fitted *y* values
  resid(m), residuals(m)          Model residuals
  summary(m)                      Key statistics
  vcov(m)                         Variance-covariance matrix of the main parameters

23. Diagnosing a Linear Regression

1
2
3
4
5

m <- lm(y ~ x)
plot(m)

library(car)
outlier.test(m)

24. Predicting New Values

1
2
3
4
5
6
7
8
9

m <- lm(y ~ u + v + w)
preds <- data.frame(u=3.1, v=4.0, w=5.5)
predict(m, newdata=preds)

preds <- data.frame(
            u=c(...),
            v=c(...),
            w=c(...))
predict(m, newdata=preds)
  • use interval="prediction" argument of predict() to obtain the confidence intervals.

25. Accessing the Functions in a Package

  • library(packagename);
  • detach(package:name);
  • higher function masks the lower function.

†

Data Mashups in R

| Comments

Ch.I Mapping Foreclosures

  • download

      download.file(url="URL", destfile="DEST")

  • regex

1
2

grep()
gsub()

  • Yahoo!’s Latitude and Longitude service: sign up

  • Parse XML

1
2
3
4
5
6

install.packages("XML")
library("XML")
xmlTreeParse(requestURL, isURL=T)

install.package("RCurl")
library("RCurl")

  • Proxy

      Sys.setenv("http_proxy" = "http://username:passwd@host:port")

  • Magic str()

    it is good practice to closely examine each package’s data structures using str()

  • Exception handling

1
2
3
4
5
6

tryCatch({
    xmlResult <- xmlTreeParse(requestURL, isURL=TRUE, addAttributeNamespaces=TRUE)
    #...other code...
    }, error=function(err){
        cat("xml parsing or http error:", conditionMessage(err), "\n")
})

1
2

library(PBSmapping)
plotPolys()
  • Exploring Data Structures
1
2

as.numeric()
level()
  • Color
1

head.colors()

Ch.II Statistics of Foreclosure

  • Data is available at FactFinder

  • skip lines when reading from file

1

read.table("FILE", skip=1, na.string="")

  • Descriptive Statistics: mean(), median(), sd(), cor(), summary().

  • lattice

      library(lattice)
  install.packages(latticsExtra)
  library(latticeExtra)
    • plot: stripplot() + bwplot()
1
2
3
4
5
6
7
8
9

print(stripplot(IncomeLevels ~ jitter(ct$FCs),
    main=list(
    "Foreclures grouped by National Median Household Income", cex=1),
    sub=list("Greater or less than $50,000", cex=1),
    xlab="foreclosures",
    ylab="household median income",
    aspect=.3, col="lightblue", pch=2 ) +
as.layer(bwplot(IncomeLevels ~ ct$FCs, varwidth=TRUE, box.ratio=0.4, col="blue", pch="|"))
)

  • Correlation

    In R, we can create multidimensional correlation graphs using the pairs() scatterplot matrix package.

†