Notes from FB: How people do postdoc research

0 Comments

Popular posts from this blog

HIVE: Both Left and Right Aliases Encountered in Join

Recently was stuck on this error while trying to do a JOIN in HIVE. Below is my SQL create table ma.internet_v2 as select A.msisdn, A.imsi, A.dt, A.start_time, A.end_time, A.url, A.ttl_connection_dur_ms, A.ttl_upload_bytes, A.ttl_download_bytes, A.ttl_cdr_cnt, coalesce(B.domain_desc,A.domain_desc) domain_desc, coalesce(B.subdomain_desc, A.subdomain_desc) subdomain_desc from db.internet A left outer join hfz_domain_name_mapping B on instr(A.url, B.url_pattern) > 0; HIVE then give me and error message "Both Left and Right Aliases Encountered in Join". After going through some mail threads it seems that HIVE can support certain types of join. Only equality joins, outer joins, and left semi joins are supported in Hive. Hive does not support join conditions that are not equality conditions as it is very difficult to express such conditions as a map/reduce job. A poster from  Stack Overflow  suggested that one should use WHERE instead of ON. Duly following his advice,...
Keep reading

Splitting value in Netezza using array_split

To split a column's value in Netezza you can use the array_split function. For example if column AB_MSISDN have a value like "01212345679|019234567679" and we'd like to split this into A number and B number, we could use the below command in Netezza: array_split(ab_msisdn,'|') Doing the above will split the values into arrays. However you wouldn't be able to access the value directly. To do this, you use the get_value_varchar function. Example below:  select ab_msisdn, get_value_varchar(array_split(ab_msisdn,'|'),1) source, get_value_varchar(array_split(ab_msisdn,'|'),2) target from telco_edgelist; Of course one could argue that there are other ways to do this such as using a substring or regex. This is just another option. For more details on the function above do visit IBM's website such this .
Keep reading

Learning GraphX

A repository of some of the places I've been to in learning Apache GraphX. Apache Spark ( http://spark.apache.org/docs/latest/ ) How can you not learn about Spark to learn GraphX? A nice introduction to Spark, and from there you can dive into other components. Download Spark, and just start playing with Scala. First Step to Scala ( http://www.artima.com/scalazine/articles/steps.html ) More tutorials on Scala. A step-by-step approach on some basic Scala skills Functional Programming Principles in Scala ( https://class.coursera.org/progfun-004/lecture ) A course on Scala by none other than Martin Odersky himself. A great way to learn Scala and work in Spark better. AMP Camp by Stanford ( http://ampcamp.berkeley.edu/stanford-workshop/index.html ) Exploring the BDAS stack by Stanford University. If you're in the States - this is probably the best way to learn about Spark, Scala, GraphX, MLib. Why do I put that restriction? It's because to be able to make full use o...
Keep reading