Using the Netezza Analytics Matrix Engine

Felt like writing something about the topic due to the lack of available examples out there in the internet.
-- initialize the engine
CALL NZM..INITIALIZE();
--create some random matrix
CALL NZM..CREATE_RANDOM_MATRIX('A', 10, 10);
CALL NZM..CREATE_RANDOM_MATRIX('B', 10, 10);
--let's try adding the matrix. Put the result in matrix C
CALL NZM..ADD('A', 'B', 'C');
--now let's see the result -> create a table from the matrix
CALL NZM..CREATE_TABLE_FROM_MATRIX('C', 'TABLE_C');
--look at the content.
select * from table_c order by row, col;
--to check the results, let's create additional tables from matrix A and B and calculate manually.
CALL NZM..CREATE_TABLE_FROM_MATRIX('B', 'TABLE_B');
select * from table_b order by row, col;
CALL NZM..CREATE_MATRIX_FROM_TABLE('TABLE_A', 'A', 10, 10);
select * from table_b order by row, col;
--let's try some real-life application.
/*  Begin Ego Network Centrality Calculation  (based on Borgatti, 2004)*/
/* formula for ego betweenness
A^2*[1-A], where A is the adjacency matrix of node 'i'.
*/
/*
suppose we have an adjacency matrix in EGO_NTWK_3. Content of the table must be in "row, col, value" format so that it could be converted to matrix later. Below is some sample data:
row,col,value
1,1,0
1,2,1
1,3,1
1,4,1
1,5,1
2,1,1
2,2,0
2,3,1
2,4,0
2,5,0
3,1,1
3,2,1
3,3,0
3,4,1
3,5,0
4,1,1
4,2,0
4,3,1
4,4,0
4,5,0
5,1,1
5,2,0
5,3,0
5,4,0
5,5,0
*/
select * from EGO_NTWK_3;
--create matrix from adjacency matrix table
call nzm..create_matrix_from_table('EGO_NTWK_3', 'EGO_NTWK', 5, 5);
--get geodesics of length 2, A^2
CALL nzm..mtx_pow2('EGO_NTWK',2,'EGO_NTWK_SQ');
--create ones matrix
call nzm..create_ones_matrix('ONES',nzm..get_num_rows('EGO_NTWK'),nzm..get_num_rows('EGO_NTWK'));
--preparing [1-A]
CALL NZM..SUBTRACT('ONES', 'EGO_NTWK', '1_MINUS_EGO');
--calculating A^2*[1-A]
CALL NZM..MULTIPLY_ELEMENTS('EGO_NTWK_SQ', '1_MINUS_EGO', 'RESULT');
--transfer result in a new table
CALL NZM..CREATE_TABLE_FROM_MATRIX('RESULT', 'TABLE_R');
--clear all matrix for future calculations
CALL NZM..DELETE_ALL_MATRICES;
--get reciprocal of matrix for values above diagonal (since below diagonal are just duplicates). The sum is the ego betweenness score for 'i'.
select sum(reciprocal) from (
select row,col,1/value as reciprocal from table_r
where col < row
and value != 0
order by row, col
) A;
/* The ego-betweenness score should be 3.5 */


For more details on the subject, check out the guides by IBM:

1. Netezza Matrix Engine Reference Guide
2. IBM Netezza Analytics Matrix Engine Developer's Guide

Comments

Post a Comment

Popular posts from this blog

A Basic Recipe for Machine Learning

Image
Ever since wrapping up the three Deep Learning courses by Andrew Ng I've been meaning to write down some of the gems that he's highlighted throughout the course. One of the nice ones that I felt needed to be written down is his general recipe to approaching a deep learning algorithm/model. I've basically summarized it in a flowchart below (because everybody loves a flowchart right?) Basic deep learning recipe What is bias and variance? The below diagram is the typical explanation that I'm sure most of us are used to.  Graphical illustration of bias and variance How can we know if we have high bias or high variance?  For high bias, we could take a look at the training set performance. A poor performance is an indicator of a poor model fit, and signals that we could try to apply a bigger network to get a better fit of the model. For high variance, we could take a look at the validation set (or dev set - as Andrew calls it) performance. ...
Keep reading

HIVE: Both Left and Right Aliases Encountered in Join

Recently was stuck on this error while trying to do a JOIN in HIVE. Below is my SQL create table ma.internet_v2 as select A.msisdn, A.imsi, A.dt, A.start_time, A.end_time, A.url, A.ttl_connection_dur_ms, A.ttl_upload_bytes, A.ttl_download_bytes, A.ttl_cdr_cnt, coalesce(B.domain_desc,A.domain_desc) domain_desc, coalesce(B.subdomain_desc, A.subdomain_desc) subdomain_desc from db.internet A left outer join hfz_domain_name_mapping B on instr(A.url, B.url_pattern) > 0; HIVE then give me and error message "Both Left and Right Aliases Encountered in Join". After going through some mail threads it seems that HIVE can support certain types of join. Only equality joins, outer joins, and left semi joins are supported in Hive. Hive does not support join conditions that are not equality conditions as it is very difficult to express such conditions as a map/reduce job. A poster from  Stack Overflow  suggested that one should use WHERE instead of ON. Duly following his advice,...
Keep reading

MVNO: Through thick and thin

The below article was taken from Silicon.com. Click here to read the actual article. By Natasha Lomas, 27 October 2008 16:23 MVNO? That'll be an abbreviation - another pointless four letter one no doubt… Right and wrong. MVNO stands for mobile virtual network operator - and it refers to a rather intriguing slice of the mobile pie. I'm listening… Building and operating a mobile network is an expensive business, not least because it requires radio spectrum, so the number of mobile network operators (or MNOs) is limited to, at most, a handful per market. For instance, the UK market has five: 3, O2, Orange, T-Mobile and Vodafone. Spectrum does not come cheap. The auction for 3G licences in the UK - way back in 2000 - ended with five companies shelling out a whopping £22.47bn for the privilege of flogging third-generation services to Brits. So the requirement for large capital expenditure to set up a mobile network goes some way to explaining why actual networks are not 10-a-penny. ...
Keep reading