http://spark.apache.org/docs/1.6.3/mllib-data-types.html

Local Vector

有兩種：dense、sparse

For example, a vector (1.0, 0.0, 3.0) can be rePResented in dense format as [1.0, 0.0, 3.0] or in sparse format as (3, [0, 2], [1.0, 3.0]), where 3 is the size of the vector.

import org.apache.spark.mllib.linalg.{Vector,Vectors}// Create a dense vector (1.0, 0.0, 3.0).val dv: Vector = Vectors.dense(1.0, 0.0, 3.0)// Create a sparse vector (1.0, 0.0, 3.0) by specifying its indices and values corresponding to nonzero entries.val sv1: Vector = Vectors.sparse(3, Array(0, 2), Array(1.0, 3.0))// Create a sparse vector (1.0, 0.0, 3.0) by specifying its nonzero entries.val sv2: Vector = Vectors.sparse(3, Seq((0, 1.0), (2, 3.0)))
import org.apache.spark.mllib.linalg.{Vector, Vectors}dv: org.apache.spark.mllib.linalg.Vector = [1.0,0.0,3.0]sv1: org.apache.spark.mllib.linalg.Vector = (3,[0,2],[1.0,3.0])sv2: org.apache.spark.mllib.linalg.Vector = (3,[0,2],[1.0,3.0])
Labeled point
本身也是一個(gè)local vector，但有l(wèi)abel。常用在監(jiān)督學(xué)習(xí)算法里。多分類問(wèn)題，lable必須從0開(kāi)始，0，1，2，。。。
import org.apache.spark.mllib.linalg.Vectorsimport org.apache.spark.mllib.regression.LabeledPoint// Create a labeled point with a positive label and a dense feature vector.val pos = LabeledPoint(1.0, Vectors.dense(1.0, 0.0, 3.0))
Local Matrix
也分稠密矩陣與 稀疏矩陣。
import org.apache.spark.mllib.linalg.{Matrix, Matrices}// Create a dense matrix ((1.0, 2.0), (3.0, 4.0), (5.0, 6.0))，按列存放val dm: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))// Create a sparse matrix ((9.0, 0.0), (0.0, 8.0), (0.0, 6.0))val sm: Matrix = Matrices.sparse(3, 2, Array(0, 1, 3), Array(0, 2, 1), Array(9, 6, 8))
// 第一個(gè)Array表示從第幾個(gè)元素開(kāi)始新起一列存放，第二個(gè)Array表示非零元素的行號(hào)，第三個(gè)Array表示非零元素的值。
dm: org.apache.spark.mllib.linalg.Matrix = 1.0 2.0 3.0 4.0 5.0 6.0 sm: org.apache.spark.mllib.linalg.Matrix = 3 x 2 CSCMatrix(0,0) 9.0(2,1) 6.0(1,1) 8.0
Distributed matrix
第一種：RowMatrix，
按行存放的，每行是Local Vector。所以列數(shù)受限于integer的范圍。
我們可以對(duì)每列進(jìn)行統(tǒng)計(jì)描述，還可以進(jìn)行QR分解。在SVD、PCA中有用到。
import org.apache.spark.mllib.linalg.Vectorimport org.apache.spark.mllib.linalg.distributed.RowMatrixval rows: RDD[Vector] = ... // an RDD of local vectors// Create a RowMatrix from an RDD[Vector].val mat: RowMatrix = new RowMatrix(rows)// QR decomposition val qrResult = mat.tallSkinnyQR(true)
第二種：IndexedRowMatrix
第三種：CoordinateMatrix
只適用于matrix的維度比較大，并且很稀疏。
Each entry is a tuple of (i: Long, j: Long, value: Double), where i is the row index, j is the column index, and value is the entry value.
第四種  BlockMatrixPS。如何將RDD中的類型轉(zhuǎn)換成自己想要的？
答：利用map，對(duì)每行數(shù)據(jù)進(jìn)行類型轉(zhuǎn)換。
import org.apache.spark.mllib.linalg.{Vector,Vectors}  val m1 = sc.textFile("/user/hadoop-generalshop/caiqi.sun/spark/ml/matrix.txt").map{ r =>    val row = r.split(" ").map(_.toDouble)    (Vectors.dense(row))}
import org.apache.spark.mllib.linalg.{Vector, Vectors}m1: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = MapPartitionsRDD[12] at map at <con
PPS。如何創(chuàng)建RDD？
1）從Hadoop文件系統(tǒng)（如HDFS、Hive、HBase）輸入創(chuàng)建。2）從父RDD轉(zhuǎn)換得到新RDD。3）通過(guò)parallelize或makeRDD將單機(jī)數(shù)據(jù)創(chuàng)建為分布式RDD。從集合創(chuàng)建RDD：var rdd = sc.parallelize(1 to 10) var rdd = sc.makeRDD(collect) 4）基于DB(MySQL)、NoSQL(HBase)、S3(SC3)、數(shù)據(jù)流創(chuàng)建。