Beyond Hadoop - Checking other options

Transcript

1. Beyond Hadoop Checking other options
2. Marcin Cylke
3. Marcin Cylke
4. Marcin Cylke
14. Tap sourceTap = new Hfs(inputScheme, inputPath); Tap sinkTap = new Hfs(outputScheme, outputPath); Pipe wcPipe = new Each("wordcount", new Fields("line"), new RegexSplitGenerator(new Fields("word"), "s+"), new Fields("word")); wcPipe = new GroupBy(wcPipe, new Fields("word")); wcPipe = new Every(wcPipe, new Count(), new Fields("count", "word"));
15. Tap sourceTap = new Hfs(inputScheme, inputPath); Tap sinkTap = new Hfs(outputScheme, outputPath); Pipe wcPipe = new Each("wordcount", new Fields("line"), new RegexSplitGenerator(new Fields("word"), "s+"), new Fields("word")); wcPipe = new GroupBy(wcPipe, new Fields("word")); wcPipe = new Every(wcPipe, new Count(), new Fields("count", "word"));
18. MapReduce łatwiej!
23. Graph landscape • memory-based graphs • disk-based graphs • cluster-based graphs
26. private boolean isSource() { return (getVertexId().get() == getContext().getConfiguration().getLong(SOURCE_ID, SOURCE_ID_DEFAULT)); } @Override public void compute(Iterator<DoubleWritable> msgIterator) { if (getSuperstep() == 0) { setVertexValue( new DoubleWritable( Double.MAX_VALUE)); } double minDist = isSource() ? 0d : Double.MAX_VALUE; while (msgIterator.hasNext()) { minDist = Math.min(minDist, msgIterator.next().get()); } if (LOG.isDebugEnabled()) { LOG.debug( "Vertex " + getVertexId() + " got minDist = " + minDist + " vertex value = " + getVertexValue()); } if (minDist < getVertexValue().get()) { setVertexValue( new DoubleWritable(minDist)); for (Edge<LongWritable, FloatWritable> edge : getOutEdgeMap().values()) { LOG.debug( "Vertex " + getVertexId() + " sent to " + edge.getDestVertexId() + " = " + (minDist + edge.getEdgeValue().get())); sendMsg(edge.getDestVertexId(), new DoubleWritable(minDist + edge.getEdgeValue().get())); } } voteToHalt();
29. • HBase as data sink • Native graph database TinkerPop & Titan
31. from http://www.slideshare.net/slidarko/titan-the-rise-of-big-graph-data - slide 69
32. Rozproszone grafy
33. ciemno listopad chłodno w radiu "Last Christmas" zima }
42. builder.setSpout("spout", new InputSpout(), 5); builder.setBolt("split", new SplitSentence(), 8) .shuffleGrouping("spout"); builder.setBolt("count", new WordCount(), 12) .fieldsGrouping("split", new Fields ("word"));
43. Rynny, rygle • storm-kafka • storm-hbase • storm-rdbms • storm-jms • storm-cassandra • storm-state - write your own hbase! • storm-ml
44. Realtime Hadoop
47. mcl@touk.pl zygm0nt http://marcin.cylke.com.pl/
48. Links Cascading - http://www.cascading.org/ Cascalog - https://github.com/nathanmarz/cascalog Titan http://thinkaurelius.github.com/titan/ http://www.slideshare.net/slidarko/titan-the-rise-of- big-graph-data Storm https://github.com/nathanmarz/storm/wiki/ https://github.com/nathanmarz/storm-contrib
50. public interface ISpout { void open(Map map, TopologyContext topologyContext, SpoutOutputCollector spoutOutputCollector); void close(); void activate(); void deactivate(); void nextTuple(); void ack(Object o); void fail(Object o); }
51. public class ExclIntegerShortSpout extends BaseRichSpout { public void nextTuple() { .... } public void declareOutputFields (OutputFieldsDeclarer declarer) { .... } }
52. public interface IBolt extends { void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector); void execute(Tuple tuple); void cleanup(); }
53. public class ExclamationBolt extends BaseRichBolt { public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) { .... } public void execute(Tuple tuple) { .... } public void declareOutputFields (OutputFieldsDeclarer outputFieldsDeclarer) { .... } }