update subquery optimized

Posted by yixu2001 on Oct 17, 2017; 1:19am
URL: http://apache-carbondata-dev-mailing-list-archive.168.s1.nabble.com/update-subquery-optimized-tp24135.html

dev
 
Version:  Spark 2.1.1 , carbondata 1.1.1  hadoop 2.7.2

test table:
                xitest2  amount of data  2Billion ,
                xitemp2   amount of data  0   ,
               xitemp   amount of data  950

run  sql
cc.sql("update xitest2  a set ( a.qqnum,a.nick,a.age,a.gender,a.auth,a.qunnum)=(select b.qqnum,b.nick,b.age,b.gender,b.auth,b.qunnum from xitemp2 b where b.pkid=a.pkid)").show;
shuffle read  336.5 KB
shuffle write 336.5 KB
 
run  sql
cc.sql("update xitest2  a set ( a.qqnum,a.nick,a.age,a.gender,a.auth,a.qunnum)=(select b.qqnum,b.nick,b.age,b.gender,b.auth,b.qunnum from xitemp b where b.pkid=a.pkid)").show;
shuffle read 1224M
shuffle WRITE 2.4G

When update subquery and subquery data = 0 shuffle too large can be optimized


yixu2001