[jira] [Created] (CARBONDATA-824) Null pointer Exception display to user while performance Testing

[Akash R Nilugal (Jira)]

Vinod Rohilla created CARBONDATA-824:
----------------------------------------

             Summary: Null pointer Exception display to user while performance Testing
                 Key: CARBONDATA-824
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-824
             Project: CarbonData
          Issue Type: Bug
          Components: data-query
    Affects Versions: 1.0.1-incubating
         Environment: SPARK 2.1
            Reporter: Vinod Rohilla


Displays null pointer exception to the user while select Query.

Steps to reproduces:

1: Create table:

CREATE TABLE oscon_new_1 (ACTIVE_AREA_ID String, ACTIVE_CHECK_DY String, ACTIVE_CHECK_HOUR String, ACTIVE_CHECK_MM String, ACTIVE_CHECK_TIME String, ACTIVE_CHECK_YR String, ACTIVE_CITY String, ACTIVE_COUNTRY String, ACTIVE_DISTRICT String, ACTIVE_EMUI_VERSION String, ACTIVE_FIRMWARE_VER String, ACTIVE_NETWORK String, ACTIVE_OS_VERSION String, ACTIVE_PROVINCE String, BOM String, CHECK_DATE String, CHECK_DY String, CHECK_HOUR String, CHECK_MM String, CHECK_YR String, CUST_ADDRESS_ID String, CUST_AGE String, CUST_BIRTH_COUNTRY String, CUST_BIRTH_DY String, CUST_BIRTH_MM String, CUST_BIRTH_YR String, CUST_BUY_POTENTIAL String, CUST_CITY String, CUST_STATE String, CUST_COUNTRY String, CUST_COUNTY String, CUST_EMAIL_ADDR String, CUST_LAST_RVW_DATE TIMESTAMP, CUST_FIRST_NAME String, CUST_ID String, CUST_JOB_TITLE String, CUST_LAST_NAME String, CUST_LOGIN String, CUST_NICK_NAME String, CUST_PRFRD_FLG String, CUST_SEX String, CUST_STREET_NAME String, CUST_STREET_NO String, CUST_SUITE_NO String, CUST_ZIP String, DELIVERY_CITY String, DELIVERY_STATE String, DELIVERY_COUNTRY String, DELIVERY_DISTRICT String, DELIVERY_PROVINCE String, DEVICE_NAME String, INSIDE_NAME String, ITM_BRAND String, ITM_BRAND_ID String, ITM_CATEGORY String, ITM_CATEGORY_ID String, ITM_CLASS String, ITM_CLASS_ID String, ITM_COLOR String, ITM_CONTAINER String, ITM_FORMULATION String, ITM_MANAGER_ID String, ITM_MANUFACT String, ITM_MANUFACT_ID String, ITM_ID String, ITM_NAME String, ITM_REC_END_DATE String, ITM_REC_START_DATE String, LATEST_AREAID String, LATEST_CHECK_DY String, LATEST_CHECK_HOUR String, LATEST_CHECK_MM String, LATEST_CHECK_TIME String, LATEST_CHECK_YR String, LATEST_CITY String, LATEST_COUNTRY String, LATEST_DISTRICT String, LATEST_EMUI_VERSION String, LATEST_FIRMWARE_VER String, LATEST_NETWORK String, LATEST_OS_VERSION String, LATEST_PROVINCE String, OL_ORDER_DATE String, OL_ORDER_NO INT, OL_RET_ORDER_NO String, OL_RET_DATE String, OL_SITE String, OL_SITE_DESC String, PACKING_DATE String, PACKING_DY String, PACKING_HOUR String, PACKING_LIST_NO String, PACKING_MM String, PACKING_YR String, PRMTION_ID String, PRMTION_NAME String, PRM_CHANNEL_CAT String, PRM_CHANNEL_DEMO String, PRM_CHANNEL_DETAILS String, PRM_CHANNEL_DMAIL String, PRM_CHANNEL_EMAIL String, PRM_CHANNEL_EVENT String, PRM_CHANNEL_PRESS String, PRM_CHANNEL_RADIO String, PRM_CHANNEL_TV String, PRM_DSCNT_ACTIVE String, PRM_END_DATE String, PRM_PURPOSE String, PRM_START_DATE String, PRODUCT_ID String, PROD_BAR_CODE String, PROD_BRAND_NAME String, PRODUCT_NAME String, PRODUCT_MODEL String, PROD_MODEL_ID String, PROD_COLOR String, PROD_SHELL_COLOR String, PROD_CPU_CLOCK String, PROD_IMAGE String, PROD_LIVE String, PROD_LOC String, PROD_LONG_DESC String, PROD_RAM String, PROD_ROM String, PROD_SERIES String, PROD_SHORT_DESC String, PROD_THUMB String, PROD_UNQ_DEVICE_ADDR String, PROD_UNQ_MDL_ID String, PROD_UPDATE_DATE String, PROD_UQ_UUID String, SHP_CARRIER String, SHP_CODE String, SHP_CONTRACT String, SHP_MODE_ID String, SHP_MODE String, STR_ORDER_DATE String, STR_ORDER_NO String, TRACKING_NO String, WH_CITY String, WH_COUNTRY String, WH_COUNTY String, WH_ID String, WH_NAME String, WH_STATE String, WH_STREET_NAME String, WH_STREET_NO String, WH_STREET_TYPE String, WH_SUITE_NO String, WH_ZIP String, CUST_DEP_COUNT DOUBLE, CUST_VEHICLE_COUNT DOUBLE, CUST_ADDRESS_CNT DOUBLE, CUST_CRNT_CDEMO_CNT DOUBLE, CUST_CRNT_HDEMO_CNT DOUBLE, CUST_CRNT_ADDR_DM DOUBLE, CUST_FIRST_SHIPTO_CNT DOUBLE, CUST_FIRST_SALES_CNT DOUBLE, CUST_GMT_OFFSET DOUBLE, CUST_DEMO_CNT DOUBLE, CUST_INCOME DOUBLE, PROD_UNLIMITED INT, PROD_OFF_PRICE DOUBLE, PROD_UNITS INT, TOTAL_PRD_COST DOUBLE, TOTAL_PRD_DISC DOUBLE, PROD_WEIGHT DOUBLE, REG_UNIT_PRICE DOUBLE, EXTENDED_AMT DOUBLE, UNIT_PRICE_DSCNT_PCT DOUBLE, DSCNT_AMT DOUBLE, PROD_STD_CST DOUBLE, TOTAL_TX_AMT DOUBLE, FREIGHT_CHRG DOUBLE, WAITING_PERIOD DOUBLE, DELIVERY_PERIOD DOUBLE, ITM_CRNT_PRICE DOUBLE, ITM_UNITS DOUBLE, ITM_WSLE_CST DOUBLE, ITM_SIZE DOUBLE, PRM_CST DOUBLE, PRM_RESPONSE_TARGET DOUBLE, PRM_ITM_DM DOUBLE, SHP_MODE_CNT DOUBLE, WH_GMT_OFFSET DOUBLE, WH_SQ_FT DOUBLE, STR_ORD_QTY DOUBLE, STR_WSLE_CST DOUBLE, STR_LIST_PRICE DOUBLE, STR_SALES_PRICE DOUBLE, STR_EXT_DSCNT_AMT DOUBLE, STR_EXT_SALES_PRICE DOUBLE, STR_EXT_WSLE_CST DOUBLE, STR_EXT_LIST_PRICE DOUBLE, STR_EXT_TX DOUBLE, STR_COUPON_AMT DOUBLE, STR_NET_PAID DOUBLE, STR_NET_PAID_INC_TX DOUBLE, STR_NET_PRFT DOUBLE, STR_SOLD_YR_CNT DOUBLE, STR_SOLD_MM_CNT DOUBLE, STR_SOLD_ITM_CNT DOUBLE, STR_TOTAL_CUST_CNT DOUBLE, STR_AREA_CNT DOUBLE, STR_DEMO_CNT DOUBLE, STR_OFFER_CNT DOUBLE, STR_PRM_CNT DOUBLE, STR_TICKET_CNT DOUBLE, STR_NET_PRFT_DM_A DOUBLE, STR_NET_PRFT_DM_B DOUBLE, STR_NET_PRFT_DM_C DOUBLE, STR_NET_PRFT_DM_D DOUBLE, STR_NET_PRFT_DM_E DOUBLE, STR_RET_STR_ID DOUBLE, STR_RET_REASON_CNT DOUBLE, STR_RET_TICKET_NO DOUBLE, STR_RTRN_QTY DOUBLE, STR_RTRN_AMT DOUBLE, STR_RTRN_TX DOUBLE, STR_RTRN_AMT_INC_TX DOUBLE, STR_RET_FEE DOUBLE, STR_RTRN_SHIP_CST DOUBLE, STR_RFNDD_CSH DOUBLE, STR_REVERSED_CHRG DOUBLE, STR_STR_CREDIT DOUBLE, STR_RET_NET_LOSS DOUBLE, STR_RTRNED_YR_CNT DOUBLE, STR_RTRN_MM_CNT DOUBLE, STR_RET_ITM_CNT DOUBLE, STR_RET_CUST_CNT DOUBLE, STR_RET_AREA_CNT DOUBLE, STR_RET_OFFER_CNT DOUBLE, STR_RET_PRM_CNT DOUBLE, STR_RET_NET_LOSS_DM_A DOUBLE, STR_RET_NET_LOSS_DM_B DOUBLE, STR_RET_NET_LOSS_DM_C DOUBLE, STR_RET_NET_LOSS_DM_D DOUBLE, OL_ORD_QTY DOUBLE, OL_WSLE_CST DOUBLE, OL_LIST_PRICE DOUBLE, OL_SALES_PRICE DOUBLE, OL_EXT_DSCNT_AMT DOUBLE, OL_EXT_SALES_PRICE DOUBLE, OL_EXT_WSLE_CST DOUBLE, OL_EXT_LIST_PRICE DOUBLE, OL_EXT_TX DOUBLE, OL_COUPON_AMT DOUBLE, OL_EXT_SHIP_CST DOUBLE, OL_NET_PAID DOUBLE, OL_NET_PAID_INC_TX DOUBLE, OL_NET_PAID_INC_SHIP DOUBLE, OL_NET_PAID_INC_SHIP_TX DOUBLE, OL_NET_PRFT DOUBLE, OL_SOLD_YR_CNT DOUBLE, OL_SOLD_MM_CNT DOUBLE, OL_SHIP_DATE_CNT DOUBLE, OL_ITM_CNT DOUBLE, OL_BILL_CUST_CNT DOUBLE, OL_BILL_AREA_CNT DOUBLE, OL_BILL_DEMO_CNT DOUBLE, OL_BILL_OFFER_CNT DOUBLE, OL_SHIP_CUST_CNT DOUBLE, OL_SHIP_AREA_CNT DOUBLE, OL_SHIP_DEMO_CNT DOUBLE, OL_SHIP_OFFER_CNT DOUBLE, OL_WEB_PAGE_CNT DOUBLE, OL_WEB_SITE_CNT DOUBLE, OL_SHIP_MODE_CNT DOUBLE, OL_WH_CNT DOUBLE, OL_PRM_CNT DOUBLE, OL_NET_PRFT_DM_A DOUBLE, OL_NET_PRFT_DM_B DOUBLE, OL_NET_PRFT_DM_C DOUBLE, OL_NET_PRFT_DM_D DOUBLE, OL_RET_RTRN_QTY DOUBLE, OL_RTRN_AMT DOUBLE, OL_RTRN_TX DOUBLE, OL_RTRN_AMT_INC_TX DOUBLE, OL_RET_FEE DOUBLE, OL_RTRN_SHIP_CST DOUBLE, OL_RFNDD_CSH DOUBLE, OL_REVERSED_CHRG DOUBLE, OL_ACCOUNT_CREDIT DOUBLE, OL_RTRNED_YR_CNT DOUBLE, OL_RTRNED_MM_CNT DOUBLE, OL_RTRITM_CNT DOUBLE, OL_RFNDD_CUST_CNT DOUBLE, OL_RFNDD_AREA_CNT DOUBLE, OL_RFNDD_DEMO_CNT DOUBLE, OL_RFNDD_OFFER_CNT DOUBLE, OL_RTRNING_CUST_CNT DOUBLE, OL_RTRNING_AREA_CNT DOUBLE, OL_RTRNING_DEMO_CNT DOUBLE, OL_RTRNING_OFFER_CNT DOUBLE, OL_RTRWEB_PAGE_CNT DOUBLE, OL_REASON_CNT DOUBLE, OL_NET_LOSS DOUBLE, OL_NET_LOSS_DM_A DOUBLE, OL_NET_LOSS_DM_B DOUBLE, OL_NET_LOSS_DM_C DOUBLE) STORED BY 'org.apache.carbondata.format';


2: Perform Select Query.
TC_010,""select * from  oscon_new_1  where CUST_ID = """"ID00000000015"""" and CUST_CITY=""""CC015"""" and CUST_LAST_RVW_DATE between """"2011-02-17 00:00:00"""" and """"2011-02-18 00:00:00""""""

TC_011,""select * from   oscon_new_1  where PROD_COLOR =""""GOLD"""" and ACTIVE_EMUI_VERSION  like""""%73"""" limit 5000""

TC_022,""SELECT *  from  oscon_new_1  where CUST_PRFRD_FLG=""""Y"""" and PROD_BRAND_NAME = """"LG"""" and PROD_COLOR = """"BLACK"""" and CUST_LAST_RVW_DATE = """"2011-07-02 00:00:00"""" and CUST_COUNTRY = """"CC009"""" and product_name = """"LG KF311 phone"""" ""

TC_019,"select * from  oscon_new_1  where CUST_CITY='CC008' and CUST_PRFRD_FLG='Y' and CUST_LAST_RVW_DATE between '2011-02-04 00:00:00' and '2011-02-05 00:00:00'"

TC_008,""select * from  oscon_new_1  where CUST_CITY=""""CC015"""" and CUST_PRFRD_FLG=""""Y"""" and CUST_LAST_RVW_DATE between """"2011-02-02 00:00:00"""" and """"2011-02-03 00:00:00""""""

Error :

WARN TaskSetManager: Lost task 0.0 in stage 69.0 (TID 24007, hadoop-slave-7, executor 8): java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at org.apache.carbondata.core.scan.processor.AbstractDataBlockIterator.updateScanner(AbstractDataBlockIterator.java:136)
        at org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:50)
        at org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32)
        at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:50)
        at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:41)
        at org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:31)
        at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:41)
        at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:79)
        at org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:204)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:99)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.carbondata.core.scan.processor.AbstractDataBlockIterator.getNextScannedResult(AbstractDataBlockIterator.java:146)
        at org.apache.carbondata.core.scan.processor.AbstractDataBlockIterator.updateScanner(AbstractDataBlockIterator.java:124)
        ... 28 more


Note : Data load file (CSV) is 32 GB so which can not be upload on the Jira.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[ZhuWilliam]

First, can you try change the create table statement ends with STORED BY
'carbondata' instead of STORED BY 'org.apache.carbondata.format';
Second , can you give some sample data instead of trying to upload 32GB CSV
file.