[ANNOUNCE] Apache CarbonData 1.4.0 release

[Liang Chen-2]

Hi

Apache CarbonData community is pleased to announce the release of the
Version 1.4.0 in The Apache Software Foundation (ASF).

CarbonData is a high-performance big data store solution that supports fast
filter lookups and ad-hoc OLAP analysis. Due to varied business driven
analysis, and the demand for flexibility of data analytics, big data domain
is shadowed with data duplication and increased data management cost.
CarbonData provides a new converged data storage to address data
de-duplication, and supports various application scenarios. CarbonData has
been deployed in 20+ enterprise production environments, largest single
cluster (100+ nodes) managing data of tens of trillions. The I/O scanning
and computing performance is improved by leveraging features such as
multi-level index, dictionary encoding, pre-aggregation, dynamic
partitioning, and quasi-real-time data query; there by achieving
second-level response to analytics query on tens of trillions of data.

We encourage everyone to download the release
https://dist.apache.org/repos/dist/release/carbondata/1.4.0/, and feedback
through mailing list.([hidden email] or
[hidden email])

This release note provides information on the new features, improvements,
and bug fixes of this release.
What’s New in Version 1.4.0?

In this version of CarbonData, following are the new features added for
performance improvements, compatibility, and usability of CarbonData.
Supports SDK

Provided Carbon SDK to write and read CarbonData files through Java API,
supporting Avro schema and JSON data.
Supports External Table with Location

Now you can create external table by specifying the location of Carbon data
files.
Supports Streaming with Pre-Aggregate Table

Now you can create pre-aggregate table on streaming tables. This enhances
OLAP type of query performance on streaming tables.
Supports Partition with Pre-Aggregate

Now when you drop the partition column in the main table, the same column
can be dropped in the aggregate table keeping both in sync.
Enhanced Data Load performance

Now the data load performance has been enhanced
Supports Lucene Index for Text Search (Alpha feature)

This feature allows you to perform text search on Carbon data.
Supports S3 Read on CarbonData FilesSupports Search Mode (Alpha feature)

Supports search mode to improve concurrent queries performance.
Supports Bloom Filter Index (Alpha feature)

This feature fastens blocklet pruning.



Please find the detailed JIRA list:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12341005&styleName=Html&projectId=12320220&Create=Create&atl_token=A5KQ-2QAV-T4JA-FDED%7C72f8d21d9927bf947fc8c0dfb7f69263d4048efb%7Clout
Sub-task

   - [CARBONDATA-1522
   <https://issues.apache.org/jira/browse/CARBONDATA-1522>] - 6. Loading
   aggregation tables for streaming data tables.
   - [CARBONDATA-1575
   <https://issues.apache.org/jira/browse/CARBONDATA-1575>] - Support large
   scale data on DataMap
   - [CARBONDATA-1601
   <https://issues.apache.org/jira/browse/CARBONDATA-1601>] - Add carbon
   store module
   - [CARBONDATA-1998
   <https://issues.apache.org/jira/browse/CARBONDATA-1998>] - Support
   FileReader Java API for file level carbondata
   - [CARBONDATA-2165
   <https://issues.apache.org/jira/browse/CARBONDATA-2165>] - Remove spark
   dependency in carbon-hadoop module
   - [CARBONDATA-2189
   <https://issues.apache.org/jira/browse/CARBONDATA-2189>] - Support add
   and drop interface
   - [CARBONDATA-2206
   <https://issues.apache.org/jira/browse/CARBONDATA-2206>] - Integrate
   lucene as datamap
   - [CARBONDATA-2247
   <https://issues.apache.org/jira/browse/CARBONDATA-2247>] - Support
   writing index in CarbonWriter
   - [CARBONDATA-2294
   <https://issues.apache.org/jira/browse/CARBONDATA-2294>] - Support
   preaggregate table creation on partition tables
   - [CARBONDATA-2301
   <https://issues.apache.org/jira/browse/CARBONDATA-2301>] - Support query
   interface in CarbonStore
   - [CARBONDATA-2359
   <https://issues.apache.org/jira/browse/CARBONDATA-2359>] - Support
   applicable load options and table properties for Non Transactional table
   - [CARBONDATA-2360
   <https://issues.apache.org/jira/browse/CARBONDATA-2360>] - Insert into
   and Insert Into overwrite support for Non Transactional table
   - [CARBONDATA-2361
   <https://issues.apache.org/jira/browse/CARBONDATA-2361>] - Refactor Read
   Committed Scope implementation.
   - [CARBONDATA-2369
   <https://issues.apache.org/jira/browse/CARBONDATA-2369>] - Add a
   document for Non Transactional table with SDK writer guide
   - [CARBONDATA-2388
   <https://issues.apache.org/jira/browse/CARBONDATA-2388>] - Avro Nested
   Datatype Support
   - [CARBONDATA-2423
   <https://issues.apache.org/jira/browse/CARBONDATA-2423>] - CarbonReader
   Support To Read Non Transactional Table
   - [CARBONDATA-2430
   <https://issues.apache.org/jira/browse/CARBONDATA-2430>] - Reshuffling
   of Columns given by user in SDK
   - [CARBONDATA-2433
   <https://issues.apache.org/jira/browse/CARBONDATA-2433>] - Executor OOM
   because of GC when blocklet pruning is done using Lucene datamap
   - [CARBONDATA-2443
   <https://issues.apache.org/jira/browse/CARBONDATA-2443>] - Multi Level
   Complex Type Support for AVRO SDK
   - [CARBONDATA-2457
   <https://issues.apache.org/jira/browse/CARBONDATA-2457>] - Add converter
   to get Carbon SDK Schema from Avro schema directly.
   - [CARBONDATA-2474
   <https://issues.apache.org/jira/browse/CARBONDATA-2474>] - Support
   Modular Plan
   - [CARBONDATA-2475
   <https://issues.apache.org/jira/browse/CARBONDATA-2475>] - Support
   Materialized View query rewrite
   - [CARBONDATA-2484
   <https://issues.apache.org/jira/browse/CARBONDATA-2484>] - Refactor the
   datamap code and clear the datamap from executor on table drop

Bug

   - [CARBONDATA-1114
   <https://issues.apache.org/jira/browse/CARBONDATA-1114>] - Failed to run
   tests in windows env
   - [CARBONDATA-1990
   <https://issues.apache.org/jira/browse/CARBONDATA-1990>] - Null values
   shown when the basic word count example is tried on carbon streaming table
   - [CARBONDATA-2002
   <https://issues.apache.org/jira/browse/CARBONDATA-2002>] - Streaming
   segment status is not getting updated to finished or success
   - [CARBONDATA-2056
   <https://issues.apache.org/jira/browse/CARBONDATA-2056>] - Hadoop
   Configuration with access key and secret key should be passed while
   creating InputStream of distributed carbon file.
   - [CARBONDATA-2080
   <https://issues.apache.org/jira/browse/CARBONDATA-2080>] - Hadoop Conf
   not propagated from driver to executor in S3
   - [CARBONDATA-2085
   <https://issues.apache.org/jira/browse/CARBONDATA-2085>] - It's
   different between load twice and create datamap with load again after load
   data and create datamap
   - [CARBONDATA-2130
   <https://issues.apache.org/jira/browse/CARBONDATA-2130>] - Find some
   Spelling error in CarbonData
   - [CARBONDATA-2147
   <https://issues.apache.org/jira/browse/CARBONDATA-2147>] - Exception
   displays while loading data with streaming
   - [CARBONDATA-2152
   <https://issues.apache.org/jira/browse/CARBONDATA-2152>] - Min function
   working incorrectly for string type with dictionary include in presto.
   - [CARBONDATA-2155
   <https://issues.apache.org/jira/browse/CARBONDATA-2155>] - IS NULL not
   working correctly on string datatype with dictionary_include in presto
   integration
   - [CARBONDATA-2161
   <https://issues.apache.org/jira/browse/CARBONDATA-2161>] - Compacted
   Segment of Streaming Table should update "mergeTo" column
   - [CARBONDATA-2194
   <https://issues.apache.org/jira/browse/CARBONDATA-2194>] - Exception
   message is improper when use incorrect bad record action type
   - [CARBONDATA-2198
   <https://issues.apache.org/jira/browse/CARBONDATA-2198>] - Streaming
   data to a table with bad_records_action as IGNORE throws ClassCastException
   - [CARBONDATA-2199
   <https://issues.apache.org/jira/browse/CARBONDATA-2199>] - Exception
   occurs when change the datatype of measure having sort_column
   - [CARBONDATA-2207
   <https://issues.apache.org/jira/browse/CARBONDATA-2207>] - TestCase
   Fails using Hive Metastore
   - [CARBONDATA-2208
   <https://issues.apache.org/jira/browse/CARBONDATA-2208>] - Pre aggregate
   datamap creation is failing when count(*) present in query
   - [CARBONDATA-2209
   <https://issues.apache.org/jira/browse/CARBONDATA-2209>] - Rename table
   with partitions not working issue and batch_sort and no_sort with partition
   table issue
   - [CARBONDATA-2211
   <https://issues.apache.org/jira/browse/CARBONDATA-2211>] - Alter Table
   Streaming DDL should blocking DDL like other DDL ( All DDL are blocking DDL)
   - [CARBONDATA-2213
   <https://issues.apache.org/jira/browse/CARBONDATA-2213>] - Wrong version
   in datamap example module cause compilation failure
   - [CARBONDATA-2216
   <https://issues.apache.org/jira/browse/CARBONDATA-2216>] - Error in
   compilation and execution in sdvtest
   - [CARBONDATA-2217
   <https://issues.apache.org/jira/browse/CARBONDATA-2217>] - nullpointer
   issue drop partition where column does not exists and clean files issue
   after second level of compaction
   - [CARBONDATA-2219
   <https://issues.apache.org/jira/browse/CARBONDATA-2219>] - Add
   validation for external partition location to use same schema
   - [CARBONDATA-2221
   <https://issues.apache.org/jira/browse/CARBONDATA-2221>] - Drop table
   should throw exception when metastore operation failed
   - [CARBONDATA-2222
   <https://issues.apache.org/jira/browse/CARBONDATA-2222>] - Update the
   FAQ doc for some mistakes
   - [CARBONDATA-2229
   <https://issues.apache.org/jira/browse/CARBONDATA-2229>] - Unable to
   save dataframe as carbontable with specified external database path
   - [CARBONDATA-2232
   <https://issues.apache.org/jira/browse/CARBONDATA-2232>] - Wrong logic
   in spilling unsafe pages to disk
   - [CARBONDATA-2235
   <https://issues.apache.org/jira/browse/CARBONDATA-2235>] - add system
   configuration to filter datamaps from show tables command
   - [CARBONDATA-2236
   <https://issues.apache.org/jira/browse/CARBONDATA-2236>] - Add SDV Test
   Cases for Standard Partition
   - [CARBONDATA-2237
   <https://issues.apache.org/jira/browse/CARBONDATA-2237>] - Scala Parser
   failures are accumulated into memory form thread local
   - [CARBONDATA-2241
   <https://issues.apache.org/jira/browse/CARBONDATA-2241>] - Wrong Query
   written in Preaggregation Document
   - [CARBONDATA-2244
   <https://issues.apache.org/jira/browse/CARBONDATA-2244>] - When there
   are some invisibility INSERT_IN_PROGRESS/INSERT_OVERWRITE_IN_PROGRESS
   segments on main table, it can not create preaggregate table on it.
   - [CARBONDATA-2248
   <https://issues.apache.org/jira/browse/CARBONDATA-2248>] - Removing
   parsers thread local objects after parsing of carbon query
   - [CARBONDATA-2249
   <https://issues.apache.org/jira/browse/CARBONDATA-2249>] - Not able to
   query data through presto with local carbondata-store
   - [CARBONDATA-2261
   <https://issues.apache.org/jira/browse/CARBONDATA-2261>] - Support Set
   segment command for Streaming Table
   - [CARBONDATA-2264
   <https://issues.apache.org/jira/browse/CARBONDATA-2264>] - There is
   error when we create table using CarbonSource
   - [CARBONDATA-2265
   <https://issues.apache.org/jira/browse/CARBONDATA-2265>] - [DFX]-Load]:
   Load job fails if 1 folder contains 1000 files
   - [CARBONDATA-2266
   <https://issues.apache.org/jira/browse/CARBONDATA-2266>] - All Examples
   are throwing NoSuchElement Exception in current master branch
   - [CARBONDATA-2274
   <https://issues.apache.org/jira/browse/CARBONDATA-2274>] - Partition
   table having more than 4 column giving zero record
   - [CARBONDATA-2275
   <https://issues.apache.org/jira/browse/CARBONDATA-2275>] - Query Failed
   for 0 byte deletedelta file
   - [CARBONDATA-2277
   <https://issues.apache.org/jira/browse/CARBONDATA-2277>] - Filter on
   default values are not working
   - [CARBONDATA-2287
   <https://issues.apache.org/jira/browse/CARBONDATA-2287>] - Add event to
   alter partition table
   - [CARBONDATA-2289
   <https://issues.apache.org/jira/browse/CARBONDATA-2289>] - If carbon
   merge index is enabled then after IUD operation if some blocks of a segment
   is deleted, then during query and IUD operation the driver is throwing
   FileNotFoundException while preparing BlockMetaInfo.
   - [CARBONDATA-2302
   <https://issues.apache.org/jira/browse/CARBONDATA-2302>] - Fix some bugs
   when separate visible and invisible segments info into two files
   - [CARBONDATA-2303
   <https://issues.apache.org/jira/browse/CARBONDATA-2303>] - If dataload
   is failed for parition table then cleanup is not working.
   - [CARBONDATA-2307
   <https://issues.apache.org/jira/browse/CARBONDATA-2307>] - OOM when
   using DataFrame.coalesce
   - [CARBONDATA-2308
   <https://issues.apache.org/jira/browse/CARBONDATA-2308>] - Compaction
   should be allow when loading is in progress
   - [CARBONDATA-2314
   <https://issues.apache.org/jira/browse/CARBONDATA-2314>] - Data mismatch
   in Pre-Aggregate table after Streaming load due to threadset issue
   - [CARBONDATA-2319
   <https://issues.apache.org/jira/browse/CARBONDATA-2319>] -
   carbon_scan_time and carbon_IO_time are incorrect in task statistics
   - [CARBONDATA-2320
   <https://issues.apache.org/jira/browse/CARBONDATA-2320>] - Fix error in
   lucene coarse grain datamap suite
   - [CARBONDATA-2321
   <https://issues.apache.org/jira/browse/CARBONDATA-2321>] - Selecton
   after a Concurrent Load Failing for Partition columns
   - [CARBONDATA-2327
   <https://issues.apache.org/jira/browse/CARBONDATA-2327>] - invalid
   schema name _system shows when executed show schemas in presto
   - [CARBONDATA-2329
   <https://issues.apache.org/jira/browse/CARBONDATA-2329>] - Non
   Serializable extra info in session is overwritten by values from thread
   - [CARBONDATA-2333
   <https://issues.apache.org/jira/browse/CARBONDATA-2333>] - Block insert
   overwrite on parent table if any of the child tables are not partitioned on
   the specified partition columns
   - [CARBONDATA-2335
   <https://issues.apache.org/jira/browse/CARBONDATA-2335>] - Autohandoff
   is failing when preaggregate is created on streaming table
   - [CARBONDATA-2337
   <https://issues.apache.org/jira/browse/CARBONDATA-2337>] - Fix
   duplicately acquiring 'streaming.lock' error when integrating with
   spark-streaming
   - [CARBONDATA-2343
   <https://issues.apache.org/jira/browse/CARBONDATA-2343>] - Improper
   filter resolver cause more filter scan on data that could be skipped
   - [CARBONDATA-2346
   <https://issues.apache.org/jira/browse/CARBONDATA-2346>] - Dropping
   partition failing with null error for Partition table with Pre-Aggregate
   tables
   - [CARBONDATA-2347
   <https://issues.apache.org/jira/browse/CARBONDATA-2347>] - Fix
   Functional issues in LuceneDatamap in load and query and make stable
   - [CARBONDATA-2350
   <https://issues.apache.org/jira/browse/CARBONDATA-2350>] - Fix bugs in
   minmax datamap example
   - [CARBONDATA-2364
   <https://issues.apache.org/jira/browse/CARBONDATA-2364>] - Remove
   useless and time consuming code block
   - [CARBONDATA-2366
   <https://issues.apache.org/jira/browse/CARBONDATA-2366>] - Concurrent
   Datamap creation is failing when using hive metastore
   - [CARBONDATA-2374
   <https://issues.apache.org/jira/browse/CARBONDATA-2374>] - Fix bugs in
   minmax datamap example
   - [CARBONDATA-2386
   <https://issues.apache.org/jira/browse/CARBONDATA-2386>] - Query on
   Pre-Aggregate table is slower
   - [CARBONDATA-2391
   <https://issues.apache.org/jira/browse/CARBONDATA-2391>] - Thread leak
   in compaction operation if prefetch is enabled and compaction process is
   killed
   - [CARBONDATA-2394
   <https://issues.apache.org/jira/browse/CARBONDATA-2394>] - Setting
   segments in thread local space but not getting reflected in the driver
   - [CARBONDATA-2401
   <https://issues.apache.org/jira/browse/CARBONDATA-2401>] - Date and
   Timestamp options are not working in SDK
   - [CARBONDATA-2406
   <https://issues.apache.org/jira/browse/CARBONDATA-2406>] - Dictionary
   Server and Dictionary Client MD5 Validation failed with
   hive.server2.enable.doAs = true
   - [CARBONDATA-2408
   <https://issues.apache.org/jira/browse/CARBONDATA-2408>] - Before
   register to master, the master maybe not finished the start service.
   - [CARBONDATA-2410
   <https://issues.apache.org/jira/browse/CARBONDATA-2410>] - Error message
   correction when column value length exceeds 320000 charactor
   - [CARBONDATA-2413
   <https://issues.apache.org/jira/browse/CARBONDATA-2413>] - After running
   CarbonWriter, there is null directory about datamap
   - [CARBONDATA-2417
   <https://issues.apache.org/jira/browse/CARBONDATA-2417>] - SDK writer
   goes to infinite wait when consumer thread goes dead
   - [CARBONDATA-2419
   <https://issues.apache.org/jira/browse/CARBONDATA-2419>] - sortColumns
   Order we are getting wrong as we set for external table is fixed
   - [CARBONDATA-2426
   <https://issues.apache.org/jira/browse/CARBONDATA-2426>] - IOException
   after compaction on Pre-Aggregate table on Partition table
   - [CARBONDATA-2427
   <https://issues.apache.org/jira/browse/CARBONDATA-2427>] - Fix
   SearchMode Serialization Issue during Load
   - [CARBONDATA-2431
   <https://issues.apache.org/jira/browse/CARBONDATA-2431>] - Incremental
   data added after table creation is not reflecting while doing select query.
   - [CARBONDATA-2432
   <https://issues.apache.org/jira/browse/CARBONDATA-2432>] - BloomFilter
   DataMap should be contained in carbon assembly jar
   - [CARBONDATA-2435
   <https://issues.apache.org/jira/browse/CARBONDATA-2435>] - SDK
   dependency Spark jar
   - [CARBONDATA-2436
   <https://issues.apache.org/jira/browse/CARBONDATA-2436>] - Block pruning
   problem post the carbon schema restructure.
   - [CARBONDATA-2437
   <https://issues.apache.org/jira/browse/CARBONDATA-2437>] - Complex Type
   data loading is failing is for null values
   - [CARBONDATA-2438
   <https://issues.apache.org/jira/browse/CARBONDATA-2438>] - Remove
   spark/hadoop related classes in carbon assembly
   - [CARBONDATA-2439
   <https://issues.apache.org/jira/browse/CARBONDATA-2439>] - Update guava
   version for bloom datamap
   - [CARBONDATA-2440
   <https://issues.apache.org/jira/browse/CARBONDATA-2440>] - In SDK user
   can not specified the Unsafe memory , so it should take complete from Heap
   , and it should not be sorted using unsafe.
   - [CARBONDATA-2441
   <https://issues.apache.org/jira/browse/CARBONDATA-2441>] - Implement
   distribute interface for bloom datamap
   - [CARBONDATA-2442
   <https://issues.apache.org/jira/browse/CARBONDATA-2442>] - Reading two
   sdk writer output with differnt schema should prompt exception
   - [CARBONDATA-2463
   <https://issues.apache.org/jira/browse/CARBONDATA-2463>] - if two insert
   operations are running concurrently 1 task fails and causes wrong no of
   records in select
   - [CARBONDATA-2464
   <https://issues.apache.org/jira/browse/CARBONDATA-2464>] - Fixed OOM in
   case of complex type
   - [CARBONDATA-2465
   <https://issues.apache.org/jira/browse/CARBONDATA-2465>] - Improve the
   carbondata file reliability in data load when direct hdfs write is enabled
   - [CARBONDATA-2468
   <https://issues.apache.org/jira/browse/CARBONDATA-2468>] - sortcolumns
   considers all dimension also if few columns specified for sort_columns prop
   - [CARBONDATA-2469
   <https://issues.apache.org/jira/browse/CARBONDATA-2469>] - External
   Table must show its location instead of default store path in describe
   formatted
   - [CARBONDATA-2472
   <https://issues.apache.org/jira/browse/CARBONDATA-2472>] - Refactor
   NonTransactional table code for Index file IO performance
   - [CARBONDATA-2476
   <https://issues.apache.org/jira/browse/CARBONDATA-2476>] - Fix bug in
   bloom datamap cache
   - [CARBONDATA-2477
   <https://issues.apache.org/jira/browse/CARBONDATA-2477>] - No dictionary
   Complex type with double/date/decimal data type table creation is failing
   - [CARBONDATA-2479
   <https://issues.apache.org/jira/browse/CARBONDATA-2479>] - Multiple
   issue in sdk writer and external table flow
   - [CARBONDATA-2480
   <https://issues.apache.org/jira/browse/CARBONDATA-2480>] - Search mode
   RuntimeException: Error while resolving filter expression
   - [CARBONDATA-2486
   <https://issues.apache.org/jira/browse/CARBONDATA-2486>] - set search
   mode information is not updated in the documentation
   - [CARBONDATA-2487
   <https://issues.apache.org/jira/browse/CARBONDATA-2487>] - Block filters
   for lucene with more than one text_match udf
   - [CARBONDATA-2489
   <https://issues.apache.org/jira/browse/CARBONDATA-2489>] - Fix coverity
   reported warnings
   - [CARBONDATA-2492
   <https://issues.apache.org/jira/browse/CARBONDATA-2492>] - Thread leak
   issue in case of any data load failure
   - [CARBONDATA-2493
   <https://issues.apache.org/jira/browse/CARBONDATA-2493>] -
   DataType.equals() failes for complex types
   - [CARBONDATA-2498
   <https://issues.apache.org/jira/browse/CARBONDATA-2498>] - Change
   CarbonWriterBuilder interface to take schema while creating writer
   - [CARBONDATA-2503
   <https://issues.apache.org/jira/browse/CARBONDATA-2503>] - Data write
   fails if empty value is provided for sort columns in sdk
   - [CARBONDATA-2520
   <https://issues.apache.org/jira/browse/CARBONDATA-2520>] - datamap
   writers are not getting closed on task failure
   - [CARBONDATA-2538
   <https://issues.apache.org/jira/browse/CARBONDATA-2538>] - No exception
   is thrown if writer path has only lock files
   - [CARBONDATA-2545
   <https://issues.apache.org/jira/browse/CARBONDATA-2545>] - Fix some
   spell error in CarbonData
   - [CARBONDATA-2552
   <https://issues.apache.org/jira/browse/CARBONDATA-2552>] - Fix Data
   Mismatch for Complex Data type Array of Timestamp with Dictionary Include
   - [CARBONDATA-2555
   <https://issues.apache.org/jira/browse/CARBONDATA-2555>] - SDK Reader
   should have isTransactionalTable = false by default, to be inline with SDK
   writer

New Feature

   - [CARBONDATA-1516
   <https://issues.apache.org/jira/browse/CARBONDATA-1516>] - Support
   pre-aggregate tables and timeseries in carbondata
   - [CARBONDATA-2055
   <https://issues.apache.org/jira/browse/CARBONDATA-2055>] - Support
   integrating Streaming table with Spark Streaming
   - [CARBONDATA-2242
   <https://issues.apache.org/jira/browse/CARBONDATA-2242>] - Support
   materialized view
   - [CARBONDATA-2253
   <https://issues.apache.org/jira/browse/CARBONDATA-2253>] - Support write
   JSON/Avro data to carbon files
   - [CARBONDATA-2262
   <https://issues.apache.org/jira/browse/CARBONDATA-2262>] - Create table
   should support using carbondata and stored as carbondata
   - [CARBONDATA-2267
   <https://issues.apache.org/jira/browse/CARBONDATA-2267>] - Implement
   Reading Of Carbon Partition From Presto
   - [CARBONDATA-2276
   <https://issues.apache.org/jira/browse/CARBONDATA-2276>] - Support SDK
   API to read schema in data file and schema file
   - [CARBONDATA-2278
   <https://issues.apache.org/jira/browse/CARBONDATA-2278>] - Save the
   datamaps to system folder of warehouse
   - [CARBONDATA-2291
   <https://issues.apache.org/jira/browse/CARBONDATA-2291>] - Add datamap
   status and refresh command to sync data manually to datamaps
   - [CARBONDATA-2296
   <https://issues.apache.org/jira/browse/CARBONDATA-2296>] - Test famework
   should take the location of local module target folder if not integrtion
   module
   - [CARBONDATA-2297
   <https://issues.apache.org/jira/browse/CARBONDATA-2297>] - Support
   SEARCH_MODE for basic filter query
   - [CARBONDATA-2312
   <https://issues.apache.org/jira/browse/CARBONDATA-2312>] - Support In
   Memory catalog
   - [CARBONDATA-2323
   <https://issues.apache.org/jira/browse/CARBONDATA-2323>] - Distributed
   search mode using gRPC
   - [CARBONDATA-2371
   <https://issues.apache.org/jira/browse/CARBONDATA-2371>] - Add Profiler
   output in EXPLAIN command
   - [CARBONDATA-2373
   <https://issues.apache.org/jira/browse/CARBONDATA-2373>] - Add bloom
   filter datamap to support precise query
   - [CARBONDATA-2378
   <https://issues.apache.org/jira/browse/CARBONDATA-2378>] - Support
   enable/disable search mode in ThriftServer
   - [CARBONDATA-2380
   <https://issues.apache.org/jira/browse/CARBONDATA-2380>] - Support
   visible/invisible datamap for performance tuning
   - [CARBONDATA-2415
   <https://issues.apache.org/jira/browse/CARBONDATA-2415>] - All DataMap
   should support REFRESH command
   - [CARBONDATA-2416
   <https://issues.apache.org/jira/browse/CARBONDATA-2416>] - Index DataMap
   should support immediate load and deferred load when creating the DataMap

Improvement

   - [CARBONDATA-1663
   <https://issues.apache.org/jira/browse/CARBONDATA-1663>] - Decouple
   spark in carbon modules
   - [CARBONDATA-2018
   <https://issues.apache.org/jira/browse/CARBONDATA-2018>] - Optimization
   in reading/writing for sort temp row during data loading
   - [CARBONDATA-2032
   <https://issues.apache.org/jira/browse/CARBONDATA-2032>] - Skip writing
   final data files to local disk to save disk IO in data loading
   - [CARBONDATA-2099
   <https://issues.apache.org/jira/browse/CARBONDATA-2099>] - Refactor on
   query scan process to improve readability
   - [CARBONDATA-2139
   <https://issues.apache.org/jira/browse/CARBONDATA-2139>] - Optimize CTAS
   documentation and test case
   - [CARBONDATA-2140
   <https://issues.apache.org/jira/browse/CARBONDATA-2140>] - Presto
   Integration - Code Refactoring
   - [CARBONDATA-2148
   <https://issues.apache.org/jira/browse/CARBONDATA-2148>] - Use Row
   parser to replace current default parser:CSVStreamParserImp
   - [CARBONDATA-2159
   <https://issues.apache.org/jira/browse/CARBONDATA-2159>] - Remove
   carbon-spark dependency for sdk module
   - [CARBONDATA-2168
   <https://issues.apache.org/jira/browse/CARBONDATA-2168>] - Support
   global sort on partition tables
   - [CARBONDATA-2184
   <https://issues.apache.org/jira/browse/CARBONDATA-2184>] - Improve
   memory reuse for heap memory in `HeapMemoryAllocator`
   - [CARBONDATA-2187
   <https://issues.apache.org/jira/browse/CARBONDATA-2187>] - Restructure
   the partition folders as per the standard hive folders
   - [CARBONDATA-2196
   <https://issues.apache.org/jira/browse/CARBONDATA-2196>] - during stream
   sometime carbontable is null in executor side
   - [CARBONDATA-2204
   <https://issues.apache.org/jira/browse/CARBONDATA-2204>] - Access
   tablestatus file too many times during query
   - [CARBONDATA-2223
   <https://issues.apache.org/jira/browse/CARBONDATA-2223>] - Adding
   Listener Support for Partition
   - [CARBONDATA-2226
   <https://issues.apache.org/jira/browse/CARBONDATA-2226>] - Refactor UT's
   to remove duplicate test scenarios to improve CI time for PreAggregate
   create and drop feature
   - [CARBONDATA-2227
   <https://issues.apache.org/jira/browse/CARBONDATA-2227>] - Add Partition
   Values and Location information in describe formatted for Standard
   partition feature
   - [CARBONDATA-2230
   <https://issues.apache.org/jira/browse/CARBONDATA-2230>] - Add a path
   into table path to store lock files and delete useless segment lock files
   before loading
   - [CARBONDATA-2231
   <https://issues.apache.org/jira/browse/CARBONDATA-2231>] - Refactor FT's
   to remove duplicate test scenarios to improve CI time for Streaming feature
   - [CARBONDATA-2234
   <https://issues.apache.org/jira/browse/CARBONDATA-2234>] - Support UTF-8
   with BOM encoding in CSVInputFormat
   - [CARBONDATA-2250
   <https://issues.apache.org/jira/browse/CARBONDATA-2250>] - Reduce
   massive object generation in global sort
   - [CARBONDATA-2251
   <https://issues.apache.org/jira/browse/CARBONDATA-2251>] - Refactored
   sdv failures running on different environment
   - [CARBONDATA-2254
   <https://issues.apache.org/jira/browse/CARBONDATA-2254>] - Optimize
   CarbonData documentation
   - [CARBONDATA-2255
   <https://issues.apache.org/jira/browse/CARBONDATA-2255>] - Should rename
   the streaming examples to make it easy to understand
   - [CARBONDATA-2256
   <https://issues.apache.org/jira/browse/CARBONDATA-2256>] - Adding sdv
   Testcases for SET_Parameter_Dynamically_Feature
   - [CARBONDATA-2258
   <https://issues.apache.org/jira/browse/CARBONDATA-2258>] - Separate
   visible and invisible segments info into two files to reduce the size of
   tablestatus file.
   - [CARBONDATA-2260
   <https://issues.apache.org/jira/browse/CARBONDATA-2260>] -
   CarbonThriftServer should support S3 carbon table
   - [CARBONDATA-2271
   <https://issues.apache.org/jira/browse/CARBONDATA-2271>] - Collect SQL
   execution information to driver side
   - [CARBONDATA-2285
   <https://issues.apache.org/jira/browse/CARBONDATA-2285>] - spark
   integration code refactor
   - [CARBONDATA-2295
   <https://issues.apache.org/jira/browse/CARBONDATA-2295>] - Add
   UNSAFE_WORKING_MEMORY_IN_MB as a configuration parameter in presto
   integration
   - [CARBONDATA-2298
   <https://issues.apache.org/jira/browse/CARBONDATA-2298>] - Delete
   segment lock files before update metadata
   - [CARBONDATA-2299
   <https://issues.apache.org/jira/browse/CARBONDATA-2299>] - Support
   showing all segment information(include visible and invisible segments)
   - [CARBONDATA-2304
   <https://issues.apache.org/jira/browse/CARBONDATA-2304>] - Enhance
   compaction performance by enabling prefetch
   - [CARBONDATA-2310
   <https://issues.apache.org/jira/browse/CARBONDATA-2310>] - Refactored
   code to improve Distributable interface
   - [CARBONDATA-2315
   <https://issues.apache.org/jira/browse/CARBONDATA-2315>] - DataLoad is
   showing success and failure message in log,when no data is loaded into
   table during LOAD
   - [CARBONDATA-2316
   <https://issues.apache.org/jira/browse/CARBONDATA-2316>] - Even though
   one of the Compaction task failed at executor. All the executor task is
   showing success in UI and Job fails from driver.
   - [CARBONDATA-2317
   <https://issues.apache.org/jira/browse/CARBONDATA-2317>] - concurrent
   datamap with same name and schema creation throws exception
   - [CARBONDATA-2324
   <https://issues.apache.org/jira/browse/CARBONDATA-2324>] - Support
   config ExecutorService in search mode
   - [CARBONDATA-2325
   <https://issues.apache.org/jira/browse/CARBONDATA-2325>] - Page level
   uncompress and Query performance improvement for Unsafe No Dictionary
   - [CARBONDATA-2338
   <https://issues.apache.org/jira/browse/CARBONDATA-2338>] - Add example
   to upload data to S3 by using SDK
   - [CARBONDATA-2341
   <https://issues.apache.org/jira/browse/CARBONDATA-2341>] - Add CleanUp
   for Pre-Aggregate table
   - [CARBONDATA-2353
   <https://issues.apache.org/jira/browse/CARBONDATA-2353>] - Add cache for
   DataMap schema provider to avoid IO for each read
   - [CARBONDATA-2357
   <https://issues.apache.org/jira/browse/CARBONDATA-2357>] - Add column
   name and index mapping in lucene datamap writer
   - [CARBONDATA-2358
   <https://issues.apache.org/jira/browse/CARBONDATA-2358>] - Dataframe
   overwrite does not work properly if the table is already created and has
   deleted segments
   - [CARBONDATA-2365
   <https://issues.apache.org/jira/browse/CARBONDATA-2365>] - Add
   QueryExecutor in SearchMode for row-based CarbonRecordReader
   - [CARBONDATA-2375
   <https://issues.apache.org/jira/browse/CARBONDATA-2375>] - Add CG prune
   before FG prune
   - [CARBONDATA-2376
   <https://issues.apache.org/jira/browse/CARBONDATA-2376>] - Improve
   Lucene datamap performance by eliminating blockid while writing and reading
   index.
   - [CARBONDATA-2379
   <https://issues.apache.org/jira/browse/CARBONDATA-2379>] - Support
   Search mode run in the cluster and fix some error
   - [CARBONDATA-2381
   <https://issues.apache.org/jira/browse/CARBONDATA-2381>] - Improve
   compaction performance by filling batch result in columnar format and
   performing IO at blocklet level
   - [CARBONDATA-2384
   <https://issues.apache.org/jira/browse/CARBONDATA-2384>] - SDK support
   write/read data into/from S3
   - [CARBONDATA-2390
   <https://issues.apache.org/jira/browse/CARBONDATA-2390>] - Refresh
   Lucene data map for the exists table with data
   - [CARBONDATA-2392
   <https://issues.apache.org/jira/browse/CARBONDATA-2392>] - Add close
   method for CarbonReader
   - [CARBONDATA-2396
   <https://issues.apache.org/jira/browse/CARBONDATA-2396>] - Add CTAS
   support for using DataSource Syntax
   - [CARBONDATA-2404
   <https://issues.apache.org/jira/browse/CARBONDATA-2404>] - Add
   documentation for using carbondata and stored as carbondata
   - [CARBONDATA-2407
   <https://issues.apache.org/jira/browse/CARBONDATA-2407>] - Removed All
   Unused Executor BTree code
   - [CARBONDATA-2414
   <https://issues.apache.org/jira/browse/CARBONDATA-2414>] - Optimize
   documents for sort_column_bounds
   - [CARBONDATA-2422
   <https://issues.apache.org/jira/browse/CARBONDATA-2422>] - Search mode
   Master port should be dynamic
   - [CARBONDATA-2448
   <https://issues.apache.org/jira/browse/CARBONDATA-2448>] - Adding
   compacted segments to load and alter events
   - [CARBONDATA-2454
   <https://issues.apache.org/jira/browse/CARBONDATA-2454>] - Add false
   positive probability property for bloom filter datamap
   - [CARBONDATA-2455
   <https://issues.apache.org/jira/browse/CARBONDATA-2455>] - Fix _System
   Folder creation and lucene AND,OR,NOT Filter fix
   - [CARBONDATA-2458
   <https://issues.apache.org/jira/browse/CARBONDATA-2458>] - Remove
   unnecessary TableProvider interface
   - [CARBONDATA-2459
   <https://issues.apache.org/jira/browse/CARBONDATA-2459>] - Support cache
   for bloom datamap
   - [CARBONDATA-2467
   <https://issues.apache.org/jira/browse/CARBONDATA-2467>] - Null is
   printed in the SDK writer logs for operations logged
   - [CARBONDATA-2470
   <https://issues.apache.org/jira/browse/CARBONDATA-2470>] - Refactor
   AlterTableCompactionPostStatusUpdateEvent usage in compaction flow
   - [CARBONDATA-2473
   <https://issues.apache.org/jira/browse/CARBONDATA-2473>] - Support
   Materialized View as enhanced Preaggregate DataMap
   - [CARBONDATA-2494
   <https://issues.apache.org/jira/browse/CARBONDATA-2494>] - Improve
   Lucene datamap size and performnace.
   - [CARBONDATA-2495
   <https://issues.apache.org/jira/browse/CARBONDATA-2495>] - Add document
   for bloomfilter datamap
   - [CARBONDATA-2496
   <https://issues.apache.org/jira/browse/CARBONDATA-2496>] - Chnage the
   bloom implementation to hadoop for better performance and compression
   - [CARBONDATA-2524
   <https://issues.apache.org/jira/browse/CARBONDATA-2524>] - Support
   create carbonReader with default projection

Test

   - [CARBONDATA-2073
   <https://issues.apache.org/jira/browse/CARBONDATA-2073>] - Add test
   cases for pre aggregate table
   - [CARBONDATA-2257
   <https://issues.apache.org/jira/browse/CARBONDATA-2257>] - Add SDV test
   cases for Partition with Global Sort
   - [CARBONDATA-2352
   <https://issues.apache.org/jira/browse/CARBONDATA-2352>] - Add SDV Test
   Cases for Partition with Pre-Aggregate table
   - [CARBONDATA-2356
   <https://issues.apache.org/jira/browse/CARBONDATA-2356>] - Adding UT for
   Lucene DataMap

Task

   -
      - [CARBONDATA-1827
      <https://issues.apache.org/jira/browse/CARBONDATA-1827>] - Add
      Support to provide S3 Functionality in Carbondata
      - [CARBONDATA-1959
      <https://issues.apache.org/jira/browse/CARBONDATA-1959>] - Support
      compaction on S3 table
      - [CARBONDATA-1960
      <https://issues.apache.org/jira/browse/CARBONDATA-1960>] - Add
      example for creating a local table and load CSV data which is
stored in S3.
      - [CARBONDATA-1961
      <https://issues.apache.org/jira/browse/CARBONDATA-1961>] - Support
      data update/delete on S3 table
      - [CARBONDATA-2135
      <https://issues.apache.org/jira/browse/CARBONDATA-2135>] -
      Documentation for Table Comment and Column Comment
      - [CARBONDATA-2138
      <https://issues.apache.org/jira/browse/CARBONDATA-2138>] -
      Documentation for HEADER option
      - [CARBONDATA-2214
      <https://issues.apache.org/jira/browse/CARBONDATA-2214>] - Remove
      config 'spark.sql.hive.thriftServer.singleSession' from
      installation-guide.md
      - [CARBONDATA-2215
      <https://issues.apache.org/jira/browse/CARBONDATA-2215>] - Add the
      description of Carbon Stream Parser into streaming-guide.md
      - [CARBONDATA-2259
      <https://issues.apache.org/jira/browse/CARBONDATA-2259>] - Add auto
      CI for examples
      - [CARBONDATA-2300
      <https://issues.apache.org/jira/browse/CARBONDATA-2300>] - Add
      ENABLE_UNSAFE_IN_QUERY_EXECUTION as a configuration parameter in presto
      integration
      - [CARBONDATA-2370
      <https://issues.apache.org/jira/browse/CARBONDATA-2370>] - Document
      for Presto cluster setup for carbondata
      - [CARBONDATA-2424
      <https://issues.apache.org/jira/browse/CARBONDATA-2424>] - Add
      documentation for properties of Pre-Aggregate tables
      - [CARBONDATA-2434
      <https://issues.apache.org/jira/browse/CARBONDATA-2434>] - Add
      ExternalTableExample and LuceneDataMapExample
      - [CARBONDATA-2507
      <https://issues.apache.org/jira/browse/CARBONDATA-2507>] - Some
      properties not validate in CarbonData, like enable.offheap.sort

[Liang Chen]

Hi

Please find the updated 1.4.0 release notes:
https://cwiki.apache.org/confluence/display/CARBONDATA/Apache+CarbonData+1.4.0+Release

Regards
Liang



--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/