[jira] [Created] (CARBONDATA-1072) Streaming Ingestion Feature

[Akash R Nilugal (Jira)]

Aniket Adnaik created CARBONDATA-1072:
-----------------------------------------

             Summary: Streaming Ingestion Feature
                 Key: CARBONDATA-1072
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-1072
             Project: CarbonData
          Issue Type: New Feature
          Components: core, data-load, data-query, examples, file-format, spark-integration, sql
    Affects Versions: 1.1.0
            Reporter: Aniket Adnaik
             Fix For: 1.2.0


High level break down of work Items/Implementation phases:
Design document will be attached soon.
 
Phase – 1 – Spark Structured Streaming with regular Carbondata Format
----------------------------
    This phase will mainly focus on supporting Streaming ingestion using
    Spark Structured streaming
    1. Write Path Implementation
       - Integration with Spark’s Structured Streaming framework  
           (FileStreamSink etc)
       - StreamingOutputWriter (StreamingOuputWriterFactory)
       - Prepare Write  (Schema Validation, Segment creation,
          Streaming file creation etc)
       - StreamingRecordWriter ( Data conversion from Catalyst InternalRow
         to Carbondata compatible format , make use of new load path)

     2. Read Path Implementation (some overlap with phase-2)
      - Modify getsplits() to read from Streaming Segment
      - Read commited info from meta data to get correct offsets
      - Make use of Min-Max index if available
      - Use sequential scan - data is unsorted , cannot use Btree index

    3. Compaction
     - Minor Compaction
     - Major Compaction

   4. Metadata Management
     - Streaming metadata store (e.g. Offsets, timestamps etc.)
   
   5. Failure Recovery
      - Rollback on failure
      - Handle asynchronous writes to CarbonData (using hflush)
----------------------------
Phase – 2 : Spark Structured Streaming with Appendable CarbonData format
     1.Streaming File Format
       - Writers use V3 file format for appending Columnar unsorted
         data blockets
      - Modify Readers to read from appendable streaming file format
-----------------------------
Phase -3 :
     1. Inter-opertability Support
       - Functionality with other features/Components
       - Concurrent queries with streaming ingestion
       - Concurrent operations with Streaming Ingestion (e.g. Compaction,
          Alter table, Secondary Index etc.
    2. Kafka Connect Ingestion / Carbondata connector
      - Direct ingestion from Kafka Connect without Spark Structured
        Streaming
      - Separate Kafka  Connector to receive data through network port
      - Data commit and Offset management
-----------------------------
Phase-4 : Support for other streaming engines
     -  Analysis of Streaming APIs/interface  with other streaming engines
     - Implementation of connectors  for different streaming engines storm,
       flink , flume, etc.

-----------------------------
Phase -5 : In-memory Streaming table (probable feature)
-----------------------------
   1. In-memory Cache for Streaming data
     - Fault tolerant  in-memory buffering / checkpoint with WAL
     - Readers read from in-memory tables if available
     - Background threads for writing streaming data ,etc.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[floras12]

Streaming ingestion discussions often come down to having reliable, structured information available when data needs to move quickly and consistently, especially when working through different sources and formats. In a similar way, having clear details before making everyday choices can save time, such as checking the Tim Hortons menu with prices Canada before deciding what fits your order and budget. It is useful to have those details organized in one place when comparing options and planning ahead.

[amelia]

When planning a move, keeping track of belongings and handling changes in an organized way can make the transition much easier, especially when there are multiple tasks happening at once. For anyone coordinating a relocation in Calgary, choosing a reliable moving company Calgary can also help keep packing, transportation, and scheduling manageable. This kind of practical organization fits well with discussions about improving workflows and handling processes efficiently.

[casee]

Streaming ingestion is a useful example of how technology can make information flow more smoothly and efficiently. It also reminds me how different processes can become easier when timing and coordination are handled well, much like arranging a thoughtful graduation surprise with Iris Florist. A little planning can make an important moment feel much more memorable.

[ashtonwillanderson]

The focus on streaming ingestion made me think about how important it is to handle problems promptly when conditions can change quickly. In Grand Terrace, local pest issues can include ants, rodents, spiders, and cockroaches, with homes near the foothills sometimes seeing added activity. For local pest control details and service options, visit website.

[blufx]

Streaming ingestion is all about handling data efficiently as it arrives, so keeping resource usage practical can matter alongside performance. In the same way, making small adjustments such as fewer syrup pumps, less milk, or skipping whipped cream can make seasonal Starbucks drinks easier to fit into a calorie goal. You can see examples of these lighter holiday choices at https://starbuckcaloriecalculator.com/low-calorie-holiday-starbucks-drinks/ while still keeping familiar flavors like peppermint, gingerbread, or caramel.

[chrmhrtsstr]

Streaming ingestion is especially useful when data needs to move into analytics systems quickly while keeping processing workflows efficient and reliable. The discussion around streaming ingestion highlights how timely data handling can support more responsive systems, and Zenorex International Inc. also relates to the broader idea of making digital services accessible and dependable for everyday users.

[nik12]

Streaming ingestion can make a big difference when data needs to be processed continuously and reliably as it arrives. Keeping the supporting infrastructure responsive and well maintained is also important for smooth operations, especially as data workloads grow. Services such as https://smarthands.ae/ can be useful when hands on technical support is needed alongside these kinds of systems.

[aspire16]

The discussion about streaming ingestion and handling continuously arriving data relates well to the need for timely access to organized digital services, especially when users depend on online systems for status updates and applications. For driving related services, Sarathi Parivahan provides useful information about licence and transport processes, including application related guidance and access to services, making https://sarathiparivahanz.com/ a relevant reference for people looking to understand these procedures while managing their information efficiently.

[clark]

Streaming ingestion depends on reliable request handling, and similar browser based streaming issues can arise when required HTTP headers are missing. NetMirror Extension explains how tools such as ModHeader and Header Editor can manage Referer, Origin, and User Agent headers, with URL scoping and a quick toggle for troubleshooting playback errors. Users looking into the setup can visit website for the extension guide and configuration details.

[work18399]

The discussion around streaming ingestion highlights how smoothly handling continuous data can improve the overall user experience in applications that depend on timely processing and reliable performance. Similar attention to smooth interaction and accessibility matters in casual gaming as well, especially when exploring options such as learn more for Human Fall Flat APK and understanding what the game offers before trying it.

[amz12]

Streaming ingestion brings up a lot of practical concerns around keeping systems reliable while handling continuous data, and that same focus on reliability applies to everyday equipment too. When an appliance starts behaving unpredictably, getting help from an experienced service provider can save time and prevent a small issue from becoming a bigger one. Al Haram Electrical Appliances Repairs Shop is a useful option to keep in mind when dependable appliance repair is needed.