[GitHub] carbondata pull request #2262: [CARBONDATA-2431] Incremental data added afte...

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5681/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4521/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4528/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5688/



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2262#discussion_r186369605
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/CommitedIndexFilesUpdatedTimestamp.java ---
    @@ -0,0 +1,73 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.readcommitter;
    +
    +import java.io.Serializable;
    +import java.util.HashMap;
    +import java.util.Map;
    +
    +public class CommitedIndexFilesUpdatedTimestamp implements Serializable {
    --- End diff --
   
    Better rename something like SegmentModifyUpdater


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2262#discussion_r186372903
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/CommitedIndexFilesUpdatedTimestamp.java ---
    @@ -0,0 +1,73 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.readcommitter;
    +
    +import java.io.Serializable;
    +import java.util.HashMap;
    +import java.util.Map;
    +
    +public class CommitedIndexFilesUpdatedTimestamp implements Serializable {
    +
    +  private Map<Long, Integer> updatedTimestampMap = new HashMap<>();
    --- End diff --
   
    It always has 2 attributes, not a map,
    1. timestamp : It has max modified time across all files in a segment.
    2. count.: the total number of files in a segment.



---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2262#discussion_r186373811
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/CommitedIndexFilesUpdatedTimestamp.java ---
    @@ -0,0 +1,73 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.readcommitter;
    +
    +import java.io.Serializable;
    +import java.util.HashMap;
    +import java.util.Map;
    +
    +public class CommitedIndexFilesUpdatedTimestamp implements Serializable {
    +
    +  private Map<Long, Integer> updatedTimestampMap = new HashMap<>();
    +
    +  private static CommitedIndexFilesUpdatedTimestamp commitedIndexFilesUpdatedTimestamp =
    --- End diff --
   
    Directly pass this object to SegmentRefresher, there it will be cached. No need to cache here


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2262#discussion_r186373935
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/CommitedIndexFilesUpdatedTimestamp.java ---
    @@ -0,0 +1,73 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.readcommitter;
    +
    +import java.io.Serializable;
    +import java.util.HashMap;
    +import java.util.Map;
    +
    +public class CommitedIndexFilesUpdatedTimestamp implements Serializable {
    +
    +  private Map<Long, Integer> updatedTimestampMap = new HashMap<>();
    +
    +  private static CommitedIndexFilesUpdatedTimestamp commitedIndexFilesUpdatedTimestamp =
    +      new CommitedIndexFilesUpdatedTimestamp();
    +
    +  public static CommitedIndexFilesUpdatedTimestamp getInstance() {
    +    return commitedIndexFilesUpdatedTimestamp;
    +  }
    +
    +  public Map<Long, Integer> getUpdatedTimestampMap() {
    +    return updatedTimestampMap;
    +  }
    +
    +  public void addUpdatedTimestampMap(Long timestampMap) {
    --- End diff --
   
    Use constructor, don't use setters


---

[qiuchenjian-2]

Github user ravipesala commented on a diff in the pull request:

    https://github.com/apache/carbondata/pull/2262#discussion_r186374044
 
    --- Diff: core/src/main/java/org/apache/carbondata/core/readcommitter/CommitedIndexFilesUpdatedTimestamp.java ---
    @@ -0,0 +1,73 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.carbondata.core.readcommitter;
    +
    +import java.io.Serializable;
    +import java.util.HashMap;
    +import java.util.Map;
    +
    +public class CommitedIndexFilesUpdatedTimestamp implements Serializable {
    +
    +  private Map<Long, Integer> updatedTimestampMap = new HashMap<>();
    +
    +  private static CommitedIndexFilesUpdatedTimestamp commitedIndexFilesUpdatedTimestamp =
    +      new CommitedIndexFilesUpdatedTimestamp();
    +
    +  public static CommitedIndexFilesUpdatedTimestamp getInstance() {
    +    return commitedIndexFilesUpdatedTimestamp;
    +  }
    +
    +  public Map<Long, Integer> getUpdatedTimestampMap() {
    +    return updatedTimestampMap;
    +  }
    +
    +  public void addUpdatedTimestampMap(Long timestampMap) {
    +    Integer c = updatedTimestampMap.get(timestampMap);
    +    if (c == null) {
    +      updatedTimestampMap.put(timestampMap, 1);
    +    } else {
    +      updatedTimestampMap.put(timestampMap, c + 1);
    +    }
    +  }
    +
    +  public void reset() {
    +    this.updatedTimestampMap = new HashMap<>();
    +  }
    +
    +  @Override public boolean equals(Object o) {
    +    if (this == o) return true;
    +    if (!(o instanceof CommitedIndexFilesUpdatedTimestamp)) return false;
    +
    +    CommitedIndexFilesUpdatedTimestamp that = (CommitedIndexFilesUpdatedTimestamp) o;
    +
    +    return updatedTimestampMap.equals(that.updatedTimestampMap);
    +  }
    +
    +  @Override public int hashCode() {
    +    return updatedTimestampMap.hashCode();
    +  }
    +
    +  public CommitedIndexFilesUpdatedTimestamp getClone() {
    --- End diff --
   
    It is not needed, create new object every time


---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4765/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5706/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4545/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5708/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5709/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4547/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4784/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5716/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4555/



---

[qiuchenjian-2]

Github user ravipesala commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4786/



---

[qiuchenjian-2]

Github user CarbonDataQA commented on the issue:

    https://github.com/apache/carbondata/pull/2262
 
    Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5717/



---