Apache CarbonData Dev Mailing List archive › Apache CarbonData JIRA issues

[GitHub] carbondata pull request #2113: [WIP][LUCENE_DATAMAP]load issue in lucene dat...

Classic

List

Threaded

124 messages Options

1234 ... 7

qiuchenjian-2

[GitHub] carbondata pull request #2113: [WIP][LUCENE_DATAMAP]load issue in lucene dat...

GitHub user akashrn5 opened a pull request:

https://github.com/apache/carbondata/pull/2113

[WIP][LUCENE_DATAMAP]load issue in lucene datamap, make multiple directory based on taskId

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

- [ ] Any interfaces changed?

- [ ] Any backward compatibility impacted?

- [ ] Document update required?

- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akashrn5/incubator-carbondata lucene

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2113.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2113

----
commit 4453e9218d2997923fa1ab3d60e4cac74cc14dba
Author: akashrn5 <akashnilugal@...>
Date: 2018-03-29T14:29:36Z

load issue in lucene datamap, make multiple directory based on taskId

----

---

qiuchenjian-2

[GitHub] carbondata issue #2113: [WIP][LUCENE_DATAMAP]load issue in lucene datamap, m...

Github user akashrn5 commented on the issue:

https://github.com/apache/carbondata/pull/2113

When the loading is triggered after creation of table and datamap , it failed with lock exception.
Analysis:
The index write location for the lucene is same, and to IndexWriter will take a lock file called write.lock in write location while writing the index files. In carbon loading the writer tasks are launched parallel and those many writers are opened,
Since the write.lock file is acquired by one writer, all other tasks will fail and dataloading will fail.

Solution
so in this PR, the write path can be configured on the fly, with task id and timestamp mapping.

Handling in reader part:
In query side, there was a single indexSearch object which does search on index file present in a single location.
Now with thsi dataload fix, there will be multiple directories which will have index files, and cannot give a single path to search object, as serch object is one.

Probable Solutions:
1. create those many number of search objects, as number of index file directories created in data load. and finally return all the final Blocklets.
2. you can have one searcher in one executor to search the local folder, this part im working how to do

---

qiuchenjian-2