[GitHub] [carbondata] xubo245 opened a new pull request #3617: [CARBONDATA-3695] Integrating deep learning framework PyTorch

[GitBox]

xubo245 opened a new pull request #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617
 
 
   Apache CarbonData should provides python interface to support deep learning framework PyTorch to ready and write data from/to CarbonData
   
    ### Why is this PR needed?
   
   Nowadays AI model training is getting more and more popular. Currently many AI framework uses raw data files or row format data files for model training, it could not provide projection, filtering, and fast scan capability like in columnar store. So, if CarbonData supports AI framework, it can speed up model training by increase IO throughput, and provide more flexible training set selection ability to AI developers
   
    ### What changes were proposed in this PR?
   
   https://github.com/apache/carbondata/pull/3479 already provide Basic framework for it and integrated TensorFlow. This PR integrates PyTorch and provide new interface for it.
   
    ### Does this PR introduce any user interface change?
    - Yes
   
   ```
   def make_data_loader(reader, batch_size=1, collate_fn=decimal_friendly_collate):
     """
     Initializes a data loader object, with a default collate.
   
     Number of epochs is defined by the configuration of the reader argument.
   
     :param reader: PyCarbon Reader instance
     :param batch_size: the number of items to return per batch; factored into the len() of this reader
     :param collate_fn: an optional callable to merge a list of samples to form a mini-batch.
     """
   ```
   
    ### Is any new testcase added?
    - Yes
   
   pytorch_example_carbon_unified_api.py . and so on.
   
       
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-585279268
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/269/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-585308624
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1972/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

ajantha-bhat commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-586823841
 
 
   Good. Can we add one documentation "carbondata AI guide" that can explain which all AI engines we support and points to simple example code for each AI engine?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

xubo245 commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-587027349
 
 
   @ajantha-bhat added, please check readme.md in python folder

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

xubo245 commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-587027486
 
 
   @jackylk @ajantha-bhat Please review it again.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-587036932
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/323/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-587072065
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2025/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#discussion_r380417707
 
 

 ##########
 File path: python/pycarbon/tests/mnist/dataset_with_unischema/pytorch_example_carbon.py
 ##########
 @@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division, print_function
+
+import argparse
+
+import jnius_config
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from torchvision import transforms
+
+from pycarbon.reader import make_reader
+from pycarbon.tests.mnist.dataset_with_unischema import DEFAULT_MNIST_DATA_PATH
+from pycarbon.tests import DEFAULT_CARBONSDK_PATH
+
+from petastorm.pytorch import DataLoader
+from petastorm import TransformSpec
+
+
+class Net(nn.Module):
 
 Review comment:
   Is this file ported from petastorm? please mention it if yes

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#discussion_r380417707
 
 

 ##########
 File path: python/pycarbon/tests/mnist/dataset_with_unischema/pytorch_example_carbon.py
 ##########
 @@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division, print_function
+
+import argparse
+
+import jnius_config
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from torchvision import transforms
+
+from pycarbon.reader import make_reader
+from pycarbon.tests.mnist.dataset_with_unischema import DEFAULT_MNIST_DATA_PATH
+from pycarbon.tests import DEFAULT_CARBONSDK_PATH
+
+from petastorm.pytorch import DataLoader
+from petastorm import TransformSpec
+
+
+class Net(nn.Module):
 
 Review comment:
   Is this file ported from petastorm? Can we call it without copy code here, at least you should mention it

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#discussion_r380417707
 
 

 ##########
 File path: python/pycarbon/tests/mnist/dataset_with_unischema/pytorch_example_carbon.py
 ##########
 @@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division, print_function
+
+import argparse
+
+import jnius_config
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from torchvision import transforms
+
+from pycarbon.reader import make_reader
+from pycarbon.tests.mnist.dataset_with_unischema import DEFAULT_MNIST_DATA_PATH
+from pycarbon.tests import DEFAULT_CARBONSDK_PATH
+
+from petastorm.pytorch import DataLoader
+from petastorm import TransformSpec
+
+
+class Net(nn.Module):
 
 Review comment:
   Is this file ported from petastorm? Can we call it without copy code here

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

jackylk commented on a change in pull request #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#discussion_r380417707
 
 

 ##########
 File path: python/pycarbon/tests/mnist/dataset_with_unischema/pytorch_example_carbon.py
 ##########
 @@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division, print_function
+
+import argparse
+
+import jnius_config
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from torchvision import transforms
+
+from pycarbon.reader import make_reader
+from pycarbon.tests.mnist.dataset_with_unischema import DEFAULT_MNIST_DATA_PATH
+from pycarbon.tests import DEFAULT_CARBONSDK_PATH
+
+from petastorm.pytorch import DataLoader
+from petastorm import TransformSpec
+
+
+class Net(nn.Module):
 
 Review comment:
   Is this file ported from petastorm? Can you call it without copy code here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

xubo245 commented on a change in pull request #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#discussion_r380735867
 
 

 ##########
 File path: python/pycarbon/tests/mnist/dataset_with_unischema/pytorch_example_carbon.py
 ##########
 @@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division, print_function
+
+import argparse
+
+import jnius_config
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from torchvision import transforms
+
+from pycarbon.reader import make_reader
+from pycarbon.tests.mnist.dataset_with_unischema import DEFAULT_MNIST_DATA_PATH
+from pycarbon.tests import DEFAULT_CARBONSDK_PATH
+
+from petastorm.pytorch import DataLoader
+from petastorm import TransformSpec
+
+
+class Net(nn.Module):
 
 Review comment:
   It can not be call it in Pycarbon because the code is in examples folder and not in pypi source.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

xubo245 commented on a change in pull request #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#discussion_r380735867
 
 

 ##########
 File path: python/pycarbon/tests/mnist/dataset_with_unischema/pytorch_example_carbon.py
 ##########
 @@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division, print_function
+
+import argparse
+
+import jnius_config
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from torchvision import transforms
+
+from pycarbon.reader import make_reader
+from pycarbon.tests.mnist.dataset_with_unischema import DEFAULT_MNIST_DATA_PATH
+from pycarbon.tests import DEFAULT_CARBONSDK_PATH
+
+from petastorm.pytorch import DataLoader
+from petastorm import TransformSpec
+
+
+class Net(nn.Module):
 
 Review comment:
   It can not be call it in Pycarbon because the code is in examples folder and not in pypi source.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

xubo245 commented on a change in pull request #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#discussion_r380739276
 
 

 ##########
 File path: python/pycarbon/tests/mnist/dataset_with_unischema/pytorch_example_carbon.py
 ##########
 @@ -0,0 +1,176 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import division, print_function
+
+import argparse
+
+import jnius_config
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.optim as optim
+from torchvision import transforms
+
+from pycarbon.reader import make_reader
+from pycarbon.tests.mnist.dataset_with_unischema import DEFAULT_MNIST_DATA_PATH
+from pycarbon.tests import DEFAULT_CARBONSDK_PATH
+
+from petastorm.pytorch import DataLoader
+from petastorm import TransformSpec
+
+
+class Net(nn.Module):
 
 Review comment:
   Done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-587525554
 
 
   Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/344/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

CarbonDataQA1 commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-587568983
 
 
   Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2046/
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

ajantha-bhat commented on issue #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617#issuecomment-590176522
 
 
   LGTM

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services

[GitBox]

asfgit closed pull request #3617: [CARBONDATA-3695]  Integrating deep learning framework PyTorch
URL: https://github.com/apache/carbondata/pull/3617
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[hidden email]


With regards,
Apache Git Services