2.3K Star 8.1K Fork 4.3K

GVPMindSpore / mindspore

 / 详情

[ST][MS][2.3]GPU环境,pynative模式下,动态shape场景使用ms_function,报错ValueError: For primitive[Reshape], the accumulate of x_shape must be equal to out_shape

TODO
Bug-Report
创建于  
2024-04-27 18:01
name about labels
Bug Report Use this template for reporting a bug kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

ValueError: For primitive[Reshape], the accumulate of x_shape must be equal to out_shape, but got x_shape: [const vector]{32, 3}, and out_shape: [const vector]{[0]:{32}, [1]:{3}, [2]:{1}, [3]:{1}, [4]:{32}, [5]:{3}, [6]:{1}, [7]:{1}, [8]:{32}, [9]:{3}, [10]:{1}, [11]:{1}, [12]:{32}, [13]:{3}, [14]:{1}, [15]:{1}} 。
在只跑graph模式,或者去掉顶层的ms_function后,就没有问题

Environment / 环境信息 (Mandatory / 必填)

  • Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device GPU

  • Software Environment / 软件环境 (Mandatory / 必填):
    -- MindSpore version (e.g., 1.7.0.Bxxx) :Mindspore 2.3 B210
    -- Python version (e.g., Python 3.7.5) :python 3.9
    -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
    -- GCC/Compiler version (if compiled from source):

  • Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative

Related testcase / 关联用例 (Mandatory / 必填)

test_ms_dynamic_shape_msfunc_multiplecell_0001.py

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

  1. 获取solution test仓
  2. export TRAIN_MODE=PYNATIVE_MODE
    source ~/workspace/TDT_deployment/solution_test/env_set.source -e CUDA11.6
    cd solution_test/cases/03subject_test/02usability/model_develop/dynamic_shape/
  3. pytest -s test_ms_dynamic_shape_msfunc_multiplecell_0001.py

Describe the expected behavior / 预期结果 (Mandatory / 必填)

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

Traceback (most recent call last):
  File "train_dynamic_net.py", line 394, in <module>
    train_net_with_model()
  File "train_dynamic_net.py", line 233, in train_net_with_model
    sink_size=config.sink_size)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/train/model.py", line 1094, in train
    initial_epoch=initial_epoch)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/train/model.py", line 115, in wrapper
    func(self, *args, **kwargs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/train/model.py", line 644, in _train
    cb_params, sink_size, initial_epoch, valid_infos)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/train/model.py", line 728, in _train_dataset_sink_process
    outputs = train_network(*inputs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/nn/cell.py", line 698, in __call__
    raise err
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/nn/cell.py", line 694, in __call__
    output = self._run_construct(args, kwargs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
    output = self.construct(*cast_inputs, **kwargs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/train/dataset_helper.py", line 109, in construct
    return self.network(*outputs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/nn/cell.py", line 698, in __call__
    raise err
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/nn/cell.py", line 694, in __call__
    output = self._run_construct(args, kwargs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
    output = self.construct(*cast_inputs, **kwargs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/nn/wrap/cell_wrapper.py", line 418, in construct
    return self._no_sens_impl(*inputs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/nn/wrap/cell_wrapper.py", line 434, in _no_sens_impl
    grads = self.grad_no_sens(self.network, self.weights)(*inputs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/ops/composite/base.py", line 389, in after_grad
    return grad_(fn, weights)(*args, **kwargs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/common/api.py", line 131, in wrapper
    results = fn(*arg, **kwargs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/ops/composite/base.py", line 378, in after_grad
    out = _pynative_executor.grad(fn, grad_, weights, self.grad_position, *args, **kwargs)
  File "/home/miniconda3/envs/tt2/lib/python3.7/site-packages/mindspore/common/api.py", line 1298, in grad
    return self._executor.grad(grad, obj, weights, grad_position, *args, *(kwargs.values()))
ValueError: For primitive[Reshape], the accumulate of x_shape must be equal to out_shape, but got x_shape: [const vector]{32, 3}, and out_shape: [const vector]{[0]:{32}, [1]:{3}, [2]:{1}, [3]:{1}, [4]:{32}, [5]:{3}, [6]:{1}, [7]:{1}, [8]:{32}, [9]:{3}, [10]:{1}, [11]:{1}, [12]:{32}, [13]:{3}, [14]:{1}, [15]:{1}}

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/core/ops/ops_func_impl/reshape.cc:78 InferShape

Special notes for this issue/备注 (Optional / 选填)

评论 (2)

田桐 创建了Bug-Report
田桐 添加了
 
attr/function
标签
田桐 添加了
 
stage/func-debug
标签
田桐 添加了
 
kind/bug
标签
田桐 添加了
 
v2.3.0
标签
田桐 添加了
 
v2.3.0.rc2
标签
田桐 添加了
 
sig/pynative
标签
田桐 添加协作者田桐
展开全部操作日志

Please assign maintainer to check this issue.
请为此issue分配处理人。
@田桐

感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:

  1. 如果您刚刚接触MindSpore,或许您可以在教程找到答案
  2. 如果您是资深Pytorch用户,您或许需要:
  1. 如果您遇到动态图问题,可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
  2. 模型精度调优问题可参考官网调优指南
  3. 如果您反馈的是框架BUG,请确认您在ISSUE中提供了MindSpore版本、使用的后端类型(CPU、GPU、Ascend)、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
  4. 如果您已经定位出问题根因,欢迎提交PR参与MindSpore开源社区,我们会尽快review
caifubi 添加协作者caifubi
caifubi 负责人caifubi 修改为gaoyong10
caifubi 里程碑B-SIG-PYNATIVE 修改为B-SIG-Runtime

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(3)
6574868 jojohw 1584546516
Python
1
https://gitee.com/mindspore/mindspore.git
git@gitee.com:mindspore/mindspore.git
mindspore
mindspore
mindspore

搜索帮助