name | about | labels |
---|---|---|
Bug Report | Use this template for reporting a bug | kind/bug |
在启智平台上910a训练SDXL网络时,recompute_Default/BatchMatMul-op6207算子启动失败
Ascend
/GPU
/CPU
) / 硬件环境: AscendPlease delete the backend not involved / 请删除不涉及的后端:
/device ascend/GPU/CPU/kirin/等其他芯片
Software Environment / 软件环境 (Mandatory / 必填):
-- MindSpore version (e.g., 1.7.0.Bxxx) :mindspore2.2.0
-- Python version (e.g., Python 3.7.5) :python3.9
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):eulerOS2.8
-- GCC/Compiler version (if compiled from source):
Excute Mode / 执行模式 (Mandatory / 必填)(PyNative
/Graph
): PyNative
Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative
/mode graph
mindone/example/stable_diffusion_xl/train.py
算子启动成功
[WARNING] KERNEL(16170,ffffa3d51440,python):2024-04-27-14:32:35.815.416 [mindspore/ccsrc/transform/acl_ir/op_api_convert.h:75] GetOpApiLibHandler] Dlopen /home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/run_check/../lib/plugin/ascend/custom_aicpu_ops/op_api/lib/libcust_opapi.so failed!/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/run_check/../lib/plugin/ascend/custom_aicpu_ops/op_api/lib/libcust_opapi.so: cannot open shared object file: No such file or directory
/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/ops/operations/array_ops.py:376: RuntimeWarning: overflow encountered in cast
value = Tensor(np.array(x).astype(np_dst_type), dtype=dst_type)
[ERROR] KERNEL(16170,ffffa3d51440,python):2024-04-27-14:33:26.077.708 [mindspore/ccsrc/plugin/device/ascend/kernel/acl/acl_kernel_mod.cc:253] Launch] Kernel launch failed, msg: The pointer[block] is null.
W11001: Op [Range] does not hit the high-priority operator information library, which might result in compromised performance.
W11001: Op [Cast] does not hit the high-priority operator information library, which might result in compromised performance.
This exception is caused by framework's unexpected error. Please create an issue at https://gitee.com/mindspore/mindspore/issues to get help.
mindspore/ccsrc/transform/acl_ir/acl_allocator.cc:40 AllocFunc
[ERROR] DEVICE(16170,ffffa3d51440,python):2024-04-27-14:33:26.077.779 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_kernel_executor.cc:583] LaunchKernel] Launch kernel failed, kernel full name: recompute_Default/BatchMatMul-op6207
Traceback (most recent call last):
File "/tmp/code/mindone/examples/stable_diffusion_xl/train_lokr.py", line 717, in
train(args)
File "/tmp/code/mindone/examples/stable_diffusion_xl/train_lokr.py", line 352, in train
train_fn(
File "/tmp/code/mindone/examples/stable_diffusion_xl/train_lokr.py", line 391, in train_txt2img
loss, overflow = train_step_fn(image, *tokens)
File "/tmp/code/mindone/examples/stable_diffusion_xl/gm/models/diffusion.py", line 181, in train_step_pynative
loss, , overflow = grad_func(x, noised_input, sigmas, w, **cond)
File "/tmp/code/mindone/examples/stable_diffusion_xl/gm/models/diffusion.py", line 161, in jit_warpper
return grad_and_update_func(*args, **kwargs)
File "/tmp/code/mindone/examples/stable_diffusion_xl/gm/models/diffusion.py", line 147, in grad_and_update_func
loss, grads = grad_fn(x, noised_input, sigmas, w, concat, context, y)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/ops/composite/base.py", line 625, in after_grad
return grad(fn_, weights)(*args, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 121, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/ops/composite/base.py", line 600, in after_grad
res = self.pynative_forward_run(fn, grad, weights, args, kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/ops/composite/base.py", line 650, in _pynative_forward_run
outputs = fn(*args, **new_kwargs)
File "/tmp/code/mindone/examples/stable_diffusion_xl/gm/models/diffusion.py", line 132, in _forward_func
model_output = model(
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/tmp/code/mindnlp/mindnlp/peft/tuners/tuners_utils.py", line 113, in construct
return self.model.construct(*args, **kwargs)
File "/tmp/code/mindone/examples/stable_diffusion_xl/gm/modules/diffusionmodules/wrappers.py", line 23, in construct
return self.diffusion_model(x, timesteps=t, context=context, y=y, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/tmp/code/mindone/examples/stable_diffusion_xl/gm/modules/diffusionmodules/openaimodel.py", line 801, in construct
h = self.middle_block(h, emb, context)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/tmp/code/mindone/examples/stable_diffusion_xl/gm/modules/diffusionmodules/openaimodel.py", line 42, in construct
x = cell(x, context)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 705, in call
raise err
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 701, in call
output = self._run_construct(args, kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 482, in _run_construct
output = self.construct(*cast_inputs, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/ops/_tracefunc.py", line 235, in _trace_wrap
return pack_func(*args, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/ops/_tracefunc.py", line 84, in call
output = super().call(*args)
File "/tmp/code/mindnlp/mindnlp/injection.py", line 176, in _op_call
return old_op_call(self, *args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/ops/primitive.py", line 314, in call
return _run_op(self, self.name, args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/ops/primitive.py", line 913, in _run_op
stub = _pynative_executor.run_op_async(obj, op_name, args)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 1186, in run_op_async
return self._executor.run_op_async(*args)
RuntimeError: Launch kernel failed, name:recompute_Default/BatchMatMul-op6207
W11001: Op [Range] does not hit the high-priority operator information library, which might result in compromised performance.
W11001: Op [Cast] does not hit the high-priority operator information library, which might result in compromised performance.
mindspore/ccsrc/runtime/pynative/run_op_helper.cc:1052 LaunchKernels
Please assign maintainer to check this issue.
请为此issue分配处理人。
@fangwenyi @chengxiaoli @Shawny
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。
感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:
您好,mindone问题建议移步mindone的代码仓进行提问,如确认是MindSpore框架问题我们会进行内部跟踪,感谢您的支持
登录 后才可以发表评论