2.3K Star 8.1K Fork 4.3K

GVPMindSpore / mindspore

 / 详情

【ST】【MS】【2.3】ascend环境,googlenet单卡训练,pynative模式下开启profiler,有大量告警重复日志

TODO
Bug-Report
创建于  
2024-04-27 17:42
name about labels
Bug Report Use this template for reporting a bug kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

ascend环境,googlenet单卡训练,pynative模式下开启profiler,有大量告警重复日志

Environment / 环境信息 (Mandatory / 必填)

  • Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend

  • Software Environment / 软件环境 (Mandatory / 必填):
    -- MindSpore version (e.g., 1.7.0.Bxxx) :
    -- Python version (e.g., Python 3.7.5) :
    -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
    -- GCC/Compiler version (if compiled from source):
    commit_id = '[sha1]:a87635b6,[branch]:(HEAD,origin/master,origin/HEAD,master)'
    runpkg_version:Milan_C17/20240414

  • Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative

Related testcase / 关联用例 (Mandatory / 必填)

test_ms_mi_profiler_pynative_googlenet_1p_cpu_abnormal_0001
test_ms_mi_profiler_pynative_yolov3_1p_0001
test_ms_mi_profiler_pynative_googlenet_1p_0001

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

取mindspore model_zoo中网络googlenet,
将训练脚本中的图模式参数设置为pynative,
然后开启profiler,在ascend上训练

用例执行流程:
source /home/miniconda3/bin/activate feature_39
export TRAIN_MODE=GRAPH_MODE
export DEVICE_TYPE=Ascend_Arm
export ENV_DEVICE=0
source solution_test/env_set.source -e ascend

cd solution_test/cases/03subject_test/02usability/perf_tuning/profiler_pynative
pytest -s test_ms_mi_profiler_pynative_googlenet_1p_cpu_abnormal_0001.py

Describe the expected behavior / 预期结果 (Mandatory / 必填)

正常训练,用例pass

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

重复日志:
ERROR 2024-04-27 11:38:04 - test_ms_mi_profiler_pynative_googlenet_1p_cpu_abnormal_0001 - base.py:check_warn_info:1069 - 以下是WARNING日志重复次数超过10的日志
WARNING日志重复次数 WARNING日志
128 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnCast_CastAiCore_Cast!
36 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnCat_ConcatD_ConcatD!
120 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnMul_MulAiCore_Mul!
228 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnRelu_Relu_Relu!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnThresholdBackward_ReluGrad_ReluGrad!
54 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Add30!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for AddN42!
44 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum49!
16 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum56!
24 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum60!
12 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum64!
12 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum66!
44 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum72!
12 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum96!
95 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingReduce5!
95 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingReduce5_MemSet!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingReduceGrad24!
96 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingUpdate6!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingUpdateGrad23!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingUpdateGrad23_MemSet!
96 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Cast3!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2D8!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2D9!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropFilter25!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropFilter25_MemSet!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropFilter27!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropFilter27_MemSet!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropInput26!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropInput28!
16 mindspore/profiler/profiling.py:1444] Failed to get launch operator for MaxPool10!
64 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Slice31!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_Cast_18!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_Cast_25!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_10!
16 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_11!
16 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_12!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_16!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_17!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_19!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_20!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_20_MemSet!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_21!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_22!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_23!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_24!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_26!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_27!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_27_MemSet!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_28!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_29!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_5!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_6!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_6_MemSet!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_7!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_8!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_9!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_9_MemSet!

Special notes for this issue/备注 (Optional / 选填)

走给 臧庆香

评论 (6)

wenli 创建了Bug-Report
wenli 添加了
 
kind/bug
标签
wenli 添加了
 
v2.3.0.rc2
标签
wenli 添加了
 
sig/visualization
标签
wenli 添加了
 
attr/function
标签
wenli 添加了
 
stage/func-debug
标签
wenli 添加协作者zangqx
展开全部操作日志

Please assign maintainer to check this issue.
请为此issue分配处理人。
@wenli

感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:

  1. 如果您刚刚接触MindSpore,或许您可以在教程找到答案
  2. 如果您是资深Pytorch用户,您或许需要:
  1. 如果您遇到动态图问题,可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
  2. 模型精度调优问题可参考官网调优指南
  3. 如果您反馈的是框架BUG,请确认您在ISSUE中提供了MindSpore版本、使用的后端类型(CPU、GPU、Ascend)、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
  4. 如果您已经定位出问题根因,欢迎提交PR参与MindSpore开源社区,我们会尽快review
wenli 修改了描述
zangqx 负责人mudongrui 修改为zangqx
zangqx 取消协作者zangqx
zangqx 添加协作者mudongrui
linzhengshu 添加了
 
device/ascend
标签

Appearance & Root Cause

调整kbyk host数据保存频次,导致host 算子信息数据丢失的情况

Fix Solution

调整kbyk 保存数据逻辑,从独立线程的方式改成设定阈值条件触发保存

Fix Description & Test Suggestion

Problem already been fixed in https://gitee.com/mindspore/mindspore/pulls/68929
https://gitee.com/mindspore/mindspore/pulls/68811

Self-test Report

Self-testing is passed.

Self-test Report & DT Review

是否需要补充ST/UT:否
已有kbyk用例

i-robot 添加了
 
gitee
标签
i-robot 添加了
 
gitee
标签
zangqx 添加协作者zangqx
zangqx 负责人zangqx 修改为wenli
zangqx 任务状态TODO 修改为VALIDATION
zangqx 添加了
 
ctl/componenttest
标签
zangqx 添加了
 
rca/codelogic
标签
zangqx 添加了
 
rct/newfeature
标签
wenli 添加协作者wenli
wenli 负责人wenli 修改为zangqx
wenli 取消协作者zangqx

回归版本:
commit_id = '[sha1]:af0f6ef7,[branch]:(HEAD,origin/master,origin/HEAD,master)'
runpkg_version:Milan_C17/20240414
回归步骤:参考issue复现步骤
基本功能:部分重复告警已修复,还有少量重复告警未修复
WARNING日志重复次数 WARNING日志
36 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnCat_ConcatD_ConcatD!
12 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnMm_TransData_TransData!
228 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnRelu_Relu_Relu!

测试结论:回归失败

wenli 任务状态VALIDATION 修改为TODO
zangqx 添加了
 
待CCB
标签

Appearance & Root Cause

动态图aclnn算子没走kbyk,无法采集此类型算子的host侧的下发耗时数据

Fix Solution

取消动态图aclnn算子无法采集host耗时的告警,动态图host侧下发耗时统计CCB申请走需求处理

Fix Description & Test Suggestion

Problem already been fixed in https://gitee.com/mindspore/mindspore/pulls/69036
https://gitee.com/mindspore/mindspore/pulls/69034

Self-test Report

Self-testing is passed.

CCB结论:profiling暂不支持动态图,后续规划需求开发。四野可以使用老的方式使用,本问题转需求处理

fangwenyi 移除了
 
待CCB
标签
fangwenyi 移除了
 
待CCB
标签
fangwenyi 添加了
 
ccb/rfc
标签

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(5)
Python
1
https://gitee.com/mindspore/mindspore.git
git@gitee.com:mindspore/mindspore.git
mindspore
mindspore
mindspore

搜索帮助