name | about | labels |
---|---|---|
Bug Report | Use this template for reporting a bug | kind/bug |
ascend环境,googlenet单卡训练,pynative模式下开启profiler,有大量告警重复日志
Ascend
/GPU
/CPU
) / 硬件环境:Please delete the backend not involved / 请删除不涉及的后端:
/device ascend
Software Environment / 软件环境 (Mandatory / 必填):
-- MindSpore version (e.g., 1.7.0.Bxxx) :
-- Python version (e.g., Python 3.7.5) :
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):
-- GCC/Compiler version (if compiled from source):
commit_id = '[sha1]:a87635b6,[branch]:(HEAD,origin/master,origin/HEAD,master)'
runpkg_version:Milan_C17/20240414
Excute Mode / 执行模式 (Mandatory / 必填)(PyNative
/Graph
):
Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative
test_ms_mi_profiler_pynative_googlenet_1p_cpu_abnormal_0001
test_ms_mi_profiler_pynative_yolov3_1p_0001
test_ms_mi_profiler_pynative_googlenet_1p_0001
取mindspore model_zoo中网络googlenet,
将训练脚本中的图模式参数设置为pynative,
然后开启profiler,在ascend上训练
用例执行流程:
source /home/miniconda3/bin/activate feature_39
export TRAIN_MODE=GRAPH_MODE
export DEVICE_TYPE=Ascend_Arm
export ENV_DEVICE=0
source solution_test/env_set.source -e ascend
cd solution_test/cases/03subject_test/02usability/perf_tuning/profiler_pynative
pytest -s test_ms_mi_profiler_pynative_googlenet_1p_cpu_abnormal_0001.py
正常训练,用例pass
重复日志:
ERROR 2024-04-27 11:38:04 - test_ms_mi_profiler_pynative_googlenet_1p_cpu_abnormal_0001 - base.py:check_warn_info:1069 - 以下是WARNING日志重复次数超过10的日志
WARNING日志重复次数 WARNING日志
128 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnCast_CastAiCore_Cast!
36 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnCat_ConcatD_ConcatD!
120 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnMul_MulAiCore_Mul!
228 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnRelu_Relu_Relu!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnThresholdBackward_ReluGrad_ReluGrad!
54 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Add30!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for AddN42!
44 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum49!
16 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum56!
24 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum60!
12 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum64!
12 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum66!
44 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum72!
12 mindspore/profiler/profiling.py:1444] Failed to get launch operator for ApplyMomentum96!
95 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingReduce5!
95 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingReduce5_MemSet!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingReduceGrad24!
96 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingUpdate6!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingUpdateGrad23!
114 mindspore/profiler/profiling.py:1444] Failed to get launch operator for BNTrainingUpdateGrad23_MemSet!
96 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Cast3!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2D8!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2D9!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropFilter25!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropFilter25_MemSet!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropFilter27!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropFilter27_MemSet!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropInput26!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Conv2DBackpropInput28!
16 mindspore/profiler/profiling.py:1444] Failed to get launch operator for MaxPool10!
64 mindspore/profiler/profiling.py:1444] Failed to get launch operator for Slice31!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_Cast_18!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_Cast_25!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_10!
16 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_11!
16 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_12!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_16!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_17!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_19!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_20!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_20_MemSet!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_21!
74 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_22!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_23!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_24!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_26!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_27!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_27_MemSet!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_28!
38 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_29!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_5!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_6!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_6_MemSet!
62 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_7!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_8!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_9!
32 mindspore/profiler/profiling.py:1444] Failed to get launch operator for trans_TransData_9_MemSet!
走给 臧庆香
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。
感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:
调整kbyk host数据保存频次,导致host 算子信息数据丢失的情况
调整kbyk 保存数据逻辑,从独立线程的方式改成设定阈值条件触发保存
Problem already been fixed in https://gitee.com/mindspore/mindspore/pulls/68929
https://gitee.com/mindspore/mindspore/pulls/68811
Self-testing is passed.
是否需要补充ST/UT:否
已有kbyk用例
回归版本:
commit_id = '[sha1]:af0f6ef7,[branch]:(HEAD,origin/master,origin/HEAD,master)'
runpkg_version:Milan_C17/20240414
回归步骤:参考issue复现步骤
基本功能:部分重复告警已修复,还有少量重复告警未修复
WARNING日志重复次数 WARNING日志
36 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnCat_ConcatD_ConcatD!
12 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnMm_TransData_TransData!
228 mindspore/profiler/profiling.py:1444] Failed to get launch operator for aclnnRelu_Relu_Relu!
测试结论:回归失败
动态图aclnn算子没走kbyk,无法采集此类型算子的host侧的下发耗时数据
取消动态图aclnn算子无法采集host耗时的告警,动态图host侧下发耗时统计CCB申请走需求处理
Problem already been fixed in https://gitee.com/mindspore/mindspore/pulls/69036
https://gitee.com/mindspore/mindspore/pulls/69034
Self-testing is passed.
CCB结论:profiling暂不支持动态图,后续规划需求开发。四野可以使用老的方式使用,本问题转需求处理
登录 后才可以发表评论