name | about | labels |
---|---|---|
Bug Report | Use this template for reporting a bug | kind/bug |
[pvt_v2_b0]训练精度不提升
Ascend
/GPU
/CPU
) / 硬件环境:Please delete the backend not involved / 请删除不涉及的后端:
/device ascend
Software Environment / 软件环境 (Mandatory / 必填):
-- MindSpore version (e.g., 1.7.0.Bxxx) :master_20240426230938_a87635b6f65bc6225173fec53883351ebf449b66
-- Python version (e.g., Python 3.7.5) :Python 3.7.6
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):4.19.90-vhulk2211.3.0.h1543.eulerosv2r10.aarch64
-- GCC/Compiler version (if compiled from source): gcc version 7.3.0 (GCC)
run包Milan_C17/20240414
mindspore包master_20240426230938_a87635b6f65bc6225173fec53883351ebf449b66
Excute Mode / 执行模式 (Mandatory / 必填)(PyNative
/Graph
):
Please delete the mode not involved / 请删除不涉及的模式:
/mode graph
test_ms_lab_pvt_v2_b0_acc3_ascend_train_8p_0008
1.get code from solution_test
2.cd solution_test/cases/02network/00cv/pvt_v2_b0/train/
3.pytest -s test_ms_lab_pvt_v2_b0_acc3_ascend_train_8p_0008.py
4.查看过训练精度是否正常
[pvt_v2_b0]训练精度正常提升
2024-04-27 05:10:13] mindcv.utils.callbacks INFO - Epoch: [34/34], batch: [1251/1251], loss: 9.847733, lr: 0.000996, time: 389.866129s
[2024-04-27 05:10:23] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.1000%, Top_5_Accuracy: 0.5840%, time: 10.636981s
[2024-04-27 05:10:25] mindcv.utils.callbacks INFO - Saving model to ./ckpt/pvt_v2_b0-34_1251.ckpt
[2024-04-27 05:10:27] mindcv.utils.callbacks INFO - Total time since last epoch: 404.670255(train: 389.894190, val: 10.636981)s, ETA: 0.000000s
[2024-04-27 05:10:27] mindcv.utils.callbacks INFO - --------------------------------------------------------------------------------
[INFO] RUNTIME(66342,python):2024-04-27-05:13:32.464.990 [engine.cc:1709] 71397 ReportTimeoutProc: report timeout! streamId=1, taskId=357, execId=355, pendingNum=9, reportCount=427, parseTaskCount=427, msec=7991049, curSec=8175284
[INFO] RUNTIME(66342,python):2024-04-27-05:16:36.785.136 [engine.cc:1709] 71397 ReportTimeoutProc: report timeout! streamId=1, taskId=357, execId=355, pendingNum=9, reportCount=427, parseTaskCount=427, msec=7991049, curSec=8359604
[2024-04-27 05:17:30] mindcv.utils.callbacks INFO - Epoch: [35/34], batch: [1251/1251], loss: 10.055557, lr: 0.000996, time: 422.013266s
[2024-04-27 05:17:40] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.1000%, Top_5_Accuracy: 0.5840%, time: 10.416377s
[2024-04-27 05:17:41] mindcv.utils.callbacks INFO - Saving model to ./ckpt/pvt_v2_b0-35_1251.ckpt
[2024-04-27 05:17:44] mindcv.utils.callbacks INFO - Total time since last epoch: 436.063551(train: 422.062537, val: 10.416377)s, ETA: -436.063551s
[2024-04-27 05:17:44] mindcv.utils.callbacks INFO - --------------------------------------------------------------------------------
[INFO] RUNTIME(66342,python):2024-04-27-05:20:48.684.866 [engine.cc:1709] 71397 ReportTimeoutProc: report timeout! streamId=1, taskId=531, execId=529, pendingNum=9, reportCount=587, parseTaskCount=587, msec=8427122, curSec=8611503
[INFO] RUNTIME(66342,python):2024-04-27-05:23:53.004.959 [engine.cc:1709] 71397 ReportTimeoutProc: report timeout! streamId=1, taskId=531, execId=529, pendingNum=9, reportCount=587, parseTaskCount=587, msec=8427122, curSec=8795824
[2024-04-27 05:24:37] mindcv.utils.callbacks INFO - Epoch: [36/34], batch: [1251/1251], loss: 10.048895, lr: 0.000996, time: 413.378937s
[2024-04-27 05:24:48] mindcv.utils.callbacks INFO - Validation Top_1_Accuracy: 0.1000%, Top_5_Accuracy: 0.5840%, time: 11.029083s
[2024-04-27 05:24:50] mindcv.utils.callbacks INFO - Saving model to ./ckpt/pvt_v2_b0-36_1251.ckpt
[2024-04-27 05:24:52] mindcv.utils.callbacks INFO - Total time since last epoch: 428.123073(train: 413.405590, val: 11.029083)s, ETA: -856.246147s
[2024-04-27 05:24:52] mindcv.utils.callbacks INFO - --------------------------------------------------------------------------------
可能与这个问题单有关联
https://e.gitee.com/mind_spore/issues/list?issue=I9HVXV
上次精度正常上升版本,也就是上一次ci下发执行该用例
run包Milan_C17/20240414
mindspore包master_20240417142516_74e1f3ea86a988bceb71bd7f02b674af4c97c89a/
Please assign maintainer to check this issue.
请为此issue分配处理人。
@chentangyu
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。
感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:
问题同:I9HVXV
问题由 PR的变更引入 https://gitee.com/bantao1/mindspore_lkrelu/commit/4de4e4b3cc4daab013f099ff840bc8f7b5a24bba
目前已回退代码 PR:
!68729:div 910A GE模式问题处理
当前问题已修复。
master分支430 ci包还未出,用开发提供的pr包
910A回归通过
登录 后才可以发表评论